Compliance: it’s got to be perfect

Smiling mushrooms being tutored by a flower.Another post about compliance training. It’s not like me to think about it twice in a month.

Andrew Jacobs put up his list of 50 big ideas to change L and D, which generated some interesting comments as different readers picked on or other idea to talk about. My attention was drawn to

26. Don’t set benchmark tests that reward 15% error rates with a pass

We’ve been a journey of learning since last year when a team was convened to regulate the flood of courses being passed off as ‘mandatory training’ for all employees and bring some common sense to it. A question was raised about pass marks: why, if this knowledge is so important to the organisation, would you accept less than 100%?

Recognising that much of this knowledge-based stuff was repeated year after year and many employees would already know it, we worked with our suppliers to create pre-testing for each module whereby the learner only had to view those parts of the course that related to questions they’d failed; the test would then be locked until they had looked at (or clicked through) those sections and they could then take only those sections of the test again and try to pass.

If you must have mandatory knowledge testing this seemed like a fairly enlightened approach. And, with some exceptions, it was popular with employees. The exceptions, however, were telling and here’s what we learned from it:

  • you have to remind and really promote the fact that you can take the test first and pretest yourself out of sections you already know. We kept hearing ‘ I didn’t know you could do that!’: people were simply so used to blah blah blah blah, test
  • the questions have to be much better written than they ever were before. Many SMEs wrote long multiple-correct questions where perhaps three out of five answers were correct. The supplier didn’t challenge this or, if they did, were reminded who was boss. Often the questions were ambiguous and confusing so the learner would fail the question, had go through the section again, but by the time they came back to that question couldn’t remember which three out of the five choices they’d picked before. You can tell people they have to go through a section again but most times they’ll just click through without reading in order to unlock the test again. We knew of email chains where people were asking ‘what did you put for that question about …?’ but nobody was fooled that this was resulting in any learning, it was just gameplaying
  • when you switch mix single-correct and multiple-correct multiple choices in one test, labelling matters. The simple switch from ‘Select four answers and select Submit‘ to ‘Select FOUR answers and select Submit‘ was a usability lesson we learned early on.

While these courses got off to a rocky start, the problems with the courses were compounded with problems introducing a new LMS and completely unforeseeable network and browser issues. When these settled down the model for mandatory course was generally approved of by most learners, with the worst noise attributable to badly-written questions. This we’ve tried to address by insisting that all questions are single-correct and asking the supplier, who should know better, to strongly challenge poor questions from SMEs, who shouldn’t.

It seems that long after sponsors and subject expects have accepted that maybe there is a skill to creating interactive learning, they retain an attitude to test-writing that says ‘multiple choice questions – how hard can it be?’ I like to remind them of the guy who trained me in test writing way back, Stan Trollip, then of the University of North Dakota, who did test writing consultancy for the airline industry on the basis of 1.5 days per question!
PS scroll down the ’50 things’ page to Helen Blundell‘s comment – it’s priceless!

Leave a Comment