Tests, exams and assessments are often done in a rush to meet a deadline. “We’ve got the content now we’d better write a test.” But they matter to employees. Maybe more than you realise.
One company I know links test passes automatically to system access – you don’t pass in the LMS, you’re locked out the system! Not good news for your boss on Monday morning. In another, users take a pre-test, and only have to do the sections of the elearning that they fail. Good stuff, but the pass mark is 100% and each time you fail, you have to go through the relevant section before you can retry. Great if the questions are clear – but if you can’t figure out what the expected answer is, you’re doomed to a frustrating loop of section, test, section, test. Until you phone a friend.
In many cases, the problem in the test boiled down to one type of question – the multiple-correct: a multiple-choice question where there are more than one correct choices. I think there’s a strong case for banning them completely from tests, or at least restricting them to one particular use.
So what’s wrong with them?
1) the rag bag question
A golden rule of test writing is that each question should test only one piece of knowledge. That goes out the window with
‘Which of the following statements about xxx are true?”
To parody some I’ve seen (click the thumbnails to see the sample questions)
I’ve seen lots of questions like this – the writer has been unable to write a question about each of these learning points individually, or thought they didn’t merit it. Or maybe they just scanned the course for facts that hadn’t been included in any other questions. The result is a rag-bag of unrelated facts. The rag-bag question is the epitome of lazy test writing. The only way to make this question worse – and I’ve seen it – is to change the stem to Which of these statements about the United Nations are not true?
2) the arguable distractor
In most LMSs a multiple-correct question is either right or wrong – you either have the right combination of boxes checked or you don’t. That means that the selection has to be clearly and unarguably correct.
You could make a case for all of them, but the expected answer was only (1) and (2); a question similar to this caused endless frustration to users I knew.
3) the obvious distractor
The role of a distractor is to distract! The aim of any test question is to sort out the people who confidently know the answer (the hoorays) from the people who aren’t sure (the mmmmms). (The people who definitely don’t know will get guess and fail according to the laws of chance.) The aim of the distractors is to tempt the middle group, the mmmmms, with something that looks plausible, but isn’t actually correct. If you’ve done your research you’ll base the distractors on what you know the common errors in the workplace are. If you haven’t, you might just fill up the question with anything that comes into your head!
This is equally true of multiple-choice questions but it plays out differently for the two question types.
In a good single-correct question the hoorays will spot the correct answer and the mmmms will weigh up all the options. Put something obvious in there and you’re just reducing the odds from 4:1 to 3:1. Put two obvious distractors in and they might as well flip a coin.
For multiple-correct, it becomes a process of elimination. In the example above clearly the last two are wrong, so you’ve got to choose three out of four. Much easier.
4) how many choices?
If a multi-correct question appears after some single-correct when you take a test, you can safely assume more than one selection is required.
Then again the most common form of multiple-correct with a single answer is where the answer is ‘all of the above’.
This should never be a multiple-correct.
If the correct answer is D then the person who selects A, B and C is wrong – see the problem?
Some test writers state in the stem how many items are correct: Which two of the following signs indicate a problem …? This isn’t necessary and arguably makes the question easier, but at least it removes some possible ambiguity.
(Some would argue it’s valid to have only one correct answer, and strictly I guess it is, but by virtue of a different question type appearing in front of them, the user knows it’s a different type of question. I’d say that after a series of single-correct, a question like this
Which of the following might indicate a breach of accessibility guidelines? where only one choice is correct is a trick question, and to be avoided.)
5) visual difference
In HTML based courses there’s no doubt about whether you’re in a single-correct question or multi-correct – single-correct has radio buttons and multi-correct has checkboxes. These are conventions we’re all used to. The problem comes with authoring systems like Storyline where choices may be placed on buttons. I’ve seen one test done in Storyline where the only visual difference when you moved from a single-correct to a multi-correct was an addition to the prompt: “Select all that apply” – easy to miss that.
So in a test where people’s jobs may be on the line I think tests have to be highly armoured against criticism that they aren’t fair, that choices are ambiguous or that it’s not clear what to do. And questions have to be of the highest quality. That’s why I would always argue against multi-correct questions for such tests.
Except …
When should you use them?
Only when they are testing selection from a list of items or actions where there is no possible ambiguity about whether something is ‘in’ or ‘out’.
For example, Which of these documents are required as proof of identity? Which of these countries are signatory to the agreement? Which of the following download sites are legal?
In each case you’d have a list of related items (point 1 above), that are right or wrong beyond argument or judgement (point 2), that don’t include obvious fillers (point 3) and where the instructions and presentation are unambiguous (points 4 and 5).