This type of question is bad for your assessment!

Snake headTests, exams and assessments are often done in a rush to meet a deadline.  “We’ve got the content now we’d better write a test.”  But they matter to employees. Maybe more than you realise.

 

One company I know links test passes automatically to system access – you don’t pass in the LMS, you’re locked out the system! Not good news for your boss on  Monday morning. In another, users take a pre-test, and only have to do the sections of the elearning that they fail. Good stuff, but the pass mark is 100% and each time you fail, you have to go through the relevant section before you can retry. Great if the questions are clear – but if you can’t figure out what the expected answer is, you’re doomed to a frustrating loop of section, test, section, test. Until you phone a friend.

In many cases, the problem in the test boiled down to one type of question – the multiple-correct:  a multiple-choice question where there are more than one correct choices.  I think there’s a strong case for banning them completely from tests, or at least restricting them to one particular use.

So what’s wrong with them?

1) the rag bag question

A golden rule of test writing is that each question should test only one piece of knowledge.  That goes out the window  with
‘Which of the following statements about xxx are true?”

To parody some I’ve seen (click the thumbnails to see the sample questions)
Which of the following statements about the United Nations are true? a) it is funded by contributions from member states b) during the Cold War the USSR accused it of favouring the USA c) In 1976 it set up the Joint Inspection Unit to deal with inefficiency d) its headquarters is in Brussels 3) it was founded in 1945 f) the secretary-general has the casting vote

I’ve seen lots of questions like this – the writer has been unable to write a question about each of these learning points individually, or thought they didn’t merit it.  Or maybe they just scanned the course for facts that hadn’t been included in any other questions. The result is a rag-bag  of unrelated facts.  The rag-bag question is the epitome of lazy test writing.  The only way to make this question worse – and I’ve seen it – is to change the stem to Which of these statements about the United Nations are not true?

2) the arguable distractor

In most LMSs a multiple-correct question is either right or wrong – you either have the right combination of boxes checked or you don’t. That means that the selection has to be clearly and unarguably correct.

You suspect one of your colleagues of passing on information to a competitior. You report it to your line manager. Which of these should your line manager do? a) take your concern seriously b) investigate your concern sensitively c) record your concerns in a blog d) not record your name as the person reporting

You could make a case for all of them, but the expected answer was only (1) and (2); a question similar to this caused endless frustration to users I knew.

3) the obvious distractor

Which three of the following are likely consequences of being caught taking bribes? a) the company's reputation will suffer b) fewer high quality candidates will apply for jobs with the company c) the company will face criminal penalties d) the individual will face criminal charges e) the company's business will grow f) the individual will receive a sales award

The role of a distractor is to distract!  The aim of  any test question is to sort out the people who confidently know the answer (the hoorays) from the people who aren’t sure (the mmmmms). (The people who definitely don’t know will get guess and fail according to the laws of chance.) The aim of the distractors is to tempt the middle group, the mmmmms, with something that looks plausible, but isn’t actually correct. If you’ve done your research you’ll base the distractors on what you know the common errors in the workplace are. If you haven’t, you might just fill up the question with anything that comes into your head!

This is equally true of multiple-choice questions but it plays out differently for the two question types.

In a good single-correct question the hoorays will spot the correct answer and the mmmms will weigh up all the options. Put something obvious in there and you’re just reducing the odds from 4:1 to 3:1.  Put two obvious distractors in and they might as well flip a coin.

For multiple-correct, it becomes a process of elimination. In the example above clearly the last two are wrong, so you’ve got to choose three out of four. Much easier.

4) how many choices?

If a multi-correct question appears after some single-correct when you take a test, you can safely assume more than one selection is required.

Which of these signs indicates a problem with the discombobulating widget? a) it emits a high pitched noise b) the red light flashes c) you can feel heat from the side of the machine d) all of the above

Then again the most common form of multiple-correct with a single answer is where the answer is ‘all of the above’.
This should never be a multiple-correct.

If the correct answer is D then the person who selects A, B and C is wrong – see the problem?
Some test writers state in the stem how many items are correct:  Which two of the following signs indicate a problem …?  This isn’t necessary and arguably makes the question easier, but at least it removes some possible ambiguity.

(Some would argue it’s valid to have only one correct answer, and strictly I guess it is, but by virtue of a different question type appearing in front of them, the user knows it’s a different type of question. I’d say that after a series of single-correct, a question like this
Which of the following might indicate a breach of accessibility guidelines?  where only one choice is correct is a trick question, and to be avoided.)

5) visual difference

In HTML based courses there’s no doubt about whether you’re in a single-correct question or multi-correct – single-correct has radio buttons and multi-correct has checkboxes. These are conventions we’re all used to. The problem comes with authoring systems like Storyline where choices may be placed on buttons. I’ve seen one test done in Storyline where the only visual difference when you moved from a single-correct to a multi-correct was an addition to the prompt: “Select all that apply” – easy to miss that.

So in a test where people’s jobs may be on the line I think tests have to be highly armoured against criticism that they aren’t fair, that choices are ambiguous or that it’s not clear what to do. And questions have to be of the highest quality.  That’s why I would always argue against multi-correct questions for such tests.
Except …

When should you use them?

Only when they are testing selection from a list of items or actions where there is no possible ambiguity about whether something is ‘in’ or ‘out’.
For example, Which of these documents are required as proof of identity?  Which of these countries are signatory to the agreement? Which of the following download sites are legal?

In each case you’d have a list of related items (point 1 above), that are right or wrong beyond argument or judgement (point 2), that don’t include obvious fillers (point 3) and where the instructions and presentation are unambiguous (points 4 and 5).

Is this how you do tests?  Please agree, disagree or challenge -I’d like to know!

Leave a Comment