One of the things I’ve noticed over the years at my former employer, which seems to be typical, is that tests are often thrown together at the end of a learning project, and sometimes not even by the instructional designers but by the sponsors or SMEs, due to time pressure or sometimes an attitude of ‘how hard can it be?’
You know the kind of questions that can be served up to learners as a result:
- questions that only test recall (easier to write)
- questions where the right answer sticks out like a sore thumb
- too few questions, focusing only on topics it’s easy to write questions about
- cop-out questions like ‘which of the following statements are true’ or worse ‘not true’
- too many questions, when the writer was on a roll, or had drunk too much coffee
Some people I’ve worked with have found this job aid useful. I adapted it from the work of Dr Stan Trollip from whom I learned a lot about testing in the early days of my career. It’s a simple table where you decide on the overall shape of a test – how many questions for each subtopic, how many questions altogether, how many factual recall and how many application questions.
The last point deserves a bit of elaboration – why even have a column for factual recall? Cathy Moore raised this objection – shouldn’t every question be an apply question? Not necessarily a scenario but something that tests recall in a specific situation. And I agree, I’d be happy if there were not entries in the Recall column, except that:
- in some jobs it is necessary to do factual recall, immediately on the spot to give you credibility with customers. To test these it’s often more straightforward to do a lot of simple recall questions thus giving a lot of practice, than only a couple of well thought out contextual questions. You may need quantity of questions as much as quality
- the table can be used as a way to evaluate an existing test, to see whether it is too heavily weighted towards recall
So very simply you decide on an overall number of questions for the test, you distribute those among the topic areas on the basis of importance, and you then split that number between Recall and Apply.
The instructions are on page 2. I hope you find it useful. Let me know.