All of the Above - how to cheat Multiple Choice questions

Forget objectives, start with questions

I never liked the 'start with the learning objectives' advice on designing content, and preferred
to start with writing and polishing these objectives as test items. This seemed that much more
real, practical and relevant than forced 'Mager-like' objectives. Get the assessment right and
the rest seemed to follow. This, I suspect, is why students much prefer to get past test papers
to establish the real contents of a course.

Of course, writing good test items is far more difficult than many imagine, which is why
many tests and not really tests of understanding, merely tests of recall. An interesting way of
coming at this problem is to do some reverse engineering. Ask how students can cheat their
way through a test.

There's the usual crib notes. The best I heard was, remove the lable from a bottle of orange
juice, writing the crib notes on the back of the label.paste it back on, then drink orange to
reveal the notes. But let's assume that pure cheating is out. What are you left with?

Second-guessing the test designer

Many multiple choice questions are poorly written. What better way to expose these errors
than write a crib sheet for learners? So here goes with my 20 ways to cheat Multiple Choice
tests:

1. Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose
marks for unanswered easy questions.

2. If in doubt choose ‘C’, poor questions designers do not truly randomise the right options and have a
bias towards ‘C’. Next best is ‘B’.

3. If in doubt choose the ‘longest option’. Question designers often cannot make a right option any
shorter, but have complete freedom with wrong options.

4. Look for similarities in options and eliminate outliers (in bold) e.g. 4p-q, 2p+q, 4p+q, 3p+q.

5. Now note that there’s only one ‘-‘, which makes 4p+q more likely. Look for these internal patterns.
6. ‘All of the above’ is likely to be correct. For it to be correct the writer has to design options that were
all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the
probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.

7. Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to
have a bias, where right answers tend to be lower than the highest and higher than the lowest option.

8. For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false
against each option. And underline the word ‘not’ as it’s sometimes missed.

9. If there’s a typo or punctuation error, the option is likely to be wrong. Writers tend to proofread
correct answers only.

10. Look for grammatical agreement between the question and its options; ‘An.....’ and words starting
with vowels or agreement between subject, object or verb.

11. Go with your first impression. The more you read, the more you tend to read into the wrong
options.

12. If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.

13. Look for clues about answers from other questions. Designers often, unintentionally, put clues, even
answers, to questions in other questions.

14. If you’ve never heard of the answer, it’s likely to be made up and incorrect.

15. First cover the options and try to answer. Prevents being misled by clever wrong options.

16. If two options are opposites, one is likely to be correct. Designers first made up option is likely to be
the correct option’s opposite.

17. Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge
usually has more finite than absolute qualities.

18. Conversely, be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are
often too definite to be reasonably correct.

19. Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.
20. Eliminate obvious answer on 4 options then guess, don’t fail to answer. This reduces the odds
from ‘1 in 4’ to ‘1 in 3’. Far better than just guessing or not answering, depending on any penalty scores
for wrong answers.

None of the above

This crib sheet can be used by question designers to improve their tests. Good students put
themselves in the shoes of the test designer to improve their chance, so the more you know
about their techniques, the better designer you’ll be.

I still see binary option questions with ‘Try again’ logic, grammatical disagreement and
stupid options. Some time back in this blog, after they refused to respond when I emailed the
mistakes through, I had a go at BBC Bitesize’s science tests, as they were riddled with these
errors. 140 comments later, it still pops up on the home page when you go to BBC Bitesize
through Google.

Writing good multiple choice questions is not easy. What’s easy is simply extracting all the
nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test
understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested.

Why questions matter

Professor Dylan Wiliam, Deputy Director of the Institute of Education, Professor of
Educational Assessment gave a brilliant ALT talk (Seb Schmoller put me on to this) on what
calls ‘hinge questions’, questions that literally diagnose poor understanding. He explains how
one can use these questions as powerful verbal test items in a classroom, where it is difficult
to diagnose 30 kids quickly. This is a technique every teacher should learn.

The ball sitting on a table is not moving. It’s not moving because:

A. No forces are pushing or pulling on the ball.

B. Gravity is pulling down, but the table is in the way.

C. The table pushes up with the same force that gravity pulls down.

D. Gravity is holding it on to the table.

E. There’s a force inside the ball keeping it from rolling off the table.

This question not only catches common misconceptions, it diagnoses between those who
have understood the ‘physics’. C is correct.
What can we do to preserve the ozone layer?

A. Reduce the amount of carbon dioxide in the air

B. Reduce the greenhouse effect

C. Stop cutting down the rainforest

D. Properly dispose of air conditioners and fridges

Looks like the designer ran out of options and put the last item in to make up the numbers,
but it’s actually the right answer. Interestingly, this was one of dozens of mistakes in BBC
Bitesize,

Which of the following is NOT an effect of burning fossil fuels?

A. global warning

B. ozone depletion

C. acid rain

D. smog

E. fog (right according to BBC)

Ozone depletion is also correct as the result of CFCs which are completely artificial (they did
not exist in nature prior to synthesis by humans). They were used in air conditioning/cooling
units, as aerosol spray propellants prior to the 1980s, and in the cleaning processes of delicate
electronic equipment. They are not the result of burning fossil fuels.

Wiliam’s point about the imortance of classroom questioning, is that;

―The variability at teacher level is about four times the variability at school level. If you get
one of the best teachers, you will learn in six months what an average teacher will take a
year to teach you. If you get one of the worst teachers, that same learning will take you two
years. There’s a four-fold difference in the speed of learning created by the most and the
least effective teachers. And it’s not class size, it’s not between class grouping, it’s not within
class grouping – it’s the quality of the teacher.‖

This led him to determine what separates good from bad teachers.

―And actually, new teachers are actually pretty bad. You don’t really learn to teach at all
well until you’re six or seven years into the profession. And some recent data from Australia
shows that the amount of value added by teachers actually carries on increasing for about
twenty years.‖

And here’s a brilliant paragraph.

―The key concept here—the big trap—is that teachers do not create learning. That’s true
teachers do not create learning, and yet most teachers behave as if they do. Learners create
learning. Teachers create the conditions under which learning can take place. Our schools
don’t function like that, which is why somebody once joked that schools are places where
kids go to watch teachers work.‖

The solution, given the fact that reducing clss sizes is incredibly expensive, is to use
diagnistic 'hinge' questions. This accelerates the teacher's knowledge of the state of learning
of the learners and accelerates the learning.

Dylan William also co-authored, with professor Paul Blackgave, the brilliant Inside the Black
Box: Raising Standards through Classroom Assessment, another ‘should be compulsory’ text
for educators. This makes the obvious point that far too much teaching is simply 'chalk and
talk'. If you don't believe this walk into any school and you'll see it in practice. It's a
manifesto for more formative assessment.

Why questions matter

What matters? All of the above and more. Questions really do matter in learning.

Questions and curiosity

First, they stimulate curiosity. Almost all of my learning as an adult has this dynamic.
Something intigues me and I follow it up as I'm curious to find the answer. This is the great
joy of having the internet as a resource. It has made this type of inquiry and research possible.

Questions and diagnosis

Good qestions diagnose your strengths and weaknesses. You don't know what you don't
know and questions uncover the often uncomfortable truth that you know less than you
thought you know.

Questions and improvement

Questions and searching for answers are fundamental to the process of learning. Roger
Schank has been using this apporach in all sorts of contexts, and this truly structured Socratic
approach, works well when used by a skilled practitioner.

Questions and motivation

To create the conditions for learning, as opposed to just delivering content, questions are the
true stimulus.
Yet, despite these advantages, few have the real skills to either construct or deliver formative
or summative feedback at the level necessary for true learning. It's a real skill.