Writing Multiple-Choice Questions that Demand Critical

Thinking
Adapted from: http://darkwing.uoregon.edu/~tep/assessment/mc4critthink.html I. Recalling
memorized information.

Practical Suggestions for Writing Exams Techniques for Creating Questions Bibliography

Important Considerations What role should testing play in the learning process? How can tests
create a real dialogue between ourselves and our students about what students do and do not
understand? How can we avoid using tests to simply punish or reward cramming?

Some suggestions:

Use frequent, small quizes and tests rather than monolithic once-or-twice per-term exams.
Give students instant feedback on their performance (for example, putting the correct answers up
on an overhead after all the tests are turned in.)

Consider allowing students to take quizzes first as individuals and then the same quiz again in
groups.
Multiple-choice questions are easiest to write when there is a definitively right or wrong answer.
Multiple-choice testing of more interpretive material should always include an appeal
mechanism in which students can and must make a written, evidence-supported case for their
answer.

II. Bloom's Taxonomy of Cognitive Levels Knowledge

1. Recalling memorized information.

May involve remembering a wide range of material from specific facts to complete theories, but
all that is required is the bringing to mind of the appropriate information. Represents the lowest
level of learning outcomes in the cognitive domain.

Learning objectives at this level: know common terms, know specific facts, know methods and
procedures, know basic concepts, know principles.
Question verbs: Define, list, state, identify, label, name, who? when? where? what?

2. Comprehension
The ability to grasp the meaning of material. Translating material from one form to

1
another (words to numbers), interpreting material (explaining or summarizing), estimating future
trends (predicting consequences or effects). Goes one step beyond the simple remembering of
material, and represent the lowest level of understanding.

Learning objectives at this level: understand facts and principles, interpret verbal material,
interpret charts and graphs, translate verbal material to mathematical formulae, estimate the
future consequences implied in data, justify methods and procedures. Question verbs: Explain,
predict, interpret, infer, summarize, convert, translate, give example, account for, paraphrase x?

3. Application
The ability to use learned material in new and concrete situations. Applying rules, methods,
concepts, principles, laws, and theories.

Learning outcomes in this area require a higher level of understanding than those under
comprehension.

Learning objectives at this level: apply concepts and principles to new situations, apply laws
and theories to practical situations, solve mathematical problems, construct graphs and charts,
demonstrate the correct usage of a method or procedure.
Question verbs: How could x be used to y? How would you show, make use of, modify,
demonstrate, solve, or apply x to conditions y?

4. Analysis

The ability to break down material into its component parts. Identifying parts, analysis of
relationships between parts, recognition of the organizational principles involved. Learning
outcomes here represent a higher intellectual level than comprehension and application because
they require an understanding of both the content and the structural form of the material.

Learning objectives at this level: recognize unstated assumptions, recognizes logical fallacies
in reasoning, distinguish between facts and inferences, evaluate the relevancy of data, analyze
the organizational structure of a work (art, music, writing).
Question verbs: Differentiate, compare / contrast, distinguish x from y, how does x affect or
relate to y? why? how? What piece of x is missing / needed?

5. Synthesis

(by definition, synthesis cannot be assessed with multiple-choice questions.

It appears here to complete Bloom's taxonomy.) The ability to put parts together to form a new
whole.

relate to y? why? how? What piece of x is missing / needed?

This may involve the production of a unique communication (theme or speech), a plan

2
of operations (research proposal), or a set of abstract relations (scheme for classifying
information).

Learning outcomes in this area stress creative behaviors, with major emphasis on the
formulation of new patterns or structure.

Learning objectives at this level: write a well organized paper, give a well organized speech,
write a creative short story (or poem or music), propose a plan for an experiment, integrate
learning from different areas into a plan for solving a problem, formulate a new scheme for
classifying objects (or events, or ideas).

Question verbs: Design, construct, develop, formulate, imagine, create, change, write a short
story and label the following elements:

6. Evaluation
The ability to judge the value of material (statement, novel, poem, research report) for a given
purpose. The judgments are to be based on definite criteria, which may be internal (organization)
or external (relevance to the purpose). The student may determine the criteria or be given them.
Learning outcomes in this area are highest in the cognitive hierarchy because they contain
elements of all the other categories, plus conscious value judgments based on clearly defined
criteria.

Learning objectives at this level: judge the logical consistency of written material, judge the
adequacy with which conclusions are supported by data, judge the value of a work (art, music,
writing) by the use of internal criteria, judge the value of a work (art, music, writing) by use of
external standards of excellence.

Question verbs: Justify, appraise, evaluate, judge x according to given criteria. Which option
would be better/preferable to party y?

III. Practical Suggestions for Writing Multiple-Choice Questions

General Suggestions

1) Do not write the test in one day. Spread the work out over time. Questions demanding high-
level thinking take longer to craft-professional item writers often write only 3 or 4 per day. Write
one or two questions after each class, so it becomes a simple matter of assembling them into an
exam. Some teachers keep a rubber-banded stack of note cards in their desk for this purpose.

2) If students are to hand-write the letters of their chosen answers, ask them to use CAPITAL
LETTERS. The handwritten, lower-case letters "a" and "d" and "c" and "e" can be difficult to
distinguish when scoring.

Writing the Stem

1) Phrase stems as clearly as possible-confusing questions can generate wrong answers
from students who do understand the material.
For example, a confusing stem like: "According to Tuckman's model, groups develop through
several stages over time. Furthermore, it contradicts Poole's activity-track model which has
groups switching among several different linear sequences. Which of the following is not one of
the stages identified in Tuckman's model?" could be cleaned up to read: "Tuckman's model of
group development includes: [Select all that apply]"

2) Avoid extra language in the stem.

Some think extraneous details make a question more complex. However, they most often just
add to the students' reading time. This reduces the number of questions you can put on a test,
therefore reducing the reliability of the test. For example, in the Tuckman question above, the
information on Poole's model had nothing to do with the information sought by the question.

3) Include any language in the stem that you would have to repeat in each answer option.
For example, a stem such as "Biology is defined as the scientific study of:" keeps you from
having to repeat "is the scientific study of" at the beginning of each option.

Answer Options

1) Avoid lifting phrases directly from text or lecture. This becomes a simple recall activity for
the student. Use new language as frequently as possible.
2) Most literature recommends writing the correct answer before writing the distracters. This
makes sure you pay enough attention to formulating the one clearly correct answer.

3) Answer options should be about the same length and parallel in grammatical structure.
Too much detail or different grammatical structure can give the answer away.

For example, the specificity and grammatical structure of the first option here are dead give-
aways:

The term "side effect" of a drug:

a) refers to any action of a drug in the body other than the one the doctor wanted to drug to have.
b) is the chain effect of a drug.
c) additionally benefits the drug.

4) Limit the number of answer options. Research shows that three-choice items are about as
effective as four-choice items. Four choice items are the most popular, and never give more than
five alternatives.

5) Distracters must be incorrect, but plausible. If you can, include among the distracters
options that contain common errors. Students will then be motivated to listen to your
explanations of why those options are incorrect.
6) To make distracters more plausible, use words that should be familiar to students. 7) If a
recognizable key word appears in the correct answer, it should appear in some or

all of the distracters as well. Don't let a verbal clue decrease the accuracy of your exam.

For example, someone with no biology background would not have to think very hard to make a
correct guess on this question:

Every organism is made of cells and every cell comes from another cell. This is the: a) Relativity
Theory
b) Evolution Theory
c) Heat Theory

d) Cell Theory

8) Help students see crucial words in the question.

For example: "Which of the following is NOT an explicit norm?" Likewise, when you ask a
similarly-worded question about two different things, always highlight the difference between
the questions.

9) It is often difficult to come up with 3 or 4 plausible distracters, and teachers will sometimes
add some that are not plausible, or even humorous. Be careful.

If it is too easy to eliminate one or two options, then the question loses much of its measurement
value. If energy or time is limited and you must come up with one more distracter, consider
either offering a true statement that does not answer the question and/or a jargon-ridden option
that is meaningless to someone who understands the concept.

10) Use Rarely:

Extreme words like "all," "always" and "never" (generally a wrong answer). Vague words or
phrases like "usually," "typically" and "may be" (generally a correct answer). "All of the above"
- eliminating one distracter immediately eliminates this, too. "None of the above" - use only
when the correct answer can be absolutely correct, such as in math, grammar, historical dates,
geography, etc.. Do not use with negatively-stated stems, as the resulting double-negative is
confusing. Studies do show that using "None of the above" does make a question more difficult,
and is a better choice when the

alternative is a weak distracter.

IV. Some Techniques for Writing Multiple-Choice Items that Demand

Critical Thinking
1) Premise - Consequence

Students must identify the correct outcome of a given circumstance.

Example: If nominal gross national product (GNP) increases at a rate of 10% per year and the
GNP deflator increases at 8% per year, then real GNP:
a) Remains constant.
b) Rises by 10%.

c) Falls by 8%. d) Rises by 2%.

Note: To increase the difficulty, provide more than one premise.

2) Analogy

Students must map the relationship between two items into a different context:

Example: E-mail is to an unmoderated listserv as office hours are to: a) Class lecture.
b) Class discussion.
c) Review sessions.

d) Tutorials.

3) Case study A single, well-written paragraph can provide material for several follow- up
questions.
Example:
2) Alice, Barbara, and Charles own a small business: the Chock-Full-o-Goodness Cookie
Company. Because Charles has many outside commitments and Barbara has a few, Alice tends
to be most in touch with the daily operations of Chock-Full-o-Goodness. As a result, when
financial decisions come down to a vote at their monthly meeting, they have decided that Alice
gets 8 votes, Barbara gets 7, and Charles gets 2-with 9 being required to make the decision.

According to minimum-resource coalition theory, who is most likely to be courted for their vote?

a) Alice
b) Barbara
c) Charles
d) No trend toward any specific person.

3) In the scenario in question 2, according to minimum-power coalition theory, who is most

likely to be courted for their vote?

a) Alice
b) Barbara
c) Charles
d) No trend toward any specific person.

4) Incomplete Scenario

Students must respond to what is missing or needs to be changed within a provided scenario.

Note: when using a graph or image, try to lay it out differently than how the students have seen
it. This is equivalent to using new language to present a familiar concept and prevents students
from using rote memorization to answer the question.

For example, the diagram below may originally have been split left to right instead of top to
bottom, and this diagram may not be as detailed as the diagram they saw in the book.)

Example: Use the diagram below to answer the following questions.

1) What belongs in the empty box in the upper right corner of the diagram? a) Hardware devices
b) Client Services for Netware
c) Logon Process
d) Gateway Services for Netware

2) If the Applications resided below the heavy black line, they would: a) be open to hackers on
the network.
b) compete with the OS for memory.
c) be preemptively multi-tasked.

d) launch in individual NTVDMs.

5) Problem/Solution Evaluation Student are presented a problem and a proposed solution. They
must then evaluate the proposed solution based upon criteria provided.

Example: A student was asked the following question: "Briefly list and explain the various
stages of the creative process."
As an answer, this student wrote the following:
"The creative process is believed to take place in five stages, in the following order: orientation,
when the problem must be identified and defined, preparation, when all the possible information
about the problem is collected, incubation, when no solution seems in sight and the person is
often busy with other tasks, illumination, when the person experiences a general idea of how to
arrive at a solution to the problem, and finally verification, when the person determines whether
the solution is the right one for the problem."

How would you judge this student' s answer?

a) EXCELLENT (all stages correct in the right order with clear and correct explanations) b)
GOOD (all stages correct in the right order, but the explanations are not as clear as they should
be)
c) MEDIOCRE (one or two stages are missing OR the stages are in the wrong order, OR the
explanations are not clear OR the explanations are irrelevant)

d) UNACCEPTABLE (more than two stages are missing AND the order is incorrect AND the
explanations are not clear AND/OR they are irrelevant)

V. Bibliography of Multiple-Choice Question Resources

Books:

Bloom, Benjamin B. (Ed.) Taxonomy of Educational Objectives: the classification of

educational goals, by a committee of college and university examiners 1st Ed. New York:
Longmans, Green, 1956.
Davis, Barbara Gross. Tools for Teaching San Francisco: Jossey-Bass, 1993.

Erickson, Bette LaSere and Diane Weltner Strommer. Teaching College Freshmen San
Francisco: Jossey-Bass, 1991.
Jacobs, Lucy Cheser and Clinton I. Chase. Developing and Using Tests Effectively: A Guide for
Faculty San Francisco: Jossey-Bass, 1992.

McKeachie, Wilbert. Teaching Tips: Strategies, Research, and Theory for College and
University Teachers (9th Ed.) Lexington, Mass: D.C. Heath and Company, 1994. Miller, Harry
G., Reed G. Williams, and Thomas M Haldyna. Beyond Facts: Objective Ways to Measure
Thinking Englewood Cliffs: Educational Technology Publications, 1978.

Articles:

Clegg, Victoria L. and William E. Cashin. "Improving Multiple-Choice Tests." Idea Paper #16,
Center for Faculty Evaluation and Development, Kansas State University, 1986.
Fuhrman, Miriam. "Developing Good Multiple-Choice Tests and Test Questions." Journal of
Geoscience Education 44 (1996): 379-384.

Johnson, Janice K. ". . . Or None of the Above." The Science Teacher 56.2 (1989) 57-61.

Web-sites:
University of Capetown's Guide to Designing and Managing Multiple Choice Questions An
excellent site from which much of our workshop material was adapted.

8
Sample multiple choice questions that test higher order thinking and application, Content areas:
economics, physics, psychology, ethics, and philosophy

Collected by Kimberly Green, Washington State University Office of Assessment of Teaching

and Learning

Developing Complex Application Questions

Devising multiple choice questions that measure higher level cognitive skills will enable you to test such
skills in large classes without spending enormous amounts of time grading.

In an evaluation question, a situation is described in a short paragraph and then a problem is posed as
the stem of the question. All the rules for writing multiple choice items described above also apply to
writing evaluation questions, but students must use judgment and critical thinking to answer them
correctly.

In the example below (adapted from Welsh, 1978), students must understand the concepts of price
inflation, aggregate private demand, and tight monetary policy. They must also be able to analyze the
information presented and, based on projected effects, choose the most appropriate policy. This
question requires critical thinking and the complex application of economic principles learned in the
course.

“ B ecause of rapidly rising national defense expenditures, it is anticipated

that Country A will experience a price inflation unless measures are taken to
restrict the growth of aggregate private demand. Specifically, the government
is considering either (1) increasing personal income tax rates or (2)
introducing a very tight monetary policy.” If the government of Country A
wishes to minimize the adverse effect of its anti-inflationary policies on
economic growth, which one of the following policies should it use?

a. The tight money policy because it restricts consumption expenditures more

than investment.
b. The tight money policy, since the tax increase would restrict consumption
expenditures.
c. The personal income tax increase since it restricts consumption
expenditures more than investment.
d. Either the tight money policy or the personal income tax rate increase
since both depress investment equally.

Instructors have developed similar questions for analysis and interpretation of poetry, literature,
historical documents, and various kinds of scientific data.
From http://cfe.unc.edu/pdfs/FYC8.pdf University of North Carolina, Chapel Hill

From CLASSROOM TEST OF SCIENTIFIC REASONING Multiple Choice Version Revised

Edition: August 2000 by Anton E. Lawson, Arizona State University. Based on: Lawson, A.E. 1978. Development and
validation of the classroom test of formal reasoning. Journal of Research in Science Teaching, 15(1): 11-24.
1. Suppose you are given two clay balls of equal size and shape. The two clay balls
also weigh the same. One ball is flattened into a pancake-shaped piece.

Which of these statements is correct?

a. The pancake-shaped piece weighs more than the ball b. The two pieces still weigh
the same
c. The ball weighs more than the pancake-shaped piece

2. because
a. the flattened piece covers a larger area.
b. the ball pushes down more on one spot.
c. when something is flattened it loses weight. d. clay has not been added or taken
away.
e. when something is flattened it gains weight.

3. To the right are drawings of two cylinders filled to the same level with water. The
cylinders are identical in size and shape.

Also shown at the right are two marbles, one glass and one steel. The marbles are the
same size but the steel one is much heavier than the glass one. When the glass
marble is put into Cylinder 1 it sinks to the bottom and the water level rises to the 6th
mark.

If we put the steel marble into Cylinder 2, the water will rise

a. to the same level as it did in Cylinder 1 b. to a higher level than it did in Cylinder 1 c.
to a lower level than it did in Cylinder 1
4. because

a. the steel marble will sink faster.

b. the marbles are made of different materials.
c. the steel marble is heavier than the glass marble.
d. the glass marble creates less pressure. e. the marbles are the same size.

19. FarmerBrown was observing the mice that live in his field. He discovered that all of
them were either fat or thin. Also, all of them had either black tails or white

tails. This made him wonder if there might be a link between the size of the mice and
the color of their tails. So he captured all of the mice in one part of his field and
observed them. Below are the mice that he captured.

Do you think there is a link between the size of the mice and the color of their tails?
a. appears to be a link
b. appears not to be a link
c. cannot make a reasonable guess

20. because
a. there are some of each kind of mouse.
b. there may be a genetic link between mouse size and tail color.
c. there were not enough mice captured.
d. most of the fat mice have black tails while most of the thin mice have white tails.
e. as the mice grew fatter, their tails became darker.

MULTIPLE CHOICE TEST – ETHICS/PHILSOPHY University of Kansas, Fall2004

Philosophy 160: Introduction to Ethics Ben Eggleslon—egglestongkuedu Tuesday, September 28,2004

III. Utilitarianism

7. Why, according to utilitarianism, should laws not impose unnecessary restrictions on individuals'
freedom?

(A) because individuals have a natural right to freedom

(B) because restrictions on freedom tend to diminish happiness
(C) because it is expensive to hire police officers to enforce freedom-restricting laws
(D) because the laws are based on a social contract, and the idea of the social contract entails

maximizing individuals' freedom

8. Suppose that a dangerous defendant who will otherwise be acquitted can be convicted through the
giving of false testimony. In the following dialogues, which statement is inconsistent with utilitarianism?

(A) Eric; "Giving false testimony is unjust and, therefore, immoral." (B) Fred: "Not all conduct
traditionally regarded as unjust is immoral." (C) Eric: "Well, giving false testimony is illegal in nearly
every society." (D) Fred: "Well, that does not necessarily mean it's immoral."

9. Which of the following statements is the strongest evidence that the person saying it is a utilitarian?

(A) Ginny; "Violations of rights are very serious, from the moral point of view."
(B) Helen: "I agree. It is always immoral to violate someone's rights."
(C) Ginny: "Well, I wouldn't say 'always'. It's o.k, to violate rights whenever the good you can produce by
doing so outweighs the harm you do by violating the person's rights."
(D) Kate: "I disagree with both of you. In my opinion, it's immoral to violate someone's rights unless
doing so is the only way to prevent that person from violating others' rights."

10. Consider the question, "What is meant by the charge that utilitarianism is too demanding?" Now
suppose the following answer is given:
"Utilitarianism requires moral people to respond to important moral concerns such as helping the less
fortunate, while allowing immoral people to pursue their careers, family lives, and personal projects."
What is wrong with this answer?

(A) Nothing - that answer is correct.

(B) It falsely describes what utilitarianism requires of moral people.
(C) It falsely describes what utilitarianism allows of immoral people.
(D) It relies on a false dichotomy between moral people and immoral people.

IV. Kant's Moral Theory

11. Suppose a person named Nick tells you to give $100 to the United Way, and you say that you are
perfectly happy with the charitable contributions that you currently mate (which do not include
anything for the United Way). What response from Nick would indicate to you that he meant his remark
categorically?

(A) "It doesn't matter whether you are already happy; you still have to give $100 to the United Way." (B)
"O.k., if you are perfectly happy with the charitable contributions you currently make, then I withdraw
what I said."
(C) "I'm not making this request of just you; I'm telling every adult human being that they have to give
$100 to the United Way, too."

(D) "You may think you are perfectly happy with the charitable contributions you currently make, but
giving $100 to the United Way will make you even happier."

Two part question using one scenario (used for discussion in a large lecture class) [from Dr. Sam
Swindell, Psychology, WSU]:

APPLICATION 1: The Scenario

Assuming the statistic is true, is the conclusion
valid? That is to say, does wearing your seatbelt
increase the likelihood that you will survive a car
• While watching TV, you see a public service announcement accident?
that claims the following:
A. Yes
“Last year, 250,000 individuals died in car accidents when they
were not wearing their seatbelts. This shows how dangerous it
B. No
is to drive without your seatbelt fastened. Be Safe. Buckle Up.”

C. Cannot determine based on insufficient

information

APPLICATION 1: The Scenario

 If you said “Yes” in response to the previous
• While watching TV, you see a public service announcement that question, you committed which of the
claims the following: following violations in rationality?

“Last year, 250,000 individuals died in car accidents when they were A. Ignoring baseline rates B. Framing effect
not wearing their seatbelts. This shows how dangerous it is to drive
C. Small sample bias
without your seatbeltfastened. Be Safe. Buckle Up.”
D. Conjunction fallacy