Writing Strong
Multiple-Choice Items
Alan Carter, IT Instructor
acarter@greenriver.edu
1
Agenda
 9:00 – 9:45
 9:45 – 10:30
10:30 – 11:00
11:00 – 11:30
11:30 – 11:45
Welcome
Presentation on Writing Strong
Multiple-Choice Exam Items
Break, Participants Write 3 Items
Meet as a Group
Introduce Multiple-Choice Item Rubric
Break, Participants Analyze Their
Own Items, Groups Analyze Each
Others’ Items Against the Rubric
Meet as a Group to Discuss Reflection/
Reactions/Results, Q & A, Wrap-Up
2
In the Beginning,
There Was Assessment...
 What Is Assessment?
According to Stephen Brookfield, “assessment is
a value-free ascertainment of the extent to which
objectives determined at the outset of a program
have been attained by participants. [It] requires no
value judgment as to their worthwhileness. It is
simply a nonjudgmental checking as to whether or
not certain purposes have been attained.”
3
Why Do We Assess Student Learning?
 To determine whether students have met the
learning outcomes specified in the CAR
 For grading purposes
 For accreditation purposes
4
Exams: A Common
Assessment Technique
 Exams are often used to assess student learning
 Discussion Question:
What general types of questions or elements
do you use in your exams (multiple-choice,
perform a hands-on task, etc.)?
5
Focus of This Session: Writing Strong
Multiple-Choice Test Items
 Multiple-choice items are test items in which
students select the correct answer from a list of
several choices. (Haladyna, 1994)
 Multiple-choice items can be used to “measure
knowledge and complex mental acts, such as
reasoning, critical thinking, and problem solving”
(Haladyna, 1994).
6
Components of a Multiple-Choice Item
What is H2O?
A.
B.
C.
D.
Fire
Water
Air
Earth
Stem
Correct Answer
Distractors
7
Components of a Complex
Multiple-Choice Item
Scenario
Jack, who is 48, took his 78-year-old mother to see a
movie. General admission tickets cost $10, and the
cost of senior admission is $8.
How much did Jack spend for movie tickets?
A.
B.
C.
D.
$ 10
$ 16
$ 18
$ 20
Stem
Correct Answer
Distractors
8
What Is a Strong
Multiple-Choice Item?
 Measures the learning outcomes for the course
 Follows the best practices for writing multiplechoice items
 Assesses student learning at the appropriate level
of Bloom’s taxonomy (which we will discuss later
in this session)
9
Learning Outcomes
 The course content learning outcomes in the
Course Adoption Revision document (CAR)
should drive assessment
 For example, if an outcome states:
“Student identifies the functions of the four
classes of biomolecules”
then the exam items relating to this outcome
should require students to identify the functions,
as opposed to analyzing or evaluating the
functions
10
Best Practices for Constructing
Multiple-Choice Items
 There should be only one correct answer
 Avoid “trick” questions
 The wording in the stem should not
telegraph the correct answer
 Avoid negatives (no, not, etc.) in the scenario
and stem. If they cannot be avoided, consider
calling attention to the word by using bold and/or
italic font
11
Best Practices for Constructing
Multiple-Choice Items
 All distractors should be plausible answers
 Distractors should be of parallel construction and
length
 Avoid “all of the above,” “none of the above,” and
“A and B, but not C”
 Avoid testing trivial material
 Items should focus on a single problem, concept,
or idea
12
Best Practices for Constructing
Multiple-Choice Items
 Balance the answer key: If there are four possible
answers, approximately 25% of the correct
answers should be A, 25% should be B, and so on
 Items should not provide the answer for other
items on the exam
 When possible, the stem should be a
question, and end with a question mark
 Avoid textbook verbatim phrasing unless asking
a knowledge-based question
13
You Be the Judge
Which sport involves players shooting a ball through
a hoop?
A.
B.
C.
D.
Soccer
Cricket
Ping pong
Basketball
Which best practice of item writing
does this item violate?
14
You Be the Judge
What is a paraplegic?
A. An animal
B. A fish
C. A person suffering from paralysis of both lower
limbs
D. A plant
Which best practice of item writing
does this item violate?
15
You Be the Judge
The textbook, Operations Management, was written
in:
A.
B.
C.
D.
2000
2002
2004
2006
Which best practices of item writing
does this item violate?
16
Introduction to Bloom’s Taxonomy
 Benjamin Bloom, an educational psychologist,
proposed this classification of learning levels in
1956
 Breaks cognitive thinking into six categories
 These categories can be thought of as
degrees of difficulty
17
Levels of Bloom’s Taxonomy
Category/Level
Description
Evaluation
Make judgments
Synthesis
Put parts together to form a whole
Analysis
Distinguishes between facts and
inferences
Application
Use a concept in a new situation
Comprehension
Understand the meaning
Knowledge
Recall data or information
Clark, D. (2001)
18
Bloom’s Examples and Key Words
Knowledge
Examples: Recite a policy. Quote prices from memory
to a customer. List the safety rules.
Key Words: defines, describes, identifies, knows,
labels, lists, matches, names, outlines, recalls,
recognizes, reproduces, selects, states.
Comprehension
Examples: Explain in your own words the steps for
performing a complex task. Translate an equation into
a computer spreadsheet.
Key Words: comprehends, converts, defends,
distinguishes, estimates, explains, extends,
generalizes, gives examples, infers, interprets,
paraphrases, predicts, rewrites, summarizes,
translates.
Clark, D. (2001)
19
Bloom’s Examples and Key Words
Application
Examples: Use a manual to calculate an employee’s
vacation time. Apply laws of statistics to evaluate the
reliability of a written test.
Key Words: applies, changes, computes, constructs,
demonstrates, discovers, manipulates, modifies,
operates, predicts, prepares, produces, relates,
shows, solves, uses.
Analysis
Examples: Troubleshoot a piece of equipment by
using logical deduction. Recognize logical fallacies in
reasoning. Gather information from a department and
select the required tasks for training.
Key Words: analyzes, breaks down, compares,
contrasts, diagrams, deconstructs, differentiates,
discriminates, distinguishes, identifies, illustrates,
infers, outlines, relates, selects, separates.
Clark, D. (2001)
20
Bloom’s Examples and Key Words
Synthesis
Examples: Write a company operations or process manual.
Design a machine to perform a specific task. Revise a process
to improve the outcome.
Key Words: categorizes, combines, compiles, composes,
creates, devises, designs, explains, generates, modifies,
organizes, plans, rearranges, reconstructs, relates,
reorganizes, revises, rewrites, summarizes, tells, writes.
Evaluation
Examples: Select the most effective solution. Hire the most
qualified candidate. Explain and justify a new budget.
Key Words: appraises, compares, concludes, contrasts,
criticizes, critiques, defends, describes, discriminates,
evaluates, explains, interprets, justifies, relates, summarizes,
supports.
Clark, D. (2001)
21
Bloom’s Level?
The Social Security Act of 1935 provides:
A.
B.
C.
D.
Emergency relief to farmers
Funding for small businesses
Financial assistance to retired people
A guaranteed minimum wage for workers
22
Bloom’s Level?
Your car suddenly stops running. When you attempt
to restart the car by turning the key in the ignition to
the start position, nothing happens.
What is the most likely cause of the problem?
A. The gas tank is empty
B. The spark plugs are fouled
C. The battery is dead
D. The fuel injectors are clogged
23
What About Test Banks?
 What is your experience with test banks?
Do you use them? Do you like the items?
 Test bank items often violate many of the best
practices for writing strong multiple-choice items
 You may find that you need to edit or revise test
bank questions
24
How Many Questions Per Exam?
 Depends on:
Complexity of exam items
Length of exam items
Amount of analysis required
Amount of time available for exam
 Consider taking the exam yourself, and
then allowing students 2–3 times as long
as it took you to complete the exam
25
Assignment: Item Writing
 Take a short break!
 Write three multiple-choice items:
Create a scenario for each item, or one
scenario that is used for multiple items
Try to target the Application or Analysis
level of learning
Use best practices for item writing
 Reconvene at 10:30 a.m.
26
Item Writing Debrief
 Was everyone able to write three items?
 Did you find it difficult to write items that followed
the best practices?
 Did you find it difficult to write items that targeted
the higher levels of learning?
27
Using Multiple-Choice Items to
Assess Critical Thinking
Critical thinking is one of the college’s campus-wide
learning outcomes:
“Critical thinking finds expression in all disciplines
and everyday life. It is characterized by an ability to
reflect upon thinking patterns, including the role of
emotions on thoughts, and to rigorously assess the
quality of thought through its work products. Critical
thinkers routinely evaluate thinking processes and
alter them, as necessary, to facilitate an improvement
in their thinking and potentially foster certain
dispositions or intellectual traits over time.”
(Green River Community College, 2005, Critical Thinking Community Rubric)
28
Using Multiple-Choice Items to
Assess Critical Thinking
 You can use multiple-choice items to assess a
student’s ability to think critically about a topic
 Items that assess critical thinking will fall in the
Application, Analysis, Synthesis, or Evaluation
level of Bloom’s taxonomy
29
Must We Always Assess
Critical Thinking?
 In general, not all multiple-choice items should
assess critical thinking, because not all course
learning outcomes are at the Application, Analysis,
Synthesis, or Evaluation levels
 For example, if a learning outcome says the
student will comprehend a topic, an item at the
Comprehension level would be appropriate
This item should not measure critical thinking
30
A Rubric for Assessing How Well MultipleChoice Items Measure Critical Thinking
Competency
Poor/No
Critical
Thinking
Required
Adequate/Easy Good/
Intermediate
Excellent/
Difficult
2.1 Apply
relevant criteria
and standards
when
evaluating
information,
claims, and
arguments
The item has
only 1 discrete
fact or statement
that must be
evaluated when
determining how
to answer the
question
The item has
two discrete
facts or
statements that
must be
evaluated when
determining how
to answer the
question
The item has
four or more
discrete facts or
statements that
must be
evaluated when
determining how
to answer the
question
The item has
three discrete
facts or
statements that
must be
evaluated when
determining how
to answer the
question
31
A Rubric for Assessing How Well MultipleChoice Items Measure Critical Thinking
Competency
Poor/No
Critical
Thinking
Required
Adequate/Easy Good/
Intermediate
Excellent/
Difficult
2.2 Use
appropriate
reasoning to
evaluate
problems, make
decisions, and
formulate
solutions.
The item:
Is knowledge
based
Has only one
fact or statement
for students to
analyze
The item:
Requires
students to
analyze multiple
facts or
statements that
are interrelated
Partially or
directly
telegraphs the
correct answer
The item:
Requires
students to
analyze multiple
facts or
statements that
are discrete
Does not
telegraph the
correct answer
The item:
Requires
students to
analyze multiple
facts or
statements that
are partially
discrete
Partially
telegraphs the
correct answer
32
You Be the Judge
What is H2O?
A.
B.
C.
D.
Fire
Water
Air
Earth
Using the rubric, does this item do a poor,
adequate, good, or excellent job of
assessing critical thinking for each competency?
33
You Be the Judge
Your car suddenly stops running. When you attempt
to restart the car by turning the key in the ignition to
the start position, nothing happens.
What is the most likely cause of the problem?
A. The gas tank is empty
B. The spark plugs are fouled
C. The battery is dead
D. The fuel injectors are clogged
Using the rubric, does this item do a poor,
adequate, good, or excellent job of
assessing critical thinking for each competency?
34
Assignment: Determining Whether
Items Assess Critical Thinking
 As we discussed previously, not all multiplechoice items should assess critical thinking
 However, in this exercise we will practice
determining how well a multiple-choice item
assesses critical thinking
35
Assignment: Determining Whether
Items Assess Critical Thinking
 Form teams of two persons
 First, individually evaluate the three items you
wrote by using the rubric. Rate each item as Poor,
Adequate, Good, or Excellent for each competency.
 Next, exchange items with your teammate and
evaluate his or her items using the rubric.
 Meet with your teammate and discuss any
differences in your ratings.
 If time, take a short break!
 Reconvene at 11:30 a.m. for discussion and
final wrap-up
36
Reflections, Reactions,
and Questions
37
Resources for Writing
Multiple-Choice Test Items
Bloom, B. S. (1956). Taxonomy of educational objectives,
Handbook I: The cognitive domain. New York: David
McKay Co. Inc.
Haladyna, T. M. (1994). Developing and validating
multiple-choice test items. Hillsdale, NJ: Lawrence
Erlbaum Associates.
38
References
Brookfield, S. D. (1986). Understanding and facilitating
adult learning: A comprehensive analysis of principles
and effective practices. San Francisco: Jossey-Bass.
Clark, D. (2001). Learning domains or Bloom’s taxonomy.
Retrieved September 9, 2006, from
http://www.nwlink.com/~donclark/hrd/bloom.html
Haladyna, T. M. (1994). Developing and validating
multiple-choice test items. Hillsdale, NJ: Lawrence
Erlbaum Associates.
39