Writing Multiple Choice
Questions
Amy S. Oxentenko, MD, FACP, FACG, AGAF
Mayo Clinic, Rochester
Disclosures
• Relevant Relationships:
 Consulting work for MKSAP 14-17
 On the IM-ITE exam committee
Outline/Objectives
• Define the test taker level before writing
• List components of a multiple-choice question
 Objective, stem, lead line, options
 Annotated description of answers
• Evaluate your questions
Who Are You Writing For?
• Med school GI pathophysiology course?
 More details; physiology on USMLEs
• Resident end-of-rotation exam?
 Presentation, testing, management
• GI Fellow board review course?
 More advanced understanding of above
• IM or GI CME course?
 Very general vs very specific
What are the
Stakes of the Questions?
• Board certification?
• Passing a class/rotation?
• Simple knowledge assessment?
• Solely for interaction?
The Stakes
Reproducible
Accurate
Learning
What Should Be Tested?
• For a clerkship/CME course
 Match goals/objectives
• For a standardized exam
 Match the targeted learner level
How Often Will Learner Encounter
Information
• Frequent
 More relevant to test often
 Atypical manifestations
• Infrequent
 Test the “can’t miss” things
 Life threatening
 Typical manifestations
Types of Questions
• Single best answer
 With clinical scenario
•
•
•
•
•
Factoid questions
All of the following EXCEPT
Matching
True/False
A and B, A and C, all of the above
What are the Components of a
Multiple Choice Question
(MCQ)?
Question Components
STEM
LEAD
LINE
OPTIONS
Question Components
ANSWER
OBJECTIVE
EXPLANATION
PLEASE TAKE THE PRETEST.
Steps to Writing the MCQ
1. Determine the objective
2. Create the lead line and options
3. Draft the stem
4. Write the explanation, if required
The Objective
Example: Objective
• Recall alcoholic hepatitis
What is wrong with this?
Example: Objective
• Treat a patient with alcoholic hepatitis
with corticosteroids in the absence of
contraindications
More focused
The Objective
• Succinct sentence
• Specifies a clinical skill to be learned
• Uses action verbs to delineate a goal
 Recognize, diagnose, treat, manage
• Avoid vague action verbs
 Remember, recall, know
• Allows you to match curriculum
Objective Format
VERB
+
TOPIC
+
MODIFIER
Action Verbs for Objectives
Examples of Objectives:
The Good, the Bad, the Ugly
• Good:
 Diagnose niacin deficiency in a patient
with carcinoid syndrome
• Bad:
 Recall features of niacin deficiency
• Ugly:
 Understand niacin deficiency
Other Objective Examples
• Select appropriate testing needed before
beginning a biologic agent on an IBD patient.
• Evaluate for splenic vein thrombosis in a
patient with isolated gastric varices.
• Manage a patient with celiac disease who has
persistent symptoms on a GFD.
• Identify peptic ulcers with stigmata for
rebleeding that require endoscopic therapy.
Lead Line
Example: Lead Line
A 56-year old alcoholic stumbles into the ER, ataxic,
with nystagmus on examination. Although he
appears confused and intoxicated, his blood alcohol
level is normal. You start IV glucose and symptoms
worsen.
The next step would be to administer
A.
B.
C.
D.
E.
Niacin
CT scan of the head
Thiamine
Insulin
Dismissal to home
What is
wrong
with this?
Example: Lead Line
A 56-year old alcoholic stumbles into the ER, ataxic,
with nystagmus on examination. Although he
appears confused and intoxicated, his blood alcohol
level is normal. You start IV glucose and symptoms
worsen.
What is the next best step in management?
A.
B.
C.
D.
E.
Niacin
CT scan of the head
Thiamine
Insulin
Dismissal to home
Keep lead line
general; avoid
syntax errors
Lead Line
• Most commonly used lead lines:
 Which of the following is the most likely diagnosis?
 What is the next best step in management?
 Half treatment options, half diagnostic options
• Avoid negative lead lines:
 All of the following EXCEPT
 Which of the following is LEAST likely
 Which would you NOT do
Lead Line
• Keep the lead line generic
 “Next step” or “management”
 Allows diagnostic & therapeutic options
• Avoid writing syntax or grammatical errors
that help exclude answers
• “Cover the options” rule
 Should be able to guess the answers
The Options
Example #1: Options
What is the next best step in management?
A.
B.
C.
D.
E.
Add IV metronidazole
Continue po vancomycin
Start cholestyramine
Begin rifaximin
Stop the po vancomycin
What is
wrong
with this?
Example #1: Options
What is the next best step in management?
A.
B.
C.
D.
E.
Add IV metronidazole
Continue po vancomycin
Start cholestyramine
Begin rifaximin
Stop the po vancomycin
Mutually
exclusive
Example #2: Options
What is the next best step in management?
A.
B.
C.
D.
Metronidazole 500 mg po TID x 14 days
Vancomycin 250 mg IV QID x 10 days
Metronidazole 500 mg IV QID x 14 days
Vancomycin 125 mg po QID x 14 days
What is wrong with this?
Example #2: Options
What is the next best step in management?
A.
B.
C.
D.
Metronidazole 500 mg po TID x 14 days
Vancomycin 250 mg IV QID x 10 days
Metronidazole 500 mg IV QID x 14 days
Vancomycin 125 mg po QID x 14 days
Testing too many concepts!
Drugs, dose, route, duration
Example #3: Options
The cause of diarrhea in most patients with
Zollinger-Ellison syndrome is:
A. Antacid use
B. High glucagon levels
C. Fat malabsorption from pancreatic lipase
inactivation by acid production
What is wrong
D. Bacterial overgrowth
with this?
Example #3: Options
The cause of diarrhea in most patients with
Zollinger-Ellison syndrome is:
A. Antacid use
B. High glucagon levels
C. Fat malabsorption from pancreatic lipase
inactivation by acid production
Correct answer
D. Bacterial overgrowth
longer, detailed
The Options
• 4 or 5 options (less may be better)
• One is not right and the rest 100% wrong
CEAB
D
Wrong
Right
The Options
• 4 or 5 options (less may be better)
• One is not right and the rest 100% wrong
• One is correct, the rest plausible
C
E
Least correct
A
B
D
Most correct
Options
• Avoid mutually exclusive options
 Increases/decreases/same
 Always/never; stop/continue
• Each option should test one concept
 Drug OR dose OR route OR duration
• Keep options brief; similar in length
• Options should be homogeneous
 Length, complexity
Stem
The Stem
• Chronologic order of presentation
 Age, gender (avoid race unless needed)
 Site of visit (ER, clinic, hospital)
 Chief complaint (add features)
 PMH/Meds/FH/SH (relevant or distracter)
 Vitals/Exam/Labs/Tests (pertinent)
The Stem
• Fine to have details to make the other
options attractive; avoid extraneous info
• It takes time to read; make sure it is
fitting in with time to administer test
• Organize labs if many
The Explanation
The Explanation
• Explain why right answer correct first
• Explain why wrong answers incorrect
– Do this chronologically
• Example for explanation order:
– If correct answer is B
– B, A, C, D, E
How Do You Analyze Your
Questions?
Question Performance
• Goal for correct answer 50-70%
 40-90% may be allowed at times
 Difficulty score of > 90% (too easy)
 Difficulty score of < 40% (too hard)
• See if it discriminates:
 Top test takers
 Bottom test takers
Performs well;
difficulty ok and discriminating
(A)
Answer
(B)
(C)
(D)
*
Top 50%
7
80
9
4
Bottom 50%
19
44
28
9
Total
13
62
18
7
Option
0.62
Performs poor;
difficulty ok, non-discriminating
Option
(A)
(B)
(C)
(D)
Answer
*
Top 50%
62
0
7
31
Bottom 50%
80
0
7
13
Total
71
0
7
22
Difficulty
0.71
Performs poor;
difficulty low (hard), non-discriminating
Option
(A)
(B)
(C)
Answer
(D)
*
Top 50%
3
45
2
50
Bottom 50%
6
51
5
38
Total
5
48
3
44
Difficulty
0.44
http://www.nbme.org/publications/item-writing-manual-download.html
The Pretest:
How Did You Do?
Avoid Duplicating Terms
1. The primary role of the denter is to move the
a)
b)
c)
d)
orme
latten
denter pin
stagle
Avoid Repeat Digits,
Look at Pattern
2. How many steps are in an average carrum?
a)
b)
c)
d)
3.7
370
240
37
Avoid Absolute Terms:
Never, Always, Every, All, None
3. The grotto is frequently humid because
a)
b)
c)
d)
every ictal is fabliterous
langers are always gantering
reslins are often dewinest
culers are never liblis
Avoid Inhomogeneous Options
4. Rigniter is a
a)
b)
c)
d)
form of glatnick
prestigious subset of the family of honorary gloutnins
signal by a sellant
crown of slessings
Avoid Repeating the Correct Option
More Than Others
5. Which of the following pairs contributed
the Sterlik track?
a)
b)
c)
d)
Smith & Johnson
Jones & Smith
Johnson & Thomson
Smith & Davis
Avoid Syntax or Grammatical Errors
6. The tarnon is proceeded by a
a)
b)
c)
d)
ofler
argling
isterin
glantern
Questions?
Summary
• Know your learner level
• Writing order: objective, lead line, options, stem
• Avoid errors in MCQ writing
• Evaluate the effectiveness of your MCQs
Thank you!
oxentenko.amy@mayo.edu