Developing Multiple Choice
Questions
David K. Coats, MD
OKAP Committee Chair
Texas Children’s Hospital in Houston


The Ophthalmic Knowledge Assessment Program
(OKAP) is an in-training ophthalmology examination
offered annually by the American Academy of
Ophthalmology for ophthalmology residency
programs in the United States and Canada.
Dr. Coats has disclosed the following financial
interests:
David J. Wilson, MD
WQE Chairman
Stacey Samuels
Program Manager, Assessment and Evaluation
Clinical Education Division of the AAO
 The
intention of this tutorial is to
educate item writers on acceptable
formats, good technique, and what to
avoid when writing questions.
STEM
Correct
Response
Which of the following phenomena increases as a
pinhole aperture gets smaller?
A.
Diffraction
B.
Chromic aberration
C.
Polarization
D.
Spherical aberration
Distractors
 Category
1: Memory or Knowledge Recall
 Category
2: Interpretive
 Category
3: Decision Making and Clinical
Management
Examples of each type of question are provided in the
training copy of this presentation
Memory or Knowledge Recall
A Hudson-Stahli line represents iron
deposition in which of the following
corneal layers?
A. Bowman’s membrane
B. Descemet’s membrane
C. Epithelium
D. Stroma
INTERPRETIVE
The lesion shown in Figure 1 is most consistent
with a diagnosis of:
A. chronic viral infection.
B. hemangioma
C. Kaposi sarcoma.
D. recurrent conjunctival
dysplasia.

INTERPRETIVE
Three months after sustaining blunt ocular trauma and
associated hypotony, a patient experiences sudden
onset of pain with corneal edema. Which of the
following is the most likely cause of this condition?
A. Corneal decompensation from acute endothelial
dysfunction
B. Late bleeding and hyphema from injured iris
vessels
C. Neovascular glaucoma
D. Spontaneous closure of an occult cyclodialysis
cleft
Decision Making and Clinical Management
Three weeks ago, the parents of a 2½-year-old girl
noticed intermittent esotropia in the child's right
eye. Examination reveals 20 prism diopters of
esotropia at 6 ⅓ m. Which of the following is the
most appropriate initial therapy?
A. Alternate occlusion
B. Base-out, 10 PD Fresnel prisms on plano
spectacles
C. Recession of the medial rectus muscles
D. Spectacles for correction of hyperopia
 Decision
Making and Clinical Management (can
include visuals)
A 24-year-old woman experiences sudden loss of vision in her
right eye with a visual acuity of 20/400. She has a right
afferent pupillary defect, optic disc swelling, and a central
scotoma. Her left eye is normal. A complete neurological
history and physical examination reveal no abnormalities.
Which of the following is the most appropriate next step?
A. Administer intravenous methylprednisolone alone
B. Administer intravenous methylprednisolone, followed by
oral prednisone
C. Perform CT scan of the orbit
D. Perform MRI of the brain
 The
ABO’s secure website contains a
visual library of images available to its
item writers
 You may upload images that will become
part of the question directly onto the
website
• Be sure to include a diagnosis and a description
for uploaded visuals
• Once submitted, visuals are copyrighted and
should not be used or have been previously
used in a publication or lecture materials.
 You
will be asked to complete a copyright. This
allows the AAO unlimited use of the images in
its educational programs.
 If
you have concerns about unlimited
copyright, work with your staff liaison to
explore alternatives.
 You
can post your images directly to a secure
area of the AAO’s ftp site. Staff will then catalog
the image as appropriate.
Due to the new “patient privacy” and HIPAA
regulations, it is essential that you have a
signed release form (see Release Form) if you
are submitting published material or
photographs in which patients may be
identifiable. This applies to photographs taken
after April 15, 2003.
 Release forms from your particular institution
also are acceptable.

 Images
such as photographs, diagrams, or
visual field results can enhance the quality
of your questions
 The
ABO and AAO encourage use of images
when they contribute required clinical
information necessary to answer the
question
 Images
should clearly illustrate only the
condition or finding described
Two examples with images are provided in the training
copy of this presentation.
The 60-year-old woman with the findings shown has pericentral
scotoma in the right eye 2 months after combined cataract and
glaucoma surgery. Which of the following is the most likely
diagnosis?
A. Age-related macular degeneration
B. Cytomegalovirus retinitis
C. Histoplasmosis
D. Photic maculopathy
The visual provided is an accurate representation of the
entity helps to improve the validity of the examination.
The 60-year-old woman with the findings shown has
pericentral scotoma in the right eye 2 months after
combined cataract and glaucoma surgery. Which of
the following is the most likely diagnosis?
A.
B.
C.
D.
Age-related macular degeneration
Cytomegalovirus retinitis
Histoplasmosis
Photic maculopathy
While the stem is well written and the distractors are plausible,
the photograph is not a good representation of photic
maculopathy.





The stem or question must provide a sufficiently
clear and focused description of the problem/case
scenario for a candidate to make a decision.
The stem should include all clinical information
necessary for the candidate to make an
unequivocal choice about the correct response.
The information should assure that there is only
one, clear correct response.
No excess verbiage irrelevant to the question.
a knowledgeable candidate should be able to
answer the question before reading the responses


Category I questions should be clearly stated and
have clinical significance
Category II and III questions should consist of:
 A description of the clinical findings, patient
symptoms, or other specific data that provides
background for the question (clinical scenario),
and
 A question that details specifically what is
being asked

As you develop the question, you have a clear idea
of exactly what the correct response will be.
A
clear question should be presented at
the end of a clinical scenario
 Good lead-in questions include:
• Which of the following is the most likely
diagnosis, cause, complication?
• Which of the following procedures, tests,
treatments is most appropriate?
Good Category 1, 2, and 3 stems are provided in the training
copy of this presentation


Which of the following is an early feature
of choroideremia (progressive
tapetochoroidal dystrophy)?
Which of the following treatments is most
likely to minimize the risk of epithelial
keratitis in a patient with herpes simplex
infection?



The histopathology report on a specimen of
posterior retina indicates thinning of the nerve fiber
layer and loss of ganglion cells, but the inner
nuclear layer, outer nuclear layer, and
photoreceptors are intact. What is the most likely
diagnosis?
Eosinophils are found in conjunctival scrapings from
a patient who has symblepharon. What is the most
likely diagnosis?
After 1% atropine drops are instilled into the
eyes of a blue-eyed, 3-year-old boy, he becomes
hyperpyretic, disoriented, and hyperactive.
Which treatment options are most appropriate?

A patient who undergoes complicated
intraocular surgery with an open lens capsule
develops secondary open-angle glaucoma
that requires a combination of a beta blocking
agents, topical carbonic anhydrase inhibitors,
and latanoprost to control. Two months after
surgery, vision has decreased to 20/200.
Which of the following is the most likely
cause of the decrease in vision?
A
70-year old with open angle glaucoma in
both eyes presents with pain, redness, and
decreased vision in his right eye for 2 days. He
is using no medication in that eye. Two years
ago a trabeculectomy was performend on the
right eye, with mitomycin C. His best corrected
visual acuity is 20/200, and slit lamp exam
shows 3+ injection with a seidel positive bleb
and yellowish discharge. There is a 1+ cell and
flare in the interior chamber. What is the most
important condition to rule out?
 Trivia, esoteric
information
 clinically irrelevant information
 Poor gauge of knowledge or clinical
competence
 The stem is an open-ended statement
 Unfocused
(True/False or Global)
 Negatively
worded

Examples of Poor Question Stems are
provided in the training copy of this
presentation
Which of the following medications can cause
palinopsia?
A. Amitriptyline
B. Doxycycline
C. Gabapentin
D. Trazodone
Problem: TOO ESOTERIC
Extremely difficult for candidates to answer correctly.
A 5-year-old boy has acute and chronic polyarthritis,
recurrent fever, pericarditis, rash, and hepatosplenomegaly.
What is the probability that iridocyclitis will develop?
A. Negligible
B. 25%
C. 50%
D. 90%
Problem: NOT CLINICALLY RELEVANT
The chances that the child will develop iridocyclitis are not
clinically relevant. Would knowing the % change the
candidate’s treatment of this patient?
Which of the following best describes the Landolt ring?
A. A device for stabilizing the lens capsule during cataract
surgery
B. A measuring tool for determining the corneal diameter
C. An optical phenomenon that results from diffraction
D. An optotype for measuring visual acuity
Problem: POOR GAUGE OF KNOWLEDGE OR CLINICAL
COMPETENCE
The question would be better if it focused on the use of the Landolt
ring Instead of asking for the description of the Landolt ring. This
question could be improved by asking:
The Landolt ring is best used in the measurement of which of the
following visual functions?
1.
2.
3.
4.
5.
6.
7.
The interference property of light:
Statin use:
Viruses:
Lung cancer is:
Which of the following statements is likely regarding XXX?
Which best describes a common occurrence in a patient
who XXX?
What would be a common practice regarding XXX?
Problem: THESE STEMS ARE OPEN-ENDED
Whether open ended or in question format , unfocused stems do not
provide candidates with enough information to formulate a
response before reading the choices. A well-constructed stem
allows knowledgeable candidates to look for the correct response.
Which of the following statements regarding lattice
degeneration is true?
A. It is equally common in emmetropes and myopes.
B. It is found in 20-30 % of all eyes that present
with a rhegmatogenous retinal detachment.
C. It frequently is associated with peripheral
retinoschisis in children.
D. It is nearly always bilateral.
Problem: True/False
The answers are a list of statements following an unfocussed
stem. A, C, and D may be related to frequency, so rewriting the
stem to ask about frequency might help.
1.
2.
Which of the statements about the reflex of the
retinoscope is true?
Which of the statements about a patient with
mixed astigmatism is correct?
3.
Identify which statement is accurate with regard to
closed-angel glaucoma:
4.
Compared to essential blepharospasm, hemifacial
spasm usually:
Problem: UNFOCUSED (TRUE/FALSE OR GLOBAL)
If the correct response and distractors are a series of
true/false statements, the stem most likely is unfocused.
Epithelial keratitis is less likely to occur after a herpes
simplex infection if the active infection is treated with:
A. minimal wiping debridement and topical
trifluridine drops.
B. topical trifluridine drops.
C. topical trifluridine drops and vidarabine ointment.
D. none of the above.
Problem: Negatively-worded
“Less likely” is an example of negative question writing.
Additionally, there is NO correct response. In reality, A, B,
and C all are correct treatments. This type of question is
confusing for candidates.
 Each
answer choice should rarely exceed more
than 1 line
 Listed
in logical, alphabetical, or numerical
order unless doing so gives away the correct
answer
 Have
both the correct response and the
distractors balanced in their plausibility, length
and amount of detail
 All
responses are parallel, e.g., all tests, all
treatments
 Wrong
answers should reflect faulty reasoning
or misconceptions. There should only be 1
defensible answer. Content that is controversial
or that varies regionally should be avoided in
order to eliminate the possibility of more than
one correct response.
 Incorrect
answers should be attractive to at
least some candidates
Question Stem: What is the most
appropriate initial management of
intraoperative bleeding at the site of a
neodymium: yttrium aluminum garnet
(Nd:YAG) laser iridotomy?
A. compression of the eye with the laser
iridotomy lens
B. pharmacologic pupillary dilation and
cycloplegia
C. cauterization of the vessel with the
Nd:YAG laser
D. bed rest and topical corticosteroid
eyedrops
 “All
of the Above” /“None of the Above” are
unacceptable to the ABO and the AAO
 attempt
to test too many areas at once, are
confusing, and when a candidates gets the
question wrong, it’s difficult to determine
which aspect of the question they did not
know.
Examples of poor distractors are provided in training
copy of this presentation
Ocular conditions associated with
hypertension included which of the following?
A. Retinal vein occlusion
B. Ischemic optic neuropathy
C. Glaucoma
D. All of the above
Problem: In addition to having an unfocused stem, “All of
the above” and “None of the Above” are unacceptable
answer choices. Similarly, “Part I only” and “Part II and III
only” types of answers are unacceptable.
Which of the following characteristics distinguishes
congenital fibrosis
syndrome from double elevator palsy?
A. Convergence on attempted elevation
B. Hypotropia in primary gaze increasing on upgaze
C. Positive forced ductions to elevation
D. Ptosis or pseudoptosis
Problem: There is more than 1 defensible answer. Although
choice A is marked as the preferred response, answer D is
arguable (ptosis or pseudoptosis is also correct.)
 What
should treatment of acute
dacryocystitis ALWAYS involve?
A. Systemic antibiotics
B. Orbital imaging
C. Immediate nasolacrimal duct probing
D. Lacrimal sac massage
Problem: Answers choices are not homogenous or parallel.
All the answer choices should be tests, symptoms, or
differential diagnoses, for example. Unparallel answer
choices are a clue that the stem needs to be focused.

Threshold retinopathy of prematurity (ROP) disease is defined
as which one of the following?
A. Five contiguous clock hours of Stage 3 or 8 total clock
hours of Stage 3 in Zone I or II with plus disease
B. Five contiguous clock hours of Stage 2 or 8 total clock hours
of Stage 2 in Zone I or II with no plus disease
C. Five contiguous clock hours of Stage 3 or 8 total clock hours
of Stage 3 in Zone II or III with plus disease
D. Five contiguous clock hours of Stage 3 or 8 contiguous clock
hours of Stage 3 in Zone I or II
Problem: Answer choices are too long.
Answer choices should rarely exceed 1 line. If the answers are too
long, it takes a disproportionate amount of time for the candidate to
answer a question even in areas which they are knowledgeable.
Which of the following pairs best describes the outflow of aqueous
humor?
Problem: Attempts to test too many areas
The question is complex in format and difficult to answer.
Moreover, if candidates get the question wrong, it’s difficult to
determine which aspect of the question they did not know.
 Implausible
or silly distractors can be easily
dismissed, increasing ability to guess the
correct response
 the same words appear in both the stem and
one answer choice.
 Incorrect answers can be too obvious (e.g.
‘smoke more cigarettes’)
 One answer is longer than the others
 One answer is grammatically different
 Mutually exclusive responses help the
candidate, because if you know one response
is false, the other must be true.
 Ambiguous
or indefinite terms like might, may,
sometimes and absolute terms like never, always
Scenarios almost always “might” or “may” be
true. Especially in the absence of qualifiers in
other answers, vague terms are a clue to the
correct response. The reverse is true of absolute
terms, where scenarios can be ruled out as
incorrect since they almost always may be true.
Examples of questions that have clues to the correct response are
provided in the training copy of this presentation.
Which of the following lesions is most likely to
cause a retinal hole or tear leading to retinal
detachment?
A. Cyst in the pars plana
B. Cystic retinal tuft
C. Enclosed oral bay
D. Peripheral cystoid degeneration
Problem: the same words appear in both the stem and
one answer choice.
Which of the following is an appropriate alternative to the
use of cyanoacrylate glues in the management of
corneal perforations?
A. Aminocaproic acid
B. Autologous plasmin
C. Erythropoietin
D. Fibrinogen and thrombin glue
Problem: The incorrect answer choices are too obvious.
While D is not a proven “appropriate alternative”, the other
distractors are so obviously incorrect that they lead the
candidate to the correct response.
Pupillary block and iris bombé:
A. occur in reverse in pseudoexfoliation syndrome.
B. are absolute in primary angle closure glaucoma.
C. are common in neovascular glaucoma.
D. may be compartmentalized in pseudophakic eyes
due to localized adhesions between the intraocular
lens surface and iris pigment epithelium, not all of
which will respond to a single iridectomy.
Problems: longer, grammatically different, vague terms.
In addition to an unfocussed stem, the correct answer is
much longer and grammatically different than the other
answers. Further, it contains the vague term “may be”.
Which one of the following is true about acquired
limbal stem cell deficiency?
A. It can be caused by preservatives in topical
medications.
B. It is always diffuse and not focal.
C. It can never be treated by limbal stem cell
transplantation.
D. It is never seen in graft-versus-host disease.
Problem: absolute terms “always” and “never”
In addition to a true/false stem, the incorrect answer choices
all contain absolute terms. This allows the candidate to
eliminate answers B, C, and D.
In the treatment of herpes simplex stromal keratitis
without epithelial keratitis, oral acyclovir:
A. alone is more effective than topical
corticosteroids alone.
B. provides no beneficial effect when added to
topical corticosteroids.
C. speeds resolution of keratitis when added to
topical corticosteroids.
D. is less effective than topical trifluridine.
Problems: Mutually excusive answer choices B and C.
In addition to an unfocused stem, B and C are mutually
exclusive. If a candidate knows one is false, the other must be true.
The globe shown was enucleated 6 weeks after cataract
extraction, which was followed by rapid growth of a
subconjunctival mass at one end of the cataract incision.
Higher magnification of the cataract incision is shown. Which
of the following is the most likely diagnosis?
A. Endophthalmitis
B. Metastatic tumor from an extraocular site
C. A primary conjunctival tumor invading the eye
D. Previously unrecognized
iris and ciliary body
melanoma extending
through the cataract
wound
The globe shown was enucleated 6 weeks after
cataract ….
A. Endophthalmitis
B. Metastatic tumor from an extraocular site
C. A primary conjunctival tumor invading the eye
D. Previously unrecognized iris and ciliary body
melanoma extending through the cataract wound
Problem: Correct response and distractors should be
approximately the same length and contain the same
detail
 have
repeated phrases in the distractors
 contain
unnecessary information
 use
unnecessary medical jargon, slang,
eponyms, and/or abbreviations
Examples and exercises to help you improve questions
are provided in the training copy of this presentation
Which of the following is most likely to be associated
with crystalline lens dislocation and/or subluxation?
A. Aniridia
B. Essential iris atrophy
C. Chandler’s syndrome
D. Cogan-Reese syndrome
Problem: C and D are eponyms.
In order to include common eponyms, reformat to :
• iridocorneal endothelial syndrome (Chandler’s)
• iris nevus syndrome (Cogan-Reese)
A patient undergoes a difficult phacoemulsification. An
opening is made in the posterior capsule, necessitating a
limited anterior vitrectomy and placement of the intraocular
lens implant in the sulcus. The postoperative vision is 20/40,
but a secondary open-angle glaucoma develops that requires
a combination of a beta blocking agent, topical carbonic
anhydrase inhibitors, and latanoprost to control the
intraocular pressure. Two months after surgery, vision has
decreased to 20/200. What is the most appropriate next step?
Problem: This stem contains unnecessary information.
Rewrite this stem and compare your effort to the up-side
down re-write provided on the next slide.
Which of the following therapies is most likely to be
associated with pseudopemphigoid?
A.
B.
C.
D.
Topical administration of cromolyn sodium
Topical administration of gentamicin
Topical administration of timolol
Topical administration of trifluridine
Problem: This question contains repeated phrases.
Rewrite this question. Compare your effort to the up-side
down re-write provided there.
Negative



question stems
ask what is false, not, least, or the exception
to the rule.
do not perform well. Statistics show that
knowledgeable test takers perform poorly
on negatively worded questions
are not useful for assessment and act much
like “trick questions”
Negative
 The
question stems
ABO does not use negative questions
 The
AAO is slowly eliminating negative
questions from its programs and will not
accept new, negative items
 “Contraindicated” is
an acceptable use of a
negative only when testing clearly
contraindicated medications or procedures.
 Questions
should not be intentionally
misleading

“Trick” questions are not good. They
tend to test savvy or reading skills
instead of clinical competence
Examples of unacceptable questions follow this slide in the
training copy of this presentation.
Which of the following agents is LEAST
effective in the treatment of ocular disease?
A. Thiabendazole
B. Clindamycin
C. Ganciclovir
D. Acyclovir
When planning injection of an eyelid hemangioma in an
infant, which of the following systemic consequences
must be considered?
A. Adrenal insufficiency
B. Central retinal artery occlusion
C. Fat necrosis
D. Skin necrosis
This is a trick question because B also is correct,
although not “systemic” as stated in the stem.
In
the following examples,
determine whether the format
shown is acceptable or
unacceptable for ABO
examinations
Which of the following is true regarding the
oculomotor nucleus?
□ Acceptable
□ Unacceptable
This question does not give candidates any idea what the correct
response might be until all of the choices are read. Unacceptable.
A persistent, clear fluid encountered when
removing bone superior to the medial canthal
tendon during a dacryocystorhinostomy most
likely would result from inadvertent penetration
of which of the following structures?
□ Acceptable
□ Unacceptable
This question provides sufficient information for a candidate to formulate a
response and quickly find it among the choices. Acceptable.
Which of the following histocompatability
leukocyte antigens is least likely to be
associated with retinal vasculitis of noninfectious
origin?
□ Acceptable
□ Unacceptable
This question asks candidates to identify what is NOT correct. Unacceptable.
Which of the following findings is most likely
to be associated with meibomian gland
dysfunction?
□ Acceptable
□ Unacceptable
This question directs candidates to look for the MOST likely
finding. Acceptable.
As compared to a normal rigid contact
lens, a translating bifocal rigid contact lens
does not have:
□ Acceptable
□ Unacceptable
This question is both unfocused AND
negatively worded. Unacceptable.
 Examples
of items that are well-written.
e.g., have good question stems and
answer choices, are provided in the
hard-copy of this hand-out after this slide.
 Prophylactic
laser photocoagulation of
asymptomatic lattice degeneration is most
indicated for a patient with which of the following
findings?
A. High myopia
B. Preoperative cataract
C. Pseudophakia
D. Retinal detachment in fellow eye
Focused question that asks candidates to recognize the
significance of retinal detachment in lattice degeneration
 Which
of the following is the most likely
diagnosis for the condition shown?
A. Chloroquine retinopathy
B. Cone dystrophy
C. Dominant drusen
D. Fundus flavimaculatus
Interpretive question that tests candidate’s ability to
recognize and diagnose conditions
Which of the following is the most appropriate
management of a corneal perforation that
remains unsealed after 3 days of aqueous
suppression therapy?
A. Aminocaproic acid
B. Autologous plasmin
C. Cyanoacrylate glue
D. Topical antibiotic
This question tests whether candidates know the most
appropriate accepted management for the problem.


References are required for every American Board of
Ophthalmology (ABO) exam item and for every OKAP
exam item.
References must be complete in order to claim CME
credit for high-stakes examination item-writing
√ BCSC section 9, 2006-07, p. 52
√ Mamalis n et al, journal of cataract and refractive surgery 2006;
32: 324-333.
Unacceptable: BCSC
Unacceptable: Shields, The Glaucomas
 Please
speak to your staff representative
from the ABO or AAO as appropriate if
you write items for a high-stakes
examination. They will provide you
appropriate information .