MCQ’s 2:
Technical Flaws to Avoid
MCQ Check-list
This presentation will focus on:
 Why we write them
 What type is best
 What flaws are to be avoided
 What content to test
 How to construct it

Writing MCQ’s
At the end of this session, the participant
will be able to
 Describe the structure of an MCQ
 List the technical flaws common to MCQ’s
Why do we test?

Formative assessment



Summative assessment



Communicate to the students what material is
important
Identify areas of deficiency in need of remediation or
further learning
Determine final grades
Make promotion decisions
Both


Motivate students to study
Identify areas where the course/curriculum is weak
Anatomy of an MCQ Item



Stem
Lead-in
Options


One correct answer
Distractors
Types of MCQ’s
True/False family
 Assess recall of isolated facts



Stems must be clear and unambiguous
Options must be absolutely true or false or
examinee must decide how true options are
Generally not recommended – usually
have subtle flaws
Types of MCQ’s: True/False
Which of the following statements is true regarding
contraception?
A)
B)
C)
D)
E)
oral contraceptives should not be initiated until six
weeks after therapeutic abortion to avoid the risk of
hypercoagulability
an intrauterine device can provide good contraceptive
protection 5 years after insertion
oral contraceptives should not be prescribed to
individuals under the age of 15 without parental
consent
lesbian women do not require counseling about
contraception or sexually transmitted diseases
oral contraceptives should not be prescribed for women
over the age of 35
Types of MCQ’s
One-Best-Answer family (A type)
 Options may be diagrammed:
D C A
Least correct


E
B
Most correct
Correct answer is the most likely correct
Better application of knowledge, integration,
synthesis and judgment
Types of MCQ’s:
One-Best-Answer family
A 24 year old university student came to see you because
she is missing several days of school each month because
of severe dysmenorrhea unresponsive to the OCP and
NSAIDS. She also complains of deep dyspareunia and has
a tender uterosacral nodule on pelvic examination. Which
one of the following conditions is most likely responsible
for her symptoms?
A)
B)
C)
D)
Chronic pelvic inflammatory disease
Endometriosis
Adenomyosis
Fibroids
MCQ ‘Items’


Should test important information – either
common or life-threatening
Should be well-structured


Flaws benefit test-wise examinees
Irrelevant difficulty does not test examinees’
specific knowledge
MCQ Technical Flaws


Issues Related to Test-Wiseness
Issues Related to Irrelevant Difficulty
MCQ Technical Flaws

Issues Related to Test-Wiseness



provide clues to the right answer
examinee can eliminate options based on
flaws and not on knowledge
Issues Related to Irrelevant Difficulty
Item Flaws: Test-wiseness






Grammatical cues
Logical cues
Absolute terms
Long correct answer
Word repeats
Convergence strategy
Item Flaws: Test-wiseness
Grammatical cues
 One or more distractors don’t follow
grammatically from the stem
Item Flaws: Test-wiseness
Grammatical cues
A 24 year old patient presents with a one year
history of dysmenorrhea, mid-cycle pain,
dyspareunia, and painful bowel movements
with her periods. The most likely diagnosis is:
A)
B)
C)
D)
Fibroid uterus
Adenomyosis
Endometriosis
Psychosomatic condition
Item Flaws: Test-wiseness
Logical cues
 A subset of the options are collectively
exhaustive
Item Flaws: Test-wiseness
Logical cues
Arterial blood gases in a normal pregnant
woman, compared to her non-pregnant state
may show which of the following?
A)
B)
C)
D)
E)
Decreased pO2
Increased pCO2
Decreased pCO2
Decreased pH
No difference in pCO2
Item Flaws: Test-wiseness
Logical cues
 One or more distractors don’t follow
logically from the stem
Item Flaws: Test-wiseness
Logical cues
An important tool for assessing fetal well-being is
the Non Stress test. A well baby demonstrates
the following fetal heart rate pattern:
A)
B)
C)
D)
E)
A baseline heart rate above 120 bpm
Evidence of fetal movement
Baseline variability
Two fetal heart rate accelerations in 20 minutes
Fetal breathing movements
Item Flaws: Test-wiseness
Absolute terms
 Always, never
 Not a problem if stem is long and lead-in
asks a precise question
Item Flaws: Test-wiseness
Absolute terms
In patients with a previous Cesarean section:
A)
B)
C)
D)
E)
Vaginal birth is NEVER an option
Repeat Cesarean section is less dangerous to the
mother
The uterine scar may rupture during labour
Care should ALWAYS be supervised by an
obstetrician
No further pregnancies should EVER be considered
Item Flaws: Test-wiseness
Long correct answer
 Correct answer is longer, more specific, or
more complete than other options
Item Flaws: Test-wiseness
Long correct answer
Which of the following problems in vaginal
breech delivery is most likely to cause serious
long-term complications?
A)
B)
C)
D)
E)
Failure to descend
Cord prolapse
Nuchal arms
Entrapment of the after coming head with an
incompletely dilated cervix
1 minute Apgar score of 4
Item Flaws: Test-wiseness
Word repeats
 A word or phase is included in the stem
and in the correct answer
Item Flaws: Test-wiseness
Word repeats
A 24 year old with a previous history of
chronic PID presents with primary infertility.
Select the procedure that would be most
appropriate to evaluate the status of her
tubes and pelvic anatomy
A)
B)
C)
D)
Post-coital test
Endometrial biopsy
Hysterosalpingogram
Laparoscopy with dye insufflation of the
fallopian tubes
Item Flaws: Test-wiseness
Convergence strategy
 The correct answer includes the most
elements in common with the other
options
Item Flaws: Test-wiseness
Convergence strategy
Normal bladder neck mobility is measured by:
A)
B)
C)
D)
A Q-tip angle measurement under 40 degrees
A Q-tip angle measurement under 30 degrees
A bladder neck ultrasound angle less than 30
degrees
A Q-tip angle measurement under 20 degrees
MCQ Technical Flaws

Issues Related to Test-Wiseness

Issues Related to Irrelevant Difficulty

do not assess knowledge of intended topic
Irrelevant Difficulty







Options are long, complicated or double
Extraneous details in stem
Numeric data not stated consistently
Vague frequency terms
Non-parallel language in options
Stems are tricky or unnecessarily complicated
‘None of the above’
Irrelevant Difficulty
Options are long, complicated or double
Extraneous details in stem
 Shifts difficulty from content knowledge to
speed reading
Irrelevant Difficulty
Options are long, complicated or double
Mrs. Brown is a 70 year old woman who had a stroke 6
months ago and is still impaired. Her daughter tells you that
her mother’s recent urinary incontinence makes her care
unmanageable. She is seeking permanent placement. Mrs.
Brown complains of urinary frequency, nocturia, and urge
incontinence. Which of the following statements is true?
A)
B)
C)
D)
E)
Neurological conditions are involved in the majority of women with
symptoms like those presented by Mrs. Brown
Diabetes has a greater impact on quality of life than Mrs. Brown’s
condition.
The pharmacological blockage of muscarinic receptors in the
detrusor muscle controls symptoms compatible with Mrs. Brown’s
condition
Overactive bladder is defined as urinary incontinence accompanied
by or immediately preceded by urgency
Alpha stimulants are effective in treating this condition
Irrelevant Difficulty
Extraneous details in stem
You see in your office a 24 year old woman for her routine
postpartum visit, at 6 weeks. Her baby is irritable and cries “all the
time’ (including during her visit.) As she is changing the diaper
during her visit, you notice that the child has a very red bum but
upon questioning, she tells you she did not notice anything
abnormal. Her partner is away in Afghanistan and she does not
have a lot of friends, as they just recently were posted in Kingston.
She states that she has a hard time getting dressed in the morning,
cries easily and feels like she does not love her child. She denies
any medical or psychiatric history. Her exam is essentially normal.
She weighs 2 kg less than at her first antenatal visit at 10 weeks.
You most likely clinical diagnosis is:
A)
Postpartum hypothyroidism
B)
Hyperthyroidism
C)
Postpartum blues
D)
Postpartum depression
E)
Psychotic depression
Irrelevant Difficulty
Numeric Data
 Not stated consistently
 Mixed format
 Illogical order
Irrelevant Difficulty
Numeric Data not stated consistently
Following a second episode of salpingitis, what
is the likelihood that a woman is infertile?
A)
B)
C)
D)
E)
20 – 30 %
Less than 20%
Greater than 50 %
90%
75%
Irrelevant Difficulty
Vague frequency terms
 Often, usually, rarely, frequently
 Not consistently defined
*may be used in stem if otherwise carefully
constructed unambiguous question
Irrelevant Difficulty
Vague frequency terms
Gonorrhea is
A) sometimes asymptomatic
B) treated with cefixime 450 mg orally
C) caused by a gram positive intracellular
diplococcus
D) diagnosed by a DNA test
E) usually associated with vulvar ulcers
Irrelevant Difficulty


Non-parallel language in options
Non-logical order
Irrelevant Difficulty
Non-parallel language in options
In a vaccine trial, 200 2-year-old boys were given a
vaccine against a certain disease and then
monitored for five years for occurrence of the
disease. Of this group, 85% never contracted the
disease. Which of the following statements
concerning these results is correct?
A)
No conclusion can be drawn, since no follow-up
was made of non-vaccinated children
B)
The number of cases (ie 30 cases over five year)
is too small for statistically meaningful
conclusions
C)
It is not possible to draw a conclusion because
the trial involved only boys
D)
Vaccine efficacy (%) is calculated as 85-15/100
Irrelevant Difficulty
“None of the above”
 Problematic where judgment is involved or
if options not absolutely true or false
 Turns question into true/false question
 Confuses very knowledgeable students
Irrelevant Difficulty
“None of the above”
The main reason that fertility is reduced in a
woman in her early 40’s is:
A)
B)
C)
D)
E)
Irregular cycles make it hard to time intercourse
Chromosomal damage adversely affects oocytes’
genetic complement
Male erectile difficulties increase
Hormone imbalance leads to poor endometrial
development
None of the above
Irrelevant Difficulty
Stems are tricky or unnecessarily complicated
Arrange the parents of the following children with Down’s
syndrome in order of the highest to lowest risk of recurrence.
Assume that the maternal age in all cases is 22 years and that a
subsequent pregnancy occurs within 5 years. The karyotypes of
the daughters are:
I: 46, XX, –14,+T (14q21q) pat
II: 46, XX, –14,+T (14q21q) de novo
III: 46, XX, –14,+T (14q21q) mat
IV: 46, XX, –21,+T (14q21q) pat
V: 46, XX, –21,+T (21q21q) (parents not typed)
A) III, IV, I, V, II
B) IV, III, V, 1, II
C) III, I, IV, V, II
D) I, V, III, I, V, II
E) III, IV, I, II, V
MCQ Check-list
Why we write them
 What type is best
 What flaws are to be avoided
 What content to test
 How to construct it

General Guidelines for Item
Construction



Include as much of the item as possible in the
stems: Long stems, short options
Avoid negatively phrased items (except or not in
the lead-in). If you must, use only with short
options
Avoid writing questions of the form:


Which of the following statements is correct?
Each of the following statements is correct EXCEPT


Unfocussed
Heterogeneous options
Basic Rules for MCQ’s (A type)*





Focus item on important concept
Item should assess application of
knowledge, not recall of an isolated fact
Stem must pose clear question – ‘Cover
the options’ test
All distractors should be homogeneous
Avoid technical item flaws
Reference
Constructing Written Test Questions For the
Basic and Clinical Sciences

Case & Swanson, NBME 1996
One last step…

Get a colleague to read over your
questions




Detects more than one right answer
Discovers the ‘what am I thinking?’ question
Identifies not enough information in the stem
…