Carolyn L. Cambor, M.D.
Department of Pathology
Jennifer R. Kogan, M.D.
Department of Medicine
FAPD Series
September 29, 2010
Materials adapted from the National Board of Medical Examiners
•
•
•
•
•
Describe the item types currently used by the NBME & the rationale for their use
•
•
Describe the steps in writing A-type items
Basic Sciences
Clinical Sciences
Identify technical flaws in test items & understand the importance of avoiding them
Advance your skills in writing A-type test items
If time permits, describe and practice writing
R-type test items
2
Test purpose
Testing time and method of administration
Test standardization
Test content
Number of items
Item format
Develop items
Item selection and evaluation
3
Why learn how to write test items?
Ensure that your items are:
Clear & understandable
Test what you want to test
Fair
Provide reproducible results
Avoid technical “flaws”
Create confusion
Add unnecessary difficulty
May aid the “testwise student”
4
“Alphabet Soup” of Item Types
True-false
X (simple true/false)
K (complex true/false) “1,2,3”, “1,3”, “2,4” “all”
C (A/B/Both/Neither)
In general, not recommended; not used in Mod1,2
If used, must be clear, unambiguous with options that are
100% T or 100% F
One-best answer
A (4 or more options)
B (4 or 5 option matching sets in sets of 2 –5 items)
R (extended matching items in sets of 2-20 items)
5
Stem Longest part of the item
Sufficient information to answer the lead-in question.
Lead-in
Options
The question being asked
The correct answer and
3-4 distractors
6
A-type Item
Stem: A 65-year-old man has difficulty rising from a seated position and straightening his trunk, but he has no difficulty flexing his leg.
Lead-in: Which of the following muscles is most likely to have been injured?
Options: A. Gluteus maximus*
B. Gluteus minimus
C. Hamstrings
D. Iliopsoas
E. Obturator internus
Answer
Distractors
7
Good, A-Type Items Have a Distinctive Shape
Stem
Lead In
Option A (distractor)
Option B (answer)
Option C (distractor)
Option D (distractor)
Option E (distractor)
8
Short stem & lead-in a. Long option b. Long option c. Long option d. Long option
9
6 Rules for Writing A-type Items
Basic Sciences
1) Focus on important topics.
Ideally based on your objectives.
2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees.
3) Assess application of knowledge, not recall.
4) Provide sufficient information in the stem and pose a clear question in the leadin. “Cover the options rule”
5) Use homogeneous distractors
6) Avoid technical flaws and unnecessary difficulty
10
6 Rules for Writing A-type Items:
Basic Sciences
1) Focus on important topics.
Ideally based on your objectives.
2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees.
3) Assess application of knowledge, not recall.
4) Provide sufficient information in the stem and pose a clear question in the leadin. “Cover the options rule”
5) Use homogeneous distractors
6) Avoid technical flaws and unnecessary difficulty
11
6 Rules for Writing A-type Items:
Basic Sciences
1) Focus on important topics.
Ideally based on your objectives.
2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees.
3) Assess application of knowledge, not recall.
4) Provide sufficient information in the stem and pose a clear question in the leadin. “Cover the options rule”
5) Use homogeneous distractors
6) Avoid technical flaws and unnecessary difficulty
12
What area of the brain is supplied by blood from the posterior inferior cerebellar artery?
A) Location 1
B) Location 2
C) Location 3
D) Location 4
Recall item: rote memory of isolated fact
13
A 62-year-old man develops left sided limb ataxia,
Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations.
Which artery is most likely to be occluded?
A) Artery A
B) Artery B
C) Artery C
D) Artery D
Application of knowledge:
Reach a conclusion, make a prediction, select a course of action
14
6 Rules for Writing A-type Items:
Basic Sciences
1) Focus on important topics.
Ideally based on your objectives.
2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees.
3) Assess application of knowledge, not recall.
4) Provide sufficient information in the stem and pose a clear question in the leadin. “Cover the options rule”
5) Use homogeneous distractors
6) Avoid technical flaws and unnecessary difficulty
15
16
A 62-year-old man develops left sided limb ataxia,
Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations.
Which artery is most likely to be occluded?
A) Artery A
B) Artery B
C) Artery C
D) Artery D
17
A 62-year-old man develops left sided limb ataxia,
Horner’s syndrome, nystagmus, and loss of appreciation of facial pain and temperature sensations.
Which artery is most likely to be occluded?
A) Artery A
B) Artery B
C) Artery C
D) Artery D
18
Avoid
“Which of the following statements is correct?”
“Each of the following statements is correct
EXCEPT…”
Because
Unfocused stems
Don’t follow “cover the options” rule
Heterogeneous options mix of epidemiology, genetics, mechanisms etc
19
True statements about cystic fibrosis (CF) include:
1. The incidence of CF is 1:2000
2. Children with CF usually die in their teens.
3. Males with CF are sterile
4. CF is an autosomal recessive disease
20
Patient vignettes are good for “stems”
Patient vignettes (stem)
brief for basic sciences
• should not require patient care expertise
• age gender site of care presenting complaint
duration
• +/- family history
•
+/- results of diagnostic tests; initial treatment, etc material needs to be taught with clinical relevance
Lead in
Clearly focused question
21
Patient Vignette Templates: Basic Sciences
A (patient description) has a (type of injury, location).
Which of the following structures is most likely to be affected?
A (patient description) has (signs & symptoms).
These observations suggest that the disease is a result of the (absence or presence) of which of the following (enzymes, mechanisms) ?
A (patient description) dies of (disease).
Which of the following is the most likely finding on autopsy?
22
Patient Vignette Templates: Basic Sciences
Which of the following is the most likely mechanism of the therapeutic effect of this (drug class) in patients with (disease) ?
(time period) after a (event, such as a trip or meal w/ certain foods) a (patient or group description) became ill with (symptoms and signs).
Which of the following (organisms, agents) is most likely to be found on analysis of (food)?
23
During an operation, the arterial PCO2, and pH of an anesthetized patient are monitored. The patient is being ventilated by a mechanical respirator, and the initial values are normal (PCO2=40 mm Hg; pH=7.42).
If the ventilation is decreased, which of the following is most likely to occur?
Arterial pCO2
A. decrease
B. decrease
C. decrease
D. increase *
E. increase
F. increase pH
A. decrease
B. increase
C. no change
D. decrease *
E. increase
F. no change
24
At a banquet, the menu included fried chicken, homefried potatoes, peas, chocolate eclairs and coffee. Within 2 hours most of the diners became violently ill, with nausea, vomiting and abdominal pain.
Analysis of the contaminated food is most likely to yield large numbers of which of the following organisms?
A.
B.
C.
D.
E.
Escherechia coli
Proteus mirabilis
Salmonella typhimurium
Staphylococcus aureus*
Streptococcus faecalis
25
Lab Vignettes are good for Basic Sciences
“Lab”-vignettes
laboratory examples
research examples
biochemical pathways
drug metabolism
26
A B
III
C
IV
D
I
II
V
E
IV
In the branched metabolic pathway, a different single enzyme catalyzes each of the individual steps. The enzyme that would be expected to be most severely inhibited by compound V is enzyme :
A. A
B. B *
C. C
D. D
E. E 27
Drug Y has a volume of distribution of 75 L in both younger and older adult men. In younger adults, it has a clearance rate of
15L / h, 50% of which is via the liver and 50% via the kidneys. For younger men, the maintenance regimen is 100 mg every 6 hours.
Which of the following regimens will produce essentially the same steady-state concentration in an older man, whose creatinine clearance is half that of younger men, but whose hepatic function is unimpaired?
A.
75 mg every 3 hrs
B.
75 mg every 6 hrs *
C.
75 mg every 9 hours
D.
100 mg every 3 hours
E.
100 mg every 6 hours
F.
100 mg every 12 hours
28
Basic Sciences: Integrated Items
Physiology / Pharmacology
Microbiology / Pharmacology
Pathology / Pharmacology
Pathology / Pathophysiology
Stem describes one component, options ask about another
1 stem is used for 2+ questions, on different topics
Guidelines:
Team preparation
Avoid “cueing”
Avoid “hinging”
29
Integrated Items - Example
An unresponsive 58 year old woman is brought to the ED after collapsing at a local shopping center.
More history. Physical signs describe a neurologic deficit.
1.
The dilated, unreactive left pupil is most consistent with injury to the left
List of anatomic structures
2.
The extensor posturing on the right is most consistent with injury to the left
List of anatomic structures
3.
Which of the following herniation syndromes is most consistent with her clinical presentation?
List of herniation syndromes
30
Patient Vignettes: Clinical Sciences
Include
– age, gender
–
– site of care presenting complaint
–
–
–
–
– duration patient history physical findings
+/- diagnostic studies
+/- initial treatment
Stems should
– not be completely based on real patients
–
–
– include reference material when it would be realistic in practice not use the patient’s or doctor’s own words not include patients who lie
31
Non-vignettes in the Clinical Sciences
The most likely renal abnormality in children with nephrotic syndrome and normal renal function is
A.
Acute poststreptococcal glomerulonephritis
B.
Hemolytic-uremic syndrome
C.
Minimal change disease
D.
Focal and segmental glomerulosclerosis
E.
Schonlein-Henoch purpura
A B C D
Hi 1
Lo 8
0
1
99
90
0
1
E
0
0
32
A 2 year old boy has a 1 week history of edema. His blood pressure is 100/60 mmHg and there is generalized edema and ascites. Labs show Cr 0.4
mg/dL, albumin 1.4 g/dL and cholesterol of 569 mg/dL.
UA shows 4+ protein and no blood. The most likely diagnosis is
A B
Hi 0
Lo 5
0
2
C D
98 2
82 8
E
0
1
33
A 2 year old black child developed swelling of his eyes and ankles over the past week. Blood pressure is 100/60 mmHg, pulse 110/min respirations 28/min. Exam shows swelling of eyes, abdominal distention and a positive fluid wave.
Labs show Cr 0.4 mg/dL, albumin
1.4 g/dL and cholesterol of 569 mg/dL.UA shows 4+ protein and no blood. The most likely diagnosis is
A
Hi 0
B
1
Lo 10 9
C D
98 1
66 10
E
0
5
34
Writing the Lead-In: Clinical Sciences
Diagnosis
Which of the following is the most likely diagnosis?
Which of the following is the most appropriate next step in diagnosis?
Management
Which of the following is the most appropriate next step in patient care?
Mechanisms of disease
Which of the following is the most likely pathogen?
Which of the following is the most likely explanation for the findings?
35
6 Rules for Writing A-type Questions
1)
Focus on important topics.
Ideally based on course / activity objectives.
2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees.
3)
4)
Assess application of knowledge, not recall.
Provide sufficient information in the stem and pose a clear question in the leadin. “Cover the options rule”
5) Use homogeneous distractors
6) Avoid technical flaws and unnecessary difficulty
36
Writing the Options
Homogeneous in content
Grammatically consistent with stem
Similar in construction and length
Alphabetized or logically ordered
Incorrect/inferior to the correct answer
choice affects question difficulty
Plausible/attractive to uninformed
no distracter should be obviously incorrect
Each should be chosen by some
37
Who was the primary author of the
Declaration of Independence?
A. Abraham Lincoln
B. Thomas Jefferson*
C. Franklin Roosevelt
D. King George II
E. Catherine the Great
38
Who was the primary author of the
Declaration of Independence?
A. George Washington
B. Thomas Jefferson*
C. Alexander Hamilton
D. Benjamin Franklin
E. James Madison
39
6 Rules for Writing A-type Questions
1) Focus on important topics.
Ideally based on course / activity objectives.
2) Pose problems, or clinical decision-making tasks that are within the education/experience of examinees.
3) Assess application of knowledge, not recall.
4) Provide sufficient information in the stem and pose a clear question in the leadin. “Cover the options rule”
5) Use homogeneous distractors
6) Avoid technical flaws and unnecessary difficulty
40
Technical Item Flaws
Issues Related to
“Testwiseness”
Issues related to
Irrelevant Difficulty
41
Testwise Analysis
The primary purpose of the stam is to remove the carm
denton menace stam bar
43
The primary purpose of the stam is to remove the carm
denton menace stam bar
“Word repeats” in stem and answer
44
Which of the following pairs has won the greatest number of Abby awards?
Jones and Smith
Smith and Taylor
Smith and White
White and Allen
45
Which of the following pairs has won the greatest number of Abby awards?
Jones and Smith
Smith and Taylor
Smith and White
White and Allen
Convergence: correct answer has the most in common with other choices
46
How many pounds of pressure are exerted by a callam?
2.6
150
260
2600
47
How many pounds of pressure are exerted by a callam?
2.6
150
260
2600
Convergence: Correct answer has the most in common with other choices
48
The stanon is aided by a instel immon octal port
49
The stanon is aided by a instel immon octal port
Grammatical cues: only “port” follows grammatically from the stem
50
The stanon frequently overheats because all grestels are bilious no immon are directly fectitious octals are usually casable ports are always critical
51
The stanon frequently overheats because all grestels are bilious no immon are directly fectitious octals are usually casable ports are always critical
Absolute terms: never found in the correct answer
52
Stammation normally occurs when the anstels rupture immon falls and the denton is in place octal rotates easily ports pass over the carm
53
Stammation normally occurs when the anstels rupture immon falls and the denton is in place octal rotates easily ports pass over the carm
Longest answer is usually the correct answer
54
Testwiseness: Grammatical Cues
The option(s) does not flow from the stem
Example
The minor differences among organisms of the same kind are known as
A. Heredity
B. Variations
C. Adaptation
D. Natural selection
55
Testwiseness: Logical Cues
A subset of options are collectively exhaustive
Example
Crime is
A. Equally distributed among the social classes
B. Overrepresented among the poor
C. Overrepresented among the middle class and rich
D. Primarily an indication of psychosexual maladjustment
E. Reaching a plateau of tolerability for the nation
56
Testwiseness: Absolute Terms
Terms such as ‘always’ or ‘never’ are used in options
Example
In patients with advanced dementia, Alzheimer’s type, the memory defect
A. Can be treated adequately with lecithin
B. Could be a sequela of early parkinsonism
C. Is never seen in patients with neurofibrillary tangles
C. Is never severe
D. Possibly involves the cholinergic system
57
Testwiseness: Long Correct Answer
The correct answer is longer, more specific, or more complete than the other options.
Example:
Secondary gain is
A. Synonymous with malingering
B. A frequent problem in obsessive-compulsive disorder
C. A complication of a variety of illnesses and tends to prolong many of them
D. Never seen in organic brain damage
58
Testwiseness: Word Repeats
A word or phrase is included in the stem and correct answer.
Example:
A 58-year-old man with a history of heavy alcohol use and previous psychiatric hospitalization is confused and agitated.
He speaks of experiencing the world as unreal. This symptom is called
A. Depersonalization
B. Derailment
C. Derealization*
D. Focal memory defect
59
Testwiseness: Convergence
The correct answer includes the most elements in common with the other options
Example:
Local anesthetics are most effective in the
A. Anionic form, acting from inside the nerve membrane
B. Cationic form, acting from inside the nerve membrane*
C. Cationic form, acting from outside the nerve membrane
D. Uncharged form, acting from inside the nerve membrane
E. Uncharged form, acting from outside the nerve membrane
60
Technical Item Flaws
Issues Related to
“Testwiseness”
Issues related to
Irrelevant Difficulty
61
Irrelevant Difficulty :
Options are long, complicated or doubled
Systematic geography differs from regional geography in that
A.
Systematic geography deals, in the main, with physical geography, while regional geography concerns itself essentially with the field of human geography
B. Systematic geography studies a region systematically while regional geography is concerned only with descriptive account of a region
C. Systematic geography studies a single phenomenon in its distribution over the earth in order to supply generalizations for regional geography, which studies the arrangement of phenomena in one given area*
62
Irrelevant Difficulty:
Numeric data are not stated consistently
Following a second episode of infection, what is the likelihood that a woman is infertile?
A. Less than 20%
B. 20% to 30%
C. Greater than 50%
D. 90%
E. 75%
63
Irrelevant Difficulty:
Frequency terms in options are vague
Severe obesity in early adolescence
A.
Usually responds dramatically to dietary regimens
B.
Often is related to endocrine disorders
C.
D.
E.
Has a 75% change of clearing spontaneously
Shows a poor prognosis
Usually responds to pharmacotherapy and intensive psychotherapy
64
Irrelevant Difficulty:
Language in options not parallel
In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and then monitored for five years for occurrence of disease. Of this group, 85% never contracted the disease.
Which of the following statements concerning these results is correct?
A.
No conclusions can be drawn since no follow-up was made of non-vaccinated children
B.
C.
D.
The number of cases (I.e. 30 cases over five years) is too small for statistically meaningful conclusions
No conclusions can be drawn because the trial involved only boys
Vaccine efficacy (%) is calculated as 85-15/100
65
Irrelevant Difficulty:
Options in an non-logical order
The population of Denmark is
A. 2 million
B. 15 million
C. 4 million
D. 7 million
66
Irrelevant Difficulty:
“None of the Above” used as option
Which city is closest to New York City?
A. Boston
B. Chicago
C. Dallas
D. Los Angeles
E. None of the above
67
Irrelevant Difficulty:
Complicated Stems and Options
Arrange the parents of the following children with Down’s syndrome in order of highest to lowest risk of recurrence.
Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years.
The karyotypes of the daughters are (see next slide)
68
I. 46, XX, -14, +T (14q21q)pat
II.
46, XX, -14, +T(14q21q) de novo
III.
46, XX, -14, +T (14q21q) mat
IV.
46, XX, -21, +T (14q21q)pat
V.
47, XX, -21, +T (21q21q) parents not typed
A.
III, IV, I, V, II
B.
IV, III, V, I, II
C.
III, I, IV, V, II
D.
IV, III, I, V, II
E.
III, IV, I II, V
69
“Window Dressing”
“Red Herrings”
70
Which one of the following sets of laboratory studies is most consistent with a diagnosis of anemia of chronic inflammation?
(Inc means increased level, Dec means decreased level)
Ferritin TIBC Serum Iron Soluble Transferrin Receptor
A. Inc Inc Dec Dec
B. Dec Inc Dec Inc
C. Inc Dec Dec Inc
D. Inc Dec Dec Normal
72
A 16 year old girl presents with a sore throat of 4 months duration. She describes feeling a lump in her throat.
Physical exam reveals a 1 cm cystic lesion at the base of the tongue.
This developmental lesion most likely arises as a remnant of which of the following anatomic structures?
A.
Auditory tube
B.
Branchial arches
C.
Facial fusion lines
D.
Rudimentary thymus
E.
Thyroglossal duct
73
All of the carbons in cholesterol are derived from:
A.
B.
C.
D.
E.
choline sphingosine acetyl CoA
HMG-CoA reductase
CO2
74
What is the most likely inheritance pattern of the following pedigree?
A.
Autosomal recessive
B.
Autosomal dominant
C.
X-linked recessive
D.
X-linked dominant
E.
Mitochondrial
75
A 64 year old male is admitted to the hospital with 3 days of progressive shortness of breath. He has a medical history of poorly controlled hypertension. He denies tobacco or alcohol use. On physical examination he is afebrile, pulse 100, respiratory rate 26, blood pressure 180/110 mmHg and room air oxygen saturation 91%.
Examination of the lungs reveals bilateral crackles and his cardiac exam reveals a diffuse, laterally displaced point of maximal impulse, a soft S3 and loud S4. Abdominal exam and extremity exams are normal.
A chest Xray reveals bilateral alveolar infiltrates and an electrocardiogram is remarkable for left ventricular hypertrophy. Which of the following is the next best step in management?
A.
Administer intravenous furosemide for diuresis
B.
Administer oral morphine
C.
Start oral beta blocker
D.
Administer chewable aspirin
E.
None of the above
77
A 72 year old female presents to her physician with progressive bilateral knee pain for 3 months. The pain is worse with ambulation and increases in severity during the day. She denies trauma or pain in other joints. On physical examination she is afebrile. Her body mass index is
31. Her knees are warm bilaterally but without effusion. There is bilateral crepitus. There is no instability. The remainder of her examination is normal. Plain xrays of the knees demonstrates narrowing of the joint space, marginal osteophytes and subcondral sclerosis. Which of the following is the most likely diagnosis?
A. Osteoporosis
B. Osteoarthritis
C. Osteogenesis imperfecta
D. Rheumatoid arthritis
E. Pseudogout
78
A 42 year old female presents to her physician for a routine medical examination. She has no specific concerns. Her blood pressure is noted to be 170/80 mm Hg. Which of the following sounds auscultatory findings corresponds to the systolic and diastolic readings?
A.
The systolic reading is the first appearance of the Korotkoff sounds and the diastolic reading is the disappearance of the
Korotkoff sounds
B.
C.
The systolic reading is the regular appearance of the Korotkoff sounds and the diastolic reading is the muffling of the Korotkoff sounds.
The systolic reading is the regular appearance of the Korotkoff sounds and the diastolic reading is the disappearance of the
Korotkoff sounds
D.
The systolic reading is the first appearance of the Korotkoff sounds and the diastolic reading is the muffling of the Korotkoff sounds.
79
A 42 year old female presents to her physician for a routine medical examination. She has no specific concerns. Her blood pressure is noted to be 170/80 mm Hg.
Which of the following sounds best corresponds to the blood pressure reading?
170
A.
First Korotkoff sound
80
Disappearance of Korotkoff sounds
B.
Regular Korotkoff sounds Muffling of Korotkoff sounds
C.
Regular Korotkoff sounds Disappearance of Korotkoff sounds
D.
First Korotkoff sounds Muffling of Korotkoff sounds .
80
Extended Matching: R-type Items
Four components
1.
2.
3.
4.
A theme
An option list
A lead-in statement
At least 2 item stems
83
R-type itemTheme: Cerebrovascular anatomy
A. Left anterior cerebral artery E. Left posterior cerebral artery
B. Right anterior cerebral artery F. Right posterior cerebral artery
C. Left middle cerebral artery G. Left lenticulostriate arteries
D. Right middle cerebral artery H. Right lenticulostriate arteries
For each patient with neurological abnormalities presented below, select the artery that is most likely to be involved from the list above.
1. A 72-year-old right-handed man has weakness and hyperreflexia of the right lower limb, an extensor plantar response on the right, normal strength of the right arm, and normal facial movements. Answer: A
2. A 68-year-old right-handed man has right spastic hemiparesis, an extensor plantar response on the right, and paralysis of the lower two-thirds of his face on the right. His speech is fluent, and he has normal comprehension of verbal and written commands. Answer: G
84
Sample Lead-ins and Topics for Option Lists
For each of the following patients, select the most likely ( cause ).
Underlying mechanism of disease, medications, toxic agents…
For each of the following patients select the (eg, nerve) that is most like to be (abnormal, defective, deficient, nonfunctioning).
-lists of nerves, muscles, enzymes, hormones, proteins, types of cells, pathologic processes, neurotransmitters
For each of the following patients, select the ( finding ) that would be expected.
Laboratory results, physical signs…
85
More sample lead-ins & topics for option lists
For each of the following patients, select the
[ eg, drug] that should be administered.
For each of the following patients with [chief complaint], select the most likely diagnosis.
For each of the following patients, select the most appropriate next step in patient care.
(drugs, lab tests, disposition choices)
86
Arteries
Nerves
Muscles
Amino acids
Peptides
Hormones
Enzymes
Cell components
Cell types
Blood components
Diagnoses
Karyotypes
Proteins
Lipids
Pathogens
Cytokines
Anatomic structures
Metabolic defects
Organelles
Drugs/drug classes
ECM components
Diagnostic tests
87
Writing the Item Stems for R-Type Items
Patient vignettes
Use for basic & clinical science questions
Homogeneous in construction
Include same demographic features in each one
All the same age group
Make sure only one best answer and usually
4 distractors
88
Identify the theme first
Write the lead-in statement
List all the options
Write the vignettes (stems)
Review the item
Be sure there is only ONE single best answer for each stem
Be sure each item has several distractors
89
Theme: Fatigue
Lead in: For each patient with fatigue, select the most likely diagnosis.
Options: Many causes of fatigue (anemia,
CHF, depression, infections etc)
Vignettes: 2 or more that correspond to diagnoses in your list
90