Question Writing Guide
ABNM SAM Questions
J. Anthony Parker, MD PhD
Associate Executive Director, ABNM
Beth Israel Deaconess Medical Center
Boston, Massachusetts
Tony_Parker@BIDMC.Harvard.edu
Caveat Lector

Psychometrics
Origin for Questions Writing Rules
Psychometrics is the field of study concerned with the theory and technique of psychological measurement, which includes the measurement of knowledge, abilities, attitudes, personality traits, and educational measurement. The field is primarily concerned with the construction and validation of measurement instruments such as questionnaires, tests, and personality assessments.
– Wikipedia

Test Goal is to Measure Ability
Logistic regression where the probability of a correct answer is a function of
F n
, the ability of person, n b i
, the difficulty of question, i
Difficulty of question, i, bi
Ability 0.3
Ability of person , n,
F n
Wikipedia

Can Also Measure Question Difficulty
Logistic regression where the probability of a correct answer is a function of
F n
, the ability of person, n b i
, the difficulty of question, i
Ability of person , n,
F n
Difficulty 0.3
Difficulty of question, i, bi
Wikipedia

Question Discrimination
The slope determines how well a question discriminates between candidate abilities a i
, ability discrimination log-odds correct answer = a i
• ( F n
- b i
)
Ability of person , n,
F n
= 0.3
Slope, a i
Discrimination
Difficulty 0.3
Wikipedia
Difficulty of question, i, bi

ABNM Procedure
Assume questions are good and correctly keyed
Measure candidate tentative ability
Using tentative abilities measure question difficulty
Discard questions that are too hard or too easy
Also measure question-ability discrimination
Correct wrong keys
Discard questions that not correlated with ability
Using improved test, measure final candidate ability
Adjust to standard independent of candidates and test

Writing Good Questions:
Questions that Correlate with Ability
Psychometricians have developed rules for writing good questions
Rules are based on question performance
Bad questions test abilities unrelated to the test goal

Some Rules for Good Questions
No negatives (not, except, none)
No absolutes (always, never, all, every, only, must)
No overlapping ranges
No “all of the above” or “none of the above”
No mutually exclusive answers (one is correct)
Logical order (numerical, alphabetical, category)
Single concept per question
No multiple true / false questions

Which of the following is true in Nuclear Medicine?
MIBG is used to image somatostatin receptors
A photon is completely absorbed during a
Compton interaction
18 F-FDG is transported into cells by GLUT
Lymphoma is never FDG-avid
Multiple true / false “which is true?”
Multiple unrelated concepts
“Never” is an absolute

With regard to the thyroid which is correct?
Pentetreotide is taken up in the parafollicular cells
Follicular cells are non-iodine-avid
Stromal cells take up iodide
Pentetreotide is not taken by the thyroid
Multiple true / false “which is correct?”
First and last answers are mutually exclusive
One of them must be correct
“Non” and “not” are negatives

In the thyroid, pentetreotide is taken up MOST avidly in: parafollicular cells (C cells) follicular cells colloid stroma

In the thyroid, pentetreotide is taken up MOST avidly in: parafollicular cells (C cells) follicular cells colloid stroma

Which of the following is not correct?
11 C-acetate is metabolized in the the TCA cycle
11 C-choline can be used for prostate cancer
18 F-fluorodexoyglucose (FDG) is not phosphorylated in the cell
18 F-fluorothymidine (FLT) is correlated with cellular proliferation
Not in stem and in 3 rd answer
Multiple true / false “which is correct?”
Multiple unrelated concepts

MOST closely correlated with proliferation?
11 C-acetate
11 C-choline
18 F-fluorodexoyglucose (FDG)
18 F-fluorothymidine (FLT)

MOST closely correlated with proliferation?
11 C-acetate
11 C-choline
18 F-fluorodexoyglucose (FDG)
18 F-fluorothymidine (FLT)
Good distractors are often partially true, but not the best answer

What is the BEST administered dose to use for a 18 F-FDG PET pediatric scan?
≤ 1 MBq/kg (≤ 0.03 mCi/kg)
1 - 4 MBq/kg (0.03 - 0.11 mCi/kg)
4 - 16 MBq/kg (0.11 - 0.43 mCi/kg)
10 - 64 MBq/kg (0.27 - 1.73 mCi/kg)
Overlapping ranges
4-16 and 10-64 both include 10-16
1-4 and 4-16 both include 4

What is the BEST administered dose to use for a 18 F-FDG PET pediatric scan?
1 MBq/kg (0.03 mCi/kg)
4 MBq/kg (0.11 mCi/kg)
16 MBq/kg (0.43 mCi/kg)
64 MBq/kg (1.73 mCi/kg)

What is the BEST administered dose to use for a 18 F-FDG PET pediatric scan?
1 MBq/kg (0.03 mCi/kg)
4 MBq/kg (0.11 mCi/kg)
16 MBq/kg (0.43 mCi/kg)
64 MBq/kg (1.73 mCi/kg)
Gelfand: Pediatric radiopharmaceutical administered
Doses: 2010 north american consensus guidelines.
J Nucl Med 2011;52:318

Parallel Answers / Avoid Teaching
Answers should be parallel in construction and length
Good test takers know that “teachers like to teach”
The correct answer is often the longest
Tendency to make correct answer exactly correct
For testing, avoid teaching

Urinary excretion of FDG is related to:
FDG concentration overwhelms GLUT renal GLUT has low affinity for FDG both of the proximal and distal convoluted tubule sodium glucose transporters (SGLT1 & SGLT2 respectively) have low affinity for FDG there is little hexokinase in the kidney
Asymmetric answers
Teaching

Urinary excretion of FDG is related to:
FDG concentration overwhelms GLUT renal GLUT has low affinity for FDG
SGLT1/2 have low affinity for FDG there is little hexokinase in the kidney

Urinary excretion of FDG is related to:
FDG concentration overwhelms GLUT renal GLUT has low affinity for FDG
SGLT1/2 have low affinity for FDG there is little hexokinase in the kidney

Glucose Reuptake in the Kidney

Stem (Question)
1. Ask a question
Given …, what is the MOST likely diagnosis?
2. Give and incomplete statement
Given …, the BEST next step is:
3. Describe a situation followed #1 or #2
A one-year-old girl had jaundice and a
5 x 6 cm mass in the porta. Total bilirubin is 7 mg/dL. …

Stem (Question)
Put most of the content in stem
Answers short
Move common words in answers to stem
Remove unnecessary words or sentences

In a 73-year-old man with metastatic melanoma, what is the MOST likely cause of the FDG uptake shown in the figure?
prostate cancer melanoma normal bladder normal urethra

In a 73-year-old man with metastatic melanoma, what is the MOST likely cause of the FDG uptake shown in the figure?
prostate cancer melanoma normal bladder normal urethra

Distractors (Incorrect Answers)
Should be plausible
Logical misconceptions
Common misunderstandings
Reasonable not the BEST answer not the MOST likely answer

Testing
Testing vs. Teaching
Wright Map candidates more able distinguish candidates difficult items judgment required low passing %
Teaching help learning easier items emphasize key points less able items hard easy

Question Taxonomy
Recall (fact)
The energy of 99m Tc is:
Interpretive (diagnosis)
A patient with … had an FDG-PET/CT shown in the figure. The MOST likely diagnosis is:
Problem solving (management, judgment)
A patient with … had an FDG-PET/CT shown in the figure. The BEST next step is:

Higher Taxonomy Preferred
Often involve more advanced reasoning
More complex medical task
Still single clinical concept
Distractors may be easier to write
Some topics require lower taxonomy questions

Stable iodine (SSKI or Lugol ’ s solution) should be administered prior to: pentetreotide (Octreoscan®) iobenguane (MIBG) a contrast FDG-PET/CT ibritumomab tiuxetan (Zevalin®)

Stable iodine (SSKI or Lugol ’ s solution) should be administered prior to: pentetreotide (Octreoscan®) iobenguane (MIBG) a contrast FDG-PET/CT ibritumomab tiuxetan (Zevalin®)
Recall question pentetreotide confused with MIBG ibritumomab tiuxetan confused with tositumomab

Based on the electrocardiogram, what is the BEST diagnosis?
Difficult to come up with 4 plausible diagnoses

A 45 y/o diabetic man with weakness and shortness of breath for 3 days presents for exercise MPI. The preexercise ECG is shown.

What should be done next?
Obtain right precordial leads.
Use dipyridamole stress instead.
Accompany the patient to the EW.
Proceed as planned.

What should be done next?
Obtain right precordial leads.
Use dipyridamole stress instead.
Accompany the patient to the EW.
Proceed as planned.

Writing Distractors
Difficulty writing reasonable distractors
Change taxonomy
Management distractors often easier to write

The BEST diagnosis is:

The BEST diagnosis is:
?
?
normal thymus mediastinal metastases
Diagnostic question

The uptake in the chest is related to:

The uptake in the chest is related to: the Na / I symporter somatostatin receptor, type 2 brown fat metastases
Diagnosis & biodistribution

A syringe of 90 Y-ibritumomab tiuxetan
(Zevalin ®) measuring 1025 MBq (27.7 mCi) needs to be discarded due to QC failure.
After transfer to a vial, measurements are vial: 815 MBq (22.0 mCi), syringe: 15 MBq
(0.4 mCi). What should be done next?
Declare a medical event
Survey the area
Call the radiation safety officer (RSO)
Store for decay

A syringe of 90 Y-ibritumomab tiuxetan
(Zevalin ®) measuring 1025 MBq (27.7 mCi) needs to be discarded due to QC failure.
After transfer to a vial, measurements are vial: 815 MBq (22.0 mCi), syringe: 15 MBq
(0.4 mCi). What should be done next?
Declare a medical event
Survey the area
Call the radiation safety officer (RSO)
Store for decay

Which Isotope is MOST Difficult to
Accurately Measure in a Dose Calibrator?
18 F
90 Y
99m Tc
111 In

Which Isotope is MOST Difficult to
Accurately Measure in a Dose Calibrator?
18 F
90 Y
99m Tc
111 In

What radiopharmaceutical was administered?
99m Tc-pertechnetate
67 Ga-citrate
111 In-pentetreotide
123 I-MIBG

What radiopharmaceutical was administered?
99m Tc-pertechnetate
67 Ga-citrate
111 In-pentetreotide
123 I-MIBG

What ’ s the Radiopharmaceutical?
Stupid Questions, but I Use Them
Should know normal and common variant biodistribution at organ, cell, and receptor level
To be able to identify abnormal biodistribution identify radiopharmaceutical QC understand unusual findings

After injection of 111 In-DTPA in a patient with normal renal function?
the half-time of excretion is 16 minutes the half-time of excretion 8 minutes the half-time of excretion 1.7 hours the half-time of excretion 36 minutes
Repeated words in answers

How long after injection of 111 In-DTPA is
1/2 excreted with normal renal function?
16 minutes
8 minutes
1.7 hours
36 minutes
Answers not in logical order

How long after injection of 111 In-DTPA is
1/2 excreted with normal renal function?
8 minutes
16 minutes
36 minutes
1.7 hours

How long after injection of 111 In-DTPA is
1/2 excreted with normal renal function?
Renogram
8 minutes
16 minutes
36 minutes
1.7 hours http://www.doseinfo-radar.com/

Some Rules for Good Questions (1/2)
No negatives (not, except, none)
No absolutes (always, never, all, every, only, must)
No overlapping ranges
No “all of the above” or “none of the above”
No mutually exclusive answers (one is correct)
Logical order (numerical, alphabetical, category)
Symmetric answers

Some Rules for Good Questions (2/2)
Put concepts and common words in stem
Distractors should be reasonable
Try to use higher taxonomy
Often easier to write distractors
Some fact questions are appropriate
Single concept per question
No multiple true / false questions
Medically, not question, complexity

Reference Guidelines
3-page guide
ABNM_Well_Formatted_SAM_Question.docx
1page list of does and don’ts
ABNM_QuestionWritingDoesAndDonts.docx
26-page guide (ABNM psychometric consultants) http://www.measurementresearch.com/
24-page guide
ABR_Item_Writers_Guide.pdf