CONSTRUCTED RESPONSE ITEMS:
MEQ & SAQ
DAVID PRIDEAUX, TIM NEILD, JENNY SAVAGE
School of Medicine, Flinders University
L of ty wa s fou
of Mo un t
nd
T h e sit u ati on
som e ot h er cross
an d fro m
f rom h en ce
th a nd 13 8
34¡ 59 ' sou
¡ 42'
b eari ng s, to be
l e so f ar to th e
w as vi si b
ea st. N o l and
rees app ea re d a bo v
ere t he t
e
n ort h as wh
ow ed th e coa st t o
hi ch sh
t he ho rizon , w
o un d ing s were
d o ur s
b e v er y low, an
.
f ast dec reasing
no on t o six o'cl ock we
Fr om
ra n th irt y
s t o t he nor th wa rd, sk irt in
m ile
g a sandy
e at th e d ista nce of five, an
sh or
d t he nce
t m i les; th e dept h w
to eigh
as then 5
m s, an d we drop p ed the
fa th o
an ch or u p on
t om o f sa nd , mi xed wit h
a b ot
pieces of
co ral.
dead
Train the Trainer Workshop
October 28, 2003
Hong Kong International Consortium
CONSTRUCTED RESPONSE ITEMS
ASSUMPTIONS
•
Higher cognitive processes
•
Increased reliability through wider sampling
ARE THE ASSUMPTIONS JUSTIFIED?
MODIFIED ESSAY QUESTIONS: MEQs
•
1970s – 1980s
•
RCGP
(Hodgkin & Knox 1975)
•
Newcastle Australia
(Feletti 1980, Feletti and Smith 1986)
•
Jefferson USA
(Rabinowitz 1987, Rabinowitz and Hojat 1989)
MODIFIED ESSAY QUESTIONS: MEQs
•
Learning objectives, concepts, deep structures
•
Clinical scenario
•
7 – 10 questions
•
Sequential
(information
•
question
information
question ...)
Integrated (content, media – Rabinowitz 1985)
MODIFIED ESSAY QUESTIONS: MEQs
HIGHER COGNITIVE ORDER
(Bloom 1956, Buckwalter et al 1981)
1.
Knowledge
Level 1 Recognition Recall
2.
Comprehension
Level 2 Comprehension
Interpretation
3.
Application
4.
Analysis
Level 3 Application
5.
Synthesis
6.
Evaluation
(Irwin and Bamber 1982)
MODIFIED ESSAY QUESTIONS: MEQs
LEVEL 1 RECOGNITION AND RECALL
Know
•
What is …..?
•
List …..
•
Identify …..
MODIFIED ESSAY QUESTIONS: MEQs
LEVEL 2 COMPREHENSION, INTERPRETATION
Understand
•
Give reasons ….
•
Explain ….
•
Interpret ….
MODIFIED ESSAY QUESTIONS: MEQs
LEVEL 3 APPLICATION
Apply, Utilise, Judge
•
How does X assist in explaining Y?
•
What are the components of X?
How are they related?
How do they affect Y?
•
How does XYZ affect treatment and management?
•
Assess, judge X in terms of ….
MODIFIED ESSAY QUESTIONS – VALIDITY
•
CONTENT
(Blueprints, objectives, deep structures)
•
CONSTRUCT
(Level 1, 2, 3 questions)
MODIFIED ESSAY QUESTIONS – VALIDITY
•
CONSTRUCT
Norman et al 1987
Effect of educational level
MCQ and MEQ not different skills
UEQ may be a different skill
Small effect of context
Feletti 1980
More Level 1 and 2 in first year undergraduate
Irwin and Bamber 1986
More Level 3 in final year undergraduate
MODIFIED ESSAY QUESTIONS – VALIDITY
•
PREDICTIVE
Irwin and Bamber 1986
Correlation with clinical examination 0.41
Rabinowitz 1987
MCQ better predictor NBME I & II
Rabinowitz and Hojat 1989
Higher correlation for MEQ than MCQ
with overall clerkship ratings
MODIFIED ESSAY QUESTIONS – RELIABILITY
•
Inter-rater reliability (model answers)
Norman et al 1987
Inter-rater reliability satisfactory
•
Internal consistency (cumulative errors)
Feletti 1980
Alpha(60) 0.57 – 0.91
Stratford and Pierce-Fenn 1985
Alpha(60) 0.39
MODIFIED ESSAY QUESTIONS – FEASIBILITY
•
Administration
•
Marking
•
Higher order questions
MINI-CASES: THE FLINDERS MEQ
•
Model PBL process
•
Determine objectives
•
Construct scenario
MINICASES: THE FLINDERS MEQ
John Taylor (age 55) has been brought to the E.D. by his wife. John
woke up this morning (one hour ago) with nausea and vertigo. He
managed to walk to the bathroom but had to hold onto the furniture
and walls to steady himself. He vomited when he reached the
bathroom.
1.
Give 3 hypotheses to account for John’s condition.
2.
Explain the mechanisms which give rise to John’s condition.
3.
What further information would you like to distinguish
between hypotheses. Explain how this information will
distinguish between hypotheses.
MINI-CASES: THE FLINDERS MEQ
•
Provide further information
•
Cumulative error cues
•
Continue scenario
MINICASES: THE FLINDERS MEQ
John has had no significant medical problems in the last few
years. He spent the previous evening at home watching TV
and did not drink any alcohol or take any other substance.
….. The left pupil is smaller than the right …..
….. John’s uvula deviates to the right …..
….. Movements of the left hand are clumsy.
MINI-CASES: THE FLINDERS MEQ
•
Further questions
•
Continue unfolding scenario
•
Page breaks, cumulative cues
•
Conclude with SAQs (Level 2/3)
MINI-CASES: THE FLINDERS MEQ
•
Face validity
•
Construct validity
•
Model answers
•
Higher order questions
SHORT ANSWER QUESTIONS - SAQ
•
Reform of RCGP MEQ
– Swanson
– Mullholand & McAleer
•
Increase number (10 – 12 / 2 hours)
– Reflect reality
•
Increase independence
•
Single marked – schedules
•
SAQ
(Lockie et al 1990)
Flinders SAQ strategies
Opportunity to target highest cognitive skills
•
application - analysis - synthesis - evaluation
Bloom 1956
Evaluation:
“The ability to make a value-judgement in relation to
relevant criteria”
response
contextual information + Q
criteria
Targeting “evaluation” response
1. Provide multiple, detailed, relevant information
•
Patient
•
Problem
•
Clinical care setting
•
Your role
response
contextual information + Q
criteria
Example scenario information
• Ron Harkin is a 32 year-old father of two and self-employed
gardener. He falls heavily. He is unable to stand and is in
considerable pain. He is taken to hospital where you are the Intern
in the Emergency Department. You note swelling and that his foot
appears to be at an unusual angle to his leg.
• A 7 year-old Aboriginal girl, Alice Mundi, presents with a mild
fever and painful knee to a small rural hospital. Her mother has
brought her 50km to the hospital. You are the GP called to see
her.
Targeting “evaluation” response
2. Pose question requiring value-judgement
• focus the response
•
assume criteria are understood
•
relate to what you would do
3. Extend information / probe further (optional)
response
contextual information + Q
criteria
Example questions targeting “evaluation”
Rationalising the diagnosis
• What further information from the history do you require?
• On what aspects of the physical examination will you
focus?
• What investigations would you undertake to establish the
diagnosis?
• What are your differential diagnoses? Which do you
consider most likely?
Example questions targeting “evaluation”
Formulating an appropriate management plan
• What would be your immediate management goals?
• Which treatments would you advise?
• For what potential complications would you monitor?
• What longer-term management strategy would you advise?
• What advice would you give in relation to this new
development?
Mr Dawes, a 58 yo man, presents to the Emergency Dept with a 24h history
of vomiting and colicky abdominal pain. He has had no bowel movement for
24h. He had an appendicectomy at age 30 to remove a gangrenous
appendix. On examination you find he has a distended abdomen with no focal
tenderness. There is no palpable hernia. He has symptoms and signs
consistent with dehydration of about 10%. Fluid resuscitation is commenced.
A plain abdominal Xray shows distended loops of small bowel with air fluid
levels and normal calibre large bowel.
Q1. What is your most likely diagnosis for this patient?
(2 marks)
Q2. What possible causes of this would you consider?
(3 marks)
After 48h of conservative management, Mr Dawes’ condition is such that
surgical intervention is warranted.
Q3. What clinical indications would suggest to you that Mr Dawes needs
surgery to resolve his problem at this time?
(5 marks)
Flinders SAQ strategies
Fairly reward sound responses
•
Model answer (checklist)
•
Writers / markers who understand:
- curriculum content
- learner stage
•
Ongoing item improvement
CONSTRUCTED RESPONSE ITEMS
•
Are the claims justified?
•
Do they measure anything different?
(Maquire, Skakun, Triska 1997)
•
What role do they have in a balanced
assessment program?
CONSTRUCTED RESPONSE ITEMS
REQUIREMENTS
•
Write to a blueprint
•
Sample widely
•
Increase level 2 and 3 questions
•
Model answers
THE END
SHORT ANSWER QUESTIONS - SAQ
You see Rebecca and Robert who are residents at a local training
centre for 27 young mentally handicapped patients. They are
accompanied by their care helper. Rebecca and Robert have
scabies.
Discuss your management
•
•
•
•
•
•
communication
clinical management
health in the community
sensitivity to the mentally handicapped
use of extended team
ethical issues
Deep/Surface Structures
(Lockie et al 1990)
SHORT ANSWER QUESTIONS - SAQ
VALIDITY
Construct
Wright et al (1983) – Independent cognitive abilities
Predictive
Wass et al (2001) – correlate with clinical tests
RELIABILITY
Low to moderate
Des Marchais and Vu (1996)
Wakeford and Roberts (1984)
Good
Wright et al (1983) – marking schedules
SHORT ANSWER QUESTIONS - SAQ
FEASIBILITY
•
Marking
•
Higher order questions
HYBRID APPROACHES
•
One clinical scenario – multiple questions
PAQ (Des Marchais and Vu 1996) – no sequence
SAQ (Flinders) sequence
•
Level 1, 2 and 3 questions on one scenario
•
Reliability
PAQ – low to moderate
•
Feasibility
THE FLINDERS HYBRID MODEL
•
Face validity
•
Construct validity
•
Model answers
•
Higher order questions