Script Concordance Test
Jean Paul Fournier
Département de Pédagogie Médicale
Faculté de Médecine de Nice Sophia Antipolis, France
Clinical reasoning
Semantic networks
Instances
Prototypes
Patient’s complaint
Problem representation
Early problem identification
(non analytic process)
Compiled knowledge
Early generation
of relevant hypothesis
Script activation
Hypothesis check
Clinical data interpretation
Micro judgements
(slots)
Analytic
process
Quest for new
clinical / paraclinical data
Tempus Program
March 12, 2012
Nendaz M. Pédagogie Médicale 2005
Pellacia Th. Annales Françaises de Médecine
d’Urgence 2011
Clinical reasoning (2)
No semantic network
High complexity case
No hypothesis
Epidemiology
Prevalence
Pathophysiology
…
Systematic approach
Forward
reasoning
Diagnosis
Tempus Program
March 12, 2012
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
Clinical reasoning (3)
Non analytic process
Description
Pattern recognition
Intuitive identification of a typical
signs configuration (= pattern),
correspounding to diagosi(e)s
Instances
Intuitive identification of a déjà vue
situation and immediate diagnosi(e)s
generation
Tempus Program
March 12, 2012
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
Clinical reasoning (4)
Analytic process
Description
Hypothetico deductive process
The physician consciously tries to
confirm
/
infirm
his
early
hypotheses, with an oriented quest
of clinical / paraclinical signs
Forward reasoning
The physician goes from clinical /
paraclinical data to the problem
solving, through application of
causal or conditionnal rules
Tempus Program
March 12, 2012
Pellacia Th. Annales Françaises de Médecine d’Urgence 2011
A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with
shortness of breath. She suffers an ovarian carcinoma, in palliatie care. She does not move from her bed,
due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 13580 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. Last creatinine
level was 150 μmol/l. She has venous disease sequellae and bilateral pitting edema. You think of a possible
pulmonary embolism.
Physician 1
Physician 2
Physician 3
D dimers assay
unesuseful in such a
context
D dimers assay
unesuseful in such a
context
D dimers assay
D dimers > 5000 g/ml
Venous ultrasound
Femoral vein thrombosis
CT pulmonary angiogram
Bilateral proximal
pulmonatry embolism
Heparin
Tempus Program
March 12, 2012
With an high pre test
probability and D dimers >
4000 : pulmonary embolism
probability : 88 p. cent
CT pulmonary angiogram
benefit/risk ?
What happened ?
 3 different pertinent approaches for the
resolution of the same problem
Script activation
 3 different pertinent approaches but with the
same result : heparin therapy ;
Clinical
reasoning
variability
 Uncertainty : D dimer results interpretation in
an old wman with ovarian cancer, CT pulmonary
angiogram risk/benefit in an old woman with
chronic kidney impairment ;
Uncertainty
 Impact of new data in initial hypotheses
(micro judgements).
Ponderation
Tempus Program
March 12, 2012
Script theory
 Script : Compiled knowledge network (clinical, biological,
imaging,…) used for a clinical problem solving
 Scripts are elaborated from early patients encounters ;
 The more frequent and diverse are encounters, the richest are
scripts ;
 Scripts cannot be transfered ;
 They vary from a physician to anaother.
Charlin B. Acad Med 2000
Tempus Program
March 12, 2012
Script Concordance Test
Standardized assessment tool of clinical reasoning through the presence
and functionnality of scripts in simulated clinical situations in an uncertainty
context
 Standardized assessment tool ;
 Clinical situations ;
 Ill-defined situation and context : no consensus,
no evidence-based medicine ;
 Assessment of compiled clinical reasoning ;
 Reference to an expert panel (aggregate scoring).
Tempus Program
March 12, 2012
SCT : Principles
 Comparing the attributed impact of a new information on initial hypothesis
between experts and students ;
 Experts take the same test as students (same questions, same timing) and
serve as the reference panel ;

 All experts answers are accepted (with the restriction of discordant answer or
expert) ;
 The closest to the experts’ choice is the students’ choice, the most performing
are students.
The SCT
Short clinical vignette
A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of
breath. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate.
Temperature is 38,2°C, BP is 135-80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face
mask. You think of a possible pulmonary embolism.
Question
Anchor
If you were planing to
prescribe…
And then you find…
The impact in your
hypothesis becomes…
A D-dimer assay
An ongoing ovarian carcinoma
in palliative care
-2 -1 0 +1 +2
A venous ultrasound
A swollen painful left calf
-2 -1 0 +1 +2
A CT pulmonary angiogram
Créatinine level of 150 µmol/l
-2 -1 0 +1 +2
-2 : Very unuseful
-1 : Unuseful
Pertinent hypothesis
0 : Neither more or less useful
New relevant information
Tempus Program
March 12, 2012
+1 : Useful
+2 : Very useful
Likert scale : assessment
of the new information
on the initial hypotheses
(Micro judgement)
Item
SCT vs rich-context MCQ
Is SCT an efficient assessment tool ?
 A valid tool ?
 Face validity : depends on test construction (blueprint) ;
 Construct validity :
 identification of juniors vs seniors ;
 prediction of scores for tests exploring the same domain ;
 A reliable test ?
 Cronbach alpha from 0,79 to 0,82.
MCQ : 0,76 à 0,93 (2 to 8 hour testing)
Patient Management Problem : 0,69 à 0,82 (4 to 8 hour testing)
Key Feature Case : 0,66 à 0,79 (4 à 8 hour testing).
Tempus Program
March 12, 2012
SCT among other assessment tests ?
 MCQ ;
 Clinical cases for ECN.
Tempus Program
March 12, 2012
SCT vs rich-context MCQ
9523
Clinical ED situations
Recommandations
NBME
Recommandations
B. Charlin 2002
60 MCQ
60 SCT
Each situation appeararing in the 2 formats
Test taken by 16 EM residents
Tempus Program
March 12, 2012
SCT vs rich-context MCQ
55
Score TCS sur 100
50
45
40
35
y = 0,3559x + 25,086
2
R = 0,3011
p = 0,0432
30
40
45
50
55
60
Score QCM sur 100
Tempus Program
March 12, 2012
65
70
75
SCT vs true/false MCQ
 106 medical students from 3rd to 6th year ;
 Endocrinology tutorial ;
 SCT vs true/false MCQ :
 Significant
correlation in 3rd
and 4th year ;
 Correlation for
the 4 years with
the knowledge
level estimate (true
vs false answers).
Tempus Program
March 12, 2012
Collard A. Med Educ 2009
SCT vs ECN clinical case
 18 medical students from 4th to 6th year ;
 ECN clinical case ;
 26 questions SCT.
Gibot S. Pédagogie Médicale 2008
Tempus Program
March 12, 2012
SCT vs ECN clinical case
R = 0,55
Gibot S. Pédagogie Médicale 2008
Tempus Program
March 12, 2012
SCT vs other assessment tools
 Significant correlation, but low clinical significance : : some
common points (assessment domain), but the SCT assesses
some other aspects ;
 Combining several assessment formats is a key-point in
assessment.
Tempus Program
March 12, 2012
Usual assessment tools
Performance
Direct observation, peers, video, audits
Action
Competences
Standardized oral examination, OSCE, mini CEX
simulation, SCT
Knowledge
utilization
Rich context MCQ, SCT, PMP, key-features
Knowledge
True/false MCQ, essay-questions
Compétence
Miller Acad. Med. 1990
Tempus Program
March 12, 2012
How is it accepted ?
Tempus Program
March 12, 2012
Bounouffe C. Am J Pharm Educ 2010
Technical aspects
 Blueprint ;
 Preparation ;
 Anchor ;
 Experts panel ;
 Scoring ;
 Quality control.
Tempus Program
March 12, 2012
Blueprint
4 points :
 What do we do ?
Certification ? Formative
aspect ?
Difficulty level ?
Prevalence ?
 Situations selection ?
Severity ?
Diagnosis
 Questions formats
Treatment
Management
Reliability
 Items and questions
number
Feasibility
Tempus Program
March 12, 2012
Validity
Blueprint
Cardio
Pneumo
Neuro
Gastro
Dg
X
X
X
X
Trt
X
X
X
X
Management
X
X
X
Metabol.
Rheumato
Geriatr.
Gyneco
Psy
Traumato
X
X
X
X
X
X
Test developers + experts
Tempus Program
March 12, 2012
X
Blueprint
Cardio
Pneumo
Neuro
Gastro
Dg
Chest
pain
Acute
shortness of
breath
Headache
Abdomin
al pain
Trt
CHF
Pulmonary
embolism
Seizure
Gastric
bleeding
Managemen
t
ACS
COPD
exacerbation
Stroke
Metabol.
Rheumato
Geriatr.
Gyneco
Psy
Trauma
Mono
arthritis
Altered
general
condition
Ectopic
pregnancy
Confusion
Neck
trauma
Dehydrat.
Test developers + experts
Tempus Program
March 12, 2012
Agitation
How many cases, how many questions ?
 Case specificity ;
 Many cases with one question ?
Format
homogeneity
 Few cases with many questions ?
 1 case with n questions ?
 20 cases and 60 questions : Cronbach alpha ≥ 0,70
Tempus Program
March 12, 2012
How many cases, how many questions ?
Tempus Program
March 12, 2012
Gagnon R. Adv Health Sci Educ 2008
How many cases, how many questions ?
Tempus Program
March 12, 2012
Gagnon R. Adv Health Sci Educ 2008
Preparation : situations, questions
Principles :
 Clinical situations selection, fitted to students level ;
 Relevant hypothesis selection in that context ;
Test developers
+ experts
 New relevant informations selection, in that
context
Uncertainty
Tempus Program
March 12, 2012
Page G. Academic Medicine 1995
Clinical vignette and questions preparation
Vignette : relevant informations :
 Age, gender ;
 Complaint, adlission,
hospitalization reason ;
1 to 2
writers
 Context : ED, office, ward,…
 Clinical data,…
Tempus Program
March 12, 2012
Clinical vignette and questions preparation
Relevant new informations :
 Clinical data ;
 Biological test ;
1 to 2
writers
 Imaging ;
 Video,…
Tempus Program
March 12, 2012
Clinical vignette and questions preparation
Vignette, hypothesis and new informations :
 Sweeping the five points of the Likert scale ;
 Well-balanced variability level ;
 All cases are independant ;
 All questions are independant ;
 One format by case (diagnosis, treatment,…).
Tempus Program
March 12, 2012
Charlin B. Medical Education 2006
Question discrimination power and
variability
Staff-students
1,5
1
0,5
0
Low variability
Moderate variability
Tempus Program
March 12, 2012
High variability
Charlin B. Medical Education 2006
Lickert scale - Anchor
Recommandations :
 Explicit the significance of each point of the Lickert scale,
particularly the neutral point (« 0 ») ;
 Keep the same presentation.
-1 : Unlikely
-2 : Very unlikely
0 : Neither more or less
likely
Tempus Program
March 12, 2012
+1 : More likely
+2 : Very likely
Lickert scale - Anchor
Lickert scales :
 Diagnosis
anchor :
If you were thinking of the
following diagnosis…
And then you find…
This new information
makes your diagnosis…
Diagnosis option
New information
-2 -1 0 +1 +2
-1 : Unlikely
-2 : Very unlikely
0 : Neither more or less
likely
Tempus Program
March 12, 2012
+1 : More likely
+2 : Very likely
Lickert scale - Anchor
Lickert scales :
 Anchor for
test(utility) :
If you were considering the
usefulness of the following
test…
And you find…
This new information makes
the following test…
Investigative option
New information
-2 -1 0 +1 +2
-1 : Useless
-2 : Less useful
 Anchor for
test(benefit/risk) :
0 : Neither morer or less useful
+1 : Useful
+2 : Very useful
If you were considering the
benefit/risk ratio of the
following test…
And you find…
This new information makes
the test…
Investigative option
New information
-2 -1 0 +1 +2
-1 Contraindicated
-2 : Strongly contraindicated
0 : Neither more or less
indicated
Tempus Program
March 12, 2012
+1 : Indicated
+2 : Strongly indicated
Lickert scale - Anchor
Non appropriate
option
Lickert scales :
 Anchor for
test(utility) :
If you were considering the
usefulness of the following
test…
And you find…
This new information makes
the following test…
Investigative option
New information
-2 -1 0 +1 +2
-1 : Useless
-2 : Less useful
0 : Neither morer or less useful
Inadequate
prescription
 Anchor for
test(benefit/risk) :
Adequate prescription
+1 : Useful
+2 : Very useful
If you were considering the
benefit/risk ratio of the
following test…
And you find…
This new information makes
the test…
Investigative option
New information
-2 -1 0 +1 +2
-1 Contraindicated
-2 : Strongly contraindicated
0 : Neither more or less
indicated
Tempus Program
March 12, 2012
+1 : Indicated
+2 : Strongly indicated
Lickert scale - Anchor
Likert scales :
 Treatment
anchor (utility) :
If you were considering the
usefulness of the following
treatment…
And you find…
This new information makes
the following treatment…
Treatment option
New information
-2 -1 0 +1 +2
-1 : Useless
-2 : Less useful
 Treatment
anchor
(benefit/risk) :
0 : Neither morer or less useful
+1 : Useful
+2 : Very useful
If you were considering the
benefit/risk ratio of the
following treatment…
And you find…
This new information makes
the following treatment…
Treatment option
New information
-2 -1 0 +1 +2
-1 Contraindicated
-2 : Strongly contraindicated
0 : Neither more or less
indicated
Tempus Program
March 12, 2012
+1 : Indicated
+2 : Strongly indicated
Lickert scale - Anchor
Non appropriate
option
Échelles :
 Treatment
anchor (utility) :
Si vous considériez l’utilité du
traitement suivant
Et qu’alors vous trouvez
Cette nouvelle information rend
le traitement
Treatment option
New information
-2 -1 0 +1 +2
-1 : Useless
-2 : Less useful
0 : Neither morer or less useful
Inadequate
prescription
 Treatment
anchor
(benefit/risk) :
Adequate prescription
+1 : Useful
+2 : Very useful
If you were considering the
benefit/risk ratio of the
following treatment…
And you find…
This new information makes
the following treatment…
Treatment option
New information
-2 -1 0 +1 +2
-1 Contraindicated
-2 : Strongly contraindicated
0 : Neither more or less
indicated
Tempus Program
March 12, 2012
+1 : Indicated
+2 : Strongly indicated
Lickert scale - Anchor
Lickert scales :
 Prognosis
anchor :
Prognosis item
New information
The prognosis becomes
Prognosis item
New information
-2 -1 0 +1 +2
-1 : Worsened
-2 : Very worsened
0 : Never more or less modified
Tempus Program
March 12, 2012
+1 : Improved
+2 : Very improved
Experts panel
Roles :
 Test preparation ;
 Clinical vignette and questions validation ;
2 different
groups
 Scoring.
Tempus Program
March 12, 2012
Experts panel
 Quality : experts in the considered domain ;
 Number : 10 to 20 minimum.
Tempus Program
March 12, 2012
Gagnon R. Medical Education 2005
Mean reliability following the
experts number
0,80
0,75
0,70
0,65
0,60
0,55
0,50
0,45
0,40
n=5
n=10
n=15
mean (sd)
n=20
n=25
min value
Tempus Program
March 12, 2012
n=30
n=38
max value
Gagnon R. Medical Education 2005
Expert panel composition
Teaching physicians vs physicians ?
 SCT in community medicine ;
 Comparison of scores ;
 Panel : teaching physicians vs physicians in a CME program ;
 Perfect concordance of the scores (R = 0.98) ;
 Scores attributed by physicians more elevated (p = NS).
Tempus Program
March 12, 2012
Charlin B. Medical Teacher 2007
Scoring
Scoring :
Panel choice :
 Modal
transformation of the
panel choice :
 Residents’ choice :
Unit 1
Unit 1
Question 1
-2
7
-1
1
0
0
+1
2
+2
0
Question 1
-2
1
-1
0.14
0
0
+1
0.28
+2
0
Unit 1 Question
1
 Residents’ scores :
Unit 1 Question
1
R1
-1
R2
-1
R3
-1
R4
2
R5
1
R6
0
R7
-2
R8
0
R1
0.14
R2
0.14
R3
0.14
R4
0
R5
0.28
R6
0
R7
1
R8
0
Tempus Program
March 12, 2012
Scoring
Excel® scoring calculator available :
www.cme.umontreal.ca/tcs
Tempus Program
March 12, 2012
Summary
Test preparation
 Blueprint
 Clinical vignettes
 Hypotheses
 New informations
 Scoring
 Pass/fail decision
Experts 1
 Clinical vignettes
 Hypotheses
 New informations
Experts 2
Validation
 Scoring (reference panel scoring)
Tempus Program
March 12, 2012
Quality control
A priori quality control :
Caire F. Neurochirurgie 2004
Tempus Program
March 12, 2012
Quality control
A posteriori quality control : item analysis :
 Difficulty index ;
 Discrimination index ;
 Question impact on the test reliability (Cronbach alpha) ;
 Etc…
Tempus Program
March 12, 2012
Thank you for your
attention !