Script Concordance Test Jean Paul Fournier Département de Pédagogie Médicale Faculté de Médecine de Nice Sophia Antipolis, France Clinical reasoning Semantic networks Instances Prototypes Patient’s complaint Problem representation Early problem identification (non analytic process) Compiled knowledge Early generation of relevant hypothesis Script activation Hypothesis check Clinical data interpretation Micro judgements (slots) Analytic process Quest for new clinical / paraclinical data Tempus Program March 12, 2012 Nendaz M. Pédagogie Médicale 2005 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011 Clinical reasoning (2) No semantic network High complexity case No hypothesis Epidemiology Prevalence Pathophysiology … Systematic approach Forward reasoning Diagnosis Tempus Program March 12, 2012 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011 Clinical reasoning (3) Non analytic process Description Pattern recognition Intuitive identification of a typical signs configuration (= pattern), correspounding to diagosi(e)s Instances Intuitive identification of a déjà vue situation and immediate diagnosi(e)s generation Tempus Program March 12, 2012 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011 Clinical reasoning (4) Analytic process Description Hypothetico deductive process The physician consciously tries to confirm / infirm his early hypotheses, with an oriented quest of clinical / paraclinical signs Forward reasoning The physician goes from clinical / paraclinical data to the problem solving, through application of causal or conditionnal rules Tempus Program March 12, 2012 Pellacia Th. Annales Françaises de Médecine d’Urgence 2011 A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She suffers an ovarian carcinoma, in palliatie care. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 13580 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. Last creatinine level was 150 μmol/l. She has venous disease sequellae and bilateral pitting edema. You think of a possible pulmonary embolism. Physician 1 Physician 2 Physician 3 D dimers assay unesuseful in such a context D dimers assay unesuseful in such a context D dimers assay D dimers > 5000 g/ml Venous ultrasound Femoral vein thrombosis CT pulmonary angiogram Bilateral proximal pulmonatry embolism Heparin Tempus Program March 12, 2012 With an high pre test probability and D dimers > 4000 : pulmonary embolism probability : 88 p. cent CT pulmonary angiogram benefit/risk ? What happened ? 3 different pertinent approaches for the resolution of the same problem Script activation 3 different pertinent approaches but with the same result : heparin therapy ; Clinical reasoning variability Uncertainty : D dimer results interpretation in an old wman with ovarian cancer, CT pulmonary angiogram risk/benefit in an old woman with chronic kidney impairment ; Uncertainty Impact of new data in initial hypotheses (micro judgements). Ponderation Tempus Program March 12, 2012 Script theory Script : Compiled knowledge network (clinical, biological, imaging,…) used for a clinical problem solving Scripts are elaborated from early patients encounters ; The more frequent and diverse are encounters, the richest are scripts ; Scripts cannot be transfered ; They vary from a physician to anaother. Charlin B. Acad Med 2000 Tempus Program March 12, 2012 Script Concordance Test Standardized assessment tool of clinical reasoning through the presence and functionnality of scripts in simulated clinical situations in an uncertainty context Standardized assessment tool ; Clinical situations ; Ill-defined situation and context : no consensus, no evidence-based medicine ; Assessment of compiled clinical reasoning ; Reference to an expert panel (aggregate scoring). Tempus Program March 12, 2012 SCT : Principles Comparing the attributed impact of a new information on initial hypothesis between experts and students ; Experts take the same test as students (same questions, same timing) and serve as the reference panel ; All experts answers are accepted (with the restriction of discordant answer or expert) ; The closest to the experts’ choice is the students’ choice, the most performing are students. The SCT Short clinical vignette A woman, aged 82, is referred to the ED from her retirement house for a right lateral acute chest pain with shortness of breath. She does not move from her bed, due to intractable pain, for which she recently began morphine sulfate. Temperature is 38,2°C, BP is 135-80 mm Hg, heart rate is 130 bpm, saturation is 90% with 6 l/min of oxygen via a face mask. You think of a possible pulmonary embolism. Question Anchor If you were planing to prescribe… And then you find… The impact in your hypothesis becomes… A D-dimer assay An ongoing ovarian carcinoma in palliative care -2 -1 0 +1 +2 A venous ultrasound A swollen painful left calf -2 -1 0 +1 +2 A CT pulmonary angiogram Créatinine level of 150 µmol/l -2 -1 0 +1 +2 -2 : Very unuseful -1 : Unuseful Pertinent hypothesis 0 : Neither more or less useful New relevant information Tempus Program March 12, 2012 +1 : Useful +2 : Very useful Likert scale : assessment of the new information on the initial hypotheses (Micro judgement) Item SCT vs rich-context MCQ Is SCT an efficient assessment tool ? A valid tool ? Face validity : depends on test construction (blueprint) ; Construct validity : identification of juniors vs seniors ; prediction of scores for tests exploring the same domain ; A reliable test ? Cronbach alpha from 0,79 to 0,82. MCQ : 0,76 à 0,93 (2 to 8 hour testing) Patient Management Problem : 0,69 à 0,82 (4 to 8 hour testing) Key Feature Case : 0,66 à 0,79 (4 à 8 hour testing). Tempus Program March 12, 2012 SCT among other assessment tests ? MCQ ; Clinical cases for ECN. Tempus Program March 12, 2012 SCT vs rich-context MCQ 9523 Clinical ED situations Recommandations NBME Recommandations B. Charlin 2002 60 MCQ 60 SCT Each situation appeararing in the 2 formats Test taken by 16 EM residents Tempus Program March 12, 2012 SCT vs rich-context MCQ 55 Score TCS sur 100 50 45 40 35 y = 0,3559x + 25,086 2 R = 0,3011 p = 0,0432 30 40 45 50 55 60 Score QCM sur 100 Tempus Program March 12, 2012 65 70 75 SCT vs true/false MCQ 106 medical students from 3rd to 6th year ; Endocrinology tutorial ; SCT vs true/false MCQ : Significant correlation in 3rd and 4th year ; Correlation for the 4 years with the knowledge level estimate (true vs false answers). Tempus Program March 12, 2012 Collard A. Med Educ 2009 SCT vs ECN clinical case 18 medical students from 4th to 6th year ; ECN clinical case ; 26 questions SCT. Gibot S. Pédagogie Médicale 2008 Tempus Program March 12, 2012 SCT vs ECN clinical case R = 0,55 Gibot S. Pédagogie Médicale 2008 Tempus Program March 12, 2012 SCT vs other assessment tools Significant correlation, but low clinical significance : : some common points (assessment domain), but the SCT assesses some other aspects ; Combining several assessment formats is a key-point in assessment. Tempus Program March 12, 2012 Usual assessment tools Performance Direct observation, peers, video, audits Action Competences Standardized oral examination, OSCE, mini CEX simulation, SCT Knowledge utilization Rich context MCQ, SCT, PMP, key-features Knowledge True/false MCQ, essay-questions Compétence Miller Acad. Med. 1990 Tempus Program March 12, 2012 How is it accepted ? Tempus Program March 12, 2012 Bounouffe C. Am J Pharm Educ 2010 Technical aspects Blueprint ; Preparation ; Anchor ; Experts panel ; Scoring ; Quality control. Tempus Program March 12, 2012 Blueprint 4 points : What do we do ? Certification ? Formative aspect ? Difficulty level ? Prevalence ? Situations selection ? Severity ? Diagnosis Questions formats Treatment Management Reliability Items and questions number Feasibility Tempus Program March 12, 2012 Validity Blueprint Cardio Pneumo Neuro Gastro Dg X X X X Trt X X X X Management X X X Metabol. Rheumato Geriatr. Gyneco Psy Traumato X X X X X X Test developers + experts Tempus Program March 12, 2012 X Blueprint Cardio Pneumo Neuro Gastro Dg Chest pain Acute shortness of breath Headache Abdomin al pain Trt CHF Pulmonary embolism Seizure Gastric bleeding Managemen t ACS COPD exacerbation Stroke Metabol. Rheumato Geriatr. Gyneco Psy Trauma Mono arthritis Altered general condition Ectopic pregnancy Confusion Neck trauma Dehydrat. Test developers + experts Tempus Program March 12, 2012 Agitation How many cases, how many questions ? Case specificity ; Many cases with one question ? Format homogeneity Few cases with many questions ? 1 case with n questions ? 20 cases and 60 questions : Cronbach alpha ≥ 0,70 Tempus Program March 12, 2012 How many cases, how many questions ? Tempus Program March 12, 2012 Gagnon R. Adv Health Sci Educ 2008 How many cases, how many questions ? Tempus Program March 12, 2012 Gagnon R. Adv Health Sci Educ 2008 Preparation : situations, questions Principles : Clinical situations selection, fitted to students level ; Relevant hypothesis selection in that context ; Test developers + experts New relevant informations selection, in that context Uncertainty Tempus Program March 12, 2012 Page G. Academic Medicine 1995 Clinical vignette and questions preparation Vignette : relevant informations : Age, gender ; Complaint, adlission, hospitalization reason ; 1 to 2 writers Context : ED, office, ward,… Clinical data,… Tempus Program March 12, 2012 Clinical vignette and questions preparation Relevant new informations : Clinical data ; Biological test ; 1 to 2 writers Imaging ; Video,… Tempus Program March 12, 2012 Clinical vignette and questions preparation Vignette, hypothesis and new informations : Sweeping the five points of the Likert scale ; Well-balanced variability level ; All cases are independant ; All questions are independant ; One format by case (diagnosis, treatment,…). Tempus Program March 12, 2012 Charlin B. Medical Education 2006 Question discrimination power and variability Staff-students 1,5 1 0,5 0 Low variability Moderate variability Tempus Program March 12, 2012 High variability Charlin B. Medical Education 2006 Lickert scale - Anchor Recommandations : Explicit the significance of each point of the Lickert scale, particularly the neutral point (« 0 ») ; Keep the same presentation. -1 : Unlikely -2 : Very unlikely 0 : Neither more or less likely Tempus Program March 12, 2012 +1 : More likely +2 : Very likely Lickert scale - Anchor Lickert scales : Diagnosis anchor : If you were thinking of the following diagnosis… And then you find… This new information makes your diagnosis… Diagnosis option New information -2 -1 0 +1 +2 -1 : Unlikely -2 : Very unlikely 0 : Neither more or less likely Tempus Program March 12, 2012 +1 : More likely +2 : Very likely Lickert scale - Anchor Lickert scales : Anchor for test(utility) : If you were considering the usefulness of the following test… And you find… This new information makes the following test… Investigative option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful Anchor for test(benefit/risk) : 0 : Neither morer or less useful +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following test… And you find… This new information makes the test… Investigative option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated Lickert scale - Anchor Non appropriate option Lickert scales : Anchor for test(utility) : If you were considering the usefulness of the following test… And you find… This new information makes the following test… Investigative option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful 0 : Neither morer or less useful Inadequate prescription Anchor for test(benefit/risk) : Adequate prescription +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following test… And you find… This new information makes the test… Investigative option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated Lickert scale - Anchor Likert scales : Treatment anchor (utility) : If you were considering the usefulness of the following treatment… And you find… This new information makes the following treatment… Treatment option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful Treatment anchor (benefit/risk) : 0 : Neither morer or less useful +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following treatment… And you find… This new information makes the following treatment… Treatment option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated Lickert scale - Anchor Non appropriate option Échelles : Treatment anchor (utility) : Si vous considériez l’utilité du traitement suivant Et qu’alors vous trouvez Cette nouvelle information rend le traitement Treatment option New information -2 -1 0 +1 +2 -1 : Useless -2 : Less useful 0 : Neither morer or less useful Inadequate prescription Treatment anchor (benefit/risk) : Adequate prescription +1 : Useful +2 : Very useful If you were considering the benefit/risk ratio of the following treatment… And you find… This new information makes the following treatment… Treatment option New information -2 -1 0 +1 +2 -1 Contraindicated -2 : Strongly contraindicated 0 : Neither more or less indicated Tempus Program March 12, 2012 +1 : Indicated +2 : Strongly indicated Lickert scale - Anchor Lickert scales : Prognosis anchor : Prognosis item New information The prognosis becomes Prognosis item New information -2 -1 0 +1 +2 -1 : Worsened -2 : Very worsened 0 : Never more or less modified Tempus Program March 12, 2012 +1 : Improved +2 : Very improved Experts panel Roles : Test preparation ; Clinical vignette and questions validation ; 2 different groups Scoring. Tempus Program March 12, 2012 Experts panel Quality : experts in the considered domain ; Number : 10 to 20 minimum. Tempus Program March 12, 2012 Gagnon R. Medical Education 2005 Mean reliability following the experts number 0,80 0,75 0,70 0,65 0,60 0,55 0,50 0,45 0,40 n=5 n=10 n=15 mean (sd) n=20 n=25 min value Tempus Program March 12, 2012 n=30 n=38 max value Gagnon R. Medical Education 2005 Expert panel composition Teaching physicians vs physicians ? SCT in community medicine ; Comparison of scores ; Panel : teaching physicians vs physicians in a CME program ; Perfect concordance of the scores (R = 0.98) ; Scores attributed by physicians more elevated (p = NS). Tempus Program March 12, 2012 Charlin B. Medical Teacher 2007 Scoring Scoring : Panel choice : Modal transformation of the panel choice : Residents’ choice : Unit 1 Unit 1 Question 1 -2 7 -1 1 0 0 +1 2 +2 0 Question 1 -2 1 -1 0.14 0 0 +1 0.28 +2 0 Unit 1 Question 1 Residents’ scores : Unit 1 Question 1 R1 -1 R2 -1 R3 -1 R4 2 R5 1 R6 0 R7 -2 R8 0 R1 0.14 R2 0.14 R3 0.14 R4 0 R5 0.28 R6 0 R7 1 R8 0 Tempus Program March 12, 2012 Scoring Excel® scoring calculator available : www.cme.umontreal.ca/tcs Tempus Program March 12, 2012 Summary Test preparation Blueprint Clinical vignettes Hypotheses New informations Scoring Pass/fail decision Experts 1 Clinical vignettes Hypotheses New informations Experts 2 Validation Scoring (reference panel scoring) Tempus Program March 12, 2012 Quality control A priori quality control : Caire F. Neurochirurgie 2004 Tempus Program March 12, 2012 Quality control A posteriori quality control : item analysis : Difficulty index ; Discrimination index ; Question impact on the test reliability (Cronbach alpha) ; Etc… Tempus Program March 12, 2012 Thank you for your attention !