Simulation based
Assessment
Training the Trainer Series
October, 2014
Assessment
• Formative
• Summative
Formative Assessment
• Assessment for learning
Summative Assessment
• Assessment of learning
Evaluation Methodology
Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483
Methodology:
Global Rating Scale
• Overall performance by categories
• Scores (Likert)
• Drawbacks
Global Rating Scale
Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41
Methodology:
Checklists
• Delineated actions
• Weighted, timed, cumulative...
• Adverse events
• Drawbacks
Checklists
Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444
Methodology:
“Time To”
• Timing of critical actions
• Pace
• Motion analysis
Methodology:
Survey Data
• Self assessments
• Psychometrics: Pre and Post Testing
Survey Data
Lammers RL, et al. Simulation-based Assessment of Paramedic Pediatric Resuscitation Skills. Prehospital Emergency Care 2009; 13:345-356
Methodology:
Biometrics
• “Technometrics?”
• Mannequin physiology
Validity
• The Assessment Method
• The Simulation
Validity:
“Face”
• General
acceptability of the
tool used for the
task required
• Does this make
sense?
Validity:
“Content”
• Does the content match what’s to be
assessed?
• Task Analysis
• The Experts
The Delphi Method
• BOGSAAT
Validity:
“Construct”
• Does this reflect true performance?
• Experts
• Role of experience
• Surveys
Validity:
“Construct”
• Statistical Analysis:
• Discrimination: “D Statistic”
• ANOVA
• “Concurrence”
Tool Validation
Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886
Validity:
“Criterion”
• Or “Extrapolation”
• Similar results to other performance
measures
• “Predictive”
Consistency:
Internal
• Requires realism
• Reflects test
quality
• Pilot Testing
Consistency:
Statistical Analysis
• Do testers do well or
poorly on the same
items?
• Generalizability: “G
statistic”
Reliability:
Inter-Participant
• Set-up
• Simulator
• The Subject
Reliability:
The Rater
• Inter-rater reliability (external
reliability)
• Statistics: IRR, ICC
Reliability:
The Rater
• Rater training
• Rater refreshing
• Scoring
Reliability:
The Observation
• Real time
• Recording
• Bias
Validated Tools
Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.
Take Away
• Rigor in realism, rehearsing and piloting
• Mix Checklists and GRS
• Supplement with survey,
psychometric, device data, etc
• More tasks with more (recorded) views
Further Sourcing
Schwid, et al. Evaluation of anesthesia residents using mannequin-based simulation, a multi institutional study. Anesthesiology 2002; 97:1434-1444
Devitt, et al. The validity of performance assessment using simulation. Anesthesiology 2001; 95:36-42
Devitt, et al. Testing internal consistency and construct validity during evaluation of performance in a patient simulator. Anesth Analg 1998; 86:1160-1164
Devitt JH, Kurrek MM, Cohen MM, Fish K, Fish P, Murphy PM, Szalai JP:
Testing the raters: Inter-rater reliability of standardized anaesthesia simulator
performance. Can J Anaesth 1997; 44:924–8
Gaba DM, Howard SK, Flanagan B, Smith BE, Fish KJ, Botney R: Assessment
of clinical performance during simulated crises using both technical and behavioral ratings. Anesthesiology 1998; 89:8-18
Boulet JR, et al. summative assessment in medicine: the promise of simulation for high-stakes evaluation. Acad Emerg Med 2008; 15:1017-1024
Cheng A, et al. Designing and Conducting Simulation-Based Research. Pediatrics 2014; 133:1090-1101.
Furman GE et al. Quality Assurance Best Practices for Simulation-Based Examinations. Qual Assurance Best Practices 2014; 5:226-231
Reid J, et al. The Simulation Team Assessment Tool (STAT): Development, reliability and validation. Resuscitation 2012; 83:879-886
Siassakos D, et al. The management of a simulated emergency: Better teamwork, better performance. Resuscitation 2011; 82:203-206
Irvine S and Martin J. Bridging the gap: from simulation to clinical practice. The Clinical Teacher 2014; 11:94-98
Blum RH, et al. Simulation-based assessment to identify critical gaps in safe anesthesia resident performance. Anesthesiology 2014; 120:129-41
Bould MD, et al. Assessment of procedural skills in anesthesia. Br J of Anaesth 2009; 103:472-483