Effectiveness of Simulation in
Healthcare
Joseph Barton, MD, MHMS
September 18, 2014
Desired Benefits of Simulation Training
• Reduce patient risk
• Broadly applicable and widely available
• Consistent training and assessment
• Reduce costs
• Provide a clinical benefit
Kirkpatrick’s 4 Levels of Learning
Evaluation
– Reaction
– Learning
– Behavior
– Results
Learning Evaluation
Level
Description
Evaluation
1
Surveys
Did the trainee like it?
2
Assessments
Did the trainee learn?
3
Transfer
Is a behavior changed?
4
Benefit
Is there a clinical outcome?
DOES SIMULATION-BASED
EDUCATION WORK?
What is the Evidence?
• How effective is simulation as a modality?
• Number of recent meta analyses
– Virtual patients
– “Technology enhanced” simulation
• Virtual patients do not require special equipment
• Standardized patients aren’t well studied
HOW EFFECTIVE IS SIMULATION
AS A TEACHING MODALITY?
Effectiveness of Simulation
• More than 600 studies comparing 36K plus
subjects
• Three recent large meta analyses
Effectiveness of Simulation
• Cook DA, Erwin PJ, Triola MM. Computerized virtual patients in
health professions education: a systematic review and metaanalysis. Acad Med. 2010;85:1589-1602.
• Cook DA, Hatala R, Brydges R, et al. Technology-enhanced
simulation for health professions education: a systematic review
and meta-analysis. JAMA. 2011;306:978-988.
• Zendejas B, Brydges R, Wang AT, Cook DA. Patient outcomes in
simulation-based medical education: a systematic review. J Gen
Intern Med. 2013.
Effectiveness of Simulation
• Virtual patients and technology enhanced simulation show
consistent, large and statistically significant benefits in
– Knowledge
– Instructor ratings
– Computer scores
– Patient care behaviors
Direct Patient Effects?
• Airway management, endoscopy, CVC insertion studied
– Smaller benefits, still significant (major complications, mortality,
length of stay) when compared to no intervention
– Non simulation instruction did not reach statistical significance
Simulation based education is more
effective than no instruction.
HOW DOES SIMULATION COMPARE
TO OTHER INSTRUCTION?
Simulation vs other intervention
• Over 100 studies and 7K participants
• Challenging question due to different
simulation interventions and due to variable
comparison strategy
Simulation vs Other Intervention
•
Cook DA, Brydges R, Hamstra SJ, et al. Comparative effectiveness of technologyenhanced simulation versus other instructional methods: a systematic review and
meta-analysis. Simul Healthc. 2012;7:308-320.
•
Cook DA, Hamstra SJ, Brydges R, et al. Comparative effectiveness of instructional
design features in simulation-based education: systematic review and metaanalysis. Med Teach. 2013;35:e844-e875.
•
Lorello GR, et al. Simulation-based training in anesthesiology: a systematic
review and meta analysis. Br J Anaesth. 2014 Feb; 112(2):231-45.
•
McGaghie, W. C., Issenberg, S. B., Cohen, E. R., Barsuk, J. H., & Wayne, D. B.
(2011). Does simulation-based medical education with deliberate practice yield
better results than traditional clinical education? A meta-analytic comparative
review of the evidence. Academic Medicine, 86, 706–711.
Simulation vs Other Intervention
• Simulation is non inferior to other approaches
• Technology enhanced simulation shows small,
statistically significant benefit for knowledge and
skills outcomes
• Approached statistical significance with direct
patient benefit and provider behaviors
Simulation vs Other Intervention
• Anesthesia review showed moderate effects for
satisfaction and skills, large effect for provider behavior,
small effect for direct patient benefit compared with non
simulation instruction
• Negligible effects comparing simulation to alternative
simulation interventions
• Analysis showed inconsistencies in measurement of non
technical skills
Pooled effect sizes for studies comparing training with no training. Effect sizes
represent Cohen’s d or the nearly-equivalent Hedges’ g from random-effects
meta-analysis; > 0.80 is large, 0.50–0.79 is moderate. Data derived from metaanalyses of Internet-based instruction (Cook et al. 2008b) virtual patients (Cook
et al. 2010a), simulation-based instruction (McGaghie et al. 2011), and
simulation-based instruction (Cook et al. 2011a)
Simulation based education is
probably as good as (but is not
necessarily better than) other types of
instruction.
There is a high degree of variability
between studies, suggesting that
certain simulation interventions may
be more effective in certain scenarios.
HOW CAN WE IMPROVE THE
EFFECTIVENESS OF SIMULATION
TRAINING?
Comparative Effectiveness
• Evaluates outcomes and processes leading to the
outcome
– Considers the costs, barriers, unforeseen consequences,
and effective strategies associated with implementing
therapies in practice.
• Studies comparing different simulation based
approaches to explain what works, for what audience
and in what context.
Comparative Effectiveness
• Sample sizes must be large
• Confounding can be a problem
• Effect size?
– The difference between teaching and no teaching
should be large, differences between teaching
types may not be as apparent
Comparative Effectiveness
• Cook DA, Hamstra SJ, Brydges R, et al. Comparative
effectiveness of instructional design features in simulationbased education: systematic review and meta-analysis. Med
Teach. 2013;35:e844-e875.
Comparative Effectiveness
• 289 studies, 20K participants
• Feedback, repetition, range of difficulty, cognitive
interactivity, clinical variation, distributed practice,
individualized training, and longer training time
significantly improve skill outcomes
• Patient outcome analysis revealed benefits of similar
direction and magnitude that approached statistical
significance
Future research should clarify how to
choose between simulation and nonsimulation approaches using rigorous
qualitative studies that explore the
strengths and appropriateness of each
approach.
IS SIMULATION TRAINING COST
EFFECTIVE?
What is the Value?
• Must consider costs
– Simulator, faculty time, training expenses, facility
fees, opportunity cost
• No study has offered a complete accounting
of simulation costs
Cost Assessment
• Zendejas B, Wang AT, Brydges R, Hamstra SJ, Cook DA.
Cost: the missing outcome in simulation-based medical
education research: a systematic review. Surgery.
2013;153:160-176.
• Norman G, Dore K, Grierson L. The minimal relationship
between simulation fidelity and transfer of learning. Med
Educ. 2012;46:636-647.
What is the Value?
• Cost reporting is infrequent and incomplete
• Low-fidelity, low-cost training models can
yield outcomes equal to much more
expensive simulators
More expensive simulators are not
necessarily better.
CONCEPTUALLY, SIMULATION
TRAINING MAKES SENSE
Simulation Training = Patient Safety?
• Ziv A, Wolpe PR, Small SD, Glick S. Simulation-based medical
education: an ethical imperative. Acad Med. 2003;78:783-788.
Simulation Education = Patient Safety?
• “Risk free” skill rehearsal
– Repetitive, deliberate and structured practice
• “Risk free” assessment and feedback
Simulation based rehearsal is
evidence supported when learning to
work with “real patients.”
SIMULATION MAY NOT ALWAYS BE
THE BEST LEARNING MODALITY
Is Simulation the Best Choice?
• Instructional design should be based on learning
objectives, learner needs, safety concerns,
resource utilization
• Virtual patients often used to teach clinical
reasoning
• Technology enhanced simulation is most often
for procedural training
How Do We Optimally Implement
Simulation Education?
• More research to clarify how to choose
between sim and non sim approaches
• Cost effectiveness research to determine true
and comparison value of sim education
How Do We Optimally Implement
Simulation Education?
• Comparative effectiveness research should focus
on what works in simulation – for which audience
– for what circumstances – and at what cost.
• Task analysis should focus on critical actions and
can help determine level of fidelity necessary for
type of training
How Do We Optimally Implement
Simulation Education?
• Careful selection and sequencing of events
surrounding the simulated task
• Appropriate faculty development and support
• Institutional commitment to simulation
QUESTIONS?