2nd July 2012
Developing scenarios and evaluating
performance in medical simulation
Steven Yule PhD
STRATUS Center for Medical Simulation
Brigham & Women’s Hospital, Boston
Department of Surgery, Harvard Medical School
The challenge: Defining how learners should act, think or
feel differently as a result of your simulation program
1. Basics of scenario development
2. Performance evaluation
Basics of scenario development
Checklist for designing a simulation scenario
① Define aim
② Needs assessment & learning objectives
③ Learning outcomes (for evaluation)
④ Develop a matrix
⑤ Work on evaluation strategy
⑥ Identify resources and people required
⑦ Programming and software
⑧ Dry run
⑨ Pilot with real participant
⑩ Run program
Performance evaluation
To successfully evaluate learners, you need to
complete these three tasks as part of the scenario
design process:
#1 Consider the
evaluation options
#2 Review learning
objectives
#3 Select strategy
Task # 1: Consider evaluation options
• Written record
• Written exam
• Log
• Oral examination
• Attendance
» Structured (oral patient
management problem)
» Short answer
• Videotape and audio tape
review
• Simulated Patient
• Objective Structured
Clinical Examination
• Case Presentations
• Direct Observations
Task # 2: Review learning objectives and select the
most appropriate evaluation method according to
these criteria:
Feasibile
Practical
Economical
Objective
Comprehensive
Valid
Reliable
Task # 3: Select strategy and implement
• How are the data to be collected?
• By what mechanism or by who?
For example:
• “Leadership behaviors will be assessed during the
scenario from video by blinded expert observers
using the Surgical Leadership Inventory”
• “Confidence in maintaining an airway during an
anaesthetic crisis will be measured by pre- and postcourse survey”
A worked example…
NOTSS skills taxonomy v1.2
Categories
Elements
Situation Awareness
Gathering Information
Understanding Information
Projecting and anticipating future state
Decision Making
Considering options
Selecting and communicating option
Implementing and reviewing decisions
Communication and
Teamwork
Exchanging information
Establishing a shared understanding
Co-ordinating team
Leadership
Setting and maintaining standards
Coping with pressure
Supporting others
Yule et al (2006) Medical Education, 40, 1098-1104
NOTSS rating form
Simulation coaching
NOTSS coaching
significantly improved
skills in a randomized
controlled trial of 16
senior surgical trainees
Simulation-based team training (2012-2015)
Designing good simulation scenarios is impossible
without clear learning outcomes and an evaluation
strategy: summary
1.
Decide what you want people to do differently after
your session
2.
Use a structured method of scenario design
3.
Select most appropriate evaluation method for learning
outcomes using criteria
Contact
» syule@partners.org
» http://scholar.harvard.edu/yule/