From Learning Objectives to
Outcomes
Assessment During Simulation
Paul E. Phrampus
Director
Winter Institute for Simulation, Education and
Research (WISER)
University of Pittsburgh
Assessment
• Why
• What
• When
• Where
• How
The Future of Simulation………
What is a simulation for?
1.
2.
3.
4.
Education
Assessment
Fun
1 and 2
The Simulation Advantage
TESTING
TEACHING
Psychomotor Skills
Decision Making
Skills and Decisions
Individuals  Teams
What is “Simulation”….. Really…..
The Job of Healthcare Educators
Psychomotor Skills
Decision Making
Base Knowledge
Communications Skills
Teamwork Skills
Professionalism Skills
Why is this so hard?
Learning Objective
• You have to have a clear reason of why you
are running any simulation.
Lets Watch
• What is most important aspect to debrief?
Where does a simulation fit?
• It depends on what your doing!
EACH Sim is part of a bigger plan!
What is this?
Does it belong here?
Course (Global) objectives v sim
objectives
Global Objectives
• Course Objectives
– 1. Demonstrate a proper patient assessment
– 2. Demonstrate following the emergency protocol
– 3. Demonstrate Airway Management Skills
Course Schedule / Flow
•
•
•
•
•
•
Sim 1
Sim 2
Lecture
Skills Lab
Sim 3
Sim 4
Course
Objectives
Session Objective
1.Ventilate w BVM
2.Recognize Narcotic Overdose
3.Activate Protocol
How do you determine “success”
of a simulation?
1. Students liked it
2. Learning objectives
achieved
3. Students passed the
scenario
4. Not Sure
Su
re
No
t
..
as
se
.
St
ud
en
ts
p
ng
o
ni
Le
ar
St
ud
e
nt
sl
ik
bj
e
c. .
.
ed
. ..
25% 25% 25% 25%
How do you determine “success”
of a simulation?
Instructor Number 1
Instructor Number 2
Instructor Number 3
How do you determine “success”
of a simulation?
Instructor Number 1
Instructor Number 2
Measure Here
Instructor Number 3
You can only keep track of so
many things at one time
How many of you feel like this ?
Tools for Success
• Make it easy for your faculty to stay on track
• “In my opinion…..”
Assessment Methodology
– Identify methods for measuring core aspects in
simulation education / assessment
– Discuss development and deployment of
measurement tools
Student
Learning
Outcomes
Curriculum
Teaching &
Learning
Assessment
Evaluation
Assessment and Evaluation
• Assessment and Evaluation
often used interchangeably
• Assessment – learner outcomes
• Evaluation – course or program outcomes
Performance Assessment
Rate the Performance
• Pass or Fail
• Score Performance
• Procedural Check
Assessment
• Why
• What
• When
• Where
• How
Rosen, MA et al. Measuring Team Performance in Simulation-Based Training: Adopting Best Practices for Healthcare. Sim
Healthcare 3:2008;33–41.
Assessment of clinical
skills/competence/performance
ACTION
PERFORMANCE
COMPETENCE
KNOWLEDGE
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Performance Assessment
• Basic to performance –
Do they know it and know how?
• Competence –
Can they do it?
• Performance –
Do they do it?
Assessment of clinical
skills/competence/performance
DOES
SHOWS HOW
KNOWS HOW
Knows When
KNOWS
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment of clinical
skills/competence/performance
DOES
Practice Review
Audits, Video, UC-SP, 360O
SHOWS HOW
Performance Tests
KNOWS HOW
Clinical Based Tests
Knows When
KNOWS
OSCE, TaskDemo, SIM/SP
Pt Mgt Qs, Essay, Oral
Knowledge Tests
MCQs, Essay, Oral exam
Miller GE, The assessment of clinical skills/competence/performance. Academic Med. 1990 Sep;65(9):S63-7.
Assessment
• Formative Assessment
–
–
–
–
Lower stakes assessment
One of several assessments over time of course or program
May be evaluative, diagnostic, or prescriptive
Often results in remediation or progression to next level
• Summative Assessment
–
–
–
–
Higher stakes assessment
Generally final course or program assessment
Primary purpose is performance evaluation
Often results in a Go-No Go outcome
Assessment
• Assessment data sources
– Live observation
– Video recording
– SIM device logs
• Instruments
–
–
–
–
AV annotation
Pencil/Paper
Scantron form
TabletPC or PDA
• Assessors
–
–
–
–
–
Instructors
Observers
SIM/Patients
Peers
Participants
Variables in Assessment
STUDENT
PATIENT
ASSESSMENT
EVALUATOR
Assessment Validity
• Validity –
– Degree to which an assessment tool measures what it is
intended to measure
– Construct validity
– Accuracy of tool to measure intended construct
• Concurrent validity
• Content validity
• Predicative validity
Assessment Reliability
• Test Reliability –
– consistency of a test to yield the same measure over
time (does not assure validity)
– Test-Retest Reliability
consistency among different administrations
– Parallel Forms Reliability
to assure that memory effects do not occur use a
different pre- and posttest
– Inter-Rater Reliability
two or more observers rate the same subjects with
strong positive correlation of their ratings
Variables in Assessment
STUDENT
PATIENT
ASSESSMENT
EVALUATOR
Assessment Improvement
• Improving Reliability and Validity
– Base assessment on outcome/objectives >
event triggers > observable behavior >
behavioral rating > assess against competencies
– Define <low – medium – high> performance
– Use of Rubric or rating metric
– Use (video) training examples of performance
– Employ a quality assurance/improvement system
Assessment Based on
Objectives and Events
Rosen, MA, et al A measurement Tool for Simulation –Based Training in Emergency Medicine: The Simulation Module for
Assessment of Resident Targeted Event Responses (SMARTER) Approach. Sim Healthcare 3:2008;170-179.
Event – Behavior – Scale - Report
Rosen, MA, et al. Measuring Team Performance in Simulation –Based Training: Adapting Best Practices for Healthcare.
Sim Healthcare 3:2008;33-41.
Assessment Metrics
• Procedural or Check List assessment
• Global Rating assessment
Assessment Metrics
• Procedural or Check List assessment
BCLS
BCLS
Y N
Y N
Open Airway
Open Airway
Check Breathing
Check Breathing
(< 5 sec of LOC)
(< 5 sec of Airway)
BCLS
Y N A
Open Airway
Check Breathing
Rating Score
+1 -1
0
*Assist
Assessment Metrics
• Global Rating assessment
Code Blue
P F
Code Blue
CPR and
CPR
ACLS
ACLS
<1(low) - 5(Hi)> points
<1(low)- 5(Hi)> points
Code Blue
H M L
CPR
ACLS
Rating Score
+1 -1
0
Pts.
Assessment Metrics
• Procedural or Check List assessment
• Global Rating assessment
< EXAMPLES>
• Pros and Cons to each
• Multiple modes & points vs. Single assessment
• QA/QI Metric – collective vs single assessment
Brett-Fleegler, MB et al. A simulator-based tool that assesses pediatric resident resuscitation competency. Pediatrics
121(3):2008;597-603.
Gaba, DM et al. Assessment of Clinical Performance during Simulated Crises Using Both Technical and Behavioral Ratings.
Anesthesiology, 89(1)July 1998:8-18.
Guise, JM et al. Validation of a Tool to Measure and Promote Clinical Teamwork. Sim Healthcare 3:2008;217–223.
Interrater Reliability
Interrater Reliability
of Data Collected with SimMan
Andrus, Phrampus, Wang
During Difficult Airway Simulations
Evaluator
9
Evaluator
1
Evaluator
2
Novices
Evaluator
8
5 Scenario
Videos
Evaluator
3
Evaluator
4
Evaluator
7
Evaluator
6
Evaluator
5
Interrater Reliability of Data Collected with SimMan
During Difficult Airway Simulations
Andrus, Phrampus, Wang
1.2
1
0.8
9 Raters
0.6
6 Raters
3 rookies
0.4
0.2
as
a
b
D
iA l
id
nt ttm
p
C
al t
lH
el
p
N
om
C
xa
m isto
in
ry
e
Ai
A
ss rwa
is
tS y
po
nt
E
to
m
P
re Su
ox
c
yg c
S
ur en
a
gi
ca te
lC
ric
h
B
VM
1
B
VM
D
ire 2
ct
La
O
ra r
la
O w1
ra
la
w
2
H
E
C
al
l
H
el
p
0
Interrater Reliability of Data Collected with SimMan
During Difficult Airway Simulations
• CONCLUSIONS:
– Simulation instructors exhibited good overall
inter-rater agreement but showed poor
agreement for some key airway tasks.
– Variability in instructor ratings may influence realtime evaluation of simulation performance.
– Likely could be improved with careful instructor
orientation / training
Andrus, Phrampus, Wang
Evaluation
• Individual Trainee Evaluation
– Per episode performance
– Course of the Day
Evaluation
• Group Trainee Evaluation
– Benchmark
Evaluation
• Instructor Evaluation
– Per episode performance
– Measure Against Benchmarks
Testing Points
Behind the Scenes
Behind the Scenes
Behind the Scenes
Behind the Scenes
Behind the Scenes
Summary
•
•
•
•
Know why you are doing the sim
Each sim is part of a bigger picture
Design tools for consistency
Make it easy for your faculty!
Questions and discussion