Designing, Developing and Evaluating Event-Based Team
Simulations: Helping Medical Educators Put Theory Into Practice
James A. Grand1, Rosemarie Fernandez2, Steve W.J. Kozlowski1, Georgia T. Chao3, Jason L. Huang1, Paul Curran1
1Michigan
State University Department of Psychology, East Lansing, MI, United States
2Wayne State University School of Medicine Department of Emergency Medicine and Office of Student Programs, Detroit, MI, United States
3Michigan State University Department of Management, East Lansing, MI, United States
Abstract
The current effort presents a methodological procedure for
constructing reliable, valid and event-based medical simulations as
well as the measurement instruments needed to assess the
teamwork skills and clinical expertise of medical teams. Illustrative
examples are provided from a simulation designed by the authors
for evaluating emergency medical teams that demonstrate the utility
and practicality of the approach. The benefits and implications of
the procedure are explicated, and practical advice to medical
educators is offered for employing the methodology across various
team training applications in the field of medicine.
Step 1: Scenario Design
A. Select a realistic medical scenario that teams might face
relative to the purpose of the training context.
• Consult hospital records or medical practitioners from various locales (i.e.,
urban/metropolitan, suburban, community/rural) to generate potential
situations and serve as justification for a scenario’s representativeness
B. Identify important events in the scenario based on objective,
reliable indicators.
• An event is any substantive team task (e.g., resuscitation, intubation)
with a clearly identifiable beginning (change in heart rhythm, lung
collapse) and ending (return to normal sinus, stabilization of O2 sats)
C. Script the ”flow” of events and/or team responses into a
standardized event-based scenario.
• Scenarios can follow a simple sequential pattern (Figure 1) or a more
complex, non-linear pattern depending upon the requirements of the
medical tasks at hand
Introduction
Beyond clinical task knowledge, teamwork and communication
skills among medical professionals are critical to effective patient
safety and care. Some recent reports have even suggested that
between 70-90% of adverse events in patient care (e.g.,
preventable deaths, etc.) can be attributed to failures in
communication and/or team-related functioning (Institute of Medicine, 2000;
Example. A patient is admitted to the emergency room with complaints of
respiratory distress and altered mental status (Event 1). After a short time, the
patient becomes hypoxic and requires intubation (Event 2). Following intubation,
the patient suddenly enters cardiac arrest and the team must resuscitate using
standard ACLS protocol (Event 3).
Shojania et al., 2001).
Although major advances in simulation technologies have improved
the assessment of medical expertise substantially, many current
strategies for evaluating and teaching team skills are less well
developed or understood. In response, a number of researchers
have proposed using event-based simulation (EBS) methodologies
to investigate team-related phenomena and performance in the
domain of medical education (e.g., Rosen et al., 2008). While undoubtedly a
positive step, implementing and taking advantage of EBSs can be
difficult for users unfamiliar with the techniques required for proper
development, and no standard procedure currently exists for
designing EBSs and any accompanying evaluative tools.
This study attempts to fill this void by providing a practical
methodology for developing EBS training/research platforms, using
examples from the authors’ work with emergency medicine teams.
Event 2
Intubation/Stabilization
Event 3
Arrest
Begin (E1)
Team enters room
End (E1)/Begin (E2)
O2 ≤ 85%  Intubate
End (E2)/Begin (E3)
V-Fib rhythm  Resuscitate
O2 reaches 85%
Intubation complete/
Consult call made
End (E3)
Sinus rhythm established
OR
OR
O2 below
85% for
1 min.
No call
within 1
min.
A. Use subject matter experts to evaluate the content validity
of the teamwork measure.
• Content validity refers to the representativeness, adequacy and
relevance of the items composing a particular measure; content validity
ensures that no important material is left out of the assessment and no
unnecessary material is included in the assessment
• Select individuals who are experts in team evaluation and research
• Provide subject matter experts with a brief description of how teams are
expected to interact in the scenario, relevant details about the team
environment and the overall flow of team activity
• Ask subject matter experts to rate the relevance/accuracy of the team
process dimensions for each event (Figure 4)
Team Processes and Example Behaviors for Event 1
This team process is necessary for the situation described thus far:





Strongly
Disagree
Disagree
Neither agree
or disagree
Agree
Strongly
Agree
Mission Analysis
Examine patient’s medical history charts and communicate findings to team; requests
updated vital signs from nurse; assesses patient’s symptoms and communicates
findings to team





Strongly
Disagree
Disagree
Neither agree
or disagree
Agree
Strongly
Agree
Goal Specification via Role Fulfillment
One/two members conduct physical exam of patient and identifies/communicates
results to team; one member monitors/communicates patient vital signs and status





Strongly
Disagree
Disagree
Neither agree
or disagree
Agree
Strongly
Agree
Strategy Formulation and Planning
Vital signs are interpreted and decision to start IV drug treatment is made





Strongly
Disagree
Disagree
Neither agree
or disagree
Agree
Strongly
Agree
Team Leadership
Individual in charge of patient’s charts solicits ideas/diagnoses from team members;
keeps track of information being delivered by other team members
Figure 4
• Select individuals who are experts in the area of relevant medical
practice
• Provide subject matter experts with a brief description of the medically
relevant details of the scenario and each event, and the overall flow of
medical activity
• Ask subject matter experts to rate the relevance/accuracy of the
performance dimensions for each event (Figure 5)
Event 1 Description
Intubation complete/
Prompt consult call
Figure 1
Step 2: Mapping and Measure Development
A. Identify observable teamwork behaviors evoked by scenario
events that are indicative of team process dimensions.
• Create a dichotomously scored checklist of behaviors that should occur
in each event related to effective TEAM functioning (Figure 2) – activities
representative of planning, helping, info sharing, etc.
• Create behaviorally anchored rating scales that can be used for global
ratings of teamwork effectiveness (Figure 3)
B. Identify observable performance achievements evoked by
scenario events that are indicative of medical expertise.
• Create a dichotomously scored checklist of objective performance
indicators in each event related to effective TASK functioning – derived
from physician standards of practice and physiologic data (e.g., correct
drugs ordered, correct dosage prescribed, etc.)
Clinical Indicators
Medical Actions
1. 34 yo patient presents from a rehabilitation
facility with a decreased level of consciousness
and respiratory distress.
2. The patient answers questions slowly, is
arousable and appropriately responds to pain.
3. The patient arrives with a chart containing:
(a) transfer note
(b) medications, allergies, and brief PMH
(c) unsigned DNR form
1. Temp = 103º F
2. BP 95/50
3. HR 110
4. RR 24
5. O2 Sat 94%
1.
2.
3.
4.
5.
CXR is ordered and interpreted
Orders Tylenol
Orders IV fluid
ECG is ordered and interpreted
Laboratory orders are placed
(a) ABG
(b) CBC
(c) Blood cx
(d) Lytes, BUN, Cr
(e) Glucose or FSBS
(f) UA / Urine culture
The events and information described in
Event 1 is representative of a patient that
might present to an emergency
department.
The clinical indicators described above
are consistent with the patient described
thus far.
The medical actions described above are
reasonable based on the patient and
clinical indicators described.
(1) Strongly agree
(2) Agree
(3) Neutral
(4) Disagree
(5) Strongly disagree
(1) Strongly agree
(2) Agree
(3) Neutral
(4) Disagree
(5) Strongly disagree
(1) Strongly agree
(2) Agree
(3) Neutral
(4) Disagree
(5) Strongly disagree
Figure 5
Step 4: Simulation Assessment
A. Run the simulation with pilot teams to ensure performance
data and team process indicators can be reliably captured.
• If possible, capture video from multiple views in the simulation room to
increase the likelihood that specific behaviors can be readily identified,
recorded, and viewed after the simulation is completed (Figure 6)
Mission Analysis
Event 2
Dimension
EBS methodologies explicitly structure a task (e.g., resuscitating a
patient, attending to a trauma victim, etc.) such that pre-scripted
events are included that require specific team responses. Coupled
with the Marks et al. (2001) taxonomy, EBSs can thus be designed
to elicit a wide range of identifiable teamwork and communication
behaviors necessary for effective team and task functioning in a
given context during certain periods of time.
Results
B. Use subject matter experts to evaluate the content validity
of the performance measure.
Event 1
Diagnosis
Prompt intubation
Team Process and Event-Based Simulations
Team processes refer to broad categories of functioning that
describe the activities of effective teams (e.g., planning, helping,
information sharing, etc.). While numerous sets of team processes
have been proposed, the present methodology works from a
temporally-based taxonomy developed by Marks et al. (2001)
because (1) the taxonomy facilitates the creation of an EBS
assessment tool by indicating when team processes should occur
and (2) empirical evidence exists linking elements of the taxonomy
to team effectiveness (LePine et al., 2008).
Step 3: Measurement Validation
Behavior
Team member(s) filled intubating role by (check all that apply):
Complete Skill
5
Very much skill
4
✔
-- inserting ET tube
GS
-- stabilizing neck if halo removed
Adequate skill
3
-- bagging (i.e., connecting bag to ET tube and squeezing)
-- monitoring/communicating pulse oxygen readings
PM/BB
PM/BB
O2 saturation displayed on the monitor are communicated to team
during intubation process
Some skill
Hardly any skill
Requested verification of ET tube placement from radiologist
Figure 2
Figure 3
Team fully and
accurately assesses
patient status and all
members are informed of
situation
Team assesses patient
status, but focuses too
exclusively on certain
pieces of information or
does not communicate
information to all
members
1
•
•
•
•
A detailed script for an event-based medical scenario describing the event
triggers and expected flow of the simulation
Measurement tools for evaluating the expected team processes and
performance criteria within each event
Validation forms for assessing the adequacy of the team process and
performance metrics
An inclusive system for monitoring teams during and after simulations that
can be readily adapted to account for unexpected occurrences in the
scenario
Practical Advice & Tips
1. Don’t expect to design the perfect simulation on your first
attempt. The process will almost certainly be an iterative one as
you update and revise the simulation’s events and measures to
best capture the teamwork and performance criteria of interest.
2. A good simulation can capture a lot—but it can’t capture
everything. Plan to make a series of simulations that require
both unique and overlapping areas of proficiency so that you can
assess participants’ skills across a variety of situations.
3. Evaluation is only the first step. Consider how the gathered
information from the simulation can best be put to use. For
novice teams, it might be most beneficial to focus on providing
feedback on performance issues rather than critiquing teamwork
behaviors; however, more experienced teams may find direct
feedback regarding breakdowns in teamwork more useful.
Discussion & Implications
Regardless of the pace at which medical simulation technology
continues to advance, accurate and valid measurement of learners’
actions and performance will remain a constant and critical need for
educators to consider. Given the importance of effective team
communication and clinical expertise for improving the overall
quality of healthcare, future simulations designed to instruct and
train students, residents and other medical specialists must be able
to adequately assess both performance and teamwork capabilities.
The present effort improves upon previous simulation design
recommendations in that it provides a concrete series of steps to
constructing EBSs predicated on an evidence-based approach to
measurement. In following the outlined procedure, medical
educators’ can ensure that the design and development of EBSs in
medical practice follow sound methodological principles, which
ultimately allows for a comprehensive evaluation and critique of the
teamwork and clinical skills required in proper medical practice.
References
Institute of Medicine (2000). To err is human: Building a safer health system. Washington, D.C.: Institute of Medicine.
LePine, J. A., Piccolo, R. F., Jackson, C. L., Mathieu, J. E., & Saul, J. R. (2008). A meta-analysis of teamwork processes: Tests of a
multidimensional model and relationships with team effectiveness criteria. Personnel Psychology, 61, 273-307.
2
Team does not reach an
overall assessment of
patient status, ignores
certain aspects of
patient status, or fails to
communicate any
mission information to
team members
The present procedure produces a systematically developed, highly
reliable EBS and accompanying measurement tools for observing
and evaluating teamwork and performance. The steps described
here result in the creation of:
Figure 6
B. Readjust events and measurement tools to capture desired
teamwork behaviors and performance criteria as necessary.
Marks, M. A., Mathieu J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of
Management Review, 26, 356-376.
Rosen, M.A., Salas, E. Wu., T.S., Silvestri, S., Lazzara, E.H., Lyons, R., Weaver, S.J., & King, H.B. (2008). Promoting teamwork: An eventbased approach to simulation-based teamwork training for emergency medicine residents. Academic Emergency Medicine, 15, 1-9.
Shojania, K.G., Duncan, B.W., McDonald, K.M., Wachter, R.M.(2001) (Eds.). Making Health Care Safer: A Critical Analysis of Patient Safety
Practices. Rockville, MD: Agency for Healthcare Research and Quality.