Defining Team Performance for Simulationbased Training: Methodology, Metrics, and
Opportunities for Emergency Medicine
Marc J. Shapiro, MD, Roxane Gardner, MD, MPH, Steven A Godwin, MD, Gregory D. Jay, MD, PhD,
David G. Lindquist, MD, Mary L. Salisbury, RN, MSN, Eduardo Salas, PhD
Abstract
Across health care, teamwork is a critical element for effective patient care. Yet, numerous well-intentioned training programs may fail to achieve the desired outcomes in team performance. Hope for the
improvement of teamwork in health care is provided by the success of the aviation and military communities in utilizing simulation-based training (SBT) for training and evaluating teams. This consensus
paper 1) proposes a scientifically based methodology for SBT design and evaluation, 2) reviews existing
team performance metrics in health care along with recommendations, and 3) focuses on leadership as a
target for SBT because it has a high likelihood to improve many team processes and ultimately performance. It is hoped that this discussion will assist those in emergency medicine (EM) and the larger
health care field in the design and delivery of SBT for training and evaluating teamwork.
ACADEMIC EMERGENCY MEDICINE 2008; 15:1088–1097 ª 2008 by the Society for Academic Emergency Medicine
Keywords: team, training, simulation, emergency medicine, metrics, leadership
H
ealth care delivery is an extremely complex
operation, and despite the best intentions of
many dedicated professionals, we are constantly reminded of the perils of the significant errors in
reported sentinel events. Newsworthy errors, such as
wrong-site surgery, highlight deficiencies in nontechnical skills and behaviors including teamwork. Across
health care, teamwork is a critical element for effective
patient care. Yet, reports reviewing the effectiveness of
healthcare teams have consistently supported the saying
From the Department of Emergency Medicine (MJS, GDJ,
DGL), the Department of Medicine (GDJ), and the Department
of Engineering (GDJ), Warren Alpert School of Medicine,
Brown University, Providence, RI; the Department of Obstetrics, Gynecology and Reproductive Biology, Harvard Medical
School (RG), Boston, MA; the Department of Emergency Medicine, Simulation Education, University of Florida (SAG), Jacksonville, FL; The Cedar Institute, Inc. (MLS), North Kingstown,
RI; and the Department of Psychology, Institute for Simulation
& Training (ES), Orlando, FL.
Received July 9, 2008; revision received July 18, 2008; accepted
July 18, 2008.
This is a proceeding from a workshop session of the 2008
Academic Emergency Medicine Consensus Conference, ‘‘The
Science of Simulation in Healthcare: Defining and Developing
Clinical Expertise,’’ Washington, DC, May 28, 2008.
Address for correspondence and reprints: Marc J. Shapiro,
MD; e-mail: mshapiro@mvhospital.org.
1088
ISSN 1069-6563
PII ISSN 1069-6563583
that ‘‘a team of experts does not necessarily constitute an
expert team.’’ The truth to this statement is demonstrated by a 2004 report indicating that 72% of the rootcause analyses of sentinel events in perinatal units across
the United States identified failures in communication
(a core teamwork skill) as a contributor.1 Similarly compelling data were obtained by a needs analysis conducted
during the MedTeams! project.2 Approximately 40% of
emergency department (ED) closed claim cases involve
teamwork errors that could have been mitigated or
prevented.2
In response to such statistics, many health care organizations have attempted to implement team training
programs. Some have succeeded, while others have
struggled to achieve desired outcomes. Hope for the
improvement of teamwork in health care is provided by
the success of the aviation and military communities in
utilizing simulation-based training (SBT) for training
and evaluating teams.3
The earliest attempt to codify teamwork behaviors
and their measurement among emergency care providers occurred during the MedTeams! project, a multicenter prospective investigation of crew resource
management (CRM) from 1996 to 2001.4 The results
from the MedTeams! project showed that an increase
in safety-related behaviors is achievable through a wellconstructed didactic curriculum designed to inculcate
team behaviors in groups of clinicians.4 Teamwork was
measured through 1) behavioral anchored rating scales
ª 2008 by the Society for Academic Emergency Medicine
doi: 10.1111/j.1553-2712.2008.00251.x
ACAD EMERG MED • November 2008, Vol. 15, No. 11
•
www.aemj.org
(BARS), 2) process measures, and 3) outcome measures
and attitudinal behaviors. BARS, accompanied by direct
observations of error, has also been used successfully
in simulated settings to study a CRM intervention.5
As teamwork training becomes more standardized,
considerations should be made to select or develop
measures that are easy to use by clinicians and possess
shared validity in both clinical and simulated settings.
This will blur the division between training and actual
ED practice. The verisimilitudes of teamwork behaviors
in emergency medicine (EM) are diverse, requiring both
didactic and SBT in multipatient settings.6 Two notable
deficiencies in the MedTeams! curriculum and measurement tools were in the domains of leadership skills
and shared team cognition. Both MedTeams!, and its
next generation, TeamStepps!, have been operationalized in many EDs, but issues of measurement and
sustainment, both within and outside of SBT, persist.
These team training programs, along with a recent
report,7 provide a foundation for core team processes
likely to have the greatest impact on team effectiveness.
The popularity of SBT is largely due to its ability to
offer a lifelike learning environment where clinicians
can practice their skills. In the absence of live patients,
health care providers can operate in a setting in which
it is safe to try new skills without risk of patient harm.8
The malleability of SBT offers health care providers
exposure to uncommon, yet critical, cases they are unlikely to see on the job.
Emergency medicine has had the vision to see the
potential for SBT for teamwork, patient safety, and
graduate medical education.5,9–11 However, despite this
enthusiasm, there is no unified teamwork taxonomy to
promote collaborative training or research efforts.
Additionally, measurement tools have received limited
validation and have not fully leveraged the extensive
body of literature on team performance. In this article,
we provide a review of the literature on team performance metrics to assist educators and researchers. In a
related article in this journal issue, Fernandez et al.12
describe a teamwork taxonomy and framework that we
integrate into our efforts to describe measurement of
team performance.
Measurement of ED team performance is truly complex because of the interrelated nature of many individuals and events. EM is challenging because of the
changing environment and resources, multiple patient
encounters, and the multidisciplinary nature of patient
care. Recognizing this challenge is instructive to our
efforts in SBT teamwork training. Successfully training
and assessing team and individual skills requires multiple focused simulations directed to specific knowledge,
skills, and attitudes (KSAs). Thus, a sound methodology
for SBT is essential to facilitate valid measurements to:
1) diagnose team and individual weaknesses and
strengths, 2) determine future training needs, 3) determine effectiveness of training and improvements in
team performance, and 4) determine improvements in
actual care-delivery processes and outcomes.
The nature of EM practice places us front and center
not only with challenging clinical demands, but with
the need to coordinate patient care across multiple specialties. Clinical expertise is necessary but not sufficient
1089
to ensure patient safety. A recent evaluation of patient
safety after the Institute of Medicine, ‘‘To Err is
Human’’ report identified high priority areas for the
next 5 years.13 This report specifically highlighted the
important role and significant potential of more effective teamwork to improve patient safety. Teamwork
training is not a specific fix for any one type of error.
Rather, teamwork training should be viewed as an
adaptable human factors program with a set of teachable skills and behaviors capable of increasing system
resilience, a hallmark of a high reliability organization.
This consensus paper: 1) proposes a scientifically
based methodology for SBT design and evaluation, 2)
reviews existing team performance metrics in health
care along with recommendations, and 3) focuses on
leadership as a target for SBT team training, because it
has a high likelihood to improve many team processes
and ultimately performance.
PRINCIPLES FOR SBT DESIGN AND EVALUATION
Four principles are critical in the development and evaluation of a methodologically sound approach to the
design of SBT for teamwork in health care. In presenting these principles, we draw largely from the science
of training and team performance14,15 and also the Simulation Module for Assessment of Resident Targeted
Event Responses (SMARTER), a methodology developed by Rosen and colleagues16 (see Figure 1). Specifically, we focus on what is required in terms of SBT
design and evaluation to improve teamwork.
Simulation-based Team Training
Principle 1: Establish the Team-based Competencies
to be Assessed and Trained—The Learning Outcomes. The science of training14 instructs us that the
first step of any training program is to identify what
needs to be trained. Training for teamwork in healthcare is no different. The Accreditation Council for
Graduate Medical Education (ACGME) requires that
residents develop proficiency in six core competencies
(e.g., KSAs). Although ‘‘teamwork’’ is not overtly specified as one of these competencies, elements of teamwork are incorporated (e.g., the development of
effective communication skills is the primary emphasis
under the ‘‘interpersonal and communication skills’’
competency). Thus, teamwork relevant aspects of these
competencies should be considered in SBT design.
Figure 1. Overview of the Simulation Module for Assessment
of Resident Targeted Event Responses (SMARTER) processes
for training teamwork in EM. Adapted from Rosen et al.16
EM = emergency medicine; KSAs = knowledge, skills, and
attitudes.
1090
Shapiro et al.
•
DEFINING TEAMS: METHODOLOGY, METRICS, AND OPPORTUNITIES
In addition to the ACGME, other sources highlight
the teamwork competencies most critical and commonly required within EM. Over 20 years of theoretical
and empirical literature on team training abound.
Emphasized across this literature are three key components for health care teamwork: communication,17
coordination,18 and cooperation.19 To obtain a more
detailed, systematic view of teamwork competencies,
team task analysis identifies the components of a
task that are related to task work versus teamwork.20
Ultimately, tasks related to teamwork can be used to
estimate the frequency and relative importance of team
member coordination during specific tasks. Input from
those considered ‘‘experts’’ in their position is often
gathered as part of this process. SBT for teamwork will
be most beneficial when scenarios are designed to
target the team-based KSAs identified as common and
critical for performance. The competencies selected for
training serve as the content for the design of SBT
scenarios (see Principle 2) and performance measurement tools (see Principle 3).
Principle 2: Provide Opportunities for Guided Teamwork Practice—Carefully Craft Scenarios. Knowing
about teamwork does not mean one will behave as a
team member. Health care providers must be able to
perform teamwork skills. SBT provides a setting for the
hands-on guided practice of teamwork behaviors to
help ensure skill acquisition. Continued guided practice
leads to learning; however, not all practice is good
practice. Mere repetition does not guarantee the accurate development of teamwork skills. Rather, practice
must be guided to ensure that teamwork skills are accurately developed. Practice opportunities should focus
on a specific skill, with immediate provision of feedback
about what areas can be improved and how these
improvements can be made.
In conducting SBT, the ability to provide structured
and focused practice opportunities begins during scenario design. An effective SBT scenario targets only a
small subset of the desired teamwork training objectives. To ensure that trainees practice and demonstrate
the specific KSAs of interest, a scenario must contain
‘‘triggers’’ designed to elicit a given teamwork
response. These triggers prompt team members to
demonstrate a specific behavior. For example, in a team
scenario designed to elicit communication, a team
member may be instructed by a confederate to administer a medication to a patient for which the team was
previously told the patient was allergic. The team member would be expected to voice concern about the medication and remind the confederate about the allergy.
Any action other than this by the team member would
potentially result in a decline in the patient’s condition.
Following the simulation, immediate feedback, including specific positive and negative examples, should be
provided.
Principle 3: Measure Team Performance—Diagnose
Teamwork Strengths and Weaknesses. Measurement
creates an opportunity for learning. SBT works when
measurement problems (e.g., observation protocols,
behavioral markers) are designed to diagnose team per-
formance. Metrics, behavioral markers, and observation
protocols are a must for effective SBT. By designing
diagnostic measures that directly parallel the trigger
events and targeted teamwork concepts embedded
within the SBT teamwork simulations, observers know
exactly what they are watching for and when a
behavior should occur (see Figure 2). The subjectivity
of performance is therefore minimized. Furthermore,
designing measurement tools assists in ensuring thorough and relevant feedback, because a tool provides a
structured template. Debriefing can focus specifically
on whether expected behaviors did or did not occur.
Principle 4: Develop Robust Debriefing Protocols—
Link Feedback to Learning Outcomes. Feedback is,
of course, essential in the team learning process. Feedback informs the team about how they are performing
and identifies strengths and weaknesses. To be most
effective, feedback must be accurate, specific, developmental, and provided immediately following scenario
participation in a structured debrief session.21 A facilitator (typically the trainer) should use a diagnostic scenario-based tool to direct debriefings; however, all
team members should actively participate and contribute to the identification of learning opportunities and
suggestions for remediation.22 For open and honest
participation to occur, debrief sessions must be perceived as safe environments where learning is the
shared objective, rather than criticism or placing blame.
Team members should discuss aspects of performance
in the scenario that went well and the factors that contributed to the desired outcomes. This allows the team
to focus on repeating these successful behaviors in
future performance sessions. Remediation opportunities
may involve the repetition of a previously performed
scenario or a new scenario requiring a similar set of
teamwork skills.
Teamwork is too complex to be taught in a single
scenario. Rather, a collection of scenarios should be
Simulation Scenario: ED Resuscitation
Teamwork competency: Leadership
Event
55 year old male is
visiting another ED
patient and suddenly
collapses. The resident
is called into the room,
but the RN begins
yelling out orders.
The patient stops
breathing and does not
have a pulse
Additional ‘team
members’ (also
confederates) arrive and
stand at the patient’s
bedside awaiting
direction/orders
Critical response
The resident identifies him/herself as the
team leader
Hits
IG
Resident calls a brief huddle to clarify the
patient’s status, the team’s plan, and the
responsibilities of each team member
Resident verbalizes the change in patient
status to those in the room
Resident calls for/activates a “CODE” to
recruit more team members to help
Resident conducts a brief huddle to update
the new team members
Delegates tasks or assignments to each
new team member, indicating specifically
what each individual is responsible for
Figure 2. Example of an observational checklist for the team
competency of ‘‘leadership.’’ ‘‘Hits’’ are a dichotomous scoring
of whether or not the targeted behavior was observed.
IG = instructor guided; used when the behavior is observed but
was coached by a trainer. Adapted from Rosen et al.16
ED = emergency department; RN = registered nurse.
ACAD EMERG MED • November 2008, Vol. 15, No. 11
•
www.aemj.org
created, with each scenario only focusing on a small,
specific subset of the teamwork KSAs. Maintaining a
library of scenarios is also beneficial in that team training is not a ‘‘one size fits all’’ process. No two teams
are identical, and individual teams will change over
time as new technologies are introduced and medical
procedures and knowledge evolve. Scenarios emphasizing a limited set of teamwork competencies allow SBT
facilitators to better accommodate and focus on the
unique training needs of a given team.
TEAM PERFORMANCE METRICS IN HEALTH CARE:
REVIEW OF THE LITERATURE AND
RECOMMENDATIONS
What Are Team Performance Metrics and Why Use
Them?
The most common reason for evaluation is to determine
the effectiveness or impact of an intervention and the
ways in which it can be improved or strengthened.23
However, the information obtained from evaluation is
only that, information. To yield insight, the quality of
information must be accurate, reliably high, and
obtained through the use of high-performing instruments.24
For the purposes of this discussion, the team is identified as a co-located group of two or more individuals
working to plan, problem solve, and carry out safe,
quality care across a population of patients, over time,
within the context of high-stress, high-stakes, timecompressed environments where information is
unavailable or uncertain, such as in the ED; or within
the confines of a structured simulated setting. Teamwork is a complex construct requiring metrics broad
enough to capture the cycles of performance, yet sensitive and precise enough to assess a single behavior.23,24
No single measure can capture overall individual or
team performance, and it is helpful to measure both
team processes and outcomes to avoid the pitfalls in
relying on ‘‘one single inadequate criterion.’’
Thus, it is helpful to begin with the end in mind. The
noise of team complexity is then reduced, precision
gained, and the yield improved when evaluation is
event-driven.24 In simulation, events are determined by
the researcher to yield specific outcomes. Scenarios are
scripted, and performance criteria are determined with
the intent to run sequential scenarios of increasing difficulty.24 Instruments commonly composing team performance evaluation are perception surveys, team
performance observations, combined with process and
outcome measures. Which instruments to choose is a
matter of science.
Brief History of Team Performance Evaluation in
Health Care
The value of conducting direct, real-time observations
as a meaningful way to assess the caliber of teamwork
behaviors in health care settings is becoming widely
appreciated. However, this methodology is still in its
infancy. Gauging performance through direct observation of team behaviors is grounded in the knowledge
gained through research in behavioral psychology,
organizational behavior, human factors, and industrial
1091
and occupational safety. Since the 1940s, the military
has studied safety in the work place, leading to similar
safety-driven initiatives by nonmilitary, high-risk industries.25–33
Lauber30 first defined the term ‘‘cockpit (crew)
resource management’’ (CRM) to mean ‘‘using all available sources—information, equipment, and people—to
achieve safe and efficient flight operations.’’ CRM has
expanded to include the identification of potential
threats of error, to avoid or mitigate these threats, and
to improve morale and enhance efficiency of operations.31–33 Aviation and other non–health care industries
supplement their technical training programs with
CRM training.34 Behavior-based aviation safety audits,
such as line-oriented flight training and line operation
safety audits (LOSA) document team skill proficiencies
of leadership, communication, workload management,
and monitor ⁄ cross-check performance. A ‘‘nontechnical
skills’’ assessment tool was developed specifically to
evaluate CRM behaviors of individual pilots during
flight deck maneuvers.35 There is tension between
observing teams versus an individual’s performance,
because an individual’s performance can affect the
team’s performance. Metrics used to assess teams may
not be appropriate for assessing an individual teammate, and vice versa.
Within the health care arena, observing teamwork
while managing routine or critical events can reveal
much about medical errors and identify ways to prevent or mitigate their effects. A contextual understanding for how teamwork skills should be demonstrated
during routine and critical events is needed to guide
the development of standardized, behavior-based metrics. Gaba et al.36 first adapted aviation-based CRM to
anesthesia in 1989 and named it ‘‘anesthesia crisis
resource management.’’ Flin and Martin37 identified five
key elements to consider when developing behavioralbased assessment tools for CRM skills applicable to aviation and anesthesia teams. These elements include the
unit of assessment, identification of team skills and their
markers, the method of assessment, rater reliability,
and rater training. Translating aviation-based CRM
principles for use in anesthesia triggered the adoption
and spread of these techniques and use of behaviorbased assessments across the medical and surgical
specialties. Morey et al.4 developed and used a CRMbased, behaviorally anchored rating scale known as the
‘‘team dimensions rating form’’ for qualitatively assessing team behaviors of EM providers. The team dimensions form grouped team behaviors in five dimensions
(maintain team structure and climate, plan and problem
solve, communicate with team, manage workload,
improve team skills) and scored them on a 7-point
Likert scale for specific team behaviors, such as team
leadership, role clarity, structured communication,
mutual accountability and respect, conflict resolution,
cross-monitoring, situational awareness, and team ⁄ shift
updates.
Healey et al.38 created an ‘‘observational teamwork
assessment of surgery’’ (OTAS) tool to assess ‘‘what
(surgical teams) do and how they do it.’’ They recorded
specific clinical, technical, and interpersonal skills and
team behaviors among themselves and with their
1092
Shapiro et al.
•
DEFINING TEAMS: METHODOLOGY, METRICS, AND OPPORTUNITIES
patient. Teams were assessed for task-related elements
(planning, availability of appropriate equipment, completion of surgical checklists, and patient-related communication) and teamwork behaviors within the five
dimensions of cooperation, coordination, leadership,
monitoring, and communication.
Thomas et al.39 developed a teamwork audit for neonatal resuscitation, based on line audits used to assess
pilots’ CRM skills. Ten behavioral markers, including
information sharing, intentions shared, managing
workload, vigilance ⁄ environmental awareness, and
overall teamwork and leadership, were identified from
interviews, surveys of providers, and video observations of neonatal resuscitations.
Flin and Maran40 developed the anesthesiologist nontechnical skills (ANTS) tool, a behaviorally anchored
teamwork assessment tool derived from attitudinal surveys of anesthesiologists, real-time observations of
anesthesiologists caring for patients, and quality assurance reviews of critical incidents with adverse outcomes. ANTS divides nontechnical skills into four
general categories of task management, team-working,
situation awareness, and decision-making, each with 15
elements or behavioral markers.
Current Use of Health Care–related Team Performance Metrics
Metrics Assessing Generic Team Work. Most behavior-based teamwork assessment tools are specialtyspecific, but Baker et al.41 recommended that all
available resources be directed toward establishing
standard, generic teamwork skills, and competencies
for the health care industry. Frankel et al.42 developed a
non specialty-specific behavior-based assessment tool
known as the ‘‘communication and team skills’’ (CATS)
assessment. This tool, based on CRM behaviors
grounded in the aviation and military industries, was
designed specifically for the purpose of gauging
teamwork skills across health care professions.31,32,36,39
Currently undergoing statistical validation, the tool
identifies whether expected team behaviors are present
and evaluates their quality. The health care team assessment domains of CATS compares favorably with that of
LOSA, OTAS, and ANTS (Table 1).42
Metrics Assessing Specialty-specific Team Work.
There is no one size fits all, criterion standard team performance metric in health care, but much research is
being directed toward identifying those that are theoretically relevant and scientifically sound, capture competencies, focus on observable behaviors, and provide
meaningful information that facilitates improved performance and outcomes.15 All forms of metrics are in use
across the medical and surgical specialties, ranging
from event-based, automated performance monitoring
and self-report measures, to BARS and behavioral
observation scales (BOS; Table 2).15,24
There is also no criterion standard approach for
the process by which specialty-specific team performance metrics are developed. However, a stepwise
approach provides a suitable example of how such
tools should be developed (Table 3). In developing
surgical team performance measures, Yule et al.43
reviewed the literature for tools assessing nontechnical skills of surgeons in the operating room. They
identified core categories of nontechnical skills that
were measured, including communication, teamwork,
leadership, and decision-making, but deemed these
metrics to be psychometrically deficient. Flin et al.44
used attitudinal surveys to assess surgeon’s perception of safety in the operating room. The adverse surgical events were analyzed via critical incident
interviews and direct observation of surgeons, which
resulted in a taxonomy of teamwork behaviors and a
behavior-based rating system to assess such skills.45
Reliability testing of their instrument is currently
under way or nearing completion. Flin et al.46 then
developed a nontechnical skills curriculum, to train
surgeons and maximize safety and quality in the
operating room. This stepwise process, although
tedious and labor-intensive, provides a comprehensive
and focused understanding of the demands of the
Table 1
Comparing LOSA Behaviors to Behaviorally Anchored Health Care Team Assessment Domains
LOSA
Briefings
Plans stated
Workload assignment
Contingency management
Monitor ⁄ crosscheck
Workload management
Vigilance
Automation management
Evaluation of plans
Inquiry
Assertiveness
Communication environment
Leadership
OTAS
Coordination
Leadership
Awareness
Coordination
Awareness
Awareness
Cooperation
Leadership
Communication
Leadership
ANTS
Task management
Task management
Situational awareness
Team working
Task management
Situational awareness
Task management
Decision-making
Team working
Team working
Team working
Team working
CATS
Situation awareness
Communication
Situation awareness
Cooperation
Situation awareness
Coordination
Cooperation
Cooperation
Communication
Coordination
ANTS = anesthesiologist nontechnical skills; CATS = communication and team skills; LOSA = line operation safety audits;
OTAS = observational teamwork assessment of surgery.
ACAD EMERG MED • November 2008, Vol. 15, No. 11
•
1093
www.aemj.org
Table 2
Summary of Team Performance Metrics*
Method
Description
Advantages
Event-based
measurements
General method involving
behavioral checklist that
are linked to scenario events and
specific KSAs being trained
Explicitly links measurement
opportunities (i.e., scenario
events), acceptable behaviors,
and KSAs being trained
BARS
Provides brief description of
behaviors as anchors associated
with each particular rating
BOS
Generally uses a Likert-type scale
to rate the frequency of
occurrence in certain team
processes
Questionnaires administered to
individuals
Focuses observer’s attention on
predefined events
Reduces latitude of judgment by
the observer by focusing on
specific observable behaviors
Can be modified
Facilitates accurate ratings by
providing concrete examples of
the behaviors
Avoids potential problem of BARS
by focusing on typical
performance
Self-report
measures
Well-suited to capture affective
factors that influence team
performance
Disadvantages
Development of measures can be
time consuming compared to
other approaches; and
measurement tools must be
developed for each scenario
Observers may focus on specific,
isolated behaviors and miss the
big picture of how the team is
performing
Requires rates to estimate
frequencies, potentially
influencing final ratings by
recency and primacy of events
Does not capture dynamic
performance; translating
individual scores to team level
scores can be problematic
*Modified from Rosen et al.15
BOS = behavioral observation scales; BARS = behaviorally anchored rating scales; KSA = knowledge, skill, attitudes.
Table 3
Developing Specialty-specific Team Performance Metrics
Stepwise process
Example
Review literature for tools assessing specialty-specific behavioral skills
Identify core categories measured—communication, teamwork, leadership, and decision-making
Survey attitudes and perceptions of safety
Develop a taxonomy of specialty-specific teamwork behaviors
Analyzing adverse specialty-specific events
Conducting critical incident reviews
Develop a behavior-based rating system
Analyzing adverse specialty-specific events
Conducting critical incident reviews
Develop a behavior-based skills curriculum
specialty. Contextually relevant, behavior-based evaluation tools can help identify opportunities to improve
team performance and determine if such educational
interventions are successful.
How Well Do These Metrics Assess Team
Performance?
There is no criterion standard team performance metric
or set of metrics, just as there is no criterion standard
team training curriculum across the health care disciplines. However, since ‘‘what gets measured gets
done,’’15 instruments that accurately and reliably characterize team performance are critical to facilitating
excellence in teamwork behaviors that ultimately mitigates medical error and prevents adverse clinical
events.
How well metrics currently in use assess team performance depends on: 1) what is being measured and
under what circumstances—during real clinical events
Yule et al.45
Flin et al.44
Flin et al.44
Flin et al.44
Flin et al.46
or simulated scenarios; 2) instrument design and how
well the tool captures the behavior in question; 3) the
frequency of measurement; 4) the evaluator’s expertise
in team performance and their understanding and
familiarity with the instrument; and 5) the interpretation
of the information gathered and the feedback provided
to team members.
It is doubtful that one metric is sufficient to adequately capture the complex nature of team performance. The work of Morey et al.4 characterizes an
approach to assessing team performance via a panoply
of process and outcome measures, a virtual scatter gun
approach of measuring a variety of specific, mutually
agreeable events sequentially, over time, and tracking
the results to monitor change in behavior. They demonstrate improved team performance and a decrease in
clinical errors. The combined works of Yule and Flin et
al. (Table 3) also underscore the importance of a solid
methodology and sound psychometrics in developing
1094
Shapiro et al.
•
DEFINING TEAMS: METHODOLOGY, METRICS, AND OPPORTUNITIES
tools for assessing specialty-specific team performance.
Well-contrived metrics can provide a robust reflection
of teamwork behaviors and reveal opportunities to promote best practices for team performance that lead to
reduction in medical errors and safer patient care.
Recommendations for Using Team Performance
Metrics
The most common reason for evaluation is to determine
first the effectiveness or impact of an intervention and
next the ways in which the attribute or impact can be
improved.23 The accuracy of determining effectiveness
hinges on engaging a solid evaluation methodology and
the use of high-performing instruments designed to
precisely assess for the attribute sought.
The majority seeking to determine the impact of a
teamwork intervention are not individuals engaged in
research. Rather, they are individuals seeking to ensure
the communication of knowledge, skills, and actions
essential to high-functioning reliably safe organizations.47 To that end, there is a need for researcher and
nonresearcher to partner and develop high performing
measures, capable of assessing the attribute sought,
while imposing little or no undue burden upon a complex dynamic environment.
SBT FOR LEADERSHIP
Strong and effective team leadership is critical to team
performance as leaders assist in team development,
problem-solving, and providing motivation.24,48 Just as
teamwork consists of behaviors that can be taught,
reinforced, and measured, team leadership may also be
defined by such behaviors. Different situations require
different leadership strengths, but there are global leadership skills and abilities that are transferable. In the
medical setting, a good leader must not only be a good
clinician, but also have a strong sense of team dynamics
and understand the impact of personal interactions and
communications.
The impact of leaders on team performance has been
described by multiple sources. Marks et al. states49 that
leaders integrate pertinent environmental cues into
their existing cognitive structures to guide problemsolving. This then culminates into a course of action
that serves to provide direction toward a given
goal.49,50 Leadership behavior in teams may be subdivided into task-focused leadership, person-focused
leadership, and task interdependence. Person-focused
leadership addresses behavioral interactions, cognitive
structures, and attitudes required for effective team
interactions to facilitate successful task performance.
Task interdependence defines the degree that team
members depend on each other to perform a desired
task. As team interdependence increases, team member
performance must be coupled more intricately and
increases the complexity of leadership.50 In 2000, Burke
et al.50 conducted a meta-analysis, which suggested that
the use of task-focused behaviors is moderately related
to perceived team effectiveness and productivity.
Person-focused behaviors were also found to be associated with perceived team effectiveness, productivity,
and learning. Although the study results regarding
team interdependence and their contribution to effectiveness, productivity, and learning lacked sufficient
effect and sample sizes, the authors concluded that
when task interdependence is higher, team leadership
is ‘‘relatively’’ more important in achieving efficacious
team performance outcomes. These findings support a
SBT-based leadership curriculum in EM, because care
of acutely ill patients has a high level of interdependent
tasks.
During a resuscitation of a critically ill patient, the
importance of leadership is paramount. The ability to
maintain control of a resuscitation room through planning and preparation can be evaluated by looking for
behaviors such as goal specification, evidenced by the
leader’s act of summarizing the current understanding
of a given situation. SBT encourages all participants to
face decisions related to team processes and leadership
challenges.
SBT to Enhance Leadership Training
Any leadership curriculum must have clear teaching
goals with appropriate measurement tools to monitor
the progress of its trainees. SBT assists in learning and
assessment. The instructors must be clear in their vision
of what constitutes an effective leader. The organizational psychology literature provides guidance for
general best-practices in leadership,51 but real-world
leadership experience in the ED is necessary to translate this into SBT scenarios. As debriefing is the key to
any useful SBT experience, those leading the debriefing
must be able to do more than merely review a scoring
sheet of observed behaviors. They must also provide a
clear connection between the concepts raised during
the SBT, to the trainees’ actual practice. It is not
enough to point out to a resident that they failed to
‘‘control the room’’ during a resuscitation. One must
provide strategies, such as voice control, eye contact,
positioning in the room, and task delegation, that can
be employed to improve such control. This discussion
is more challenging if the instructor has not had applicable leadership training or significant experience managing resuscitations. SBT is a means for initiating
discussions and reflecting on performance, but debriefing activities related to leadership also need to be conducted after challenging clinical situations to reinforce
key concepts.
Creating Leadership Scenarios
How does one create simulation scenarios to address
leadership behaviors, permitting their measurement?
As SBT can be absorbing and overwhelming, scenarios
that are ‘‘clean,’’ i.e., do not provide multiple pathways
for different outcomes, are more easily controlled to
keep the teamwork and leadership issues at the forefront.
As noted, the SMARTER methodology provides a
framework for scenario development that focuses on
the desired KSAs in EM.52 Although not validated for
leadership training purposes, it is reasonable to assume
this system can be utilized for leadership-specific
scenarios. As leadership is often manifested by
decision-making, decisions the participants will face in
the simulation should be defined. Scenarios that force
ACAD EMERG MED • November 2008, Vol. 15, No. 11
•
www.aemj.org
participants to address critical decisions or actions
should be created. The scenarios should be crafted so
that the patient’s progression provides additional triggers that elicit desired participant reactions. In most
settings for leadership SBT, it is important not to make
the outcome of the scenario dependent on the participants’ making the ‘‘right’’ decisions. It is possible that
the team succeeds and the leader has fallen short of the
desired training goals. To avoid unnecessary confusion
in the learners, complex medical situations should only
be used where the appropriate treatment is apparent.
Immediately following the scenario, a teamwide
debriefing should be conducted in which the focus is
on the team leadership teaching points. The use of
straightforward clinical cases minimizes digressions
over the case’s appropriate management and boosts
participants’ confidence in their own clinical abilities,
permitting a less defensive, more open debriefing session. Measurement of the behaviors of interest requires
that they be outlined prior to the case’s creation. An
example observational checklist is provided in Figure 2.
The more clearly defined the behaviors are, the more
easily a measurement tool can be utilized. Tight control
of the scenario’s progression increases the odds of generating the desired behaviors.
Metrics for Leadership Assessment
Once leadership behaviors have been defined, one can
approach their metrics. This is the most challenging
aspect of leadership training, teamwork training, and
SBT in general. There are no universally applicable validated leadership measurement tools. Different settings
require different skill-sets. Different types of teams have
different goals, requiring distinctly unique metrics.
What is measurable may not yield useful information.
Examples of available measurement tools include the
anesthesia nontechnical skills,53 which allows for individual member assessment and therefore may be used
as a tool for leadership behavior assessment, compared
to the communication and teamwork skills42 that evaluates whole team performance. More recently, an EMspecific system has been proposed as an event-based
measurement approach. The SMARTER approach52
systematically links performance measures to ACGME
core competencies. Each measurement tool applies to
different arenas; none is universally applicable, especially in the evaluation of leadership skills. Measuring
behavior often involves quantifying subjectivity; hence
there is a need for metrics specific to a given situation
and team member that can also permit standardized
comparisons.
CONCLUSIONS
The health care community is interested in improving
teamwork. Yet, numerous well-intentioned programs
may fail to achieve the desired outcomes in team performance. We address the need for teamwork-focused
SBT in EM and detail a process for its development.
Teamwork is dynamic and complex, requiring an evaluation strategy that is dynamic as well. While no criterion standard metric or measure exists, there is a solid
methodology for developing tools designed to assess
1095
the attributes sought. More research is needed to
develop high-performing metrics. It is hoped that this
discussion will assist those in EM, and the larger health
care field, in the design and delivery of SBT for training
and evaluating teamwork.
References
1. Joint Commission. Preventing infant death and
injury during delivery. Sentinel Event Alert, Issue30.
Available at: http://www.jointcommission.org/Sentinel
Events/SentinelEventAlert/sea_30.htm. Accessed Aug
27, 2008.
2. Risser DT, Rice MM, Salisbury ML, Simon R,
Jay GD, Berns SD. The potential for improved
teamwork to reduce medical errors in the
emergency department. The MedTeams Reasearch
Consortium. Ann Emerg Med. 1999; 34:373–83.
3. Salas E, Wilson KA, Lazzara EH, et al. Simulationbased training for patient safety: 10 principles that
matter. J Pat Saf. 2008; 4:3–8.
4. Morey JC, Simon R, Jay GD, et al. Error reduction
and performance improvement in the emergency
department through formal teamwork training:
Evaluation results of the MedTeams project. Health
Serv Res. 2002; 37:1553–81.
5. Shapiro MJ, Morey JC, Small SD, et al. Simulation
based teamwork training for emergency department staff: does it improve clinical team performance when added to an existing didactic
teamwork curriculum? Qual Saf Health Care. 2004;
13:417–21.
6. Kobayashi L, Shapiro MJ, Gutman DC, Jay G.
Multiple Encounter Simulation Scenario (MESS) for
clinical skills and patient safety training of
personnel working in high acuity multi-patient
medical environments. Acad Emerg Med. 2007;
14:1141–8.
7. Salas E, Sims DE, Burke CS. Is there a ‘‘big five’’ in
teamwork? Small Group Res. 2005; 36:555–99.
8. Kneebone RL. Crossing the line: Simulation and
boundary areas. Simul Healthc. 2006; 1:160–3.
9. Reznek M, Smith-Coggins R, Howard S, et al.
Emergency medicine crisis resource management
(EMCRM): Pilot study of a simulation-based crisis
management course for emergency medicine. Acad
Emerg Med. 2003; 10:386–9.
10. McLaughlin SA, Doezema D, Sklar DP. Human simulation in emergency medicine training: a model
curriculum. Acad Emerg Med. 2002; 9:1310–8.
11. Bond WF, Deitrick LM, Arnold DC, et al. Using simulation to instruct emergency medicine residents in
cognitive forcing strategies. Acad Med. 2004;
79:438–46.
12. Fernandez R, Kozlowski SW, Shapiro MJ, Salas E.
Toward a definition of teamwork in emergency
medicine. Acad Emerg Med. 2008; 15:1104–12.
13. Leape LL, Berwick DM. Five years after To Err is
Human: what have we learned? JAMA. 2005;
293:2384–90.
14. Salas E, Cannon-Bowers JA. The science of training: a decade of progress. Annu Rev Psychol. 2001;
52:471–99.
1096
Shapiro et al.
•
DEFINING TEAMS: METHODOLOGY, METRICS, AND OPPORTUNITIES
15. Rosen MA, Salas E, Wilson KA, et al. Measuring
team performance in simulation-based training:
adopting best practices for healthcare. Simul
Healthc. 2008; 3:33–41.
16. Rosen MA, Salas E, Wu TS, et al. Promoting teamwork: an event-based approach to simulation-based
teamwork training for emergency medicine residents. Acad Emerg Med. 2008; 15:1190–98.
17. Awad S, Fagan S, Bellows C, et al. Bridging the
communication gap in the operating room with
medical team training. Am J Surg. 2005; 190:770–
4.
18. Young GJ, Charns MP, Daley J, Forbes MG,
Henderson W, Khuri SF. Best practices for managing surgical services: the role of coordination.
Health Care Manag Rev. 1997; 22:72–81.
19. Clemmer TP, Spuhler VJ, Berwick DM, Nolan TW.
Cooperation: the foundation of improvement. Ann
Intern Med. 1998; 128:1004–9.
20. Burke CS. Team task analysis. In: Stanton N,
Hendrick H, Konz S, Parsons K, Salas E (eds.). Handbook of Human Factors and Ergonomics Methods.
London, England: Taylor & Francis, 2005, pp 1–8.
21. Cannon-Bowers JA, Tannenbaum SI, Salas E, Volpe
CE. Defining team competencies and establishing
team training requirements. In: Guzzo R, Salas E, &
Associates (eds.). Team Effectiveness and Decision
Making in Organizations. San Francisco, CA:
Jossey-Bass, 1995, pp 333–80.
22. Salas E, Klein C, King H, et al. Debriefing medical
teams: 12 evidence-based best practices and tips.
Jt Comm J Qual Pat Saf. 2008; 34(9):518–27.
23. Kirkpatrick D. Evaluating Training Programs: The
Four Levels. 2nd ed. San Francisco: Berrett-Koehler
Publishers Inc., 1998.
24. Kendall D, Salas E. Measuring team performance:
review of current methods and consideration of
future needs. In: Ness JW, Tepe V, Ritzer D (eds.).
The Science and Simulation of Human Performance; Advances in Human Performance and Cognitive Engineering Research. Philadelphia, PA:
Elsevier, 2004, pp 307–26.
25. Bennet JB. The essence of morale. Mil Rev. 1942;
22:50–2.
26. Briggs GE, Naylor JC. Team versus individual training, training task fidelity and task organization
effects on transfer performed by three-man teams.
J Appl Psychol. 1965; 49:387–92.
27. Beaubien JM, Baker DP, Holzman MA. How military
research can improve team training effectiveness in
other high risk industries. In: Proceedings of the 45th
Annual Conference of the International Military
Testing Association. Pensacola, FL, Nov 3–5, 2003;
679–87. Available at: http://www.internationalmta.
org/2003/2003Proceedings/03IMTAproceedings.pdf.
Accessed May 5, 2005.
28. Brannick ME, Salas E, Prince C (eds.). Team Performance Assessment and Measurement: Theory,
Methods and Applications. Mahwah, NJ: Lawrence
Erlbaum Associates, 1997.
29. Cooper GE, White MD, Lauber JK. Resource management on the flightdeck: Proceedings of a
NASA ⁄ Industry Workshop. NASA Conference
30.
31.
32.
33.
34.
35.
36.
37.
38.
39.
40.
41.
Publication No. CP-2120. Moffett Field, CA: NASAAmes Research Center, 1980.
Lauber JK. Cockpit resource management: background and overview. In: Orlady HW, Foushee HC
(eds.). Cockpit Resource Management Training:
Proceedings of the NASA ⁄ MAC Workshop. NASA
Conference Publication No. 2455. Moffett Field, CA:
NASA-Ames Research Center, 1987.
Helmreich RL, Wilhelm JA. Outcomes of crew
resource management training. Int J Aviat Psychol.
1991; 1:287–300.
Helmreich RL, Merritt AC, Wilhelm JA. The evolution of crew resource management training in commercial aviation. Int J Aviat Psychol. 1999; 9:19–32.
Wiener EL, Kanki BG, Helmreich RL. Cockpit
Resource Management. San Diego, CA: Academic
Press, 1993.
Klinect JR, Murray P, Merritt A, Helmreich RL. Line
operations safety audit (LOSA): definition and operating characteristics. In: Proceedings of the 12th
International Symposium on Aviation Psychology.
Dayton, OH: Ohio State University, 2003, pp 663–8.
Available
at:
http://www.homepage.psy.utexas.
edu/HomePage/Group/HelmreichLAB/Publications/
pubfiles/Klinect.Operating.Characteristics.2003.pdf.
Accessed Jun 30, 2008.
Avermaete JAG van. NOTECHS: Non-Technical
Skill Evaluation in JAR-FCL; National Lucht-en
Ruimtevaartlaboratorium, National Aerospace Laboratory NLR-TP-98518, 1988. Available at: http://
www.siaa.asn.au/simtect/2005/NOTECHS%20report.
pdf. Accessed Jun 30, 2008.
Gaba DM, Fish KJ, Howard SK. Crisis Management
in Anesthesiology. New York, NY: Churchill Livingstone Inc, 1994, pp 5–45.
Flin R, Martin L. Behavioral Markers for Crew
Resource Management: A review of current practice. Int J Aviat Psychol. 2001; 11:95–118. Available
at:
http://www.informaworld.com/smpp/content~
content=a784772428~db=all~order=page. Accessed
Jun 30, 2008.
Healey AN, Undre S, Vincent CA. Developing
Observational Measures of Performance in Surgical
Teams. Qual Saf Health Care. 2004; 13:ii33–i40.
Available at: http://www.pubmedcentral.nih.gov/
picrender.fcgi?artid=1765793&blobtype=pdf. Accessed
Jun 30, 2008.
Thomas EJ, Sexton JB, Helmreich RL. Translating
teamwork behaviours from aviation to healthcare:
development of behavioural markers for neonatal
resuscitation. Qual Saf Health Care. 2004; 13:i57–
i64.
Flin R, Maran N. Identifying and training non-technical skills for teams in acute medicine. Qual Saf
Health Care. 2004; 13:i80–i84. Available at: http://
qshc.bmj.com/cgi/content/abstract/13/suppl_1/i80.
Accessed Jun 30, 2008.
Baker DP, Gustafson S, Beaubien J, Salas E, Barach
P. Literature Review: Medical Teamwork and
Patient Safety: The Evidence-based Relation. Publication No. 05-053, Agency for Healthcare Research
and Quality, Apr. 2005. Available at: http://www.
ahrq.gov/qual/medteam/. Accessed Jun 30, 2008.
ACAD EMERG MED • November 2008, Vol. 15, No. 11
•
www.aemj.org
42. Frankel A, Gardner R, Maynard L, Kelly A. Using
the Communication and Teamwork Skills (CATS)
Assessment to Measure Health Care Team Performance. Jt Comm J Qual Patient Saf. 2007; 33:549–
58.
43. Yule S, Flin R, Paterson-Brown S, Maran N.
Non-technical skills for surgeons in the operating
room: a review of the literature. Surgery. 2006;
139:140–9.
44. Flin R, Yule S, McKenzie L, Paterson-Brown S,
Maran N. Attitudes to teamwork and safety in the
operating room. Surgery. 2006; 4:145–51.
45. Yule S, Flin R, Paterson-Brown S, Maran N, Rowley
D. Development of a rating system for surgeons’
non-technical skills. Med Educ. 2006; 40:1098–104.
46. Flin R, Yule S, Paterson-Brown S, Maran N, Rowley
D, Youngson G. Teaching surgeons about nontechnical skills. Surgery. 2007; 5:86–9.
47. Institute of Medicine. To Err Is Human: Building a
Safer Health System. Washington, DC: National
Academy Press, 1999.
48. Kozlowski SW, Gully SM, Nason ER, Smith EM.
Developing adaptive teams: a theory of compilation
and performance across levels and time. In: Ilgen
DR, Pulakos ED (eds.). The Changing Nature of
49.
50.
51.
52.
53.
1097
Work and Performance. San Francisco, CA: JosseyBass, 1999.
Marks MA, Zaccaro SJ, Mathieu JE. Performance
implications of leader briefings and team interaction
training for adaptation to novel environments. J
Appl Psychol. 2000; 85:971–86.
Burke SC, Stagl KC, Klein C, Goodwin GF, Salas E,
Halpin SM. What type of leadership behaviors are
functional in teams? A meta-analysis. Leadership Q.
2000; 17:288–307.
Stagl KC, Salas E, Burk CS. Best practices in team
leadership: what team leaders do to facilitate team
effectiveness. In: Conger J, Riggo RE (eds.). The
Practice of Leadership: Developing the Next Generation of Leaders. San Francisco, CA: Jossey and
Bass, 2007, pp 172–97.
Rosen MA, Salas E, Silvestri S, Wu T, Lazzara EH.
A measurement tool for simulation-based training
in emergency medicine: the Simulation Module for
Assessment of Resident Targeted Event Responses
(SMARTER) approach. Simul Healthc. 2008; in
press.
Fletcher G, Flin R, McGeorge P, et al. Anaesthetists’
non-technical skills (ANTS): evaluation of a behavioural marker system. Br J Anaesth. 2003; 90:580–8.