Individual Performance in Airline Simulations
Artyom Ivakh & Amber Fritsch, Master’s Candidates in I/O Psychology
Faculty Advisors: Dr. Glenn Littlepage, Dr. Michael Hein, and Dr. Richard Moffett III
Middle Tennessee State University
NASA Flight Operation Center-Unified Simulation Lab
Introduction
• This research was conducted at the Flight Operations
Center Simulation (FOCUS) Lab where aerospace
students from various disciplines (flight dispatch,
technology, maintenance/management, professional
pilot, and airport administration) participate in three
high fidelity flight simulations that are each followed
by team debriefings (i.e., After Action Reviews AARs).
• Students are asked to treat the simulations as if it was
their real job and their goal is to reach high levels of
safety, efficiency, teamwork, and individual task
performance while bringing profit to an airline.
• Within the FOCUS Lab, there are nine
individual roles that represent the main functions
of today’s dispatching teams at flight operations
centers. The positions include: Flight Operations
Coordinator, Crew Scheduling, Maintenance Control,
Weather Operations, Flight Scheduling, Flight
Planning, Ramp Tower Coordinator, Pseudo Pilot, and
Pilot.
• Because there was no measure of performance that
emphasized individual level contributions (taskwork)
at the FOCUS lab, such measures were developed and
validated for purposes of this study. We validated the
measures by providing evidence of their reliability, as
well as other substantive findings that would provide
evidence for their construct validity.
• Because the essence of individual (taskwork) is
central for teams to be effective in a complex
environment (Krokos, Backer, Alonso, & Day, 2009),
we hypothesize that individual performance measures
are positively related to teamwork as well as team
performance.
• Little research exists on transition phase process
(processes that take place before and/or after a
mission/simulation - AAR), so we investigated how
individual performance relates to team debriefings
after the FOCUS Lab via a research question (Marks,
Mathieu, & Zaccaro, 2001).
Hypothesis 1: The average score for individual
performance will be positively related to teamwork
performance across all flight simulations. (Krokos,
Backer, Alonso, & Day, 2009)
Hypothesis 2: The average score for individual
performance will be positively related to the team
performance across all flight simulations.
Research Question 3: The average score for individual
performance on a prior flight simulations will be
positively
related
to
the
following
team’s
transition
performance
across
all
flight simulations.
Discussion
Measures Comprised
Individual Performance Measures: A measure that assessed the frequency of specific job relevant behaviors expected from each position (Behavioral Observation
Scales). It was developed for all nine position. Following each flight simulation, individual performance measures for each position, were filled out by two or three
members of the FOCUS Lab staff who were assigned to observe their specific position based upon their unique areas of expertise.
Teamwork Performance Measures:
1) Action Phase Teamwork Performance: A self-report measure used to assess teamwork performance during the simulation. This measure contains four subdimensions: monitoring progress toward goals, systems monitoring, team monitoring and backup responses, and coordination activities (Marks, Mathieu, & Zaccaro,
2001). Following each flight simulation, participants completed the measure via an electronic survey.
2) Interpersonal Teamwork Performance: A self-report measure used to assess participants’ perception of their team’s interpersonal processes. This measure contains
four sub-dimensions: conflict management, motivating and confidence building, and affect management (Marks, Mathieu, & Zaccaro, 2001). Following each flight
simulation, participants completed the measure via an electronic survey.
3) Teamwork Grading Rubrics: A measure used to evaluate teams’ level of teamwork during a simulation. This measures contains three sub-dimensions: problem
solving, information utilization, and coordination. Following each simulation, FOCUS Lab staff who observed the flight simulation completed this measure.
Team Performance Measures:
1) Financial Revenue: A measure of quantified revenue generated teams after each flight simulation. The revenue is mostly influenced by the total amount of time loss
due to flight delays, and each minute of delay can be translated into a dollar amount. Following each simulation, team performance metrics were collected and
recorded.
2) Trigger Response Effectiveness: The trigger response effectiveness measure targets a specific situation that was defined to have a critical significance to team
outcomes by the staff of the lab. After issue has been identified, lab’s faculty discuss what would be the optimal response to this situation, and compared it to the
response exhibited by the team. Following this discussion, which occurs after each flight simulation has ended, the trigger response effectiveness measure is filled out
by the FOCUS lab observers that were present during the flight simulation
Transition Performance Measure: A self-report and observer measure that is used to evaluate the quality of processes that occurred during the After Action Review.
This measure contains three main sub-dimensions: mission analysis, goal specification, strategy formulation, as well as learning occurred (Marks, Mathieu, &
Zaccaro, 2001). Following each flight simulation, student-participants as well as the facilitator of the After Action Review completed this measure.
Research Design:
SIM1 ->AAR1
SIM2->AAR2
SIM3->AAR3
Methods
Aerospace students enrolled in AERO 4040 were assigned into six teams with nine or ten students in each team. Each of the six teams participated in three flight
simulations throughout the semester. Six to ten members of the FOCUS Lab staff observed each simulation to rate teamwork as well as individual performance.
Each one of the three simulations was followed by an After Action Review (AAR), which is a facilitated discussion of positive and negative events and
opportunities for improvement. Data in this study will be analyzed at the team level. To test Hypothesis 1 & 2, and the Research Question, average scores for all
nine individual performance measures during flight simulation one, two, and three were computed and then correlated to the subsequent teamwork, team, and
transition performance measures in their respective flight simulations and then averaged across all three flight simulations (illustration below).
r1
r2
r3
SIM1 Ave
Individual
Performance
Measures
SIM1 Team Performance
SIM1 Teamwork Performance
AAR1 Transition Performance
H1 SIM2 Ave
H2 Individual
RQ1 Performance
Measures
SIM3 Team Performance
H1 SIM3 Ave
SIM3 Teamwork Performance
H2 Individual
AAR3 Transition Performance
RQ1 Performance
Measures
Results for Hypothesis 1&2, as well as Research Question were computed by averaging r1 , r2 , and r3 values for each performance measure utilizing r to z transformations.
H1
H2
RQ1
SIM2 Team Performance
SIM2 Teamwork Performance
AAR2Transition Performance
Results
Hypothesis 1
r = .587*
Average Individual
Performance Measures
r = .357*
r = .731*
Action Phase
Teamwork
Improvement
Interpersonal
Teamwork
Improvement
Hypothesis 2
*Results were not
significant at the
p<.05 level; df = 5.
r = .671*
Financial Revenue
r = .650*
Trigger Response
Effectiveness
Average Individual
Performance Measures
Teamwork Grading
Rubric
• Findings of this study suggest that individual
performance measures are valid and are related with
teamwork, team, and transition performance as it was
expected.
• Because there were up to three raters that assessed
performance of individual roles at the NASA FOCUS
Lab, inter-rater reliability of the individual performance
measures were assessed by providing correlations
between raters for each positions. It was found that interrater reliability was very high for some positions (Crew
Scheduling r = .733, Maintenance Operations r = .798,
Flight Operations Coordinator r = .640, and Weather
Operation r = .624) modest for other positions (Pseudo
Pilot r = .433, Ramp Tower Operations r = .472, and
Pseudo Pilot r = .433), and very low for the flight
planning position (r = -.117). On average inter-rater
reliability estimates for the individual performance
measure were r = .492, which is representative of the fact
that for the most part individual performance measures
are reliable, and that for most positions there was
moderate to strong agreement among raters.
• The teamwork patterns discussed in this study are
essential in today’s dynamic environment, especially the
in highly interdependent work-teams that exist in today's
aerospace, fire-fighting, medical, and military units, and
many other industries and/or domains. Findings of this
research can contribute to other of those that examined
effectiveness of team debriefings (i.e., transition
performance) and shown its positive relationship with
team/individual performance effectiveness (Tannenbaum
& Cerasoli, 2012).
Limitations
• Because there are a limited number of people who
possess skills that are to run NASA FOCUS Lab, we
could only recruit a restricted number of participants for
this study. Due to this, results of the correlations
examined in this study are based on a small number of
teams (six teams total; df = 5), and since all correlations
are analyzed at a team level, they may provide weak
statistical power while answering most hypotheses and
research questions.
•We would like to note that this is only an exploratory
study, and we would suggest that results of its
correlational findings are to be interpreted in terms of the
magnitude of their r values, opposed to paying attention
to significance (which was not reported).
• Finally, it is important to interpret the findings of this
study with caution. One should only try seeking an
overall understanding of a correlations pattern (i.e.,
associations), instead of making causal judgments in
answering the following hypotheses.
.
Related Work
1.
2.
3.
Research Question
4.
Average Individual
Performance Measures
r =.672*
Student Ratings of Transition
Performance
Average Individual
Performance Measures
r =.484 *
Facilitator Ratings of
Transition Performance
5.
Humphrey, S. E., Morgeson, F. P., & Mannor, M. J. (2009). Developing a theory of the
strategic core of teams: A role composition of team performance. Journal of Applied
Psychology, 94, 48-61.
Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and
taxonomy of team processes. Academy of Management Review, 26, 356-376.
Marks, M. A., DeChurch, L. A., Mathieu, J. E., Panzer, F.J., & Alonso, A. (2005).
Teamwork in multiteam systems. Journal of Applied Psychology, 90, 964-971
Smith-Jentsch, K. A., Johnston, J. H. & Payne, S. C. (1998). Measuring team-related
expertise in complex environments. In J. A. Cannon-Bowers & E. Salas (Eds.), Making
decisions under stress: Implications for individual and team training. (pp. 61-87)
Washington, DC APA Press.
Tannenbaum, S. I., & Cerasoli, C.P. (2012). Do team and individual debriefs enhance
performance? A meta-analysis. Human Factors: The Journal of the Human Factors and
Ergonomics Society, 1-15.