Evaluating self-analysis as a strategy for learning crew resource management (CRM) in undergraduate
flight training
by Guy Mario Smith
A thesis submitted in partial fulfillment of the requirements for the degree of Doctor of Education
Montana State University
© Copyright by Guy Mario Smith (1994)
Abstract:
Crew Resource Management (CRM) is training in crew coordination and teamwork. CRM was added
to airline programs in the late 1970’s because it became evident that human performance was a
significant contributor to aviation accidents. In pilot preparatory programs, emphasis is usually placed
on individual performance and crew skills are not addressed until airline training. College aviation
programs are in a unique position to include CRM training in the curriculum to meet industry demands
for pilots with a high degree of competence in interpersonal skills. CRM training is usually a student’s
first exposure to crew operations, requiring the college to modify airline training procedures to create
meaningful learning for. inexperienced pilots. Modern simulators have transformed airline training
because full-mission simulation or Line-Oriented Flight Training (LOFT) allows airline training
departments to reproduce the line, or operational, environment in the simulator. Research with airline
pilots has found that LOFT was most effective for teaching CRM and that LOFT was best when airline
crews debriefed themselves using self-analysis to evaluate their CRM performance. The study sought
to determine if undergraduate flight students could effectively learn CRM skills by using self-analysis
of LOFT as a debriefing strategy, despite their inexperience with crew operations. Eight men and two
women, paired into five flight crews, completed CRM and LOFT training. Self-analysis was randomly
inserted into their training using an alternating treatments research design. Crew effectiveness was
assessed by measurements of crew attitudes, observations by trained observers, crew reflections on
their performance, and communications analysis. It was found that at least one self-analysis session
was effective for each crew and overall gains were noted for two of the five crews. Self-analysis was
effective when crews had the prerequisite technical skills and was ineffective if technical skills were
lacking or if the scenario was too complex. Results suggest that self-analysis should not be applied
universally in undergraduate flight training, but it is a valuable supplementary strategy to focus
attention on personalities, roles, team dynamics, or specific CRM skills. EVALUATING SELF-ANALYSIS AS A STRATEGY FOR LEARNING
CREW RESOURCE MANAGEMENT (CRM) IN
UNDERGRADUATE FLIGHT TRAINING
by
Guy Mario Smith
i>"
A thesis submitted in partial fulfillment
of the requirements for the degree
of
. Doctor of Education
MONTANA STATE UNIVERSITY
Bozeman, Montana
April, 1994
© COPYRIGHT
by
Guy Mario Smith
1994
All Rights Reserved
APPROVAL
of a thesis submitted by
Guy Mario Smith
This thesis has been read by each member of the thesis committee and
has been found to be satisfactory regarding content, English usage, format,
citations, bibliographic style, and consistency, and is ready for submission to the
College of Graduate Studies.
Dater
Approval for the Major Department
-LO
Date
/9 ? '-/
Head, Major Department
Approved for the College of Graduate Studies
Date
'
7
Graduate Dean
iii
STATEMENT OF PERMISSION TO USE
In presenting this thesis in partial fulfillment of the requirements for a
doctoral degree at Montana State University, I agree that the Library shall make it
available to borrowers under rules of the Library. I further agree that copying of
this thesis is allowable only for scholarly purposes, consistent with "fair use" as
prescribed in the U.S. Copyright Law. Requests for extensive copying or
reproduction of this thesis should be referred to University Microfilms
International, 300 North Zeeb Road, Ann Arbor, Michigan 48106, to whom I have
granted "the exclusive right to reproduce and distribute my dissertation for sale in
and from microform or electronic format, along with the right to reproduce and
distribute my abstract in any format in whole or in part."
Signature
Date
ACKNOWLEDGEMENTS
The author is grateful for the leadership of Dr. John W. Kohl, committee
chairman, and for the support and encouragement of committee members
Dr. Robert A. Fellenz, Dr. Gary J. Conti, Dr. Leroy J. Casagranda, and
Dr. Jana R. Noel.
The work was funded by the 1992-93 (GRAD VII) Graduate Research
Award Program sponsored by the Federal Aviation Administration and
administered by the Transportation Research Board (TRB). Line-Oriented
Flight Training (LOFT) was conducted in Billings, Montana at the Rocky
Mountain College Flight Simulator Laboratory.
A special note of affectionate gratitude to MaryJo O. Smith for her
research assistance and devoted confidence.
V
TABLE OF CONTENTS
Page
LIST OF T A B L E S ..................................................................................................
ix
A B STR A C T ........................................................................................................... ..
x
1. IN T R O D U C T IO N ...........................................................................................
I
Purpose Statement ....................................................................................
Significance of the S tu d y .............................................................................
Research Questions ...................................................................................
General Procedures ....................................................................................
D elim itations................................................................................................
Definition of Terms ....................................................................................
4
4
7
7
8
10
2. LITERATURE REVIEW .............................................................................
12
Crew Resource Management (C R M ).......................................................
CRM S k ills .................................
CRM Learning S trategies.............................................................. .. .
Line-Oriented Flight Training (L O F T )....................................
Behavior Modification Model ...................................................................
Discovery-Learning Model ........................................................................
Theoretical or Conceptual Framework ...........................................
Learning Strategy - Cooperative L e a rn in g ......................................
Practice Strategy - L O F T .............................................
Debriefing Strategy - Self-Analysis ..................................................
Evaluating E ffectiveness.................................
A ttitu d e s................................................................................................
Systematic Observation of Crew B e h a v io r......................................
Survey Data on Crew R e a c tio n s.....................
Communications A nalysis...................................................................
CRM and LOFT in Undergraduate Flight Training .............................
12
13
18
18
21
23
24
27
27
29
30
32
34
36
37
39
vi
TABLE OF CONTENTS-Continued
. Page
3. METHODOLOGY .........................................................................................
42
Research O bjectives..........•.......... ..............................................................
Research Design ...........
Single-Subject D e sig n s........................................................................
A-B-A-B and Multiple Baseline D e s ig n s ............ ............................
Alternating Treatments D e sig n ................................................
Analytical T echniques..................................................................................
Population and Site Description ...............................................................
Research Subjects ...............................................................................
Research S e ttin g ..................................................................................
Simulator P rocedures...........................................................................
The LOFT S cen ario s...........................................................................
Procedures for Training and Evaluation .........................................
Individual Orientation to the Simulator ............................
Evaluation of Flying Skills and Assignment to C rew s............
Learning Procedures: Cooperative Learning ........................
Practice Procedures: LOFT .....................................................
Debriefing Procedures: Conventional ...................
Debriefing Procedures: Self-Analysis . . . ................................
D ata Collection Methods .....................................................................
The Cockpit Management Attitudes Questionnaire (CMAQ) . . .
Validity of Attitude M easurem ents...........................................
Reliability of Attitude M easurem ents......................................
Validity and Reliability of the CMAQ as a Measure
of Performance .............................................................. .. .
CMAQ M ethodology...................................................................
Individual Background Data .....................................................
The LINE/LOS Checklist ........................................
Validity of the LINE/LOS Checklist ............................
Reliability of the LINE/LOS C hecklist....................................
LINE/LOS Checklist Methodology .........................................
The CRM Survey...........................................................
Validity of the CRM Survey .....................................................
Reliability of the CRM S urvey.......................
Methodology of the CRM S urvey........................................
42
42
43
44
44
45
47
48
50
51
51
53
53
54
54
54
55
55
56
56
57
59
59
60
60
61
61
62
63
63
64
65
65
vii
TABLE OF CONTENTS-Continued
Page
Communications A nalysis...................................................................
Validity of Communications A n a ly sis.......................................
Reliability of Communications Analysis ..................................
Communications Analysis Methodology ..................................
Qualitative Data: Lessons-Learned ................................................
Supplementary or Supportive D a t a ................................................ .
4. ANALYSIS OF DATA ..............................................................
A le x /A r t................................................
Betty/Bob .....................................................................................................
C arl/C ath y ..........................
D a n /D a v e .....................................................................................................
E d /E r ic ....... ..................................................................................................
S u m m ary ...................................
Summary of F in d in g s...................
Summary of Lessons-Learned............................................................
66
67
68
68
69
70
71
71
77
82
87
92
96
96
98
5. CONCLUSIONS AND RECOM M ENDATIONS....................................... 101
Conclusions ..................................................................................................... 10.1
The Cockpit Management Attitudes Questionnaire (CMAQ) . . . 101
The LINE/LOS ChecMist .................................
102
The CRM Survey...............................
103
Communications A nalysis....................................................
104
CRM and Technical SMlls ................................................................. 105
Women Pilots ...................................................................................... 105
Scenarios .................................................................................................106
Role Definition and Crew D ynam ics................................................ 107
Self-Analysis Skills ..................................................
108
Communications Analysis and T ran scrip ts..........................................HO
Summary of Conclusions ................................................................................ I l l
V lll
TABLE OF CONTCNTS-Continued
Page
Recommendations ............................................................ ; ....................... 113
Improving Self-Analysis ...................................................................... 113
Metacognitive L e a rn in g .........................................................................114
Guided Self-Analysis ........................................................................... 114
Role M o d e ls ............................................................................................ 115
For Further Research ................................................ i ..................... 116
REFERENCES CITED ....................................................................................... 119
A P P E N D IC E S ........................
128
Appendix A--The Cockpit Management Attitudes
Questionnaire (CMAQ) -...................................................................... 129
Appendix B-Background Q uestionnaire................................................... 132
Appendix C--The LINE/LOS C h e c k list......................................................137
Appendix D--The CRM S u rv e y ................................................................. 149
Appendix E-Communications Analysis ................................................... 153
Appendix F -L essons-L earned..................................................
155
LIST OF TABLES
Table
Page
1.
Group Process Communication Research . ..............................................
38
2.
Summary of Students’ Background and S k ills .........................................
49
3.
Alex/Art Evaluation.R e s u lts ......................................................................
73
4.
Betty/Bob Evaluation Results . ...................... ..........................................
79
5.
Carl/Cathy Evaluation Results ............................ : ................................
83
6.
Dan/Dave Evaluation R e su lts...................................................................
88
7.
E d/E ric Evaluation R esults........................................................................
93
8.
Summary: Effectiveness of Self-Analysis............ ....................................
97
9.
Summary of Lessons-Learned .............. .. . . . .................................... .. .
99
X
ABSTRACT
Crew Resource Management (CRM) is training in crew coordination and
teamwork. CRM was added to airline programs in the late 1970’s because it
became evident that human performance was a significant contributor to
aviation accidents. In pilot preparatory programs, emphasis is usually placed on
individual performance and crew skills are not addressed until airline training.
College aviation programs are in a unique position to include CRM training in
the curriculum to meet industry demands for pilots with a high degree of
competence in interpersonal skills. CRM training is usually a student’s first
exposure to crew operations, requiring the college to modify airline training
procedures to create meaningful learning for. inexperienced pilots. Modern
simulators have transformed airline training because full-mission simulation or
Line-Oriented Flight Training (LOFT) allows airline training departments to
^reproduce the line, or operational, environment in the simulator. Research with
airline pilots has found that LOFT was most effective for teaching CRM and
that LOFT was best when airline crews debriefed themselves using self-analysis
to evaluate their CRM performance. The study sought to determine if
undergraduate flight students could effectively learn CRM skills by using selfanalysis of LOFT as a debriefing strategy, despite their inexperience with crew
operations. Eight men and two women, paired into five flight crews, completed
CRM and LOFT training. Self-analysis was randomly inserted into their
training using an alternating treatments research design. Crew effectiveness was
assessed by measurements of crew attitudes, observations by trained observers,
crew reflections on their performance, and communications analysis. It was
found that at least one self-analysis session was effective for each crew and
overall gains were noted for two of the five crews. Self-analysis was effective
when crews had the prerequisite technical skills and was ineffective if technical
skills were lacking or if the scenario was too complex. Results suggest that selfanalysis should not be applied universally in undergraduate flight training, but it
is a valuable supplementary strategy to focus attention on personalities, roles,
team dynamics, or specific CRM skills.
-//
/
CHAPTER I
INTRODUCTION
Modern aircraft, loaded with computers and automation, demand highly
trained pilots with finely tuned motor skills, the ability to follow involved
procedures, and an extensive information base. As technology places additional
demands on pilot skills and airmanship, the milieu of cognitive, behavioral, social,
and organizational psychology requires more pilot education in interpersonal and
teamwork skills. Statistics on the causes of accidents from 1959 to 1989 indicate
that flight crew actions were causal in more than 70% of worldwide accidents in the
public transport sector (Helmreich & Foushee, 1993). Recognition that human
performance is as vital to aircraft operations as technical proficiency caused airlines
to regard team skills as a vital part of the pilot selection and training process. [rThe
X
ideal airline candidate is a technical expert and a master of teamwork. For most of
this century, however, pilot selection and training was based on technical
proficiency aloney Airlines recognized this deficiency and have made substantial
investments into human factors research, resulting in the development of advanced
aircrew training programs such as Crew Resource Management (CRM).
It is argued that CRM is advanced training, not appropriate for beginning
students who should concentrate on "stick and rudder" skills. Others contend that
k
2
teamwork is an indispensable pilot skill and that it is a disservice to students to
postpone crew training until they reach the airlines (Trollip & Jensen, 1991).
_/
European "ab initio" programs, where nonpilots are taught from the beginning to be
airline pilots, have successfully included CRM in initial flight training for years
(Nash, 1992). College aviation programs are in a unique position to develop
effective CRM training for initial flight students.
In 1993, the Federal Aviation Administration (FAA) published three
guidelines for an effective CRM program, for airlines operating under Federal
Aviation Regulations (FAR) Parts 121 and 135:
1. The course content should emphasize the CRM skills.
2. Students should experience and practice these skills.
3.
Students should get feedback on their CRM performance.
To apply these guidelines to undefgraduate flight students, a "content" model,
concerned with transmitting information and skills, was insufficient. An
experiential or "process" model, concerned with providing resources to help
learners acquire CRM skills, was required. Moreover, to evaluate the outcomes of
this model, the primary effectiveness measure had to be "performance." There are
numerous CRM instructional methods to choose from. Of the 16 listed in Sams "
(1987), the most effective for airline pilots was Line-Oriented Flight Training
(LOFT), an experiential-learning method in which flight crews fly a complete
scenario in a high fidelity simulator in real tim ej Airlines have achieved striking
results with LOFT, but systematic research is necessary to ensure that LOFT is
I
3
equally effective for teaching CRM to undergraduate students. For most
undergraduates, LOFT training is their first exposure to nonroutine, high-stress,
high-workload, and emergency situations which require teamwork.
Self-analysis of LOFT, in which the debriefing is lead by the crew
^
themselves, has been a highly effective technique for improving CRM performance
in airline pilots (Butler, 1993). Self-analysis is a discovery-learning strategy based
on the Theory of Objective Self-Awareness (Duval & Wicklund, 1972), which
proposes that self-focusing stimuli often force objective appraisals of oneself that
may lead to attitude and behavior changes. For college crews, self-analysis could
give powerful insights into their CRM performance, offsetting some of their
inexperience (Duval, Duval, & Mulilis, 1992).
To expand technical skills into higher-order CRM skills, students must be
actively involved in each stage of the learning process (Bonwell & Eison, 1991).
Active-learning theory implied that learning CRM in a classroom was not
sufficient; students had to gain practical experience with the CRM skills in a crew
setting. Active-learning strategies, especially LOFT and self-analysis, were
employed in a college CRM training program which progressed through three
distinct phases: learning sessions where CRM skills were introduced, practice
sessions where CRM skills were exercised, and feedback sessions where behaviors
were reinforced or corrected.
4
Purpose Statement
The purpose of this study was to determine if undergraduate flight students
could learn Crew Resource Management (CRM) through self-analysis of LineOriented Flight Training (LOFT) despite their inexperience with crew operations.
Significance of the Study
This study concerns CRM and LOFT training at the undergraduate level
with flight students who have no experience with crew procedures. It cannot be
assumed that any of the results gathered from airline pilots are applicable to
college flight students because of the vast differences in their technical and crew
experiences. When designing CRM training for undergraduate flight students,
many of the issues already explored by airline researchers should be re-examined.
This study was limited in scope to only one facet of teaching strategies: Could
undergraduate flight students learn CRM through self-analysis of LOFT despite
their inexperience with crew operations?
The philosophy underlying CRM training is that heightened awareness of
CRM skills will produce tangible behavior change in cockpit performance. The fact
is that these principles and techniques are being "preached as the gospel" without
much research dealing with the evaluation or relative efficacy of such programs
(Foushee & Helmreich, 1988). Beyond the goal of developing awareness of
teamwork skills, the bottom line is that continued research is necessary to ensure
5
that CRM and LOFT training are applicable to aviation safety (Lofaro, 1990).
Awareness of CRM skills can be taught in the classroom and knowledge can be
assessed. But, if the goal is to teach students how to use CRM skills and to evaluate
their effectiveness, students must actually fly. LOFT allows undergraduate flight
)(_
students to experience the crew environment many years before envisioned.
Validation of the effectiveness of LOFT in crew coordination and resource
management requires proof that it produces the desired outcomes: increased
teamwork, reduced pilot errors, and ultimately enhanced safety. "The critical
research task becomes the determination of the relative effectiveness of different
instructional techniques and the development of more effective ones" (Foushee &
Helmreich, 1988, p. 224). There is already evidence that personality and attitudes
are linked to pilot performance, including crew management (Cook, 1991).
Whereas personality traits are considered stabilized and only marginally affected by
training, crew attitudes are less stable and are responsive to change through
training and experiences. Therefore, if CRM training and especially LOFT are to
be effective in changing pilot’s attitudes, they must have a positive effect on actual
cockpit management and flight performance. Group investigations generally
concentrate on attitudes, on output, or on performance factors. They commonly
ignore the process variables linking the training program to the outcomes (Foushee
& Helmreich, 1988).
Measuring attitudes is only an indirect assessment of effectiveness. "The
effect of crew attitudes on process variables and line performance remain issues in
6
need of further investigation" (Gregorich, Helmreich, & Wilhelm, 1990). An
important question is whether changes in attitude as a result of CRM training
actually result in increased crew effectiveness. Irwin (1991) notes that it is difficult
to examine this link in air transport environments because organizations rarely
permit the acquisition of the types of data needed for investigation. However, with
college flight students, there are fewer apprehensions that performance information
could threaten their status as airline pilots (Foushee & Helmreich, 1988). The
issue of assessment may be addressed in this environment where it is acceptable to
gather attitude and behavioral data at an individual level.
Airlines struggle with the problem of CRM failures, those crew members
who are unable or unwilling to adopt CRM concepts. "What do individual crew
members and organizations do when confronted with the individual who represents
a continuing threat to safety because of poor resource management practices?"
(Helmreich & Wilhelm, 1989). CRM training and evaluation at the undergraduate
level could help to revise the career choices of untrainable students and provide an
input into their career selection process. Once linkages between attitudes,
effectiveness, and performance are better understood, selection of students for airline
training can be based on superior rankings in these areas. Traditional approaches
to psychological screening have focused on "selecting out" those few individuals who
showed signs of psychopathology (Gregorich, Helmreich, Wilhelm, & Chidester,
1989). Equivalent measures could be taken towards "selecting in" individuals who
demonstrate superior performance in undergraduate CRM and LOFT training.
7
Research Questions
1. Does self-analysis of LOFT change undergraduate flight students’
attitudes as measured by the Cockpit Management Attitudes Questionnaire (CMAQ) ?
2. Does self-analysis of LOFT change undergraduate flight crews’
effectiveness as measured by the LINE/LOS Checklist?
3. Does self-analysis of LOFT change undergraduate flight crews’ reactions
to CRM training as measured by the CRM Survey?
4. Does self-analysis of LOFT change undergraduate flight crews’
performance of CRM skills as measured by Communications Analysis?
5. Are there factors or circumstances that strengthen or inhibit the learning
of CRM skills by self-analysis?
General Procedures
The research was essentially five single-subject experiments, each conducted
with one flight crew composed of two undergraduate students. The performance of
each crew was analyzed independently, similar to case study research. Due to the
small number of students involved, comparisons between crews was only
speculative and non-inferential.
Each crew received five lessons in CRM with each lesson focusing on a
specific CRM skill. CRM lessons were taught using active learning strategies which
required students to be actively involved in each stage of the learning process
8
(Bonwell & Eison, 1991). Each lesson progressed through three distinct phases: a
learning session where the CRM skill was introduced, a practice session where the
CRM skill was exercised in the LOFT simulator, and a debriefing session where
specific behaviors were reinforced or corrected. Learning and practice sessions
were the same throughout the five lessons, but two distinctly different debriefing
methods were employed; When conventional or traditional debriefing was used,
the instructor provided the majority of the input while the students were basically
passive recipients (Butler, 1993). When self-analysis debriefing was used, the crew
themselves conducted the debriefing, and the instructor monitored the discovery­
learning process (Butler, 1993). The crews were alternately exposed to
conventional debriefing and self-analysis debriefing; multiple measurements were
made after each lesson to determine if changes in effectiveness could be attributed
to self-analysis debriefing.
Delimitations
Due to the size of the population, generalizability to any other population
cannot be made. In this aspect, the study has the characteristics of a case study in
that it attempts to describe outcomes in depth. According to Guba (1978),
generalizability in a case study is left up to the reader who analyzes the findings of
the reported case and generalizes to his own context, if appropriate.
Guba and Lincoln (1981) note a limitation of case studies that also pertains
to single-subject designs: "Case studies can oversimplify or exaggerate a situation,
9
leading the reader to erroneous conclusions about the actual state of affairs"
(p. 377). In this study, no global conclusions about the overall effectiveness of selfanalysis can be drawn.
Because the number of participants was small, the quantitative data from
the CMAQ, the LINE/LOS Checklist, the CRM Survey, and the Communications
Analysis could not produce statistically significant results. Qualitative data from
open-ended questions asking students what they had learned during CRM sessions
were used to reinforce or diminish the findings.
Cockpits may have more diverse methods for communication than many
other arenas. Due to the nature of information present and to the specialized
training of pilots, crews can significantly interact in ways other than verbal such as
gestures, pointing, body language, and adjusting instruments. Due to the nature of
the data collection devices (videotapes and audio backup tapes), communications
analysis ignored all non-verbal crew communications. However, significant
non-verbal communications were noted in the instructor field notes.
A Frasca 142 two-place flight simulator was used for student LOFT training
to emulate a typical two-place multi-engine aircraft. A limitation of the Frasea 142
simulator is that it does not provide displays and controls on the right side of the
cockpit, requiring the co-pilot to share instruments with the pilot and occasionally
to reach in front of the pilot to manipulate controls, In a typical aircraft of this
type, the co-pilot would have some displays and controls on the right side of the
cockpit, and the pilots would be able to swap flying duties. This limitation might
11 >;
10
have affected decision making because some of the normal crew options were not
available.
Definition of Terms
A b initio Pilot Training: Pilot training programs that teach airline procedures from
the beginning of training (Trollip & Jensen, 1991, pp. 9-15).
Active Learning: Instructional strategies that get students involved in doing things
and thinking about what they are doing. To be actively involved, students
must engage in higher-order thinking tasks such as analysis, synthesis, and
evaluation (Bonwell & Eison, 1991).
Baseline Training: Conventional or traditional debriefing that was immediate,
quantifiable, and objective where the instructors provided the majority of
the input and students were basically passive recipients (Butler, 1993).
Performance effectiveness after conventional debriefing provided the
baseline measurements for this study.
Captain: The pilot designated to have overall responsibility for the flight crew
V
actions and for the safety of the flight (Campbell & Bagshaw, 1991, p. 143).
The Captain is often identified as the Pilot-in-Command or the Aircraft
Commander. Normally the Captain sits in the left seat of an airplane’s
cockpit.
. Cooperative Learning: The instructional use of small groups so that students work
together to maximize their own and each other’s learning (Johnson,
Johnson, & Smith, 1991).
Co-pilot: The pilot designated to assist the Captain in flight deck responsibilities,
Also known as the First Officer. Normally the co-pilot sits in the right seat
of an airplane’s cockpit.
V
y
Crew Resource Management (CRM): The process of managing all resources for
V
the benefit of flying an airplane safely. These resources may be on or off the
airplane and include both humans and mechanical systems (Trollip &
Jensen, 1991, p. G-3). The effective use of available human, technical, and
information sources (Lauber & Foushee, 1983). In earlier literature, CRM
meant Cockpit Resource Management.
I
11
11
Crew: The essential people required to operate an aircraft. For this study, a crew
consists of a Captain (left-seat pilot) and a Co-pilot (right-seat pilot) who
share the flying duties.
General Aviation: That portion of civil aviation which encompasses all facets of
aviation except scheduled airlines and large aircraft commercial operators
(AGFA, 1991).
Human Factors: The technology concerned to optimize the relationships between
people and their activities by the systematic application of human sciences,
integrated within the framework of system engineering (Edwards, 1988,
p. 9). It is the study of how people interact with their environments (Trollip
& Jensen, 1991, pp. 1-2).
Interpersonal Skills: The ability to work effectively as a team member and to build
cooperative effort within a team as a leader (Butler, 1991).
Line: An aviation industry term that encompasses the total operational
environment distinct from the training environment (Hawkins, 1987).
Line-Oriented Flight Training (LOFT): A flight training scenario in which flight
crews fly a complete trip in a high-fidelity simulator. These flights are made
in real time, and no intervention is made by the instructor regardless of the
actions of the crew. LOFT is often an integral part of or a supplement to
Crew Resource Management (CRM) training (Foushee & Helmreich, 1988,
p. 221).
Line-Oriented Simulation (LOS): Same meaning as Line-Oriented Flight Training
(LOFT). The term was developed to emphasize that the training occurred
in a simulator and not in a line flight.
Mission-Oriented Simulator Training (MOST): The military counterpart of LineOriented Flight Training (Caro, 1988).
SHEL Model: The interrelationship between the three types of system resource
(Software, Hardware, and Liveware) and their Environment (Edwards,
1988, p. 12). '
Technical Skills: A pilot’s knowledge of aircraft systems and an understanding of
and proficiency in skills involving piloting procedures and techniques
(Butler, 1991).
I'.
I
12
CHAPTER 2
LITERATURE REVIEW
Crew Resource Management ICRMl
Hawkins (1987) defines Crew Resource Management as the management
and utilization of all the people, equipment, and information available on the
aircraft. The concept originated from two conclusions: (a) that some accidents
occur because the resources available are not adequately used, and (b) that
sufficient resources were available to avoid the accident if they had been utilized in
the optimum way.
Edwards (1972) developed a conceptual model that delineates the four types
of resources available to the cockpit crew:
1. Software: Collection of documents which include rules, regulations,
laws, orders, standard operating procedures, customs, practices, and
habits.
2. Hardware: Physical property - the aircraft itself as well as terminals,
vehicles, equipment, materials, and so forth.
3. Environment: Physical, economic, political, and social factors that affect
the crew from outside the airplane.
13
4. Liveware: Humans involved - flight and cabin crew, passengers,,
controllers, aircraft and baggage handlers, and dispatchers.
These four system resources interact with each other; the relationships between
them are represented in the model by their initial letters (SHEL) (Edwards, 1988).
Crew Resource Management is concerned with interactions between all four
of these resources. The Liveware (Human) interactions, called "teamwork,"
constitute almost the entirety of CRM training courses, because successful
interactions between other resources hinge on teamwork. Two central issues in
implementing effective teamwork training are: (a) the specific skills (content) to be
learned, and (b) the learning methodology (strategy) to gain the needed behavioral
changes (Telfer & Biggs, 1988).
CRM Skills
A quality CRM program should identify the specific interpersonal skills that
will be taught and evaluated. These group interaction (or teamwork) skills are
»
•
'■
distinctly different from the personal skills and technical skills that constitute the
subject matter of undergraduate flight training. In CRM, students Ieam the
dynamics of group interaction and how group members communicate with each
other in order to coordinate their individual actions. Bolman (1979) proposes a
"Theory of the Situation" which shows how various forces interact with each other
in an attempt to gain an awareness of the situation. The following definitions are
offered for the elements of this model:
14
1. Theory of the Situation: What you assume to be true, based on your
perception of the facts you have at any point in time.
2. Reality: The situation as it is in the "real world" - often not fully known
until after the fact.
3. Theory in Use: Your predictable behavior in a given situation that has
been developed since birth.
4. Espoused Theory: Your account or explanation of your behayior.
5. Theory in Practice: The set of skills, knowledge, and experidnc^you call
upon according to your theory of the situation (Trollip & Jensen, 1991,
pp. 9-17).
If the flight crew’s assumptions are correct, safe flight decisions are likely; but if a
discrepancy exists between their theory of the situation and reality, danger exists.
Most CRM accidents, according to Trollip and Jensen (1991), are a result of these
discrepancies going unrecognized and magnified under stress. Proper interactions
(teamwork skills) are the tools for handling these discrepancies.
In discussions of teamwork, the concept of synergy among flight crew
members is an important factor. Synergy is combined action, the notion that two,
three, or more crew members working together in a sound way can produce a more
effective solution to a problem than any one person (Telfer & Biggs, 1988).
Interaction and communication is the key to synergism which is best achieved when
five concepts (inquiry, advocacy, listening, conflict resolution, and critique) are used
by all individuals within the team (Trollip & Jensen, 1991). Other authors discuss
15
synergy using different terms such as "information flow" (Foushee & Helmreich,
1988) and "information sharing" (Stone, 1988) as crucial variables of teamwork.
Stone (1988) stresses division of duties between pilots, information sharing, and
communication of intentions as essential teamwork skills. Foushee and Helmreich
(1988) link information flow with two other indispensable teamwork variables:
feedback and acknowledgement of information provided by other crew members.
The first attempts to identify the essential content elements of a CRM
curriculum came from Blake and Mouton, who in 1982 customized their
Managerial Grid Theory to CRM as a means of understanding teamwork dynamics
in the cockpit. They identified five elements of teamwork: inquiry, advocacy,
conflict resolution, critique, and decision making. In 1985, a group of 13 aviation
professionals compiled a more extensive list of 20 teamwork skills that they
considered vital to CRM training:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
listening
communication
assertiveness
awareness of the situation
ability to deal with conflict
problem solving
problem definition
priority and analysis
open-mindedness
personality awareness
managing distractions
fatigue management
use of checklists
decision making
pattern recognition
crew incapacitation recognition
workload assessment
16
18. division of attention
19. stress management
20. ability to critique. (Teller & Biggs, 1988, p. 139)
Individual airlines developed their own training programs, each listing teamwork
skills that constitute the essential elements of their individualized CRM program.
A typical offering is the list of principles of flight resource management developed
by American Airlines. They include:
1.
2.
3.
4.
5.
6.
7.
appropriate delegation of tasks and assignment of responsibilities,
establishment of a logical order of priorities,
continuous monitoring and cross checking of essential instruments and
systems,
careful assessment of problems and avoidance of preoccupation with
minor ones,
utilization of all available data to conduct an operation,
clear communication among crew members of all plans and intentions,
and
assurance of sound leadership by the pilot in command. (Telfer &
Biggs, 1988, p. 139)
Perhaps the most definitive list, though not a complete one, comes from Federal
Aviation Administration (1993) which defines seven teamwork skills which should
be acquired:
1.
2.
3.
4.
5.
6.
7.
Communication
Situational Awareness
Problem Solving, Decision Making, Judgement
Team Management
Stress Management
Team Review
Interpersonal Skills
In the Line Operational Simulations Advisory Circular. AC 120-35B. the
Federal Aviation Administration in cooperation with the Air Transport Association
17
formulated a taxonomy for CRM skills (FAA, 1992). Three main clusters of CRM
X
skills were defined as: (a) Communication Processes and Decision Behavior,
(b) Team Building and Maintenance, and (c) Workload Management and Situational
Awareness. The clusters were broken down further into subcomponents as follows:
1.
Communication Processes and Decision Behavior
Briefing (conduct and quality)
Inquiry/Assertion/Advocacy
Crew self-critique (decisions and actions)
Conflict resolution
Communications/Decisions
2.
Team Building and Maintenance
Leadership, followership, and concern for tasks
Interpersonal relationships/Group climate
3.
Workload Management and Situational Awareness
Preparation/Planning/Vigilance
Workload distribution/Distraction avoidance
In this study, these teamwork skills will be considered the essential content
elements of any CRM curriculum regardless of the delivery method or proposed
learning strategy.
V
■:
18
CRM Learning Strategies
Methods for teaching and learning CRM are also varied, and many useful
strategies have been developed. The Blake and Mouton (1982) CRM Grid Model
defined two phases of training:
1.
Phase I: A home study program that provides a common language for
use in applying succeeding phases of the program, and
2.
Phase II: A structured learning process in a seminar environment where
crews analyze how they react to various leadership styles and how their
own behavior can affect operational outcomes.
To extend the training into the flight environment and to optimize teamwork and
cockpit behavior, the use of critique and feedback was added to the Grid Model
(Telfer & Biggs, 1988). United Airlines developed the first formal airline CRM
training program in 1979, and later adopted the Grid Model as a teaching strategy.
As programs developed, many techniques evolved from both professional training
groups and the airlines. The following strategies were incorporated into CRM
training programs: classroom lectures, group exercises, seminars, personality
assessments, feedback techniques, role-playing, case studies, and interpersonal
encounter drills (Foushee & Helmreich, 1988).
Line-Oriented Flight Training (LOFT). One strategy that has developed
harmoniously with CRM and has now been widely adopted is LOFT. It is the
instruction, and sometimes testing, of flight crews as a team rather than as
X
19
individuals. Hawkins (1987) credits the development of LOFT to the realization
that individual proficiency is not sufficient when the skill must be practiced in a
group activity where personality and situational factors interact to create overall
performance. Training programs should not concentrate on changing attitudes
alone; it is critical to reinforce changes in attitude by providing opportunities, such
as LOFT, to place new attitudes into practice (Helmreich, Foushee, Benson, &
Russini, 1986). Helmreich (1984) considers the most effective CRM learning
strategy to be presentations of CRM skills, followed by behavioral exercises such as
LOFT.
LOFT scenarios are flights flown in a mission-oriented simulator with a full
crew complement. They are flown in real time, and no intervention is made by the
instructors regardless of the actions of the crew (Foushee & Helmreich, 1988). As
crews are faced with routine as well as nonroutine, emergency, and judgement
situations, they use CRM principles to bring the flight to a successful conclusion.
The strength of LOFT training is that it combines CRM principles with vital
performance and judgement skills in a single learning environment. The findings
suggest that more emphasis should be placed on task behavior in CRM training
(Cook, 1991). Using LOFT to evaluate the outcomes of a CRM training program
requires the program to focus on task behavior and crew performance.
Massey (1990) outlines the procedures used: as part of the CRM training
program, LOFT exercises are flown in a simulator and the exercise is videotaped
for debriefing after the LOFT has been flown. The crew debriefs itself in
X
20
cooperation with the flight instructor; then the tape is erased by the crew to ensure
that it is used only for training and not for evaluation or employment decisions.
LOFT is recognized as a valid method for teaching CRM, but it is not an
essential strategy element of CRM training; many smaller airlines, which cannot
afford simulators, conduct CRM training programs without LOFT exercises
(Massey, 1990). The Federal Aviation Administration (1993) acknowledges that
there are many approaches and learning strategies useful in CRM training. They
list certain features that are necessary for effective CRM programs:
1.
The program should focus on the functioning of crews as intact teams.
2.
The program should provide opportunities for crew members to
practice the skills that are necessary to be good team leaders and
members.
3.
The program should help crew members learn that how they behave
during normal, routine circumstances can have a powerful impact on
how well a crew functions during high workload, stressful situations.
Since the Federal Aviation Administration (1993) presents federal guidelines for
developing, implementing, and evaluating a CRM training program, these three
principles are considered the essential strategy elements for,all CRM training
programs, including undergraduate flight training.
21
Behavior Modification Model
The predecessor to this study was a CRM course that was taught according
to the Behavior Modification Model. The first challenge was to modify CRM
training for the inexperienced pilot and to develop effective strategies which
cultivate CRM skills early in a pilot’s career. A college CRM course was developed
based on guidelines established for FAR Parts 121 and 135 (FAA, 1993). The
objective was to give these students realistic CRM and crew training despite their
inexperience. The Behavior Modification Model was chosen because the overall
goal of CRM training is to modify behavior. It was evident that operant
conditioning is a powerful tool for behavior modification; it reinforces desired
behaviors and immediately corrects adverse behaviors. In this model, three phases
were repeatedly utilized to reinforce learning:
1.
learning sessions where CRM skills were introduced,
2.
practice sessions where CRM skills were exercised, and
3.
debriefing sessions where feedback was used to reinforce or correct
behaviors.
Most current undergraduate flight training concentrates on technical skills:
the knowledge base, procedural activities, and motor activities required to fly an
airplane. These individual technical skills must be paired with crew-related skills to
ensure safe flight (FAA, 1993). In learning sessions, students attained awareness of
CRM skills through a variety of methods. The objective was for the students to
22
identify CRM skills, to define them operationally, and to recognize their own
positive or negative utilization of them.
Awareness of CRM skills was not sufficient to change behavior; students
had to perform them in an operational environment. Simulator LOFT provided a
viable opportunity to practice CRM skills in a learning environment. In LOFT, a
realistic flight simulator and highly structured scenarios were used to reflect realworld line operations. The flights were flown in real-time with no external
assistance except what would be supplied by Air Traffic Control (ATC) (Foushee &
Helmreich, 1988). Simulator sessions were videotaped for use in debriefing
sessions.
Feedback is the essence of behavior modification; it is the reinforcer of
positive skills and the correction tool for negative behavior. Feedback can come
from the crew members themselves in the form of self- or peer-critique, from
videotapes, or from third-party evaluators. Feedback is most effective if it is
immediate, quantifiable, and objective (Butler, 1992). In the behavior, modification
model, the debrief was scheduled immediately following a LOFT simulation, they
were interactive between students and instructors, and observations were confirmed
by field notes, plotter data, and videotapes.
The behavior modification model had significant limitations because it dealt
only with manifest behaviors without inquiry into personality, attitudes, learning
processes, common sense, or judgement (Butler, 1992). Most of the shortcomings
of the behavior modification model occurred in debriefing sessions:
23
1.
Students focused on technical skills rather than CRM skills.
2.
Errors were emphasized rather than positive performance.
3.
Constructive feedback caused students to act defensively.
4.
Videotapes centered on technical skills, particularly errors.
5.
Students tended to passively observe videotapes.
For these reasons, the behavior modification model had only limited success
as a learning strategy for an undergraduate CRM program. Technical and crew
errors are expected to occur in LOFT training at this level; LOFT training is most
effective when errors are tolerated without penalty (Hawkins, 1987). The limited
success of the behavior modification model made it clear that a different strategy
was needed. Though the guiding principles appeared sound, another model for
debriefing was explored.
1
Discovery-Learning Model
Discovery-learning was chosen as an alternative to the behavior modification
model for this study. Discovery-learning is a model where the learner is placed in
the environment which is structured to allow the learner to manipulate it, to learn
its limits, and to apply concepts to other situations and environments (Butler, 1992).
Numerous aircrew learning contexts have been surveyed by Butler and simulator
LOFT appears to be the best environment for discovery-learning. If the reduction
of human factor errors in the cockpit is the goal, then CRM skills should be
discovered by individuals rather than poured out from a fountain of knowledge.
24
Formal CRM training and LOFT are designed not only to instruct but also to
encourage each pilot to examine his/her own management skills and to implement
other CRM techniques that may seem even more effective (Butler, 1991).
Theoretical or Conceptual Framework
The innovative learning method (treatment) employed in this study was a
discovery-learning strategy called self-analysis. Self-analysis is a LOFT debriefing
strategy that was compared to conventional (behavior modification) debriefing to
determine if effectiveness could be improved by self-analysis. When self-analysis
was in effect, students analyzed videotapes of their own LOFT simulator flights,
concentrating on their performance of CRM skills. From a theoretical, viewpoint,
this approach may very well produce some tangible changes in attitudes and
behavior (Foushee & Helmreichi 1988). The Theory of Objective Self-Awareness
(Duval & Wicklund, 1972) professes to explain numerous behaviors falling within
the domain of human performance. The theory assumes that self-focusing stimuli
often forces objective appraisals of oneself that may lead to attitude and behavior
changes (Wicklund, 1975). The theory draws heavily from Piaget’s (1966)
Cognitive Theory and Festinger’s (1957) Cognitive Dissonance Theory. According
to Festinger, if a person is led to focus on discrepant cognitive elements, dissonance
reduction will result. Other researchers, Jones and Nisbett (1971), proposed that
two people, the actor and the observer, will offer two different explanations of the
actor’s behavior; the actor explains his behavior based on environmental
25
contingencies, but the passive observer is likely to interpret it based on the actor’s
personality. Duval and Wicklund (1972) have made the actor and the observer the
same person, by making the actor an observer of himself, and found greater
attribution to self when the actor is turned into a self-observer.
Wicklund (1975) explains how the Theory of Objective Self-Awareness
works: The initial reaction to self-focused attention is self-evaluation, which can be
favorable or unfavorable, depending on the nature of the discrepancy noted. It is
assumed that discrepancies in this context are mostly negative. There are two
possible reactions to a negative discrepancy in addition to the self-evaluation: to
avoid the self-focusing stimuli, or to create an objective self-awareness that will
result in attempted discrepancy reduction. If escape from the self-focusing stimuli
is not possible, discrepancy reduction will follow.
Wicklund’s (1975) research has demonstrated changes in both attitude and
performance, specifically, both lowering and raising of aggression, improved rate of
performance on simple tasks, dissonance reduction, and conformity. WicMund and
Brehm (1976) reported a meta-analysis of research projects that show positive
results for objective self-awareness in distinct fields such as marketing,
interpersonal persuasion, politics, religion, disconfirmation of cherished beliefs,
clinical psychology, marriage, smoking, and gambling.
In a study by Gibbons and Wicklund (1982) the result of self-directed
attention on prosocial behavior was examined; they concluded that self-focus and
prosocial behavior are closely related to one another only to the extent that the
26
individual is "self-reflective." Recently, Duval, Duval, and Mulilis (1992) examined
factors that determine the decision to withdraw or to progress towards discrepancy
reduction after an objective self-awareness. They drew heavily on the work of
Carver and Scheier (1981) who assume that the decision to match or withdraw is
determined by outcome expectancy favorability. Combining the two theories,
Duval et al. (1992) assessed the match or withdraw response in relationships to the
magnitude of the discrepancy. They confirmed that larger discrepancies lead to
withdrawal more than to discrepancy reduction. More importantly, they found that
the objective self-awareness group, who believed they were making rapid progress
towards discrepancy reduction, clearly evidenced greater efforts to match their
behavior and attitudes to the standard; even if the initial discrepancy was very large.
Objective self-awareness theory has implications for the present study. The
goal of the study is to determine if self-analysis can shape attitudes and increase
performance of CRM skills. It should be noted that a possible result of a negative
self-awareness stimuli is withdrawal from the self-analysis situation; however, if
escape from the self-focusing stimuli is made very difficult, discrepancy reduction
should follow. The Gibbons and Wicklund (1982) study shows that self-awareness
is not sufficient to produce results; the students need time and incentive to engage
in "self reflection" in order to observe significant results. The article by Duval et al.
(1992) showed that evidence of progress towards discrepancy reduction can
motivate students to continue their efforts to learn and practice the CRM
principles, even if they perceive that their errors are numerous or severe. The
27
implication is that undergraduate flight students can learn CRM through selfanalysis despite their lack of experience in crew operations.
Learning Strategy - Cooperative Learning
As in the behavior modification model, learning sessions should provide
awareness of the CRM skills. In the discovery-learning model, however, learning
becomes an active process, removed from the classroom setting and placed in the
crew context (Bonwell & Eison, 1991). One active learning method is the
Cooperative Learning procedure called "Jigsaw" (Johnson, Johnson, & Smith,
1991). Each student is individually responsible for a number of readings or
assignments. When both are prepared, they are scheduled for a learning session to
interchange information with each other by sharing their readings, discussing case
studies, analyzing accident reports, and writing team response papers. Students are
responsible for their own learning, and the instructor becomes a facilitator and a
learning resource.
Practice Strategy - LOFT
In the discovery-learning model, simulator LOFT is the focal point of
training as it was in the behavior modification model. Simulator scenarios
deliberately focus on CRM skills such as crew communication, situational
awareness, judgement, decision making, and leadership skills. Faced with routine
as well as. nonroutine, emergency, and judgement situations, crews must use CRM
skills to bring the flight to a successful conclusion. Situations are subjective and
%
28
usually do not have a "right" or "wrong" outcome; instead, students discover for
themselves the relationships between CRM skills, safety and error. In the
discovery-learning process, students experience both success and error in the LOFT
simulator.
Ruffell-Smith (1979) in a simulation study provided strong evidence for the
importance of experiencing group performance, finding that crew interactions were
significantly related to safety. He found that most problems were related to
breakdowns in crew coordination, not to a lack of technical knowledge and skill.
"High-error" crews experienced difficulties in the areas of communication, crew
interaction, and integration.
Several studies delineate group processes that jeopardize safety. Caro
(1988) identifies safety factors such as aircrew preoccupation with minor
mechanical problems, failure to delegate tasks and assign responsibilities, failure to
set priorities, failure to communicate intent and plans, failure to utilize available
data, and inadequate monitoring of other crew member’s performance. Focusing
on the vital area of communication, Campbell and Bagshaw (1991) list
communication blocks that impact safe flight: pre-occupation, resentment, status
differential, and strongly held opinions. Foushee and Helmreich (1988) discuss a
malady some refer to as "Captainitis," a situation where a Captain does not achieve
the full potential of the crew because of ego or lack of leadership. All of these
factors are behaviors that can be experienced tangibly in LOFT simulations.
29
Hawkins (1987) warns that there are some factors that probably will not be
affected by CRM or LOFT training. Personality traits and the effects of domestic
stress, fatigue, circadian rhythm disturbance, drugs or medication will probably
change very little with LOFT training. Also, LOFT training will not put an end to
errors in information processing, eliminate visual illusions, or do away with designinduced error.
Debriefing Strategy - Self-Analysis
Instead of having instructors debrief students after LOFT training,
discovery-learning allows pilots to examine their own performance and to analyze
the CRM skills they have learned (Butler, 1991). Recognizing that college crews,
may need assistance in analyzing their LOFT performance, self-analysis of LOFT
Training was developed. It is a discovery-learning strategy that provides focusing
tools and specific learning tasks to support the self-analysis.
Though the use of videotapes had shortcomings in the behavior modification
model, they still provide the most objective criteria for self-analysis (Hawkins,
1987). In self-analysis of LOFT, two strategies enable videotapes to be used for
discovery: learning tools and learning tasks. The learning tools and the learning
task transform the debrief session from "watching videotapes" to an "active learning
process" that focuses on CRM skills, places the locus of control within the students,
and emphasizes positive use of CRM skills rather than focusing on errors (FAA,
1992).
30
Evaluating Effectiveness
The concept of effectiveness implies demonstrated proficiency or evaluation
of performance. Measurements of teamwork skills are not only evaluations of the
effectiveness of the CRM training program, but also assessments of the
performance of crew members. Presently, in the airline community, the training
associated with CRM and the simulator experience associated with LOFT are
entirely nonpunitive and current philosophy encourages them to remain that way
(Foushee & Helmreich, 1988). Especially since LOFT training has expectations of
errors, this kind of training is effective only in a situation where errors are allowed
to occur without penalizing the trainees (Hawkins, 1987). There remains a definite
separation between training and evaluation at this juncture, however, there is a
logical extension into required demonstration of proficiency in crew resource
management. Should this occur, there is an expected outcry from pilots who have
apprehensions that performance in this area could be used as a condition of gaining
or maintaining a license to operate in multipilot aircraft (Foushee & Helmreich,
1988).
In situations where a high level of teamwork is required, the individual skills
of team members are often not enough to assure effective performance of the
group (Jones, 1974). Group performance criteria has received relatively little
attention in the aviation research and training communities until recently; this
heightened awareness has been stimulated, unfortunately, by a number of tragic
accidents relating to flight crew performance. Accidents such as these have
prompted some researchers to believe that the "group effectiveness problem"
cannot be solved in the near future (Foushee & Helmreich, 1988). But those
claiming that CRM training and LOFT scenarios are ineffective tend to overlook
evidence from two accidents in which both Captains credited their crew’s CRM
training for reducing loss of life (Cook, 1991).
CRM training assessment requires more defined techniques to assess the
direct or indirect effects of training programs on the outcomes they wish to
produce. Unfortunately, current knowledge is not advanced enough to establish
absolute standards of measurement (Gregorich & Wilhelm, 1993). Training
programs are currently assessed using multiple measures which provides less
ambiguous assessment of program elements and outcomes as well as more reliable
and more interpretable findings (Helmreich & Wilhelm, 1987). Rather than using
a single instrument to measure effectiveness, the crew effectiveness research
methodology emphasizes the assessment of crew performance via converging
sources of data (Chidester, Kanki, & Helmreich, 1989). Helmreich (1991) lists six
methodological approaches that can be used to measure crew effectiveness. Four
of these are within the scope of undergraduate flight training and are addressed in
the research questions of this study:
1.
Survey data on crew member attitudes as indicators of culture and
training effects: The CMAQ.
2.
Systematic observation of crew behavior: The LINE/LOS Checklist.
32
3.
Survey data on crew reactions to LOFT: The CRM Survey.
4.
Micro-coding of communications: Communications Analysis.
A
Each of the above evaluation methods measure a different aspect of effectiveness.
Reliability can be maximized by establishing that the evaluation methods converge
on a global measure of effectiveness (Chidester, Kanki, & Helmreich, 1989).
Attitudes
The thrust of CRM training is crew dynamics and the target of CRM
evaluation is crew effectiveness. Under aircrew team dynamics, Vandermark
(1989) lists three vital components: personality, attitude, and communication.
Similarly, when discussing crew effectiveness, Gregorich, Helmreich, Wilhelm, and
Chidester (1989) state that it is largely determined by technical skills, attitudes, and
personality characteristics. Personality and attitudes must be considered both as
objectives of crew dynamics training and as measures of crew effectiveness.
Personality, which Helmreich (1983a) calls traits are deep-seated
dispositions to behave in a certain way (Vandermark, 1989). They are acquired
relatively early in life and are stable and resistant to change (Helmreich, 1983a).
Personality must be a factor in the selection of pilots, but it is "unrealistic to believe
that training can effect lasting changes in personality" (Helmreich, 1983a, p. I).
There is considerable evidence that personality traits are directly linked to flight
deck performance. If these traits are the major determinant, then the best strategy
I
33
would be to concentrate on selection and not to worry about teaching old dogs new
tricks (Helmreich, 1984).
Attitudes, on the other hand, are less stable characteristics that are learned
patterns of responding (Helmreich, 1983a). They can be acquired at any time
during the lifespan and are relatively open to modification through intellectual or
emotional appeals. Though both personality and attitudes affect crew
performance, the data show that they are relatively independent (Helmreich,
1983b). This independence suggests that CRM training can reshape attitudes
through intellectual or emotional appeals and that CRM assessments can use
attitudes as a measure of actual performance (Helmreich, 1984).
Examining the link between attitudes and performance, Helmreich (1983a)
concluded, "Effecting change in management attitudes can produce marked
changes in behavior and can improve the effectiveness and safety of flight
personnel" (p. 10). Another study (Helmreich, Foushee, Benson, & Russini, 1986)
established a direct linkage between self-reported attitudes and independent
evaluations of performance. Measuring attitudes provides an indirect approach to
assessing the potential effectiveness of resource management.
While research shows significant positive increases in attitudes resulting
from CRM training, ten to fourteen percent of the individuals tested were either
unaffected by the training or showed a negative result of training with attitudes
"boomeranging" in the direction opposite of that intended. Helmreich and Wilhelm
(1989) pinpoint personality factors and group dynamics as the critical determinants
Ii
34
of the direction of the change in attitudes. Studies revealing the "boomerang effect"
were done with airline pilots, but it is reasonable to consider that it could be
present in undergraduate students as well.
Training designed to instill or enhance the desired attitudes in flight crew
members is now commonplace in both commercial and military aviation. To.
evaluate the effectiveness of such training, the CMAQ (see Appendix A) was
designed as an index for measuring attitudes (Helmreich, 1984). "The items on the
survey were chosen to tap issues that are conceptually and/or empirically related to
cockpit resource management" (Helmreich, 1984, p. 585). It has three functions in
CRM training: providing an evaluative pre- and post-training snapshot of attitudes,
identifying individual areas that may need special attention, and evaluating the
impact of CRM training (Gregorich, Helmreich, & Wilhelm, 1990).
Systematic Observation of Crew Behavior
The overall objective of LOFT is to improve total flight crew performance,
meaning that, in parallel with technical skills, crew coordination and
communication skills must be both learned and assessed (FAA, 1992). Traditional
evaluation of pilot performance concentrates on assessing individual technical
skills. However, considering that crew coordination problems have played a
significant role in more than half of aviation’s accidents and incidents (Gregorich,
Helmreich, & Wilhelm, 1990), evaluation of individual flying skills is not enough.
35
Detailed data on the cognitive and interpersonal skills of crew members are needed
(Helmreich, Wilhelm, Kello, Taggart, & Butler, 1991).
Consensus seems to be against using LOFT as an evaluation tool primarily
because pilots perceive that any formal evaluation includes the risk of
decertification, regardless of assurances to the contrary (Butler, 1992). Yet there is
a need to reinforce and evaluate the impact of training in CRM (Helmreich,
Wilhelm, Kello et al., 1991). Otherwise, most crew performance data come from
reports where things end badly, the formal investigations of accidents and serious
incidents. Helmreich, Wilhelm, Kello et al. (1991) maintain that a better
understanding of crew effectiveness could be obtained from analysis of situations
where crews performed successfully.
In order to evaluate, the overall effectiveness of CRM /LOFT training, a
joint research project between the National Aeronautics and Space. Administration
(NASA) Ames Research Lab and the University of Texas developed the
NASA/UT LINE/LOS Checklist (see Appendix C). The LINE/LOS Checklist was
designed to provide accurate systematic data on crew performance including the
positive aspects (Helmreich, Wilhelm, Kello et al., 1991). It defines eight specific
crew effectiveness markers and two global ratings that are each made specific by a set
of behavioral indicators. Research data indicate that the use of behavioral markers
to help evaluators classify observed behavior greatly increases the reliability of
observations (Helmreich & Wilhelm, 1991).
36
An issue of concern is whether reliable judgements of CRM behaviors can
be made with the LINE/LOS Checklist through the observation of only one flight
segment. It is the subjective feeling of Helmreich and Wilhelm (1991) that first
impressions of crew interaction are sufficiently clear that single observations are
suitable for analysis.
Survey Data on Crew Reactions
Whereas the LINE/LOS Checklist is an evaluation of performance by
observers, flight crews can report their own reactions to the training in a self-report
sheet called the CRM Survey (see Appendix D). The questionnaire asks questions
regarding familiarity with the scenario, its value for technical and crew coordination
training, the effectiveness of the instructor and debriefing, workload imposed, and
evaluation of team performance and use of CRM concepts (Helmreich, 1991).
Survey data from more than 20,000 flight crew members showed
overwhelming acceptance of CRM training with the vast majority of pilots
endorsing the training as both relevant and useful (Helmreich & Wilhelm, 1991).
A similar pattern was found in a survey from more than 8,000 participants in LOFT
exercises where crew members overwhelmingly felt that LOFT was important and
useful training, and that it had value for technical training as well as training in
CRM skills. Clearly, acceptance of the training program is a necessary but not
sufficient indicator of its effectiveness, but if the crew members do not perceive the
37
training to be useful, the result may be increased awareness of CRM concepts but
little change in observable behavior (Helmreich & Foushee, 1993).
Communications Analysis
Another approach to evaluation of crew performance follows a research
tradition known as ethnography and involves a trained observer making extensive
notes on the actions of crews during extended observations (Helmreich & Wilhelm,
1991). They agree that this type of data provides an extremely rich record of the
characteristics of particular crews and is an invaluable research technique. The
I I
drawbacks that Helmreich and Wilhelm note is that it is labor intensive, it requires
detailed content coding, and it is limited to a small sample of crews. However, it
provides more detailed data than any of the other evaluation measures.
Effective crew coordination depends on verbal communication to transmit
and receive information; the foundation for achieving effective teamwork in the
cockpit resides within the information transfer process (Kanki, 1991). In order to
understand these group process issues, researchers are implementing ethnograhic
methods using communications-based research that analyzes the information
transfer process in the cockpit.
Kanki (1991) describes five studies that used a variety of techniques to
analyze the speech records from actual line flights or LOFT simulators. In each
case the basic data are the verbatim transcripts of interactive speech sequences that
are coded according to a particular classification system. "Each coding system is
,
38
tailored to meet the needs of a particular research/ theoretical focus as it relates to
communication/information transfer process" (Kanki, 1991, p. 247).
There are four phases of the research that apply to all the studies in Kanki
(1991): designing the data source, transcribing the speech record, coding according
to a model, and analyzing the data. Table I is a summary of these phases.
Table I. Group Process Communication Research.
I
Design Phase
. D ata source (e.g., field versus simulation study
. Level of analysis (macro/micro)
. Define critical contrasts
II
Transcribing Phase
. Identify portion of videotaped records to be transcribed
. Transcribe verbal data in real-time sequences
. Demarcating critical events and flight phases
III
Coding Phase
. Speech act sequences
. Problem solving functions
. Managerial functions
. Information model
. Standard versus nonstandard terminology
IV
Analysis Phase
. Identify communication and other group process patterns that relate
to input and/or outcome variables
______________ ___________________________(Kanki, 1991, p. 251)
Kanki (1991) calls the coding phase the most crucial and creative because
the research question must be translated into a discriminating coding system by
which all speech will be characterized. Several studies describe a variety of coding
LI
K
39
strategies, each aimed at uncovering important differences in communication
patterns. The majority of researchers begin with a primary coding strategy
developed by Foushee and Manos (1981) in which each communication act is
assigned to one of 18 communications categories (see Appendix E). Secondary
codes are often assigned to the same speech acts to uncover important
communications patterns relating to the research question. For example, Kanki,
Palmer and Veinott (1991) created a secondary code of speech sequences to
determine initiators and responders in cockpit communications.
Communications Analysis has potential to be far more than an assessment
tool. It can.provide direct, objective evidence to crew members on their
communications and their effectiveness in crew coordination. Mostly, it provides
an opportunity to forsake the negative aspects of evaluation and to concentrate on
positive skills reinforcement. One of the reasons we lack a full understanding of
effective crew performance is that the database on crew performance is built on
accident or serious incidents. We would have a better understanding of crew
effectiveness if we could analyze crew behavior in those situations where the crews
were particularly effective (Helmreich, Wilhelm, Kello et al., 1991).
CRM and LOFT in Undergraduate Flight Training
CRM and LOFT are usually considered to be advanced training programs
for experienced airline crews. Some undergraduate programs offer a CRM seminar
for upper division flight students, but combining it with LOFT training is rare. Very
40
few research projects have been conducted with undergraduate crews using both
LOFT and CRM. Ross, Slotten, and YeazeFs (1990) study incorporates a "LOFT"
scenario into an initial (undergraduate) flight training program. Since it uses only
single-pilot crews, there are no cockpit interactions involved; it is individual
performance training and does not evaluate the learning of CRM principles.
The best models for LOFT training in undergraduate programs come from
team training included as part of ab initio training (Trollip & Jensen, 1991). These
programs break away from the strong emphasis on individual performance typical
of primary flight training. From the beginning of their flying careers, ab initio
students fly as part of a team, initially with an instructor in the right seat and a
second student in the back seat who is also an active participant in the flight. "By
encouraging teamwork and constructive criticism from the beginning, students build
a habit of cooperation rather than competition" (Trollip & Jensen, 1991, pp. 9-15).
In a CRM training program for Air Traffic Control (ATC) Specialists at
Hampton University, Weisman (1991) found that teamwork skills and individual
competence can be developed in an undergraduate student simultaneously. She
recommends that a prototype CRM curriculum be developed for undergraduate
programs to be used with all of the aviation specialties, including pilots.
The military has been the traditional source of airline pilots for years.
Defense cutbacks will reduce the number of military pilots available to U.S. airlines
in the future. General Aviation, where pilots build flight hours by flight instructing
and flying charters and light cargo, is emerging as the principal source of
41
journeyman pilots. General Aviation creates a group of pilots with entry skills that
are more diverse than the typical expertise of military pilots. Airline standards
cannot broaden to accommodate such differences; new technology and automation
demand greater technical competence than ever, and crew resource management
introduces additional standards for pilots to master.
College aviation programs are in a unique position to fulfill these
requirements. Because colleges have more educational requirements than other
General Aviation programs, college flight training is a broader, more diverse
education. Colleges influence their students for several years longer than most
flight schools allowing time to integrate interpersonal skills into "airmanship" and
"professionalism" technical training. Some experts uphold that CRM is too
advanced for undergraduate flight training; others contend that it is wrong to defer
crew training until a pilot reaches the airlines (Trollip & Jensen, 1991). Perhaps,
the strongest argument in favor of including CRM in initial flight training is that it
is successfully being taught in European ab initio programs (Nash, 1992).
42
CHAPTER 3
METHODOLOGY
Research Objectives
Beginning with the premise that self-analysis of LOFT is the most effective
way for airline pilots to learn CRM, the purpose of this study was to determine if
undergraduate flight students can learn CRM through self-analysis of LOFT
despite their inexperience with crew operations. "Performance" was selected as the
measure of effectiveness. The Cooperative Learning sessions and LOFT practice
sessions were common to all crews, but debriefing sessions (conventional or selfanalysis) were uniquely distinctive so that differences in CRM performance could
be measured:
Research Design
The design for this research was an alternating treatments design (Gay,
1992), a type of single-subject design. In the alternating treatments design the
subject is alternately exposed to a nontreatment (baseline) and a treatment phase,
and performance is repeatedly measured in each phase. In this research, the
treatment (self-analysis debriefing) was alternated with the baseline (conventional
43
debriefing). Repeated measurements of attitudes, effectiveness, performance, and
self-reporting were made to determine if changes were a function of self-analysis.
Single-Subject Designs
The single-subject design is a variation of the time-series design (Gay, 1992)
or the longitudinal time design (Kerlinger, 1973) in which each subject serves as his
or her own control. Single-subject designs, sometimes called single-case designs,
are studies of the effect of interventions on an individual (Best & Kahn, 1989), but
they can also be applied when several individuals are considered as one group, such
as a flight crew (Gay, 1992). Most experimental research that compares treatments
requires a traditional group comparison design; however, a single-case design may
be more appropriate if there are not enough subjects of a given kind to permit the
formulation of two equivalent groups (Gay, 1992). Single-subject design differs
from case study research in the degree of experimental control (Borg & Gall,
1989); case studies rely heavily on qualitative data and the researcher’s subjective
impressions whereas single-subject design is an experimental method that is
typically concerned with manipulation of a treatment in order to draw inferences
about the effectiveness of the treatment (Cohen & Manion, 1985). Single-subject
research requires careful assessment, repeated observation or measurements, and
conscientious applications of the treatment (Best & Kahn, 1989).
44
A-B-A-B and Multiple Baseline Designs
The characteristics of single-subject research are discussed in terms of
A-B-A-B designs which determine the effects of an intervention by alternating the
baseline condition (the A phase) with the intervention condition (the B phase)
(Kazdin, 1982). Kazdin states that the effects of the intervention are clear if
performance improves during the first intervention phase, reverts back to baseline
levels when the intervention is removed, and improves again when the intervention
is recommenced at the second intervention phase.
Sometimes the effects of the treatment may not disappear when the
treatment is removed. Actually, in many cases, as in this research, it is highly
desirable if they do not. Multiple baseline design allows for conditions other than
the naturally occurring target behavior as a control for assessing treatment effects
(Borg & Gall, 1989). They are used whenever a treatment can be withdrawn but
when the effects of the treatment carry over into the second baseline phase and a
return to original baseline conditions is difficult or impossible (Gay, 1992). In this
research, a multiple baseline design was used; instead of collecting one target
behavior for one crew in one setting, several behaviors were collected for each crew
in each of the LOFT simulations.
Alternating Treatments Design
Alternating treatments design is a special application of either the A-B-A-B
design or the multiple baseline design; it is a relatively rapid alternation of
45
treatments for a single subject (Gay, 1992). "The design currently represents the
only valid approach to assessing the relative effectiveness of two (or more)
treatments, within a single-subject context" (p. 346). The design allows the for the
comparison of two or more treatments without controlling for possible order
effects. Treatments can be alternated for each session or they can be randomly
assigned to sessions (Best & Kahn, 1989). Additionally, the design may be used to
alternate between treatment and no-treatment without, having to establish a
baseline (Gay, 1992). In this study, the alternation was between treatment and no­
treatment for each LOFT scenario. For each crew, the first LOFT scenario was
randomly assigned to either conventional debriefing (no-treatment) or to
self-analysis debriefing (treatment).
Analytical Techniques
In single-subject designs, statistical analysis is rarely used (Best & Kahn,
1989; Borg & Gall, 1989). One criticism is that inferential statistics are not
compatible with the logic of the single-subject experiment; another is that
inferential statistics assume that observations are independent of each other which
is seldom satisfied in single-subject designs. In single-subject designs, data analysis
typically involves visual inspection and graphical analysis of the results (Gay, 1992).
Single-subject designs have their roots in clinical psychology. For single­
subject designs, Gay (1992) maintains that the primary criterion of treatment
effectiveness is clinical significance of the results rather than statistical significance.
I
Clinical significance means that the treatment must produce meaningful outcomes
in performance, not just mathematically significant results.
To tabulate the graphical results, a summary method of classifying individual
results to show magnitude and direction of a change was employed (Irwin, 1991).
Difference scores between observations were computed for each factor and subjects
were placed into one of seven groups:
Symbol
Amount of Change
those observations decreasing more than two standard
deviations since the last observation.
those observations decreasing between one and two standard
deviations since the last observation.
those observations decreasing less than one standard deviation
since the last observation.
nc
those observations that have not changed since the last
observation.
+
those observations increasing less than one standard deviation
since the last observation.
+ +
those observations increasing between one and two standard
deviations since the last observation.
+ + +
those observations increasing more than two standard
deviations since the last observation.
47
When alternating between self-analysis and conventional debriefing, an
effectiveness measure was considered to favor self-analysis if it showed gains for a
self-analysis session that was higher than the previous conventional debriefing
session. For example, if a self-analysis measurement showed that performance
increased more than two standard deviations since the previous conventional
session, it would be classified as "+ + +" signifying that one measurement
indicated that self-analysis session was more effective than the conventional
debriefing session.
Replication is a vital part of all research and especially, single-subject
research since initial findings are generally based on one or a small number of
subjects (Gay, 1992). Three types of replication are: direct, systematic, and
clinical. In this study, direct replication was used by the same investigator with
different subjects (different crews) in a specific setting (the same LOFT scenarios).
Population and Site Description
The study was conducted at Rocky Mountain College (RMC) in Billings,
Montana. RMC is a four-year, private, liberal arts college with an aviation
department which offers two undergraduate degree programs in aviation:
Aeronautical Science and Airway Science. RMC is an institutional member of the
University Aviation Association (UAA). The LOFT training was conducted at the
college’s Flight Simulator Laboratory in a Frasca Model 142 ground trainer.
48
Research Subjects
A two credit course in CRM and LOFT was offered at RMC in Spring, 1993.
The course prerequisites required students to be juniors or seniors, to possess a
Commercial Pilot Certificate and to be finished with multi-engine ground
instruction. Twelve students were enrolled. Prior to any CRM instruction, each
student completed a locally-developed questionnaire to document pilot experience,
education, and exposure to CRM (see Appendix B). Students received an
individual orientation session in the Frasca 142 simulator and were separately
evaluated on their flying skills. The Flying Skills Test score was used to assign
individuals to crews so that the technical skill level of each crew was balanced. Of
the twelve, nine were fully qualified to participate in the study; two were
unqualified and were excluded; and one was marginally qualified (Ed) but had
acceptable technical skills and was included to produce an even number for crew
assignments. None of the students had previous experience with CRM or LOFT.
Table 2 (with pseudonyms) summarizes their background and scores on the Flying
Skills Test. Eight men and two women were teamed as follows:
One crew with above-average skills (Alex/Art).
Two crews with average skills (Betty/Bob, Carl/Cathy).
One crew with mixed skills (Dan/Dave).
One crew with below-average skills (Ed/Eric).
Each crew completed five sessions of CRM and LQFT training; they had two selfanalysis debriefing sessions randomly inserted into their schedule.
Table 2. Summary of Students’ Background and Skills.
Pilot
Ratings
Year
Instru­
ment
Hours
MultiEngine
Hours
Total
Hours
Skills
Test
Alex
Comm. Instructor
Single-Engine
Multi-Engine
Instrument
Jr.
100
320
630
8
Art
Priv.
Single-Engine
Instrument
Jr.
33
0
168
8
Betty
Priv.
Single-Engine
Instrument
Sr.
36
0
206
6
Bob
Comm. Single-Engine
Multi-Engine
Instrument
Sr.
58
23
218
7
Carl
Priv.
Single-Engine
Instrument
Jr.
39
2
194
7
Cathy
Comm. Single-Engine
Instrument
Jr.
32
0
252
6
Dan
Comm. Single-Engine
Seaplane
Instrument
Jr.
56
0
1015
8
Dave
Priv.
Single-Engine
Instrument
Sr.
33
0
166
4
Ed
Priv.
Single-Engine
Jr.
19
0
128
5
Eric
Priv.
Single-Engine
Instrument
Sr.
41
0
199
3
Skills Test: 10, 9 ,8= above average technical skills
7, 6
= average technical skills
5 ,4 ,3 = below average technical skills
2 ,1 ,0 = unsatisfactory technical skills
50
Research Setting
Though airline LOFT training tends to use multimillion dollar simulators
with a six-axis motion system, wrap-around visual representation, sound effects, and
high-fidelity re-creation of operational parameters; it is possible that lower-fidelity
simulation using video recording of behavior might provide much of the needed
training at a lower cost (Foushee & Helmreich, 1988). In General Aviation, it is
not necessary to reach the level of fidelity required by the airlines; it is crucial to
identify and simulate only those cues that are necessary and sufficient for the high
transfer of learning (Campbell & Bagshaw, 1991). The Frasca 142 twin engine
simulator is designed to support the requirements of advanced collegiate flight
training providing the handling and performance characteristics similar to that of a
typical light, twin-engine General Aviation airplane (Frasca, 1986). It is possible to
conduct controlled studies of group process variables in this type of simulator with
confidence that the results are strongly representative of the real world (Foushee &
Helmreich, 1988).
The communication system in the Frasca 142 was modified for LOFT
training to provide the following:
1. Monitoring of radio communications by the simulator instructor,
2. monitoring of both radio and internal communications by the
researcher,
3. video-recording that includes both radio and internal communications,
and
51
4. backup audio recording that includes both radio and internal
communications.
Simulator Procedures
Before the simulator flights, students were provided with complete preflight
information including the scenario, flight planning information, en route and
approach charts, and a printout of the current and forecasted weather. The crews
had an opportunity to. ask the instructors questions and to clarify any requirements.
From the time they entered the simulator room, they were considered to be
operational flight crews and no assistance was given except that which is normally
provided by Air Traffic Control. Simulator sessions were videotaped from start to
finish with an audio tape backup. Researchers agree that videotaping markedly
increases the impact of LOFT training (Foushee & Helmreich, 1988; Hawkins,
1987; O’Hare & Roscoe, 1990). The researcher and simulator instructor took
extensive collaborative field notes to record significant events, to support the
videotape data, and to provide objective criteria for evaluation.
The LOFT Scenarios
The LOFT scenarios were based on Federal Aviation Regulations (FAR)
Part 91 and 135 operations requiring Commercial Pilot skills. Instrument flight
rules (IFR) were required throughout. No scenario forced students to choose a
solution that would violate regulations. Flights took place in the United States
inter-mountain Northwest, a unique area that requires extreme vigilance because of
52
mountainous terrain and intermittent radar coverage. To preclude rehearsal
errors, unfamiliar airports and routes were chosen. Flights were designed to last
45 minutes, including 15 minutes of normal workload followed by an occurrence
triggering a high-workload phase.
LOFT I was designed as a crew training flight because it was their first crew
experience. The scenario required normal crew interactions for instrument flight;
there were no critical occurrences. There were two similar legs allowing each
student the opportunity to fly as Captain. Two crews (Alex/Art and Carl/Cathy)
received self-analysis debriefing.
LOFT 2 was a communications flight concentrating on advocacy. The
scenario was a medical support flight that was requested to divert by an urgent
request for blood replacements. It required crew interaction and radio
communication to choose a divert airport that was above weather minimums and
that could deliver the required blood. Self-analysis debriefing was used for
Betty/Bob, Dan/Dave and Ed/Eric.
LOFT 3 was a decision-making flight focusing on prioritizing and analyzing
alternatives. The crew was on a long distance flight which encountered arrival
deadlines, departure delays, and unsuitable weather at the destination. It required
consideration of operational commitments, weather complications, and fuel
constraints. Self-analysis debriefing was used for Alex/Art and Carl/Cathy.
LOFT 4 was designed as a situational awareness flight emphasizing situation
monitoring and cross-checking. While transporting high-priority medical supplies,
53
minor mechanical difficulties progressively developed into a total loss of electrical
power. The scenario required attentive monitoring of the aircraft’s capabilities and
awareness of external factors: weather, operational requirements, navigation
capabilities, and alternatives. Communication with Air Traffic Control (ATC) and
radar services were lost about 30 minutes after takeoff; Self-analysis debriefing was
used for Betty/Bob, Dan/Dave, and Ed/Eric.
LOFT 5 was a team management flight highlighting workload assessment
and management. The crew was exposed to operations in a high-density (Class "B"
airspace) environment where the weather was unsuitable for their destination but
was above minimums for several nearby alternates. They lost communication with
Air Traffic Control requiring crew interaction and leadership skills to select a
course of action from a large number of alternatives. Because of the complex
airspace, marginal weather, and faulty radios, LOFT 5 became known as the
"LOFT from Hell!"
Procedures for Training and Evaluation
Individual Orientation to the Simulator. All of the participants had
previously flown the Frasca 142 simulator in various configurations. To assure that
they were entirely familiar with every display and control before any CRM training
or evaluation began, every person was individually instructed in the configuration
used throughout the study. An oral debrief was given and errors were corrected to
ensure 100% comprehension of the simulator displays and controls.
54
Evaluation of Flying Skills and Assignment to Crews. Before any crew
training or evaluation began, each student was given an Evaluation of Individual
Flying Skills. This evaluation was used to assign individuals to crews so that the
level of experience in each crew was balanced. They were assigned to perform 10
instrument flying tasks which were graded as "Pass" or "Fail." Their performance in
these tasks was used to match students of equal technical skill level.
Learning Procedures: Cooperative Learning. It was important to establish
that the full content domain was addressed and that appropriate knowledge about
the CRM skills (FAA, 1992) was imparted to each crew. The theoretical content
was mastered by a Cooperative Learning procedure called "Jigsaw" (Johnson,
Johnson, & Smith, 1991). Students individually read assigned material and then
shared information with their crew member by discussing case studies, analyzing
accident reports, and writing team response papers. The instructor acted as a
resource person and evaluated learning by observation and oral quizzing.
Practice Procedures: LOFT. After learning about the CRM skills, students
practiced them in the LOFT simulator flights where they were required to actively
use the learned CRM skills in an operational environment. LOFTs were flown in
real-time without interruption or assistance and were videotaped from start to
finish.
55
Debriefing Procedures: Conventional. Debriefing sessions were periods to
provide feedback to the crew on performance of CRM skills in the simulator. Two
distinct debriefing methods were used: conventional and self-analysis.
Conventional debriefing was NOT an active-learning strategy; feedback was
immediate, quantifiable, and objective, and the instructors provided the majority of
the input (Butler, 1993). Performance effectiveness after conventional debriefing
provided the baseline measurements for this study.
Debriefing Procedures: Self-Analysis. LOFT simulators were videotaped; a
verbatim transcript of the tape was prepared; and the transcription was. coded by
Communications Analysis. The alternating treatments design required two sessions
of self-analysis debriefing for each crew. When a crew was scheduled for selfanalysis debriefing, they did not debrief immediately after the simulator session,
but they waited until the transcription and communication analysis of the videotape
was complete, usually within two days. Guided by learning tools (the videotape, the
transcription, and the communication analysis), the crew objectively evaluated their
own performance. They were instructed to do a specific learning task (to highlight
the transcript whenever they used one of the CRM skills). They were challenged to
count the number of positive CRM skills that occurred in a half hour of simulator
time. Their attention was solidly focused on CRM skills, and they were encouraged
to avoid any discussion of fault or blame.
I
56
All self-analysis sessions were videotaped for data collection. Crews mostly
ignored the video recorder, but there were occasions when it noticeably detracted
from the openness of the self-analysis discussion. The instructor visited the
debriefing room intermittently to personally observe and to answer questions, but
he usually deferred to the self-analysis process.
These self-analysis debriefs constituted the "treatment" in this study and
were the keystone events of this research. A unique feature of this self-analysis
training was that the verbatim transcripts and the Communications Analysis
strengthened the crew’s examination of the videotapes and fostered more effective
self-analysis (Hawkins, 1987).
Data Collection Methods
The Cockpit Management Attitudes
Questionnaire fCMAOl
Aviator’s attitudes are typically measured before and after CRM training
with the CMAQ (Helmreich, Wilhelm, & Gregorich, 1988). Appendix A contains
the revised version of the CMAQ (Gregorich, Helmreich, & Wilhelm, 1990) used in
this study. On the CMAQ, students express the extent that they agree with a series
of statements on a five-point Likert-type scale where 5 signifies strong agreement
and I signifies strong disagreement with the statement (Gregorich & Wilhelm,
1993). The CMAQ measures three attitude clusters that are either conceptually or
empirically related to CRM (Gregorich, Helmreich, & Wilhelm, 1990):
57
1. Communication and Coordination (COMCOOR): eleven statements
that measure a pilot’s beliefs about appropriate degrees of interpersonal
awareness, communication, and coordination in cockpit operations.
2. Command Responsibility (COMRES): four statements which measure
a belief in the appropriateness of shared responsibilities of crew
members as opposed to the wholly dominant Captain.
3. Recognition of Stressor Effects (RSE): four statements that denote an
individual’s appreciation for the debilitating effects of situational
stressors on performance.
Besides being used in a pretest - posttest manner to assess training induced changes
in attitude, the CMAQ can further be used as an indirect measure of crew
performance. Helmreich, Foushee, Benson, & Russini (1986) established a linkage
between self-reported attitudes and independent evaluations of performance, thus
attitudes provide an approach to assessing effective crew resource management.
Validity of Attitude Measurements. Content validity involves establishing
that the items on the CMAQ representatively sample the domain of aviator’s
attitudes (Brown, 1983). Distinctions are drawn between personality traits which
are viewed to be relatively stable and attitudes which are considered to be receptive
to training. Helmreich (1984) found that there were non-significant correlations
between personality traits, measured by the Personal Attributes Questionnaire
(Helmreich, Spence & Holahan, 1979), and attitudes, measured by the CMAQ.
58
Criterion-related validity involves establishing an empirical relationship
between test scores and some external measure (Brown, 1983). In related research
with the same battery, few significant correlations were found between personality
and attitudes (Helmreich, Spence, & Holahan, 1979). The available evidence
supports the conclusion that cockpit management attitudes measured by the
CMAQ are relatively independent of personality attributes.
Construct validity is the degree to which a test measures the psychological
trait or characteristic it is designed to measure (Brown, 1983). In a factor analysis
using data provided by crew members from three commercial airlines, Gregorich,
Helmreich, and Wilhelm (1990) investigated the exogenous variables, organization
and history effects and established that the three clusters (COMCOOR, COMRES,
and RSE) were stable and exhibited strong correlations across samples. In a study
of six airlines, Helmreich and Wilhelm (1991) found a consistent gain in attitudes,
suggesting that the CMAQ has construct validity because crews do relate the
concepts being taught to specific attitudes.
The CMAQ has been revised so that all items would have predictive validity,
the degree to which the test can predict how an individual will perform in a future
situation (Helmreich, Wilhelm, & Gregorich, 1988). Using this revised version,
studies have typically demonstrated significant positive shifts in attitudes as a result
of training impact (Helmreich & Foushee, 1993).
Though the CMAQ may be a valid test for airline pilots, there is no data to
establish it as a valid test of the attitudes of undergraduate flight students. This is
v
59
evident when considering the individual items on the CMAQ which assume that the
pilots have prior experience in crew operations. In the pretest, particularly,
untrained college students can only imagine how they would function in a crew
operation.
Reliability of Attitude Measurements. The factor-analytic study of
Gregorich, Helmreich, and Wilhelm (1990) exhibited strong correlations across the
three samples demonstrating reliability of the CMAQ. They conclude that the
CMAQ is a reliable test because their data suggest the existence of three stable
factors that exhibit strong correlations across samples. Gregorich (1993) used a
sample of 1191 crew members from a major air carrier to compute reliability
coefficients for the three attitude clusters. The reliability coefficient is the
correlation between scores from two administrations of the test to the same sample
of people (Brown, 1983). For Communication and Coordination the pre-training
and post-training reliability coefficients were .63 and .71, respectively.
Corresponding reliability coefficients for Command Responsibility were .38 and .42
and for Recognition of Stressor Effects were .50 and .64. A consistent pattern of
intercorrelations established the internal reliability of each attitude dimension.
Validity and Reliability of the CMAO as a Measure of Performance. Aside
from attitude stability, validation of the effects of attitudes on performance is
meager (Gregorich, 1993). Two investigations using expert ratings of performance
in line operations have provided some validation that attitudes measured by the
X
60
CMAQ are predictors of crew performance (Helmreich, Foushee, Benson, &
Russini, 1986; Helmreich, Wilhelm, Gregorich, & Chidester, 1990). Because the
linkage between attitudes and performance is less than perfect, the CMAQ should
be used with other performance data in a multiple measures design to measure
outcomes or to determine whether training programs such as CRM are effective
(Helmreich & Foushee, 1993).
CMAO Methodology. The CMAQ was completed by each crew member
individually as a pretest-posttest evaluation of the entire program. It was also
completed by each crew (scored by consensus) after each LOFT session as a
measure of the effectiveness of the debriefing strategy employed (conventional or
self-analysis). Though the CMAQ is designed as an individual instrument,
consensus scoring of one form by two crew members was used in order that the
attitude /performance relationship could be based on an aircrew score (Simon,
Pawlik, & Bronkhorst, 1991).
Individual Background Data. Section II of the CMAQ (Background
Information) was designed for airline pilots to document their airline and military
experience. The questions were not appropriate for undergraduate students, so a
modified questionnaire was developed to collect background information on the
students and to document their pilot experience, education, and exposure to CRM,
(see CMAQ Personal Questionnaire, Appendix B).
61
The LINE/LOS Checklist
The LINE/LOS Checklist (see Appendix C) is an instrument for evaluating
crew performance of CRM skills on the line and in Line-Oriented Simulations
(Helmreich, Wilhelm, Kello et al., 1991). The LINE/LOS Checklist is used by
instructors in LOFT and by other line evaluators to evaluate the performance of
full crews rather than the skills or behavior of a particular crew member. The
instrument taps multiple dimensions of performance and utilizes interval scales,
providing a greater range of variation than binary, pass-fail ratings (Helmreich &
Wilhelm, 1987).
The LINE/LOS Checklist consists of eight Crew Effectiveness Markers and
two Global Ratings. The Crew Effectiveness Markers were factored into three
conceptual groupings that are indicators of crew performance (Helmreich &
Wilhelm, 1991; FAA, 1993):
GROUP I: Communications Processes/Decision Behavior.
GROUP 2: Team Building and Maintenance.
GROUP 3: Workload Management and Situational Awareness.
The two global ratings are:
GLOBAL RATING I: Overall Technical Proficiency.
GLOBAL RATING 2: Overall Crew Effectiveness.
Validity of the LINE/LOS Checklist. Under a grant from the National
Aeronautics and Space Administration (NASA), Dr. Robert Helmreich at the
62
University of Texas (UT) compiled a significantly large database on the
effectiveness of CRM training. Through this research, he developed the behavioral
markers for the NASA/UT LINE/LOS Checklist to be used in the evaluation of
CRM skills of cockpit crews. Through accurate and systematic data collection that
included more than 15,000 completed LINE/LOS Checklists, the NASA/UT Crew
Performance Project established the LINE/LOS Checklist to be a valid measure of
effectiveness of crew performance (Helmreich & Wilhelm, 1991). Current CRM
training programs are built on the principles developed in the NASA/UT studies.
The Federal Aviation Administration has issued an advisory circular (FAA, 1993)
which specifies the three clusters of the LINE/LOS Checklist as the basic CRM
skills that should be taught and evaluated in each CRM program of instruction
(Driskell & Adams, 1992)
Reliability of the LINE/LOS Checklist. To establish reliability of the
LINE/LOS Checklist, the NASA/UT Crew Performance Project used multiple
observers to rate a large number of crews and compared results (Helmreich &
Wilhelm, 1991). There was general consensus on the ends of the rating scales
(I and 5 ratings) with some variability in the middle of the rating range.
Differences in trained observer ratings of the same crew seldom exceeded I point
on a 5-point scale. The research data show that the use of behavioral markers to
evaluate CRM performance greatly increases the reliability of the observations
(Gregorich & Wilhelm, 1993).
Ji
63
LINE/LOS Checklist Methodology. During the LOFT exercise,
comprehensive field notes were independently taken by two instructors. The
LINE/LOS Checklist was scored immediately following each LOFT by the
instructors using the Evaluator/LOS Instructor Reference Manual (Helmreich,
Wilhelm, Kello et al., 1991) as a guide. Each behavior marker was deliberated
until a consensus score could be agreed upon by both instructors. Conscientious
efforts were made to evaluate the crew as undergraduate flight students and not as
airline pilots. Deliberations included the performance of technical skills, but each
behavioral marker was considered on its own merit independent of evaluations in
other areas. The LINE/LOS Checklist was used as a measure of crew effectiveness
for this study.
The CRM Survey
Typically, efforts to evaluate training programs have included a student’s
evaluation of the curricula and instruction. The CRM Survey (see Appendix D)
was designed as a team consensus instrument to measure crew opinions about their
LOFT experience. Since simulator training is expensive and excellence in training
is the objective, evaluating the training should include input from the viewpoint of
the crew member (Wilhelm, 1991). The CRM Survey includes a section where
crew members can contribute feedback and information about what they found
most useful about the training and what they feel might be done to increase the
I
64
usefulness of the training. Responses on the CRM Survey have been factored into
six categories (Wilhelm, 1991) to obtain student’s views on:
1. Value of LOFT as a training technique
2. Quality of the LOFT scenario
3. Workload imposed by the LOFT scenario
4. LOFT instructor ratings
5. Self-evaluation of the crew’s overall performance
6. Self-report on their use of CRM skills
Validity of the CRM Survey. The self-evaluations of training provided by
the CRM Survey cannot be considered valid evidence for training effectiveness
because of low correlations between self-reports and behavior (Helmreich &
Wilhelm, 1987). The enthusiastic endorsement of a training program by
participants does not mean that their performance will change in any significant
way. Its validity is also suspect because there is a significant degree of variation in
acceptance of CRM training found in self-report evaluations (Helmreich &
Wilhelm, 1989). Whereas the CRM survey may not be a valid measure of the
effectiveness of self-analysis, it has strong construct validity as a measure of the
crew’s acceptance of self-analysis training and their evaluation of its relevance and
usefulness (Helmreich Sc Foushee, 1993). Though "buy-ons" and measures of
student motivation are not sufficient evidence of a program’s success, it is unlikely
X
65
that behavior changes will occur without these expressions of motivation and
agreement (Helmreich & Wilhelm, 1987).
Reliability of the CRM Survey. The CRM Survey is a very reliable measure
of training effectiveness at the extreme ends of the scale, both negative and
positive. If the participants reported that a program had little value, it was
reasonable to conclude that it had not achieved its goals (Helmreich & Wilhelm,
1989). Also, good instructor ratings correlated highly with good scenario ratings,
with good CRM behaviors, and with overall ratings of LOFT as a training
technique (Wilhelm, 1991).
The most important aspect of the CRM Survey is that it provides input data
directly from the viewpoint of the program participants. It increases the reliability
of the multiple measures assessment because it may provide converging data that
will support other data sources (Gregorich & Wilhelm, 1993).
Methodology of the CRM Survey. The crew completed the CRM Survey as
soon as conventional or self-analysis debriefing was complete. They were given the
survey together in the debriefing room and were asked to complete it together
before leaving. They were instructed on the concept of consensus so that the survey
would truly be a crew response rather than the opinion of a singular individual.
They were allowed to be absolutely straightforward and were assured that the
results would have no effect on their grades.
66
Communications Analysis
Communication has dozens of functions, but this research concentrates only
on communication as a means to effectively accomplish a task. Evaluation of crew
communications can be used to determine how crew communications relate to
performance. Listening to the communications during a flight provides indicators
whether the crew is performing functionally or if problems are occurring (Kanki &
Palmer, 1993). Past research has shown that cockpit communications patterns are
related to flight crew performance (Foushee, Lauber, Baetge, & Acomb, 1986;
Foushee & Manos, 1981).
/
The procedure used to analyze within-cockpit communications was adapted
from the Foushee and Manos (1981) procedure that coded each statement by the
Captain or First Officer into one of 20 categories of communication (see
Appendix E). Although it would have been desirable to analyze all 20 categories,
only four that have been empirically related to performance were used as measures
of crew effectiveness:
1. Total Communications: The sum of all types of verbal communication
by the Captain and First Officer, including radio communications.
2.
Commands by the Captain: Specific assignment of responsibility by the
Captain to the First Officer.
3. Acknowledgements by the First Officer: Verbal recognition of the
Captain’s command, inquiry, or observation.
67
4. Observations by Both Crew Members: Remarks aimed at orienting the
other pilot to some aspect of flight status.
Validity of Communications Analysis. Prior studies (Foushee & Manos,
1981; Kanki, Lozito, & Foushee, 1987) have shown communication patterns to be
linked to quality of crew performance. Foushee and Manos (1981) found that more
effective crew performance was associated with more task-related talk, more
commands, and more acknowledgements. Kanki, Lozito, and Foushee (1987)
found that more effective crews made greater use of command-acknowledgement
sequences initiated by Captains while less effective crews used more questionresponse sequences initiated by First Officers.
Several studies link the performance of crews with the four patterns of
communication used in this investigation. Commands by the Captain appear to
have a coordinating effect on crew performance because of their strong influence
on subordinate crew member’s actions (Foushee, Lauber, Baetge, & Acomb, 1986;
Foushee & Manos, 1981; Kanki, Lozito, & Foushee, 1987). Acknowledgements by
the First Officer to the communications of the Captain were associated with fewer
crew performance errors and with a stronger personal interaction process
(Foushee, Lauber, Baetge, & Acomb, 1986; Foushee & Manos, 1981; Kanki,
Lozito, & Foushee, 1987). Observations by crew members, particularly by the First
Officer, were significantly related to effective crew performance (Foushee, Lauber,
Baetge, & Acomb, 1986; Foushee & Manos, 1981; Kanki, Lozito, & Foushee, 1987).
68
Increased total communications are considered an indicator of crew effectiveness
because there is a tendency for crews who do not perform well to communicate less
(Foushee, Lauber, Baetge, & Acomb, 1986; Foushee & Manos, 1981).
Reliability of Communications Analysis. Using a procedure adapted from
Foushee and Manos (1981), cockpit communications for a period of exactly
30 minutes were transcribed, and each statement was coded into one of
20 categories. Two coders, working independently, coded all of the tapes. A pointby point comparison of their coding established an interrater reliability of 81%. To
further refine reliability, the coders jointly reviewed each differently-coded
statement, debating their position until a consensus code was reached.
Communications Analysis Methodology. LOFT flights were designed to last
at least 45 minutes with about 15 minutes of normal workload followed by a highworkload phase. In order to make comparisons between LOFTs, communications
for each LOFT were analyzed for a timed 30-minute segment that began when a
planned event triggered the high-workload phase of the flight. The variable, Total
Communication, was the numerical sum of all of the Captain’s communications and
the First Officer’s communications that were spoken during that exact period. Each
crew member’s total communication was further sorted into the 20 categories
designed by Foushee and Manos (1981). The three categories which were
analyzed in this study were Commands (by the Captain), Acknowledgements (by
the First Officer), and Observations (by both crew members). These three
69
variables were analyzed as a percentage of the total communication rather than as
individual frequencies.
Qualitative Data: Lessons-Learned
Learning criteria in LOFT has received almost no attention; research has
predominantly focused on crew behaviors (Gregorich & Wilhelm, 1993). In an
effort to measure what was learned by individual crews, open-ended questions
asked students what lessons they had discovered during the learning, practice, and
debriefing sessions. The Lessons-Learned (see Appendix F) was a self-evaluation
generated by each crew after their debriefing session. Students reflected on the
entire experience, listed their lessons-learned relating them to a specific CRM skill,
and specified the source of learning for each lesson. The lessons-learned were
classified into five categories that defined the cockpit behaviors that resulted from
the training program:
1. Communication
2. Decision Making
3. Situational Awareness
4. Team Building
5. Technical
The sources of learning were sorted into six categories that defined where
the crew derived the lesson from:
70
1. Debrief
2. Instructor
3. LOFT
4. Preflight
5. Readings
6. Self-analysis
The Lessons-Learned documents are sources of qualitative data to evaluate the
student’s learning experience. The task was not criteria-based, rather it was a
formative learning task. Crews were encouraged to concentrate their lessonsleamed on CRM skills rather than the technical and proficiency aspects of the
flight. The lessons-learned were included in the study as descriptive evidence of the
effectiveness of the CRM training program.
Supplementary or Supportive Data
Excerpts from their transcripts illustrated a finding or demonstrated the
level of sophistication that a crew achieved in integrating CRM principles into
cockpit dialogue. The quality of the communications is a good indication of the
extent that the crew has internalized the CRM skills.
The researcher and the simulator instructor took extensive field notes
throughout the study especially during the LOFT practice sessions and the selfanalysis debriefing sessions. Where appropriate, these notes were included in the
study to clarify or document a finding.
71
CHAPTER 4
ANALYSIS OF DATA
Alex/Art
The Alex/Art crew was well matched in technical skills, but mismatched in
experience. Art was a low-time private pilot while Alex was an active flight
instructor, Alex struggled with role definition throughout the training program
because he thought of himself as a flight instructor, but the course expected him to
perform in a team, sometimes even as the junior crew member. Whenever Alex
was First Officer, there were occasions when he would instruct rather than assist,
for example, in LOFT 5:
PO:
CAP:
PO:
CAP:
PO:
More flaps?
Oh! Yup! 40.
You need more than that, (laugh)
(laugh) We need ... just dive!
Be ready to bring power in at the bottom.
When Alex was Captain, the struggle with role definition was different. Art
had a tendency to regard himself as subordinate to Alex and deferred to Alex’s
decisions even if he differed. They discussed these relationships in their selfanalysis sessions and addressed them in their Lessons-Learned:
72
Learned that just because of different experience levels, that one
pilot should not assume the more experienced does not make
mistakes. (LOFT I)
Captains must delegate authority, assign tasks, during emergency and
normal operations. (LOFT 4)
Both crew members became aware of role definition challenges in their selfanalysis sessions and role issues were an important part of their self-analysis
deliberations. It was apparent from their crew interactions in subsequent LOFTs
that they worked hard to resolve role differences. Consequently, they achieved
many positive results. Their final set of Lessons-Learned after LOFT 5 revealed
their insights on role issues:
Crew members need to work together on decisions.
Someone needs to take control and be a leader.
Delegate duties of flying and option review.
The Captain needs to take more control in emergency situations.
Table 3 shows substantial gains in both self-analysis sessions.
In the crew CMAQ, none of the attitude scores showed differences, except
Recognition of Stressor Effects. Thus, little insight into self analysis was gained
from their CMAQ. Lack of any variation indicates that the crew probably
responded mechanically with remembered scores. One exception was a dramatic
gain in Recognition of Stressor Effects following LOFT 5, a particularly difficult
LOFT for them. They apparently departed from their methodical scoring of the
73
Table 3. Alex/Art Evaluation Results.
1st
A le x /A r t
2nd
Self-A nalysis
Self-A nalysis
LO FT 2
LOFT 4
A le x /A r t
A le x /A r t
C om m unication and C oordination
nc
nc
C om m and R esponsibility
nc
nc
C A P T A IN /F IR S T O FF IC E R
CM AQ
R ecogn ition o f Stressor E ffects
nc
L IN E /L O S C H EC K LIST
C om m u n ication s/D ecision Behavior
+
+
T eam Building and M aintenance
W orkload M gm t/Situational A w areness
nc
nc
O verall T echnical Proficiency
+
++
O verall Crew Effectiveness
nc
++
CRM SU R V EY
Training V alu e o f LO FT
nc
Scenario Q uality
+++
+
W orkload Im posed
++
++
Instructor R ating
++
Self-E valuation o f Perform ance
+
Self-R eport on C R M Skills
nc
+
C O M M U N IC A T IO N S A N A L Y SIS
T otal C om m unications
+ + +
+
C om m ands (by Capt)
+
+ + +
A cknow ledgem ents (by P O )
+
+ +
O bservations (C apt & F O )
+ +
+
Key: Standard deviations since the last observation: —
= < -2, - - = - l< - 2 ,
- = 0 < T , nc = no change, + = 0 > 1 , + + = I > 2 , + + + = 2 > 3
74
CMAQ to express a notable change in their understanding of flying abilities during
stressful flight situations.
The LINE/LOS Checklist showed moderate gains for the crew in the first
four LOFTs. Overall Technical Proficiency showed strongest gains, consistent with
the crew’s high technical ability and their expressed concern for "looking good."
They progressed steadily through both self-analysis sessions up to LOFT 4. One
effectiveness group, Communication Process and Decision Behavior, decreased in
LOFT 4 because they had decided, possibly from self-analysis, that limited
communications were the key to effectiveness. In LOFT 4, there were long periods
of non-communication when the crew should have been sharing information,
assimilating circumstances, and deciding between alternatives.
In LOFT 5 the crew made some bad decisions which threatened the safety
of the flight. The consequence was decreased performance in every category of the
LINE/LOS Checklist. These declines were supported by the increased
Recognition of Stressor Effects in the CMAQ and by the crew’s CRM Survey selfevaluation scores. Complacency did not appear to be the key to their setback,
because this crew was involved, dedicated, and expressed a preflight ambition to
"be the best." Self-analysis was not a factor in their declines in LOFT 5. Also, Art
was Captain in both LOFT 3, where they were very successful, and in LOFT 5,
where they had difficulties, indicating that Captain was not the determining factor.
LOFT 5 was a complex scenario, but it should have been within the ability level of
this crew. Two plausible reasons for their decline were that the scenario was too
75
complex or that the crew’s continuing struggle with role definition interfered with
their ability to improve in LOFT 5.
The CRM Survey showed that self-analysis increased their workload but
resulted in scenarios of higher quality. Self-analysis instructor ratings decreased in
the second session, indicating that more instructor inputs were desired. A
suggestion for improvement read:
Possibly more actual teaching of CRM skills rather than self-taught,
trial-and-error teaching. (Survey, LOFT 2)
It appears that self-analysis challenged this crew but.influenced their technical skills
more than CRM behavior.
Their Communications Analysis showed gains in all four categories in both
sessions the strongest evidence that self-analysis motivated this crew. The data
suggest that the crew focused on communications in their sessions, though they
reported concentrating on team building. The result supports the CRM principle
that communications is instrumental toward achieving other CRM skills (Kanki &
Palmer, 1993).
In their Lessons-Learned, Alex/Art focused on Team Building, though the
subject was not formally taught until LOFT 5. They recorded that their main
learning sources were LOFT, readings, and self-analysis. It appears that selfanalysis challenged this crew and made an important contribution to their learning.
The Lessons-Learned that Alex/Art attributed to self-analysis were:
76
Lesson I: Communication
Understand or question - departure procedure from Livingston when
First Officer asked the Captain if he was using Navigation # 2.
Lesson 2: Decision Making
Learned to analyze the questions and to know what our answer
means.
Lesson 3: Situational Awareness
Briefings are important for each phase of the flight —failing to
discuss departure procedures from Livingston.
Lesson 4: Situational Awareness
Knowledge of job description - both doing jobs of single pilots, not
allowing specific job applications.
Lesson 5: Situational Awareness
There is a time and place for each action, a particular action should
not interrupt a previous action.
Lesson 6: Team Building and Maintenance
Assumption is bad - assuming that the Captain always knows what’s
going on.
Lesson 7: Team Building and Maintenance
Need to learn how to do own job, double check other without doing
his job. Flight flown as two IFR pilots in same airplane.
Lesson 8: Team Building and Maintenance
Need to not assume both people are on the same track, that one
knows what the other one is thinking.
Lesson 9: Team Building and Maintenance
Learned that, just because of different experience levels, one pilot
should not assume the more experienced does not make mistakes.
Lesson 10: Team Building and Maintenance
Alex was flight instructor again (sometimes).
Lesson 11: Team Building and Maintenance
Art was not making decisions (sometimes).
77
These Lessons-Learned, particularly those that addressed Team Building
and Maintenance, revealed that self-analysis made the crew fully aware of the role
issues and that they made earnest efforts to manage them. It appears that
discovery-learning in self-analysis made these lessons more tolerable than if they
came from an instructor. Self-analysis may be more effective for learning about
personality matters or about private concerns that involve individual differences.
Betty/Bob
The Betty/Bob crew had great difficulty accepting the training environment.
They were very aware that they were in a research project, though they did not
know the research question or how the data would be analyzed. It was necessary to
remind them, especially Bob, to disregard the research and to concentrate on
learning. The crew also had difficulty accepting the realism of the simulator.
When something occurred, they would sometimes conclude that the simulator
malfunctioned or that the instructors were "trying to pull something." They did not
seem capable of "suspending reality" and treating the simulator sessions as line
flights. Instead they focused on training aspects which often do not occur in the
LOFT environment:
When we make a mistake, the LOFT session should terminate,
because at that point or soon after, CRM is kind of ignored, and we
find ourselves thinking back to our own training and skill. (CRM
Survey, LOFT 2)
78
Crew dynamics may have influenced this crew’s learning ability more than
methods or strategy. Bob was confident and capable and tended to be dominant in
most situations. He was not overtly sexist, but occasionally his manner or speech
inferred that Betty was a "girl" rather than a crew member. Betty was equally
capable but was more acquiescent. She responded professionally to him, but her
body language and responses exhibited sensitivity. Evidently, their crew dynamics
were more manifest to the instructors than to the crew, and it was apparently not an
important topic in their self-analysis sessions. Table 4 shows the changes in
effectiveness measures for Betty/Bob.
In the crew CMAQ, only losses in attitude corresponded with self-analysis.
Declines in attitude after self-analysis sessions were contrary to observations and
were not confirmed by findings from other instruments in the first self-analysis
session (LOFT 3). This crew appeared to be very enthusiastic about self-analysis
and claimed to gain significantly from it. The crew CMAQ more closely resembles
Bob’s individual CMAQ indicating that his attitudes probably had more leverage in
the consensus process.
In LOFT 2 there was a noticeable change in crew dynamics manifested in
body language and noted by instructor field notes:
0818 Betty is just flying - where is the thinking?
0828 Bob, you just agreed she could go to 7,000 feet without
thinking about it or discussing it.
79
Table 4. Betty/Bob Evaluation Results.
B e tty /B o b
C A P T A IN /F IR S T O FF IC E R
1st
2nd
Self-A nalysis
Self-A nalysis
LOFT 3
LOFT 5
B o b /B e tty
B e tty /B o b
CM AQ
C om m unication and Coordination
C om m and R esponsibility
+
R ecogn ition o f Stressor E ffects
L IN E /L O S C H EC K LIST
C om m u n ication s/D ecision Behavior
+++
M ■
T eam B uilding and M aintenance
+++
ma ■
W orkload M gm t/Situational A w areness
+++
_
O verall Technical Proficiency
++
_
O verall Crew Effectiveness
+++
_
_
CRM SURVEY
Training V alu e o f L O FT
nc
Scenario Quality
++
W orkload Im posed
nc
■ •
■
nc
Instructor R ating
** e,
Self-E valuation o f Perform ance
++
Self-R eport on C R M Skills
+++
_
C O M M U N IC A T IO N S A N A L Y SIS
+ + +
T otal C om m unications
C om m ands (by Capt) .
+ + +
■ —
+
A cknow ledgem ents (by F O )
O bservations (C apt & F O )
+ +
+ + +
Key: Standard deviations since the last observation: —
= < -2 , - - = - l < - 2 ,
- = 0 < - l , nc = no change, + = 0 > 1 , + + = 1 > 2 , + + + = 2 > 3
80
0830 What are you doing, Betty? Share the "big picture."
0839 Hello, Bob, are you there? Would you like a martini and a
more comfortable seat in First Class?
In the beginning, impaired crew dynamics were evident to the instructors but
did not show in the effectiveness measures. Every item on the Line/LOS ChecMist
and three CRM Survey items showed strong gains after the first self-analysis
session. It appeared that crew dynamics was not a concern. However, in LOFT 4,
impaired crew dynamics again were noted by both instructors and were manifested
in the crew’s Lessons-Learned:
Do not assume that your partner knows what you mean.
Recognize problems. Don’t fixate on them —fly the plane.
Share decision making. Don’t let Captain override the crew.
Their poor performance after the second self-analysis session did not relate to
Captain, self-analysis, or other responses on the CRM Survey. Their decline may
be attributable to the complexity of the scenario, but inferior crew dynamics was
still detectable in their comments:
The need for CRM sMlls was not practiced. We used a lot of
non-verbal communication and that was a mistake. (CRM Survey)
First Officer learned to wait for Captain decisions or make verbal
suggestions before taMng action. (Lessons-Learned)
In the second self-analysis session, the Communications Analysis shows
gains, contrary to the other measures of effectiveness. Gains in communications
skills suggest that the crew may have been processing their crew dynamics issue and
81
that self-analysis was starting to work. Their poor performance in LOFT 5 could
have been more attributable to the difficulty level of the scenario than to
interpersonal disparities. However, reduction in Commands by Captain Betty
correlate with other results, signifying that she may not have been completely in
charge of the situation.
In their Lessons-Learned, Betty/Bob concentrated on Communication and
Situational Awareness. None of their self-analysis Lessons-Learned addressed
Team Building and Maintenance and only three of their Lessons-Learned from
conventional debriefing concerned teamwork issues:
Coordinate better crew coordination during emergencies and missed
approach.
Learned to not take the simulator for granted - that it is more like an
actual airplane.
Share decision making. Don’t let Captain override the crew.
The crew primarily learned from flying the LOFT with only five of their 50 lessonslearned coming from self-analysis. The lessons-learned that Betty/Bob attributed
to self-analysis were:
Lesson I: Communication
Use complete sentences so that we understand each other.
Lesson 2: Communication
Limit talking to ourselves.
Lesson 3: Communication
Use complete sentences, explaining what you mean (hot engine or
generator).
82
Lesson 4: Situational Awareness
Scan even more often - should have picked up failure earlier.
Lesson 5: Situational Awareness
Look carefully at alternate approaches to determine what’s available.
Their self-analysis Lessons-Learned indicate that crew dynamics was not
discussed in their self-analysis sessions. These data suggest that the crew was
presumably unaware of the dynamics issues. In this regard, it is reasonable to
conclude that self-analysis was not effective in illuminating the personality
characteristics and human relations skills that influenced the performance of this
crew.
Carl/Cathy
The Carl/Cathy crew was equally matched in skills and were compatible in
personality, performing well as a male/female crew. They were preoccupied with
the technical aspects of flying which limited their absorption of CRM skills. They
struggled with the basics of crew coordination. They had difficulty deciding who
was to talk on radios or how to do checklists. They confused preflight planning
(technical) with preflight briefing (CRM), giving the impression that they were
unprepared for the flights. Table 5 shows the changes in effectiveness for
Carl/Cathy.
The crew CMAQ does not show that self-analysis was effective for this crew.
They continually struggled with Command Responsibility, experiencing difficulty
deciding the basics of crew flight, such as who was to talk on radios or how to do
83
Table 5. Carl/Cathy Evaluation Results.
1st
C arl/C athy
C A P T A IN /F IR S T O FF IC E R
2nd
Self-A nalysis
Self-A nalysis
LO FT 2
LOFT 4
C athy/C arl
C athy/C arl
CM AQ
C om m unication and C oordination
+
C om m and R esponsibility
R ecogn ition o f Stressor E ffects
L IN E /L O S C H EC K LIST
C om m unications /D e c isio n Behavior
+ ++
T eam B uilding and M aintenance
+++
W orkload M gm t/Situational A w areness
++ +
Overall Technical Proficiency
+ ++
O verall Crew E ffectiveness
++
CRM SU R V EY
Training V alu e o f LO FT
nc
Scenario Quality
++
+ +
W orkload Im posed
++
+
Instructor R ating
nc
Self-E valuation o f P erform ance
S elf-R eport bn C R M Skills
++
nc
C O M M U N IC A T IO N S A N A L Y SIS
T otal C om m unications
+ + +
+ + +
C om m ands (by Capt)
A cknow ledgem ents (by F O )
nc
O bservations (Capt & F O )
+
Key: Standard deviations since the last observation: —
-
= < -2 , - - = - ! < - 2 ,
- = 0 < - l , nc = no change, + = 0 > 1 , + + = 1 > 2 , + + + = 2 > 3
I
checklists. Their Recognition of Stressor Effects steadily decreased because they
were preoccupied with the technical aspects of flying and had little time to reflect
on their CRM skills.
The LINE/LOS Checklist showed negative results for the first self-analysis
session but strong gains were registered for the second one. In the first self-analysis
session the crew was concentrating on technical errors, not CRM. Instructor field
notes reflect this:
0806 Two separate people in the cockpit, not a crew yet.
0856 Cathy, you’re the Captain, but you’re asking Carl questions as if
he had all the answers.
The crew was making efforts but did not know what to do. Self-analysis did not
provide any answers; they needed a role model. To assist them, the instructor
decided to deviate from self-analysis procedures because this was primarily a
training activity and the learning needs of the students took precedence over the
a priori research design. Thus, their second self-analysis session was closely
monitored by the instructor who facilitated the self-analysis, keeping them away
from technical discussions. It became a guided self-analysis session which
manifested strong gains in the LINE/LOS Checklist scores. The gains did not
continue into LOFT 5 which was apparently an unmanageable challenge for both
their technical and CRM skills.
In the CRM Survey, the crew recorded an increase in CRM skills in the first
self-analysis session, contrary to the results of the LINE/LOS Checklist. Gains they
85
perceived were probably in technical performance confirming that the crew was
unable to discern between technical and CRM skills at. that point. In the second
self-analysis session the crew reported that they sensed the workload had increased
and instructor helpfulness decreased during the guided self-analysis session. Their
low Self-Evaluation of Performance and Self-Report on CRM Skills did not agree
with the results of the LINE/LOS Checklist which showed strong gains for guided
self-analysis.
In the first self-analysis session Communications Analysis showed a notable
increase in total communications, usually an indication of increased performance.
Foushee and Manos (1981) warn that more communication among flight crew
members does not necessarily translate into better performance. In the second selfanalysis session, increased commands by the Captain does support the results of the
LINE/LOS Checklist, evidence that guided self-analysis was an effective learning
method.
Crews were briefed that Lessons-Learned should not address technical skills,
but nevertheless many of Carl/Cathy’s comments addressed technical lessons:
Should trim out the aircraft; lack of trim leads to unfavorable flight
attitudes. (LOFT I)
Learned to be aware of I minute legs in the holding pattern.
(LOFT 2)
Don’t use out-of-date charts. (LOFT 4)
It was evident that the crew did not distinguish between technical and CRM skills.
The crew reported that their Lessons-Learned focused on Situational Awareness.
86
They learned mostly from instructor debriefings, LOFT, and self-analysis. The
Lessons-Learned that Carl/Cathy credited to self-analysis were:
Lesson I: Communication
Learned what advocacy was, Carl explained to Cathy that we could
descend after intercepting the localizer.
Lesson 2: Communication
Learned when to declare minimum fuel.
Lesson 3: Communication
Using more standard phraseology. Leave the crap outside.
Lesson 4: Decision Making
If not sure, don’t be afraid to ask questions.
Lesson 5: Decision Making
Learned to take into consideration all available airports in the area.
Lesson 6: Situational Awareness
Learned to be vigilant. Being on top of things such as engine
instrument checking.
Lesson 7: Situational Awareness
Learned not to focus on one problem.
Lesson 8: Situational Awareness
Double check approaches: 9,000 feet instead of 10,000 feet.
Lesson 9: Situational Awareness
In heavy workload situations, First Officer should talk on radios.
Lesson 10: Team Building and Maintenance
Both crew members should copy the clearances.
Lesson 11: Technical
Carl learned to scan better because of his erratic attitude situations.
Lesson 12: Technical
Learned to figure out holding patterns before entering one.
87
Lesson 13: Technical
To slow down the situation, for instance, slowing the plane.
Numerous Lessons-Learned documented by Carl/Cathy are of a technical
nature even when they pertain to a CRM skill. This finding suggests that CRM
skills are higher-order skills that can only be mastered when the crew has acquired
a high level of technical expertise in individual flying skills.
Dan/Dave
The Dan/Dave crew was the only crew mismatched in skills; Dan’s skills
were above-average, and Dave’s skills were below-average. This combination,
especially when Dan was Captain, produced the best overall gains in effectiveness.
They worked hard in self-analysis sessions and appeared to make excellent
discoveries. They openly expressed their dislike for self-analysis and preferred to
integrate the instructor into the discussion. Table 6 shows the changes in
effectiveness measures for Dan/Dave.
The crew CMAQ shows very little evidence that self-analysis was an
effective learning environment for this crew. The gains in Recognition of Stressor
Effects that occurred in LOFT 5 are probably more characteristic of the difficulty
of that scenario than of self-analysis.
The LINE/LOS Checklist results made a persuasive case for self-analysis.
Gains were achieved in all global scores except in Overall Technical Proficiency in
both self-analysis sessions regardless of who was Captain. The crew was definitely
88
Table 6. Dan/Dave Evaluation Results.
D a n /D a v e
1st
2nd
Self-Analysis
Self-A nalysis
LOFT 3
LOFT 5
D a v e /D a n
D a n /D a v e
C om m unication and Coordination
nc
■
C om m and R esponsibility
nc
nc
R ecognition o f Stressor Effects
nc
++
C om m u n ication s/D ecision Behavior
+
++
T eam B uilding and M aintenance
++
++
W orkload M gm t/Situational A w areness
++
+ +
C A P T A IN /F IR S T O FF IC E R
CM AQ
L IN E /L O S C H EC K LIST
++
O verall T echnical Proficiency
+
+++
Training V alu e o f LO FT
nc
nc
Scenario Quality
nc
nc
W orkload Im posed
++
_
_
_
_
O verall Crew E ffectiveness
CRM SU R V E Y
Instructor R ating
Self-E valuation o f Perform ance
++
++
Self-R eport on C R M Skills
+ +
■■—
C O M M U N IC A T IO N S A N A L Y SIS
_
——
C om m ands (by Capt)
__
_
A cknow ledgem ents (by F O )
+ +
_
O bservations (Capt & F O )
+ + +
+
T otal C om m unications
Key: Standard deviations since the last observation: —
- = 0 < -I,
= .< -2 , - - =
n c = n o change, + = 0 > 1 , + + = 1 > 2 , + + + = 2 > 3
-l< -2,
89
more technically efficient when Dan, the more experienced pilot, was Captain
because they concentrated more on CRM skills. When Dave was Captain, they
were more susceptible to technical problems, but Dan usually advocated his concerns
strongly enough to keep the crew from making poor decisions. Unlike the other
four crews, they were not overwhelmed in LOFT 5 and established considerable
gains in every crew effectiveness marker. From another viewpoint, however, the
LINE/LOS Checklist basically stands alone in support of self-analysis in the second
session. It is possible that the evaluators were unknowingly inflating the evaluation
of Dan/Dave because their performance in LOFT 5 was superior to the other
crews. Conceivably, Dan/Dave did well in the LINE/LOS Checklist only in
comparison to the others, but the remaining effectiveness measurements indicated
that their gains were not substantiated. Dan/Dave were the last crew to fly LOFT 5.
The crew’s preference for. the conventional debriefing method is reflected in
their CRM Survey where the Instructor Rating dropped for both self-analysis
sessions. However, the crew’s Self-Evaluation of Performance for both self-analysis
sessions and their Self-Report on CRM Skills for session one showed positive
effects for self-analysis despite their dislike of the method.
Two variables in the Communications Analysis supported gains during the
first session of self-analysis. They seemed to work particularly hard on increasing
observations, keeping each other well informed. Communications Analysis did not
give evidence of the gains for self-analysis that were shown in the LINE/LOS
Checklist in the second session of self-analysis.
90
In the Lessons-Learned, this crew documented a large variety of lessonslearned in Communications, Decision Making, Situational Awareness, and Team
Building. Their learning sources were predominately LOFT and self-analysis,
indicating that they felt self-analysis was an effective learning agent for them. The
lessons-learned that Dan/Dave attributed to self-analysis were:
Lesson I: Communication
Inquiry - making sure that both people understand when the other
person speaks.
Lesson 2: Communication
Advocacy - when "Missoula" was said instead of "Butte," while in
communication with Salt Lake Center.
Lesson 3: Communication
Assertiveness - turned around right after losing second generator.
Lesson 4: Communication
Checklist - went to the flight manual to verify generator emergency
procedures.
Lesson 5: Communication
Advocacy - both tried to state our ideas on where to go.
Lesson 6: Decision Making
Decision making is important.
Lesson?: Decision Making
Problem solving - reading approach plates correctly.
Lesson 8: Decision Making
Problem solving/assessment - where, when, how to get the blood.
Lesson 9: Decision Making
Problem solving, workload management, dealt with cylinder head
temperature problems.
91
Lesson 10: Decision Making
Problem assessment. Saw left generator fail and worked together to
reduce overload on right generator.
Lesson 11: Decision Making
Tried to use other resources to solve another problem - knob to turn
off the ADF.
Lesson 12: Decision Making
Exploring possibilities about where to go in our situation.
Lesson 13: Situational Awareness
Avoiding fixation - we kept trying to make the weather better, when
we knew we couldn’t get into Butte.
Lesson 14: Situational Awareness
Stress management - laughing at weather that we knew we couldn’t
get into.
Lesson 15: Situational Awareness
Double check - repeating pertinent information.
Lesson 16: Situational Awareness
Double-check each other. System checking and monitoring.
Lesson 17: Situational Awareness
Fixated on knob for ADF.
Lesson 18: Situational Awareness
Situational awareness, decided to go to 11,000 feet from 12,000 after
going to East heading.
Lesson 19: Team Building and Maintenance
Cockpit crew coordination - readying approach plates, etc. for
Captain.
Lesson 20: Team Building and Maintenance
Leadership directing cockpit components, communication, what
happens next, etc.
The Lessons-Learned by the Dan/Dave crew had variety; they displayed
willingness to experiment in the LOFT simulator. Their lessons-learned were
92
driven by practicality because they reflected on how to make the CRM lesson work
for them. They applied theory in practice, looking for a new tool that would work.
Perhaps they functioned most like an airline crew because both students considered
Dan to be the senior crew member no matter what seat he occupied. There was
never a question of who was in charge and who made the major decisions for the
crew. On the other hand, Dan always endeavored to get an input from Dave,
particularly when Dave was Captain.
Ed/Eric
The E d/E ric crew had a very positive attitude; they were dedicated and
conscientious and were very anxious to learn. Both had excellent academic records,
yet their technical performance was below-average. The evaluation process
documents a laborious journey from textbook knowledge to a practical (technical or
CRM) skill. For example, their Lessons-Learned show their struggle with checklist
management:
Standardization needed with regards to use of checklists in the
cockpit. (LOFT I)
Need to work more on the "do-list" and the "checklist" concept.
(LOFT 2)
Still haven’t established the "do-list" and "checklist." (LOFT 3)
"Do-list" and "checklist" still need more work. (LOFT 4)
"Do-list" and "checklist" needs a hell of a lot more work. (LOFT 5)
Table 7 shows the changes in effectiveness measures for Ed/Eric.
93
Table 7. E d/E ric Evaluation Results.
E d /E r ic
C A P T A IN /F IR S T O FFIC E R
1st
2nd
Self-A nalysis
Self-A nalysis
LOFT 3
LOFT 5
E d /E r ic
E r ic /E d
++
+
CM AQ
C om m unication and C oordination
+
C om m and R esponsibility
R ecogn ition o f Stressor Effects
*
— —
L IN E /L O S C H EC K LIST
C om m u n ication s/D ecision Behavior
+
_
_
T eam Building and M aintenance
++
•
—
W orkload M gm t/Situ ation al A w areness
+
■ ■
O verall T echnical Proficiency
+
O verall Crew Effectiveness
+
_
CRM SU R V EY
nc
Training V alu e o f L O FT
nc
Scenario Quality
++
*
W orkload Im posed
++
nc
■ —
Instructor R ating
— — —
Self-E valuation o f Perform ance
»
■ ■
Self-R eport on C R M Skills
_
_
C O M M U N IC A T IO N S A N A L Y SIS
T otal C om m unications
_
_
_
«, —
C om m ands (by Capt)
+ +
— — —
A cknow ledgem ents (by F O )
+ + +
_
O bservations (Capt & F O )
+ + +
-
Key: Standard deviations since the last observation: —
= < -2 , - - = - l < - 2 ,
- = 0 < -I , n c = no change, + = 0 > 1 , + + = 1 > 2 , + + + = 2 > 3
94
The crew CMAQ showed positive results for the first self-analysis session in
communication and coordination, indicating that self-analysis influenced their
learning of interpersonal awareness, communication, and crew coordination. The
crew became aware of communications deficiencies in the first self-analysis session;
subsequently communication became the nucleus of CRM for them.
The LINE/LOS Checklist showed very moderate gains attributable to selfanalysis except in Team Building and Maintenance for the first session. The crew
did not use CRM in LOFT 3 because they were fully engaged by the technical
requirements of the flight. Their performance declined as the difficulty increased
in subsequent LOFTs. Since they concentrated their energy on communication and
judgement processes, they were unable to manage the additional demands of
situational awareness (LOFT 4) and team processes (LOFT 5). LOFT 4 and
LOFT 5 both involved "lost communications" with air traffic control, a stumbling
block for Ed/Eric. They had a conceptual understanding of "lost communications"
procedures, but were unable to transfer their knowledge into appropriate action.
The CRM Survey Report for the first self-analysis session showed that the
crew perceived an increase in the scenario quality and in the workload imposed, but
they did not sense any gains in performance or use of CRM skills over conventional
debriefing. Their perception of progress apparently declined when they
encountered explicit feedback in self-analysis. Their negative entries in the second
self-analysis session probably reflected that the crew was "over their head" in a
scenario that was too difficult for their skill level.
95
Communications Analysis substantiated the results of the other effectiveness
measures for this crew. After the first self-analysis session, the crew’s concentration
on cockpit communication generated considerable progress in communication
effectiveness measures. The Communications Analysis also confirmed their lack of
progress in LOFT 5.
The Lessons-Learned for Ed/Eric focused on Communication and Team
Building. They reported that the majority of their lessons were learned from
LOFT, but seven of their Lessons-Learned were attributed to self-analysis. The
Lessons-Learned that E d/E ric credited to self-analysis were:
Lesson I: Communication
Need to listen to one another and not cut in on each other.
Lesson 2: Communication
Too much extraneous talking, need to concentrate on the "sterile
cockpit."
Lesson 3: Communication
During the problem solving, need to work on "thinking first." Still too
much communication to oneself out loud.
Lesson 4: Communication
Advocacy is very important part of CRM!
Lesson 5: Communication
"Do" list and "check" list still need more work.
Lesson 6: Communication
Sterile cockpit and extraneous conversation needs more work. (Has
improved though).
Lesson 7: Situational Awareness
Situational awareness needs some more improvement (not
recognizing temperature gauge).
I I
n
w
./
>
.
i
96
The Lessons-Learned document that E d/Eric were considerably
preoccupied with cockpit communication skills. The self-analysis sessions made
them acutely aware of the confusion persisting in their communications. They
became aware of interrupting each other’s thoughts, extraneous talking, thinking
out loud, and incomplete or interrupted communication. Being aware is the first
step, but not the solution. This crew, more than any other, needed more time and
less complicated scenarios so they could continue to make progress in effective
communications. They were not ready for the advanced CRM skills that were
introduced in LOFT 4 and LOFT 5.
Similar to the situation of Alex/Art, it appears that discovery-learning in
self-analysis made the communications issues more visible to E d/E ric than if they
came from an instructor. Self-analysis may be more effective for learning about
personal communications habits that are not readily apparent from other sources.
Summary
Summary of Findings
Due to the small number of students involved, comparisons between crews
were only speculative and non-inferential. Table 8 provides a summary statement
of the results for each measurement instrument, stating whether or not it supports
the hypothesis that self-analysis was an effective learning strategy for each crew. It
was compiled by graphical analysis and visual inspection of the data; by combining
the magnitude and direction of change for each factor, a global appraisal was
I
97
Table 8. Summary: Effectiveness of Self-Analysis.
L ESSO N S-
COMM
SE SSIO N
CMAQ
L IN E /L O S
SURVEY
LEARNED
A N A L Y SIS
1st A le x /A r t
No
No
[Yes]
N /A
Y es
No
[Yes]
[No]
Y es
[Yes]
1st B etty /B o b
No
Y es
[Yes]
N /A
[Yes]
2nd B etty /B o b
No
No
No
No
[Yes]
1st C arl/C athy
No
No
[Yes]
N /A
[Yes]
2nd C arl/C athy*
No
Y es
[No]
Y es
[Yes]
1st D a n /D a v e
No
[Yes]
[Yes]
N /A
[Yes]
2nd D a n /D a v e
N eutral
Y es
[No]
Y es
No
1st E d /E r ic
[No]
[Yes]
[Yes]
N /A
[Yes]
2nd E d /E r ic
No
No
No
No
No
SE L F -A N A L Y SIS
2nd A le x /A r t
r
Y es
= Supports that self-analysis is effective.
No
= D o e s not support that self-analysis is effective.
[Y es]
= M ostly Y es.
[N o]
= M ostly N o.
N /A
= N o t A pplicable.
*
= G uided self-analysis.
produced for the entire instrument. Only significant changes, observations that
changed more than one standard deviation since the last observation, were
considered in computing the global evaluation. The results were considered to
definitely support self-analysis only if all factors of an instrument showed gains.
Conversely, if all factors showed losses, the results were judged to definitely not
98
support self-analysis. Often, the data were not definitive, but the factors could be
judged to be mostly supportive or mostly non-supportive of self-analysis as an
effective learning strategy for that crew.
Individually, the instruments gave a different response to the effectiveness
question. Using multiple instruments to measure effectiveness took advantage of
varying sources of data. Reliability was maximized by collecting data from each
source and estimating the correlations among the sources (Chidester, Kanki, &
Helmreich, 1989). There was not absolute correlation between data sources;
instead they converged on an answer that reflected the variability in learning styles
of these students.
Summary of Lessons-Learned
The final assignment for the crew at the end of each CRM session was to
complete the Lessons-Learned worksheet (see Appendix F). The worksheet details
ten CRM lessons that the crew learned during that session. Though it is artificial
that each crew should learn exactly ten CRM lessons in a session, the worksheet
was designed to stimulate extended reflection on the learning process.
Consequently, each crew wrote exactly 50 lessons-learned in the course. These
were later analyzed and categorized into the four CRM skills that were taught.
Though directions were explicit that crews should write only about CRM lessonslearned, several wrote about technical lessons-learned, indicating that they did not
thoroughly distinguish between CRM skills and technical skills.
99
The crews were also asked to name the source of learning for each lessonlearned. Without exception, LOFT proved to be a valuable learning source. This
clearly indicated that these students learned CRM by doing it. Self-analysis was a
helpful learning source for three crews, indicating that it also had value. The
strongest support for self-analysis came from Dan/Dave who frankly acknowledged
that they did not like self-analysis but attributed 40% of their learning to it. For the
50 lessons-learned by each crew, Table 9 shows the skills that were learned and the
sources of learning for those skills.
Table 9. Summary of Lessons-Learned.
Lessons-Learnec
C om m u­
D ecision
Sit.
Team
Crew
nication
M aking
A w are
Building
T ech
T otal
A /A
8
10
10
22
0
50
B /B
14 .
9
21
3
3
50
C /C
9
6
16
5
14
50
D /D
9
18
12
11
0
50
E /E
24
4
9
13
0
50
•
W here Learned
SelfCrew
D eb rief
Instructor
LO FT
Preflight
R dgs
analysis
A /A
3
0
27
0
9
11
B /B
3
2
35
I
4
5
C /C
15
0
20
2
0
13
D /D
0
I
28
0
I
20
E /E
0
6
34
0
3
7
.
100
Each crew focused their lessons-learned on a specific CRM skill, and four
crews had a CRM skill they neglected:
CREW
FOCUSED ON
NEGLECTED
Alex/Art
Team Building
None
Betty/Bob
Situational Awareness
Team Building
Carl/Cathy
Situational Awareness
Team Building
Dan/Dave
Decision Making
Communication
E d/E ric
Communication
Decision Making
All of the CRM skills were not taught in the beginning of training; instead they
were introduced in the following sequence: Communications (LOFT 2), Decision
Making (LOFT 3), Situational Awareness (LOFT 4), and Team Building (LOFT
5). It would be logical for Communications and Decision Making to have more
references in the lessons-learned and for Situational Awareness and Team Building
to be neglected. Crews with lower technical ability (Dan/Dave and Ed/Eric)
neglected one of the earlier lessons of Communications or Decision Making.
Crews with more technical ability (Alex/Art, Betty/Bob, and Carl/Cathy) were
able to focus on Situational Awareness and Team Building, though these CRM
skills were taught towards the end of the curriculum.
101
CHAPTER 5
CONCLUSIONS AND RECOMMENDATIONS
Conclusions
The Cockpit Management Attitudes
Questionnaire CCMAOl
Research Question #1: Does self-analysis of LOFT change
undergraduate flight students’ attitudes, as measured by the CMAQ.
For every crew, the CMAQ indicated only slight variability and provided
essentially no evidence for effectiveness of self-analysis. Two crews showed
nominal increases for isolated CMAQ variables in self-analysis sessions, and these
gains supported results from other measurement instruments. Relationships
between attitudes and performance have been validated for airline crews
(Helmreich, Foushee, Benson, & Russini, 1986), but the instrument may be
unsuitable for undergraduates because they lack crew experience on which to base
attitudes. Also, the CMAQ was taken by a crew as a consensus measure of crew
attitude, though it was designed as an individual instrument. A "crew attitude" may
not even exist. It is also conceivable that the CMAQ showed small variations
because it was completed so often (every two weeks) and crews remembered
J
102
previous responses. Thus, for several reasons, the CMAQ did not render an
acceptable measure of self-analysis effectiveness.
The LINE/LOS Checklist
Research Question #2: Does self-analysis of LOFT change
undergraduate flight crews’ effectiveness, as measured by the
LINE/LOS Checklist?
In the first self-analysis session, only one crew showed decisive gains in the
LINE/LOS Checklist, two crews showed modest gains, and two crews showed no
gains at all. In the second self-analysis session, two crews showed considerable
gains, one crew showed moderate gains, and two crews showed no gains at all. The
LINE/LOS Checklist is consistent with other measures when the crew’s
performance is either superior or substandard. It did not discriminate well for
crews with intermediate performance.
The LINE/LOS Checklist was probably the most objective measure of
effectiveness because it required systematic data collection of CRM skills distinct
from technical performance. Consensus grading compelled justification for every
grade and reduced the possibility of grading by instinct, crew reputation, or
preferred results. Of all the measurements taken, the LINE/LOS Checklist is the
best summary of the study. It shows three significant gains and three limited gains
for the ten self-analysis sessions.
I
103
The CRM Survey
Research Question #3: Does self-analysis of LOFT change
undergraduate flight crews’ reactions to CRM training, as measured
by the CRM Survey.
The CRM Survey was designed as a self-analysis instrument. In the CRM
Survey, none of the crews rated self-analysis highly as a training technique. One
factor in the survey, Self-Report on CRM Skills, is probably the most direct
measure of self-analysis. Three of the five crews showed a step increase in this
factor after the first application of self-analysis, but none reported gains in the
second session. It appears that crews became more discerning and critical as they
gained awareness of the CRM skills.
For self-analysis sessions, two other trends were evident in the survey:
1. Instructor Ratings declined. Four crews rated the instructor lower in selfanalysis sessions showing that self-analysis was harder than having the instructor
give the answers. Instructor ratings were markedly higher for baseline sessions
noting that instructor "knowledge and helpfulness" were decidedly missing in the
self-analysis sessions.
2. Workload Imposed increased. Three crews said that self-analysis was
more work; two said it was about the same as baseline training. Crews apparently
preferred conventional debriefing with the instructor; the extra work was perceived
as a negative feature of self-analysis.
104
Communications Analysis
Research Question #4: Does self-analysis of LOFT change
undergraduate flight crews’ performance of CRM skills, as measured
by Communications Analysis?
Multiple measures of effectiveness were used because each data source has
its strengths and weaknesses. A data source has merit if it consistently validates or
disproves the results from other measures. In Communications Analysis,
frequencies are an equivocal measure of effectiveness because communication
must be interpreted within a task, environment, or interpersonal context (Kanki &
Palmer, 1993). In this study, three of the four communications categories
confirmed results of other measures. However, Total Communications was not
consistent as a measure of effectiveness. It appears that well-intentioned crews, in
an effort to practice communications skills, "talked" more but "communicated" less.
When a crew was scheduled for self-analysis debriefing, they did not debrief ,
immediately after the simulator session but waited until the transcript and
communication analysis of the LOFT videotape were complete. This process was
expedited, but there was usually a two-day delay between the simulator session and
the self-analysis debrief. This unfortunate detainment allowed the crews to lose
some of the intensity and emotions of the LOFT simulator, even though all the
details were still available through the videotape and transcript. It is possible that
the affective results of self-analysis would be different if the debriefings were
I
105
immediate as they were in the conventional debriefings. The conventional
debriefings were not delayed for two days after the simulator session, because
feedback is best if it is immediate (Butler, 1993).
CRM and Technical Skills
Research Question #5: Are there factors or circumstances that
strengthen or inhibit the learning of CRM skills by self-analysis?
Throughout this research, the focus was on CRM skills, leaving the
impression that CRM skills are superior to or more desirable than technical skills.
A high degree of technical proficiency is essential for safe and efficient flight
operations (FAA, 1993). In this research, crews with lower technical ability had
considerable difficulty learning CRM skills, confirming the conventional wisdom
that mature technical skills are the foundation for developing CRM skills.
Assessment of technical skills prior to CRM training was essential; however, a
higher qualifying grade on the technical skills test should have been established,
excluding more individuals.
Women Pilots
Research Question #5: Are there factors or circumstances that
strengthen or inhibit the learning of CRM skills by self-analysis?
The two women pilots that participated in the study were as professional and
competent as the men, supporting the reality that women belong in aviation and
■f
106
should be encouraged to participate equally with men in all domains of the
industry. This study confirmed that all pilots should be treated equally; if even
suspicions of discriminatory or preferential behavior are raised, the issue must be
confronted. In the case of Betty/Bob, the learning needs of the students should
have taken precedence over the research design and the crew dynamics issues
should have been discussed and resolved. Sexist behavior by women or men,
students or instructors, must be addressed and eliminated as soon as it is
discovered.
Scenarios
Research Question #5: Are there factors or circumstances that
strengthen or inhibit the learning of CRM skills by self-analysis?
The objective of LOFT is to provide crew members with the opportunity to
practice both technical and CRM skills in a realistic scenario. The scenarios for
this research were created, field-tested, and evaluated by experienced aviators
based on perceived skills of commercial pilots. "Realistic and reasonable" for
designers may not be viable for the students. In retrospect, two unanticipated
factors may have influenced the results: students needed more low-workload time
in all scenarios, and LOFT 4 and LOFT 5 were too difficult for most of the crews.
With the exception of Dan/Dave, overall technical performance in LOFT 5 was
deficient, making it difficult to determine if outcomes were attributable to selfanalysis or to the scenario itself. Undergraduate flight students in a CRM class are
107
uniquely different from airline pilots reporting for initial or recurrent CRM
training. For many, it is likely that CRM training was their first exposure to
situations where they were required to coordinate their activities with another
person. An effective CRM curriculum for them might require less content material
and more simplified scenarios to accommodate for their inexperience.
In the LOFT, no scenario required students to choose a solution that
violated Federal Aviation Regulations (FARs). However, in the 25 flights, two
crews deliberately chose to violate FARs, both while in "lost communication"
situations. All five crews had some difficulty with "lost communication" which they
encountered in LOFT 4 and LOFT 5, and these scenarios were particularly
difficult, especially for the less skilled. They all seemed comfortable with textbook
answers but had difficulty transferring the knowledge to appropriate action when
they were denied assistance or feedback. LOFT requires higher-order thinking, just
as life’s situations do. This may be one of the strongest arguments in favor of
including LOFT in undergraduate programs.
Role Definition and Crew Dynamics
Research Question #5: Are there factors or circumstances that
strengthen or inhibit the learning of CRM skills by self-analysis?
Reflecting on the crews that struggled with role definition and crew
dynamics reveals a important difference between airline CRM training and
undergraduate training. Airline crews are expected to have such issues resolved
108
beforehand, but these contentions are natural learning encounters for college
students. The outcome for Alex/Art was positive because self-analysis made them
aware of the role definition problem and they struggled with it, though it was not
totally resolved. On the other hand, self-analysis did not expose the crew dynamics
issue to Betty/Bob, so it was not addressed forthrightly and the outcome is
uncertain. It may have been more appropriate if I had temporarily relinquished my
role as researcher and reverted to my role as educator to examine the crew
dynamics issues with Betty/Bob so they could resolve it.
Self-Analysis Skills
Research Question #5: Are there factors or circumstances that
strengthen or inhibit the learning of CRM skills by self-analysis?
The expectation of the research was that crews would know.how to direct
their own thinking and learning processes in self-analysis sessions. This assumes
that they have previously developed skills in metacognitive strategies and reflective
learning. Metacognition refers to reflecting on and regulation of one’s own
thinking (Flavell, 1981). Reflective learning in a self-analysis setting means being
aware of and expertise in the use of interpersonal skills (Telfer & Biggs, 1988). The
structure and guidelines of self-analysis require students to use "higher-order"
learning skills such as analysis, synthesis, and evaluation (Bloom, Englehart, Furst,
Z
Hill, & Krathwohl, 1956).
109
The findings indicate that some of the expectations of self-analysis were not
realized because students lacked training and experience with metacognitive or
"higher-order" learning processes. Specifically, some crews exhibited deficiency in
the skills of self-disclosure or expressing feelings and emotions (Betty/Bob),
attending and listening (Ed/Eric), and placing legitimate demands on each other
(Carl/Cathy). All of the crews showed inadequacies in the skills of self-critique and
confronting others. The learning sessions of the course concentrated on four CRM
skills (Communication, Situational Awareness, Decision Making, and Team
Building) which prepared them for the skills required in the LOFT scenarios, but
were probably not adequate preparation for the task of self-analysis. Even if crews
had mastered the interpersonal, skills directed at the structured environment of the
cockpit, it is not assured that they would be able to apply them in the group climate
established in a self-analysis session. Flavell (1981) identified three metacognitive
abilities that a learner must achieve in order to exercise "higher-order" learning,
Le., a self-knowledge about personal cognitive processes, an insight into the
requirements for successful completion of the task, and adeptness with the
strategies of achieving goals in the learning process. Without specific training and
experience in metacognitive processes and reflective learning, the undergraduate
students in this study probably were not highly skilled in self-analysis processes.
no
Communications Analysis and Transcripts
Research Question #5: Are there factors or circumstances that
strengthen or inhibit the learning of CRM skills by self-analysis?
One limitation of the self-analysis method was that verbatim transcriptions
were expensive and time-consuming to produce. In order to have similar segments
of transcript for communications analysis, exactly 30 minutes of communication
from each LOFT was transcribed. Though these lengthy transcripts were valuable
learning tools, they were too extended and often overloaded the students with work.
Ten or fifteen minutes of transcribed communication would have been sufficient to
keep a crew diligently involved in self-analysis training for over an hour.
Expenses can be decreased significantly by limiting transcriptions to
segments where CRM skills are specifically required by the scenario. The segments
can be selected by the instructor and customized for the specific needs of a crew.
Graduate students or work study students can learn to do transcripts and they cost
significantly less than professional transcribers. My transcriber for this research
was a graduate student who was a skilled typist but had no experience with
transcriptions. She became a wizard at the task, averaging 2.3 hours to transcribe
15 minutes of a LOFT.
Ill
Summary of Conclusions
The research has some practical implications for the design and conduct of
flight training in general. None of the crews rated self-analysis highly, indicating
that crews preferred conventional debriefing to self-analysis. Substantial evidence
weighed against self-analysis as a stand-alone strategy for undergraduate flight
students. The results are characteristic of initial flight students who are accustomed
to more guidance and rely heavily on feedback from instructors to evaluate their
performance. However, there are sufficient data supportive of self-analysis,
especially for some crews, that it should not be completely rejected. Though airline
strategies are not necessarily applicable to undergraduate flight students, airline
research shows that debriefings handled in a "teacher-tell" manner are less effective
than discussions led by the crew themselves with the instructor and the videotape as
resources for the self-critique (Butler, 1993).
Self-analysis seems to be more effective as a supplemental strategy to be
employed by an instructor when certain conditions exist. Further research is
needed to determine the circumstances such as personalities, team dynamics, or
experience that would make it successful. Self-analysis may be more effective for
learning about personality matters or about individual differences (Alex/Art). It
also appears that self-analysis may facilitate understanding of communications
issues (Ed/Eric).
112
In general, the first exposure to self-analysis was more effective than the
second. This conclusion is partially explained by the complexity of the scenarios in
the second self-analysis session (LOFT 4 or LOFT 5) which caused a decrease in
performance. The data show a step increase in many of the effectiveness measures
after the first application of self-analysis that is not as apparent after the second
application. If the students considered self-analysis techniques to be worthwhile
and useful, they presumably would continue to use them even for conventional
debriefing sessions, thereby decreasing the effect of second or subsequent
applications of self-analysis.
The crew’s assessment of self-analysis as a learning tool matched the data in
the LINE/LOS Checklist in four out of five cases. The crews were apparently
aware of the requirements for effective CRM skills and were able to judge their
performance according to valid criteria. Their self-evaluation on the CRM Survey
showed that three crews were able to discriminate between technical performance
and performance of CRM skills. This, in itself, is a measure of their understanding
of the subject matter as it relates to performance criteria.
The effectiveness of self-analysis is strongly influenced by individual
technical ability and by the complexity of the scenarios. Self-analysis may also
become more effective as students accumulate experiences with crew operations
and with self-critique. Butler (1993) supports the concept that learning to perform
operational review and analysis establishes a platform for effective post-LOFT
discussion that consists of more than technical issues. In the beginning of this
113
course, students had no experience base, but after several sessions they had ample
exposure to several CRM skills. As the course progressed, their self-evaluations
became more discerning, their lessons-learned became more discriminating, and
their cockpit communication reflected active practice of the CRM skills.
Recommendations
Improving Self-Analysis
After reflecting on the procedures used for self-analysis in this study, I
recommend changes in the design that might improve the study. Instead of.forming
crews with students of equal skill, I would use mismatched crews, similar to
Dan/Dave, so that one person obviously had seniority in the crew. I would not
swap roles between Captain and First Officer but would organize the crews so that
the more experienced pilot was always the Captain. I would considerably reduce
the complexity of the scenarios and even tailor scenarios to match the learning pace
of individual crews. I would decrease the length of the self-analysis sessions to
about one hour and would provide fewer pages of transcript for analysis. I would
allot more time for learning effective communication skills and would discriminate
between communications skills needed in self-analysis sessions and cockpit
communications skills. I would have a higher standard for admission into the
course, denying admission to anyone whose technical skills score was less than six.
114
Metacognitive Learning
Smith (1982) explored metacognition, the concept of learning how to learn,
and concluded: "A central task of learning how to learn is developing awareness of
oneself as a learner" (p. 57). This requires that the learner engage in reflection on
his own learning process such as an inventory of learning styles. Flavell (1981)
identified two other metacognitive skills that a learner must achieve in order to
engage in "higher-order" learning activities such as self-analysis, i.e., an insight into
the requirements of the task and adeptness with "higher order" learning strategies.
I recommend training and experience in metacognitive processes and reflective
learning before undergraduate students embark on self-analysis. This could be a
supplementary course in interpersonal skills which includes the skills of self­
disclosure, self-critique, expression of feelings and emotions, attending and
listening, accurate empathetic understanding, genuineness and respect, challenging,
confrontation and immediacy (Egan, 1976). The minimum preparation for selfanalysis should be a module on metacognition that furnishes the students with skills
required for effective self-analysis: self-awareness, an insight into the learning task,
and proficiency with interpersonal skills such as self-critique, attending and
listening, and confrontation (Flavell, 1981).
Guided Self-Analvsis
Because one crew centered on technical discussions, a variant of selfanalysis was employed; the instructor closely monitored the session and
115
participated in the exchange. It became a guided self-analysis session and produced
strong gains in the LINE/LOS Checklist. Based on the outcomes from this flight, I
recommend using a facilitated approach to self-analysis, a teaching strategy that
combines self-analysis and conventional debriefing. This strategy theoretically
employs the benefits of self-analysis and standard debriefings, providing a very
powerful debrief method for undergraduate crews. Research with this design might
be more complex because it would be difficult to discriminate between guided selfanalysis and conventional debriefing.
Role Models
Participants in this study were sometimes frustrated because they did not
always know "the right way" to do things. They were given readings, particularly
sections of airline operations manuals, but they were not always able to apply
theory to practice in the LOFT. Self-analysis did not help either, unless they knew
beforehand what to do. It became clear that these students needed a standard for
their comparison. They needed to imitate a Captain or First Officer, a role model
with considerable experience in crew operations to demonstrate effective crew
performance. In this study, the only role model a student experienced was another
student. I recommend a research design where a self-analysis session is preceded
by an exposure to role model crew members who illustrate effective crew
performance. Students could observe role models on videotapes or role plays, but
'I
116
the best training would be achieved by flying a LOFT scenario with a pilot
experienced in crew operations.
For Further Research
Crews in this study experienced that the cockpit can be a confining and
sometimes emotional environment and that male/female relationships can add
CRM issues that must be considered. Further research is needed to understand
perceptions of male dominance, male/female dynamics, and the seniority of
Captains regardless of age, sex, and often skill. These issues are compelling
reasons why CRM should be included in initial flight training to educate men and
women of the appropriate paradigm: men and women are equal; performance, not
gender, is the decisive factor.
Preferably students should randomly be assigned to flight crews, but
diversity of experience and differences in technical ability of these students dictated
that they should be assigned to crews by technical ability rather than randomly.
Ability grouping where experienced pilots were teamed together left at least one
crew that was too inexperienced to handle the complexities of some scenarios. The
data show that excellent gains in effectiveness were achieved by the crew that had
mixed skills (Dan/Dave). Further research should consider forming crews using
one pilot with high skills and one with lower skills, allowing for greater
heterogeneity of the crews.
117
Each learning session focused on a new CRM skill using the Cooperative
Learning procedure called "Jigsaw" (Johnson, Johnson, & Smith, 1991). Students
individually studied material about that particular CRM skill and then shared their
learning with the other crew member. It is uncertain whether one exposure to each
CRM skill is sufficient for thorough learning. LOFT engages the students in such
higher-order thinking tasks as analysis, synthesis, and evaluation (Bonwell & Eison,
1991). Future CRM programs could attempt a longer program of study so that
each CRM skill could be discussed several times at different levels of
comprehension.
In self-analysis sessions, crews evaluated their own performance in a
secluded room using videotapes and transcripts that were not shared with other
crews. It was suggested that group analysis of LOFT videotapes and transcripts
might be a valuable learning method. Admitting that much could be learned from
serious group analysis of another crew’s successes and mistakes, I envision possible
negative outcomes such as frivolity, crew embarrassment, or individual mockery if
the scrutinized crew members are classmates. Further research might consider if it
is worthwhile for the entire group to analyze videotapes and transcripts from
LOFTs performed by unfamiliar crew members.
Training requirements compelled students to exchange roles between
Captain and First Officer for each scenario. The significance of role changing for
the research was that crew performance could vary with the Captain. Assigning one
crew member to be Captain for the entire study would have been better for
118
research; however, it is unethical in an educational environment because students
require exposure to both paradigms. An alternative approach would be to teach
CRM in two semesters with beginner students flying First Officer and experienced
ones flying Captain. A potential benefit is that experienced students could be a
role model for novice students. Further research could determine if a student with
one semester of LOFT experience is an adequate role model.
Future research should recognize that college students need acclimation to
crew operations; scenarios should be uncomplicated and should include significant
low-workload periods. Undergraduate flight students require high stress, high
workload, and emergency experiences that stimulate the use of CRM skills, but
their level of experience dictates that scenarios should not be overwhelming.
Guidelines and scenarios developed for airline pilots may be top complex for
undergraduate flight students.
119
REFERENCES CITED
120
AOPA’s Aviation USA. (1991). Frederick, MD: Aircraft Owners and Pilots
Association.
Best, J. W., & Kahn, J. V. (1989). Research in education (6th ed.). Englewood
Cliffs, NI: Prentice-Hall.
Blake, R. R., & Mouton, J. S. (1982). Cockpit resource management. Denver, CO:
Scientific Methods, Inc.
Bloom, B. S., Englehart,.M. B., Furst, E. J., Hill, W. H., & Krathwohl, D. R. (1956).
Taxonomy of educational objectives: The classification of educational goals.
Handbook I: The cognitive domain. New York: Longman.
Bolman, L. (1979). Theory of the situation. Boston: Harvard University School of
Education.
Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the
classroom. ASHE-ERIC Higher Education Report No. I. Washington, DC:
The George Washington University, School of Education and Human
Development.
Borg, W. R., & Gall, M. D. (1989). Educational research: An introduction (5th
ed.). New York: Longman.
Brown, F. G. (1983). Principles of educational and psychological testing. New York:
Holt, Rinehart, and Winston.
Butler, R. E. (1991). CRM. LOFT, and Bernard Shaw: Flight crew evaluator
training workshop presentation (NASA/ University of Texas Technical
Manual 91-3). Moffett Field, CA: NASA-Ames Research Center.
Butler, R. E. (1992). AOP Committee: LOFT design focus group report (Report of
ATA LOFT Design Focus Group Meeting, February 6,1992;
NASA/University of Texas Crew Performance Project). Austin, TX:
University of Texas.
Butler, R. E. (1993). LOFT: Full-mission simulation as crew resource management
training. In E. L. Weiner, R. L. Helmreich, & B. G. Kanki (Eds.), Cockpit
resource management (pp. 231-259). San Diego: Academic Press.
Campbell, R. D., & Bagshaw, M. (1991). Human performance and limitations in
aviation. Oxford: Blackwell Scientific Publications, Inc.
121
Caro, P. W. (1988). Flight training and simulation. In E. L. Weiner, & D. C. Nagel
(Eds.), Human factors in aviation (pp. 229-262). New York: Academic Press.
Carver, C. S., & Scheier, M. F. (1981). Attention and self-regulation: A control
theory approach to human behavior. New York: Springer-Verlag.
Chidester, T. R., Kanki, B. G., & Helmreich, R. L. (1989). Performance evaluation
in full-mission simulation: Methodological advances and research
challenges. In R. S. Jensen (Ed.), Proceedings of the Fifth International
Symposium on Aviation Psychology (pp. 16-21). Columbus, OH: Ohio State
University Press.
Cohen, L., & Manion, L. (1985). Research methods in education (2nd ed.).
London: Croom Helm.
Cook, G. N. (1991). Cockpit resource management training: Are current
instructional methods likely to be successful? The Journal of
Aviation/Aerospace Education and Research. 1(3), 26-35.
Driskell, J. E., & Adams, R. J. (1992). Crew resource management: An
introductory handbook (NTIS no. PB93-101525). Springfield, VA: National
Technical Information Service.
Duval, S., & Wicklund, R. A. (1972). A theory of objective self awareness. New
York: Academic Press.
Duval, T. S., Duval, V. H., & Mulilis, J. P. (1992). Effects of self-focus, discrepancy
between self and standard, and outcome expectancy favorability on the
tendency to match self to standard or to withdraw. Journal of Personality
and Social Psychology. 62(2), 340-348.
Edwards, E. (1972). Man and machine: Systems for safety. In Proceedings of
British Airline Pilots Association Technical Symposium, (p. 21-36). London:
British Airline Pilots Association.
Edwards, E. (1988). Introductory overview. In E. L. Weiner, & D. C. Nagel (Eds.),
Human factors in aviation (pp. 3-26). New York: Academic Press.
Egan, G. (1976). Interpersonal living: A skills/contract approach to human
relations training in groups. Monterey, CA: Wadsworth.
122
Federal Aviation Administration. (1992). Line operational simulation advisory
circular (AC no. 120-35B). Washington, DC: U.S. Department of
Transportation.
Federal Aviation Administration. (1993). Cockpit resource management training
(AC no. 120-51A). Washington, DC: U.S. Department of Transportation.
Festinger, L. (1957). A theory of cognitive dissonance. Stanford, CA: Stanford
University Press.
Flavell, J. H. (1981). Cognitive monitoring. In W. P. Dickson (Ed.), Children’s oral
communication skills. New York: Academic Press.
Foushee, H. C., & Helmreich, R. L. (1988). Group interaction and flight crew
performance. In E. L. Weiner, & D C. Nagel (Eds.), Human factors in
aviation (pp. 189-228). New York: Academic Press.
Foushee, H. C., & Manos, K. L. (1981). Information transfer within the cockpit. In
C. E. Billings and E. S. Cheaney (Eds,), Information transfer problems in the
aviation system (NASA Technical Paper 1875). Moffett Field, CA: NASA
Ames Research Center.
Foushee, H. C., Lauber, J. K., Baetge, M. M., & Acomb, D. B. (1986). Crew factors
in flight operations: III. The operational significance of exposure to shorthaul air transport operations (NASA Technical Memorandum 88322).
Moffett Field, CA: NASA Ames Research Center.
Frasca model 142 fact sheet. (1986). Urbana, IL: Frasca International, Inc.
Gay, L. R. (1992). Educational research: Competencies for analysis and
application (4th ed.). New York: Macmillan.
Gibbons, F. X., & Wicklund, R. A. (1982). Self-focused attention and helping
behavior. Journal of Personality and Social Psychology. 43(3), 462-474.
Gregorich, S. E. (1993). The dynamics of CRM attitude change: Attitude stability.
In R. S. Jensen (Ed.), Proceedings of the Seventh International Symposium
on Aviation Psychology (pp. 509-512). Columbus, OH: Ohio State University
Press.
Gregorich, S. E., Helmreich, R. L., & Wilhelm, J. A. (1990). The structure of
cockpit management attitudes. Journal of Applied Psychology. 75(6), 682690.
123
Gregorich, S. E., Helmreich, R. L., Wilhelm, J. A , & Chidester, T. (1989).
Personality based clusters as predictors of aviator attitudes and
performance. In R. S. Jensen (Ed,), Proceedings of the Fifth International
Symposium on Aviation Psychology (pp. 686-691). Columbus, OH: Ohio
State University Press.
Gregorich, S. E., & Wilhelm, J. A. (1993). Crew Resource Management Training
Assessment. In E. L. Wiener, R. L. Helmreich, & B. G. Kanki (Eds.),
Cockpit Resource Management (pp. 173-198). San Diego: Academic Press.
Guba, E. G. (1978). Toward a methodology of naturalistic inquiry in educational
evaluation. Los Angeles: Center for the study of Evaluation, UCLA
Graduate School of Education.
Guba, E. G. & Lincoln, Y. S. (1981). Effective evaluation: Improving the usefulness
of evaluation results through responsive and naturalistic approaches. San
Francisco: Jossey-Bass.
Helmreich, R. L. (1983a, October). What changes and what endures: The
capabilities and limitations of training and selection. Paper presented at the
Irish A r Line Pilots/AER Lingus Flight Operations Symposium, Dublin,
Ireland.
Helmreich, R. L. (1983b). Cockpit management attitudes. Paper presented at the
Second International Symposium on Aviation Psychology, Ohio State
University. Columbus, OH.
Helmreich, R. L. (1984). Cockpit Management Attitudes. Human Factors, 26(5),
583-589.
Helmreich, R. L. (1989). Evaluating the effectiveness of cockpit resource
management training. In Proceedings of the Second Conference on Human
Error Avoidance Techniques (pp. 17-22). Warrendale, PA: Society of
Automotive Engineers.
Helmreich, R. L. (1991). Strategies for the study of flight crew behavior. In R. S.
Jensen (Ed.), Proceedings of the Sixth International Symposium on Aviation
Psychology (pp. 338-343). Columbus, OH: Ohio State University Press.
Helmreich, R. L., & Foushee, H. C. (1993). Why crew resource management? In E.
L. Wiener, R. L. Helmreich, & B. G. Kanki (Eds.), Cockpit resource
management (pp. 3-45). San Diego: Academic Press.
124
Helmreich, R. L., Foushee, H. C., Benson, R., & Russini, W. (1986). Cockpit
resource management: Exploring the attitude-performance linkage.
Aviation. Space, and Environmental Medicine. 57. 1198-1200.
Helmreich, R. L., Spence, J. T., & Holahan, C. K. (1979). Psychological androgyny
and sex-role attitudes: A test of two hypotheses. Journal of Personality and
Social Psychology. 3 7 ,1631-1644.
Helmreich, R. L., & Wilhelm, J. A. (1987). Evaluating cockpit resource
management training. In R. S. Jensen (Ed.), Proceedings of the Fourth
International Symposium on Aviation Psychology (pp. 440-446). Columbus,
OH: Ohio State University Press.
Helmreich, R. L., & Wilhelm, J. A. (1989). When training boomerangs: Negative
outcomes associated with Cockpit Resource Management programs. In R. S.
Jensen (Ed.), Proceedings of the Fifth International Symposium on Aviation
Psychology (pp. 692-697). Columbus, OH: Ohio State University Press.
Helmreich, R. L., & Wilhelm, J. A. (1991). Methodological issues in the use of the
TINF, /I OS Checklist and Behavioral Markers: Notes for evaluators.
(NASA/University of Texas Technical Memorandum 91-1). Austin, TX:
University of Texas.
Helmreich, R. L., Wilhelm, J. A., & Gregorich, S. E. (1988). Revised version of the
Cockpit Management Attitudes Questionnaire (CMAO) and CRM Seminar
Evaluation Form (NASA/University of Texas Technical Report 88-3).
Austin, TX: University of Texas.
Helmreich, R. L., Wilhelm, J. A., Gregorich, S. E., & Chidester, T. (1990).
Preliminary results from the evaluation of cockpit resource management
training: Performance ratings of flight crews. Aviation, Space, and
Environmental Medicine. 61. 576-579.
Helmreich, R. L., Wilhelm, J. A., Kello, J. E., Taggart, W. R., & Butler, R. E.
(1991). Reinforcing and evaluating crew resource management:
Evaluator/LOS instructor reference manual (NASA/ University of Texas
Technical Manual 90-2, Revision I). Austin, TX: University of Texas.
Hawkins, F. H. (1987). Human factors in flight. Brookfield, VT: Gower Publishing
Company.
125
Irwin, C. M. (1991). The impact of initial and recurrent cockpit resource
management training on attitudes. In R. S. Jensen (Ed.), Proceedings of the
Sixth International Symposium on Aviation Psychology (pp. 344-349).
Columbus, OH: Ohio State University Press.
Johnson, D. W., Johnson, R. T., & Smith, K. A. (1991). Cooperative learning:
Increasing college faculty instructional productivity. ASHE-ERIC Higher
Education Report No. 4. Washington, DC: The George Washington
University, School of Education and Human Development.
Jones, M. B. (1974). Regressing group on individual effectiveness. Organizational
Behavior and Human Performance. 11.426-451.
Jones, E. E., & Nisbett, R. E. (1971). The actor and the observer: Divergent
perceptions of the causes of behavior. New York: General Learning Press.
Kanki, B. G. (1991). Session overview: Information transfer and crew performance.
In R. S. Jensen (Ed.), Proceedings of the Sixth International Symposium on
Aviation Psychology (pp. 247-252). Columbus, OH: Ohio State University
Press.
Kanki, B. G., & Palmer, M. T. (1993); Communication and crew resource
management. In E. L. Weiner, R. L. Helmreich, & B. G. Kanki (Eds.),
Cockpit resource management (pp. 99-136). San Diego: Academic Press.
Kanki, B. G., Palmer, M. T., & Veinott, E. (1991). Communication variations
related to leader personality. In R. S. Jensen (Ed.), Proceedings of the Sixth
International Symposium on Aviation Psychology (pp. 253-259). Columbus,
OH: Ohio State University Press.
Kanki, B. G., Lozito, S., & Foushee, H. C. (1987). Communication indexes of crew
coordination. In R. S. Jensen (Ed.), Proceedings of the Fourth International
Symposium on Aviation Psychology (pp. 406-412). Columbus, OH: Ohio
State University Press.
Kazdin, A. C. (1982). Single-case research designs. New York: Oxford University
Press.
Kerlinger, F. N. (1973). Foundations of behavioral research (2nd ed.). New York:
Holt, Rinehart and Winston.
126
Lauber, J. K., & Foushee, H. C. (1983, October). Line-oriented flight training. In
The Flight Operations Symposium. Irish Air Line Pilot’s Association/AER
Lingus, Dublin, Ireland.
. Lofaro, R. I. (1990). Cockpit resource management: Research, development,
status. Airworthy Aviator. 1(3), I, 8.
Massey, D. (1990). CRM: Teaching crew interaction. Career Pilot. 10,13-17.
Nash, T. (1992). LOFT: Where is it going? Civil Aviation Training. 3(1), 26-30.
O’Hare, D., & Roscoe, S. (1990). Flight deck performance: The human factor.
Ames,LA: Iowa Statb University Press.
Piaget, J. (1966). Judgement and reasoning in the child. Totowa, NI: Littlefield,
Adams.
Ross, L. E., Slotten, P. W., & Yeazel, L. M. (1990). Pilots’ evaluation of the
usefulness of full mission IFR simulator flights for general aviation pilot
training. The Journal of Aviation/Aerospace Education and Research. 1(2),
16-25.
Ruffell-Smith, H. P. (1979). A simulator study of the interaction of pilot workload
with errors, vigilance, and decisions (NASA Technical Memorandum
78482). Moffett Field, CA: NASA-Ames Research Center.
Sams, T. L. (1987). Cockpit resource management concepts and training strategies:
Developing an analysis of training needs. In R. S. Jensen (Ed.), Proceedings
of the Fourth International Symposium on Aviation Psychology (pp. 360371). Columbus, OH: Ohio State University Press.
Simon, R., Pawlik, E. A., & Bronkhorst, T. M. (1991). Aircrew coordination for
Army helicopters: An exploration of the attitude-behavior-performance
relationship. In R. S. Jensen (Ed.), Proceedings of the Sixth International
Symposium on Aviation Psychology (pp. 290-295). Columbus, OH: Ohio
State University Press.
Smith, R. M. (1982). Learning how to learn. Chicago: Follett Publishing.
Stone,,R. B. (1988). Airline Pilots’ perspective. In E. L Weiner, & D. C. Nagel
(Eds.), Human factors in aviation (pp. 189-228). New York: Academic Press.
127
Teller, R., & Biggs, J. (1988). The psychology of flight training. Ames, IA: Iowa
State University Press.
Trollip, S. R., & Jensen, R. S. (1991). Human factors for general aviation.
Englewood, CO: Jeppesen Sanderson.
Vandermark, M. J. (1989). Aircrew Team Dynamics: A comprehensive crew
management program for America West Airline’s pilots and flight
attendants. In R. S. Jensen (Ed.), Proceedings of the Fifth International
Symposium on Aviation Psychology (pp. 640-645). Columbus, OH: Ohio
State University Press.
Weisman, G. (1991). A new partnership in CRM training. ICAO Journal. 9,10-12.
Wicklund, R. A. (1975). Objective self-awareness. In L. Berkowitz (Ed.), Advances
in experimental social psychology (Vol. 81. New York: Academic Press.
Wicklund, R. A , & Brehm, J. W. (1976). Perspectives on cognitive dissonance.
Hillsdale, NJ: Lawrence Erlbaum, Associates.
Wilhelm, J. (1991). Crew member and instructor evaluations of Line Oriented
Flight Training. In R. S. Jensen (Ed.), Proceedings of the Sixth International
Symposium on Aviation Psychology (pp. 362-367). Columbus, OH: Ohio
State University Press.
APPENDICES
129
Appendix A
The Cockpit Management Attitudes Questionnaire (CMAQ)
130
Cockpit M anagem ent Attitudes Q uestionnaire
A l part of NASA tpootortd rdearth, we art
H«n on •ttimrlM about fiightdeck management
in commeroai openiiont. You will greatly auiit our reiearch u you complete the iunrey. AU data are strictly
confidential Please answer by writing beside each item the letter that best reflects v o u r personal a t t i t u d e .
Choose the letter from the scue below.
**Scale**
D is a g re e
Strongly
IiI?K!5
D
s}!|..ly
^
strongly
L
Crew members should avoid disagreeing with others because conflicts create tension and reduce crew
cEectivcncis.
Z
Crew members should feel obligated to mention their own psychological stress or physical problems
to other flight crew personnel before or during a flight
3.
If is important to avoid negative comments about the procedures and techniques of other crew
members.
4.
Ciptmns should cut dictate technique to their first officers.
5.
Casual, social conversation in the cockpit during periods of low workload can improve crew
coordination.
6.
Each crew member should monitor other crew members for signs of stress or fatigue and should
d iin in the situation with the crew member.
7.
'
Good communications and crew coordination are as important as technical proficiency for the safety
of flight
&.
TOots should be aware a and sensitive to the personal problems of other crew members.
9.
The captain should take physical control and fly the aircraft in emergency and non-standard
10. The pilot flying the aircraft should veI f o l i y plans for procedures or manuevers and should be sure
ihai the information is understood ana acknowledged by the other crow members.
UL Crew members should not question the decisions or actions of the captain except when they threaten
the safety of the flight
12. Crew members should alert others to their actual or potential work overloads.
13. Even when fatigued, I perform effectively during critical flight maneuvers.
14. Captains thnnM encourage crewmember questions during normal flight operations and in
emergencies.
15. There are no circumstances (except total incapacitation) where the first officer should assume
command of the aircraf t
_____ 16. A debriefing and critique of procedures and Hwfonru after each flight is an important part of
developing »nH Twtnfining effective crew coordination.
_____ 17. My performance is not adversely affected by working with an inexperienced or less capable crew
member.
18. Overall, successful RighrHwIr management is primarily a function of the captain’s flying proficiency.
19. Training is one o f the captain's most important responsibilities.
-
20. Because individuals function less effectively under high stress, good e e w coordination is more
important in emergency or abnormal situations.
2L The pre-flight crew briefing is important for safety and for effective crew management
22. Effective crew coordination requires crew members to take into account the personalities of other
ctcw members.
23. The captain’s responsibilities include coordination between flight and cabin crews.
24. A truly professional crew member can leave personal problems behind when flying the line.
25. My decision making abillry is as good in emergencies as in routine Qying situations.
P lease go on to the next page
131
II. Background Infonnatlon
Airline Experience:
Year o f birth
Y ean in 121 ops _ _ _ Y e m in 135 o p t_____
Sex (M or F) _____ Y ean with present organization _ _ _ _ _ _ Nationality___________________
Status and Position: Complete both status gad position. Check those that apply. (If you are a new hire, or are
in upgrade or transition training fill m thc appropriate item, then complete status and position in
reference to your old aircraft/position.)
New hire training for________ _position in
U p grad e training to
positi'on
aircraft
"Transition training to
Status:
aircraft
Position:
Line pilot
instructor (pilot or FE)
lin e check (pilot or F tJ
Management
--------Other (specify)________
Y ea n in present position:
Captain
“ First Officer
flight Engineer (S/O )
Flight boon in present position_
Current aircraft: _______ Hours in current aircraft
Total airline flight hours (all airlines, aircraft
positions)____
MUltaiy Experience
Aircraft flown
Position(s)__________
Y e a n in military flying
Total military flight houn
O ther Civilian Pilot Experience
Aircraft flown
P ositions)__________
Nature o f flying (e x pleasure, cargo, FBO, etc.)
Y e a n in other
CrviEan operation^
Total civilian non-airline flight houn
This completes the questionnaire. Thanks for your help!
132
Appendix B
Background Questionnaire
133
BACKGROUND QUESTIONNAIRE
Date:
Name:
(NOTE: ALL RESPONSES WILL BE HANDLED ANONYMOUSLY)
I) Do you now hold, or have you ever held the following FAA Pilot Certificates:
Recreational
Private
Commercial
Air Transport Pilot
Cert. Flight Instructor
Cert. Instr. Instructor
Multi-Engine Instructor
2) Do you now hold, or have you ever held the following FAA Ratings:
Single-Engine Land
Single-Engine Sea
Multi-Engine Land
Multi-Engine Sea
Glider
Lighter-than-air
Helicopter
Instrument
3) Have you ever had formal (classroom) ground school in the following:
Private Pilot
Commercial Pilot
Air Transport Pilot
Glider Pilot
Balloon Pilot
Helicopter Pilot
Cert. Flight Instructor
Cert. Instr. Instructor
Multi-Engine Instructor
Instrument Rating
Seaplane Rating
Multi-Engine Rating
134
4) From your logbook, please enter the following logged hours:
Airplane (SEL)
Airplane (MEL)
Helicopter
Dual Given as CFI
Cross-country
Total Day
Total Night
Actual Instrument
Simulated Instrument
Link or Simulator
Dual Received
Pilot-in-Command (Solo)
Second-in-Command
# of Approaches
# of landings
Total Flight Hours
5) From your logbook, please enter:
Date of first solo:_________________________
Date of Private Pilot Certificate:_____________
Date of last FAA certificate:___________Type:
Date of Medical certificate:__________ Class: _
Date of Last Biannual Flight Review:________
6) List all makes and models of aircraft you have ever flown as a pilot. List total
hours in each type (Include simulators).
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
Hours:
135
7) Briefly state your goals and ambitions in aviation.
8) PERSONAL INFORMATION:
a) Year of Birth:_______________
b) Sex: M F
c) Home State:__________________
d) Family Background (Circle One):
RURAL, URBAN, SUBURBAN
e) Check all that apply:
Full-time student
Part-time Student
Aeronautical Science Major
Airway Science Major
Other M ajor:____________
___Freshman
___Sophomore
___Junior
___Senior
___Graduate
f) M inor:___________
g) Do you work while attending college? Yes No
IfYes,
___Full Time? or ___ Part Time?
Work Description:
136
9)
Have you had any previous experience with Cockpit Resource Management
(CRM)? ___Yes ___ No.
If Yes, Explain briefly:_______ _____________________________________
10) Have you had any previous experience with Line-Oriented Flight Training
(LOFT)? ___Yes ___ No.
If Yes, Explain briefly:___________________________________________
Appendix C
The LINE/LOS Checklist
138
N A S A /U T C re w P e rfo rm a n c e In d ic a to rs
I.
Briefing (conduct and quality).
The effective briefing will be operationally
thorough, interesting, and will address coordination, planning, and problems. (Although
primarily a Captain responsibility, other crewmembers may add significantly to planning
and definition of potential problem areas.)
E v alu atio n S eale
I
Poor
Psrformonce
Significantly
Below Expectations
2
Minimally
Acceptable Performance
Improvement Needed
4
Very Good. Above
Average
Performance
j
Satisfactory
or
Standard
Performance
S
Exceptional
Performance
Significantly
Above Standard
Crew Performance Markers
(Bold, Italicized Markers apply to Advanced Technology Flightdecksl
Scoring
a.
e s t a b l i s h e s e n v i r o n m e n t f o r o p e n / i n t e r a c t i v e c o m m u n i c a t i o n s (e g . , c a l l s f o r q u e s t i o n s o r
c o m m e n t s , a n s w e r s q u e s t i o n s d i r e c t l y , l i s t e n s w i t h p a t i e n c e , d o e s n o t i n t e r r u p t or ‘ t a l k
o v e r", d o e s n o t r u s h th r o u g h th e briefing, m a k e s e y e c o n ta c t a s a p p ro p ria te ).
b.
is i n t e r a c t i v e , 2 - w a y , a n d e m p h a s i z e s t h e
offering of in fo rm a tio n
c.
e s t a b l i s h e s " t e a m c o n c e p t " (e g . , u s e s " w e " l a n g u a g e , e n c o u r a g e s all t o p a r t i c i p a t e a n d
im p o rta n c e of q u e s t io n s ,
critiq u e a n d th e
h e l p w i t h t h e flig h t)
d.
co v ers p ertin en t sa fe ty and op eratio n al iss u e s
e.
identifies p o te n tia l p r o b le m s s u c h a s w e a t h e r , d e la y s , a n d a b n o r m a l s y s t e m o p e r a tio n s
f.
p r o v i d e s g u i d e l i n e s f o r c r e w a c t i o n s -- d i v i s i o n o f l a b o r a n d c r e w w o r k l o a d a d d r e s s e d
g.
i n c l u d e s c a b i n c r e w a s p a r t o f t e a m in t h e b r i e f i n g , a s a p p r o p r i a t e
h.
S e t s e x p e c t a t i o n s f o r h o w d e v i a t i o n s f r o m S O P. a r e t o b e h a n d l e d
i.
establishes guidelines for the operation of automated systems ti.e. when system will be
disabled, programming actions that must be verbalized and acknowledged)
j.
specifies RF and PNF duties and responsibilities with regard to Automated Systems
Overall Rating
None/Poor I
Comments:
2
3
4
5 Excellent
139
N A S A /U T C r e w P e rfo rm a n c e In d ic a to rs
2.
Inquiry/Advocacy/Assertion practiced. This rating assesses the extent to which
crewmembers advocate the course of action they feel best, even when it involves
conflict and disagreements with others.
E v alu atio n S c a le
I
Poor
Performance
Significantly
Below Expectations
2
Minimally
A cceptable Perform ance
Improvement Needed
Scoring
3
S e tn fie ie n r
or
Standard
Perform ance
4
Very Good. Above
Average
Perform ance
S
Exceptional
Performance
Significantly
Above Standard
Crew Performance Markers
___________ [Bold,
a.
Italicized Markers apply to Advanced Technology Flightdecksl
r r e w m e m h e r t e p e a l r ti n . a n d s t a t e t h e i r i n f o r m a t i o n w i t h a o o r O D r i a t e p e r s i s t e n c e , u n til
t h e r e is s o m e c l e a r r e s o l u t i o n a n d d e c i s i o n
b.
"challenge a n d re s p o n s e " e n v iro n m e n t d ev elo p e d
c.
q u e s tio n s are e n c o u ra g e d , a n d are a n s w e r e d o p en ly a n d n o n d e fe n siv e ly .
d.
c r e w m e m b e r s are e n c o u r a g e d to a s k q u e s t io n s re g ard in g c r e w a c t i o n s a n d d e c is io n s
e.
c r e w m e m b e r s s e e k in fo rm atio n a n d d irectio n from o th e rs w h e n n e c e s s a r y
/.
crewmembers question status and programming of Automated Systems to verify
situational awareness
Overall Rating
None/Poor I
Comments:
2
3
4
5 Excellent
140
N A S A /U T C r e w P e rfo rm a n c e In d ic a to rs
3.
C rew
se lf-c ritiq u e
(d e c isio n s an d
crew m em bers, conduct and
activ ities,
w h ic h
in clu d es
a c tio n s ). T h is ite m
e v a lu a te s
th e
e x te n t to
p a r t i c i p a t e in a d e b r i e f i n g , o p e r a t i o n a l r e v i e w , a n d
th e
p ro d u c t, th e
p ro cess, and
th e
p e o p le
w h ic h
c ritiq u e of
in v o lv e d .
C ritiq u e
c a n , a n d sh o u ld , o c c u r d u rin g a n a c tiv ity , a n d /o r a fte r c o m p le tio n of th e ac tiv ity .
E v alu atio n S cale
I
Poor
Performance
Significantly
Below Excectations
2
Minimally
A cceptable Performance
Improvement Needed
C rew
S co rin g
(So/d,
3
Satisfactory
or
Standard
Performance
P erfo rm an ce
4
Very Good. Above
Avereg#
Performance
S
Exceptional
Performance
Significantly
Above Standard
M ark ers
Italicized Markers apply to Advanced Technology Flightdecksl
a.
g iv e n a t a p p r o p r ia te tim e s , b o th lo w a n d high w o rk lo a d
b.
d e a l s w i t h p o s i t i v e a s w e l l a s n e g a t i v e a s p e c t s o f c r e w p e r f o r m a n c e d u r i n g f l ig h t
c.
in teractiv ely in v o lv es th e w h o le c r e w
d.
m ade
a p o sitiv e learning
ex p erien ce
-
f e e d b a c k is s p e c i f i c ,
o b jective,
based
se rv a b le b eh av io r, a n d g iv en c o n s tru c tiv e ly .
e.
c r i t i q u e is a c c e p t e d o b j e c t i v e l y a n d n o n d e f e n s i v e l y .
f.
O v e ra ll R a tin g
None/Poor I
C omm ents:
2
3
4
5 Excellent
on
ob­
141
N A S A /U T C r e w P e rfo rm a n c e In d ic a to rs
4.
C o m m u n ic a tio n s /D e c is io n s . T h is
c o m m u n ic a tio n
sary
to
and
in fo rm a tio n
d e v e lo p in g
a t th e a p p ro p ria te tim e
p ro b le m s).
p ra c tic e d . Q u e stio n in g
co m m u n ic a te d
ra tin g
re fle c ts th e
e x te n t to
w h ic h
is p r a c t i c e d . It i n c l u d e s t h e e x t e n t t o w h i c h c r e w m e m b e r s
and
A c tiv e
free
and
open
p ro v id e n e c e s ­
(for e x a m p l e , in itia tin g c h e c k l i s t s , a le r tin g o t h e r s
p artic ip a tio n
in
d ecisio n
m a k in g
process
encouraged
o f a c t i o n s a n d d e c i s i o n s is p r o p e r . D e c i s i o n s m a d e
a re clea rly
ack n o w le d g e d .
E v alu atio n S e a le
I
Poor
Perform ance
Significently
Below Expectetione
2
Minimelly
Acceptable Performance
Improvement Needed
3
Satiefectory
or
S u n d ird
Performence
4
Very Good. Above
Aversge
Performance
S
Exceptional
Performance
Significently
Above Stenderd
Crew Performance Markers
(Bold, Italicized Markers apply to Advanced Technology Flightdecksl
S coring
a.
o p e r a tio n a l d e c is io n s a r e clearly s t a te d to o th e r c r e w m e m b e r s
b.
c r e w m e m b e r s a c k n o w le d g e u n d e rsta n d in g of d ecisions m a d e
Ce
" b o t t o m lines" a re e s t a b l is h e d a n d c o m m u n i c a t e d for s a f e ty o f o p e r a t i o n s
d.
t h e " b i g p i c t u r e " a n d t h e g a m e p l a n a r e s h a r e d w i t h i n t h e t e a m i n c l u d i n g f l ig h t a t t e n ­
d a n ts and o th ers
e.
c r e w m e m b e r s a re e n c o u r a g e d to s t a t e their o w n id e a s , o p in io n s , a n d r e c o m m e n d a t i o n s
f.
e f f o r t is m a d e t o p r o v i d e a n a t m o s p h e r e c o n d u c i v e t o o p e n a n d f r e e c o m m u n i c a t i o n s
9-
entries and changes to Automated Systems' parameters are verbalized and
acknowledged
Overall Rating
None/Poor I
Comments:
2
3
4
5 Excellent
142
N A S A /U T C r e w P e rfo rm a n c e In d ic a to rs
5.
Leadership-Followership/Concern for tasks. This rating evaluates the extent to
which appropriate leadership and followership is practiced. It reflects the extent to which
the crew is concerned with the effective accomplishment of necessary tasks.
E v alu atio n S cale
I
Poor
Perform ance
Significently
Below Expectebone
2
Mimmelly
A cceptable Performance
Improvement Needed
4
Very Good, Above
Avortgo
Performance
i
Sotitfoctory
or
Standard
Performance
5
Exceptional
Performance
Significantly
Above Standard
Crew Performance Markers
{Bold, Italicized Markers apply to Advanced Technology Flightdecks)
S co rin g
a.
u t i l i z e s all a v a i l a b l e r e s o u r c e s t o a c c o m p l i s h j o b a t h a n d
b.
co o rd in ates
Ilig h td e c k
activ ities
to
e sta b lish
proper
b alance
b etw een
a u th o rity
and
assertiv en ess
c.
a c t s decisively w h e n th e situ atio n requires
d.
d e m o n s tr a te s d esire to a c h ie v e m o s t effe ctiv e p o ssib le o p e ra tio n
e.
re c o g n iz e s n e e d to m ain tain a d h e re n c e to S O P s
f.
e n s u r e s t h a t g r o u p c l i m a t e is a p p r o p r i a t e t o o p e r a t i o n a l s i t u a t i o n ( i. e . s o c i a l c o n v e r s a t i o n
in l o w w o r k l o a d c o n d i t i o n s b u t n o t h ig h )
Q
re c o g n iz e s e ffe c ts of s tr e s s a n d fatigue on p e rfo rm a n c e
h.
m a n a g e s tim e available for ta s k a c c o m p lis h m e n t
i.
recognizes and deal with demands on resources posed by opereVon of Automated
Systems
J-
disengages Automated Systems operation when programming demands could reduce
situetional awareness or create work overloads
Overall Rating
None/Poor I
Comments:
2
3
4-
5 Excellent
143
N A S A /U T C r e w P e rfo rm a n c e In d ic a to rs
6.
In terp erso n al
ob serv ed
re la tio n sh ip s/G ro u p
clim ate.
in te rp e rso n a l relatio n sh ip s a m o n g
is i n d e p e n d e n t o f d e m o n s t r a t e d c o n c e r n
T h is
ev alu atio n
a n d th e o v e ra ll
climate
re fle c ts
th e
q u ality
of
o f t h e f lig h td e c k . T h is
w ith a c c o m p lis h m e n t o f re q u ire d
ta sk s.
E v alu atio n S c a le
I
Poor
Perlorm ence
Significantly
Below Expectation#
2
Mimmelly
Acceptable Performance
Improvement Needed
C rew
3
Satiefactory
or
Standard
Performance
S
Exceptional
Performance
Sigmficentiy
Above Standard
4
Vary Good. Above
A vsregi
Performance
P erfo rm an ce M ark ers
(Bo/d, Italiciied Markers apply to Advanced Technology Flightdecksl
S coring
a.
c r e w m e m b e r s rem ain calm u n d e r stre ssfu l c o n d itio n s
b.
s h o w s s e n s i t i v i t y a n d a b ility t o a d a p t t o o t h e r c r e w m e m b e r s ' p e r s o n a l i t i e s a n d p e r s o n a l
ch aracteristics
c.
r e c o g n i z e s s y m p t o m s o f p s y c h o l o g i c a l s t r e s s a n d f a t i g u e in s e l f a n d o t h e r s (e g . , n o t e
w h e n a c r e w m e m b e r is n o t c o m m u n i c a t i n g , a n d d r a w h i m / h e r b a c k i n t o t h e t e a m ;
re c o g n iz e w h e n th e y are ex p e rie n c in g "tu n n e l vision", a n d s e e k h e lp fro m th e team )
d.
" t o n e " in t h e c o c k p i t is f r i e n d l y , r e l a x e d , s u p p o r t i v e
e.
d u r i n g t i m e s o f l o w c o m m u n i c a t i o n , c r e w m e m b e r s c h e c k in w i t h e a c h o t h e r t o s e e h o w
th e y are doin g
O v e ra ll R a tin g
N o n e /P o o r
C o m m en ts:
I
2
3
4
5
E x c e lle n t
144
N A S A /U T C re w P e rfo rm a n c e In d ic a to rs
7.
P r e p a r a tio n /P la n n in g /V ig ila n c e . T h is ra tin g
tic ip a te
"ahead
c o n tin g e n c ie s
of th e
ap p ro p ria te
curve"
a tte n tio n
and
w h ile
to
ac tio n s
th at
poor crew s
re q u ire d
m ay
in d ic a te s th e
be
re q u ire d .
co n tin u ally
task s
and
p lay
resp o n d
e x te n t to
E x c e lle n t
c a tc h
up.
im m e d ia te ly
w h ich c re w s a n ­
crew s
are
V ig ilan t c r e w s
to
new
alw ay s
d ev o te
in fo rm a tio n .
A
crew indulging in casual social conversation during periods of low workload is not
necessarily lacking in vigilance if flight duties are being discharged properly.
E v alu atio n S cale
I
Poor
Perform ance
Significantly
Below Expecieuons
2
Minimally
A cceptable Performance
Improvement Needed
C rew
Satisfactory
Standard
Performance
Exceptional
Performance
Significently
Above Stenderd
Vary Good. Above
Average
Perform ance
P erfo rm an ce M ark ers
{Bold, Italicized Markers apply to Advanced Technology Flightdecksl
S coring
d e m o n s t r a t e s a n d e x p r e s s e s s i t u a t i o n a l a w a r e n e s s — t h e " m o d e l " o f w h a t is h a p p e n i n g
a.
is s h a r e d w i t h i n t h e c r e w
m o n i t o r i n g o f all i n s t r u m e n t s a n d c o m m u n i c a t i o n s , s h a r i n g r e l e v a n t i n f o r m a t i o n w i t h t h e
b.
re st of th e c re w
m o n ito r clim atic a n d traffic c o n d itio n s , sh a rin g r e le v a n t in f o r m a tio n w ith th e re s t of th e
c.
crew
d.
a v o i d s " t u n n e l v i s i o n " o f s t r e s s - s t a t i n g or a s k i n g f o r t h e
e.
is a w a r e
of f a c to rs s u c h a s s t r e s s th a t c a n r e d u c e
big p ic tu r e
v igilance -
th u s,
m on ito rin g th e
p e rfo rm a n c e of o th er c re w m em b ers
s t a y s " a h e a d o f c u r v e " in p r e p a r i n g f o r e x p e c t e d o r c o n t i n g e n c y s i t u a t i o n s ( i n c l u d in g
f.
a p p r o a c h e s , w e a th e r , etc.)
0
verbally in su re s th a t c o c k p it a n d cab in cre w are a w a re of p lan s
h.
i n c l u d e s all a p p r o p r i a t e c r e w m e m b e r s in p l a n n i n g p r o c e s s
-
/
f0r sufficient time prior to maneuvers for programming of Flight Management
Computer
ensures that all crewmembers are aware of status end changes in FMS parameters
O v e ra ll R a tin g
N o n e /P o o r
C o m m en ts:
I
2
3
4
5
E x c e lle n t
145
N A S A /U T C r e w P e rfo rm a n c e In d ic a to rs
8.
Workload distributed/Distractions avoided. This is a rating of time and workload
management. It reflects how well the crew managed to distribute the tasks and avoid
overloading individuals. It also considers the ability of the crew to avoid being distracted
from essential activities and how work is prioritized.
E v alu atio n S cale
I
Poor
Performance
Significantly
Below Exoectetione
2
Minimally
A cceptable Performance
Improvement Needed
4
Very Good, Above
A v in g i
Performance
3
S»ti»l»ctorv
or
Standard
Performance
S
Exceptional
Performance
Significantly
Above Standard
Crew Performance Markers
[Bold, Italicized Markers apply to Advanced Technology Flightdecksl
S co rin g
a.
c r e w m e m b e r s rep o rt a n d adm it w o rk o v e rlo a d s
b.
a c t i o n is t a k e n t o d i s t r i b u t e t a s k s t o m a x i m i z e e f f i c i e n c y
c.
w o r k l o a d d i s t r i b u t i o n is c l e a r l y c o m m u n i c a t e d a n d a c k n o w l e d g e d
d.
m akes
sure
that
n o n -o p era tio n al
facto rs
such
as
social
interaction
do
not
in terfere w ith n e c e s s a r y t a s k d u tie s
e.
c o m m u n i c a t e s t h e w o r k priorities clearly to t h e c r e w .
f.
m a k e s s u r e t h a t s e c o n d a r y o p e r a t i o n a l t a s k s ( i. e . d e a l i n g w i t h p a s s e n g e r n e e d s ,
c o m p a n y c o m m u n i c a t i o n s ) are prioritized s o a s t o a llo w s u f f ic ie n t r e s o u r c e s for
d e a l i n g e f f e c t i v e l y w i t h p r i m a r y f l ig h t d u t i e s
g.
r e c o g n i z e s a n d r e p o r t o v e r l o a d s in o t h e r s
h.
crewmembers recognize potential distractions posed by Automated Systemss
and take appropriate preventive action, including disengaging
Overall Rating
None/Poor I
Comments:
2
3
4
5 Excellent
146
Cluster D.
Overall Technical proficiency
Item 9. Overall Technical Proficiency. This is a rating of how well the crew as a
unit discharged the technical aspects of the flight. It reflects awareness that a high degree
of technical proficiency is essential for safe and efficient operations. Demonstrated mastery
of CRM concepts cannot overcome a lack of proficiency. Similarly, high technical proficiency
cannot guarantee safe operations in the absence of effective crew coordination. This rating
can be thought of as a more fine grained evaluation of the technical performance of a crew
than the typical "S" or *U* employed in a Line Check or other evaluation. A "5" represents
an unusual demonstration of proficiency while a "I* would reflect seriously substandard
behavior. Refer to the scale definitions in the previous section.
Behavioral Indicators for
Overall Technical Proficiency
(Bold, Italicized Indicators apply to Advanced Technology Flightdecksl
1.
adheres to FAR's and ATC requirements, and follows company established
procedures including checklist management and standard call-outs.
2.
observes and effectively manages sterile cockpit environment .
3.
demonstrates a high level of basic (stick and rudder) flying skills .
4.
required briefings include all pertinent safety and operational issues as defined in the
AOM and FOM.
5.
demonstrates knowledge of aircraft systems and normal, abnormal, and emergency
procedures
Item 10. Overall CREW effectiveness. This item is a composite, global judgment of
the crew's performance taking into account both demonstrated technical proficiency and the
CRM concepts assessed above. To receive a "5", a crew must be at least at the midpoint
("3") on Item 9 and must show clearly above average performance on a number of CRM
items with no "Vs. A crew rated *V overall might receive this on the basis of low technical
proficiency that threatens the conduct of the flight or on the basis of low ratings on one or
more critical CRM items. Reference to the definition of the scale will help in determining
this judgement.
147
NASA/UT LINE/LOS CHECKLIST
L FLIGHT AND EQUIPMENT INFORMATION
Airline_______
LOFT Q
Date (Mo,Yrl______
LOE
Q
SPOT □
LINE CHECK IS or Ul
Scenario ID
ROUTING
R A T E R ID .
Q
PUot flying: CA
A/C Type & S e n e s
D em o g rap h ics
U ne O bservation
|
| FO
□
[
HOURS OBSERVED
C a p ta in
* LEGS OBSERVED
□
F i g h t E n g in eer
F r s t O fficer
C o m p l e t e d I n itia l C R M (Y, N , o r ?)
N um ber of CRM R ecurrent S e ssio n s
N u m b e r of L O S /L O F T S e s s i o n s
C r e w m e m b e r D om icile
F lig h t H o u r s in t h i s A i r c r a f t
F lig h t H o u r s in t h i s P o s i t i o n
A p p ro x im ate A ge
II. CREW EFFECTIVENESS MARKERS
1.
B riefings ( c o n d u c t a n d quality)
N o n e/P o o r
1
2
3
4
5
E x cellent
2.
Inqu iry /A ssertio n /A d v o cacy
N o n e/P o o r
1
2
3
4
5
E x cellent
3.
C r e w se lf-critiq u e ( d e c isio n s an d a c tio n s)
N o n e/P o o r
1
2
3
4
5
E x cellent
4.
C o m m u n icatio n s/D ecisio n s
Poor
1
2
3
4
5
E x cellent
5.
L e a d e rsh ip -F o llo w e rs h ip /C o n c e rn for ta s k s
Poor
1
2
3
4
5
E x cellent
6.
In te rp e rs o n a l re la tio n sh ip s/G ro u p clim ate
Poor
1
2
3
4
5
E xcellent
7.
P re p ara tio n /P lan n in g /V ig ilan ce
Poor
1
2
3
4
5
E xcellent
8.
W o rk lo a d d istrib u te d /D istra c tio n s avoided
Poor
1
2
3
4
5
E x cellent
9.
O v e ra ll T E C H N IC A L p r o fic ie n c y
Poor
1
2
3
4
5
E x cellent
O verall C R EW e ffe c tiv e n e s s
Poor
1
2
3
4
5
E xcellent
Low
1
10.
III. SUPPLEMENTARY INFORMATION:
11.
E n v iro n m en tal w o rk lo a d
2
3
4
5
H igh
O th e r c o n d itio n s w h ic h sig n ific a n tly in flu e n c e d th e flig h t (in clu d e w e a t h e r , A T C , p re -e x is tin g m e c h a n ic a ls ,
in - f l i g h t a b n o r m a l e v e n t s , IO E, e x t r a c r e w m e m b e r s , e t c ).
D escribe below .
148
IV. SPECIAL CIRCUMSTANCES:
T h is s e c t i o n p r o v id e s d a t a o n n o n - s ta n d a r d s itu a tio n s or
b e h a v io rs t h a t m a y in flu e n ce c r e w p e rfo rm a n c e .
If c o n f l i c t s o c c u r , r a t e h o w e f f e c t i v e l y t h e y w e r e r e s o l v e d .
12.
Severity of abnormal or emergency situation
Low
1 2
13.
C onflict resolution
Poor
1
2
3 4 5 High
3
4
5
E xcellent
In s o m e c a s e s t h e a c t i o n s o f a p a r t i c u l a r c r e w m e m b e r m a y b e p a r t i c u l a r l y s i g n i f i c a n t t o t h e o u t c o m e o f t h e
f l i g h t . In c a s e s w h e r e t h i s h a p p e n s , e n t e r t h e r e l e v a n t i t e m n u m b e r f r o m a b o v e , c h e c k t h e p o s i t i o n o f t h e
c r e w m e m b e r in v o lv e d , a n d circle th e ratin g a s s i g n e d .
D
Item No.
□
Item No.
□
Item No.
□
□
C a p ta in
□
C a p ta in
D
C a p ta in
D
Item No.
C a p ta in
V. COMMENTS:
in S e c t i o n II.
Item #
D
First O fficer
□
F irst O fficer
D
First O fficer
D
First O fficer
D
Poor
I
2
3
4
5
E xcellent
D
Poor
I
2
3
4
6
E xcellent
D
Poor
I
2
3
4
5
E xcellent
D
Poor
I
2
3
4
5
E xcellent
E n g in e er
E n g in e e r
E ngine er
E n g in e e r
D e s c r ib e a b n o r m a l or e m e r g e n c y c o n d i t i o n s , c o n f lic ts , o r individual b e h a v i o r s r a te d
A l s o c o m m e n t o n e x t r e m e (I o r SI r a t i n g s f r o m S e c t i o n II.
Comments
Appendix D
The CRM Survey
150
P le a s e f i l l out t h i s sho rt survey to g iv e us your o p in io n about the
u s e fu ln e s s o f LOPT In g e n e ra l and o f the LOFT sce na rio s you have Just
com pleted.
This survey Is p a r t o f the NASA-sponsored research e v a lu a t in g
the ln p a c t o f Cockpit Resource Management t r a i n i n g .
The q u e s tio n n a ire is
anonymous and w i l l be used to develop summary s t a t i s t i c s and recommendations
r e g a r d in g CRM and LOFT.. , Please p la ce the con p leted f o r a in the envelope
p ro v id e d and place i t in Company n a i l .
A i r c r a f t _____________
LOFT f r o n _____________
to _____________
' I n f l i g h t problem ______________________ ______________________________________________
P o s itio n :
___ CA
_
PO
___ FE
Conpleted CRM Seminar?
How nany LOFT sessions have you completed b e fo re today?
1.
•_____________________
aspect o f i t .
p ro b le n and s c e n a r io .
w it h p ro b len and s c e n a r io . .
p ro b len and s c e n a r io .
I f you in d ic a t e d sone awareness o f the s c e n a r io , p le a s e check the
s ta te m e n t below which is c lo s e s t to your o p in io n .
_____ T h is awareness g r e a t l y reduced the t r a i n i n g v a lu e .
_____ T h is awareness s l i g h t l y reduced the t r a i n i n g v a lu e .
_____ T h is awareness had no e f f e c t on the t r a i n i n g v a lu e .
2.
How r e a l i s t i c was the scenario ?
U n re a lis tic In
e very way
3.
1 2
3
( C i r c l e a nunber on the s c a le )
4
S
6
7
R e a l i s t i c in
e v e ry way
4
3
6
7
E x trem e ly
d iffic u lt
How d i f f i c u l t was the scenario?
Extremely
easy
4.
___ NO
Were you aware o f the n a tu re o f the s c e n a rio and p ro b le n in t h i s LOFT?
_____ No l n f o r n a t l o n about any
_____ S l i g h t f a n l l l a r l t y w it h
_____ C onsiderable f a n i l l a r i t y
_____ D e t a i l e d in fo rm a tio n on
la .
___ YES
1 2
3
How w e l l d id your crew' perform the mission?
Extremely
poor
1 2
3
4
3
6
7
Extrem e ly
w e ll
151
3.
Hot# w e l l d id you p e r s o n a lly p e r fo rn ?
Extrem ely
poor
6.
4
3
6
Extrem ely
w ell
7
1
2
3
4
5
6
7
C o a p le te ly
u s e fu l
O v e r a l l , how would you r a t e the t e c h n i c a l t r a i n i n g v alu e o f t h i s LOFT
sessio n ?
C o n p le t e ly
useless
8.
3
O v e r a l l . how would you r a t e th e v a lu e o f t h i s LOFT session f o r crew
c o o r d in a t i o n t r a i n i n g ?
. C o a p le te ly
useless
7.
1 2
1
2
3
4
5
6
7
C o a p le te ly
useful
O v e r a l l , how nuch have you le a rn e d In t h i s s i n u l a t l o n t h a t you w i l l
a c t u a l l y use on the lin e ?
A b s o lu t e ly
n o th in g
1
2
3
4
5
6
A trenendous
amount
7
Use a number fro n the f o l lo w in g s c a le t o I n d i c a t e your l e v e l o f agreement
w i t h each o f the f o l lo w in g s ta te m e n ts .
W r i t e the number in the space t o the
l e f t o f th e s ta te m e n t.
I
S t r o n g ly
D is a g re e
9.
2
D isagree
3
S lig h tly
Disagree
4
N eu tral
3
S lig h tly
Agree
6
Agree
7
S tro n g ly
Agree
Our crew r e a l l y worked as a team.
D u rin g the LOFT we spent f a r too nuch t i n e t a l k i n g or a rg u in g .
D e a lin g w it h members o f the crew l e f t me f e e l i n g i r r i t a t e d and
fru strate d .
Our crew p r a c t ic e d I n q u i r y / q u e s t I o n l ng.
The C ap tain made most o f the d e c is io n s about our f l i g h t w it h o u t
i n v o l v i n g o th e r crewmembers.
Our crew p r a c t ic e d a d v o c a c y /a s s e r tio n .
Our crew shared r e s p o n s i b i l i t y f o r i t s
Our crew p r a c t ic e d s e l f - c r i t i q u e
leadership'.'-'
o f d e c is io n s and a c tio n s .
152
_____
17.
Our LOFT I n s t r u c t o r was knowledgeable and h e l p f u l .
_____
18.
The d e b r i e f i n g f o l l o w i n g t h e LOFT was h i g h l y u s e f u l f o r a l l
crewmembers.
_ _ _ _ _ 19.
The videotape o f the LOFT p ro vide d Im portant feedback to the crew.
_____
O v e r a l l . LOFT is an e x tre m e ly u s e fu l t r a i n i n g t e c h n i q u e .
20.
P le a s e d e s c r ib e what you found Dost u s e fu l about the LOFT.
P le a s e d e s c r ib e anything t h a t you f e e l n ig h t be done to in c r e a s e the
u s e f u ln e s s o f LOFT.
Appendix E
Communications Analysis
154
COMMUNICATIONS ANALYSIS CODES
1.
Commands/Command Decisions
2.
Observations/Supplying Unsolicited Information
3.
Suggestions
4.
Statements of Intent
5.
Inquiries/Requests for Clarification/Requests
6.
Agreements
7.
Disagreements
8.
Acknowledgements
9.
Answers supplying information
10.
Response uncertainty
11.
Tension release
12.
Frustration/anger/derisive remarks/surprise
13.
Embarrassment
14.
Repeats
15.
Checklist
16.
Non-task related
17.
Non-codeable
18.
ATC communications
19.
Talking to self aloud
20.
Incomplete/interrupted communication
Foushee, H. C. & Manos, K. L. (1981). Information transfer within the cockpit. In
C. E. Billings and E. S. Cheaney (Eds.) Information transfer problems in the
aviation system. (NASA Technical Paper 1875). Moffett Field, CA: NASA
Ames Research Center.
Foushee, H. C , Lauber, J. K., Baetge, M. M., & Acomb, D. B. (1986). Crew factors
in flight operations: III. The operational significance of exposure to short-haul
air transport operations. (NASA Technical Memorandum 88322). Moffett
Field, CA: NASA Ames Research Center.
Appendix F
Lessons-Learned
156
AVS-404
Lessons-Learned
Names:
■
______________________ & _____
Lesson:____________________________________
We learned that from:
Lesson:____________
We learned that from:
Lesson:___ ________
We learned that from:
Lesson:____________
We learned that from:
Lesson:
We learned that from:
_________
M
O
N
T
A
N
A
S
T
A
T
E
U
N
IV
E
R
S
IT
Y
L
IB
R
A
R
IE
S
3
1762
10269131
6