Lundberg, J. (2002): Operationalising civil pilot's process of understanding
instrument failure events. In Humans in a complex environment Proceedings
of the 34th Annual Congress of the Nordic Ergonomics Society, Kålmården,
Sweden, 569-574.
Operationalising civil pilot’s process of understanding
instrument failure events
J. Lundberg
Department of Computer and Information Science
Linköpings Universitet
SE-581 83 Linköping, Sweden
E-mail: jonlu@ida.liu.se
Abstract
Instrument failure in civil aircraft is potentially very dangerous. It is
therefore imperative to understand where breakdowns in the distributed
cognitive process of understanding occur in the cockpit. In this paper I
present a model of distributed situational understanding using eye-point-ofgaze equipment. The model includes a range of constructs, from perceivable
stimuli in the cockpit, to eye-point-of-gaze data, to situational awareness,
and finally understanding and action. To test the model, in series of
simulated flights, conducted in the VINTHEC project, experienced pilots
were presented with situations of instrument failure. In the experimental
setting, the pilot’s understanding was found to be adequate to deal with an
altitude bust event, but not with a map shift event. It is concluded that the
complicated situational awareness construct does not contribute to the
model, whereas pilot actions and eye-point-of-gaze data are important
measuring points for understanding the pilots’ process of understanding.
Keywords
Situational awareness, eye-point-of-gaze
1 INTRODUCTION
Flying a civil airliner today is a semi-automatic process, where the pilots to a large
extent interact with and monitor automations that directly manoeuvre the plane. To
support them, there are other automations, such as altitude warning systems and
electronic maps. In these systems, breakdowns may occur in steering automation, in
advanced automated instrumentation, in the interaction and in the understanding of
events by and between pilots and flight control. To fly the plane, the pilots must
understand what goes on in the systems, and the situation that the plane is in. They have
to understand whether any anomalities are present in the system, and they have to
predict the future situations based on the currently available information, to ensure that
the plane doesn’t get into a situation were there are no possible actions left, to ensure
safe flight. It has been claimed that this is best understood as the situational awareness
(SA) of the individual pilots or as the system of humans and artefacts mediating the
information that distributes and coordinates the awareness of team members. The SA
can be about the current situation, about the predicted future, far or near. It can also be
a retrospective awareness about past situations. (Endsley 1988; Jones and Endsley,
1996; Artman and Garbis, 1998). This paper addresses the problem of maintaining SA
in the cockpit by proposing a model starting at the perceivable stimuli in the cockpit,
indicating normality, abnormality and presenting situational data, continuing to the SA
of the pilots, resulting in understanding and action. The SA is here operationalised as
eye-point-of-gaze data, and action performed or omitted by the pilots. This
operationalised model is applied to data from a series of ten flights conducted in the
VINTHEC (Visiual interaction in the cockpit) project, reported in Lundberg (1999).
Furthermore it is compared to the statements of the pilot’s of their subjective view on
their SA, to see whether they are aware of their level of SA.
The question of this paper is whether the SA construct contributes to the model of
pilot’s understanding in the cockpit, and whether the operationalised model contributes
to an explanation of the events in the experimental setting.
1.1 Operationalising situational awareness
Seen from the individual, SA corresponds to Neissers (1976) model of the perceptual
cycle. In its simplest form, it consists of three entities, an object, a schema and
perceptual exploration. The object is in this case the world, the schema is in most cases
a mental situation model, and perceptual exploration is the sampling of sensory data
from the world. In Neisser’s (1976) model, the schema directs perceptual exploration,
causing the components of the situation model to change. To catch the eye movements
involved in visual perceptual exploration, eye-point of gaze equipment can be used
(VINTHEC WP 4). It measures what a human is looking at. It measures a time history
of fixations, where the environment is divided into areas of interest. The equipment
records the time, duration and position of fixations on these areas. For each fixation,
some EPOG measurement equipment may measure pupil diameter and blink rate as
well. The pupil diameter and blink rates are usually used as measures of workload. For
a thorough discussion about the use and calibration of EPOG equipment, see Alfredson
(2001). Several EPOG equipped pilots in the same environment give rise to parallel
time histories. These can be compared, to find differences in the time distribution of the
attended areas, and to examine whether simultaneous dwells on the same area occurs.
Dwelling on an area with important information does not necessarily mean that the
situational awareness of the dweller has been updated. However, both not having
looked at an instrument providing important information, and looking at it, but either
not attending to it, or not becoming aware of vital changes to current or projected future
situations, indicates a SA problem in the cockpit.
The correctness of action based on information may be in indication of levels of
comprehension, and the quality of SA, providing that SA is required for correct action.
However, high performance is not always accompanied by high SA. System
performance with for example an active autopilot may be excellent, whereas pilot SA
may be quite low (Tenney et. al, 1992; Shively and Goodman, 1994). As discussed,
when performance of cognitive functions and performance of relevance processes are
distributed from human to machine, performance may improve, whereas pilot SA may
decrease (Shively and Goodman, 1994). Indeed, it has been claimed that SA does not
cause anything since it is merely an abstraction (Billings, 1996).
1.2 Limitations of the operationalisation
The operationalisation presented here focuses on the interdependencies between
instrumentation (including automation), visual perception, SA, and action.
Instrumentation can be seen as a substitute for short-term memory, as Mogford (1997)
states, being sampled on demand, to use the human working memory optimally.
However, verbal interaction is a valuable source for SA. (Heath and Luff, 1992; Jones
and Endsley, 1996; Artman and Garbis, 1998). Using the operationalisation presented
here, verbal interaction must be controlled, or included in the analysis. Pilot interaction
is operationalised as simultaneous or divided visual sampling of visual data. Thus,
group SA is operationalised as the actions of the team of pilots and the way they divide
visual sensory exploration. What is brought forward as relevant SA here thus is
awareness about changes in the current situation presented by instruments. It includes
expected and unexpected changes, both relating to the functioning of the equipment and
to the flight of the airplane. Other conceptualisations bring forth other things as relevant
SA, and which to use must always depend on the situation under study. Indeed, a
survey presented by Fracker (1988), present several different definitions of SA, and
several new ones have emerged since the time of that survey. Thus, the distributed
cognitive process of understanding in the cockpit is operationalised as the co-ordinated
perceptual exploration of the pilots, and of their action or lack of action. From this their
SA may be induced.
2 THE EXPERIMENT
2.1 Test site
The experiment was performed at the NLR (National Aerospace Laboratories) in
Amsterdam, the Netherlands.
2.2 Participants
The participants were 20 active pilots normally flying civil passenger jets, most of them
familiar with Boeing 737 procedures. Their flight experience ranged from 2000 to
20000 flight hours. In the pre-experiment questionnaire, no pilot reported any activities
within twenty-four hours, judged as likely to interfere with experiment behaviour.
2.3 Procedure
The pilots were divided into ten crews of pilot and co-pilot. Each crew flew two
simulated flights, from Amsterdam to London, and then back again. The entire flights,
except landing and takeoff were simulated to be above thick clouds, to avoid relevant
information to appear on the outside view. Before each flight, the briefing questionnaire
was handed out. During each flight the four subjective questionnaires were handed out.
After each flight a debriefing questionnaire was handed out.
Before each flight, the in flight observer gave the pilots the flight log. The log
described nothing of consequence for the functionality of the aircraft for the first,
Amsterdam-London, flight. For the second flight, the log described an autopilot failure
where the autopilot failed to capture the selected altitude.
There were four abnormal events in the experiment, two occurring in each flight, in
same order for each crew. A map shift event occurred during the cruise phase in the first
flight. It was presented to the pilots as an unlikely wind strength on the map display,
which in the simulation resulted in the map misrepresenting the position of the aircraft.
A gear unsafe event occurred during the landing of the flight. This was shown as a
missing green light, of the three lights for the landing gear, indicating that the landing
gear has not been safely extended. An altitude bust occurred at the end of the climb in
the second flight. This was constructed as a failure of the autopilot to catch the selected
altitude. It could be seen as a continued rising altitude on the altitude indicators. The
fourth and final event occurred during the London-Amsterdam landing. The pilots were
given a flap asymmetry warning, consisting of a warning light, a sound, and a flap
asymmetry warning text on the engine display.
Regarding SA, the pilots were warned in the techlog, that the autopilot had failed to
capture the desired altitude during a previous flight, and the gear unsafe was practised
during the practice flights, making these two events high SA events because of the
preparedness. The map shift and flap asymmetry were neither practised nor prepared
for, making them low SA events. In this paper the altitude bust event and the map shift
events were selected for analysis.
2.5 Equipment
The Fokker 100 simulator was based on a full-scale multipurpose flight simulator with
motion. The simulator had seats for two pilots and one in flight observer. Two headsets
were used for the EPOG data collection, one for each pilot. For the EPOG data
collection, the cockpit was divided into ten areas of interest, and the marker signals for
the events and flight phases of the experiment were attached to the timeline of the copilot. Pupil diameter was used as an indicator of mental workload while eye blink rate
indicated the visual workload. Video and audio was also recorded from the cockpit.
2.3 Results
The evaluation of the EPOG calibration data resulted in the exclusion of two crews
from the analysis. Furthermore, the analysis of the remaining pilots showed the desired
high between-area point of gaze accuracy. The within-area precision varied between
pilots, and since some pilots had too low precision, no within area data is used here.
Since the simultaneous dwells measures method was not valid on data with the time
line synchronisation accuracy obtained in this experiment, this data was not used either.
Mean attention on OUT (AL)
80
60
40
Value
20
MEAN_PNF
MEAN_PF
0
24.00
Segment
25.00
26.00
27.00
28.00
Figure 1. Division of attention on the outside view. (climb - 24) ( cruise - 25) (descent 26) (approach - 27) (landing - 28) (PF - Pilot flying) (PNF - Co-pilot)
There was a significant mean dwell time difference between the crew members for the
landing flight segment on the Amsterdam-London flight (Dunnett C: df = 9, F =
123.456, p < 0.05) (see Figure 1). (A non-significant trend to the same effect was
visible for the London-Amsterdam flight.) In Figure 5, the attention during the climb
(24), the cruise (25), the descent (26) the approach (27) and the landing (28) can be
seen on the Y axis, as the percentage of time the eyes of each pilot were focused on the
outside view.
For the altitude bust event -> +2 min, the groups of pilots were divided using the
performance measure, time to correct the altitude bust. There was a significant
difference between the groups regarding the percentage of time spent on the primary
flight display, the relevant area of interest (ANOVA: df = 1, F = 7.105, p < 0.05). When
further divided into groups of pilots flying and co-pilots, there was a significant dwell
time percentage difference for the co-pilots (ANOVA: df = 1, F = 8.038, p < 0.05).
For the map shift event, no significant correlation was found. However, the pilots did
spend time looking at the display during the event.
3 DISCUSSION
Clearly, the cockpit crew has a division of attention, as the analysis of the landing
segment shows. Attention furthermore affects situational understanding, and SA, as the
handling of the altitude bust event shows. For this event, the amount of time spent by
the co-pilot on the relevant display correlates with swift action. However, for all pilots
the distributed system of situational understanding, having the aircraft log, as mediator
of retrospective SA, was sufficient to handle the event. The map shift event, in contrast,
was not recognized, or recognized after some time. Since all crews attended to the
relevant display, clearly they either did not understand it, or they did not notice the
relevant data in it. The system supporting understanding here was not satisfactory, and
either training is necessary to learn to handle the event, or the display must be
redesigned to make this kind of error more salient. Regarding SA, in this experiment
deciding whether the pilots are aware of situational aspects of the situation, current or
projected, is less important than their perception, understanding and action. Thus, what
has been studied here is the process of distributed situational understanding. And rather
than searching for breakdowns in awareness, here the focus has been breakdowns in
visual perceptual exploration, and the salience and comprehensibility of the visual data.
This corresponds to Billings (1996) statement, that SA is a theoretical construct, which
does not cause anything. This study indicates that the question of pilots’ awareness is
irrelevant, indeed the concept of SA is irrelevant, when discussing situational
understanding, perception and action, whereas Neissers (1976) model is highly
relevant.
For this experiment to be valid the experimental setup was crucial, since these results
rely on an assumption of realistic eye scan patterns. This motivates having relatively
few real pilots in a simulator environment, rather than having more subjects without
pilot training in a less realistic setting.
4 CONCLUSION
It is concluded that the complicated SA construct did not contribute to the model. It was
important whether the pilots had perceived the information, and whether they
comprehended it. It was neither important whether the pilots were aware of the events
at the time of their occurrence, nor when they were aware of them as they acted to
correct them (this instead was rather obvious).
Applying a commonsense notion of SA, an “awareness” of what was going on, was
necessary to act correctly in all situations. It is concluded that the map shift could not
be easily detected due, to its manifestation on the map, which should be redesigned if it
is considered important to detect this event, when the computer measurement of wind
strength malfunctions. Furthermore, it is concluded that attending more to the primary
flight display was sufficient to discover this event more swiftly, in particular for the copilots.
References
Alfredson, J. (2001): Aspects of situational awareness and its measures in an aircraft
simulation context. Licentiate of engineering thesis: Linköping Institute of
Technology, Division of Industrial Ergonomics, Linköping.
Artman, H. and Garbis, C. (1998): Situation Awareness as Distributed Cognition. In T.
Green, L. Bannon, C. Warren, J. Buckley (Eds): Cognition and cooperation. In
Proceedings of 9th Conference of Cognitive Ergonomics, Limerick, 151-156
Billings, C. E. (1996). Situation Awareness Measurement and Analysis: A
Commentary. In D. J. Garland, M. R. Endsley (Eds): Experimental Analysis and
Measurement of Situation Awareness. Embry-Riddle Aeuronautical University
Press, Daytona Beach, FL
Endsley, M. R. (1988): SA Global Assessment Technique (SAGAT). In Proceedings of
the IEEE National Aerospace and Electronics Conference.
Fracker, M. L. (1988): A Theory of Situation Assessment: Implications for Measuring
SA. In Proceedings of the Human Factors Society -32nd Annual Meeting, Anaheim,
CA, 102-106.
Heath, C. and Luff, P. (1992): Collaboration and Control - Crisis management and
multimedia technology in London underground line control rooms. CSCW - An
international journal, 1, 69-94.
Jones, D. G. and Endsley, M. R. (1996). Sources of SA in Aviation. Aviation, Space
and Environmental Medicine, 67, 507-512.
Lundberg, J. (1999). Distributed Situation Awareness for the Civil Cockpit: Theory and
operationalisation. Masters thesis in social science: Linköpings Universitet, LIUKOGVET-D-0013-SE,
Mogford, R. H. (1997): Mental Models and SA in Air Traffic Control. The
Internaltional Journal of Aviation Psychology, 6(4), 331-341.
Neisser, U. (1976): Cognition and Reality: Principles and Implications of Cognitive
Psychology. W H Freeman and Company, San Fransisco.
Shively, R. J., and Goodman, A. D. (1994): Effects of Perceptual Augmentation of
Visual Displays: Dissociation of Performance and Situational Awareness. In
Proceedings of the Human Factors and Ergonomics Society 38th Annual Meeting,
Nashville, TE, 1271-1274.
Tenney, Y.J., Adams, M. J., Pew, R. W. Huggins, A. W. F, and Rogers, W. H. (1992):
A Principled Approach to the Measurement of SA in Commercial Aviation. NASACR-4451
VINTHEC Consortium. (1998). Testplans for Pilot Experiments. Technical Report:
NLR Amsterdam, VINTHEC-WP4-TR 01.