Distributed Situation Awareness for the Civil Cockpit: Theory
and Operationalisation
Abstract
This thesis combines the theoretical frameworks of internal human
cognition with the framework of distributed cognition for the civil
cockpit, to create a combined situation awareness theory of distributed
situation awareness. The theory is operationalised by being related to
measurement methods. An experiment was conducted to test the theory
and the distribution of attention in the cockpit as well as the adequacy of
the situation awareness system in an altitude bust anomaly. The theory
was found to be adequate for the operationalisation of the situation
awareness system of the experiment. Distributed attention was found to
exist for the landing phase of one flight, and the situation awareness
system was found to be adequate for safe handling of the altitude bust
event, but inadequate for the safe handling of the map shift event.
Reference notice:
Lundberg, J. (1999). Distributed Situation Awareness for the Civil
Cockpit: Theory and operationalisation. LIU-KOGVET-D-0013-SE,
Linköpings Universitet, Linköping.
i
Table of contents
1. Introduction.......................................................................................................... 1
2. Theory .................................................................................................................. 3
2.1. Theoretical grounding of the SA system ....................................................... 5
2.1.1. SA as a product....................................................................................... 5
2.1.2. Updating SA ........................................................................................... 7
2.1.3. Examples of distributed SA.................................................................... 9
2.1.4. Compatibility with SA definitions of the literature .............................. 10
2.2. The proposed SA theory.............................................................................. 13
2.2.1. Situation models, mediations and function........................................... 14
2.2.2. Updating and sampling processes......................................................... 15
2.2.3. The presentation process....................................................................... 16
2.2.4. The monitoring process ........................................................................ 17
2.2.5. The relevance process........................................................................... 18
2.2.6. Properties of distributed and localised systems .................................... 19
2.2.7. SA Error................................................................................................ 20
2.3. Measurement techniques for SA systems.................................................... 22
2.3.1. Eye point of Gaze ................................................................................. 22
2.3.2. Questionnaires ...................................................................................... 23
2.3.3. Physiological measures......................................................................... 23
2.3.4. Event-related potentials ........................................................................ 24
2.3.5. Transcripts of verbal messages ............................................................. 24
2.3.6. Performance measures .......................................................................... 25
2.4. Theory summary and discussion ................................................................. 26
2.4.1. Concluding operationalisation model for SA ....................................... 26
2.4.2. Relevant SA.......................................................................................... 27
2.4.3. Safe distributions of labour between actors and artefacts..................... 28
2.4.4. Safe practices, tradition and training .................................................... 28
2.4.5. Optimal SA distribution and component failure................................... 29
2.4.6. Consequences for design ...................................................................... 30
3. The experiment .................................................................................................. 34
3.5. Method ........................................................................................................ 34
3.5.1. Test site................................................................................................. 34
3.5.2. Participants ........................................................................................... 34
3.5.3. Procedure .............................................................................................. 34
3.5.4. Design................................................................................................... 37
3.5.5. Equipment............................................................................................. 37
3.5.6. Research questions ............................................................................... 38
3.5.7. Operationalisation of the research questions ........................................ 39
3.6. Measurement results.................................................................................... 42
3.7. Conclusions ................................................................................................. 44
4. General discussion ............................................................................................. 47
5. References.......................................................................................................... 49
6. Appendix A........................................................................................................ 52
7. Appendix B ........................................................................................................ 55
ii
1. INTRODUCTION
To have situational awareness (SA) has traditionally been to know what
is going on, and to predict what will happen. Recent developments of the
study of SA has presented a more complex, yet still simple and
straightforward view of what it is to have good SA. It is now widely
accepted to view SA as a dynamic process rather than a product. The
next step has been to view SA as something localised to the individual,
yet co-ordinated between, built in concert with, and maintained in, a
team of actors and tools. Tool free environments are very scarce, and so
are environments where there is just one actor, making these issues
essential to a theory of SA. To have good SA, then, is now to know what
is going on, how that came to be, and to predict what will happen in the
near and more distant future. For a system of actors and tools, to have
good SA is to have a sufficiently redundant SA divided over the actors,
to have accurate SA, and to have a good knowledge of the quality of the
SA in the system. To have an environment for good SA, is to have a safe
distribution of labour, with safe practices for maintaining SA, between
actors and tools.
The aim of this thesis
The aim of this thesis is to operationalise the concept of situation
awareness (SA) for individuals in co-operation with each other in a
dynamic environment, the civil cockpit. In the first part of this thesis I
will present a theory and a descriptive framework for SA systems with
actors and artefacts. It is common to discuss how to operationalise the
concept of SA, how to relate theory to measurement tools useable in real
and experimental situations. This will be done in this thesis as well, to
anchor the theory in practice. The model will finally be applied to an
experimental flight situation, to evaluate the SA in the situation, and to
evaluate the suitability of the theory to a practical problem.
VINTHEC
This thesis is made in co-operation with the VINTHEC (Visual
interaction in the cockpit) project, sponsored by the European
Commission. The VINTHEC project is a joint research effort between
companies1 in several countries. The project aim is to clarify the concept
of situational awareness and to test and evaluate measurement tools for
1
National Aerospace Laboratory (NL) British Aerospace (UK), Defence Evaluation & Research
Agency (UK), National Defence Research Establishment (SE), Luleå University of Technology (SE),
Saab AB (SE), H.A. Mooij Holding B.V (NL)
1
situational awareness, pilot workload and performance in simulation
environments.
2
2. THEORY
After this short introduction to SA, there is a chapter reviewing SA
theory from the literature. In the next chapter the combined theory of
localised and distributed SA, which emerged from the review, will be
described. Then I will present a review of measurement methods for SA.
The theory part of this thesis will end with a summary and a discussion
of the proposed SA system.
Situation awareness (SA)
SA is in short knowledge of what is going on. It is conscious knowledge
of what is happening, what has happened, what will happen if, and what
will happen if not. But SA is not the ability or knowledge of how to act.
Since SA incorporates awareness it is not automatic, unconscious use of
data. As part of SA an estimation of the quality of the awareness can be
included, an estimation of its correctness and completeness. But since
this can also be excluded form the concept of SA, it is simply termed
meta-SA, to distinguish quality from content.
Device models, to know how some device works, or how a human will
act in a given situation, is not part of SA. However, to have an accurate
device model is essential to get SA, since if one knows how some
artefact or human will act in a situation, then it is possible to predict what
will happen, in short, to get SA of future states.
SA consists of, first, unstructured aspects of the dynamic situation, such
as for an aircraft the current airspeed and the current weight. Second, SA
is made of what gives the aspects meaning, situation models, such as the
landing situation model.
Active situation knowledge (ASK)
For devices, it is pointless to discuss awareness. Instead devices can
actively process data about a dynamic situation, thus devices can have
active situation knowledge. Since the quality and number of situations a
device recognises and handles may vary, the weaker term active situation
knowledge (ASK) will be used for devices taking part in the SA system.
By using a different term for mechanical information processors
executing SA processes, simple mechanical agents can be recognised as
contributing to the SA of the system without attributing full fledged
awareness, equivalent to human awareness, to them.
Mediation
3
SA or ASK is always mediated, either in a mental model, such as in the
human working memory, or in a physical media, such as a display of
some kind
Situation assessment
Furthermore, the maintenance of SA relies on several processes.
Together these are called situation assessment processes. First, there is an
updating process, responsible for the maintenance of the model, to keep
it up to date. Second, there may be a presentation process, which presents
the model for some other agent to perceive. Third, there may be a
monitoring process, which evaluates whether the two processes work,
and whether the SA is adequate.
Scope of this thesis
The model of this thesis can be used to model the distribution of
processes between agents, by what physical medium the situation models
and aspects are mediated. But the physical possibilities for relevant
action, the exact properties of the physical medium used, and the
physical possibilities for manipulation of the situation models, mental or
physical, are beyond the scope of this thesis. These properties can and
should be discussed with the resulting situation model for which they are
relevant, but the properties are not described in this thesis, but have to be
found elsewhere.
4
2.1. Theoretical grounding of the SA system
The objective of this chapter is to ground the theoretical model presented
in the subsequent chapters in the models and observations of SA related
research.
2.1.1. SA as a product
Endsley (1988, 1990; Jones & Endsley, 1996) proposes three
components, or levels, of SA, the first level being knowledge of aspects
of the environment, the second being comprehension of the meaning of
the aspects, and the third being projection of future events.
Related to the model of this thesis, the first level aspects can be found as
non interpreted, that is not as parts of any situation model, or as
interpreted, that is as parts of one or more situation model. The second
level of comprehended models, can be the result of comprehending
elements of the situation, or comprehending a represented situation
model. The level three, future events can be the result of an internal
projection process or the result of comprehending an external
representation of future situations.
Artman and Garbis (1998) propose three functional types of SA models,
contemporary (current states) SA, prospective (future states) and
retrospective (past states) SA. The current states and future states SA
corresponds with the level two and level three SA of Endsley’s (1988)
model, whereas the past states SA is added. The focus of Artman and
Garbis study is on the usage and mediation of SA in artefacts.
Device models
The importance of a correct device model for pilots of passenger jet
aircraft is illustrated by an auto pilot failure incident (Tenney, Adams,
Pew, Huggins, Rogers, 1992). As one engine of the aircraft failed, and
the aircraft started to turn, the auto pilot started to compensate for this by
turning the control wheel of the aircraft to keep the programmed flight
path. Oblivious to this tug-of-war between the aircraft and the autopilot,
after four minutes, the pilot took manual command of the wheel and
turned of the autopilot. Instantly the aircraft won the tug of war, and was
only stabilised after a fall of 10 000 feet. This illustrates how an
inaccurate or missing device model of what was going on, contributed to
5
an incident when the control was transferred from automated equipment
to a crew member.
Hutchins (1990) describes how the crew of the ship form a chain of
information processing, where the simple perception is done at one end,
and the highest level interpretation is done at the other end. The human
components of this processing chain, as individuals start in the
perception end of the chain, and may move up the chain of processing
only when the current task is mastered. This gradual ascent of the chain
of processing causes the individual to know what has been done with the
information received, and how it has been processed. Thus, the
individual has a device model of the team navigation chain covering all
steps before him or her.
Meta-SA
In the VINTHEC report (forthcoming) SA is described as consisting of
first, all the information that concerns the dynamic situation. Then there
is the subset of information about the situation that a person is aware of.
This can be compared with the subset of information that is required to
be fully aware of what is relevant in the current situation. The area
covered both by the pilot awareness and required awareness constitutes
the actual SA.
From the VINTHEC model, the relative awareness can be seen as the
quotient between the required awareness and the actual awareness. Then,
the estimation of the relative awareness by the pilot is called the
awareness of relative awareness. It may have four outcomes, a high SA
with a correct estimation, a low SA with a correct estimation, a high SA
with an underestimation of SA, or a low SA with an overestimation of
SA. This awareness is what is in this thesis called meta-SA, and the
result of some monitoring process.
Summary
SA consists of six kinds of components, models of present events,
models of future events, models of past events, and knowledge of aspects
of the environment. Device models are knowledge about how actors and
tools function. This is not SA, but essential to achieve SA, especially
when the functionality of actors or tools change. A sixth component of
SA is made of meta-SA, awareness about the quality of the SA.
6
2.1.2. Updating SA
In its simplest form, the perceptual cycle (Neisser, 1976) consists of three
entities, an object, a schema2 and perceptual exploration. The object is in
this case the world, the schema is in most cases a mental situation model,
and perceptual exploration is the sampling of sensory data from the
world. In Neisser’s model, the schema directs perceptual exploration,
causing the components of the situation model to change.
Endsley (1988, 1990; Jones & Endsley, 1996) presents SA as emerging
in three levels, the first level being perception of elements in the
environment, the second being comprehension of the situation, and the
third being projection of future status. This process is guided by
expectation, by preconceptions, objectives, experience and training and
restricted by the ongoing task workload.
In this thesis, the level three projection does not have to be mental
projection, it can be mechanical projection as in the case with the GPWS
(Bateman, 1997). The resulting situation model can then be presented for
another actor to perceive.
The model of this thesis is partly analogous to the Wellens (1993)
extended dynamic decision-making model for group decision making. In
the model for the single actor, all raw data, including current and past
events makes an information space. Integration and assessment creates a
situation space. Decision rules then select action from an action space
consisting of all behavioural options, which can effect the environment.
A feedback loop is then connecting the action space and the information
space. Several decision-making units can be combined via an information
bridge.
Compared to the model of this thesis, the information space most closely
resembles the unordered situation aspects. Wellens also includes events
among the aspects; in this thesis events consist of groups of aspects, and
are not separate entities. The integration and assessment component, is in
this thesis the updating processes. The situation space, is a collection of
all situation models. The decision rules as well as the action space is part
of the relevance process. In this thesis, the limitations of mental and
2
In the present thesis, the schema (Ashcraft, 1994, Sarter & Woods,
1991), is a situation model skeleton that has to be filled with relevant
aspect values to become a situation model.
7
physical manipulation are mentioned as safe or unsafe combinations of
mental, physical and device systems. The exact limitations and properties
of mental, device and physical manipulation is beyond the scope of this
thesis.
The communication bridge is in this thesis specified as co-ordination of
presentation processes and relevant attention. This corresponds to the
Clark (1996) first level of symbol based communication, execution of
behaviour by one agent co-ordinated with attention to that behaviour by
another agent.
Figure 1 (adapted from Ashcraft, 1994) shows the medium for human
situation models. The three components, sensory memory, working
memory and long term memory, and their interconnections have different
properties. The sensory memory data are short lived. Sensory memory
contains the rawest form of information that has currently been sampled
from the environment. This information is available to working memory
and to long term memory for identification, categorisation and further
processing. Its content is therefore highly dependent on the processes of
attention focusing. What is in sensory memory is not automatically
attended to by working memory. The working memory, where most of
the processing of the situation models is done has limited resources. If
too many tasks are given, there will not be enough capacity available in
working memory to perform them all and some tasks will have to be
ignored. The long-term episodic memory contains situation schemes,
memories of past events used to interpret the sensory data to build a
situation model. It also contains a history of events. For example, it
contains the events of the current flight, and how acted upon. It should be
noted that what is stored in long term memory may be distorted and
difficult to retrieve.
8
PHYSICAL MEDIUM FOR
MENTAL MODELS
WORKING MEMORY
mental model & mental
manipulation
perceived physical model
SENSORY
MEMORY
Currently attended
environment
LONG TERM
MEMORY
Model structure
(schemes)
History of past events
Figure 1. Physical medium for mental models
Summary
SA models in a dynamic environment must constantly be updated. Three
kinds of updating processes can be found in the literature; aspect
sampling processes, structure deciding processes and projective
processes.
In the human cognitive system, attention focusing, directed by the
situation schemes from long term memory, decides the contents of the
sensory memories. What is in the sensory memories does not
automatically become part of SA.
2.1.3. Examples of distributed SA
For SA to be distributed, it may be co-ordinated using shared
representations. Private representations may also be actively presented to
some other agent.
Ballard & Rippy (1994) describes a knowledge based decision aid, a
mechanical tool, sampling and interpreting data, thus constituting a
localised mechanical ASK system. This example illustrates that not only
humans may contain situation models, and that ASK systems can
potentially do many things a human SA system can do.
9
The research of Mogford (1997) on air traffic control (ATC), suggests
how aspects of a situation model may be distributed between the
environment, and the human working memory. It is suggested that it may
be sufficient to maintain directions and altitudes of aircraft in the human
working memory, and to sample further information from the
environment on demand. In this way the limited human working-memory
capacity is optimally used. Mogford suggest that three kinds of
information are present in the ATC environment, relevant information
that has to be maintained in working memory, relevant information that
can be sampled when needed and then forgotten, and third irrelevant
information that should be ignored.
Artman and Garbis (1998) has documented presentation and sharing of
representations in Rescue Agency Centres. Sharing of model aspects
allowing reconstruction of the situation model by co-agents has been
documented in London underground control rooms (Heath & Luff,
1992). Ground proximity warning systems present their internal model, a
map to the pilots (Bateman, 1997) and Heads up display systems (HUDS)
present aspects of the environment replacing information usually
perceivable from the outside view during bad weather (Hartman &
Moylan, 1994).
Together these examples shows that the aspects and situation models of
the SA system are distributed between different agents, and that the
models and aspects are presented for other agents to perceive.
2.1.4. Compatibility with SA definitions of the literature
Several definitions of SA, or as it is sometimes termed, situational
awareness, can be found in the literature. These will be related to the
model of this thesis, to show if they are consistent with it, if the model of
this thesis can indeed be claimed to concern what has previously been
named situation(al) awareness.
The definitions of Endsley (1988, 1990) of Jones and Endsley (1996) and
of Artman & Garbis (1998) has been discussed earlier in this chapter and
will therefore not be discussed further. Metalis (1993) describes SA as
the mental processes involved in the development, maintenance and use
of the incomplete and constantly changing variables of a complex
system. In this thesis, the mentioned mental process has been
categorised, and their importance emphasised.
10
Fracker (1988) reviews several SA definitions from the literature.
First, SA can be;
“the ability to envision the current and future disposition of both red and
blue aircraft and surface threats”.
In the model of this thesis, this would be an example of situation aspects
of a current and a future situation model, specified for relevance in some
relevance process.
Second, SA can be;
“Where: the pilot’s knowledge of the spatial relationships among aircraft
and other objects
What: the pilot’s knowledge of the presence of threats and their
objectives, and of his own aircraft system state variables
When: The pilots knowledge of the evolution of events over time.”
The ‘where’ and ‘what’ would in the model of this thesis be current
situation models, unspecified for usage by any relevance process, and the
‘when’ projections of the models in the future. Since the what and where
parts of the above SA definition can not be translated into two concepts,
but to one concept, the situation model, this is a difference between the
above definition and the definition suggested in this thesis.
Third, SA can be;
“the pilot’s knowledge about his surroundings in light of his mission
goals”
Mission goals would have to be translated to the reasons why the
relevance processes are considered relevant. In this thesis, the
surroundings must be specified as the surroundings at the present, future
or at some past point in time.
However, these definitions have only concerned individual SA,
definitions of group SA has been made by Wellens (1993), Artman &
Garbis (1998) and Endsley (1988).
Wellens defines group SA as;
“The sharing of a common perspective between two or more individuals
regarding current environmental events, their meaning and projected
future status.”
Artman & Garbis defines group SA as;
“The active construction of a model of a situation partly shared and
partly distributed between two or more agents, from which one can
anticipate important future states in the near future.”
11
Endsley defines group SA as the;
“degree to which every crew member posses the SA required for his
position.”
The SA model of the present thesis suggests that the sharing of a
common perspective is defined as the degree to which the participants
are aware of the same situation aspects, and the degree to which use them
as parts in equivalent situation models. The meaning of the aspects is
thus defined as the situation models they are part of. Their projected
future status is the future situation model they are related to.
In a dynamic environment, every situation model has to be constantly
reconstructed. For future situations, in this thesis every future situation
model can be included as relevant SA, not only SA models for the near
future.
Finally, in this thesis the degree to which every crew member posses the
SA required for his position, decides how much SA is missing, how
much is redundant, and how much SA is misdistributed.
12
2.2. The proposed SA theory
From the perspective of distributed cognition, SA becomes a system of
agents, humans or artefacts, mediating, creating, transforming and
presenting situation models, using the situation models, making them
relevant to the destiny of the aircraft (Figure 1).
The situation model is what is commonly referred to as the SA whereas
the processes to maintain it are termed situation assessment processes.
The processes related to the models, are the updating, presentation,
monitoring and relevance processes. The updating processes place the
unstructured aspects of the situation in structures to give them meaning,
update the aspect values, and can project the model into the past or the
future. Presentation processes can make the models available to other
agents of the system and monitoring process evaluate whether there is
something wrong with the models, creating meta-SA. Finally, the
relevance process uses the model to execute some task.
With an analysis of distribution, it becomes clear to what extent the
processes are performed by the same or by different agents and which
agents are co-operatively performing the same process. It can also be
seen what or who mediates the models, which will influence whom, can
access the models, and how easily these can be distorted. This gives
insight into how reliable the system is and where measurements must be
made clarify how well the system works.
ASPECTS
(unstructured)
UPDATING
• of structure
• of aspects
• of projection
SITUATION
MODEL
(structured aspects)
PRESENTATION
MONITORING
13
USAGE IN
RELEVANCE
PROCESS
Figure 2. Components of the SA system
2.2.1. Situation models, mediations and function
All situation models are physically mediated, by for example a computer
memory, a human working memory, or a verbal utterance. The physical
form constrains who can perceive and alter the model, how easily it can
be perceived, comprehended, and altered, and how the model can be
disrupted. The physical form of the model and the location of the agents
also allow or disallow monitoring of attention to the model, by some
agent for some other agent.
Physical situation models
Physical models accessible to several agents allow distribution of labour,
since different agents can update the model, and several agents can use
the same model to achieve SA. In the cockpit, if one pilot could program
the flight path to be visible on the screen of the other pilot, then the
shared representation might save some work.
The physical model can be modelling the current situation, for example
the current position of the aircraft. Even the outside view of the cockpit
can be seen as a situation model of the current environment. When it is
blocked the relevant contents has to be modelled in some other way. It
can also model some previous situation, and in this way work as a
memory. In aviation, models of future situations are of importance. A
model could, for example, combine a presentation of the current position
of the aircraft with a possible future situation. The ideal is to model the
relevant aspects of the situation, and their values at the relevant time,
which may be any combination of the current state, and possible future
and past states.
Mental situation models
For human cognition it may be interesting to distinguish between present
models and mental3 models, since they rely on different mental resources.
A mental model resides in the human working memory, and can be
3
Here, the term ‘mental’, in ‘mental model’ does not refer to the
functional type of the model, but to the physical medium of the model. In
the literature, the term ‘mental model’ has been used to denote situation
awareness, as well as device models, and what is in this model denoted
situation schemes.
14
transformed by mental manipulation, whereas a present model relies on
the perception of models present in the environment, therefore the model
may be physically manipulated, by the perceiver or by someone else.
2.2.2. Updating and sampling processes
There are three kinds of updating, of structure, of aspect values and of
relative time. Updating of structure changes the meaning of the situation.
Updating of aspects provides the contents in the structure. The entire
situation model, or parts of it, can be projected either forwards in time,
for a prospective SA, or backwards in time, for a retrospective SA. These
three things can be distributed between several agents, several agents
may provide structures and contents, and projection may be done by the
same or different agents.
Sampling of the environment is the basis for updating SA. Situation
elements can be sampled and comprehended into a situation model.
Alternatively, a mediated situation model can be sampled, this model can
be of future, present, past states, or a combination of the three.
There can be a combination of physical manipulation of a physical
model, and mental manipulation of a mental model. A mental model that
is a perceived physical model is then co-represented by the world and the
perceiver. In this way, if the mental model of the world model is not
perceived, it is not lost, but can easily be reconstructed by attending to
the model in the world. On the other hand, if the perceived model is
manipulated mentally, for example projected into the future, this
projected model will be lost if it is not maintained in the working
memory of the agent, or externalised by manipulation of the physical
model.
Updating by ASK devices
As an example of ASK updating, consider the ground proximity warning
system (GPWS). It collects the aspects it needs, the current position of
the aircraft and the current surrounding terrain. It also provides a
structure for the aspects where it can project the future position of the
aircraft in relation to the current terrain.
Human updating
The updating process in the human cognitive system relies on the
environment and on the situation schemes of long-term memory. For
sampling of the environment, it relies on attention attraction of the
environment, such as light and sound, and by expectation created by the
15
situation scheme, for example the pilot expects to see information about
the landing gear on the relevant indicator, and is assumed to know when
to attend to the landing gear indicator.
The reliance on situation schemes to interpret sensory data can be
illustrated by the example of a crew warned of possible windshear at the
takeoff from the runway (Tenney, Adams, Pew, Huggins, Rogers, 1992).
This made them interpret the information using the windshear scheme,
which resulted in a crash, because the symptoms were caused by the flap
settings of the aircraft. In this case the symptoms could also be
interpreted using the scheme of inappropriate flap settings. As Tenney et
al. interprets this event, the crew used the wrong scheme because it had
been primed by the windshear alert.
The situation scheme thus has two functions, it directs attention, and it is
used to make sense of what is perceived. In this case, the windshear
situation model was not consistent with all possible information in the
environment, since the flap settings were wrong. Apparently, the
windshear scheme does not direct attention to the flap settings.
In the accident described above, the cause of the inappropriate flap
settings was an interruption during the pre flight checklist, which caused
the checklist not to be completed. At the time of the interruption, this can
be viewed as a problem with the current situation model. In this case the
SA is incomplete. This in turn caused a prospective SA error, since the
problems of the inappropriate flap settings were not foreseen. At the time
of the accident, it is transformed to a retrospective SA problem, since the
pilots’ model of how the current state came to be was flawed at the time
of the accident.
2.2.3. The presentation process
A presentation process can do two things. First it can present a situation
model for some agent to perceive. Second, it can direct the attention of
agents to the situation model. The task of presenting the model and the
task of directing attention to it may be distributed between several
agents.
Presentation of a situation model may be done for optimal own cognition,
since then the model does not have to be maintained in memory. It may
be done for other agents, to reduce their workload, and it may be done to
co-ordinate situation models of different agents, when it is of importance
16
that they agree on one model of the situation for co-ordinated action.
Furthermore, it may be presented to allow some other agent monitor the
situation model. The other agent can then evaluate the accuracy,
completeness and relevance of the model, and also use it to understand
the action of the other agent.
Presentation by ASK devices
In the case of the GPWS, a correct projection of the future aircraft
relative the terrain does not automatically mean that a crash can be
avoided. If the projection is not presented to the process that controls the
navigation of the aircraft, the modelling is done for nothing. To make
sure that it is attended to when needed, the GPWS evaluates the
consequences of the future position, and, if needed, directs the attention
of the pilot to the result of the evaluation. If the GPWS presents the map
with the projection, the pilot can always decide to look at it to evaluate
what can be a future problem, or if there already is an identified future
problem, evaluate how to avoid it.
Human presentation
One disadvantage of mental models is that other agents cannot directly
perceive them. They can, however, be implicitly or explicitly
communicated. By acting, a pilot may reveal how the situation is
perceived. What information a pilot attends to, can reveal what the pilot
currently finds important. If a co-pilot perceives something important,
and also perceives the pilot flying does not attend to it or acts upon it, he
/ she can assume that the pilot flying is unaware of it.
Of course the pilots may also use explicit means of communication, such
as discussing the events with each other. If a third party overhears such
discussion, this third party may use it to infer what the pilots are aware
of.
Presentation is subject to distortion, and the perceiver is subject to
possible misperception, misunderstanding, or distraction. Therefore, it is
a good idea to test the presentation, such as a digital map, in a realistic
situation, if the presentation must work.
2.2.4. The monitoring process
In a multi-agent system with human and machine agents, the agents can
monitor each other, to evaluate how well the system as a whole, or some
specific agent, is functioning. For this to work, an agent can either
17
monitor the actions, or results of actions, from other agents.
Alternatively, the internal processes of the agent can be monitored, if
these are presented.
Monitoring by ASK devices
In the modern cockpit a good example of a non-human agent is the
autopilot. It can for example be programmed to hold a specified altitude.
This task may be monitored by checking the current altitude, to see if it
deviates from the selected altitude. In this case, what is monitored is the
effects of the actions of the agent.
Monitoring by humans
An agent may monitor and evaluate its own SA, to decide how complete
and accurate it is. This meta-SA can then have four outcomes. Either the
awareness is both complete and accurate, and the pilots believes that it is,
or it can be complete and accurate when the pilot believes that it is not.
Furthermore, the awareness can be incomplete or inaccurate when the
pilot also believes that it is incomplete and inaccurate. The least desirable
outcome is that the SA is incomplete or inaccurate, when the pilot
believes it to be complete (VINTHEC WP 1).
The pilots may monitor the load on the own cognitive system, this
subjective assessment of workload can be used to decide when to
delegate tasks or when to ignore some tasks in favour of the most
important ones.
2.2.5. The relevance process
The relevance process are not part of SA, but makes use of the situation
model. For example, a situation model containing the future and current
position of the own aircraft, an enemy aircraft, and the ground, may be
used in two processes. One process may need the own aircraft position
and the enemy aircraft position to lock some weapon at the enemy.
Another process may need the position of the own aircraft and the
position of the ground, to avoid crashing into the ground. This way the
military awareness as well as the navigation awareness is based upon the
same model. Thus, different processes make use of aspects of the same
model.
Sometimes, as Endsley (1988) observes, one of the processes, but not the
other is, performed correctly. If hunting the enemy is done at the cost of
maintaining awareness of the ground, a crash may result. This may
according to this model be either because the ground position of the
model is not maintained, or because the ground position relative the
18
aircraft is not correctly projected into the future, or simply because the
consequences of the model, a crash, is not evaluated.
A situation model is only relevant if the information is needed to perform
or to evaluate the need for some current or future action. Action can,
however, be done without complete SA. For example, one crew taxied to
the wrong runway, by habit, instead of the assigned runway (Jones &
Endsley, 1996).
Awareness relevant for future actions is for example to know the weather
conditions of the destination airport, before arrival. If there is a risk of
weather closing, this awareness can be used to plan ahead, to decide in
advance what to do if there should be weather closing. An experiment
where some crews obtained such awareness, whereas some pilots failed
to obtain it, showed that such awareness reduces the mental workload of
the pilots during the weather closing. (Tenney et. al. 1992)
2.2.6. Properties of distributed and localised systems
The properties of distributed systems
The processes and models have to be found and located to some agent or
physical representation. When this is done, it is possible to see what
processes each agent is responsible for. Then it can be examined whether
the processes are done with conditional or continuos updating.
With continuos processing, a process is performed continuously, or with
regular intervals, unconditionally, it is not possible for the agent to leave
the process undone to execute some other process instead. Agents with
continuous processing, such as the GPWS, can dramatically increase
aircraft security, as seen in the reduction of accidents with the
introduction of GPWS systems (Bateman, 1997). Here, ASK, although
weaker than human SA, is sufficient to increase the SA of the entire SA
system.
With conditional processing, the process can be left undone. Then care
has to be taken to ensure that vital processes are still maintained even
during high workload. Human agents, in particular, have to switch
between different processes. This may mean that some process is
performed at the cost of another process not being performed. It may
happen that some process is not performed, even though the resources
are present. Such an action, or lack of action, depends on traditions and
19
training. Since monitoring processes are only vital when something goes
wrong, these are especially vulnerable to the problems of processes being
left undone during high workload.
The properties of localised systems
The main properties of the internal human cognitive system are first that
it relies on situation schemes to direct what aspects of the environment to
attend to, and how to interpret these aspects. The second main property is
the limited capacity of the working memory to maintain the mental focus
on the situation models, or on the focus on the task. If the human
working memory is overloaded some situation model has to be neglected,
or the execution of some task has to be abandoned. These sometimes
conflicting tasks of maintaining awareness, may therefore result in a
maintenance of SA on cost of maintenance of relevant tasks. For
example, if a model of a future situation which is not immediately
relevant (but will be relevant in a later stage of the flight) is maintained
in a high workload situation, and the execution of a relevant task
therefore has to be abandoned, high SA is then achieved at the expense
of performance, instead of providing the basis for good performance
(Wellens, 1993).
If compared with the situation of distributed cognition, all the aspects of
SA cognition may be localised to one individual. The individual then
represents the model internally, updates and manipulates it internally,
and can also monitor its quality and present the model when needed.
Localised cognition is not only a possible property of a human agent. A
mechanical system may also work independently of other agents. For
example it is possible to imagine an ASK system such as the GPWS,
which instead of directing the attention of the pilot to a possible future
danger, is designed to assume control of the aircraft and to perform the
evasive manoeuvres without communicating this to the pilots.
2.2.7. SA Error
In a system of distributed SA, when some agent has the relevant SA, but
not the agent that should have it, then the SA is misdistributed, as
compared to missing SA.
There can also be redundant SA, that no agent needs, or that is shared
between an agent that needs it and one that doesn’t need it. When SA is
20
shared between two or more agents that need it, then the SA is not
redundant. Redundant SA is necessary as a precaution against device
failure, or in the case of a human agent for some reason not being able to
act as desired. In these cases, the redundant SA becomes vital for the
remaining agents to be able to take over the processes performed by the
missing agents.
The SA for the individual may be incomplete, when some aspect of the
relevant situations, or some relevant situation, is missing. It may be
inaccurate when some aspect is interpreted using the wrong situation
scheme, or when some aspect is inaccurately perceived.
21
2.3. Measurement techniques for SA systems
For the operationalisation of SA, several measurement tools are
available. In this chapter eye-point of gaze equipment, voice transcripts,
physiological measures, event-related potentials, and questionnaires as
well as performance measures will be discussed.
2.3.1. Eye point of Gaze
Eye point of gaze (EPOG) equipment (VINTHEC WP 4) measures what
a human is looking at. It measures a time history of fixations, where the
environment is divided into areas of interest. The equipment records the
time, duration and position of fixations on these areas. For each fixation,
pupil diameter and blink rate may be measured as well, by some EPOG
measurement equipment.
Translated into the model of this thesis, the fixated areas correspond to
the contents of the visual sensory memory of the human participants. The
time history corresponds to the changing contents of the visual sensory
memory. The LTM scripts together with other attention guiding devices
guide head- and eye movement.
The time history may be transformed into dwell time, or fixation number,
percentages for some period of time on the defined areas. It may also be
transformed into a transition matrix, where the numbers of transitions
from area X to area Y, from area X to area Z, e t c, are described.
Given one GazeTracker and one agent, the equipment can still be seen
not only as a measure of internal cognition, but of usage of external
information sources. In a real situation some source of information
supposedly contains the relevant information. The EPOG measure can be
used to evaluate to what extent the areas with relevant information are
attended to. This is particularly important when information is only
visible a short time, or has to be reacted to as soon as it is presented.
Assuming that a pilot knows what to do when the information is
comprehended, in the case of a pilot not acting, it can be inferred if the
information was attended to at all. If so, it was not comprehensible. If it
was not attended to, then it was not salient enough.
Several EPOG equipped pilots in the same environment give rise to
parallel time histories. These can be compared, to find differences in the
time distribution of the attended areas, and to examine whether
simultaneous dwells on the same area occurs.
22
The pupil diameter and blink rates are usually used as measures of
workload. The workload measures of the EPOG equipment are mostly a
measure of internal cognition, but even this is important from a holistic
perspective, since a too high workload means that the work distribution
has to be altered.
2.3.2. Questionnaires
Questionnaires can be used to measure meta-SA, subjective workload
and subjective performance, as well as more objective data such as
working memory contents.
The meta-SA questionnaires measure the result of some monitoring
process. (VINTECH WP 3) Consequently, one can measure the meta- SA
of ones own SA, assess the SA of some other agent(s) or the SA of the
system as a whole. However, it has been argued that an observer rating of
SA quality and completeness must be incomplete, since the observer can
only observe behaviour and does not have access to what other agents are
thinking (Berggren, 1998).
There are two approaches to questionnaire measures of SA. The
subjective SA method asks the human agent for a SA score on a scale.
The other method queries the agent for specific aspects of the
environment, when the relevant aspects in the environment are removed.
An example of such a method is the SAGAT technique (Endsley, 1988).
These two methods measure very different things. While the SA score
method attempts to measure the result of a monitoring process, subjective
meta-SA, the other method measures the components of sensory memory,
working memory or long term memory.
The subjective workload measures the experienced workload of the
agent. This can be compared with physiological measures such as the
mentioned blink rates, and measures such as heart rate variability.
2.3.3. Physiological measures
Physiological measures such as heart rate, heart rate variability,
perspiration, and blink rates can be used as indicators of mental
workload. Unfortunately, they can be indicators of physical work as well,
which means that physiological measures has to be interpreted with care
(VINTHEC WP 1,3).
23
2.3.4. Event-related potentials
EEG Event-related potentials (ERP) can be used to shed light on the
processes of understanding of stimuli (Derks & Gillkins, 1993). In short,
using electrodes attached to the human skull, the magnetic fields of the
brain are measured. The fields are then analysed after the presentation of
stimuli to examine the changing electrical field in different parts of the
brain as recognition of stimuli proceeds.
The comprehension processes are assumed to occur in parallel, with a
negative peak at 100 ms for “orientation”, and at 200 ms for “labelling”.
These peaks are different for each modality. The positive peak at 300 ms
is associated with categorisation, where unexpected but relevant stimuli
give a high peak. There is a highly interesting 400 ms-depolarisation
peak associated with categorisation failure, it is reported to be
specifically sensitive to violation of semantic expectancies. At 600 ms,
there may appear a positive polarisation that corresponds to incongruity
resolution, however this is not well documented.
In the model of this thesis, what is used as the known variable is the
content of sensory memory. This has to be known to be able to discuss
what is being labelled and categorised. A categorisation failure would
indicate that the pilot is unable to find a situation model in which the
sampled sensory data would make sense, or fails to decide which
conflicting model the data would fit. A high categorisation “surprise”
peak would indicate that the pilot finds the event surprising but
comprehensible. The 600 ms resolution peak, could, if this peak is
correctly understood, indicate that the subject has managed to make
sense of the stimulus after all.
Thus, an ERP analysis will not, even after the appearance of a resolution
peak, tell us which model the pilot is using. Using ERP alone does not
tell us whether the pilot has an accurate situation model, or if the model
is inaccurate, but rather if the pilot has at all got a model, whether the
pilot has been considering alternative models and how surprising the
situation is.
2.3.5. Transcripts of verbal messages
Transcripts of communications between aircraft and air traffic control
can illuminate the number of errors occurring due to misperception or
24
miscomprehension of voice based messages (Jones & Endsley, 1996). It
has furthermore been used to analyse the interactions between coordinators of the London underground (Heath & Luff, 1992) and of the
interactions in an emergency co-ordination centre (Artman & Garbis,
1998).
2.3.6. Performance measures
High performance is not always accompanied by high SA. System
performance with for example an active autopilot may be excellent,
whereas pilot SA may be quite low (Tenney et. al, 1992; Shively &
Goodman, 1994). As discussed, when performance of cognitive functions
and performance of relevance processes are distributed from human to
machine, performance may improve, whereas pilot SA may decrease
(Shively & Goodman, 1994). Care must therefore be taken to ensure that,
for the measured situation, SA is required for performance.
25
2.4. Theory summary and discussion
In this chapter I will summarise the SA model, and discuss the
consequences for designing SA critical systems.
2.4.1. Concluding operationalisation model for SA
Scheme directed sampling &
externally directed sampling
AGENT A
MENTAL MODEL &
SCRIPTS
Mental
manipulation
AGENT B
MENTAL MODEL &
SCRIPTS
Mental
manipulation
Physical
manipulation
PHYSICAL MODEL AVAILIABLE
TO A AND B
Physical
manipulation
Scheme directed sampling &
externally directed sampling
Figure 3. Combined SA system
To model an SA system, first, the agents are added, then the physical
representations of interest. In this distributed system there can be
different physical models, modelling the same information, such as in
cockpits with double instrumentation, one for each pilot. Sampling is, as
illustrated in Figure 3, either guided by mental situation schemes or by
external attention allocation mechanisms. As we see, there can also be a
combination of physical manipulation of a physical model, which may be
shared, and mental manipulation of a mental model. The mental model
can be a perceived physical model, or a comprehended set of situation
aspects.
When we want to predict the SA of a modelled situation, first we can
examine if the relevant dynamic aspects are present in the environment.
If not, then we can stop here, because then there can be no SA. If they are
present then we want to know if they will be attended to at the right time,
26
and if so, whether they will be comprehended. For this we may design an
experimental setting and measure the SA system.
If SA is required for performance, and the pilots perform well, then we
know the dynamic aspects have been comprehended. Comprehension of
situations and situation elements can also be measured with the SAGAT
method. The process of comprehension can be investigated with ERP
methods. To measure comprehension in verbal interaction, the review
suggests transcripts of verbal messages.
If comprehension is not achieved or if we want to know where in the
environment the dynamic aspects were sampled, we can measure
attention. To measure visual attention, the review suggests the usage of
Eye Point of Gaze tools.
To measure Meta-SA the review suggests the usage of a subjective SA
quality rating scale.
To measure the pilot workload affecting SA, the review suggests
physiological measures or a subjective workload rating scale.
2.4.2. Relevant SA
Having defined a model for SA, relevant SA can be defined as those
models and aspects present in a team that are relevant for some relevance
process. For the SA to be correctly distributed, it has to be present for the
agent executing the relevance process. For the SA to be accurate, the
aspects must be interpreted using accurate situation schemes. For the SA
to be consistent there can be no situation models that gives the same
aspect conflicting meanings or values. For the SA to be sufficiently
redundant, if some agent should be unable to function, the remaining
agent(s) should have sufficient SA to maintain the relevant tasks.
To say that this abstract description leaves the concept of SA too general
to be useful is merely a play of words, since the model framework of this
thesis shows that SA is a label for any system containing situation
models. This means that anyone interested in some specific SA system,
be it a distributed or localised system, can compare the system of interest
to already examined systems to seek similarities that may make
measurement redundant, or that may show how measurement can be
performed. It allows sharing of measurement tools, and of modelling
27
frameworks, as well as sharing of observations about the properties of
some physical situation representation.
2.4.3. Safe distributions of labour between actors and
artefacts
Device usage in the cockpit transforms the task of remembering cockpit
speeds (Hutchins, 1995) from one which heavily relies on the unreliable
human working memory to one that relies on simple manipulation of
simple devices, and a far more reliable visual comparison task. Basically,
a list of landing speeds given an aircraft weight and flap settings, serve as
a long term memory of landing speeds. Given the current weight of the
aircraft read from a display, a sheet with relevant numbers can be
selected and put in front of the pilots. The sheet is functioning as a
working memory, and a means for both pilots to monitor the task,
whenever they want they can compare the weight specification on the
card, with that of the weight indicator of the aircraft. Next, the pilots set
the speed bugs on their airspeed indicators, which show the airspeed
where the flap settings have to be changed. As the settings are cross
checked by both pilots, the reliance on human working memory is
removed, and the task is successfully transformed into a simple visual
task of observing the airspeed needle and compare it to the position of
the speed bugs. In this way, the cognition is distributed over several
artefacts and a reliable part of the human cognitive system.
The point of this example was to show a maintenance of a physical future
situation device, the speed bugs. They do not predict the future situation,
they are used to compare a physical model of the current speed, with the
desired future speeds at which the flap settings are to be changed. In this
way, the model is maintained by physical manipulation, sampled and
used for mental evaluation.
The external usage of speed bugs at the same time allows the pilots to
monitor each others representations of the future situation, and the
appropriate monitoring process was observed in the above example.
2.4.4. Safe practices, tradition and training
That a set of components and practices exists that can constitute a safe
SA system, does not mean that the practices are followed or that the
human agents use the components properly.
28
Sollenbergher & Stein (1995) discuss the problem with unused memory
aids in air traffic control environments. The concern is that usages of
memory aids were resisted due to controller culture. Weick & Roberts
(1993) discusses the phenomenon as it is manifested on aircraft carriers.
They name it heedful interrelating, and includes in it not only acting with
high SA, but also the individual usage of practices leading to a high SA.
On air carrier decks heedful interrelating is preserved as experienced
personnel introduce newcomers to the tradition of heedful interrelating
through stories of the consequences of heedless interrelating.
How training can theoretically increase SA trough training the pilots how
to recognise signs of low SA, is described by several authors (Schwartz,
1997; Caretta & Zelinsku, 1996; Regal, Rogers & Boucek, 1988).
Related to the SA system model of this thesis, such training would, if
successful, increase the accuracy of the human monitoring processes.
This would lead to a more accurate meta-SA and consequently to safer
flight.
2.4.5. Optimal SA distribution and component failure
Distributing actions between man and machine may have very subtle, yet
potentially very grave consequences, for the pilot’s ability to handle
automation failures. Shively and Goodman (1994) used an arcade game
like simulation of a star trade cruiser to examine these subtle effects. In
the simulation, the calculation needed for a successful landing was to
evaluate the distance from the sun to the own position. The pilots either
had to calculate this themselves, or the correct position was calculated by
the game computer and presented as an X on the navigation display. The
consequence of having a computer presenting the correct position, was
an increase in performance, but a decreased awareness of how the correct
position should be calculated. In the framework this corresponds to
decreased device awareness for the process. It also corresponds to a
potentially decreased awareness of the correct distance to the sun, a
critical aspect of the situation.
Shively and Goodman theorise that automatic systems producing this
effect may leave the pilot without the necessary device knowledge of the
process, to perform it during a device failure. In the model of this thesis,
this corresponds to an optimal distribution of cognitive processing for the
fully functional system, since the performance with this distribution is
improved compared to a system without it. Then it corresponds to a
system where one agent is suddenly removed. The result may then be that
29
the pilot is unable to perform the cognitive process of the device, or that
performance is poor. This means that optimal distribution of cognitive
processing may lead to poor device models of the human agents. How to
perform some action, such as a calculation is not part of SA, but if the
result, awareness of the correct distance to the sun is considered part of
SA, then this decreased device knowledge results in decreased SA.
This result is to be contrasted to display enhancements where cognitive
processing is not delegated, but where the action space is made more
salient, such as making the storage gauge more comprehensible. With
this kind of enhancement SA was enhanced as well as performance
(Shively & Goodman, 1994).
The SA system may consist of combination of agents using conditional
and continuos processes. Then the properties of the combined system
will depend on the reliability of this combination. The mechanical agents
most often responsible for the continuos updating may of course fail, but
they may none the less be less likely to fail than if the task is delegated to
a human agent, especially if the human agent simultaneously has to
attend to other tasks. If action is not delegated to the mechanical agent
with continuos processing, then the agent should direct the attention of
the pilot to the relevant process, when needed.
The advantage of mechanical agents is therefore that they can often
reliably maintain continuos processes, whereas the advantage of a human
processor is that they often can handle unexpected processes, or
processes that are yet too complex to be automated.
2.4.6. Consequences for design
The measured and modelled system will show to what extent the system
is optimised. Design efforts can then be directed at the weakest spots of
the system. Furthermore it may give insight to the distributed and
localised effects of the introduction of new technology.
The design of situation models for mechanical agents to achieve ASK,
and the design of situation models to present for the human agents,
reflects the understanding of the SA needs of the designer. This is true
especially for non-human agents since these normally do not have the
option to decide what aspects to sample, or how to combine the aspects
to a situation model. They normally do not decide how to evaluate the
resulting situation model either. Thus, the performance of the cognitive
30
processes is delegated to a mechanical device, whereas the design of the
process resides in the hands of the designer. Thus, all situation models
delegated to a non-human agent contain only what is put there by the
designer. Even for self-learning systems, the designer controls what can
be learned. A weaker dependence on the designer is present for the
human actor, which has to rely on what is presented to him or her by
aircraft equipment. The dependence is weaker only since the human
agent can decide to sample what a designer did not intend to be sampled,
and put it into a situation model the designer does not know of, and
evaluate the resulting model in a way unknown by the designer. Still,
however, the availability and comprehensibility of situation aspects and
models is largely in the hands of the designer.
The applied model together with appropriate measurement may show the
reliance of the system on internal attention management, what aspects
and models are maintained internally, and whether they are projected
internally in time. Different design solutions may have to be used to
make world aspects more comprehensible, attended to at the appropriate
time, to reduce the need for human internal projection of aspects and
models, and to reduce the effort needed to either perceive a model, to
perceive the relevant aspects of that model, or to construct a model from
presented aspects.
The theory does not deal with all design problems, of course, but only
those associated with SA. A major problem for cockpit design is the
limited space. There can’t be too many big displays, and decisions may
be time critical. That means, if some critical information proves to be
incomprehensible, and there is no space to present a comprehensible
picture, or if some information will be unattended with no means of
attention attraction, this model suggest three options. First, routines may
be changed, training may direct attention and increase the
comprehensibility of messages. Second, the monitoring task may be
distributed, either to a new human agent, or to a mechanical agent. Third,
if neither of these options are possible, then if the experiment was valid,
as it should, then the problem will remain, and if it is considered too
dangerous to fly with this problem, then there is always the option to stay
on the ground. But the limited space and the time pressure is exactly
what makes an analysis of the SA system vital, since it may suggest what
the limited space should be used to present, and how to manage the
distribution of labour in the SA system.
31
This model, thus, is to be seen as complementary to models for the
design of comprehensible and space economical representations, and to
models addressing other design issues for the cockpit. This model is
complementary, since it is of no importance that a representation is
comprehensible, if it is either irrelevant or unattended. It is furthermore
of no help to have an attended but incomprehensible representation of a
situation.
32
33
3. THE EXPERIMENT
The experiment was made in the framework of the VINTHEC (Visual
interaction and human effectiveness in the cockpit) project. The
experiment was designed to evaluate the accuracy and usefulness of eye
point of gaze (EPOG) equipment, and for measuring SA, and workload
during the experimental conditions. Due to the fact that there were two
pilots in each crew, the experimental data could be used for the analysis
of this thesis as well.
Relevant measures from the experiment were selected for inclusion in the
data analysis of this thesis.
3.5. Method
3.5.1. Test site
The experiment was performed at the NLR (National Aerospace
Laboratories) in Amsterdam, the Netherlands.
3.5.2. Participants
The participants were 20 active pilots normally flying civil passenger
jets, most of them familiar with Boeing 737 procedures. Their flight
experience ranged from 2000 to 20000 flight hours. In the preexperiment questionnaire, no pilot reported any activities within twentyfour hours, judged as likely to interfere with experiment behaviour. The
questionnaires are described in appendix B.
3.5.3. Procedure
The pilots were divided into ten crews of pilot and co-pilot. Each crew
flew two simulated flights, the first from Amsterdam to London, the
second from London to Amsterdam. The entire flights, except landing
and takeoff were simulated to be above thick clouds, to avoid relevant
information to appear on the outside view.
Each flight crew was briefed and made familiar with the simulator during
the morning. In the afternoon the two experimental flights were made.
Before the experiment the EPOG equipment was calibrated, to ensure
that dwells on areas of interest were also recorded as such. Without
34
calibration, the EPOG equipment does not work up to standard. The
calibration was checked twice during each flight, one during the cruise
and once on the ground after the taxi phase of each flight, with a
possibility of recalibration between the two flights. The calibration
sequences were removed from the experimental data and analysed
separately.
Before each flight, the briefing questionnaire was handed out. During
each flight the four subjective questionnaires were handed out. After
each flight a debriefing questionnaire was handed out.
Before each flight, the in flight observer gave the pilots the flight log.
The log described nothing of consequence for the functionality of the
aircraft for the first, Amsterdam-London, flight. For the second flight, the
log described an autopilot failure where the autopilot failed to capture the
selected altitude.
In addition to the physiological measures made by the EPOG equipment,
heart rate was recorded, and heart rate variability was derived from the
heart rate.
Each flight may be divided into six flight segments, the takeoff, climb,
cruise, descent, approach and landing. The flight was segmented because
the workload, the actions and goals of the pilots and their looking
behaviour are assumed to differ between the segments, but stay the same
within the segments, if nothing unusual happens. The takeoff segment
was excluded from the present study, leaving five segments for the
analysis.
There were four abnormal events in the experiment, two occurring in
each flight. They will here be presented in the order they occurred, the
same order for each crew.
The map shift event occurred during the cruise phase in the Amsterdam
London flight. It was manifested by a short period of strong wind. It took
place in a situation with almost no wind, and returned to almost no wind
after a short time. The strength of the wind was displayed on the map
display, and was a simulated computer error, which made the computer
project the aircraft on the map at the location it would have been if there
had been a strong wind. This means that the projection error remained
after the disappearance of the wind. Such a strong wind is extremely
35
unlikely, if not impossible, if it occurs and disappears as was indicated
on the map display.
A gear unsafe event occurred during the landing in the Amsterdam
London flight. This was shown as a missing green light, of the three
lights for the landing gear. The missing green light indicates that the
landing gear has not been safely extended.
An altitude bust occurred at the end of the climb in the second flight.
This was constructed as a failure of the autopilot to catch the selected
altitude. This means that the aircraft continued the climb beyond the
selected altitude. It could be seen as a continued rising altitude on the
altitude indicators.
The fourth and final event occurred during the London-Amsterdam
landing. The pilots were given a flap asymmetry warning, consisting of a
warning light, a sound, and a flap asymmetry warning text on the engine
display. A flap asymmetry causes a situation where the flap settings can
not be changed during the landing without risk of an increased
asymmetry. The function of the flaps is to change the weight/speed ratio
with which the aircraft can fly.
The events were grouped as follows
Low
workload
High
workload
Low SA
Map Shift
Flap
Asymmetry
High SA
Altitude
Bust
Gear
Unsafe
The map shift and the altitude bust were designed to be low workload
events, whereas the gear unsafe and flap asymmetry occurred during the
landing, and were therefore considered high workload events. Regarding
SA, the pilots were warned in the techlog, that the autopilot had failed to
capture the desired altitude during a previous flight, and the gear unsafe
was practised during the practice flights, making these two events high
SA events because of the preparedness. The map shift and flap
asymmetry were neither practised nor prepared for, making them low SA
events. For this thesis the altitude bust event and the map shift events
were selected for analysis. The gear unsafe event was ill suited for the
analysis of the present thesis since the gear unsafe warning lights were
not included as an area of interest for the gazeTracker recordings. The
36
flap asymmetry event was not analysed statistically but will be mentioned
briefly in the discussion section.
3.5.4. Design
A 2*2*2*5*20 mixed design was used. The first variable refers to Role,
as pilot flying or as co-pilot, the second refers to Workload (high/low),
the third refers to SA (high/low), the fourth refers to the five Flight
phases (climb, cruise, descent, approach and landing) and the fifth to the
number of Subjects (20).
3.5.5. Equipment
The Fokker 100 simulator was based on a full-scale multipurpose flight
simulator with motion. The simulator had seats for two pilots and one in
flight observer. Communications using headsets was recorded, and a
small stationary video camera was used to record the events in the
cockpit. On the same video tape also the primary flight display (PFD) ,
the navigation display (NAV) and engine display was recorded, together
with all audio that went via the cockpit headphones.
For the EPOG data collection, the cockpit was divided into ten areas of
interest. The primary flight display (PFD)(*2), the navigation display
(NAV)(*2), the outside view (OUT)(*2), the engine display (ENG), the
CDU (*2) and the autopilot (A/P).
Two headsets were used for the EPOG data collection, one for each pilot.
The EPOG equipment used at the NLR consists of a head mounted
tracking device. Eye Point-Of-Gaze (EPOG) is the point on a predefined
surface were a line coming straight from the centre of the eye crosses that
surface via the lens of the eye. As such this is the central point in the
pilot’s field of vision. This point was measured by means of the EPOGrecorder called GazeTracker (VINTHEC WP4). The co-ordinates on the
surface, together with the duration the pilot looked at a particular area of
interest, is called a fixation and has been recorded in a computer file.
Besides the scanning pattern, the pupil diameter and eye blink rate were
recorded.
The marker signals for the events and flight phases of the experiment
were attached to the timeline of the co-pilot. For this experiment, the
synchronisation accuracy was +/- one second due to the time accuracy of
the ethernet connection. It should be noted that a much higher
synchronisation accuracy can be obtained with other now available
37
methods. This accuracy is only of importance for the simultaneous dwell
analysis.
The scanning behaviour was considered to be an indicator of the pilot’s
mental preoccupation. Pupil diameter was used as an indicator of mental
workload while eye blink rate indicated the visual workload.
The accuracy used to discriminate between fixations was 1,5 degree
visual angle from the eye and a minimum fixation duration of 150 ms.
This means, for the analysis it was assumed, only if the subject looked at
a point more than 1,5 degrees from the previous fixation and for longer
than 150 ms. a new, cognitive meaningful, fixation was made.
For the determination of the eye blink rate the blink had to be longer then
40 ms. Shorter blinks are impossible for human beings to make and as
such are considered to be artefacts. Blinks are all further situations
during which the pupil diameter was zero (closed eyelids).
It has to be noted that pupil diameter is dependent on workload as well as
other variables amongst which the amount of light falling on the eye. In
order to compensate for this illumination effect care has been taken to
use, for the analyses, only the pupil diameter measured when the pilot
looked at one display only for each analysis. This way, knowing that the
illumination within the cockpit didn’t change, the pupil diameter can be
used as an indicator of mental workload.
Questionnaires for SA, workload and performance were used as
subjective measurement techniques before, after and during flights.
Questionnaires for sleep quality, and other individual variables were used
as well as briefing and debriefing questionnaires before and after each
flight respectively. All questionnaires were handed out on sheets of
paper.
3.5.6. Research questions
The first research question which can be partly answered by an
experiment, is whether the operationalisation model for distributed
situation awareness presented in this thesis, is sufficiently clear,
complete, and correct to explain found results. That is, to explain what
components of the SA system are responsible for the achieved SA. If the
achieved SA is too low, it should shed light on the reasons for this, as
well as the reasons for a sufficient SA. In the case of no results regarding
38
SA, it should explain whether this is due to quality problems with the
equipment or the number of subjects, the scenario, or the choice of
measurement techniques. This question will be answered as an
evaluation of whether the following research questions can be answered.
The second main question the experiment should answer is for this thesis
whether relevant attention is a factor for achieving relevant SA.
If the second question, as expected, is confirmed, an additional question
is posed. The question, then, is whether this particular attention
management system is sufficient for achieving or for not achieving the
relevant SA. In particular, if the system is sufficient to achieve relevant
SA in the altitude bust and map shift events.
The third main question is whether there exists divided attention in the
cockpit. If the pilots look at different things, this can mean that they
update different situation models, for themselves, or to be shared with
each other. If they look at the same things but at different times, in a
dynamic environment this means that the team as a whole has a more
complete coverage of what has been present in the environment. If the
perceiver can also comprehend what has been attended to can, as noted
before, not be evaluated using attention measures alone.
A side question, which will be answered, is whether the measurement
techniques were sufficient for the desired measurements, or if some
question demands some alterations to the equipment or entirely different
techniques. A part of this side question is whether the scenario was
appropriate for the questions in this thesis. When this side question is
answered, the reader should keep in mind that the experiment was not
made exclusively for this thesis, but as part of a larger research project.
3.5.7. Operationalisation of the research questions
To operationalise the questions, they were reformulated as questions
related to measurement data and interpreted using the SA system model
presented in this thesis, applied to the experimental situations.
The first step is to relate the measurements to the applied SA system
model. Below is a model (Figure 4) of the two human agents, the pilot
flying and the pilot non-flying and of the physical areas of interest, a
selection of the physical representation surfaces, in the cockpit. The
cockpit has duplicated Primary Flight Displays (PFD), two Navigation
39
Displays (NAV), two front windows (OUT), two CDUs, a shared
Autopilot (A/P) and one Engine display.
The content of sensory memory were estimated by the EPOG measures.
The arrows represent the EOPG measures of visual attention. The
attention of one pilot on the areas of interest of the other pilot could not
be measured, as indicated in the illustration. This could not be done due
to low precision of the EPOG equipment with such an angle. As
indicated in the illustration the EPOG measures tell us what raw display
information is present to the visual sensory memory. It should be noted
that mere presence to visual sensory memory does not mean that the
information is attended to by working memory, or that the information is
comprehended.
The working memory situation models were not measured directly.
Instead the subjective quality of the situation models, the meta-SA, as
estimated by the pilots, was operationalised using the subjective SA
questionnaire. The load on working memory was recorded using a
subjective workload measures questionnaire. In the case of the altitude
bust abnormality, by observed attention to the aircraft techlog, the
altitude bust scheme is assumed to be present, allowing an altitude bust
situation model to be formed. The maps shift scheme had, instead, to be
evoked only with the help of the cues of a map shift visible on the map
shift display.
The quality of the long term memory schemes were estimated as the
hours of flight experience of each pilot.
The aircraft log was, as noted above, handed out on each flight, and
attended to by the pilots. The log represents the physical medium for the
retrospective SA of the aircrew.
The team performance was measured, for the altitude bust, as the time
elapsed from the onset of the altitude bust, to the time of altitude capture.
For the map shift event, time was measured from the onset of the event to
the moment the map was reset, either the crew discovered it or it was
revealed to not interfere with the next event. For the individual pilots,
pilot performance was estimated by the pilots using a subjective
performance questionnaire. Performance is not a part of the situation
awareness system, but a control variable, which under specific
circumstances can be used as an estimation of the quality of the situation
model, or the quality of the SA system.
40
Sampled sensory data of the team can be measured as the total
percentage of time an area was attended to by at least one pilot. For each
flight segment or event, the time the pilots spend simultaneously on each
display can be calculated. Together with the times they each spend
fixating on each display, the total time they monitored each area together
can be calculated.
AIRCRAFT LOG
Observed usage
PILOT FLYING
SAMPLED
SENSORY DATA
PHYSICAL
AREAS OF INTEREST
PFD
NAV
WORKING MEMORY
Comprehension:
sSA sWL sP
Primed scheme
Performance (event
resolution time)
LONG TERM MEMORY
Flight experience
OUT
CDU
A/P
Engine
SAMPLED
SENSORY DATA
PFD
NAV
OUT
CDU
PILOT NON FLYING
WORKING MEMORY
Comprehension:
sSA sWL sP
Primed scheme
Performance (Event
resolution time)
LONG TERM MEMORY
Flight experience
Figure 4. Applied SA system model to the experiment. sSA stands for subjective SA, sWL stands for
subjective workload, and sP stands for subjective performance.
The first research question was operationalised as a discussion over the
results of the other research questions.
41
The second research question will be operationalised first as an analysis
of the possible dwell time percentage differences on the various areas of
interest between the flight phases. If there are no differences then
attention can not be a good measure for situation models, because clearly
the landing demands a different situation model than the cruise phase,
since the situations are very different. If no differences are found, then
the differences will still be assumed to exist. The conclusion will instead
be that the differences can not be caught using this experiment.
Second, to get a more clear demonstration of the importance or lack of
importance of relevant attention, the altitude bust and map shift events
will be examined. For these events, are there any differences in dwell
time on the relevant display (PFD) between pilots flying and co-pilots? Is
there a mean difference between the groups, when divided into groups of
fast and slow action: subjective workload, dwell time, subjective SA,
subjective performance and flight hours?
If there are any results regarding the altitude bust and the map shift, these
results, or lack of results, will be discussed to answer the additional
question regarding the attention management system.
If there are any differences in the attention between the flight phases, the
third question will be operationalised as two subquestions.
Are there differences on the time spent on the same display, for the pilots
of the same crew, when analysed for simultaneous dwells, that can not be
accounted for by assuming that the co-ordination is completely random?
Are there differences in the percentage of time spent on the displays
when the crews are divided into pilots flying and co-pilots?
The side question, regarding the validity of the experiment will be
addressed in the general discussion part of this thesis.
3.6. Measurement results
The evaluation of the EPOG calibration data resulted in the exclusion of
two crews from the analysis. Furthermore, the analysis of the remaining
pilots showed the desired high between area point of gaze accuracy. The
within area precision varied between pilots, therefore precautions must
be taken when selecting pilots if a high within area precision is desired.
No within area analyses are performed in the current thesis.
Between the main flight segments there were significant mean dwell time
differences, for both the pilots flying and the co-pilots. For example, in
42
the Amsterdam-London flight for the co-pilots there was a significant
dwell time difference on the outside view for the climb-cruise-descent
phase compared to the approach-landing phase (Dunnet C: df = 9, F =
123.456, p < 0.05). For the pilot flying, there is an additional dwell time
difference between the approach and the landing (Dunnet C: df = 9, F =
123.456, p < 0.05). A list of significant differences can be found in
appendix A.
Mean attention on OUT (AL)
80
60
40
Value
20
MEAN_PNF
MEAN_PF
0
24.00
25.00
26.00
27.00
28.00
Segment
Figure 5. Division of attention on the outside view. (climb - 24) ( cruise - 25) (descent - 26) (approach
- 27) (landing - 28) (PF - Pilot flying) (PNF - Co-pilot)
There was a significant mean dwell time difference between the crew
members for the landing flight segment on the Amsterdam-London flight
(Dunnett C: df = 9, F = 123.456, p < 0.05) (see Figure 5). (A nonsignificant trend to the same effect was visible for the LondonAmsterdam flight.) This difference was expected and reflects the
assumed division of attention between the pilot flying and the pilot nonflying. In Figure 5, the attention during the climb (24), the cruise (25),
the descent (26) the approach (27) and the landing (28) can be seen on
the Y axis, as the percentage of time the eyes of each pilot were focused
on the outside view.
There were no significant mean differences between the measured
simultaneous dwell time and the expected simultaneous dwell time based
on random co-ordination of attention, for the main flight segments. Since
the simultaneous dwells measures method was not valid on data with the
43
time line synchronisation accuracy obtained in this experiment, these
results should be viewed as a demonstration of the simultaneous dwell
time analysis procedure only.
For the altitude bust event -> +2 min, the groups of pilots were divided
using the performance measure, time to correct the altitude bust. There
was a significant difference between the groups regarding the percentage
of time spent on the primary flight display, the relevant area of interest
(ANOVA: df = 1, F = 7.105, p < 0.05). When further divided into groups
of pilots flying and co-pilots, there was a significant dwell time
percentage difference for the co-pilots (ANOVA: df = 1, F = 8.038, p <
0.05).
No significant dwell time percentage differences (Dunett C: p < 0.05)
were found when groups were divided into groups of pilot flying / copilot, into groups with high / low subjective SA, subjective workload or
subjective performance, or into groups with high/low flight hours.
However, the co-pilot flying had significantly longer dwells on the
relevant display (PFD) than the pilot flying (df = 1, F = 5.636, p < 0.05).
For the map shift event, no significant correlation was found. However,
the pilots did spend time looking at the display during the event.
3.7. Conclusions
The answer to the first research question, is that the operationalisation
model is sufficient to explain the found results, as we shall se below. It
will be discussed in more detail in the concluding general discussion
chapter.
The second research question was as expected confirmed. As there were
differences between the dwell time percentages on the relevant displays
for the flights segments, the null hypothesis of no differences was
disproved. Furthermore, the significant dwell time differences on the
PFD for the altitude bust event, which showed the fast agents to spend
more time on the PFD, confirm this further, by disproving that the time
spent on the relevant display does not matter.
The answer to the additional question then, whether the SA system was
sufficient to handle the altitude bust event, is that even for the slowest
agents, the attention was sufficient to ensure safe handling of the event.
For the map shift event, despite the time spent looking at the display, the
44
event was either not discovered at all, or after several minutes of flight
with inaccurate SA regarding the position of the aircraft.
The third research question could not be completely answered since the
analysis of simultaneous dwells could not be carried out on the available
data. There was however a significant division of attention during the
Amsterdam-London landing.
The answer to the additional question, regarding the attention
management system is that, for the altitude bust event that the long-term
memory schemes were sufficiently good. There was no need for further
training. Furthermore, local attention distribution was sufficient. This
implicates that there was no need for attention direction by some aircraft
system. There was also no need for any mechanical system to be installed
to monitor the altitude capture, in case the one in the autopilot would fail.
It can also be concluded that the relevant aspects on the display were
comprehensible and salient enough. The aircraft log, finally, was in this
case functioning adequately as a mediator of retrospective SA, since it
was in all cases properly attended to and comprehended.
For the map shift event, the SA system did not work properly. The
relevant display was looked at and attended to by the crews during the
event and despite this the instrument malfunction was not discovered.
The conlusion then is that the malfunction was neither comprehensible
nor salient enough to be recognised by an unprepared crew. Here either
an additional mechanical monitoring system has to be installed, or the
display has to be redesigned, or the pilots has to receive more training to
discover the event.
The flap asymmetry event was not statistically analysed in the present
thesis, however, all crews discovered the event swiftly. The SA question
raised by the flap asymmetry event is somewhat different from the other
events. Whereas there are not many alternative ways of reacting to a gear
unsafe event, than to try and extend the landing gears again, for the flap
asymmetry there are at least two possible actions, to make a go-around
and land later, and to land immediately. The correct choice depends on if
the flaps will be increasingly asymmetrical with time or if the safest thing
to do is to delay the landing. This relates to the to the knowledge of how
the aircraft works. This knowledge will influence the projected future
situation, either to give the pilots an accurate prospective SA or an
inaccurate prospective SA. This illustrates that device models are
necessary means to reach accurate SA. In this report, the focus is not on
the impact of device models for accurate SA. Therefore this was not
45
examined more closely. As a remark, it can be mentioned that the
procedure for a flap asymmetry for this experiment was available and
used by the pilots, as mediated by a written instruction in the cockpit.
46
4. GENERAL DISCUSSION
The present thesis was conducted in order to scrutinise SA in the civil
cockpit. The model proposed in this thesis is intended to be sufficient to
model SA systems with varying degrees of localisation and distribution
of SA processes and models. The model has been fully applied to one
experimental situation, the one of this thesis. Further research is needed
to establish the general usefulness of the model.
The model could however be successfully applied to the experimental
situation described in this thesis. The research questions could be
operationalised by relating the measurement techniques to the applied SA
model and the questions could be answered as well. This does not prove
that the model will be successful for every experimental situation
regarding distributed SA, but this has to be examined with further
research.
A careful reader may have noticed that the data collection in the
experiment section is based on a very small group of pilots and crews. Of
course, a larger group would have been desirable. For economical
reasons, however, a larger group was not an option. Instead, a simpler
simulation could have been made, with more crews. Then, however, it
would have been uncertain how the results would have transferred to a
real world situation, since the actual layout, the design of the
instrumentation, and the feeling of real flight may influence both
workload and the use of normal procedures and eye scan patterns. The
last reason being the stronger for this thesis, since the assumption of
realistic eye scan patterns is what grounds the applied SA system model
in reality. If the eye scan patterns were not realistic, this model can’t tell
us this, and it will only show the properties of a system with the
measured eye scan patterns and the measured performance.
For further research, it is recommended to make more detailed artefact
studies of the instrumentation mediating the cues for the events. This
would give a more detailed description of what in the presentation was,
or was not, sufficiently good. Furthermore, since statistical analyses on
small groups are of questionable value, the emphasis on these can be
reduced. Not removed, since it is of importance to know whether the
instrumentation was at all attended. Instead a qualitative interpretation
could be of value, questioning how much time before onset to
recognition to action, would be acceptable, and of how much training is
dedicated to the discovery of the anomalies. Furthermore it would be
47
possible to analyse whether the discovered attention patterns and
behaviours are expected by pilots and by pilot trainers.
As this experiment was made within the VINTHEC project, the
experiment could not be optimally arranged to answer the questions of
this thesis. A compromise had to be made to get acceptable data for all
involved in the project. This meant that not all aspects of the SA system
could be measured. This could not have been done in any case, since too
many measures would have been too intrusive, and made the experiment
too unrealistic. To reach the local situation model of the agent, for
example, a probing technique such as Endsley’s SAGAT could have been
used, but to use this measure on top of the other measures would have
made the entire mass of measures too intrusive.
Considering the accuracy of the EPOG equipment in a real simulator
environment, this was in itself a research question for the VINTHEC
group, and was therefore not completely known in advance. The
accuracy, as expected, turned out to be sufficient both to answer the
VINTHEC theoretical questions, and to answer the general questions of
this thesis. To answer more detailed questions of the role of attention for
the updating process of situation models, higher measurement accuracy
than needed for the general conclusions, will have to be achieved. As a
result, parts of the SA definition of this thesis has not been grounded in
the empirical work of this thesis, but has been grounded in SA theory and
in the results of previously made SA research.
48
5. REFERENCES
Artman, H., & Garbis, C. (1998). SA as Distributed Cognition.
Proceedings of ECCE’98, Limerick.
Ashcraft, M. H. (1994). Human Memory and Cognition. New York:
Harper Collins
Ballard, D., & Rippy, L. (1994). A knowledge-based decision aid for
enhanced SA. IEEE-94CH-3573-0.
Bateman, D. (1997) Enhanced GPWS Improves Situational Awareness
and Provides More Effective Protection Near Terrain. ICAO Journal,
52(4), 7-8
Berggren, P. (1998). Situational Awareness, mental workload, and pilot
performance. Magisteruppsats, Institutionen för pedagogik och
psykologi, Linköpings Universitet.
Caretta, T. R., Zelenski, M. W. (1996) Situational Awareness: There is
no substitute for experience. Flying safety, 52(3), 18-20.
Derks, P.L., & Gillkin L. S. (1993). Incongruity, Incongruity Resolution,
and Mental States: The measure and Modification of Situational
Awareness and Control. NASA-CR-194568
Clark, H. C. (1996). Using Language. Cambridge: Cambridge University
Press.
Endsley, M. R. (1988). SA Global Assessment Technique (SAGAT).
Proceedings of the IEEE National Aerospace and Electronics
Conference.
Endsley, M. R. (1990). Predictive Utility of an Objective Measure of SA.
Proceedings of the Human Factors Society - 34th Annual Meeting, p 4145.
Fracker, M. L. (1988). A Theory of Situation Assessment: Implications
for Measuring SA. Proceedings of the Human Factors Society -32 nd
Annual Meeting, p 102-106.
49
Hartman, B. K., Moylan, P. (1994) Head-Up Guidance Systems: The
Foundation Of SA Systems. IEEE 0-7803-2425-0, p88-92
Heath, C. & Luff, P. (1992). Collaboration and Control - Crisis
management and multimedia technology in London underground line
control rooms. CSCW - An international journal, 1, 69-94.
Hutchins, E. (1990). The technology of team navigation. In J. Galegher,
R. E. Kraut & C. Egido (Eds.) Intellectual Teamwork - Social and
Technological Foundations of Co-operative Work. (pp 22-51). Hillsdae,
NJ:Erlbaum.
Hutchins, E. (1995) How a cockpit remembers its speeds. Cognitive
Science, 19, 265-288.
Jones, D. G. & Endsley, M. R. (1996). Sources of SA in Aviation.
Aviation, Space and Environmental Medicine, 67, p 507-512.
Metalis, S. A. (1993). Assessment of Pilot Situational Awareness:
Measurement Via Simulation. Proceedings of the Human Factors Society
- 37th annual meeting.
Mogford, R. H. (1997). Mental Models and SA in Air Traffic Control.
The Internaltional Journal of Aviation Psychology, 6(4), 331-341.
Neisser, U. (1976). Cognition and Reality: Principles and Implications of
Cognitive Psychology. San Fransisco: W H Freeman and Company.
Regal, D. M., Rogers, W.H., Boucek, G. P. (1988). Situational
Awareness in the Commercial Flight Deck: Definition, Measurement and
Enhancement. Aerospace Technology Conference and Exposition.
Sarter, N. B. & Woods, D. D. (1991). SA: A Critical But Ill-defined
Phenomenon. The International Journal of Aviation Psychology, 1(1),
45-57.
Schwartz, D. (1994). Training for SA. Air Line Pilot, 20-23, May edition.
Shively, R. J., Goodman, A. D. (1994). Effects of Perceptual
Augmentation of Visual Displays: Dissociation of Performance and
Situational Awareness. Proceedings of the Human Factors and
Ergonomics Society 38th Annual Meeting.
50
Sollenbergher, R.I., & Stein, E. S. (1995). The Effects of Structured
Arrival and Departure Procedures on TRACON Air Traffic Controller
Memory. DOT/FAA/CT-TN95/27
Tenney, Y.J., Adams, M. J., Pew, R. W. Huggins, A. W. F, & Rogers, W.
H. (1992). A Principled Approach to the Measurement of SA in
Commercial Aviation. NASA-CR-4451
VINTHEC Consortium. (1996). Technical Report: Human Awareness
Criteria. VINTHEC-WP1-TR 01, NLR, Amsterdam
VINTHEC Consortium. (1997). Technical Report: Review of Eye-pointof-gaze Equipment and Data Analysis. VINTHEC-WP2-TR 01, NLR,
Amsterdam
VINTHEC Consortium. (1997). Technical Report: Dynamic Measures of
Pilot Mental Workload (PMWL), Pilot Performance (PP), and
Situational Awareness (SA). VINTHEC-WP3-TR 01, NLR, Amsterdam
VINTHEC Consortium. (1998). Technical Report: Testplans for Pilot
Experiments. VINTHEC-WP4-TR 01, NLR, Amsterdam
VINTHEC Consortium. (1998). Technical Report: Technical
Implementation. VINTHEC-WP5-TR 01, NLR, Amsterdam
VINTHEC Consortium. (1998). Technical Report: Pilot Studies.
VINTHEC-WP6-TR 01, NLR, Amsterdam
Weick, K. E. & Roberts, K. H. (1993). Collective minds in organisations:
Heedful interrelating on flight decks. Administrative Science Quarterly,
38, 357-381.
Wellens, A. R. (1993). Group SA and distributed decision making: From
military to civilian applications. In N. J. Castellan (ed. ). Individual and
Group Decision Making: Current issues. (pp. 267-87). Erlbaum.
51
6. APPENDIX A
Significant dwell time percentage differences (Dunnet C: p < 0.05)
between the flight phases, for the pilot flying (PF) and the pilot nonflying (Where PF is not indicated).
AL PFD
Cruise <> Approach
PF Climb <> PF Approach & PF Landing
PF Cruise <> PF Approach & PF Descent
PF Descent <> PF Cruise & PF Approach & PF Landing
PF Approach <> PF Climb & PF Cruise & PF Descent & PF Landing
AL NAV
Cruise <> Approach
Descent <> Approach
Approach <> Cruise & Descent
PF Climb <> PF Approach & PF Landing
PF Cruise <> PF Approach & PF Landing
PF Descent <> PF Approach & PF Landing
PF Approach <> PF Climb & PF Cruise & PF Descent
PF Landing <> PF Climb & PF Cruise & PF Descent
AL OUT
Climb <> Approach & Landing
Cruise <> Approach & Landing
Descent <> Approach & Landing
Approach <> Climb & Cruise & Descent
Landing <> Climb & Cruise & Descent
PF Climb <> PF Approach & PF Landing
PF Cruise <> PF Approach & PF Landing
PF Descent <> PF Approach & PF Landing
PF Approach <> PF Climb & PF Cruise & PF Descent & PF Landing
PF Landing <> PF Climb & PF Cruise & PF Descent & PF Approach
AL ENG
AL CDU
52
Climb <> Approach & Landing
Descent <> Landing
Approach <> Climb
Landing <> Climb & Descent
AL A/P
Descent <> Landing
Landing <> Descent
PF Climb <> PF Landing
PF Cruise <> PF Landing
PF Descent <> PF Approach & PF Landing
PF Approach <> PF Descent
PF Landing <> PF Climb & PF Cruise & PF Descent
LA PFD
Climb <> Approach
Cruise <> Approach
Approach <> Climb & Cruise
PF Climb <> PF Approach & PF Landing
PF Cruise <> PF Descent & PF Approach & PF Landing
PF Descent <> PF Cruise & PF Landing
PF Approach <> PF Climb & PF Cruise & PF Landing
PF Landing <> PF Climb & PF Cruise & PF Descent & PF Approach
LA NAV
Climb <> Approach
Cruise <> Approach
Descent <> Approach
Approach <> Climb & Cruise & Descent
PF Climb <> PF Approach & PF Landing
PF Cruise <> PF Approach & PF Landing
PF Descent <> PF Approach & PF Landing
PF Approach <> PF Climb & PF Cruise & PF Descent
PF Landing <> PF Climb & PF Cruise & PF Descent
LA OUT
Climb <> Approach & Landing
Cruise <> Approach & Landing
Descent <> Approach & Landing
53
Approach <> Climb & Cruise & Descent
Landing <> Climb & Cruise & Descent
PF Climb <> PF Landing
PF Cruise <> PF Landing
PF Descent <> PF Landing
PF Landing <> PF Climb & PF Cruise & PF Descent
LA ENG
LA CDU
Cruise <> Approach & Landing
Approach <> Cruise
Landing <> Cruise
PF Cruise <> PF Approach & PF Landing
PF Approach <> PF Cruise
PF Landing <> PF Cruise
LA A/P
Climb <> Landing
Cruise <> Landing
Descent <> Landing
Landing <> Climb & Cruise & Descent
PF Cruise <> PF Approach & PF Landing
PF Approach <> PF Cruise
PF Landing <> PF Cruise
54
7. APPENDIX B
Nationaal Lucht- en Ruimtevaartlaboratorium
National Aerospace Laboratory (NLR)
VINTHEC Questionnaire
Beoordelings Schaal Mentale Inspanning
(BSMI)
Wilt u door middel van het zetten van een streepje op de verticale lijn aangeven hoeveel inspanning
het u heeft gekost om de zojuist door u verrichtte taak te voltooien.
150
140
130
120
Ontzettend inspannend
110
Heel erg inspannend
100
90
Erg inspannend
80
Behoorlijk inspannend
70
60
Tamelijk inspannend
50
40
Enigzins inspannend
30
Een beetje inspannend
20
Nauwelijks inspannend
10
Helemaal niet inspannend
0
Volgorde number: …….
Vlieger: flying / non-flying
Datum: ….... april 1998
Crew nummer: …....
Vlucht: AMS - LHR / LHR - AMS
Vlieger nr.: ……. (Links / Rechts) Tijd ….... : …....
55
Nationaal Lucht- en Ruimtevaartlaboratorium
National Aerospace Laboratory (NLR)
VINTHEC Questionnaire
Situational Awareness (SA)
Pilot Rating Scale
Yes
Was your SA on a
satisfactory level?
No
Yes
Was your SA on an
acceptable level?
No
My SA included all aspects of importance as well as a lot of other
aspects of the dynamic situation.
10
My SA included all aspects of
importance as well as some other
aspects of the dynamic situation.
9
My SA included all aspects of
importance (neither more nor
less) of the dynamic situation.
8
My SA was insufficient. A few
important aspects were out
of my control.
7
My SA was reduced. Some
important aspects were
out of my control.
6
My SA was low. A lot of
important aspects were
out of my control.
5
My SA was very low. About half
of the important aspects were
out of my control.
4
My SA was very, very low.
Most important aspects
were out of my control.
3
My SA was extremely low.
Almost all important aspects
were out of my control.
2
Yes
Was it possible to perform the
task with respect to your SA?
No
No SA at all. All important
aspects were out of my control.
Start here!
Pilot: flying / non-flying
Date: April ….... 1998
Crew number: …....
Pilot number: ……. (Left / Right)
1
Order number: …….
Flight: AMS - LHR / LHR - AMS
Time: ….... : …….
Remark: The actual size of this questionnaire is bigger. The size has
been adjusted so that it fits on a report page together with the standard
headers that are used throughout the entire report.
56
Sleep questionnaire
Length of sleep
• At what time did you go to bed last night?
• At what time did you get up this morning?
………………..
………………..
Scale for quality of sleep
The following 14 questions refer to your sleep during the past
night. The questions can provide a good impression of one's
quality of sleep. Please answer every question by ticking 'yes' or
'no', according to the alternative that best describes your rest of
last night.
Scale for Quality of Sleep (SQS)
1. I did not sleep at all last night
2. Last night, I woke up and had trouble getting back to sleep
3. I got out of bed last night
4. I find that I slept badly last night
5. Last night, I fell asleep very easily
6. I slept less than five hours last night
7. Last night, before I fell asleep, I was awake for at least half an hour
8. Last night, I woke up several times
9. Last night, I tossed and turned all night
10 I find that I slept well last night
.
11 I feel as if I only slept a couple of hours
.
12 After getting up this morning, I felt tired
.
13 I feel that I did not get enough sleep last night
.
14 After getting up this morning, I felt well rested
.
Please make sure that you’ve answered all questions!
57
ye
s










no


















Pre flight questions
1. How well do you think you will succeed with this flight?





Not at all
Average


Very
2. How motivated are you to perform this flight?




Not at all
Average



Very
3. How difficult do you think this flight will be?




Very simple
Average



Very difficult
4. How would you characterise this flight, what are typical aspects of this flight that you can
expect? (You can think of the holding, the workload that ATC might generate, and so on.)
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
…………………………………………………………………………………………………………
58
During flight ratings
1. I would describe the mission performance during this period as:


HopelessPoor



Faulty Mediocre Fair


Good Excellent
2. Did something peculiar occur during this period?
No
Go to question 2A
 Yes
Stop
2A. If yes, at what amount of relevant displays had you been
looking immediately before the odd situation occurred?


None


Some


Most

All
2B. If yes, how long did it take you to recognise that something
odd has been going on?







< 5 sec.½ min.> 1 min. > 2 min.> 3 min.> 4 min.> 5 min.
Post flight questions
1. How difficult did you find the flight?



Very simple

Average



Very difficult
2. How well would you say you succeeded with the flight?





Not very well
Average


Very well
3. How would you assess the level of realism of the flight?





Very poor
Average


Very good
4. How much ‘spare time’ did you have during the flight?





None at all
Average


A great amount
5. How much physical effort was required for the flight?







59

None at all
Average
Excessive
6. How much mental effort was required for the flight?




None at all
Average



Excessive
7. What was the overall level of physical workload?




None at all
Average



Excessive
8. What was the overall level of mental workload?




None at all
Average



Excessive
9. Did you find it difficult to evaluate any of the available information?






Very simple
Average

Very difficult
10. To what extend were you disturbed by information other than that required for the task?







Not at all
Average
A great deal
11. Concerning the flight control task…



I need to consciously think what to do

Average



I acted without conscious consideration
12. Concerning the navigation task…



I need to consciously think what to do

Average



I acted without conscious consideration
13. Considering the entire flight the helmet is
comfortable - uncomfortable
/
heavy - light
60
/
steady - sliding a bit
Post Amsterdam - London questions
1. Have you experienced a map shift before?
 Yes
 No
1A. If yes, can you give an estimation about how often you experienced a map shift?
………………………………………………………………………………………………….
1B. If yes, do you remember the last time you experienced a map shift? ……………
………………………………………………………………………………………………….
2. Did you ever experience a flap asymmetry before?  Yes
 No
2A. If yes, can you give an estimation about how often you experienced a flap asymmetry?
………………………………………………………………………………………………….
2B. If yes, do you remember the last time you experienced a flap asymmetry? ……………
………………………………………………………………………………………………….
61
Post London - Amsterdam Questions
1. Have you experienced an altitude bust before?
 Yes
 No
1A. If yes, can you give an estimation about how often you experienced a altitude bust?
………………………………………………………………………………………………….
1B. If yes, do you remember the last time you experienced a altitude bust? ……………
………………………………………………………………………………………………….
2. Have you experienced a gear unsafe before?
 Yes
 No
2A. If yes, can you give an estimation about how often you experienced a gear unsafe?
………………………………………………………………………………………………….
2B. If yes, do you remember the last time you experienced a gear unsafe? ……………
………………………………………………………………………………………………….
62
Questionnaires instructions
Before, after and during the simulator
questionnaires will be handed to you.
experiment
some
The responses to the questions as well as the psychophysiological
measurements will be stored separately from your personal
information such as your name or employer. Besides this
separation no one except for the experiment operators is allowed
and will be able to access this information. All the information will
be used for the purpose of this experiment only and will not be
used for medical purposes or become available to third parties.
Besides the measurements and recordings that will be made in the
simulator we are interested in your background, and how you
perceive the experiment. This information will be compared with
the experimental results and psychophysiological measurements.
Therefore we kindly ask you to take your time to fill in these
questionnaires.
Thanking you in advance,
The experiment management.
63
Flight experience
Please answer the following questions about your flight
experience:
Aircraft type
In the
function of
Hour
s
Period (MM/YY MM/YY)
….. / ….. - ….. /
1.
…..
….. / ….. - ….. /
2.
…..
….. / ….. - ….. /
3.
…..
….. / ….. - ….. /
4.
…..
….. / ….. - ….. /
5.
…..
….. / ….. - ….. /
6.
…..
….. / ….. - ….. /
7.
…..
….. / ….. - ….. /
8.
…..
….. / ….. - ….. /
9.
…..
….. / ….. - ….. /
10
…..
.
Please make sure that you’ve answered all questions!
If you have any experience with PC-based computer games that
simulate aircraft like Microsoft flight simulator, flight unlimited, and
so on, please give a brief description below of this experience and
the period you used those simulations regularly.
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
……………………………………………………………………………
…
64
Use of substances
Did you use any of the following substances in the past 24 hours:
Substance
Ye
s

No
If yes, what and how much

1.
Alcohol
2.
Caffeine


3.
Medication


4.
Nicotine


5.
Other


•
•
•
•
coffee: ……………….. cups
tea: ……………….. cups
cola: ……………….. glasses
other:………………..
Please make sure that you’ve answered all questions!
65