Rappel Accident
Human Performance Analysis
Jim Saveland, Ph.D.1
Ivan Pupulidy, M.Sc.2
“The trouble starts when I fail to notice that I see only whatever confirms my
categories and expectations but nothing else. The trouble deepens even further if I
kid myself that seeing is believing. That’s wrong. It’s the other way around. You
see what you expect to see.”
Karl Weick & Kathleen Sutcliffe
Managing the Unexpected
“Discourse and research around situation awareness may so far have shed little
light on the actual processes of attentional dynamics.”
Sidney Dekker
Ten Questions About Human Error
OVERVIEW
This paper has three main sections: an introduction, an explanation of events, and a
discussion of potential future improvements. The introduction covers some of the problems with
our language and the thinking behind typical attributions of cause and labeling of human error, and
a recap of the essential elements of the accident.
The explanation of events section begins where most analyses typically end – a discussion
of situational awareness. Going deeper leads to a discussion of visual awareness and attention. Not
seeing elements in our visual field, i.e. “blindness,” is the way the human visual system works. In
particular, the impact of expectations (confirmation bias) and the phenomenon of change blindness
are discussed. Since Rappeller #2 was relatively inexperienced, this section concludes with a brief
discussion of expertise.
The potential improvements section discusses the typical interventions of introducing
barriers (physical, functional, symbolic, and incorporeal) and enhancing training. These
interventions exist within the context of a continuum from “sharp end” factors at work in the here
and now to “blunt end” factors removed in space and time. A central characteristic of high
reliability organizations (HROs) is mindfulness. A central tenet of resilience engineering is that
people create safety through adaptive practice in context. A reasonable hypothesis worth testing is
that mindfulness training and psychological skills training adapted from sports psychology
(explored in appendix III) may result in improved safe performance (adaptive practice in context).
The paper concludes with a summary and recommendations. There are three appendices:
the methodology used in this analysis, background information on expertise, and a section on
mindfulness and psychological skills training.
1
Program Manager for Human Factors & Risk Management RD&A, Rocky Mountain Research Station; and current
Chair of Forest Service National Safety & Health Committee; Fort Collins, CO.
2
Human Performance Specialist, WO Fire & Aviation Management, Boise, ID
INTRODUCTION
There is an inherent dilemma created by the fact that policy always lags behind the science.
It takes time for science to translate into operational practice. Then it takes time to determine best
practices and convert best practices into corporate policy; and in that time, the science has
evolved. From the perspective of the science, the policy is out of date the minute it is published
and then it takes time to update the policy. This analysis attempts to follow the science rather than
policy. Policy as expressed in the Accident Investigation Guide (MTDC 2005) simply provides a
Human Factors Accident and Incident Analysis checklist that has its origins in Wiegmann and
Shappell’s (2003) Human Factors Analysis and Classification System (HFACS). Paletz et al.
(2009) point out the inadequacies of HFACS for handling social psychological phenomena and
make suggestions for expansion. Through the use of any of these checklists, we might be tempted
to simply categorize error, such as looking under Sensory and Perceptual Factors and stating “loss
of situational awareness,” “attention failure,” or “conditions that affect attention and situational
awareness.” While this may give a classification and an ability to count “human error,” it is a
result of hindsight bias and not all that useful in explaining how and why the accident occurred or
in preventing future accidents. Such a finding is a starting point for further analysis, not a
conclusion.
There are problems with the language we use and the thinking behind our search for
causality and attributions of human error. Leplat and Rasmussen (1984) suggest that the analysis
of accident reports should have the specific aim of identifying the potential for future
improvements rather than causes. Rasmussen (1990) details the problems with attributions of
human error and causality in the analysis of accident investigations. Woods and Cook (2002)
provide a series of maxims and corollaries to move forward from the problems of labeling human
error. One of those maxims is the attempt to escape hindsight bias. Knowledge of outcome tends
to overly simplify accounts of apparent ‘cause’ and narrows and distorts our view of practice –
how things are done. Hollnagel (2004) traces the evolution of accident models with the movement
away from potentially limiting notions of causality and human error.
The main types of accident models (Hollnagel 2004)
Sequential Models
Epidemiological
Models
Search Principle
Specific causes and
Carriers, barriers, and
well-defined links
latent conditions
Analysis Goals
Eliminate or contain
Make defenses and
causes
barriers stronger
Systemic Models
Tight couplings and
complex interactions
Monitor and control
performance
variability
Examples
Chain or sequence of
Latent conditions
Control theoretic
events (domino)
Carrier-barriers
models,
Tree models
Pathological systems
Chaos models,
Network models
Stochastic resonance
“Both the sequential and the epidemiological accident models represent thinking in terms
of clear cause-effect links. In these models accidents are resultant phenomena, in the sense that the
consequences are predictable – at least in principle – from knowledge about their constituent parts.
In contrast to that, the systemic models see accidents as emergent phenomena, as something that
arises out of the complex of conditions but which cannot be predicted in a similar manner. This
difference in views has important consequences for how accident analysis is done. And since the
2
way in which we understand and explain accidents determines how we respond, the differences
are also important for accident prevention.” (Hollnagel 2004)
Causal attributions are constructed, not discovered. “What you look for is what you find.”
Causal attributions are an artifact of the accident model we hold that guides our search for
information and how we make sense of that information. Current policy as expressed in the
Accident Investigation Guide is based on sequential models. The evolution of accident
investigations over the years can be summarized as employing sequential models in an attempt to
find a root cause resulting in first blaming the individual and later blaming supervisors, then to
employing epidemiological models where the focus of blame shifted to the organization. The time
has come to develop systems thinking and question the utility of a blame paradigm for preventing
future accidents.
This analysis will move in that direction by focusing on an explanation of events and
identifying potential future improvements, while avoiding attributions of cause and labeling of
human error. The reader should keep in mind the rarity, extreme impact, and only retrospective
predictability of this event.
Essentials of the accident:
Mission: Perform a proficiency rappel from a Bell 212 helicopter that is required every 14 days to
maintain technical competency.
A buddy check of equipment uncovered a broken Kong Clip on Rappeller #2’s (R-2) Trilink. This was R-2’s first season rappelling. R-2 had 10 rappels prior to the accident. The Kong
Clip is used to center the Forgecraft “J” hook at the forward corner of the Tri-link. R-2 went to a
member of the crew overhead, a spotter trainee, who opened his Tri-link and replaced the
damaged Kong Clip with a rubber O-ring. R-2 closed the Tri-link. The evidence suggests that the
Foregecraft “J” hook was no longer inside the Tri-link at this point. R-2 returned to the aircraft and
went through the pre-rappel procedures of a buddy check and a spotter check prior to loading the
aircraft. This was also the first season rappelling for the buddy who had nine rappels prior to the
accident. This was the spotter’s ninth season rappelling. An additional equipment check was made
in the air by the spotter when R-2 prepared for the rappel by hooking his Forgecraft “J” hook into
the Sky Genie. R-2 fell to his death a short-time later.
So the essential question to pursue is how could R-2, the spotter trainee, the buddy, and the
spotter not see the misconfigured harness (“J” hook not inside the Tri-link)?
EXPLANATION OF EVENTS
Situational Awareness (SA)
Situational awareness has become a very popular term in the last decade. There is a temptation
to say this is a case of multiple instances of “loss of situation awareness,” and stop there. We need
to dig deeper. Exactly what constitutes SA is a difficult undertaking. Situational awareness is
commonly thought of as “knowing what is going on around you.” “SA represents a continuous
diagnosis of the state of a dynamic world” Parasuraman et al. 2008). A more formal definition is:
“The perception of the elements in the environment within a volume of time and space, the
comprehension of their meaning and the projection of their status in the near future” (Endsley
1995, Endsley and Garland 2000). Thus, this definition has three levels (Wickens 2008):
Level 1 SA: Noticing events in a dynamic environment (perception/attention)
Level 2 SA: Understanding the meaning of those events (comprehension)
Level 3 SA: Predicting or projecting their implications for the future (simulation/projection)
3
“The way in which a person deploys his or her attention in acquiring and processing information
has a fundamental impact on situational awareness” (Endsley and Garland 2000). Situation
awareness can also be studied at the group/team level (e.g. Garbis and Artman 2004).
Human actions almost always have to meet multiple, changing, and often conflicting goals.
“Goal-directed behavior in a changing environment requires a context-sensitive balance between
two antagonistic challenges: On the one hand, goals [such as complete the mission] must be
shielded from interference, and distracting information should be inhibited (goal shielding); on the
other hand, the environment must be monitored for potentially significant information that may
afford a goal switch, as when, for instance, one notices the smell of fire while preparing a talk
(background monitoring). Goal shielding and background monitoring incur complementary
benefits and costs: Whereas goal shielding prevents interference, it increases the risk of
overlooking significant information. Conversely, whereas background monitoring facilitates
noticing task-irrelevant but potentially significant stimuli, it increases susceptibility to
interference” (Goschke and Dreisbach 2008).
People constantly try to optimize performance. This is done by “trying to achieve an
acceptable balance or trade-off between thoroughness and efficiency. Thoroughness means that
they try as best they can to do the right thing and do it in the right way, i.e., to choose the correct
action and to carry it out as well as possible. Efficiency means that they try to do this without
spending too much effort in order to meet the demands of the situation, regardless of whether
these demands are imposed by an external source or of their own making. This efficiencythoroughness trade-off (ETTO) is a common feature of human performance that seems to play a
role on the level of individuals and on the level of organizations alike” (Hollnagel 2004).
Thus, to understand situational awareness and the dynamic balancing acts of ETTO and
goal shielding – background monitoring in this particular context, we need to delve a little deeper
into the underlying psychological processes and systems, especially level 1 SA, including visual
awareness and attention.
Visual Awareness System
Our eyes contain a fovea and a periphery. Foveal vision is very slow and serial. The fovea
lets us see fine detail. When we read, we focus the fovea of our eyes on the letters we want to see.
In contrast, peripheral vision is useful for providing the overall perspective that lets us keep
ourselves well oriented in space. Peripheral vision is fast, but very inaccurate. We need both the
fovea and the periphery to carry on our lives. If you hold out your arm and focus on your
thumbnail, that is about the area covered by your fovea.
Popular metaphors of visual attention include a spotlight (enabling selective processing of
the ‘illuminated’ region of the visual field) and later as a zoom lens. For example, increases in
emotional arousal narrow the attentional field. Unfortunately, these metaphors have many
implications that are unconfirmed or disproven.
“Human vision seems so fast, effortless and reliable that most people mistake their
perceptual experience for a direct reflection of reality—after all, ‘seeing is believing.’ In reality,
visual perception is an interpretative act that consists of two key components: information analysis
and subjective awareness. Unlike a camera, which simply collects and stores raw visual
information projected from the environment, we experience the world as a detailed analysis of
vivid visual features, forms and objects. Our visual system analyses low-level feature information
such as color, orientation, texture, motions, depth and form, as well as the high-level structure and
meaning of visual objects” (Tong 2003).
4
The visual system has been the subject of research for decades, and the visual cortex is
probably the best studied area of the primate brain, yet it is still not well understood, with a variety
of explanatory models and theories that are controversial. It remains an area of active research,
especially the neural basis of conscious vision.
Attention
Attention is necessary but not sufficient for visual awareness. Attention is also a complex
area of ongoing research with a long history, and is now being studied at the cognitive,
neurosystem, cellular, synaptic, and genetic levels.
Attention is multifaceted. There is really no such thing as attention, but rather several
attentions. Wickens and McCarley (2008) talk about the varieties of attention being focused,
selective, switched, divided, and sustained. From a cognitive neuroscience perspective, “There are
three main functionally and anatomically distinct types of supramodal attentional control
systems—selection, orientation, and alertness…More recently, Posner and his colleagues have
refined the typology to orienting, alerting, and executive control” (Posner 2004).
Alerting attention is sometimes referred to as vigilance or sustained attention and has little
application in the context of this accident. Executive control is sometimes called supervisory
attention and includes top-down willful processes. One highly controversial issue with respect to
visual selective attention concerns the degree to which the top-down attentional mechanism
modulates attentional allocation to bottom-up deviation (Sawaki and Katayama 2008). In this case,
doing a buddy check or a spotter’s check is an example of a top-down attentional mechanism of
executive control. The check is meant to specifically direct attention. The misaligned harness is a
bottom-up deviation.
Orienting attention is an adjustment of position relative to a stimulus. It usually involves a
body, head, and/or eye movement. Pavlov called it the “investigating” or “what-is-it” reaction.
Traditionally it was studied as a reflex-like response. When orienting occurs, attention is also
directed to the stimulus. When confronted with a novel stimulus, animals show a strong orienting
response toward it. When repeated continually, animals habituate to it and the responses
progressively weaken. Orienting attention is either overt or covert. Overt shifts of attention occur
in synchrony with a physical movement, such as a saccade (eye movement). Covert shifts of
attention occur in the absence of any physical movement. A saccade cannot be executed until
covert attention has shifted to the target location. In viewing a naturalistic scene, observers
generally make two to three eye movements per second, pausing in between for fixations of 250300 ms. This is another area of active research with conflicting theories. For example, “To
understand the mechanisms of visual attention, it is crucial to know the relationship between
attention and saccades. Some theories propose a close relationship, whereas others view the
attention and saccade systems as completely independent.” (Belopolsky and Theeuwes 2009). As
the buddy/spotter check is progressing overt shifts of attention are occurring in synchrony with
eye movements and covert shifts of attention may also be occurring. Executive control as well as
overt and covert orienting attention in four people did not pick up the misconfigured harness. Let’s
turn our attention to how that’s possible.
Blindness
A misconfigured harness is a very rare event. Wolfe et al. (2005) demonstrated that when a
target is rare, participants are surprisingly poor at detecting it. The rarity of the target leads to
“disturbingly inaccurate performance.” Rich et al. (2008) explored some of the possible
mechanisms for why we miss rare targets, one of them being ending the visual search prematurely.
5
Research in visual attention has also revealed several ways that people don’t see what is in their
visual field. This is simply how the human visual system works.
Inattentional blindness is the “looked-but-failed-to-see” effect. It occurs when attention is
focused on one aspect of a scene and overlooks an object that is prominent in the visual field and
is well above sensory threshold. The “Gorilla basketball video” (Simons 2008), which has been
shown in several training sessions is an example. Inattentional blindness is easy to produce and
sometimes startling in strength. Inattentional blindness has been implicated as a common cause of
traffic accidents.
Change blindness is a failure to notice that something is different from what it was. Large
changes to a visual scene are very likely to go unnoticed if they occur during saccades (eye
movements) because visual analysis is suppressed during that time. Changes may even go
unnoticed if subjects are visually tracking the object at the moment of change (Triesch et al. 2003).
Rensink (2002) points out that it is important to distinguish between change and difference. “To
the degree that they are not the same, trying to ‘spot the difference’ between two side-by-side
images will be a rather different activity than trying to detect the change in a pair of sequentially
presented images.” This is particularly important when providing an explanation of what happened
in this accident. Showing side by side photos comparing proper and improper rigging will lead the
reader to incorrectly assume that they could easily spot the difference. A better method of
presentation to demonstrate the possibility for change blindness would be to show a picture of the
complete harness rigged correctly for a few seconds, then remove the picture and show the exact
same view but with the O-ring instead of the kong clip for just a few seconds and ask people if
they can spot the difference. “The surprisingly large variety of conditions that induce failures of
conscious perception reinforces the broad conclusion that we are often unaware of what would
otherwise be fully-visible stimuli” (Simons and Rensink 2003). Despite our remarkable blindness
to completed changes, people incorrectly tend to assume that such changes will be noticed easily,
a phenomenon that has been termed change blindness blindness.
Confirmation Bias
Confirmation bias is a tendency to search for or interpret information in a way that
confirms our preconceptions/expectations and to ignore, not look for, or undervalue what
contradicts our preconceptions. Hollnagel (2004) states that this feature of human thinking (a
strong tendency to look for confirming evidence) is an example of the Efficiency-Thoroughness
Trade-Off (ETTO) principle. Klayman and Ha (1987) state that some confirmation bias results are
due to over-application of a positive test strategy. A repeated review of an established condition
(e.g. a correctly configured harness) reinforces the expectation that the system is in that specific
configuration. This is an example of reinforcement through experience which leads one to miss
potentially critical anomalies.
The existence of inattention and change blindness, as well as the influence of expectations
on vision (confirmation bias), helps us make sense of this tragic event.
Expertise
This was Rappeller #2’s first season rappelling, with 10 rappels prior to the accident. This
was the spotter’s ninth season rappelling. Using the Dreyfus and Dreyfus (1986) five stage model
(appendix II), we can make a reasonable assumption that R-2 and the buddy were novice’s,
transitioning to advanced beginner. The spotter is most likely proficient, perhaps transitioning to
expert. While people with different levels of skill will pay attention to different cues in their
6
environment and differ in the way they make sense of the situation, both novice and expert alike
are susceptible to change blindness and confirmation bias.
POTENTIAL FUTURE IMPROVEMENTS
Taleb (2007) introduced the concept of a Black Swan event. Black Swan events have three
attributes: rarity (it lies outside the realm of regular expectations), extreme impact, and
retrospective (though not prospective) predictability. We often times concoct explanations for the
occurrence of Black Swans after the fact making it explainable and predictable. Taleb (2007) goes
on to talk about Near-Black Swan events, or “Gray” Swans: events that are somewhat tractable
scientifically – knowing about their incidence should lower our surprise. Gray Swans events are
rare, consequential, and somewhat predictable (expected). In many respects this accident can be
considered a Black Swan event. With the knowledge of change blindness and the effect of
expectations on vision we can think of it as a Gray Swan event. Taleb (2007) recommends that we
need to adjust to the existence of Black Swan events rather than naively try to predict them. The
strategy is to maximize tinkering (aggressive trial and error) and try to collect as many Black
Swan opportunities as possible. We are presented here with an opportunity to learn.
Sports psychologists (Vealy 2007, Weinberg and Gould 2007) often talk about attention in
terms of width (from broad, e.g. focused on the entire field of play, to narrow, e.g. focused on the
ball) and direction (from external, e.g. other players actions, to internal, e.g. feelings, thoughts,
sensations). Likewise, we can think about potential interventions along these dimensions of width
(blunt end / sharp end) and directions (barriers / training). Hollnagel (2004) depicts the continuum
from the sharp end to the blunt end:
Hollnagel (2004)
Barriers
“A barrier is, generally speaking, an obstacle, an obstruction, or a hindrance that may
either: (1) prevent an event from taking place, or (2) thwart or lessen the impact of the
consequences if it happens nonetheless. In the former case the purpose of the barrier is to make it
impossible for a specific action or event to occur. In the latter case the barrier serves, for instance,
to slow down uncontrolled releases of matter and energy, to limit the reach of the consequences, or
to weaken them in other ways” (Hollnagel 2004). Thus it is useful to make a distinction between
barriers that prevent and barriers that protect. Barriers may also be either active or passive, as well
7
as permanent or temporary. Hollnagel (2004) characterizes barrier systems as physical (e.g. design
of hardware), functional (e.g. a lockout), symbolic (warnings, signs, and symbols), and incorporeal
(e.g. rules or procedures).
Prior to this event, there was a reliance on redundant incorporeal barriers (procedures for
self check, buddy check, and spotter check) that are all susceptible to change blindness and
confirmation bias.
At the blunt end, the agency has already invested resources (e.g. the Missoula and San
Dimas Technology Development Centers) in the capacity to develop effective physical, functional,
symbolic, and incorporeal barriers.
Training
The training for helicopter rappelling to accomplish the mission appears rigorous.
Firefighters have some exposure to human factors concepts in the L-courses, but overall,
psychological skills training is minimal and ad hoc.
The existence of change blindness could be made widely known through experiential
training that would allow people to experience the phenomena for themselves. A DVD (Simons
2008) exists that could facilitate such training. While such training would reduce change blindness
blindness – the fact that people don’t think they are susceptible – it would not reduce the actual
phenomena of change blindness. While a knowledge of change blindness can not be expected to
prevent the phenomena, such knowledge may spur the invention of effective barriers.
The typical response to finding a “loss of situational awareness” is to make exhortations
and reminders to “maintain SA!” This will have a small and fleeting impact, if that. A central
characteristic of high reliability organizations (HROs) is mindfulness. We may also be tempted to
make equally ineffective exhortations and reminders to “be more mindful!” That being said, there
is mindfulness training that has been developed from two different perspectives and this is
explored more fully in appendix III. While we don’t know of any research that explicitly looks at
the effect of mindfulness training on change blindness or confirmation bias, this would be a
potentially productive area of future research.
A central tenet of resilience engineering is that people create safety through adaptive
practice in context. Hollnagel’s (2004) systemic accident model includes human performance
variability (both individual and organizational) as a main component. “Performance is variable for
many different reasons prime among which is the human tendency to adjust performance to
current conditions,… lack of constancy of perceptual and cognitive functions (such as attention),
etc.” (Hollnagel 2004). A reasonable hypothesis worth testing is that psychological skills training
in general and mindfulness awareness practices in particular that are adapted from sports
psychology (see appendix III) may result in improved safe performance (adaptive practice in
context).
Blunt End of Training
At the blunt end, organizational and cultural levels, training is often given short shrift,
especially when budgets get tight. There is often minimal investment in training, no more than
what is absolutely necessary to get the job done. There is a constant search and pressure for
cheaper, though questionably effective, methods such as on-line course material. Mental models of
training taking away time from production are common. Sport psychologists often lament the
disparity between time and effort spent on physical training in comparison with time and effort
spent on psychological skills, even though the difference that separates good from elite
8
performance is widely attributed to mental skills. In the wider culture, we seem reluctant to invest
very much in human performance and psychological skills, except in the training of elite athletes.
Most of the mindfulness training mentioned in appendix III is modeled after the MindfulnessBased Stress Reduction Program (MBSR) which is an eight week program, the common time
frame to establish habits, yet most corporate training is focused on three days or less.
In safety and accident investigation, there is a shifting focus away from finding and
correcting human error to recognizing humans create safety through adaptive practice in context.
This mirrors the larger shift in psychology to recognize the importance of positive psychology. In
introducing the field of positive psychology, Seligman and Csikszentmihalyi (2000) pointed out
that psychology had, since World War II, become a science largely about healing; concentrating
on repairing damage within a disease model of human functioning. “Psychology is not just the
study of pathology, weakness, and damage; it is also the study of strength and virtue. Treatment is
not just fixing what is broken; it is nurturing what is best” (Seligman and Csikszentmihalyi 2000).
Positive psychology has three central concerns: positive emotions, positive individual traits, and
positive institutions. It studies the strengths and virtues that enable individuals and communities to
thrive. Seligman and Csikszentmihalyi (2000) point to a major cultural hurdle that must be
overcome to the advancement of the field: “One legacy of the humanism of the 1960s is
prominently displayed in any large bookstore: The ‘psychology’ section contains at least 10
shelves on crystal healing, aromatherapy, and reaching the inner child for every shelf of books that
tries to uphold some scholarly standard.”
One of the functions of training is to pass along the latest science. When compared to other
sciences, funding for psychological science is at the bottom of the list. Human factors research is
only a small portion of that and is primarily focused on the aviation and health care industries.
Thus, the application of human factors knowledge in wildland fire must rely on research done
elsewhere in different contexts that hopefully can be applied to the wildland fire environment.
SUMMARY
The misconfiguration of a rappel harness that was not seen by four people was rare,
extreme, and only retrospectively predictable – a Black Swan event. With a greater understanding
of visual awareness, attention, change blindness and confirmation bias, we simply have a
retrospective explanation of events. With this understanding comes a great challenge and
responsibility: to maximize our individual and collective learning to prevent similar accidents
from happening in the future by developing appropriate barriers and improving training at both the
sharp end and the blunt end..
The typical response to lament the “loss of situational awareness” and exhort the troops to
“maintain SA” or be more “mindful” is inadequate and ineffective. Effective barriers are needed.
While there is currently no research linking mindfulness training (cognitive and reflective) to
change blindness, such training can be expected to increase the rate of acquisition of expertise
thereby improving performance. Reflective mindfulness training has also been demonstrated to
modify subsystems of attention and has the beneficial side effect of improving immune function.
The study of the impact of mindfulness training on change blindness would be a fruitful path of
research and development. R&D capacity in this area is limited.
Recommendation #1. Develop appropriate physical, functional, and/or symbolic barriers to
misconfiguration of the harness system that overcomes change blindness and confirmation bias.
9
Physical barriers such as redesign of hardware and functional barriers such as a witness seal on the
Tri-link are already being talked about.
 Include field personnel in re-design.
 Consciously assess increased levels of complexity in the re-design process, as the
introduction of more complexity could have unintended consequences.
Recommendation #2. Create a widespread awareness of change and inattention blindness. All
firefighter crews can incorporate the video, Surprising Studies of Visual Awareness from Viscog
Productions (Simons 2008) in their refresher training. Rappel crews could develop and incorporate
their own examples, using paired photos presented sequentially, as well as scenarios that
demonstrate what could go wrong so that personnel are trained to look for anomalies.
Recommendation #3. Recognize the importance of the development of professional expertise
(especially intuition) and that safety is created by our adaptive, innovative personnel.
 Focus on creating optimal learning environments.
 Explore ways to incorporate cognitive mindfulness learning concepts in all training
courses.
 Experiment with the application of reflective mindfulness, especially bringing mindful
awareness practices to movement (such as running and other physical training). Integrate
psychological skills training into physical fitness training, including providing information
through FireFit.
Acknowledgements
We thank Ron Small and Chris Wickens with Alion Science and Technology for our
conversation and their helpful perspectives and input. We thank Anne Black, Sidney Dekker,
Amishi Jha, Deanne Shulman, and Dave Thomas for their critical review of the draft analsysis.
References
Banbury, S. and S. Tremblay, eds. 2004. A Cognitive Approach to Situation Awareness: Theory
and Application. Ashgate, Burlington.
Begley, S. 2008. Train Your Mind, Change Your Brain. Ballantine Books, New York.
Belopolsky, A.V. and J. Theeuwes. 2009. When are attention and saccade preparation dissociated?
Psychological Science, Sep 24, Epub ahead of print.
Dekker, S.W.A. 2005. Ten Questions About Human Error. Lawrence Erlbaum Assoc., New York.
Dreyer, D. and K. Dreyer. 2009. Chi Running. Simon & Schuster, New York.
Dreyfus, H.L. and S.E. Dreyfus. 1986. Mind Over Machine. Free Press, New York.
Endsley, M.R. 1995. Toward a theory of situation awareness in dynamic systems. Human Factors
37:32-64.
10
Endsley, M.R. and D.J. Garland, eds. 2000. Situation Awareness Analysis and Measurement. CRC
Press, New York.
Ericsson, K.A. ed. 2009. Development of Professional Expertise: Toward Measurement of Expert
Performance and Design of Optimal Learning Environments. Cambridge University Press, New
York.
Ericsson, K.A., N. Charness, R.R. Hoffman, and P.J. Feltovich (eds.) 2006. The Cambridge
handbook of expertise and expert performance. Cambridge University Press, New York.
Ericsson, K.A., R.T. Krampe, and C. Tesch-Romer. 1993. The role of deliberate practice in the
acquisition of expert performance. Psychological Review 100(3):363-406.
Fitzgerald, M. 2007. Brain Training for Runners. Penguin Group Inc., New York.
Garbis, C. and H. Artman. 2004. Team situation awareness as communicative practices. In:
Banbury, S. and S. Tremblay, eds. A Cognitive Approach to Situation Awareness: Theory and
Application. Ashgate, Burlington.
Goschke, T. and G. Dreisbach. 2008. Conflict-triggered goal shielding: response conflicts
attenuate background monitoring for prospective memory cues. Psychological Science 19(1):2532.
Jackson, P. and H. Delehanty. 1995. Sacred Hoops. Hyperion, New York.
Jha, A.P., J. Krompinger, and M.J. Baime. 2007. Mindfulness training modifies subsystems of
attention. Cognitive, Affective and Behavioral Neuroscience 7(2): 109-119.
Hollnagel, E. 2004. Barriers and Accident Prevention. Ashgate. Burlington.
Kabat-Zinn, J. 1990. Full Catastrophe Living. Random House Inc., New York.
Kahneman, D. and G. Klein. 2009. Conditions for intuitive expertise: A Failure to Disagree.
American Psychologist 64(6):515-526.
Klayman, J. and Young-Won Ha. 1987. Confirmation, disconfirmation and information in
hypothesis testing. Psychological Review 94(2):211-228.
Langer, E.J. 1997. The Power of Mindful Learning. Da Capo Press, Cambridge.
Laughlin, T. 2006. Extraordinary Swimming for Every Body. Total Immersion Inc., New Paltz.
Laughlin, T. 2002. Triathlon Swimming Made Easy. Total Immersion, Inc., New Paltz.
Leplat, J. and J. Rasmussen. 1984. Analysis of human errors in industrial incidents and accidents
for improvement of work safety. Accident Analysis and Prevention 16(2): 77-88.
11
Mack, A. and I. Rock. 1998. Inattentional Blindness. MIT Press, Cambridge.
MTDC. 2005. Accident Investigation Guide, 2005 edition. 0567-2806-MTDC. U.S. Forest
Service. Washington, D.C.
Paletz, S.B.F., C. Bearman, J. Orasanu, and J. Holbrook. 2009. Socializing the human factors
analysis and classification system: Incorporating social psychological phenomena into a human
factors error classification system. Human Factors 51(4):435-445.
Parasuraman, R., T.B. Sheridan, and C.D. Wickens. 2008. Situation awareness, mental workload,
and trust in automation: Viable, empirically supported cognitive engineering constructs. Journal of
Cognitive Engineering and Decision Making 2(2):140-160.
Posner, M.I. ed. 2004. Cognitive Neuroscience of Attention. The Guilford Press, New York.
Rasmussen, J. 1983. Skills, rules, and knowledge; signal, signs, and symbols, and other
distinctions in human performance models. IEEE Transactions on Systems, Man, and Cybernetics
SMC-13(3):257-266.
Rasmussen, J. 1990. Human error and the problem of causality in analysis of accidents.
Philosophical Transactions of the Royal Society of London. Series B, Biological Sciences
327(1241): 449-462.
Rensink, R.A. 2002. Change detection. Annual Review of Psychology 53:245-477.
Rich, A.N., M.A. Kunar, M.J. Van Wert, B. Hidalgo-Sotelo, T.S. Horowitz, and J.M. Wolfe. 2008.
Why do we miss rare targets? Exploring the boundaries of the low prevalence effect. Journal of
Vision 8(15):15, 1-17.
Sawaki, R. and J. Katayama. 2008. Top-down directed attention to stimulus features and
attentional allocation to bottom-up deviations. Journal of Vision 8(15):4, 1-8.
Seligman, M.E.P. and M. Csikszentmihalyi. 2000. Positive psychology: An introduction.
American Psychologist 55(1):5-14.
Siegel, D.J. 2007. The Mindful Brain. W.W. Norton & Co., New York.
Simons, D.J. 2008. Surprising Studies of Visual Awareness, DVD. VisCog Productions,
Champaign.
Simons, D. J., and R.A. Rensink. 2003. Induced failures of visual awareness. Journal of Vision,
3(1):i.
Stanley, E.A. 2009. Neuroplasticity, mind fitness and military effectiveness. In The ‘New’
Biological Warfare: Biotechnology and the Future of America’s Military, Mark Drapeau, Ed.
National Defense University Press, Washington, D.C.
12
Stanley E.A., and A.P. Jha. In Press. Mind fitness and mental armor: Training the mind to improve
operational effectiveness and build warrior resilience. Joint Force Quarterly.
Styles, E.A. 2006. The Psychology of Attention, 2nd Edition. Psychology Press, New York.
Taleb, N.N. 2007. The Black Swan. Random House, New York.
Tharp, L. and L. Ferguson. 2007. Overachiever’s Diary. Total Immersion Inc., New Paltz.
Tong, F. 2003. Primary visual cortex and visual awareness. Nature Reviews 4:219-229.
Triesch, J., D.H. Ballard, M.M. Hayhoe, and B.T. Sullivan. 2003. What you see is what you need.
Journal of Vision 3:86-94.
Vealey, R.S. 2007. Mental skills training in sport. In: Handbook of Sport Psychology, 3rd ed. G.
Tenenbaum and R.C. Eklund eds. John Wiley & Sons Inc., Hoboken.
Weick, K. and T. Putnam. 2006. Organizing for mindfulness: Eastern wisdom and western
knowledge. Journal of Management Inquiry 15(3):275-287.
Weick, K.E. and K.M. Sutcliffe. 2007. Managing the Unexpected. John Wiley & Sons Inc., San
Francisco.
Weinberg, R.S. and D. Gould. 2007. Foundations of Sport and Exercise Psychology. Human
Kinetics, Champaign.
Wickens, C.D. 1984. Engineering Psychology and Human Performance. Charles E. Merrill
Publishing Co., Columbus.
Wickens, C.D. 2008. Situation awareness: Review of Mica Endsley’s 1995 articles on situation
awareness theory and measurement. Human Factors 50(3):397-403.
Wickens, C.D. 2009. The psychology of aviation surprise: An 8 year update regarding the noticing
of black swans. ISAP Keynote Address.
Wickens, C.D. and J.S. McCarley. 2008. Applied Attention Theory. CRC Press, New York.
Wiegmann, D. and S. Shappell. 2003. A Human Error Approach to Aviation Accident Analysis:
The human factors analysis and classification system. Ashgate, Burlington.
Wolfe, J. M., T.S. Horowitz, and N.M. Kenner. 2005. Rare items often missed in visual searches.
Nature 435: 439–440.
Woods, D.D. and R.I. Cook. 2002. Nine steps to move forward from error. Cognition, Technology
& Work 4:137-144.
Wright, R.D. and L.M. Ward. 2008. Orienting of Attention. Oxford University Press, New York.
13
Appendix I. Methodology
For this analysis, we read the draft accident investigation report, but did not access original
documents such as witness interviews. We then consulted the relevant literature [including the
psychology of attention (Rensink 2002, Simons 2008, Simons and Rensink 2003, Styles 2006,
Wickens 1984, 2009, Wickens and McCarley 2008), situational awareness (Banbury and
Tremblay 2004, Dekker 2005, Endsley and Garland 2000), cognitive neuroscience (Begley 2008,
Posner 2004, Wright 2008), sports psychology (Vealey 2007, Weinberg and Gould 2007), expert
performance (Dreyfus and Dreyfus 1986, Ericsson 2009, Ericsson et al. 1993, Kahneman and
Klein 2009), and mindfulness (Jha et al. 2007, Langer 1997, Siegel 2007, Stanley 2009, Weick
and Sutcliffe 2007)], requested some additional information from the investigation team, talked
with some rappellers, and conferred with some colleagues. A draft analysis was sent out for peer
review and substantial revisions were made.
14
Appendix II. Expertise
“The ability to recognize that a situation is anomalous and poses a novel challenge is one
of the manifestations of authentic expertise” (Kahneman and Klein 2009). Sports psychologists
have long noticed differences in attentional processing between experts and novices (Weinberg
and Gould 2007). The level and range of skill brings us to the study of expertise (see Ericsson et
al. 2006, Ericsson 2009). Dreyfus and Dreyfus (1986) presented a five stage model for the
acquisition of expertise.
1 Novice



Rigid adherence to taught rules or plans
Little situational perception
No discretionary judgment
2 Advanced Beginner



Guidelines for action based on attributes or aspects.
Situational perception still limited
All attributes and aspects are treated separately and given
equal importance
3 Competent




Coping with "crowdedness"
Now sees actions at least partly in terms of longer-term goals
Conscious deliberate planning
Standardized and routine procedures
4 Proficient





Sees situations holistically rather than in terms of aspects
Sees what is most important in a situation
Perceives deviations from the normal pattern
Decision-making less labored
Uses maxims for guidance, whose meaning varies according to
the situation
5 Expert



No longer relies on rules, guidelines or maxims
Intuitive grasp of situations based on deep tacit understanding
Analytic approaches used only in novel situations or when
problems occur
Vision of what is possible

Dreyfus and Dreyfus (1986) five stage model of expertise
The amount of attention needed by a task depends on the level of skill. Rasmussen (1983)
states there are three levels of performance: skill-, rule-, and knowledge-based performance.
“The skill-based behavior represents sensory-motor performance during acts or activities
which, following a statement of an intention, take place without conscious control as smooth,
automated, and highly integrated patterns of behavior…. Characteristically, skilled performance
rolls along without conscious attention or control. The total performance is smooth and integrated,
and sense input is not selected or observed: the senses are only directed towards the aspects of the
environment needed subconsciously to update and orient the internal map. The man looks rather
than sees” (Rasmussen 1983).
15
“At the next level of rule-based behavior, the composition of such a sequence of
subroutines in a familiar work situation is typically controlled by a stored rule or procedure which
may have been derived empirically during previous occasions, communicated from other persons’
know-how as instruction or a cookbook recipe, or it may be prepared on occasion by conscious
problem solving and planning. The point here is that performance is goal-oriented but
structured…through a stored rule. Very often, the goal is not even explicitly formulated but is
found implicitly in the situation releasing the stored rules…. The boundary between skill-based
and rule-based performance is not quite distinct, and much depends on the level of training and on
the attention of the person. In general, the skill-based performance rolls along without the person’s
conscious attention, and he will be unable to describe how he controls and on what information he
bases the performance” (Rasmussen 1983).
“During unfamiliar situations, faced with an environment for which no know-how or rules
for control are available from previous encounters, the control of performance must move to a
higher conceptual level, in which performance is goal-controlled and knowledge-based. In this
situation, the goal is explicitly formulated, based on an analysis of the environment and the overall
aims of the person. Then a useful plan is developed—by selection—such that different plans are
considered, and their effect tested against the goal, physically by trial and error, or conceptually by
means of understanding the functional properties of the environment and prediction of the effects
of the plan considered. At this level of functional reasoning, the internal structure of the system is
explicitly represented by a ‘mental model’ which may take several different forms” (Rasmussen
1983).
Kahneman and Klein (2009) state that two conditions must be satisfied for skilled intuition
to develop: an environment of sufficiently high validity and adequate opportunity to practice the
skill. Ericsson et al. (2006) describe a range of factors that influence the rate of skill development
and include the type of practice people employ, their level of engagement and motivation, and the
self-regulation processes they use. Ericsson et al. (1993) talk about the importance of “deliberate
practice” in developing expertise. Expert performance is acquired slowly over a very long time as
a result of deliberate practice. It typically takes 10 or more years experience and the accumulation
of 10,000 hours of practice over that time. Yet, expertise is not attained automatically as a function
of extended experience. It requires repeated experiences in which the individual can attend to the
critical aspects of the situation and incrementally improve her or his performance in response to
knowledge of results, feedback, or both from a teacher. Deliberate practice requires highly
structured activities, the explicit goal of which is to improve performance. It requires effort and is
not inherently enjoyable. The motivation to practice comes from a desire to improve performance.
16
Appendix III. Mindfulness and Psychological Skills Training
Mindfulness
Being “mindful” has a range of definitions, from the common everyday notion of “bearing in
mind or inclined to be aware” to specific educational, clinical, and scientific definitions of the
term. Siegel (2007) categorizes mindfulness into two categories: cognitive mindfulness and
reflective mindfulness. Weick and Putnam (2006) make a similar distinction.
Cognitive Mindfulness
Siegel (2007) uses the term “mindful learning” or “cognitive mindfulness” to refer to Ellen
Langer’s conceptualization of mindfulness. For Langer (1997), mindfulness is a flexible state of
mind in which we are actively engaged in the present, noticing new things and sensitive to
context. People act less mindfully when they rely on past categories, act on “automatic pilot,” and
fixate on a single perspective without awareness that things could be otherwise. “The essence of
this approach is to offer learning material in a conditional format rather than as a series of absolute
truths. The learner in this way is required to keep an ‘open mind’ about the contexts in which this
new information may be useful. Involving the learner in the active process of education also is
created by having students consider that their own attitude will shape the direction of
learning…Langer suggests that the point of conditional learning is to leave us in a healthy state of
uncertainty, which will result in our actively noticing new things” (Siegel 2007). Teachers can use
terms such as “may,” “might be,” or “sometimes” instead of “is” to promote conditional
uncertainty. Mindful learning consists of openness to novelty; alertness to distinction; sensitivity
to different contexts; implicit, if not explicit, awareness of multiple perspectives; and orientation
to the present. This form of mindfulness is a flexible state of mind in which we actively notice
new things, are sensitive to context, and engage in the present. Weick and Sutcliffe (2007) build
on Langer’s work to present mindfulness as the foundation for organizing for high reliability. “By
mindfulness we mean the combination of ongoing scrutiny of existing expectations, continuous
refinement and differentiation of expectations based on newer experiences, willingness and
capability to invent new expectations that make sense of unprecedented events, a more nuanced
appreciation of context and ways to deal with it, and identification of new dimensions of context
that improve foresight and current functioning” (Weick and Sutcliffe 2007). Weick and Sutcliff
(2007) contrast mindfulness with mindlessness. “A tendency toward mindlessness is characterized
by a style of mental functioning in which people follow recipes, impose old categories to classify
what they see, act with some rigidity, operate on automatic pilot, and mislabel unfamiliar new
contexts as familiar old ones. A mindless mental style works to conceal problems that are
worsening” (Weick and Sutcliff 2007).
Reflective Mindfulness
Siegel (2007) uses the term “reflective mindfulness” to refer to mindful awareness
practices (MAPs) that have there origins in various forms of meditation. “Direct experience in the
present moment has been described as a fundamental part of Buddhist, Christian, Hindu, Islamic,
Jewish, and Taoist teaching. The clinical application of the practice of mindfulness meditation
derived from the Buddhist tradition has served as a focus of intensive study on the possible neural
correlates of mindful awareness. These studies, across a range of clinical situations, from medical
illness with chronic pain to psychiatric populations with disturbances of mood or anxiety, have
shown that effective application of secular mindfulness meditation skills can be taught outside of
any particular religious practice or group membership” (Siegel 2007). Probably the best known of
17
these secular practices in Mindfulness-Based Stress Reduction (MBSR) developed by Jon KabatZinn (1990) 30 years ago at the University of Massachusetts Medical School. Kabat-Zinn (1990)
defines mindfulness as “paying attention, on purpose, in the present moment, non-judgmentally, as
if your life depended on it.”
Reflective mindfulness has influenced a variety of approaches to psychotherapy for the
treatment of a wide range of conditions from eating disorders to anxiety, posttraumatic stress
disorder, and obsessive-compulsive disorders. Mindfulness based cognitive therapy (MBCT) has
been shown to prevent relapse in cases of chronic depression. Mindfulness has also become a part
of Dialectical Behavior Therapy (DBT) and Acceptance and Commitment Therapy (ACT).
“Studies have shown that specific applications of mindful awareness improve the capacity
to regulate emotion, to combat emotional dysfunction, to improve patterns of thinking, and to
reduce negative mindsets” (Siegel 2007). Research on some dimensions of mindful awareness
practices reveals that they “greatly enhance the body’s functioning: healing, immune response,
stress reactivity, and a general sense of physical well-being are improved with mindfulness”
(Siegel 2007).
The educational use of cognitive mindfulness concepts, and the clinical use in medical and
mental health practices of reflective mindful awareness; while quite distinct in practice, may share
common neural pathways. The bottom line is that mindful awareness is a form of experience that
seems to promote neural plasticity and has been shown to modify subsystems of attention (Jha et
al. 2007). Begley 2008 chronicles the radical paradigm shift in cognitive neuroscience over the
past few decades. It is now firmly established that the brain remakes itself throughout life, in
response to outside stimuli—to its environment and to experience, a phenomena known as
neuroplasticity. In the last twenty years the research on neuroplasticity has gone from: not being
able to get published, to being published in obscure journals, then to published in mainstream
journals, and finally to being published in Science and Nature while moving into the lexicon of
mainstream society. Neuroplaticity occurs only when the mind is in a particular mental state, one
marked by attention and focus. This is related to the concept of deliberate practice, essential to the
development of expertise.
Psychological Skills Training
What is missing from Siegel’s (2007) overview of mindfulness is another field of
application: the training of elite athletes. For sports psychologists, psychological skills training
(PST) is defined as “systematic and consistent practice of mental or psychological skills for the
purpose of enhancing performance, increasing enjoyment, or achieving greater sport and physical
activity self-satisfaction” (Weinberg and Gould 2007). Attentional focus (concentration) is
considered the most important mental skill needed in sport (Vealey 2007), and is actively
developed through psychological skills training (PST). Exercises to increase concentration skills
are often very similar if not identical to mindfulness awareness practices. “Many athletes have
used meditation to mentally prepare for competition, asserting that it improves their ability to
relax, concentrate, and become energized” (Weinberg and Gould 2007). Mindfulness awareness
practices have been used in running (Dreyer and Dreyer 2009), and swimming (Laughlin 2002,
2006, Tharp and Ferguson 2007). Phil Jackson, current coach of the world champion Los Angeles
Lakers, teaches mindfulness skills to his teams. As a coach he has won more NBA titles (10) than
any other coach. He credits his success to the triangle offence, team play, and mindfulness
(Jackson and Delehanty 1995). His speaker’s profile on the internet states, “Phil Jackson’s system
centers on the power of awareness and selfless teamwork to achieve success on the court and to
18
cultivate mindfulness in everyday life. As he puts it, ‘Awareness is everything. Being aware is
more important than being smart.’”
Sports psychologists often don’t refer to teaching focus skills as meditation, yet the
exercises are sometimes identical, including the nonjudgmental focus on the present moment.
“Concentration is the ability to maintain focus on relevant environmental cues. When the
environment changes rapidly, attentional focus must also change rapidly. Thinking of the past or
the future raises irrelevant cues that often lead to performance errors” (Weinberg and Gould
2007). There is also the recognition of the importance of non-judgment: “one of the biggest
obstacles athletes face in maintaining concentration is the tendency to evaluate performance and
classify it as good or bad” (Weinberg and Gould 2007).
Tharp and Ferguson (2007) work with the highly successful Army triathlon team at West
Point. The following excerpt from Tharp and Ferguson (2007) on the concept of focus could just
as easily be an introduction to a mindfulness awareness practice:
“The concept of focus is tricky to understand because there are really three parts to
it. The first is what everyone thinks is focus—a single-minded concentration on the
task at hand with the conscious and unconscious mind dedicated to the correct
completion of the task. The second part of focus is losing focus. Yes, the second
part of focus is losing focus. [Thought sampling studies reveal that the median
length of time during which thought content remains on target is approximately 5
seconds.] The third part is beating yourself up for losing focus. Being focused and
losing focus is human. Beating yourself up over losing focus is learned behavior
and can be unlearned on the way to becoming a champion. If you think about fly
fishing, it’s easier to understand how the concept of focus and regaining focus
works. A fisherman casts the line into the river and almost immediately it begins
floating downstream. To counteract this, a simple wrist movement brings the line
back upstream. The key here is that the fisherman expects the line to go
downstream, away from where it was originally cast. If you think of focus as the
line being cast, and losing focus as the line going downstream, it is easy to look at
regaining focus as a simple matter of bringing the line back upstream—without a
lot of drama. If you accept that you will lose focus, it is much easier to regain it and
you don’t waste time being angry over losing focus, which, of course, just makes
you lose focus.” (Tharp and Ferguson 2007)
The essence of a mindfulness awareness practice is a four step process:
1. Directing and sustaining attention on a selected object
2. Detecting mind wandering and distractions
3. Disengaging attention from distracters and shifting attention back to the selected
object
4. Cognitive reappraisal of distracter that is non-judgmental (e.g. “just a thought,”
“it’s OK to be distracted”)
The practice is noticing drifting away from present moment awareness and coming back as
quickly as possible. Objects of focus can be quite varied. The breath and sounds are common
objects of focus and when done while sitting cross-legged with eyes closed give the stereotypical
impression of meditation. However, mindfulness awareness practices are also associated with
movement. Yoga and tai chi have a long tradition of incorporating a focus on movement as a
mindfulness awareness practice. Focusing on the tastes and textures of food while eating is also a
traditional mindfulness awareness practice, as is slow walking. Running is the primary method
19
that firefighters use to develop and maintain aerobic fitness and can be used as a mindfulness
awareness practice. Dreyer and Dreyer (2009) have four pages of “form focuses” for running.
Similarly, Fitzgerald (2007) stresses the importance of using “proprioceptive cues” to improve the
biomechanics of running form and thus enhance performance. The visual field could also be used
as an object of focus. Here attention could be deployed on foveal vision or peripheral vision.
Another traditional mindful awareness practice is a body scan, making a systematic scan of
sensations in the body. This can be done while sitting, lying still, or while moving. A body scan
can be employed in what sports psychologists call an associative attentional strategy (monitoring
bodily functions and feelings, such as heart rate, muscle tension, level of perceived exertion, and
breathing rate); in contrast to a dissociative attentional strategy (distraction and tuning out). Early
research indicated that elite runners tended to use an associative strategy while non-elites used a
dissociative strategy. It is now seen along a continuum rather than a strict dichotomy. An
associative attentional strategy can be cultivated using the body scan as a mindful awareness
practice, for example monitoring the level of relaxation throughout the body during a run.
In addition to the training of elite athletes, the U.S. Army and Marines have begun to
investigate the application of mindfulness awareness practices (Stanley 2009. Stanley and Jha In
Press). Region 2 and Region 4 of the Forest Service have also begun to explore the application of
mindfulness awareness practices.
Finally, while mindfulness training has been shown to modify subsystems of attention (Jha
et al. 2007), we are not aware of any research yet to see if training in mindfulness awareness
practices will affect change blindness, though that would be an interesting line of research.
20