Humans in safety critical systems
Erik Hollnagel, 2000
Estimated number of “human
errors”
100
% Human action attributed as
cause
90
The diagram
shows the
attribution of
“human errors”
as causes, which
may be
different from
the contribution
of “human
errors” to
incidents /
accidents.
80
70
60
50
40
30
20
10
1960
1965
Erik Hollnagel, 2000
1970
1975
1980
1985
1990
1995
What is an “error”?
Incorre
ct
actions
Correctly performed actions
Detected and recovered
Detected but tolerated
Detected but not recovered
Undetected
Erik Hollnagel, 2000
Actual
outcomes =
intended
outcomes
Overt
effects
Latent
effects
Humans and system safety
Technology centredview
Humans are a major source of
failure. It is therefore
desirable to design the human
out of the system.
Automation permits the
system to function when
the limits of human
capability have been
reached.
Human-centred
view
Humans are the main resource
during unexpected events. It is
therefore necessary to keep
them in the system.
The conditions for
transition between
automation and human
control are often vague
and context dependent.
Automation does not use humans
effectively, but leaves them
with tasks that cannot be
automated - because they are
too complex or too trivial.
Automation is costeffective because it
reduces the skillrequirements to the
operators.
Conclusion: Humans are necessary to ensure
safety
Erik Hollnagel, 2000
Ironies of automation
The basic automation “philosophy” is that
the human operator is unreliable and
inefficient, and therefore should be
eliminated from the system.
1
“Designer errors can be a
major source of operating
problems.”
2
“The designer, who tries to eliminate the
operator, still leaves the operator to do the
tasks which the designer cannot think how to
automate.”
Lisanne Bainbridge (1987), “Ironies of
automation”
Erik Hollnagel, 2000
Automation double-bind
Safety
critical
event
Design teams
are fallible,
therefore
humans are
required in the
system
Erik Hollnagel, 2000
Humans are fallible, and
should therefore be
designed “out” of the
system
Maintaining control
What can help
maintain or regain
control?
What causes
the loss of
control?
Unexpected
events
Acute time
pressure
Not knowing
what happens
Not knowing
what to do
Not having the
necessary
resources
Erik Hollnagel, 2000
Being in control
of the situation
means:
Knowing
what will
happen
Knowing what has
happened
Sufficient time
Good
predictions of
future events
Reduced task
load
Clear
alternatives or
procedures
Capacity
to
evaluate and
plan
Cyclical HMI model
Goals for what
to do when
something
unusual
happens:
Goals [Identify,
Diagnose, Evaluate,
Action]
Tea
m
Erik Hollnagel, 2000
Informati
on /
feedback
Provides
/
produce
s
Modifi
es
Next
action
Current
understandin
g
Directs /
controls
Effects of misunderstanding
The dynamics of
the process only
leaves limited
time for
interpretation
Increases
demands to
interpretatio
n
Operator
may lose
control of
situation
Erik Hollnagel, 2000
Unexpecte
d
informatio
n /
feedback
Provides
/
produce
s
Inadequat
e actions
Loss of
accuracy increases
unexpected
information
Incorrect or
incomplete
understanding
Leads
to
Prevention and protection
Initiating event
(incorrect
action)
Prevention (control
barriers):
Protection
(safety
barriers):
Protection
(boundaries):
Active or passive barrier
functions that prevent the
initiating event from
occurring.
Active barrier
functions that
deflect
consequences
Passive barrier
functions that
minimise
consequences
Erik Hollnagel, 2000
Types of barrier systems
 Material barriers
 Physically prevents an action from being carried out,
or prevents the consequences from spreading
 Functional (active or dynamic) barriers
 Hinders the action via preconditions (logical,
physical, temporal) and interlocks (passwords,
synchronisation, locks)
 Symbolic barriers (perceptual, conceptual
barriers)
 requires an act of interpretation to work, i.e. an
intelligent and perceiving agent (signs, signals
alarms, warnings)
 Immaterial barriers (non-material barriers)
 not physically present in the situation, rely on
internalised knowledge (rules, restrictions, laws)
Erik Hollnagel, 2000
Barrier system types
 Physical, material
 Obstructions, hindrances, ...
 Functional
 Mechanical (interlocks)
 Logical, spatial, temporal
 Symbolic
 Signs & signals
 Procedures
 Interface design
 Immaterial
 Rules, laws
Erik Hollnagel, 2000
Barriers systems on the road
Symbolic:
requires
interpretatio
n
Physical: works
even when not
seen
Symbolic:
requires
interpretation
Erik Hollnagel, 2000
Symbolic:
requires
interpretation
Classification of barriers
Material,
physical
Containing
Restraining
Keeping
together
Dissipating
Walls,fences, tanks, valves
Safety belts, cages
Safety glass
Air bags, sprinklers
Function
al
Preventing
(hard)
Preventing
(soft)
Hindering
Symbolic
Countering
Regulating
Indicating
Permitting
Communicating
Locks, brakes, interlocks
Passwords, codes, logic
Distance, delays,
synchronisation
Function coding, labels,
warnings
Instructions, procedures
Signs, signals, alarms
Work permits, passes
Clearance, approval
Immateri
al
Monitoring
Prescribing
Monitoring
Rules, restrictions, laws
Erik Hollnagel, 2000
Barrier evaluation criteria
 Efficiency: how efficient the barrier is
expected to be in achieving its purpose.
 Robustness: how resistant the barrier is w.r.t.
variability of the environment (working
practices, degraded information, unexpected
events, etc.).
 Delay: Time from conception to implementation.
 Resources required. Costs in building and
maintaining the barrier.
 Safety relevance: Applicability to safety
critical tasks.
 Evaluation: How easy it is to verify that the
barrier works.
Erik Hollnagel, 2000