HFA
Human Factors
Associates
Assessing Organizational Safety Effectiveness
as an Integral component of a
Safety Management System (SMS)
Anthony Ciavarelli, ED.D.
September 2012
HFA
Human Factors
Associates
AGENDA
•
•
•
•
•
•
•
Organizational Accidents
Sources of Human Error
High - Reliability Organizations
Organizational Effectiveness and Safety Culture
Assessing Safety “culture” -- Climate
Safety Status of Aviation and Aerospace
SMS Implementation and Safety Culture
Copyright © HFA
HFA
Human Factors
Associates
Aviation and Aerospace Accidents
• Air Florida Washington DC “start up airline”
• ValuJet Florida Everglades “outsourced cargo services”
• Challenger Space Shuttle “risk perception -- decision”- o-rings
• Columbia Space Shuttle “risk perception—decision”- foam
• Buffalo “aircraft control to stall” – pilot training & fatigue
• Air France 447 – aircraft design & pilot training
Copyright © HFA
HFA
Human Factors
Associates
Sources of Human Errors
• Attitudes about job safety and performance
• Culture of Safety is deficient in key areas, such
leadership commitment and non-punitive reporting
• Inadequate standards, training and monitoring of
practitioner qualifications
• Poor communications and information flow
• Non-compliance with rules or best practices
• Stress, fatigue, production pressure, short staffing and
lack of resources
Copyright © HFA
HFA
Human Factors
Associates
Supervisory Contributions
•
•
•
•
•
•
•
•
Failure to instill strong safety values/culture
Pressure to complete a job or meet schedule.
Failure to establish or enforce standards.
Over-tasking pilots or maintainers
Failure to manage fatigue or stress.
Inadequate resources to perform job safely.
Poor equipment operating status/reliability.
Failure to manage known risks, including high-risk pilots or
maintainers, or other front line personnel.
Copyright © HFA
HFA
Human Factors
Associates
Naval Aviation Major (Class A) Flight Mishaps
7 aircraft
destroyed in
FY-11
all in flight
mishaps
776 aircraft
destroyed in 1954
Angled Decks
Aviation Safety Center
Naval Aviation Maintenance Program
RAG (FRS) Concept Initiated
NATOPS Program
Squadron Safety Program
System Safety Aircraft Design
CRM (crew-resource-mgmt)
Aircrew reviews
Risk management (ORM)
Safety climate-culture
October 2011
Fiscal Year
HFA
Human Factors
Associates
High Reliability - US Naval Aviation
Copyright © HFA
HFA
Human Factors
Associates
High Reliability - US Naval Aviation
Copyright © HFA
HFA
Human Factors
Associates
Naval Aviation Risk Mitigation
•
•
•
•
•
•
•
•
Recruitment and selection
Aeromedical screening
Training standardization and qualifications
Command supervision and risk management
Aircrew performance reviews
Human Factors reviews (Boards & Councils)
Crew Resource Management (CRM)
Safety Climate and Culture Assessments
Copyright © HFA
HFA
Human Factors
Associates
Key Organizational Accident Theories
Human Factors, Inc.
•
•
•
•
•
•
Incubation Periods (Turner,1978)
The Normal Accident (Perrow, 1984)
The High-Reliability Organization (Roberts, 1993)
Organizational Culture (Shein, 1990)
Normalization of deviance (Vaughn, 1997)
Organizational Sense-making (Weick, 1999)
Copyright © HFA
HFA
Human Factors
Associates
High Reliability - Safety Culture
Human Factors, Inc.
Copyright © HFA
HFA
Human Factors
Associates
Organizational Culture
Shared Values (What is important) and Beliefs (How
things work) that interact with an organizations
structures and control systems to produce
Behavioral Norms (The way things work around
here).
(Adapted: DuPont Culture 98 -- originally Reason 1990)
Copyright © HFA
HFA
Human Factors
Associates
High - Reliability Organization (HRO)
• A culture of trust, shared values, and risk
mitigating communication processes.
• Communication that provides opportunities
for open discussion and improvement.
• Distributed decision-making, “where the buck
stops everywhere.”
(Roberts, 1997)
Copyright © HFA
HFA
Human Factors
Associates
Safety Culture
• Shared values about what is safe and unsafe
• Common beliefs about how to conduct safe
operations
• Behavioral norms that govern risk-taking,
everyday procedures and precautions
• Transmission of values, beliefs and accepted
practices to others.
Copyright © HFA
HFA
Human Factors
Associates
Emergence of Just Culture Culture Concept
The phrase just culture refers to the principles for achieving a culture in
which frontline personnel feel comfortable disclosing errors—including
their own—while maintaining professional accountability... A just culture
recognizes that individual practitioners should not be held accountable for
system failings over which they have no control.
A just culture also recognizes anyone can produce errors as a consequence
of predictable interactions between human operators and the systems in
which they work. However, in contrast to a culture that turns to blame as
its governing principle, a just culture does not tolerate conscious disregard
of rules, reckless behavior or gross misconduct (AHRQ 2008)
Copyright © HFA
HFA
Human Factors
Associates
A Concise Definition of Safety Culture
Strictly speaking, the survey metrics are measuring “safety
climate”, which is considered a snapshot of employee attitudes
that reflect the underlying culture of the organization Flin,
Mearns, O’Connor and Bryden (2000). We use the findings of
the survey to address specific issues regarding the state of a
company’s safety culture, which is defined as:
“The values, beliefs, and prevailing practices regarding safety
that are routinely demonstrated and passed on to all
employees from generation to generation.”
Copyright © HFA
HFA
Human Factors
Associates
Copyright © HFA
HFA
Human Factors
Associates
Copyright 2001-2007 HFA, Inc
Sample Survey Items
.
Copyright © HFA
HFA
Human Factors
Associates
Survey Items
Copyright 2001-2007 HFA, Inc.
Copyright © HFA
HFA
Human Factors
Associates
Copyright 2001-2007 HFA, Inc.
Survey Items
HFA
Human Factors
Associates
Copyright © HFA
HFA
Human Factors
Associates
Copyright © HFA
HFA
Human Factors
Associates
High Reliability Organizational (HRO) Model
SPA
SCRS
RSKM
Copyright © HFA
QA
LDSHP
HFA
HRO Performance Dashboard
Human Factors
Associates
SPA
QA
SCRS
RSK MNGT
LDSHP
Diagnostic Feedback: Fictitious Data
100%
5,0
90%
4,5
80%
4,0
70%
60%
3,5
50%
3,0
40%
2,5
30%
20%
2,0
1
3
5
7
9
11
13
15
17
19
21
23
25
Copyright 2001-2007 Human Factors Associates, Inc.
27
29
31
33
Copyright © HFA
35
37
39
41
43
45
47
49
51
53
55
57
59
HFA
Human Factors
Associates
Placement of a specific rating
on Normal Distribution Curve
Rating is 1 standard deviation below mean
Rating is in 14th percentile
86% of ratings are above this one
Copyright © HFA
HFA
Human Factors
Associates
Copyright 2001-2007 HFA, Inc.
Feedback Display
(fictitious data)
HFA
Human Factors
Associates
Typical Demographic Statistics
Higher Ranking Personnel Give Higher Ratings
Average CSA Scores by Military Rank
5
4.5
4
3.5
Military Rank
3
O4 - O6
O1 - O3
E6 - E9
E1 - E5
2.5
Senior Commissioned
Overcommitted
Short
Resources
Junior Enlisted
Rank differences statistically significant (p<.001)
Survey Items
Copyright © HFA
61
58
55
52
49
46
43
40
37
34
31
28
25
22
19
16
13
10
7
4
1
2
HFA
Human Factors
Associates
Normative Data: Aviation or Aerospace
42.56%
76.03% Favorable
Survey Return Rate: 50%
33.47%
9.65% Problematic
3.05%
14.23%
6.60%
Copyright © HFA
HFA
Human Factors
Associates
Normative Benchmark Comparison Across
Domains
Comparison
% Problematic
Mean
Major Airlines
Data restricted to Airline Client
US Hospitals
US Navy
12.10
4.10
not reported
not reported
O&G off shore transport
Air medical One
Air medical Two
Energy/Power
Aerospace
19.22%
8.9%
7.5%
10.6%
7.97 %
3.5
4.0
4.0
3.8
4.1
Copyright © HFA
HFA
Human Factors
Associates
What the People on the ground have to say -- over 12
years of survey data in aviation and aerospace.
• Flight Operations
–
–
–
–
–
–
• Maintenance
–
–
–
–
–
–
–
Schedule pressure
Pilot shortage
Lack of trust
Low morale
Reluctance to report
Low time pilots
Copyright © HFA
Schedule pressure
Inadequate staffing
Lack of trust
Low morale
Reluctance to report
Hi workload - fatigue
Low experience service
personnel
HFA
Human Factors
Associates
International Civil Authority Study
Selected Issues Identified
•
•
•
•
•
•
•
•
•
•
Some employees are not comfortable reporting deficiencies
Not all workforce trust senior management
Not all senior managers are supportive of SSP – SMS activities
ICAO standards – guidelines are not completely clear
Legal framework may not support non-punitive reporting
Management and workers do not perceive need for change
There are no obvious incentives to reward change acceptance
Cultural and language diversity complicates communication
Employee training in SMS – and cultural impact is deficient
Additional resources (budget and staffing) may be needed
Copyright © HFA
APEC study by HFA 2010
HFA
Human Factors
Associates
ICAO Accident Statistics
YEAR
Accidents
Fatalities/Rate
2005
119
824 / 4.2
2006
112
806 / 3.9
2007
122
645 / 4.0
2008
138
534 / 4.6
2009
113
670 / 3.9
2010
121
707 / 4.0
ICAO 2011 State of Global Safety
HFA
ICAO Statistics
Human Factors
Associates
ICAO 2011 State of Global Safety
State
Accident Rate
Safety Indicators
(practice effectiveness}
Africa
16.8
41%
Asia
3.1
61%
Europe
3.3
72%
Lat. Amer/Car.
5.4
64%
North America
3.3
93%
Oceana
4.8
47%
Rates and %
Effectiveness are
Correlated.
Note: The accident rate is defined
as number of accidents per million
departures.
HFA
Human Factors
Associates
ICAO Global Areas of Concern
•
•
•
•
Fatigue risk management
Runway safety
Accident investigation (flight recordings)
Next generation safety professionals
What about – UAVs, cockpit automation, and next gen
ATC, global economy and resources?
HFA
Human Factors
Associates
Survey Application Process
1.
2.
3.
4.
5.
6.
7.
HFA representatives gather safety documentation and conduct interviews
with a cross section of company/agency employees.
HFA constructs tailored survey based on the use of previously validated
survey items and inputs from document review and personnel interviews.
A draft survey is presented to safety personnel for review and revision,
then finalized for survey application.
The final survey and demographic information are uploaded to the web or
a scanable paper version is created, .
The survey is administered in accordance with preplanned schedule of
events, including announcements/survey instructions, survey distribution,
collection and analysis of survey data, and results reporting.
Results are presented along with recommendations for safety
improvement interventions where warranted.
A time frame is established for a follow on second survey round which will
provide results to be compared to the original “benchmark” Survey.
Copyright © HFA
HFA
Human Factors
Associates
END
Summary and Conclusions
•
Survey process for use as a risk assessment tool – validation of survey
instrument is continuous – US Navy, major airline and NASA study.
•
Approach incorporates a complete measurement, analysis and diagnostic
display “dashboard” system for supervisors that provides diagnostic
feedback and normative “benchmark” comparisons for a given company,
industry or domain.
•
Survey process was successfully used by civilian organizations,
helicopter transport, aerospace, airline, electric power, and critical medical
care facilities.
•
Approach applies to other high-risk industries, that desire to asses their
•
HRO performance effectiveness and risk exposure.
Demonstration Web Site:
https://www.hfa-clients.com/demosite/login.html
Copyright © HFA