Practical steps to proactively
measure safety performance
Canada Chapter
System Safety Society
April 19, 2012
Presented by
Terry Kelly & Tom Moir
SMS Aviation Safety Inc.
Outline
 Introduction and assumptions
 Steps in developing a
measurement protocol
 A case study
 Concluding thoughts
Assumptions
 We are dealing with complexly interrelated systems

People, equipment, organization (s)
 Continuous & dynamic state(s) of change
 Goal is to explicitly manage risks to
ALARP (performance-based)
 ALARP is achieved through a blend of
prescribed and performance-based
requirements/standards
Achieving ALARP
Five forms of functionality:
1.
2.
3.
4.
5.
Proactive & comprehensive
Reliability
Explicit safety-risk management
Safety culture
Safety measurement
Requirements
(1)
 Measures (system) performance, not
just processes
 Measures positive performance, as well
deficiencies in performance
 Is common across programs and
functions in a company/organization
 Focuses mainly on proactive indicators of
performance, as well as reactive
measures
 Focuses on hazards as well as risks
Requirements
(2)
 Accounts for human and organizational
factors as well as technical and operational
factors
 Evaluates the results of day-to-day
decision-making, as well as strategic
decisions
 Discriminates in terms of safety significance
 Information is easily/efficiently collected
and applied
 The results can be replicated, and have
validity over time
Building a Measurement Tool
FOUR EASY STEPS!
1. Define performance requirements
2. Identify individual & organizational
behaviours/outcomes
3. Choose a measurement scale
4. Determine collection methods
1. Define Performance
Requirements
 Research:




Industry standards & regulations
Industry association literature (e.g.,
good practices)
Texts on risk management in complex
systems (e.g., Reason)
Management and leadership texts
 Use knowledge and experience
Safety Culture
Everyone in the company understands their role
and is committed to sharing information so that
the risks related to organizational, human and
operational factors are actively managed.
Proactive safety management is embraced
throughout the organization. There is a free flow
of safety-related information vertically and
laterally, within and outside the organization. The
positive safety culture provides organisational
resiliency, which enables the organisation’s SMS
to remain appropriate and effective during times
of change.
2. Identify Behaviours
and Outcomes
 What are you going to look for?

How people behave


How people think


e.g., pre-job briefings, raise hazard
reports, follow procedures
e.g., company priorities, risk
How the companies manage
operations

e.g., metrics, documents up to date and
accurate
Function 4 - Safety Culture
A. Safety information is valued and used at all levels of the
organisation. It is obtained from consistent, unfettered and
active (3-way) communication throughout the organisation.
B. Safety information is actively sought and communicated
with stakeholders, including the regulator.
C. Staff have confidence in non-punitive reporting.
D. Managers trust and employ validated, safety-related
information from all sources
E. Everyone from all levels “walks the talk” and are actively
engaged in proactive safety management.
Caution
 You don’t necessarily want to spell
out too precisely the actions or
outcomes you will see.
 Why not?
3. Choose a
Measurement Scale
 Rating scale


Allows you to assess change in
performance over time
Allows evaluator to translate and
solidify an otherwise ambiguous
perception
 Rating scale needs to be
accompanied by guidance
Evaluation Table
Function 1 - Proactive & Comprehensive Safety-risk Management.
All components of the organisation are actively engaged in or support the proactive
management of safety-risks (e.g. operational, technical, financial, HR components, etc.).
Safety-risk management is top-down, and integrated with strategic, business and HR
planning….
Function
Proactive & comprehensive
safety management
a. Performance-based safety policy
(e.g. strategic safety goal) and
principles (e.g. sharing and using
safety-related information) guide
the organisation’s direction, and its
staff’s behaviours
b. Safety goals & objectives are
explicitly set (safety planning) and
measured, and progress reported
(accountability), to continuously
improve safety management and
performance.
Not
Present
Exists in
Part
1
2
3
4
5
6
7
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
☐
Exists
Completely
Observations / Evidence (cite
sources)
Caution
 Watch for prescriptive tools
masquerading as performancebased tools!
What’s Wrong with This
Picture?
4. Determine Collection
Method
 Choose from a combination of:



Document & Data review
Interviews (management & staff)
Observation (of work & meetings)
 Choose sample sizes to ensure
validity
 Multiple lines of evidence
Report
 Performance ratings supported by
evidence
 High-level strengths
 System Safety Deficiencies
(prioritised in terms of risk)
Case Study
 Offshore helicopter operator
 SMS (not regulated) in place since
2009
 3-day SMS Evaluation
Function 4 C
Score
c. Staff have confidence in nonpunitive reporting
1
2
3
4
5
6
7
Supporting evidence:
•There were indications that staff are comfortable raising items in the event
reporting program when they do not feel they are getting results through other
avenues.
•Several interviewees expressed that they did not feel their careers would be
jeopardized in any way by their raising safety related concerns.
•Several persons interviewed expressed the opinion that AMEs were less
trusting of the non-punitive reporting program than other people, and that they
were also less likely to raise event reports. This is in contrast to operations,
where reporting was felt to be strong.
•Management expressed the opinion that people’s attitudes were shifting (for
the better) since the introduction of the SMS.
Function 5 C
c. System safety deficiencies are assessed in
terms of safety significance to guide priorities
to improve safety management/performance
Score
1
2
3
4
5
6
7
Supporting evidence:
•Company practices do not rigorously identify and
address organizational factors.
•This weakness has been recognized by company
management.
•See also 2.3.2.b. and 2.3.2.f.
System Safety Deficiencies
Robust Analysis(B – 2)
Presently, all events reported through the electronic
system receive some form of analysis. This normally
consists of company management discussing the
event, either in person or over the Internet, and
developing action plans. More significant events
receive more in-depth analysis based on Tap-root
terminology or the S-H-E-L(L) model. However, these
analyses do not thoroughly assess HF or OF. This
consistently results in safety actions that target the
behaviours of frontline workers, often by calling for
increased vigilance.
Key Points
 Measures (system) performance,
focussing explicitly on an
organization’s capability to manage
risk
 Focuses mainly on proactive
measures, using multiple lines of
evidence drawn from different types
of data
Conclusion
Our objective has been to
demonstrate how a simple tool can
be developed and employed with
relative ease to proactively measure
the safety performance of even a
large and complex organization