Leading Indicators
for ES&H Performance:
Measuring How Well Our
Management System Is Performing
ISM Best Practices Workshop
September 12, 2006
Steven R. Woodbury
DOE Office of Nuclear Safety
and Environmental Policy
Overview
 The mix of ES&H performance measures
changes as an ES&H program matures.
 DOE currently uses many lagging measures.
 Our ‘next level’ of performance measurement
should focus on how well key elements of our
ES&H management system are performing.
 There are instructive examples in the literature
of best practices on which we can build.
 But nobody says this is easy.
2
Three Levels of Performance
Measurement
Grace Wever identifies three types of performance measures
that organizations may employ:
1. Operational indicators (lagging)
For example:
• waste generation
• accident/injury rates
• regulatory violations
2
2. Identifying underlying management system deficiencies (leading)
For example, the quality of:
• training
• audits
• communications
• documentation
• review processes
3
Three Levels of Performance
Measurement (continued)
3. Identifying gaps in internal competencies
and external relationships (leading)
For example,
• Organization begins to measure:
– Impacts of products, operations, or services
on public health and the environment
– EHS research and development
– Capital investments
– Stakeholder and customer partnerships
• Organization strives to link financial, human resource,
operation, EHS, and other goals and measures
“The mix of an organization's performance indicators
tends to reflect the maturity of its EHS program.”
Grace Wever, Strategic Environmental Management, pp 158-9
4
Importance of Leading Indicators
The importance of developing good leading indicators
goes beyond
“It’s hard to steer by looking in the rearview mirror.”
The importance goes beyond continuous improvement
“We should do what leading organizations are doing.”
We need to be concerned because
even good performance, on good lagging indicators,
can’t assure us that risks are really being controlled.
5
Outcome Measures
are Not Sufficient
“When safety is good and injury and loss rates are low,
then [outcome] measurements are not sufficient to
provide adequate feedback for managing safety.
For operations where there may be potential for
severe accidents, the likelihood of such events must
be extremely low. This means that the absence of
very unlikely events is not, of itself, a sufficient
indicator of good safety management.”
--European Process Safety Center, 1996, p. 3.
6
Outcome Measures
are Not Sufficient
“Health and safety differs from many areas measured by
managers because success results in the absence of an
outcome (injuries or ill health) rather than a presence.
But a low injury or ill-health rate, even over a period of
years, is no guarantee that risks are being controlled
and will not lead to injuries or ill health in the future.
This is particularly true in organizations where there is
a low probability of accidents but where major
hazards are present.”
– A Guide to Measuring Health & Safety Performance
Health & Safety Executive (UK), p 5
7
Some Best Practices in “Next Level”
Performance Measurement
 Guide to Measuring Health & Safety Performance
(British Health and Safety Executive)
 Total Quality Environmental Management Matrix
(Grace Wever, Council of Great Lakes Industries)
 Positive Performance Indicators
(Australia, Department of Employment
and Workplace Relations)
 General Electric Company
 ProSmart software
(Chemical Process Safety Council)
8
Health & Safety Executive (UK)
 Guide discusses performance
measurement in the context
of a health and safety
management system
 Discusses the limitations of
traditional “lagging” measures
 Provides a systematic
approach to deriving
measures which link to the
H&S management system
9
Why Measure Performance?
“The primary purpose of measuring
H&S performance is
to provide information on
the progress and current status
of the strategies, processes
and activities
used by an organization
to control risks to health and safety”
– A Guide to Measuring Health & Safety Performance
Health & Safety Executive (UK), p 7
10
HSE Guide –
a Balanced Approach
Performance measurement should be based on a
balanced approach, which addresses:
 Input
Monitoring the scale, nature and distribution of hazards created
by the organization’s activities
 Process
Active monitoring of the adequacy, development, implementation
and deployment of the H&S management system
 Outcomes
Reactive monitoring of adverse outcomes resulting in injuries, ill
health, loss and accidents with the potential to cause injuries ill
health or loss (‘measures of failures’)
11
HSE Guide –
Measure the Management System
 The H&S management system is the process which
turns uncontrolled hazards to controlled risks
 The key elements of
•
•
•
•
•
Policy
Organizing
Planning and implementation
Measuring performance
Audit and review
all need to be in place to control risks effectively
 The performance measurement system must cover
each element of the H&S management system
12
HSE Guide –
What and How to Measure
 Measuring elements of the H&S management system
should cover three aspects:
• Capability
• Implementation
• Deployment
 The measurement process can gather information
through:
• Direct observation of conditions and people’s behavior
• Talking to people to elicit facts and their experiences as well
as gauging their views and opinions
• Examining written reports, documents and records
13
HSE Guide – Observations
 Performance measurement is approached in the
context of the health and safety management system
 Guide acknowledges the need to measure outcomes
– but this is not sufficient:
• ‘measures of failure’
• doesn’t guarantee that risks are really controlled
 Approach is not limited to Health and Safety –
completely applicable to management across ES&H
 Focus is on the performance of key management
system elements
 Measures are largely qualitative
 Provides a clear framework for thinking about ES&H
management and performance measurement
14
 TQEM identifies 7 performance
categories, based on Baldrige Criteria
most relevant to EHS management, as
well as the ISO 14001 standard
 Identifies performance levels (1–10)
(based on descriptive text)
 Detailed self-assessment questionnaire
is provided to support evaluation.
 Within a category, one cannot score at a
‘higher’ level unless performance is fully
satisfactory at all ‘lower’ levels
 Developed in the 1990’s by Grace
Wever, and the Council of Great Lakes
Industries
15
Ten Performance Levels
Total Quality Environmental
Management Matrix
Seven Categories
Total Quality Environmental
Management Matrix – Categories
Seven categories (different weights)
 Leadership (15%)
 Information and Analysis (7.5%)
 Strategic Planning (7.5%)
 Human Resource
Development (10%)
 Process Management (15%)
 Environment, Health & Safety
Results (30%)
 Customer/Stakeholder
Satisfaction (15%)
16
TQEM Performance Levels –
Example: ‘ES&H Results’ Category
10. Benchmarking shows unit is ‘best-in-class’
7. EHS improvements contribute to financial
and business improvements
5. Positive improvement trends
in units’ key EHS measures
4. EHS measures reviewed and improved
at least annually
3. EHS results compared with objectives and
targets; used to improve effectiveness of
management systems and performance
1. EHS performance measures identified;
baseline data and trends collected
17
EH&S Results
Environmental Health and Safety Results
Example: ‘Strategic Planning’ Category
9. Information on competitors’ EHS strategies used
to improve units’ strategies and performance
8. EHS integrated into long and short-term
business plans for all of unit’s products,
processes and services.
6. EHS plans and deployment consistently aligned
at all levels of the unit
4. Long and short-term plans that include EHS
objectives are reviewed and improved at least
annually. Key measures include EHS measures
1. Long-term and short-term planning process used
that addresses EHS needs; annual operating
plan includes EHS management needs
18
Strategic Planning
TQEM Performance Levels –
Strategic Planning
Example: ‘HR Development’ Category
9. EHS needs fully integrated into unit’s HR
development plan. Training/education for
EHS staff include key business knowledge
7. Employees proactively initiate activities to
improve EHS performance
5. Measures of EHS training effectiveness in
place; employees with potential to impact EHS
are competent to perform EHS responsibilities
3. All employees have received appropriate EHS
training; employees aware of EHS compliance
requirements.
1. EHS training needs identified, resources
committed.
19
Human Resource
Development
TQEM Performance Levels –
Human Resource Development
TQEM Matrix – Observations
 Not limited to environment – completely applicable
to management across ES&H
 System is based on explicit, but qualitative, criteria
for ES&H management
 Focus is on qualitative measures of performance in
key management categories
 Published matrix is fully defined (not open-ended)
 Matrix was developed more than 10 years ago –
it should be updated on the basis of additional
experience
 Its importance is as a conceptual framework;
the specific content can be adapted
20
Positive Performance
Indicators
‘Positive Performance Indicators’
(PPIs) focus on assessing how
successfully an organization is
performing by monitoring the
processes that provide good
OHS outcomes.
PPIs are to be developed in addition
to traditional ‘outcome’ indicators.
PPI program was developed by the
Australian Government’s
Department of Employment
and Workplace Relations
21
Positive Performance Indicators –
Based on Quality Management
PPI is based on a Quality Management approach. PPIs may be
 Inputs (Key Activities)
•
•
•

Processes (Monitoring Key Risks)
•
•
•

Identify key risks
Identify key contributors to outcomes of concern
Develop measures to monitor behaviors and practices
Outputs (Milestones)
•
•
22
Measures of actions or initiatives undertaken to improve OHS
Provide indicators of commitment and effort, but not of
effectiveness
Can provide useful information on participation, leadership and
communication
Achievement of objectives
Progress toward achievement of higher-level OHS goals and
targets
Positive Performance Indicators –
Part of a Management System
 PPIs are only useful if an organization has a
systematic approach to management of OHS
 Many variations exist, but all have the following
principles:
• Commitment and policy
• Planning
• Implementation
• Measurement and evaluation
• Review and improvement
23
Positive Performance Indicators –
Categories of PPIs
 Commitment and policy
Measures demonstrated commitment to improve OHS performance
 Planning
Measures what procedures are in place to eliminate workplace injury
and disease
 Implementation
Measures the capability and support mechanisms that are necessary
to achieve OHS objectives and targets
 Measurement and Evaluation
Measures the extent to which workplace health and safety is monitored
and evaluated so that issues can be identified and corrective action
taken
 Review and improvement
Measures the effectiveness of the OHS management system, and its
continuing suitability
24
Examples of PPIs –
Commitment and Policy
25
Performance Indicators
How to Measure
Evidence of OHS policy statement
signed by CEO
Frequency and quality of OHS reporting
by or to Senior Management
Senior managers’ performance appraisals
include OHS
Percentage of workforce and contractors
covered by consultation processes
and OHS representation
Rating of effectiveness of employee
participation on OHS management
Examination of
records
Employee
questionnaire/survey
Examples of PPIs –
Implementation
Performance Indicators
How to Measure
Percentage of workplace inspections conducted
over a specified timeframe
Percentage of high risks identified
over a specified timeframe
Proportion of items identified through safety walks
and inspections that are repeat items
Proportion of reported incidents that do not
result in injury compared with those that do
Percentage of planned management visits conducted
over a specified timeframe
Percentage of managers and employees that have
received OHS training (e.g. induction, job-specific,
hazard management, emergency procedures….)
Observation –
walk through
inspections/ audits
Examination of
hazard reports
Examination of
hazard logs
Review of
maintenance log
Analysis of
accident and
incident reports
26
Positive Performance
Indicators – Observations
 PPIs are not limited to Occupational Health and Safety –
completely applicable to management across ES&H
 PPIs focus on identifying key elements of the ES&H
management system, and measuring how well these key
elements are functioning
 PPIs are non-prescriptive and open-ended: approach
provides great flexibility at the facility and site level
 Most PPIs are qualitative, based on assessments,
record reviews, surveys, or audits
 The Guidance document assumes a fairly basic
management system. DOE sites generally have more
sophisticated ES&H management systems in place, with
the opportunity to develop more sophisticated measures.
27
General Electric –
Health and Safety Framework
 At the April ISM meeting in Albuquerque, Kurt Krueger
described GE’s corporate-wide Health and Safety program
 “One standard, one program, one set of metrics
for every GE facility around the world – No exceptions!”
 GE’s Health and Safety Framework includes a mix of 21
• Management system elements
For example:
– Training
– Accident investigation and follow-up
• Subject matter areas
For example:
– Industrial hygiene
– Chemical management
– Motor vehicle safety
28
General Electric
 Detailed
questions
 GE-specific
guidance &
expectations
29
(continued)
General Electric
(continued)
 Scorecard ratings are based on
• Plant self-assessment (540 questions, twice a year)
• Periodic corporate audits
 Tied to monthly site operational metrics and supervisor scorecards,
• “tailored to site operations”
• “designed to drive supervisor behaviors that will find
and fix H&S issues before an accident finds them”
 GE has a parallel system for assessing
environmental performance
 GE also continues to track “trailing metrics”
For example:
• Recordable injury and illness rates
• Lost time injury and illness rates
• Permit exceedances
• Notices of noncompliance
30
General Electric –
Observations
 Integrated into GE’s broader management system
 Reflects a balance between
frequent site self-assessment and reporting,
and periodic headquarters audits
 GE chose to implement a uniform corporate system.
While this might not be DOE’s choice,
there is still a lot to learn from the overall framework,
and from the measures GE’s chose to include
31
ProSmart –
Center for Chemical Process Safety
 In 1993, CCPS began a project to measure the
effectiveness of process safety management systems
 Their objective was
“to provide management with the tools for
assessing the health of process safety
management systems on a real-time basis”
 Based around 12 management system elements
 Extensive set of computer-based questions,
using qualitative and quantitative measures
 Rolls up responses into a single score or ‘index’
32
CCPS ProSmart – Based on 12
Management System Elements
 CCPS identified 12 key management
system elements, such as
•
•
•
•
•
•
•
Accountability
Process knowledge and documentation
Capital project hazard review
Training and performance
Management of change
Incident investigation
Audits and corrective action
 ProSmart generates a score,
which is continually updated
33
CCPS ProSmart – Based on 12
Management System Elements
34
CCPS ProSmart – Drills Down to
Detailed Evaluation Questions
35
ProSmart –
Observations
 Derived from a quality management approach
 Focus is on specific management system elements
 Established for chemical process safety,
but in principle could encompass environment,
safety and health
 Like TQEM and GE, ProSmart provides a set of
detailed assessment questions, for qualitative
assessment against established criteria
36
Key Aspects of “Next Level”
Performance Measures
 “Next Level” performance measures identify
and measure how well key elements of the
ES&H management system are functioning
 Measures tend to involve
• Precursor analysis and trending
• Observations, assessments and audits
 Measures tend to be qualitative
rather than just counting events
(This doesn’t mean they can’t be
rigorous and replicable)
37
DOE Has Many Pieces in Place
 Analysis and trending
• Data collection
• Trending and analysis
• Root cause analysis
 Observations, assessments, audits
• Contractor self-assessment processes
• DOE oversight activities
• Annual ISM assessments should systematically
address management system elements
• Inspections by the Office of ES&H Inspections (HS-64,
formerly SP-44) address ES&H management system
functioning at a high level (ISM core functions)
• Individual IG audits address some management
system elements
38
Audits and Inspections
39
ES&H Inspections
(colors added)
Define Scope of Work
Analyze the Hazards
Identify and Implement Controls
Perform Work Within Controls
Feedback and Continuous
Improvement
(contractor)
(DOE)
Essential System Functionality
40
ES&H Evaluations
of 17 DOE Sites 2002-2004
Management System
Area
Analyze the Hazards
Develop and
Implement
Controls
Work within
Controls
Feedback and
Improvement
41
Effective Performance
Needs Improvement
10
7
5
11
12
5
2
15
Significant
Weakness
1
Summary
 There is a common thread of best practice, across many different
organizations: to move beyond outcome (lagging) indicators,
we need to start systematically measuring how well
key elements of our management system are performing.
 This is not as easy as counting occurrences (injuries, enforcement
actions): it must involve walk-throughs, audits, surveys, and other
varied approaches.
 This ‘next level’ of leading performance measurement
complements, and does not replace, outcome (lagging) measures.
 Approaches developed for process safety, worker safety,
environment, can be applied across ES&H.
We need to be eclectic and ecumenical in our thinking.
 Nobody has a turnkey system that’s right for DOE.
We need to identify and build on the best of these various approaches.
 Nobody says it will be easy.
42
Next Steps
 Identify key ES&H management system elements
• We currently categorized some information (e.g., ORPS
reports; Independent Oversight Assessments) around
ISM’s five broad core functions
• Within these core functions, we can identify more specific
management system elements, including:
43
Identification of requirements - Corrective action management
Hazard identification
- Lessons Learned sharing
Communication
- Performance Measurement
Training
- Worker involvement
Documentation
- Audits and assessments
Periodic top management review
Next Steps (continued)
 Identify the various things we are already doing to
assess how well these elements are functioning
 Identify and fill gaps as appropriate
 Organize and present the information in a way which
provides current feedback to management on how
effectively key management system elements are
functioning at each site and each facility
 In Sum:
Use the health of these key management system
elements as the conceptual and organizing
framework to strengthen our ability to know
how well we are managing risks
44
In Closing …
“Far better an approximate answer
to the right question
which may be difficult to frame,
than an exact answer
to the wrong question
which is always easy to ask.”
– statistician John W. Tukey
45
References
Grace Wever, 1996, Strategic Environmental Management: Using TQEM
and ISO 14000 for Competitive Advantage, John Wiley and Sons, NY
European Process Safety Centre (Jacques van Steen, ed.), 1996, Safety
Performance Measurement, Institution of Chemical Engineers, Rugby (UK)
(distr by Gulf publishing Co., PO Box 2608, Houston, TX 77252-2608)
Health and Safety Executive (United Kingdom), 2001, A Guide to Measuring
Health & Safety Performance http://www.hse.gov.uk/opsunit/perfmeas.pdf
Commonwealth of Australia (Department of Employment and Workplace
Relations), 2005, Guidance on the Use of Positive Performance
Indicators to Improve Workplace Health and Safety
http://www.nohsc.gov.au/PDF/Standards/GuidanceNotes/ASCCPPIGuidanceBooklet.pdf
Kurt Krueger, “Creating a Culture of Safety Excellence: The Journey and the
Prize,” talk at DOE ISM Champions Workshop, Albuquerque NM, April 2006
http://www.eh.doe.gov/ism/workshops/Creating_a_Culture_of_Safety_Excellence.pdf
George Eckes, The Six Sigma Revolution: How GE and Others Turned
Process into Profits, New York: John Wiley and Sons, 2000.
Center for Chemical Process Safety, ProSmart software
http://www.aiche.org/CCPS/Publications/Software/ProSmart/index.aspx
46
Contact Information:
Steven R. Woodbury
DOE Office of Nuclear Safety
and Environmental Policy
202-586-4371
steven.woodbury@eh.doe.gov
47