Developing Effective Performance Measurements
Introduction- The Theme
Begin to Think Differently About
Performance Measurement Using
• Why You Need to Measure
• How You Can Measure with a Purpose
• What Risks You Need to Consider
2
2
Introduction- Agenda
The Red Bead Experiment:
• Is an interactive exercise to expose everyone to
the advantages of a systems approach to
measuring
• It will demonstrate that traditional management
practices will not affect or produce the desired
outcome
• Emphasize the importance of top management
decisions to ensuring safety as an outcome
• Show how statistical analysis can identify
inherent variation
3
3
Introduction- Agenda
• How to link statistical analysis and
performance measurement
– Overview of sequential steps
– Identifying your organizational risks
– Recognize Leading and Lagging performance
indicators
– Making the links to your management System,
goals and objectives
4
4
Introduction- Agenda
Some Real-Life Examples from our
aviation community
• Focussing on examples of analysis (accident rates or
hazard frequencies)
– Counting and trending-Power of control charting
– Using Statistical control to recognize opportunities
for continuous improvement
5
5
Introduction- Agenda
Interactive Exercise using a Process Example
• Aligning your Performance Measures to
Your Risks
–
–
–
–
–
–
How it fits into the mgmt review
When do you measure
What are your risks
How will you change what you do
How do you monitor
How do you feedback
6
6
Introduction- Agenda
Summary and Closure
 Discuss the Main Take-Aways from the
session
7
7
Deming’s Red Bead Experiment:
With a Twist
Who is Edward Deming?
• American Statistician – best known in Japan
• Taught Management in Post-War Japan to
Improve Design and Quality using Stats
• The Red Bead Experiment
– A Teaching Tool
– Historically used to demonstrate quality
Management System principals
– Can this be applied to Safety?
9
9
What we Need:
• 6 Volunteers!
• 4 “Willing” Workers
• 2 Safety Inspectors
10
10
TO PRODUCE WHITE BEADS, SAFELY!
OUR MISSION
11
11
THE EXPERIMENT:
• The “Production” of each White Bead
follows a Process
• Processes inherently have “risks” (or Red
Beads) that can impact on Safety
• The White Beads can be a metaphor for
anything – they are an output (Flight to X,
Repair of Y, Painting of Z, Storage of W,
etc)
12
12
THE EXPERIMENT:
• OBJECTIVE:
Produce Something (White Bead),
following a pre-determined Process, safely
(ie. Without Red Beads)
13
13
TRAINING
1. Each “willing worker,” dips the paddle
into the box of beads for his/her first
day of production
2. Safety inspectors identify and count
the number of risks in “Producing”
3. Chief Safety Inspector of the company
announces the result, records and
dismisses
14
14
PRODUCTION STAGE 1
15
15
PRODUCTION STAGE 2
• Production Standard: No more than 3 Red
Beads today per worker!
16
16
AWARDS CEREMONY
17
17
PRODUCTION STAGE 3
• Motivation!
18
18
PRODUCTION STAGE 4
19
19
THE RESULTS
• Chart
20
20
WHAT DOES THIS TELL US
• The system is at fault, not the “Workers”
• Management must change the system
• Performance Measures need to look at
the system and process, not just at the
people
21
21
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
It's the system, not the workers. If you
want to improve performance, you
must work on the system.
22
22
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
Safety is made at the top. Safety is an
outcome of the system. Top
management owns the system.
23
23
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
Numerical goals and production
standards can become meaningless.
The number of red beads produced is
determined by the process, not by the
standard.
24
24
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
Rewarding or punishing the Willing
Workers had no effect on the outcome.
Extrinsic motivation is not effective.
25
25
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
Rigid and precise procedures are not
sufficient to produce the desired
Safety.
26
26
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
People are not always the dominant
source of variability .
27
27
WHAT MESSAGES ARE IN
THIS EXPERIMENT?
Slogans, Exhortations and Posters Are
Not Always Useful To The Willing
Worker.
28
28
IN SUMMARY
•
Be Cautious about misusing
performance data > Needs to focus on
the process
29
29
NOW WHAT?
WHEN YOU NEED A FUNDAMENTAL
CHANGE TO YOUR SYSTEM:
HOW CAN YOU USE
STATISTICAL ANALYSIS TO SHOW IF
YOU ARE ACHIEVING WHAT YOU
WANT?
30
30
BREAK (15 Minutes)
31
31
Linking Performance Measures
&
Statistical Analysis
32
32
1.0
2.0
Appendix A
3.0
4.0
5.0
6.0
Safety Management Plan
1.1 Safety Policy
1.2 Non-punitive Policy
1.3 Roles, Responsibilities & Employees
1.4 Communication
1.5 Safety Planning – Objectives & Goals
1.6 Performance Measurement
1.7 Management Review
Document Management
2.1 Identification & Maintenance of Regulations
2.2 SMS Documentation
2.3 Records Management
Safety Oversight
3.1 Reactive Processes
3.2 Proactive Processes
3.3 Investigation and Analysis
3.4 Risk Management
Training
4.1 Training, Awareness & Competence
Quality Assurance
5.1 Operational
Quality Assurance
33
Emergency Preparedness
33
6.1 Emergency Preparedness and Response
SMS Operational Measurement Elements
1.0
3.0
5.0
Safety Management Plan
1.1 Safety Policy
1.5 Safety Planning – Objectives & Goals
1.6 Performance Measurement
1.7 Management Review
Safety Oversight
3.1 Reactive Processes
3.2 Proactive Processes
3.3 Investigation and Analysis
3.4 Risk Management
Quality Assurance
5.1 Operational Quality Assurance
34
34
Essential SMS Measurement Elements
1.0
3.0
Safety Management Plan
1.1 Safety Policy
1.5 Safety Planning – Objectives & Goals
1.6 Performance Measurement
1.7 Management Review
Safety Oversight
3.1 Reactive Processes
3.2 Proactive Processes
3.3 Investigation and Analysis
3.4 Risk Management
35
35
Where to look to find measurable
•Safety policy objectives
•Process purpose statement
•Management decision duties
•Hazard registry – risks
•Control graphs
•Aviation publications
•Safety case
•Industry associations
•Intuition
•Feedback
36
36
SP Objectives Examples
1. Ensure that aviation safety is our number one priority.
2. Document, implement and maintain and effective SMS
3. Meet or exceed all applicable regulatory requirements.
4. Establish and sustain a continual improvement philosophy.
5. Encourage employee participation and support their efforts.
6. Reduce and eradicate the real and potential causes of incidents or accidents.
7. Identify and control all known and potential hazards to aviation safety.
8. Implement a non-punitive policy for the security of all employees.
37
37
Purpose
Measurable
1) To define the method and process
to be used for the scheduling and
implementing of aircraft maintenance
services to ensure consistent quality
and safety of the service.
2) To establish a method that allows for
continual improvement, including
preventive actions, throughout all
aspects of the organization that
have an affect on quality, safety,
service and price
•On-time scheduling
•Qty of recalls
•Qty of emergency repairs
•Duration of repairs
•Spare parts inventory value
•Reported hazards
•Preventive actions
•Qty of reported hazards
•CADORS reduction
•Incidents / accident rate
•Positive impacts on processes
•Cost reductions
•Improved margins
•Efficiency
38
38
Purpose
Measurable
•Qty of reports
•Cost of damages
•Down time for A/C
•Types of damage – how
•Types of maintenance
•Competency
•Procedure impacts
To review/analyze reports and provide
feedback to maintenance managers so
that they can make change to
procedures/training to reduce damage
to aircraft by employees conducting
maintenance.
This above example is an actual purpose statement currently
in use today
39
39
Example of an actual measurement practiced
in industry to reduce or eliminate hanger rash
Purpose Statement:
To review/analyze reports and provide feedback
to maintenance managers so that they can make
change to procedures/training to reduce damage to
aircraft by employees conducting maintenance.
40
40
Portion of the Excel spreadsheet and the capturing of data
Damage To Aircraft In The Completion Of Tasks
Date
Report #
23-Jan-06
1483
1-Feb-06
1516
21-Feb-06
1606
16-Feb-06
1584
14-Sep-06
2047
12-Mar-07
2624
4-Feb-07
2705
15-Nov-07
3393
20-Nov-07
3401
21-Nov-07
3409
26-Nov-07
3431
3-Dec-07
3463
11-Jan-08
3597
25-Jan-08
3643
5-Feb-08
3687
8-Feb-08
3699
26-Feb-08
3773
29-Feb-08
3771
5-Mar-08
3793
25-Mar-08
3889
20-Mar-08
3899
8-Jul-08
4176
Year
Month
6
6
6
6
6
7
7
7
7
7
7
7
8
8
8
8
8
8
8
8
8
8
41
1
2
2
2
9
3
4
11
11
11
11
12
1
1
2
2
2
2
3
3
3
7
Q1 2006
Q2 2006
Q3 2006
Q4 2006
Q1 2007
Q2 2007
Q3 2007
Q4 2007
Q1 2008
Q2 2008
Q3 2008
Q4 2008
Q1 2009
Q2 2009
Q3 2009
Q4 2009
Q1 2010
Q2 2010
Q3 2010
2
0
1
0
1
1
0
5
9
3
5
6
11
2
4
7
4
2
1
41
Q3 2010
Q2 2010
Q1 2010
Q4 2009
Q3 2009
Q2 2009
Q1 2009
Q4 2008
Q3 2008
Q2 2008
Q1 2008
Q4 2007
Q3 2007
Q2 2007
Q1 2007
Q4 2006
Q3 2006
Q2 2006
Q1 2006
NUMBER OF REPORTS
Captured data displayed in a bar graph
DAMAGE TO AIRCRAFT
15
14
13
12
11
10
9
8
7
6
5
4
3
2
1
0
QUARTER
DAMAGE REPORTS
42
42
DAMAGE TO AIRCRAFT
Action Taken
15
14
NUMBER OF REPORTS
13
1st Qtr -09
12
11
10
9
8
7
6
5
4
3
2
1
Q3 2010
Q2 2010
Q1 2010
Q4 2009
Q3 2009
Q2 2009
Q1 2009
Q4 2008
Q3 2008
Q2 2008
Q1 2008
Q4 2007
Q3 2007
Q2 2007
Q1 2007
Q4 2006
Q3 2006
Q2 2006
Q1 2006
0
QUARTER
DAMAGE REPORTS
Observations
•
corrective actions to employee filed reports may not be effective.
•
potential trend existed
Corrective Actions
•
mined reports from the previous years and plotted them in the graph
•
specific training was provided
•
heightened awareness
43
43
DAMAGE TO AIRCRAFT
Action Taken
15
14
NUMBER OF REPORTS
13
12
11
4th Qtr -09
10
9
8
7
6
5
4
3
2
1
Q3 2010
Q2 2010
Q1 2010
Q4 2009
Q3 2009
Q2 2009
Q1 2009
Q4 2008
Q3 2008
Q2 2008
Q1 2008
Q4 2007
Q3 2007
Q2 2007
Q1 2007
Q4 2006
Q3 2006
Q2 2006
Q1 2006
0
QUARTER
DAMAGE REPORTS
Observations
•
Were reports involving damage while transitioning around aircraft on an
elevating device?
•
Is the use of spotters effective
Actions
4) Training department will review the “Working at Height” presentation
44
44
DAMAGE TO AIRCRAFT
Action Taken
15
14
NUMBER OF REPORTS
13
12
11
10
9
8
7
6
5
4
3
2
1
Q3 2010
Q2 2010
Q1 2010
Q4 2009
Q3 2009
Q2 2009
Q1 2009
Q4 2008
Q3 2008
Q2 2008
Q1 2008
Q4 2007
Q3 2007
Q2 2007
Q1 2007
Q4 2006
Q3 2006
Q2 2006
Q1 2006
0
QUARTER
DAMAGE REPORTS
Actions
5) The working at height training presentation has been amended to better
focus on clearance and the use of spotters
6) Center console covers have been completed for each type of aircraft
7) The training department will modify the human factors training program
to more effectively provide a clear and concise message to employees.
Comment; Seven (7) CAPs / changes were discussed, reviewed and
implemented to control and reduce damage to aircraft
45
45
Trend line beginning in year 2006 – 1st Qtr
DAMAGE TO AIRCRAFT
15
13
12
Reports Not Usually
Submitted
11
10
9
8
7
6
5
4
3
2
QUARTER
Q3 2010
Q2 2010
Q1 2010
Q4 2009
Q3 2009
Q2 2009
Q1 2009
Q4 2008
Q3 2008
Q2 2008
Q1 2008
Q4 2007
Q3 2007
Q2 2007
Q1 2007
Q4 2006
Q3 2006
Q2 2006
1
0
Q1 2006
NUMBER OF REPORTS
14
DAMAGE REPORTS
Linear (DAMAGE REPORTS)
46
46
Trend line beginning in year 2008 – 1st Qtr
12
10
8
6
4
2
0
Q1
2008
Q2
2008
Q3
2008
Q4
2008
Q1
2009
Q2
Q3
2009 2009
Q4
2009
Q1
2010
Q2
2010
Q3
2010
47
47
Exercise
48
48
List several safety critical processes
Process Name
(Safety Critical Process)
Securing Cargo
Loading and
securing of cargo
Description of
Measurable /
KPI
Unit of
Measure
(pcs., qty., lbs,
%, etc.)
Target
SP Objective
(Increase
/decrease /
number)
(Link to or draft a SP
objective)
Number of 10% reduction Identify and control
all known and
events
Per quarter
potential hazards
to aviation safety.
Log the qty. of
of incidents
49
49
1) Identify a process(s)
Safety critical processes
2) Identify a measureable for each
identify KPI
identify continual improvement item and unit of measure
3) Note the unit of measure
How and what will you measure to demonstrate improvement
4) Establish a Target
Can be a percent increase / decrease,
Size
Nbr of pcs
Qty
Lbs
5) Link processes to objectives
Note an objective from earlier slide
50
50
SP Objectives Examples
1. Ensure that aviation safety is our number one priority.
2. Document, implement and maintain and effective SMS
3. Meet or exceed all applicable regulatory requirements.
4. Establish and sustain a continual improvement philosophy.
5. Encourage employee participation and support their efforts.
6. Reduce and eradicate the real and potential causes of incidents or accidents.
7. Identify and control all known and potential hazards to aviation safety.
8. Implement a non-punitive policy for the security of all employees.
51
51
When organizations are measuring their SMS,
they need to:
1) Identify what they want to control (KPI)
2) Choose and establish the unit of measure
3) Identify a goal or target
4) Interpret / study the difference
5) Take action on the difference
52
52
Transport Canada
Captured data on aircraft accidents in Ontario Region
53
53
Historical Trend of Accidents in Ontario
180
168
160
153
149
140
128
122
119
Number of Accidents
120
113
102
100
86
89
YTD
81
80
74
64
80
74
68
63
58
60
40
20
0
93
94
95
96
97
98
99
00
01
02
03
04
05
06
07
08
09
10
Year
Counting
54
54
Trending of Ontario Accidents
1999 - 2010 (YTD)
180
168
160
149
140
122
Number of Accidents
120
119
113
102
100
81
80
74
80
74
63
58
60
40
20
0
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
YTD 2010
Year
Trending
55
55
Historical Trend of Accidents in Ontario
180
168
160
153
149
140
128
122
Number of Accidents
120
119
113
102
100
86
89
YTD
81
80
74
64
80
74
68
63
58
60
40
20
0
93
94
95
96
97
98
99
00
01
02
03
04
05
06
07
08
09
10
Year
Instead of trending…….
Lets put this56data through a Control Chart
56
Control Chart
Ontario Accidents – 1993 -2010
Upper Control Limit
Mean
Lower Control Limit
57
57
Interpretation
-Seven (7) points outside the control limits
-Eight (8) points in a row - same side of mean
Comment
Process unstable
Process out of control
Process may not be sustainable
Concerns
Would you be concerned??
58
58
Historical Trend of Accidents in Ontario
180
168
160
153
149
140
128
122
119
113
102
100
89
86
YTD
81
80
74
80
74
68
64
63
58
60
40
20
94
95
96
97
98
99
00
01
02
03
04
05
06
2023
93
2022
0
07
08
09
10
Ye a r
Hypothetical for next 17 years
180
160
140
120
100
80
60
40
20
2027
2026
2025
2024
2021
2020
2019
2018
2017
2016
2015
2014
2013
2012
0
2011
Number of Accidents
120
59
59
np Chart
90
80
70
60
50
60
40
30
20
10
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Is this a stable process and is it in control?
60
Is this process sustainable?
60
100
61
61
Hypothetical Example
Recording of Data
Developing Graphs
62
62
Incidents Report
Operations Incidents
A
B
C
D
E
F
G
Description of Incident
Local air traffic
Foreign object damage
Bird strikes
Runway incursion
Aborted takeoff
Open
Open
Target
Actual Month Total
Jan'07
3
1
1
3
1
Feb
Mar
2
10
9
Apr
May
Jun
3
1
2
2
1
1
3
3
0
1
1
3
1
2
9
8
7
8
7
9
Jul
Aug
Sept
4
1
0
1
1
3
1
1
2
2
2
0
1
1
2
2
0
6
6
5
7
5
6
4
6
Oct
4
0
Nov
3
0
Dec
3
0
Year to Date
12
7
20
15
5
0
0
3
0
Maintenance Incidents
A
B
C
D
E
F
G
Description of Incident
Malfunction
Hazardous cargo
Communication
Component failure
Shift change
Open
Open
Target
Actual Month Total
Jan'07
2
1
1
2
7
6
Feb
Mar
Apr
May
Jun
Jul
Aug
1
2
4
2
2
2
2
2
1
2
3
2
3
2
3
1
3
7
9
6
8
6
6
6
5
6
5
4
5
Sept
Oct
Nov
Dec
Year to Date
6
7
17
18
0
0
0
1
1
3
4
4
4
0
3
0
3
0
3
0
Overall Operations & Maintenance
Overall Performance
Total Number of Flights
Total Incidents
Uncontrollable Incidents
Score
Jan'07 Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
150
200
225
300
400
175
210
360
15
17
16
15
11
12
11
10
0
3
5
3
5
4
4
5
2
0
92.0% 94.0% 94.2% 96.7% 98.3% 95.4% 97.1% 97.8%
63
Oct
Nov
0
0
Dec
0
0
13
5
63
Operations Incidents
A
B
C
D
E
F
G
H
I
Description of Incident Jan'10 Feb Mar Apr May Jun Jul Aug Sept Oct
Collision
3
2
1
3
1
1
1
Foreign object damage
1
1
1
1
2
1
Bird strikes
1
3
3
2
4
3
2
2
Runway incursion
3
3
1
2
1
1
2
2
Aborted takeoff
1
0
2
1
0
1
0
0
Open
Open
Target
ActualMonth Total
10
9
9
8
7
8
7
9
6
6
5
7
5
6
4
6
4
0
Nov Dec Year to Date
12
7
20
15
5
0
0
3
0
3
0
3
0
64
64
Maintenance Incidents
A
B
C
D
E
F
G
H
I
Description of Incident Jan'10 Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec Year to Date
Hydraulic Incident
2
1
2
1
6
Engine Failure
1
2
1
1
1
6
Communication
1
4
2
2
2
2
1
3
17
Component failure
2
2
2
3
3
3
3
18
Electrical Incident
3
0
2
0
1
6
Open
0
Open
0
Target
ActualMonth Total
7
9
7
9
6
9
6
6
6
5
6
5
4
5
4
5
4
0
3
0
3
0
3
0
65
65
Overall Operations & Maintenance
Overall Performance Jan'10 Feb Mar Apr May Jun Jul Aug Sept Oct Nov Dec
Total Number of Flights 200 200 225 300 400 300 225 360
Total Incidents
18 17 17 15 11 12 11 11 0 0 0 13
Uncontrollable Incidents
1 3 3 2 4 3 2 2 0 0 0 5
Score
90.5% 90.0% 91.1% 94.3% 96.3% 95.0% 94.2% 96.4%
66
66
Operations Incidents
A
B
C
D
E
F
G
H
I
Description of Incident
Collision
Foreign object damage
Bird strikes
Runway incursion
Aborted takeoff
Open
Open
Jan'10
3
1
1
3
1
Target
Actual Month Total
Feb
Mar
2
10
9
Apr
May
Jun
3
1
2
2
1
1
3
3
0
1
1
3
1
2
9
8
7
8
7
9
Jul
Aug
Sept
4
1
0
1
1
3
1
1
2
2
2
0
1
1
2
2
0
6
6
5
7
5
6
4
6
Oct
4
0
Nov
3
0
Dec
3
0
Year to Date
12
7
20
15
5
0
0
3
0
Flight Operations Incidents
Year to Date
25
20
Collision
Foreign object damage
Bird strikes
Runway incursion
Aborted takeoff
Open
Open
Target
Month Total
Quantity
15
10
5
0
Jan'10
Feb
Mar
Apr
May
Jun
Jul
67
Aug
Sept
Oct
Nov
Dec
Year to
Date
Month
67
Maintenance Incidents
Description of Incident
Hydraulic Incident
Engine Failure
Communication
Component failure
Electrical Incident
Open
Open
A
B
C
D
E
F
G
H
I
Jan'10
2
1
1
2
3
Target
Actual Month Total
Feb
7
9
Mar
Apr
May
Jun
1
2
4
2
0
2
1
2
2
2
1
2
3
0
2
3
7
9
6
9
6
6
6
5
Jul
Aug
Sept
Oct
Nov
Dec
Year to Date
6
6
17
18
6
0
0
1
2
3
1
3
1
3
1
6
5
4
5
4
5
4
0
3
0
3
0
3
0
Maintenance Performance Incidents
20
18
16
Hydraulic Incident
Engine Failure
Communication
Component failure
Electrical Incident
Open
Open
Target
Month Total
Quantity
14
12
10
8
6
4
2
0
Jan'10
Feb
Mar
Apr
May
Jun
68
Jul
Aug
Sept
Oct
Nov
Dec
Year to
Date
Month
68
Overall Operations & Maintenance
Overall Performance
Total Number of Flights
Total Incidents
Uncontrollable Incidents
Score
Jan'10 Feb
Mar
Apr
May
Jun
Jul
Aug Sept
200
200
225
300
400
300
225
360
18
17
17
15
11
12
11
11
0
1
3
3
2
4
3
2
2
0
90.5% 90.0% 91.1% 94.3% 96.3% 95.0% 94.2% 96.4%
Oct
Nov
0
0
Dec
0
0
13
5
Overall Performance - Hazard Control
97%
96%
Percentage
95%
94%
93%
92%
91%
90%
Jan'10
Feb
Mar
Apr
May
Jun
Jul
Aug
Sept
Oct
Nov
Dec
Month
69
69
Bill James
Sabermetrics
Boston Red Sox
70
60 Seconds
70
Watch for signs of the following:
•Identifying of measureable
•Measuring activities that are not out of control
•Proactive actions
•Performance results
•Management commitment
•Confidence levels
71
71
Thank You
Bill LaPorte
72
72