Jim Stuart
Kevin White
Manager, Applied Statistics
Eastman Chemical Company
jestuart@eastman.com
Senior Statistician
Voridian, a Division of Eastman
Chemical Company
kwhite@eastman.com
E A ST M A N
1
Statistical Thinking...
A habitual way of looking at work that:
 recognizes all activities as PROCESSES,
 recognizes that all processes have VARIABILITY,
 uses DATA to understand variation, and to drive
effective DECISION MAKING.
2
Outline
 Process Thinking Principles
 8 Lessons for Visualizing Variability
 Databased Decision Making
– Control Charts
– Special Cause Rules
– Change Point Analysis
3
Process Thinking
Managerial Data Should:
• Summarize performance on what is key to business success.
• Provide history of how the business has performed.
• Help predict the future.
• Provide the foundation for improvement. (Gap Identification)
• Provide a signal for reinforcement of accomplishments.
• Serve as a means for holding the gains.
4
Principles for Selecting Measures
• Sufficient number to adequately cover all the important facets of
the business. If it isn’t important to the business, don’t track it.
• Each measure should impact at least one stakeholder including
suppliers, publics, investors, customers, or employees (SPICE).
• Needs to be an appropriate mix of leading and lagging measures.
• Lend themselves to charts that are easy to read and interpret.
• Measures should be analyzed for appropriateness if the situation
or strategy changes.
Well-charted measures can provide a mechanism
for concise communication with stakeholders.
5
Indicators of Performance
Outputs
Inputs
Process
Customers
Products or Services
4
3
2
1
Gauge
Gauge
Gauge
Gauge
Quality
of Inputs
Process Quality/
Reliability
Supplier
Quality
(Leading Indicators
of Product and
Service Quality)
Product and
Service Quality
Customer
Satisfaction
Customer
Dissatisfaction
5
6
7
Gauge
Gauge
Gauge
Financial/Cost
People
Health, Safety &
Environmental
6
His project is 10% over budget…
Good News?
Bad News?
No Earthly Idea?
7
Statistical Thinking Lesson #1:
“It Depends”
• Was the budget set at the best current estimate or was it a
“guaranteed not to exceed number”?
• What are the implications of financial planning if everyone
uses guaranteed not to exceed numbers?
• What would you suspect if a particular project manager
finished every project exactly on schedule?
8
Statistical Thinking Lesson #2:
“Variation Happens…At Least It Should”
120
Distribution of Project Cost Variances
ESTIMATED COST
NUMBER OF PROJECTS
100
80
60
40
overrun
overrun
20
0
-20
-10
0
10
MANAGER A
20-30
-20
-10
0
10
MANAGER B
20
-20
-10
0
10
20
MANAGER C
9
Statistical Thinking Lesson #3:
“Show Data in Time Order”
215
215
210
210
205
205
200
200
195
195
190
190
185
185
1
31
61
91
121
151
181
211
241
271
301
1
215
215
210
210
205
205
200
200
195
195
190
190
185
185
1
31
61
91
121
151
181
211
241
271
301
1
31
31
61
61
91
121
151
181
211
241
271
301
91
121
151
181
211
241
271
301
10
Statistical Thinking Lesson #4:
“Beware Your Axes”
The selection of the scale of your vertical axis can
have a profound effect on the interpretation by the
audience…particularly if it is not their data.
Daily Sales in Thousands
Daily Sales in Thousands
207
206
200
205
180
204
160
203
140
202
201
120
200
100
199
80
198
197
60
196
40
195
20
194
0
193
1
31
61
91
121
151
1
31
61
91
121
151
11
Statistical Thinking Lesson #5:
“Don’t Over-Summarize”
Collect and display data at sufficient frequency to understand
the variation, and beware the trappings of bar-charts!
250
215
210
200
205
150
200
100
195
50
190
185
0
1
31
61
91
121
151
181
211
241
271
Opportunity Seeking
Improvement Motivating
(Let’s fix the dips!)
301
1
2
3
4
5
6
7
8
9
10
Management Review
And Presentation
(I’m OK, you’re OK)
12
I’m OK, You’re OK Slide
250.0
Summary
presentations utilizing
averages, ranges or
histograms should not
mislead the user into
taking action that
would not have been
taken if presented as a
time series.
200.0
150.0
100.0
50.0
0.0
2001
13
Statistical Thinking Lesson #6:
“Display History to Provide Context”
Key Result Area:
Stakeholder:
Date Prepared
Chart Title
Brief Description (Measure & Scope)
GOOD
Measure (Clear Description & Units)
500
450
400
350
300
250
200
150
100
50
Comparison 1
Comparison 2
Goal
0
J M
1998
Plot
sufficient
history to
visualize
trends
relative
to the
variation
M
J
S
N
J M
1999
M
J
S
N
J M
2000
Month/Year
Source of Data, How Measure is Calculated
Population (i.e., Entire Company, All U.S., etc.)
M
J
S
N
J M
2001
M
J
S
N
List of Supporting Information
(Tables, Bar Charts, Pie Charts, etc.)
14
Statistical Thinking Lesson #7:
“Provide Comparisons to Enable Gap Analysis”
Key Result Area:
Stakeholder:
Date Prepared
Chart Title
Brief Description (Measure & Scope)
GOOD
Measure (Clear Description & Units)
500
450
400
350
300
250
200
150
Relevant
comparisons
should be
placed in the
appropriate
locations on
the graph
100
50
Comparison 1
Comparison 2
Goal
0
J M
1998
M
J
S
N
J M
1999
M
J
S
N
J M
2000
Month/Year
Source of Data, How Measure is Calculated
Population (i.e., Entire Company, All U.S., etc.)
M
J
S
N
J M
2001
M
J
S
N
List of Supporting Information
(Tables, Bar Charts, Pie Charts, etc.)
15
Statistical Thinking Lesson #8:
“Use Moving Averages with Caution”
• Helps visualize trends through the
noise.
• Length should cover expected cycles.
Annual is most common.
• Tend to be sluggish.
• Can generate the appearance of cycles
or shifts which are not truly present.
– Cannot use run rules to signal
special causes
• Control limits for moving averages
can be calculated, but prefer not to
place them on the graph itself.
16
The Headlines Scream - Great News!
17
What We All Imagine…
‘cause this is what newspaper graphs look like
1995
1996
1997
1998
18
Reality
Projected 1998
19
Statistical Thinking...
A habitual way of looking at work that:
 recognizes all activities as PROCESSES,
P
 recognizes that all processes have VARIABILITY,
P
 uses DATA to understand variation, and to drive
effective DECISION MAKING.
20
Databased Decision Making
Managers are routinely faced with interpreting their metrics
and making a real-time decision as to whether the latest data
point tells them to do something.
 Good graphical depiction goes a long way
 Seasoned managers can see signals through the noise
 Statistics can take the subjectivity out of such
decisions
 One size does not fit all
21
Control Charts - The Two
Mistakes
 The False Alarm - Interpreting noise as a signal
 The Missed Alarm - Failure to detect a signal
22
Control Charts in Data Rich
Environments
 Control limits set at 3 standard errors
 Approximate 0.3% risk of a false alarm
 The risk of the missed alarm is often overlooked
 In parts manufacturing, greater sensitivity can be
obtained by giving consideration to the selection of
the rational subgroup
23
The “Run of 8” Rule
 Sometimes 7 and sometimes 9
 Also provides low risk of false alarms
 Used with 3 standard error limits, sensitivity is
improved
 Takes 8 points to initiate signal
Many other rules are also often used in
the data rich environment for greater
sensitivity but the tradeoff is a higher
false alarm rate.
24
Average # Points to Detect Shift
Average Run Lengths for Typical
Data Rich Environments
400
350
3 Std. Error Limits Only
300
3 Std. Error Limits + Run of 8
250
200
150
100
50
0
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 1.80 2.00 2.20 2.40 2.60 2.80 3.00
Mean Shift (Standard Errors)
25
Average # Points to Detect Shift
Average Run Lengths for Typical
Data Rich Environments
(Reduced Scale)
20.00
3 Std. Error Limits Only
3 Std. Error Limits + Run of 8
15.00
10.00
5.00
0.00
1.40
1.60
1.80
2.00
2.20
2.40
Mean Shift (Standard Errors)
2.60
2.80
3.00
26
Why These Rules Work for Data
Rich Environments
 High false alarm rates would lead to wasted time
doing investigation and possibly excessive process
adjustments.
 Poor sensitivity is often an acceptable trade-off
because for a lower false alarm rate
 And the next point is never far behind
27
Why Managerial Data Is Different
 The Obvious - less frequent data
 Detection of large process shifts is not as
important
 Actions taken are different
 Improve mindset, not maintain
28
Traditional Rules Applied to Low
Frequency Managerial Data
 A shift of 1.5 standard errors takes eight points on
average to detect
 This is little comfort if dealing with monthly
managerial data
29
The Individuals Chart
 An excellent all-purpose tool
 Very robust - low false alarms for virtually any data
distribution (typically < 1%)
 A single option for managers will get more use
 But, don’t forget the poor sensitivity
30
Sensitivity for Managerial Data
 Data is usually individual observations (cannot
subgroup)
 With traditional special cause rules, there is no
control over risk of the missed alarm
 User can control the width of the control limits
 User can employ some modified run rules
These modifications do come
with a higher false alarm rate!
31
Alternative Special Cause Rule Sets
A - Control limits set at 2.5 std. errors from the centerline.
B - Control limits set at 2.5 std. errors from the centerline
plus two points past 1.5 standard errors.
C - Control limits set at 2.0 std. errors from the centerline.
D - Control limits set at 2.0 std. errors from the centerline
plus a run of 6.
E - Control limits set at 2.0 std. errors from the centerline
plus three points past 1.0 standard errors.
F - Runs of 6 consecutive points on one side of the centerline.
32
Why Alternative Rules?
 Greater sensitivity is desired with an acceptable
number of false alarms
 What’s acceptable? It depends (See Lesson #1)!
– The data frequency
– Time to do investigation
– Importance of detecting quickly
– Magnitude of change deemed important
33
Average # Points to Detect Shift
Average Run Lengths for
Alternative Rules - Chart #1
100
90
(A) 2.5 Std Error Limits Only
80
70
(B) 2.5 Std Error Limits + 2 Past 1.5
60
50
40
30
20
10
0
0.00 0.20 0.40 0.60 0.80 1.00 1.20 1.40 1.60 1.80 2.00 2.20 2.40 2.60 2.80 3.00
Mean Shift (Standard Errors)
34
Average # Points to Detect Shift
Average Run Lengths for
Alternative Rules - Chart #2
85
80
75
70
65
60
55
50
45
40
35
30
25
20
15
10
5
0
(C) 2 Std. Error Limits Only
(D) 2 Std. Error Limits + Run of 6
(E) 2 Std. Error Limits + 3 Past 1
(F) Run of 6 Only
0.00
0.25
0.50
0.75
1.00
1.25
Mean Shift (Standard Errors)
1.50
1.75
2.00
35
Average False Alarms Per Year
SPECIAL CAUSE RULE SET
Data
Obs. Per
Frequency
Year
Hourly
4 Hours
8 Hours
Daily
Weekly
Monthly
Quarterly
8760
2190
1095
365
52
12
4
ARL for No Shift
2.5 Std.
2 Std.
Error
2 Std.
Error
3 Std. 2.5 Std. Limits 2 Std.
Error
Limits
3 Std.
Error
Error plus two Error
Limits
plus Run of 6
Error
Limits + Limits past 1.5 Limits
plus
three
(F)
Limits
Run of 8
(A)
Std.
(C)
Run of 6 past 1
Errors
(D)
Std.
(B)
Error (E)
23.64
5.91
2.96
0.99
0.14
0.03
0.01
57.48
14.37
7.19
2.40
0.34
0.08
0.03
108.59
27.15
13.57
4.52
0.64
0.15
0.05
171.50
42.87
21.44
7.15
1.02
0.23
0.08
401.28
100.32
50.16
16.72
2.38
0.55
0.18
509.01
127.25
63.63
21.21
3.02
0.70
0.23
435.60
108.90
54.45
18.15
2.59
0.60
0.20
139.45
34.86
17.43
5.81
0.83
0.19
0.06
370.50
152.40
80.67
51.08
21.83
17.21
20.11
62.82
36
Situational Recommendations
Situation
Recommendation
Goal is to Maintain
Use 3 Std Error Limits and
consider Run of 6
Goal is to Improve
Alternative Rules
Strong Slope in the Metric
Predictable Slope - Place
limits around sloped center
line.
Highly Unstable Process Where’s the Average?
Control Charts will not
apply.
37
Change Point Analysis
 The general principle is Monte Carlo simulation
 Advantages include:
– Very easy to use
– Detects mean and variation changes
– Excellent graphics
38
Change Point Analysis
 Confidence Levels for the probability a change is real
 Confidence Levels for when the change occurred.
 Handles any type of data
 More sensitive than control charts
 Not confused by outliers
39
Change Point Analysis
Example Graph
Change-Point Analysis of % of AR$ Past Due
% of AR$ Past Due
22
14
6
Jan-1990
Dec-1990
Nov-1991
Oct-1992
Sep-1993
Aug-1994
Jul-1995
Jun-1996
Month/Yr
40
Change Point Analysis
Example Table
Table of Significant Changes for % of AR$ Past Due
Confidence Level = 90%, Confidence Interval = 95%, Bootstraps = 1000, Sampling Without Replacement
Month/Yr
Confidence Interval
Conf. Level
From
To
Level
Dec-1990
(Nov-1990, Feb-1991)
98%
13.527
16.8
2
Jul-1991
(Jun-1991, Oct-1991)
98%
16.8
13.083
1
Jun-1993
(Feb-1992, Jan-1994)
94%
13.083
11.306
3
Dec-1994
(Sep-1994, May-1995)
100%
11.306
14.392
2
41
Change Point Analysis
Serial Dependency
Change Point Analysis of $ Accounts Receivable
$ Accounts Receivable
440000
330000
220000
Jan-1990 Dec-1990 Nov-1991 Oct-1992 Sep-1993 Aug-1994 Jul-1995 Jun-1996
Month/Yr
42
Conclusions
 Process thinking and careful selection of measures
can help keep managers focused
 Appropriate plotting is 90% of the battle
 Traditional control charts may not be optimal
 Alternative special cause rule sets should be
considered
 Change point analysis may be the closest thing to
an all-purpose tool for managers.
43
REFERENCES
1) Stuart J, White K, Methods for Handling Low Frequency Managerial Data, 2002
ASQ Annual Quality Congress Proceedings
2) Balestracci D, Data “Sanity”: Statistical Thinking Applied to Everyday Data, ASQ
Statistics Division Special Publication, Summer, 1998 (Available through ASQ,
www.asqstatdiv.org)
3) Britz G, Emerling D, Hare L, Hoerl R, Shade J, Statistical Thinking, ASQ Statistics
Division Special Publication, Summer, 1996 (Available through ASQ,
www.asqstatdiv.org)
4) Leitnaker, MG, Using the Power of Statistical Thinking, ASQ Statistics Division
Special Publication, Summer 2000 (Available through ASQ, www.asqstatdiv.org)
5) Wheeler DJ, Understanding Variation: The Key to Managing Chaos. Knoxville, TN:
SPC Press, Inc. 1986. (www.spcpress.com)
6) Taylor WA, "Change-Point Analysis: A Powerful New Tool For Detecting Changes,"
WEB: www.variation.com/cpa/tech/changepoint.html, 2000
7) Wheeler DJ, Building Continual Improvement, Knoxville, TN: SPC Press, Inc., 1998.
(www.spcpress.com)
44