Measurement Systems Analysis:
What is it and why should I care?
Dec. 11, 2012
Barry Kulback
Global Lean Six Sigma Leader
Trane and Thermo King, brands of
Ingersoll Rand
Agenda
• About Ingersoll Rand
• Measurement systems and Measurement
System Error
• What is Measurement System Analysis
• Types of Measurement Systems Analysis
and examples
• Lessons Learned
Feel free to ask questions at any time!
2
About Ingersoll Rand
• A $14 billion diversified industrial company
• Publicly-held; NYSE:IR
• Operations in every major geographic region
• Strategic brands are #1 or #2 in their markets
• Products and services for commercial, industrial
and residential markets
3
What do you think about my measurement system?
4
We do a lot of measuring in our
businesses and in any process
improvement methodology…
• Running the business
 Monitor process performance
• Improving the business
 Baseline
 Set the levels of the adjustments
 Getting better or worse
 Validate improvement results
… and many of the measurement
systems we use for this have
similar problems as the one shown!
5
When not measuring well…
• Current process performance may be
misjudged
• Process improvement results may be
misinterpreted
• Missed opportunities
• Wasted effort, $
6
5 years of college in physics
& chemistry labs….
• Not once was it ever discussed… how
does your ability to measure influence the
results you think you are seeing?
7
My ‘aha’ moment
came in my Six Sigma
Black Belt Training
DMAIC process improvement
methodology
•
•
•
•
•
Define
Measure
Analyze
Improve
Control
Measurement System Analysis
Collect data
Six Sigma practitioners are known as Green Belt, Black
Belts and Master Black Belts
8
Premise for Six Sigma Methods
Sources of variation can be
– Identified
– Quantified
– Eliminated by control or prevention
Y = f(x)
Data driven decisions with a known level of confidence…
… we do a lot of measuring in Six Sigma
9
What is an MSA?
• When measuring there is always Measurement
System Error
• An MSA is a procedure to assess a Measurement
System
– Quantifies the Measurement System Error
– Acceptable? Yes or no
• If ‘no’  improve the measurement system
– MSA output can tell you where to look
10
Conducting an MSA
• For 2 of the 3 types of MSA’s we’ll cover today
• Guidelines
– Trained Operator(s)
– Proper Method
– Representative Samples
• Generally two to three operators
• Each unit is measured or assessed 2-3 times
by each operator
• Results are then analyzed
• Often with statistical software like Minitab
• Analytical and graphical outputs explain the results
11
Types of Data
Continuous Data (Quantitative)
–
–
–
–
–
–
Decimal subdivisions are meaningful
Time (seconds)
Pressure (psi)
Conveyor Speed (ft/min)
Rate (inches)
Temperature (degrees)
Attribute Data (Qualitative)
–
–
–
–
–
Categories
Good / Bad
Inventory Classification Code A, B or C
Shift number
Counted things (# receipt errors, # units shipped, etc.)
12
Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA
13
… but first just a wee bit of ‘technical’
just
the
Histogram
Dotplot
fitted
curve
Histogram
with
fit
18
16
14
Frequency
12
10
11.6
8
6
11.8
12.0
board
21 boards
4
2
0
11.70
11.64
11.85
11.76
12.00
50 boards
11.88
12.00
50 boards
14
12.2
12.15
12.12
12.4
12.30
12.24
12.36
so…
Dotplot
11.70
11.85
12.00
50 boards
12.15
12.30
just the fitted curve
18
16
16
14
14
12
12
10
=
8
6
4
2
0
Frequency
=
Frequency
Histogram
18
10
8
6
4
2
11.64
11.76
15
11.88
12.00
50 boards
12.12
12.24
12.36
0
11.64
11.76
11.88
12.00
50 boards
12.12
12.24
12.36
Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA
16
How does measurement system error
appear?
LSL
USL
15
Frequency
Actual process variation No measurement error
10
5
0
30
40
50
60
70
80
90
100
110
P ro c e s s
15
LSL
Observed process
variation With measurement error
USL
Frequency
10
5
0
17
30
40
50
60
70
O b s e rv e d
80
90
100
110
Why do we care?
Can’t really understand the true variation
present! And if it isn’t understood, it can’t
be fixed.
LSL
x
USL
x
x
A ‘bad’ part
can measure
good
x
A ‘good’ part
can measure
bad
Observed process variation With measurement error
18
Possible Sources of Process Variation
15
Long Term
Frequency
10
Actual
Process
Variation
5
Short Term
0
30
40
50
60
70
80
90
100
110
O b s e rv e d
Within Sample
Observed
Process
Variation
Repeatability
Calibration
Due to
instrument
Measurement
Variation
Stability
Due to
operators
Linearity
19
Possible Sources of Process Variation
15
Long Term
Frequency
10
Actual
Process
Variation
5
Short Term
0
30
40
50
60
70
80
90
100
110
O b s e rv e d
Within Sample
Observed
Process
Variation
Repeatability
Calibration
Due to
instrument
Measurement
Variation
Stability
Due to
operators
Linearity
20
Possible Sources of Process Variation
15
Long Term
Frequency
10
Actual
Process
Variation
5
Short Term
0
30
40
50
60
70
80
90
100
110
O b s e rv e d
Within Sample
Observed
Process
Variation
Repeatability
Calibration
Due to
instrument
Measurement
Variation
Stability
Due to
operators
Linearity
21
Possible Sources of Process Variation
15
Long Term
Frequency
10
Actual
Process
Variation
5
Short Term
0
30
40
50
60
70
80
90
100
110
O b s e rv e d
Within Sample
Observed
Process
Variation
Repeatability
Calibration
Due to
instrument
Measurement
Variation
Stability
Due to
operators
Linearity
‘Repeatability’ and ‘Reproducibility’ are the two main contributors to
Measurement System Error – hence ‘Gage R&R’
22
If Measurement System Error always
exists, when should we be concerned
with it?
-- When it it too large.
Too large compared to what?
-- That depends on what you
are using the measurement
system for!
23
Gage R&R – case study
• Process:
Compressor Machining
• Project:
Six Sigma Black Belt Project
Scrap Reduction
$375,000 in scrap / year
Capacity bound 
lost margins on lost sales
Considering $1M CAPX
to increase capacity
24
Inspection;
CMM –
measures to
.000001 inch
machined casting
assembly into compressor
Define
Measure
Analyze
Improve
Control
Actual
10
5
0
30
50
60
70
80
90
100
110
USL
LSL
15
Observed
10
5
0
30
Measurement system improved  project complete!
25
40
P ro c e s s
Frequency
•
•
•
•
•
Frequency
15
40
50
60
70
O b s e rv e d
80
90
100
110
Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA
26
Attribute Measurement Systems
Assessing attribute data often involved
judgment
… sometimes a little
• “it’s broke / it isn’t” or “it fits / it doesn’t”
… sometimes a lot
•
“ it dented too bad, scrap it”
27
Attribute Measurement Systems
It all starts with an Operational Definition:
•
•
•
•
•
The ‘spec’
Describes what the defects or categories are
Describes how to perform the appraisal assessment
Used to train those performing the assessment
Should be applied with a high degree of consistency
28
Attribute Agreement Analysis –
case study
• Process: 2” Copper Tube
Bending
• Project: Six Sigma
Green Belt Project
Scrap Reduction
29
Scrap Reason Pareto from
Baseline Measurement
100
150
100
60
40
50
20
0
Defect
Count
Percent
Cum %
0
nt
ram
e
g
e
l
g
o
r
stm
pr
ho
on
f
wn
es
r
l
e
u
j
o
o
d
k
d
p
w
d
n
e
in
p
t
kn
ut
be
wr
pa
en
co
un
urn
to
b
g
b
m
o
d
n
r
n
cla
ha
wro
llet
co
61
40.1
40.1
16
10.5
50.7
14
9.2
59.9
5/8
2
-
9
5.9
65.8
8
5.3
71.1
30
6
3.9
75.0
5
3.3
78.3
rs
he
Ot
4
29
2.6
19.1
80.9 100.0
Percent
Count
80
Spec = “No Wrinkles”
What’s a wrinkle? How bad is bad?
31
How good are our Operators at assessing
if a tube is wrinkled and should not be used?
•
•
•
•
•
Not wrinkled, use it.
Slight wrinkle, use it?
That’s not a wrinkle, it’s a tool mark! Use it?
Not a wrinkle, a stretch mark! Use it?
Wrinkled, scrap it.
32
An Attribute MSA was conducted:
 30 Samples
 4 Operators
 1 ‘Expert’
 2 Trials
33
Date of study:
Reported by:
Name of product:
Misc:
Assessment Agreement
Within Appraiser
100
[ , ] 95.0% CI
P e rc e n t
Percent
90
want 90% level of
agreement or higher
80
‘Expert’!
70
Albner
Danny
Jim
Appraiser
34
Michael
Ted
Spec = “No Wrinkles” – OK!
Would not use
Would use
?
35
Visual Aid Added
Prep for future new employees...
36
Types of Measurement Systems Analysis
Continuous Data
Gage R&R
Attribute Data
Attribute Agreement Analysis
Data Scrub MSA
37
Sometimes we get our data out
of the ‘system’
… is it right?
38
Data Scrub MSA – case study
• Process:
• Project:
Cooling the office
Six Sigma Green Belt Project
Reduce Energy Consumption
39
Lessons Learned…
40
An expensive gage
does not always
mean good
measurements
41
Remember this?
Inspection;
CMM –
measures to
.000001 inch
 $145,000 used
 Must operate in a environmentally
controlled room
 Strict procedures on part handling,
cleanliness, controlling local
conditions, controlling part
temperature…
42
A measurement
system is more
than a gage or a
operational
definition
(specification)
43
A continuous variable measurement
system is composed of:
• the gage / measuring device
• operator techniques
• set-up and handling techniques
• the environment in which the measurements
are being done (ex. lighting, access)
• recording of measurement results
44
The Attribute Measurement System
Operational
Definition
Problems in any of
these areas can lead
to too high a degree
of inconsistent /
incorrect
assessments
Training of
Operators
Application
45
Just because a
measurement system
has been in use
‘forever’ doesn’t mean
it is very good
46
Attribute MSA – case study
• Process: Invoicing –
Application of
appropriate tax
• Project: Six Sigma
Black Belt Project
Improve DSR
47
Tax Codes are applied to invoices
being sent out to Customers:
Sales Tax -- a percentage added to invoice,
customer pays
Use Tax -- a percentage of the cost of the
goods, company pays
Non-Tax -- government, hospital, etc., where
neither customer or company
pays
What tax should be applied?
48
How good are the Accountants at
applying the correct tax code?
(been doing it for years…)
49
An Attribute MSA was conducted:
 10 Samples (more would be better)
 3 Operators -- who do the job
every day
 1 ‘Expert’
 2 Trials
50
Date of study:
Reported by:
Name of product:
Misc:
Assessment Agreement
Appraiser vs Standard
100
100
90
90
80
80
70
70
Percent
Percent
Within Appraiser
60
50
50
40
30
30
20
20
10
10
2
3
Percent
60
40
1
[ , ] 95.0% CI
1
Appraiser
2
Appraiser
51
3
Date of study:
Reported by:
Name of product:
Misc:
Assessment Agreement
Appraiser vs Standard
100
100
90
90
80
80
70
70
Percent
Percent
Within Appraiser
60
50
50
40
30
30
20
20
10
10
2
3
Percent
60
40
1
[ , ] 95.0% CI
1
Appraiser
2
Appraiser
52
3
Just because a it came
out of the computer
doesn’t mean it is
accurate
53
Data Scrub MSA’s –
case studies
• Focused Management Team monthly
published results
54
Recap
• About Ingersoll Rand
• Measurement systems and Measurement
System Error
• What is Measurement System Analysis
• Types of Measurement Systems Analysis
and examples
• Lessons Learned
55
56
Final lesson:
You cannot assume
you are measuring
well
57
• Measurement error is
always present
• You don’t know if it is
small enough to ignore
unless you assess it!
58
Addendum
59
If using the measurement system to see if the item
being measured is within the spec, you want the
Measurement System Error (MSE) to be small
compared to that spec:
specification range, or tolerance
sample
observed
measurements
LSL
USL
x
MSE
This..
should be small when
compared to this...
The P/T ratio quantifies this...
A ratio of MSE to the
60 tolerance range
If using the measurement system to analyze the
variation present or control the process with
Statistical Process Control, you want MSE to be small
compared to that variation:
sample
observed
measurements
LSL
USL
x
MSE
This..
should be small when
compared to this...
61
The %R&R quantifies this...
A ratio of MSE to process
variation
Dec. 11, 2012 Professional Development Meeting
TOPIC AREA: MEASUREMENT SYSTEMS
Measurement System Analysis: What is it and why should I care?
SPEAKER – Barry Kulback, Master Black Belt, Six Sigma
For our December Professional Development Meeting we are
pleased to have Barry Kulback, Master Black Belt.
Measurement System Analysis (MSA) is an often overlooked
critical step in any process improvement process and is the
linchpin of Six Sigma’s Measure phase in the DMAIC (Define,
Measure, Analyze, Improve, and Control) methodology. This
non-technical presentation will review what MSA’s are and why
they are important. The major components of measurement
error will be discussed along with how measurement error may
color your perception of process performance. Examples of the
three most common types of MSA’s will be shared along with
some lessons learned. Come find out why you should assess
your measurement systems the next time you embark on a
process improvement journey.
62
Sound interesting? Come join us! We look forward to seeing you!
Reserve your place today by clicking here or by emailing
programs@apicsnashville.org.
DATE: December 11, 2012
PLACE: Holiday Inn Select Opryland Airport/Briley Parkway
2200 Elm Hill Pike
Nashville, TN 37214
(615) 883-9770
Click http://www.hinashville.com/directions.html for directions
TIME: 5:30 to 8:00 PM
COST: $20.00 for Dinner - Free if an unemployed member - bring your resume
You do not need to be an APICS member to attend
Remember! Attend Four (4) APICS Nashville Professional Development Meetings from
September 2012 thru April 2013 (excluding Tours) and attend the 5th PDM free! New
members to our chapter attend their first meeting at no charge when they bring their APICS
welcome letter!
Reserve your place today by clicking here or by emailing programs@apicsnashville.org.
Stay Informed! Click to Join our LinkedIn group APICS Middle Tennessee Chapter Nashville
63
Speaker Bio:
Barry Kulback’s 33 year professional career has been entirely with
Trane/American Standard now Ingersoll Rand. He is currently the
Global Lean Six Sigma leader for the Climate Solutions Sector of
Ingersoll Rand and very engaged in IR’s Lean Transformation. He
gained his BS in 1979 at Austin Peay State University majoring in
Physics and minoring in Computer Science. After spending 20 years
in Information Technology Barry joined Operations as a Six Sigma
Black Belt and progressed to a certified Six Sigma Master Black Belt in
2004. He was named Tennessee Academy of Science’s Industrial
Scientist of the year in 2006 and is the holder of 4 patents related to
algorithms for delivering to customer request by optimizing the match
of demand to supply. He maintains his membership in the American
Society for Quality as a Senior Member.
Reserve your place today by clicking here or by emailing
programs@apicsnashville.org.
Stay Informed! Click to Join our LinkedIn group APICS Middle Tennessee
Chapter Nashville
Thank you for your support of APICS.
64
Bring
•
•
•
•
•
•
•
1 copper tube
1 measuring tape
PC
Remote mouse
Backup – on data stick, on disk
Pad, pen to take notes re parking lot
45-50 minutes, leaving 10-15 minutes at
the end for question/answer
65
Title
• Topic
• Sub-topic
• Sub-sub-topic
• Sub-topic
• Sub-sub-topic
• Sub-sub-topic
66
Broad and Global Portfolio of HVAC Systems and Services
SERVICES / CONTROLS
TRANSPORT REFRIGERATION
Vision
To make building owner and transport customers more profitable and
efficient for life through innovative HVACR systems and services
67