BB 01 - 06 Measurement System Analysis V 2.1

Blended Lean Six Sigma
Black Belt Training –
ABInBev
Measurement System Analysis
©2010 ASQ. All Rights Reserved.
Module Objectives
 Analyze, and interpret measurement system
capability using:
•
•
•
•
•
Bias,
Linearity,
Stability,
Discrimination,
Repeatability and reproducibility (GR&R)
• Variable Gage R&R (GRR)
• Attribute Gage R&R (GRR)
This review module is aligned with your Moresteam Web Training,
Session 5, Measure II: Measurement Systems Analysis
2
© 2010 ASQ. All Rights Reserved.
Where are we now?
•
•
•
•
We now understand our process and its associated risks through
the use of Process Maps and FMEA.
We have brainstormed potential causes that could be contributing
to the problem and created a Cause and Effect diagram
We have collected data and now we must begin our analysis to
validate the cause and effect relationship and baseline process
performance.
We will look at some of the graphical analysis tools and also
evaluate whether the data we are collecting is repeatable and
reproducible (Measurement System Analysis)
3
© 2010 ASQ. All Rights Reserved.
Measurement Systems Analysis
Measurement System Analysis (MSA) identifies and quantifies
the different sources of variation that affect a measurement
system.
A measurement system includes both the measurement
instrument and the items being measured.
Gage Repeatability and Reproducibility (GRR) is used as an
interchangeable reference of MSA.
4
© 2010 ASQ. All Rights Reserved.
Why Measure - GRR
Terminology
Observed
Variation
Actual Process
Variation
Long Term
Process
Variation
Repeatability
Measured
Variation
Short Term
Process
Variation
Calibration
Variance
due to
Instrument
Stability
Linearity
Variance
due to
Operators
Reproducibility
Repeatability and Reproducibility are typically the primary
contributors to measurement error
5
© 2010 ASQ. All Rights Reserved.
Accuracy - Bias
Terminology
Accuracy – Bias: Bias is the difference between the output of the
measurement method and the true value. Bias effects include:
• Operator bias - different operators get detectably different
averages for the same thing.
• Machine bias - different machines get detectably different
averages for the same thing, etc.
• Others - day to day (environment), fixtures, customer and supplier
(sites).
6
© 2010 ASQ. All Rights Reserved.
Accuracy – Linearity
Accuracy – Linearity: Linearity measures the bias across the
operating range of a tool or instrument.
Poor
Linearity
Accuracy
(Difference between true
value and mean of the
instruments)
Good
Linearity
Low
High
Measured Value
7
© 2010 ASQ. All Rights Reserved.
Accuracy – Stability
Terminology
Accuracy – Stability: The consistency of measurements over time.
Master Value
(Reference Standard)
Time One
Time Two
8
© 2010 ASQ. All Rights Reserved.
Precision
Terminology
Precision = s2 Repeatability + s2 Reproducibility
•
•
Total variation in the measurement system
Measure of variation of repeated measurements
Repeatable
not reproducible
Precise not
Accurate
Not repeatable and
not reproducible
Neither Precise or Accurate
9
Repeatable and
reproducible
Accurate and
Precise
© 2010 ASQ. All Rights Reserved.
Precision - Repeatability
Terminology
Repeatability and Reproducibility represent two aspects of precision and help
describe the variability of a measurement method:
Precision - Repeatability – (a.k.a., “equipment variation”) is the
variation in measurements obtained when one operator uses the
same gauge for measuring the identical characteristics of the same
parts.
10
© 2010 ASQ. All Rights Reserved.
Precision - Reproducibility
Terminology
Precision - Reproducibility – (a.k.a., “appraiser variation”)
is the variation in the measurements made by different
operators using the same gage while measuring the
identical characteristic on the same parts.
Inspector A
Inspector A
Inspector B
Inspector C
Inspector B Inspector C
11
© 2010 ASQ. All Rights Reserved.
Precision - Discrimination
Terminology
Precision – Discrimination
The number of significant digits that can be measured by the system.
Increments should be about 0.1 of the product specification or the
process variation.
Poor discrimination
Good discrimination
12
© 2010 ASQ. All Rights Reserved.
Basics of Data Collection for the GRR
• Develop specific operational definitions
for the data collected to ensure
everyone is defining the elements the
same way.
• Develop a data collection plan to
provide a roadmap for data collection
and manage how the data is gathered.
• Be consistent – follow the plan!
• Create templates and / or check sheets
to control how the data is entered.
• Resist the temptation to analyze as
you collect – remain objective and
capture data exactly as it comes to
you.
13
© 2010 ASQ. All Rights Reserved.
Creating the Data Collection Sheet
Stat>Quality Tools>Gage Study
Randomization is critical for a valid Gage R&R study.
15
© 2010 ASQ. All Rights Reserved.
A GRR Example
\Datafile\Gageaiag.mtw*
Stat>Quality Tools>Gage Study>Gage R&R (Crossed)
(Crossed)
*This data file is owned by
Minitab, Inc., and is used with
permission.
16
© 2010 ASQ. All Rights Reserved.
GRR (Continued)
Gage R&R (ANOVA) for Response
Gage name:
Gage R&R %
Date of study:
Contribution
should be <9%
Components of Variation
100
P er cent
All parts
should have the
same pattern
Reported by:
Tolerance:
Misc:
Response by Part
% Contribution
% Study Var
% Tolerance
50
1.00
0.75
0.50
0
G age R&R
Repeat
Reprod
1
P art-to-P art
2
3
Sample Range
All points
should
be within the
control limits
2
7
8
9
0.10
0.75
_
R=0.0383
0.05
0.00
0.50
LC L=0
1
3
1.00
2
O per ator
0.75
3
Operator * Part Interaction
1.00
_
X=0.8075
U_C L=0.8796
A ver age
2
10
All operators
should have
the same pattern
1.00
Xbar Chart by Operator
Sample M ean
6
Response by Operator
3
U C L=0.1252
1
Most points
should be
outside of
the control
limits
5
P ar t
R Chart by Operator
1
4
LC L=0.7354
0.50
Operator
1
2
3
0.75
0.50
1
Worksheet: Gageaiag.MTW
2
3
4
5
6
P ar t
7
8
9
10
Diverging points
indicate part by
operator interaction
17
© 2010 ASQ. All Rights Reserved.
The Session Window
Gage R&R
%Contribution
Source
VarComp
(of VarComp)
0.0044375
10.67
Repeatability
0.0012917
3.10
Reproducibility
0.0031458
7.56
Operator
0.0009120
2.19
Operator*Part
0.0022338
5.37
Part-To-Part
0.0371644
89.33
Total Variation
0.0416019
100.00
Total Gage R&R
This is the % Contribution shown on the graphic analysis. Note the hierarchy
i.e. Operator and the Operator by Part interaction sum to the Reproducibility
and the Repeatability and Reproducibility sum to the Total Gage R&R.
18
© 2010 ASQ. All Rights Reserved.
The Session Window (Continued)
Study Var
%Study Var
StdDev (SD)
(6 * SD)
(%SV)
0.066615
0.39969
32.66
Repeatability
0.035940
0.21564
17.62
Reproducibility
0.056088
0.33653
27.50
Operator
0.030200
0.18120
14.81
Operator*Part
0.047263
0.28358
23.17
Part-To-Part
0.192781
1.15668
94.52
Total Variation
0.203965
1.22379
100.00
Source
Total Gage R&R
Number of Distinct Categories = 4
The number of distinct categories should be >4.
19
© 2010 ASQ. All Rights Reserved.
More Session Window
Joint Probability
Part is bad and is accepted
0.000
Part is good and is rejected
0.002
Conditional Probability
False Accept
False Reject
0.265
0.002
20
© 2010 ASQ. All Rights Reserved.
Notes About Minitab’s Assistant
In version 16 Minitab introduced the Assistant as shown
on the next slide.
The assistant as the name implies assists with many of
the frequently performed statistical analysis. It tests the
prerequisites and provides detailed reports in one step.
This training uses the traditional and assistant analysis
interchangeably so you become familiar with both
approaches.
The method is a matter of user preference.
21
© 2010 ASQ. All Rights Reserved.
Using the Assistant
Assistant>Measurement System Analysis
22
© 2010 ASQ. All Rights Reserved.
Assistant Setup
23
© 2010 ASQ. All Rights Reserved.
Assistant Output 1
Gage R&R Study for Response
Report Card
Check
Status
Description
Amount
of Data
i
To determine if a measurement system is capable of assessing process performance, you need good
estimates of the process variation and the measurement variation.
-- Process variation: Comprised of part-to-part and measurement variation. It can be estimated from a
large sample of historical data, or from the parts in the study. You chose to estimate from the parts.
Although the number of parts (10) satisfies the typical requirement of 10, the estimate may not be
precise. If the selected parts do not represent typical process variability, consider entering a historical
estimate or using more parts.
-- Measurement variation: Estimated from the parts, it is broken down into Reproducibility and
Repeatability. The number of parts (10) and operators (3) meets the typical requirement of 10 parts and
3 operators. This is usually adequate for estimating Repeatability, but the estimate of Reproducibility is
less precise. If the %Study Variation for Reproducibility estimate is large, you may want to examine the
differences between operators and determine if these differences are likely to extend to other operators.
Xbar
Chart
i
The control limits are based on Repeatability. Ideally, the variation from repeated measurements is
much less than the variation between parts. Guidelines suggest that approximately 50% or more should
fall outside the limits. In this study, 73.3% are outside.
R
Chart
i
Each point is the range of the measurements for a part. In this study, no points are above the upper
control limit, indicating all parts were measured with similar consistency.
No issues identified by the report card
24
© 2010 ASQ. All Rights Reserved.
Assistant Output 2
Gage R&R Study for Response
Variation Report
Xbar Chart of Part Averages by Operator
At least 50% should be outside the limits. (actual: 73.3%)
2
3
1
Diagnostic
information
similar to the
previous
analysis
1.00
0.75
0.50
R Chart of Test-Retest Ranges by Operator (Repeatability)
Operators and parts with larger ranges have less consistency.
0.10
0.05
0.00
Reproducibility — Operator by Part Interaction
Look for abnormal points or patterns.
1.00
0.75
0.50
Reproducibility — Operator Main Effects
Look for operators with higher or lower averages.
1.00
Variation by Source
Source
StDev
%Study
Variation
%Tolerance
Total Gage
Repeatability
Reproducibility
Operator
Operator by Part
Part-to-Part
0.067
0.036
0.056
0.030
0.047
0.193
32.66
17.62
27.50
14.81
23.17
94.52
33.31
17.97
28.04
15.10
23.63
96.39
Study Variation
0.204
100.00
101.98
Tolerance (upper spec - lower spec): 1.2
0.75
0.50
1
2
3
25
© 2010 ASQ. All Rights Reserved.
Assistant Output 3
Gage R&R Study for Response
Summary Report
Can you adequately assess process performance?
0% 10%
30%
Study Information
100%
Yes
No
32.7%
The measurement system variation equals 32.7% of the
process variation. The process variation is estimated from the
parts in the study.
(Replicates: Number of times each operator measured each part)
Comments
Can you sort good parts from bad?
0% 10%
30%
100%
Yes
No
33.3%
The measurement system variation equals 33.3% of the
tolerance.
Variation by Source
%Study Var
%Tolerance
45
30
30
Number of parts in study
Number of operators in study
Number of replicates
General rules used to determine the capability of the system:
<10%: acceptable
10% - 30%: marginal
>30%: unacceptable
Examine the bar chart showing the sources of variation. If the
total gage variation is unacceptable, look at repeatability and
reproducibility to guide improvements:
-- Test-Retest component (Repeatability): The variation that
occurs when the same person measures the same item multiple
times. This equals 54.0% of the measurement variation and is
17.6% of the total variation in the process.
-- Operator and Operator by Part components (Reproducibility):
The variation that occurs when different people measure the
same item. This equals 84.2% of the measurement variation
and is 27.5% of the total variation in the process.
10
3
2
Shows the
measurement
system is
unacceptable and
why. Also shows
the ability of the
measurement
system to
accurately
determine the
acceptability of the
associated
products.
15
10
0
Total Gage
Repeat
Reprod
26
© 2010 ASQ. All Rights Reserved.
Gage R&R Study Interpretation
Table is used to determine the acceptability of the measurement system
Characterization
Relative Utility
Operator Bias
Operator
Consistency
Method
Value
%GRR < 10%
The measurement system is acceptable.
%GRR >10% and <=30%
The measurement system may be
acceptable based on importance of
application, cost of gauge, cost of repairs,
etc.
%GRR > 30%
The measurement system needs
improvement. Make every effort to identify
the problems and have them corrected.
All appraisers have at least 50% of values
outside the control limits
No bias present
%GRR
Averages Chart
Range Chart
Meaning
One or more appraisers have less than 50% of
Bias is present
values outside the control limits
All appraisers ranges are within the control
limits
Results are consistent across appraisers
One or more appraisers ranges are outside the Results are not consistent across
control limits
appraisers
27
© 2010 ASQ. All Rights Reserved.
An ABI MSA Example 1 – Variable Data
Spare Parts Process – GRR for max levels
Gage R&R (ANOVA) for MAX
G age name:
D ate of study :
Parts should
have the same
pattern
Unacceptable
% contribution
and variation
Reported by :
Tolerance: 2
M isc:
P art max lev el setting
M ay 12, 2010
Katie S chiro
Components of Variation
MAX by Part-Brwy
160
% Contribution
100
80
0
0
Gage R&R
Repeat
Reprod
R Chart by Operator
Sample Range
Brwy
All points should
be within the
control limits
Part-to-Part
Expert
V CL V C F T JK M W SL M F T JK W M V CL C J K V CL C F T JK LA M W S L M V CL V C F T JK LA M W SL M
1B 1 1 C1 F 1F 1H 1 M N 1 W 1F 1H 1 N W 1 B 1 1F 1 1B 1 1F 1F 1H 1 1 M N 1 W 1 B 1 1C 1F 1F 1H 1 1 M N 1 W
3063030 30 30 306303 0130 163 001 40 4034040101 7037070370907 9090 90 9079079090190179001 4034040 40 40 4034 034 04014013400 1
6
6
0
0 9 0 06 06 0690 06 06 90 6 353 5385 53 34 0370 0370 9709 97 979 709 09 97 97 09 79 933 9 93 93 93 9339 39 93 93 39 34
1901 19 19 19 1901 19 19 01 90 98 98099 8 85 9739 9739 1 041 10 10 1041 41 10 104 1 09 6346 63 636 36346 46 63 63 46 39
10 1 1 01 01 0 10 1 10 10 1101 10 10 1 1 01 09 13 1 13 1 14 1 14 14 14 1 11 4 14 11 4 11 4 1 14 14 14 14 1 11 41 4 1 146
Brewery and
Expert should
have the same
pattern
Part-Brwy
MAX by Operator
100
50
50
25
_
UCL=7.75
R=2.37
LCL=0
0
Xbar Chart by Operator
80
Sample Mean
50
% Tolerance
Brwy
Expert
0
Brwy
Most points
should be
outside the
control limits
40
0
Expert
Operator
Operator * Part-Brwy Interaction
Av erage
Percent
% Study Var
80
O perator
Brwy
40
Expert
0
VCL V C F TJ KM W SL M F TJKW M VCL CJK VCL C F TJKL A MW SL M VCL V C F TJKL A MW SL M
1B1 1C1F1F 1H1 M N 1W 1F1H 1 N W1 B11 F 11 B11 F1F1 H1 1 M N 1 W1B1 1C1F1F 1H1 1 M N 1 W
306303030303063300130613001404034040101703707037 09079 090909079709900910719 00140340404040403 4304400410314001
6
6
6
0
09 0 0 06069 0060 690 635353855 33 40370 0370 9709 979 7970909979709 799339 939393933939939339 34
1901 1919191901191901 9098980998 859739 9739 1041 1010104141101041 096346 63636363464 6636 346 39
10 11010101011 010 1101101 01101091 311 311 411 4141 41 11414 11 41141 141414141 11414 1146
_
_
UCL=11.64
X=7.17
LCL=2.71
Part-Brwy
Conclusion: Variation is unacceptable at 78%
© Anheuser-Busch InBev. All Rights Reserved.
28
Katie Shiro© Belt
Project,
Zone
NA
2010 ASQ.
All Rights
Reserved.
An ABI Example 2 – Variable Data
Promotional Differentiation
Plan
•
•
•
Measure variance between internal STR data and external IRI Case
sales data
Results should show variance between internal and external (IRI) data is
repeatable and reproducible
Any anomalies should be investigated and corrected
How
•
•
•
•
•
Market: Arizona, Albertsons Chain
Measurement: Case sales data for Bud Light 30pks
Parts: Sales Levels 1) Avg Weekly Sales levels 2) Avg Weekly Sales
levels +10% 3) Avg Sales Weekly sales levels -10%
Operators: IRI or BudNET
Repeated measures are instances with the same sales levels historically
Chris Walker Belt Project – Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
29
© 2010 ASQ. All Rights Reserved.
An ABI Example 2 – Variable Data (Continued)
Part to part
variation
anticipated
Gage R&R
contribution
only 4.59%
-10% IRI
outsie of
control limits
Operators
have similar
pattern
Minimal
operator
interaction
Most points
outside of
control limits
GRR validates IRI data is able to detect a 10% sales lift in ABI Internal Sales-toRetailer data and therefore can be confidently used to measure retail execution
Chris Walker Belt Project – Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
30
© 2010 ASQ. All Rights Reserved.
Attribute Definitions
• An attribute gage compares each part to a specific set of
limits:
o
o
Accepts the part if limits are satisfied
Rejects the part if limits are not satisfied
• An attribute gage does not indicate how good or how bad a
part is, only if the part has been accepted or rejected.
Example: Is an employee on time or
late?
31
© 2010 ASQ. All Rights Reserved.
Attribute MSA Study: Preparation
• Select 20 – 30 parts
o
o
o
Must represent full range of variability
Some acceptable
Some not acceptable
• Measurements
o
o
o
3 Assessors
One “expert” who represents the reference standard
Evaluate each part at least 2 times
32
© 2010 ASQ. All Rights Reserved.
Attribute Acceptability
• Determine a minimum acceptable level of agreement
– Ideally all decisions should agree
• If the assessment method is not adequate and cannot be
improved, develop an alternate means of measurement
• Assessments may be made by:
– Ordinal (“On a scale of 1 to 5”)
– Classification (Color, Texture …)
– Attribute (Go-No go, Good-Bad, Yes-No)
Let’s look at an ABI Attribute example.
33
© 2010 ASQ. All Rights Reserved.
Attribute Gage R&R – An ABI Exercise
• Help desk at the call center assigns incidents to 11 global
types.
• We sent 11 participants a spreadsheet with the sample
incident information
• We also included as a standard/expert gage, the description
and definition of the Standard 11 Global problem types
• We then asked each of the 11 participants and one Expert
to review the incident information provided and, per the
control definition, determine if it was correctly assigned “1”
or incorrectly assigned “0”
• The process was repeated twice.
Datafile/Attribute ProblemType.mtw
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
34
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Datafile/Attribute ProblemType.mtw
Stat> Quality Tools > Attribute Agreement Analysis
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
35
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Date of study :
Reported by :
Name of product:
Misc:
Assessment Agreement
Where do we go from here?
Within Appraisers
100
95.0% C I
P ercent
80
80
60
60
Percent
Percent
100
Appraiser vs Standard
40
20
0
0
Appraiser
Agreement within
appraisers was fair
Agreement
vs. standard
was poor
40
20
A B C D E F G H I J K
95.0% C I
P ercent
A B C D E F G H I J K
Appraiser
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
36
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Within Appraisers
Assessment Agreement
Appraiser
A
B
C
D
E
F
G
H
I
J
K
# Inspected
11
11
11
11
11
11
11
11
11
11
11
# Matched
7
8
9
8
10
5
10
10
11
10
10
Percent
63.64
72.73
81.82
72.73
90.91
45.45
90.91
90.91
100.00
90.91
90.91
95% CI
(30.79, 89.07)
(39.03, 93.98)
(48.22, 97.72)
(39.03, 93.98)
(58.72, 99.77)
(16.75, 76.62)
(58.72, 99.77)
(58.72, 99.77)
(76.16, 100.00)
(58.72, 99.77)
(58.72, 99.77)
# Matched: Appraiser agrees with him/herself across trials.
What does ‘within appraisers’ measure?
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
37
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Within Appraisers
Fleiss' Kappa Statistics
Appraiser
A
B
C
D
E
F
G
H
I
J
K
Response
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
Kappa
0.26667
0.26667
0.43590
0.43590
0.60714
0.60714
0.45455
0.45455
0.79048
0.79048
-0.10000
-0.10000
0.74118
0.74118
0.81197
0.81197
1.00000
SE Kappa
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
0.301511
Z
0.88443
0.88443
1.44571
1.44571
2.01367
2.01367
1.50756
1.50756
2.62171
2.62171
-0.33166
-0.33166
2.45820
2.45820
2.69299
2.69299
3.31662
P(vs > 0)
0.1882
0.1882
0.0741
0.0741
0.0220
0.0220
0.0658
0.0658
0.0044
0.0044
0.6299
0.6299
0.0070
0.0070
0.0035
0.0035
0.0005
1.00000
0.81197
0.81197
0.81818
0.81818
0.301511
0.301511
0.301511
0.301511
0.301511
3.31662
2.69299
2.69299
2.71360
2.71360
0.0005
0.0035
0.0035
0.0033
0.0033
What is the Kappa
value telling us?
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
38
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Each Appraiser vs. Standard
Assessment Agreement
Appraiser
A
B
C
D
E
F
G
H
I
J
K
# Inspected
11
11
11
11
11
11
11
11
11
11
11
# Matched
4
5
3
4
7
2
8
6
5
6
5
Percent
36.36
45.45
27.27
36.36
63.64
18.18
72.73
54.55
45.45
54.55
45.45
95%
(10.93,
(16.75,
( 6.02,
(10.93,
(30.79,
( 2.28,
(39.03,
(23.38,
(16.75,
(23.38,
(16.75,
CI
69.21)
76.62)
60.97)
69.21)
89.07)
51.78)
93.98)
83.25)
76.62)
83.25)
76.62)
# Matched: Appraiser's assessment across trials agrees with the
known standard.
What does Appraiser vs. Standard tell us?
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
39
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Each Appraiser vs. Standard
Assessment Disagreement
Appraiser
A
B
C
D
E
F
G
H
I
J
K
# 1 / 0
*
*
*
*
*
*
*
*
*
*
*
Percent
*
*
*
*
*
*
*
*
*
*
*
# 0 / 1
3
3
6
4
3
3
2
4
6
4
5
Percent
27.27
27.27
54.55
36.36
27.27
27.27
18.18
36.36
54.55
36.36
45.45
# Mixed
4
3
2
3
1
6
1
1
0
1
1
Percent
36.36
27.27
18.18
27.27
9.09
54.55
9.09
9.09
0.00
9.09
9.09
# 1 / 0: Assessments across trials = 1 / standard = 0.
# 0 / 1: Assessments across trials = 0 / standard = 1.
# Mixed: Assessments across trials are not identical.
What does assessment disagreement mean?
© Anheuser-Busch InBev. All Rights Reserved.
40
Jeff Cassidy and Dan Davis Belt Project,
Zone
NA Reserved.
© 2010 ASQ.
All Rights
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Each Appraiser vs Standard
Fleiss' Kappa Statistics
Appraiser
A
B
C
D
E
F
G
H
I
J
K
Response
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
1
0
Kappa
-0.298611
-0.298611
-0.258170
-0.258170
-0.473214
-0.473214
-0.334559
-0.334559
-0.190058
-0.190058
-0.396825
-0.396825
-0.128947
-0.128947
-0.258170
-0.258170
-0.375000
SE Kappa
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
0.213201
Z
-1.40061
-1.40061
-1.21092
-1.21092
-2.21957
-2.21957
-1.56922
-1.56922
-0.89145
-0.89145
-1.86128
-1.86128
-0.60482
-0.60482
-1.21092
-1.21092
-1.75891
P(vs > 0)
0.9193
0.9193
0.8870
0.8870
0.9868
0.9868
0.9417
0.9417
0.8137
0.8137
0.9686
0.9686
0.7273
0.7273
0.8870
0.8870
0.9607
1
0
1
0
1
-0.375000
-0.258170
-0.258170
-0.334559
-0.334559
0.213201
0.213201
0.213201
0.213201
0.213201
-1.75891
-1.21092
-1.21092
-1.56922
-1.56922
0.9607
0.8870
0.8870
0.9417
0.9417
What is the Kappa
value telling us?
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
41
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
Between Appraisers
Assessment Agreement
# Inspected
11
# Matched
1
Percent
9.09
95% CI
(0.23, 41.28)
# Matched: All appraisers' assessments agree with each other.
Fleiss' Kappa Statistics
Response
0
1
Kappa
0.296505
0.296505
SE Kappa
0.0198380
0.0198380
Z
14.9463
14.9463
P(vs > 0)
0.0000
0.0000
What is the Kappa value telling us?
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
42
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
(Continued)
All Appraisers vs. Standard
Assessment Agreement
# Inspected
11
# Matched
0
Percent
0.00
95% CI
(0.00, 23.84)
# Matched: All appraisers' assessments agree with the
known standard.
Fleiss' Kappa Statistics
Response
0
1
Kappa
-0.300571
-0.300571
SE Kappa
0.0642824
0.0642824
Z
-4.67579
-4.67579
P(vs > 0)
1.0000
1.0000
What is the Kappa value telling us?
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
43
© 2010 ASQ. All Rights Reserved.
Attribute Agreement Analysis – ABI Exercise 1
Conclusion
Interpreting measurement system capability for attributes
• There are Statistical programs that will run the data
and provide a graph
o
o
o
If Kappa value = 1, then everyone agrees
Kappa = .9 is excellent
Kappa value < or = .7; means that operators need to be
retrained
• If, after retraining, the Kappa value does not show
improvement, then develop a new training method
Jeff Cassidy and Dan Davis Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
44
© 2010 ASQ. All Rights Reserved.
An ABI Attribute Example - MSA Example 1
Price Coordinators
responsible for
“measuring” a PCR to
make approval
decisions.
What is happening in this
example?
There is variation in repeatability between coordinators
and variation in actual measurement vs. the expert.
Michael Zacharias Belt Project, Zone NA
© Anheuser-Busch InBev. All Rights Reserved.
45
© 2010 ASQ. All Rights Reserved.
Attribute GRR Workshop
• Break out into your groups for this class exercise
o
o
o
You work at a brewing facility and have been given the
responsibility of approving the color of the finished product
(beer)
As a team select a brand of beer that you would like to work
with (example – Stella Artois; Bud Light, Becks, Brahma, Leffe,
etc…)
25 to 30 Beer color swatches have been provided to you for
beer color evaluation.
M
46
M
© 2010 ASQ. All Rights Reserved.
Attribute GRR Workshop
• Perform attribute GRR study using beer color swatches
o Assign an “expert” to both classify and accept/reject beer
color
o Classify the beer by color (Accept / Reject)
o Identify the defective beer color based on the brand of
beer that has been selected by your team (example –
Stella Artois; Bud Light, etc…)
o Use 3 operators/inspectors
• Complete attribute GRR analysis and report results (45
minutes).
Part 2 of the workshop - improve classification and acceptance
criteria, rerun attribute GRR study/analysis and report results
(30 minutes).
47
© 2010 ASQ. All Rights Reserved.
Inspect Beer Color
48
© 2010 ASQ. All Rights Reserved.
Variable Data GRR Workshop – Exercise 2
(Optional)
Verifying Our Measurement System
• We need to verify the measurement system by
assessing the system.
• How robust is our class measurement system?
49
© 2010 ASQ. All Rights Reserved.
Conduct a GRR with Objects in the Class
(Optional)
The Reliable Measurement Exercise
1. Objects in the classroom have been
‘assigned’
2. On a sheet of paper, write down each of
the objects in your group—including yours
3. Measure and record the length, width, and
depth of your object
4. No talking or sharing data with others
5. Pass your object to the person next to you
6. Measure and record the length, width, and
depth of the object given to you
7. Repeat steps 4, 5, and 6 until you have
measured and recorded all of the object s
in your group
8. Multiply the length x width x depth of each
object to obtain a single value of measure
for that object. Define this value as the
“DSF”. NO CALCULATORS-NO TALKING
9. We’ll report out and debrief in 30 minutes
50
© 2010 ASQ. All Rights Reserved.
GRR Key Actions
• Pick the right measurement system to evaluate
• Map the measurement process
• Pay attention to the likely cause of measurement
variation
• Conduct the data collection rigorously
• Think about the measurement process when drawing
conclusions from the Gage R&R
• Implement the countermeasures
• Hold the gains: Control and improve the measurement
system forever!!
51
© 2010 ASQ. All Rights Reserved.
Measurement System Analysis Summary
• There are two ways to do MSA (Gage R&R) based on the
type of data.
• Gage R&R studies can be conducted for Variable data
and Attribute Data
• Precise, accurate, repeatable and reproducible measures
are critical for effective process management
• Gage R&R evaluates Repeatability and Reproducibility
• For Variable data a Gage R&R value of > 10% is
undesirable
• For Attribute data a Kappa value of .7 or below is
undesirable
52
© 2010 ASQ. All Rights Reserved.
What We Covered

Accuracy, Linearity, Stability and Discrimination of
measuring instruments

How to conduct a Gage RR using Minitab for both
Variable and Attribute Data

Analyze and interpret measurement system
capability for Variable data.

Analyze and interpret measurement system
capability for Attribute data.
53
© 2010 ASQ. All Rights Reserved.
In the Next Module
• We will learn about the importance of some additional
tools for Black Belts focused on Analyzing the Process:
Correlation and Regression
o
o
Correlation enables you to develop and interpret the correlation
between variables
Teaches how to develop a mathematical model expressing the
relationship – regression
54
© 2010 ASQ. All Rights Reserved.