Blended Lean Six Sigma Black Belt Training – ABInBev Measurement System Analysis ©2010 ASQ. All Rights Reserved. Module Objectives Analyze, and interpret measurement system capability using: • • • • • Bias, Linearity, Stability, Discrimination, Repeatability and reproducibility (GR&R) • Variable Gage R&R (GRR) • Attribute Gage R&R (GRR) This review module is aligned with your Moresteam Web Training, Session 5, Measure II: Measurement Systems Analysis 2 © 2010 ASQ. All Rights Reserved. Where are we now? • • • • We now understand our process and its associated risks through the use of Process Maps and FMEA. We have brainstormed potential causes that could be contributing to the problem and created a Cause and Effect diagram We have collected data and now we must begin our analysis to validate the cause and effect relationship and baseline process performance. We will look at some of the graphical analysis tools and also evaluate whether the data we are collecting is repeatable and reproducible (Measurement System Analysis) 3 © 2010 ASQ. All Rights Reserved. Measurement Systems Analysis Measurement System Analysis (MSA) identifies and quantifies the different sources of variation that affect a measurement system. A measurement system includes both the measurement instrument and the items being measured. Gage Repeatability and Reproducibility (GRR) is used as an interchangeable reference of MSA. 4 © 2010 ASQ. All Rights Reserved. Why Measure - GRR Terminology Observed Variation Actual Process Variation Long Term Process Variation Repeatability Measured Variation Short Term Process Variation Calibration Variance due to Instrument Stability Linearity Variance due to Operators Reproducibility Repeatability and Reproducibility are typically the primary contributors to measurement error 5 © 2010 ASQ. All Rights Reserved. Accuracy - Bias Terminology Accuracy – Bias: Bias is the difference between the output of the measurement method and the true value. Bias effects include: • Operator bias - different operators get detectably different averages for the same thing. • Machine bias - different machines get detectably different averages for the same thing, etc. • Others - day to day (environment), fixtures, customer and supplier (sites). 6 © 2010 ASQ. All Rights Reserved. Accuracy – Linearity Accuracy – Linearity: Linearity measures the bias across the operating range of a tool or instrument. Poor Linearity Accuracy (Difference between true value and mean of the instruments) Good Linearity Low High Measured Value 7 © 2010 ASQ. All Rights Reserved. Accuracy – Stability Terminology Accuracy – Stability: The consistency of measurements over time. Master Value (Reference Standard) Time One Time Two 8 © 2010 ASQ. All Rights Reserved. Precision Terminology Precision = s2 Repeatability + s2 Reproducibility • • Total variation in the measurement system Measure of variation of repeated measurements Repeatable not reproducible Precise not Accurate Not repeatable and not reproducible Neither Precise or Accurate 9 Repeatable and reproducible Accurate and Precise © 2010 ASQ. All Rights Reserved. Precision - Repeatability Terminology Repeatability and Reproducibility represent two aspects of precision and help describe the variability of a measurement method: Precision - Repeatability – (a.k.a., “equipment variation”) is the variation in measurements obtained when one operator uses the same gauge for measuring the identical characteristics of the same parts. 10 © 2010 ASQ. All Rights Reserved. Precision - Reproducibility Terminology Precision - Reproducibility – (a.k.a., “appraiser variation”) is the variation in the measurements made by different operators using the same gage while measuring the identical characteristic on the same parts. Inspector A Inspector A Inspector B Inspector C Inspector B Inspector C 11 © 2010 ASQ. All Rights Reserved. Precision - Discrimination Terminology Precision – Discrimination The number of significant digits that can be measured by the system. Increments should be about 0.1 of the product specification or the process variation. Poor discrimination Good discrimination 12 © 2010 ASQ. All Rights Reserved. Basics of Data Collection for the GRR • Develop specific operational definitions for the data collected to ensure everyone is defining the elements the same way. • Develop a data collection plan to provide a roadmap for data collection and manage how the data is gathered. • Be consistent – follow the plan! • Create templates and / or check sheets to control how the data is entered. • Resist the temptation to analyze as you collect – remain objective and capture data exactly as it comes to you. 13 © 2010 ASQ. All Rights Reserved. Creating the Data Collection Sheet Stat>Quality Tools>Gage Study Randomization is critical for a valid Gage R&R study. 15 © 2010 ASQ. All Rights Reserved. A GRR Example \Datafile\Gageaiag.mtw* Stat>Quality Tools>Gage Study>Gage R&R (Crossed) (Crossed) *This data file is owned by Minitab, Inc., and is used with permission. 16 © 2010 ASQ. All Rights Reserved. GRR (Continued) Gage R&R (ANOVA) for Response Gage name: Gage R&R % Date of study: Contribution should be <9% Components of Variation 100 P er cent All parts should have the same pattern Reported by: Tolerance: Misc: Response by Part % Contribution % Study Var % Tolerance 50 1.00 0.75 0.50 0 G age R&R Repeat Reprod 1 P art-to-P art 2 3 Sample Range All points should be within the control limits 2 7 8 9 0.10 0.75 _ R=0.0383 0.05 0.00 0.50 LC L=0 1 3 1.00 2 O per ator 0.75 3 Operator * Part Interaction 1.00 _ X=0.8075 U_C L=0.8796 A ver age 2 10 All operators should have the same pattern 1.00 Xbar Chart by Operator Sample M ean 6 Response by Operator 3 U C L=0.1252 1 Most points should be outside of the control limits 5 P ar t R Chart by Operator 1 4 LC L=0.7354 0.50 Operator 1 2 3 0.75 0.50 1 Worksheet: Gageaiag.MTW 2 3 4 5 6 P ar t 7 8 9 10 Diverging points indicate part by operator interaction 17 © 2010 ASQ. All Rights Reserved. The Session Window Gage R&R %Contribution Source VarComp (of VarComp) 0.0044375 10.67 Repeatability 0.0012917 3.10 Reproducibility 0.0031458 7.56 Operator 0.0009120 2.19 Operator*Part 0.0022338 5.37 Part-To-Part 0.0371644 89.33 Total Variation 0.0416019 100.00 Total Gage R&R This is the % Contribution shown on the graphic analysis. Note the hierarchy i.e. Operator and the Operator by Part interaction sum to the Reproducibility and the Repeatability and Reproducibility sum to the Total Gage R&R. 18 © 2010 ASQ. All Rights Reserved. The Session Window (Continued) Study Var %Study Var StdDev (SD) (6 * SD) (%SV) 0.066615 0.39969 32.66 Repeatability 0.035940 0.21564 17.62 Reproducibility 0.056088 0.33653 27.50 Operator 0.030200 0.18120 14.81 Operator*Part 0.047263 0.28358 23.17 Part-To-Part 0.192781 1.15668 94.52 Total Variation 0.203965 1.22379 100.00 Source Total Gage R&R Number of Distinct Categories = 4 The number of distinct categories should be >4. 19 © 2010 ASQ. All Rights Reserved. More Session Window Joint Probability Part is bad and is accepted 0.000 Part is good and is rejected 0.002 Conditional Probability False Accept False Reject 0.265 0.002 20 © 2010 ASQ. All Rights Reserved. Notes About Minitab’s Assistant In version 16 Minitab introduced the Assistant as shown on the next slide. The assistant as the name implies assists with many of the frequently performed statistical analysis. It tests the prerequisites and provides detailed reports in one step. This training uses the traditional and assistant analysis interchangeably so you become familiar with both approaches. The method is a matter of user preference. 21 © 2010 ASQ. All Rights Reserved. Using the Assistant Assistant>Measurement System Analysis 22 © 2010 ASQ. All Rights Reserved. Assistant Setup 23 © 2010 ASQ. All Rights Reserved. Assistant Output 1 Gage R&R Study for Response Report Card Check Status Description Amount of Data i To determine if a measurement system is capable of assessing process performance, you need good estimates of the process variation and the measurement variation. -- Process variation: Comprised of part-to-part and measurement variation. It can be estimated from a large sample of historical data, or from the parts in the study. You chose to estimate from the parts. Although the number of parts (10) satisfies the typical requirement of 10, the estimate may not be precise. If the selected parts do not represent typical process variability, consider entering a historical estimate or using more parts. -- Measurement variation: Estimated from the parts, it is broken down into Reproducibility and Repeatability. The number of parts (10) and operators (3) meets the typical requirement of 10 parts and 3 operators. This is usually adequate for estimating Repeatability, but the estimate of Reproducibility is less precise. If the %Study Variation for Reproducibility estimate is large, you may want to examine the differences between operators and determine if these differences are likely to extend to other operators. Xbar Chart i The control limits are based on Repeatability. Ideally, the variation from repeated measurements is much less than the variation between parts. Guidelines suggest that approximately 50% or more should fall outside the limits. In this study, 73.3% are outside. R Chart i Each point is the range of the measurements for a part. In this study, no points are above the upper control limit, indicating all parts were measured with similar consistency. No issues identified by the report card 24 © 2010 ASQ. All Rights Reserved. Assistant Output 2 Gage R&R Study for Response Variation Report Xbar Chart of Part Averages by Operator At least 50% should be outside the limits. (actual: 73.3%) 2 3 1 Diagnostic information similar to the previous analysis 1.00 0.75 0.50 R Chart of Test-Retest Ranges by Operator (Repeatability) Operators and parts with larger ranges have less consistency. 0.10 0.05 0.00 Reproducibility — Operator by Part Interaction Look for abnormal points or patterns. 1.00 0.75 0.50 Reproducibility — Operator Main Effects Look for operators with higher or lower averages. 1.00 Variation by Source Source StDev %Study Variation %Tolerance Total Gage Repeatability Reproducibility Operator Operator by Part Part-to-Part 0.067 0.036 0.056 0.030 0.047 0.193 32.66 17.62 27.50 14.81 23.17 94.52 33.31 17.97 28.04 15.10 23.63 96.39 Study Variation 0.204 100.00 101.98 Tolerance (upper spec - lower spec): 1.2 0.75 0.50 1 2 3 25 © 2010 ASQ. All Rights Reserved. Assistant Output 3 Gage R&R Study for Response Summary Report Can you adequately assess process performance? 0% 10% 30% Study Information 100% Yes No 32.7% The measurement system variation equals 32.7% of the process variation. The process variation is estimated from the parts in the study. (Replicates: Number of times each operator measured each part) Comments Can you sort good parts from bad? 0% 10% 30% 100% Yes No 33.3% The measurement system variation equals 33.3% of the tolerance. Variation by Source %Study Var %Tolerance 45 30 30 Number of parts in study Number of operators in study Number of replicates General rules used to determine the capability of the system: <10%: acceptable 10% - 30%: marginal >30%: unacceptable Examine the bar chart showing the sources of variation. If the total gage variation is unacceptable, look at repeatability and reproducibility to guide improvements: -- Test-Retest component (Repeatability): The variation that occurs when the same person measures the same item multiple times. This equals 54.0% of the measurement variation and is 17.6% of the total variation in the process. -- Operator and Operator by Part components (Reproducibility): The variation that occurs when different people measure the same item. This equals 84.2% of the measurement variation and is 27.5% of the total variation in the process. 10 3 2 Shows the measurement system is unacceptable and why. Also shows the ability of the measurement system to accurately determine the acceptability of the associated products. 15 10 0 Total Gage Repeat Reprod 26 © 2010 ASQ. All Rights Reserved. Gage R&R Study Interpretation Table is used to determine the acceptability of the measurement system Characterization Relative Utility Operator Bias Operator Consistency Method Value %GRR < 10% The measurement system is acceptable. %GRR >10% and <=30% The measurement system may be acceptable based on importance of application, cost of gauge, cost of repairs, etc. %GRR > 30% The measurement system needs improvement. Make every effort to identify the problems and have them corrected. All appraisers have at least 50% of values outside the control limits No bias present %GRR Averages Chart Range Chart Meaning One or more appraisers have less than 50% of Bias is present values outside the control limits All appraisers ranges are within the control limits Results are consistent across appraisers One or more appraisers ranges are outside the Results are not consistent across control limits appraisers 27 © 2010 ASQ. All Rights Reserved. An ABI MSA Example 1 – Variable Data Spare Parts Process – GRR for max levels Gage R&R (ANOVA) for MAX G age name: D ate of study : Parts should have the same pattern Unacceptable % contribution and variation Reported by : Tolerance: 2 M isc: P art max lev el setting M ay 12, 2010 Katie S chiro Components of Variation MAX by Part-Brwy 160 % Contribution 100 80 0 0 Gage R&R Repeat Reprod R Chart by Operator Sample Range Brwy All points should be within the control limits Part-to-Part Expert V CL V C F T JK M W SL M F T JK W M V CL C J K V CL C F T JK LA M W S L M V CL V C F T JK LA M W SL M 1B 1 1 C1 F 1F 1H 1 M N 1 W 1F 1H 1 N W 1 B 1 1F 1 1B 1 1F 1F 1H 1 1 M N 1 W 1 B 1 1C 1F 1F 1H 1 1 M N 1 W 3063030 30 30 306303 0130 163 001 40 4034040101 7037070370907 9090 90 9079079090190179001 4034040 40 40 4034 034 04014013400 1 6 6 0 0 9 0 06 06 0690 06 06 90 6 353 5385 53 34 0370 0370 9709 97 979 709 09 97 97 09 79 933 9 93 93 93 9339 39 93 93 39 34 1901 19 19 19 1901 19 19 01 90 98 98099 8 85 9739 9739 1 041 10 10 1041 41 10 104 1 09 6346 63 636 36346 46 63 63 46 39 10 1 1 01 01 0 10 1 10 10 1101 10 10 1 1 01 09 13 1 13 1 14 1 14 14 14 1 11 4 14 11 4 11 4 1 14 14 14 14 1 11 41 4 1 146 Brewery and Expert should have the same pattern Part-Brwy MAX by Operator 100 50 50 25 _ UCL=7.75 R=2.37 LCL=0 0 Xbar Chart by Operator 80 Sample Mean 50 % Tolerance Brwy Expert 0 Brwy Most points should be outside the control limits 40 0 Expert Operator Operator * Part-Brwy Interaction Av erage Percent % Study Var 80 O perator Brwy 40 Expert 0 VCL V C F TJ KM W SL M F TJKW M VCL CJK VCL C F TJKL A MW SL M VCL V C F TJKL A MW SL M 1B1 1C1F1F 1H1 M N 1W 1F1H 1 N W1 B11 F 11 B11 F1F1 H1 1 M N 1 W1B1 1C1F1F 1H1 1 M N 1 W 306303030303063300130613001404034040101703707037 09079 090909079709900910719 00140340404040403 4304400410314001 6 6 6 0 09 0 0 06069 0060 690 635353855 33 40370 0370 9709 979 7970909979709 799339 939393933939939339 34 1901 1919191901191901 9098980998 859739 9739 1041 1010104141101041 096346 63636363464 6636 346 39 10 11010101011 010 1101101 01101091 311 311 411 4141 41 11414 11 41141 141414141 11414 1146 _ _ UCL=11.64 X=7.17 LCL=2.71 Part-Brwy Conclusion: Variation is unacceptable at 78% © Anheuser-Busch InBev. All Rights Reserved. 28 Katie Shiro© Belt Project, Zone NA 2010 ASQ. All Rights Reserved. An ABI Example 2 – Variable Data Promotional Differentiation Plan • • • Measure variance between internal STR data and external IRI Case sales data Results should show variance between internal and external (IRI) data is repeatable and reproducible Any anomalies should be investigated and corrected How • • • • • Market: Arizona, Albertsons Chain Measurement: Case sales data for Bud Light 30pks Parts: Sales Levels 1) Avg Weekly Sales levels 2) Avg Weekly Sales levels +10% 3) Avg Sales Weekly sales levels -10% Operators: IRI or BudNET Repeated measures are instances with the same sales levels historically Chris Walker Belt Project – Zone NA © Anheuser-Busch InBev. All Rights Reserved. 29 © 2010 ASQ. All Rights Reserved. An ABI Example 2 – Variable Data (Continued) Part to part variation anticipated Gage R&R contribution only 4.59% -10% IRI outsie of control limits Operators have similar pattern Minimal operator interaction Most points outside of control limits GRR validates IRI data is able to detect a 10% sales lift in ABI Internal Sales-toRetailer data and therefore can be confidently used to measure retail execution Chris Walker Belt Project – Zone NA © Anheuser-Busch InBev. All Rights Reserved. 30 © 2010 ASQ. All Rights Reserved. Attribute Definitions • An attribute gage compares each part to a specific set of limits: o o Accepts the part if limits are satisfied Rejects the part if limits are not satisfied • An attribute gage does not indicate how good or how bad a part is, only if the part has been accepted or rejected. Example: Is an employee on time or late? 31 © 2010 ASQ. All Rights Reserved. Attribute MSA Study: Preparation • Select 20 – 30 parts o o o Must represent full range of variability Some acceptable Some not acceptable • Measurements o o o 3 Assessors One “expert” who represents the reference standard Evaluate each part at least 2 times 32 © 2010 ASQ. All Rights Reserved. Attribute Acceptability • Determine a minimum acceptable level of agreement – Ideally all decisions should agree • If the assessment method is not adequate and cannot be improved, develop an alternate means of measurement • Assessments may be made by: – Ordinal (“On a scale of 1 to 5”) – Classification (Color, Texture …) – Attribute (Go-No go, Good-Bad, Yes-No) Let’s look at an ABI Attribute example. 33 © 2010 ASQ. All Rights Reserved. Attribute Gage R&R – An ABI Exercise • Help desk at the call center assigns incidents to 11 global types. • We sent 11 participants a spreadsheet with the sample incident information • We also included as a standard/expert gage, the description and definition of the Standard 11 Global problem types • We then asked each of the 11 participants and one Expert to review the incident information provided and, per the control definition, determine if it was correctly assigned “1” or incorrectly assigned “0” • The process was repeated twice. Datafile/Attribute ProblemType.mtw Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 34 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Datafile/Attribute ProblemType.mtw Stat> Quality Tools > Attribute Agreement Analysis Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 35 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Date of study : Reported by : Name of product: Misc: Assessment Agreement Where do we go from here? Within Appraisers 100 95.0% C I P ercent 80 80 60 60 Percent Percent 100 Appraiser vs Standard 40 20 0 0 Appraiser Agreement within appraisers was fair Agreement vs. standard was poor 40 20 A B C D E F G H I J K 95.0% C I P ercent A B C D E F G H I J K Appraiser Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 36 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Within Appraisers Assessment Agreement Appraiser A B C D E F G H I J K # Inspected 11 11 11 11 11 11 11 11 11 11 11 # Matched 7 8 9 8 10 5 10 10 11 10 10 Percent 63.64 72.73 81.82 72.73 90.91 45.45 90.91 90.91 100.00 90.91 90.91 95% CI (30.79, 89.07) (39.03, 93.98) (48.22, 97.72) (39.03, 93.98) (58.72, 99.77) (16.75, 76.62) (58.72, 99.77) (58.72, 99.77) (76.16, 100.00) (58.72, 99.77) (58.72, 99.77) # Matched: Appraiser agrees with him/herself across trials. What does ‘within appraisers’ measure? Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 37 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Within Appraisers Fleiss' Kappa Statistics Appraiser A B C D E F G H I J K Response 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 Kappa 0.26667 0.26667 0.43590 0.43590 0.60714 0.60714 0.45455 0.45455 0.79048 0.79048 -0.10000 -0.10000 0.74118 0.74118 0.81197 0.81197 1.00000 SE Kappa 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 0.301511 Z 0.88443 0.88443 1.44571 1.44571 2.01367 2.01367 1.50756 1.50756 2.62171 2.62171 -0.33166 -0.33166 2.45820 2.45820 2.69299 2.69299 3.31662 P(vs > 0) 0.1882 0.1882 0.0741 0.0741 0.0220 0.0220 0.0658 0.0658 0.0044 0.0044 0.6299 0.6299 0.0070 0.0070 0.0035 0.0035 0.0005 1.00000 0.81197 0.81197 0.81818 0.81818 0.301511 0.301511 0.301511 0.301511 0.301511 3.31662 2.69299 2.69299 2.71360 2.71360 0.0005 0.0035 0.0035 0.0033 0.0033 What is the Kappa value telling us? Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 38 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Each Appraiser vs. Standard Assessment Agreement Appraiser A B C D E F G H I J K # Inspected 11 11 11 11 11 11 11 11 11 11 11 # Matched 4 5 3 4 7 2 8 6 5 6 5 Percent 36.36 45.45 27.27 36.36 63.64 18.18 72.73 54.55 45.45 54.55 45.45 95% (10.93, (16.75, ( 6.02, (10.93, (30.79, ( 2.28, (39.03, (23.38, (16.75, (23.38, (16.75, CI 69.21) 76.62) 60.97) 69.21) 89.07) 51.78) 93.98) 83.25) 76.62) 83.25) 76.62) # Matched: Appraiser's assessment across trials agrees with the known standard. What does Appraiser vs. Standard tell us? Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 39 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Each Appraiser vs. Standard Assessment Disagreement Appraiser A B C D E F G H I J K # 1 / 0 * * * * * * * * * * * Percent * * * * * * * * * * * # 0 / 1 3 3 6 4 3 3 2 4 6 4 5 Percent 27.27 27.27 54.55 36.36 27.27 27.27 18.18 36.36 54.55 36.36 45.45 # Mixed 4 3 2 3 1 6 1 1 0 1 1 Percent 36.36 27.27 18.18 27.27 9.09 54.55 9.09 9.09 0.00 9.09 9.09 # 1 / 0: Assessments across trials = 1 / standard = 0. # 0 / 1: Assessments across trials = 0 / standard = 1. # Mixed: Assessments across trials are not identical. What does assessment disagreement mean? © Anheuser-Busch InBev. All Rights Reserved. 40 Jeff Cassidy and Dan Davis Belt Project, Zone NA Reserved. © 2010 ASQ. All Rights Attribute Agreement Analysis – ABI Exercise 1 (Continued) Each Appraiser vs Standard Fleiss' Kappa Statistics Appraiser A B C D E F G H I J K Response 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 1 0 Kappa -0.298611 -0.298611 -0.258170 -0.258170 -0.473214 -0.473214 -0.334559 -0.334559 -0.190058 -0.190058 -0.396825 -0.396825 -0.128947 -0.128947 -0.258170 -0.258170 -0.375000 SE Kappa 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 0.213201 Z -1.40061 -1.40061 -1.21092 -1.21092 -2.21957 -2.21957 -1.56922 -1.56922 -0.89145 -0.89145 -1.86128 -1.86128 -0.60482 -0.60482 -1.21092 -1.21092 -1.75891 P(vs > 0) 0.9193 0.9193 0.8870 0.8870 0.9868 0.9868 0.9417 0.9417 0.8137 0.8137 0.9686 0.9686 0.7273 0.7273 0.8870 0.8870 0.9607 1 0 1 0 1 -0.375000 -0.258170 -0.258170 -0.334559 -0.334559 0.213201 0.213201 0.213201 0.213201 0.213201 -1.75891 -1.21092 -1.21092 -1.56922 -1.56922 0.9607 0.8870 0.8870 0.9417 0.9417 What is the Kappa value telling us? Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 41 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) Between Appraisers Assessment Agreement # Inspected 11 # Matched 1 Percent 9.09 95% CI (0.23, 41.28) # Matched: All appraisers' assessments agree with each other. Fleiss' Kappa Statistics Response 0 1 Kappa 0.296505 0.296505 SE Kappa 0.0198380 0.0198380 Z 14.9463 14.9463 P(vs > 0) 0.0000 0.0000 What is the Kappa value telling us? Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 42 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 (Continued) All Appraisers vs. Standard Assessment Agreement # Inspected 11 # Matched 0 Percent 0.00 95% CI (0.00, 23.84) # Matched: All appraisers' assessments agree with the known standard. Fleiss' Kappa Statistics Response 0 1 Kappa -0.300571 -0.300571 SE Kappa 0.0642824 0.0642824 Z -4.67579 -4.67579 P(vs > 0) 1.0000 1.0000 What is the Kappa value telling us? Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 43 © 2010 ASQ. All Rights Reserved. Attribute Agreement Analysis – ABI Exercise 1 Conclusion Interpreting measurement system capability for attributes • There are Statistical programs that will run the data and provide a graph o o o If Kappa value = 1, then everyone agrees Kappa = .9 is excellent Kappa value < or = .7; means that operators need to be retrained • If, after retraining, the Kappa value does not show improvement, then develop a new training method Jeff Cassidy and Dan Davis Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 44 © 2010 ASQ. All Rights Reserved. An ABI Attribute Example - MSA Example 1 Price Coordinators responsible for “measuring” a PCR to make approval decisions. What is happening in this example? There is variation in repeatability between coordinators and variation in actual measurement vs. the expert. Michael Zacharias Belt Project, Zone NA © Anheuser-Busch InBev. All Rights Reserved. 45 © 2010 ASQ. All Rights Reserved. Attribute GRR Workshop • Break out into your groups for this class exercise o o o You work at a brewing facility and have been given the responsibility of approving the color of the finished product (beer) As a team select a brand of beer that you would like to work with (example – Stella Artois; Bud Light, Becks, Brahma, Leffe, etc…) 25 to 30 Beer color swatches have been provided to you for beer color evaluation. M 46 M © 2010 ASQ. All Rights Reserved. Attribute GRR Workshop • Perform attribute GRR study using beer color swatches o Assign an “expert” to both classify and accept/reject beer color o Classify the beer by color (Accept / Reject) o Identify the defective beer color based on the brand of beer that has been selected by your team (example – Stella Artois; Bud Light, etc…) o Use 3 operators/inspectors • Complete attribute GRR analysis and report results (45 minutes). Part 2 of the workshop - improve classification and acceptance criteria, rerun attribute GRR study/analysis and report results (30 minutes). 47 © 2010 ASQ. All Rights Reserved. Inspect Beer Color 48 © 2010 ASQ. All Rights Reserved. Variable Data GRR Workshop – Exercise 2 (Optional) Verifying Our Measurement System • We need to verify the measurement system by assessing the system. • How robust is our class measurement system? 49 © 2010 ASQ. All Rights Reserved. Conduct a GRR with Objects in the Class (Optional) The Reliable Measurement Exercise 1. Objects in the classroom have been ‘assigned’ 2. On a sheet of paper, write down each of the objects in your group—including yours 3. Measure and record the length, width, and depth of your object 4. No talking or sharing data with others 5. Pass your object to the person next to you 6. Measure and record the length, width, and depth of the object given to you 7. Repeat steps 4, 5, and 6 until you have measured and recorded all of the object s in your group 8. Multiply the length x width x depth of each object to obtain a single value of measure for that object. Define this value as the “DSF”. NO CALCULATORS-NO TALKING 9. We’ll report out and debrief in 30 minutes 50 © 2010 ASQ. All Rights Reserved. GRR Key Actions • Pick the right measurement system to evaluate • Map the measurement process • Pay attention to the likely cause of measurement variation • Conduct the data collection rigorously • Think about the measurement process when drawing conclusions from the Gage R&R • Implement the countermeasures • Hold the gains: Control and improve the measurement system forever!! 51 © 2010 ASQ. All Rights Reserved. Measurement System Analysis Summary • There are two ways to do MSA (Gage R&R) based on the type of data. • Gage R&R studies can be conducted for Variable data and Attribute Data • Precise, accurate, repeatable and reproducible measures are critical for effective process management • Gage R&R evaluates Repeatability and Reproducibility • For Variable data a Gage R&R value of > 10% is undesirable • For Attribute data a Kappa value of .7 or below is undesirable 52 © 2010 ASQ. All Rights Reserved. What We Covered Accuracy, Linearity, Stability and Discrimination of measuring instruments How to conduct a Gage RR using Minitab for both Variable and Attribute Data Analyze and interpret measurement system capability for Variable data. Analyze and interpret measurement system capability for Attribute data. 53 © 2010 ASQ. All Rights Reserved. In the Next Module • We will learn about the importance of some additional tools for Black Belts focused on Analyzing the Process: Correlation and Regression o o Correlation enables you to develop and interpret the correlation between variables Teaches how to develop a mathematical model expressing the relationship – regression 54 © 2010 ASQ. All Rights Reserved.