advertisement

Establishing the Integrity of Data: Measurement Systems Analysis prepared by Jeffrey T. Luftig, Ph.D. W. Edwards Deming Professor of Management Lockheed Martin Engineering Management Program University of Colorado at Boulder Topics Measurement Scales and Types of Data Establishing the Reliability and Validity of Instrumentation, or Precision and Accuracy of Instrumentation Measurement Scales and Data Measurement Scales and Data Measurement as a Process As in any process, regardless of the nature of data collected or generated, measurement systems must demonstrate Stability through time, or control Minimal variation as a proportion of specifications, or capability Minimal variation as a proportion of process variation Measurement as a Process Standard Product or Process to be Measured MEASUREMENT Procedure Equipment PROCESS Operator Ambient Environmental Characteristics Measurement Definition of Terms Reference Value The theoretically or agreed upon correct value of the characteristic being measured, traceable to some standard Resolution The smallest increment, or unit of measure, available from a measurement process Generally at least 1/10th of the specification range Definition of Terms Precision The degree of agreement (or variability) between individual measurements or test results from measuring the same specimen(s) Accuracy (Bias) The difference between the average of the measurement error distribution and the reference value of the specimen measured Precision Precision vs. Accuracy Accuracy Definition of Terms Repeatability The variation in repeated measurements of the same items with a single measurement system Within appraiser/system variation Reproducibility The variation in the average measurements by different appraisers or systems measuring the same items Between appraiser/system variation Measurement Error Distribution of repeated measures on a single specimen or part Precision - Repeatability - Reproducibility Accuracy (Bias) Reference Value Terms Linearity The degree to which bias changes with changes in the magnitude of the characteristic measured Stability The dependability, or consistency of the measurement process over time Measurement Systems Capability The variability resulting from measurement error must not exceed a significant proportion of the intended specifications said to be capable In addition, it is not desirable for measurement error to exceed a significant proportion of the total process variability Capability is not the same as acceptability, acceptability must be determined on a case by case basis Measurement Systems Capability Measurement Error Distribution LSL 2 2 E Rpt Rpd 5.15E (USL - LSL) 5.15 E %R &R 100 USL LSL USL Measurement System Studies Potential Studies Assess potential of a measurement system to be capable over the long term 10 parts measured 2–3 times by one or more appraisers A “quick and dirty” study to find out if you are in the ballpark Assesses repeatability and reproducibility Often called an R&R study Measurement System Studies Potential Studies Error Through Time Tests of Between-Subjects Effects Dependent Variable: ADAS ADA within Period Source Time Period Specimen Res idual Total Type III Sum of Squares 54.021 325.873 431.604 811.498 df 3 7 21 31 Mean Square 18.007 46.553 20.553 F .876 Sig. .469 F .097 Sig. .961 Bias Through Time Tests of Between-Subjects Effects Dependent Variable: MEASURE Gram Weight Source Time Period Specimen Res idual Total Type III Sum of Squares 21.844 925.719 1571.406 2518.969 df 3 7 21 31 Mean Square 7.281 132.246 74.829 Measurement System Studies Potential Studies (continued) Multiple Comparisons Dependent Variable: Gram Weight Tukey HSD (I) Time of Day Parts Were Measured 1 Early Morning 2 Late Morning 3 Early Afternoon 464.0 4 Late Afternoon 463.5 Mean (J) Time of Day Parts Difference Were Measured (I-J) Std. Error 2 Late Morning -.75 4.33 3 Early Afternoon 1.50 4.33 4 Late Afternoon -.13 4.33 1 Early Morning .75 4.33 3 Early Afternoon 2.25 4.33 4 Late Afternoon .63 4.33 1 Early Morning -1.50 4.33 Gram Weight Values Repeated Through Time4.33 2Mean Late Morning -2.25 4 Late Afternoon -1.63 4.33 1 Early Morning .13 4.33 2 Late Morning -.63 4.33 3 Early Afternoon 1.63 4.33 Estimated Means Bas ed on observed463.0 means. 462.5 462.0 461.5 Early Morning Late Morning Early Afternoon Late Afternoon Sig. .998 .985 1.000 .998 .953 .999 .985 .953 .981 1.000 .999 .981 95% Confidence Interval Lower Bound Upper Bound -12.81 11.31 -10.56 13.56 -12.18 11.93 -11.31 12.81 -9.81 14.31 -11.43 12.68 -13.56 10.56 -14.31 9.81 Gram Weight -13.68 10.43 a, b Tukey HSD -11.93 12.18 Time -12.68of Day Parts11.43 Were Measured N -10.43 13.68 3 Early Afternoon 1 Early Morning 4 Late Afternoon 2 Late Morning Sig. Means for groups in homogeneous s ubs ets are dis played. Bas ed on Type III Sum of Squares The error term is Mean Square(Error) = 74.829. a. Uses Harmonic Mean Sample Size = 8.000. b. Alpha = .05. Time of Day Parts Were Measured 8 8 8 8 Subset 1 461.63 463.13 463.25 463.88 .953 Results & Conclusions: Evaluating the Precision & Accuracy of the Measurement System This result of the previous analysis allows us to calculate the average variance of the repeated measures, which when we take its square root gives us the estimate of the standard deviation due to measurement error: 2 = 66.39 = 8.15 Results & Conclusions: Evaluating the Precision & Accuracy of the Measurement System Using the estimate of measurement error, we can calculate the Precision-Tolerance ratio, which in the case of short-term studies, should be less than 10%. Assuming the engineering tolerance for this process is 470 (USL) – 450(LSL) = 20: 2 = 66.39 = 8.15 P/T = Precision-Tolerance Ratio = = 6() / USL-LSL = 6(8.15) / 470 – 450 = 2.44 = 244% > 10% Requirement (S-T) Results & Conclusions: Evaluating the Precision & Accuracy of the Measurement System Likewise, we can estimate the Accuracy (amount of Bias) in the scale by calculating the average of the differences between the Means of the Repeated Measures and the True Values for the associated specimens: = -3.09 Estimate Bias at 3.09 Grams; as compared to the Precision estimate, this is arguably an inconsequential value. Measurement System Studies Short-term Studies 25 parts measured 5-8 times by one or more appraisers A more thorough short-term assessment Long-term Studies 8-10 parts measured 25+ times by one or more appraisers Assesses through time stability Measurement System Studies Long-term Studies Measurement System Studies Long-term Studies Measurement Systems Requirements Summary Regardless of the type of data gathered by an instrument, and the assessment methodology employed, the instrument or device utilized to obtain criterion data must meet three requirements before the experiment should proceed: The instrument must be precise or reliable; The instrument must be accurate or valid; and The instrument should be / must be operating in a state of statistical control. Sources and References The material used in the PowerPoint presentations associated with this course was drawn from a number of sources. Specifically, much of the content included was adopted or adapted from the following previously-published material: Luftig, J. A Quality Improvement Strategy for Critical Product and Process Characteristics. Luftig & Associates, Inc. Farmington Hills, MI, 1991 Spooner-Jordan, V. Understanding Variation. Luftig & Warren International, Southfield, MI 1996 Luftig, J. and Petrovich, M. Quality with Confidence in Manufacturing. SPSS, Inc. Chicago, IL 1997 Littlejohn, R., Ouellette, S., & Petrovich, M. Black Belt Business Improvement Specialist Training, Luftig & Warren International, 2000 Ouellette, S. Six Sigma Champion Training, ROI Alliance, LLC & Luftig & Warren, International, Southfield, MI 2005 Luftig, J. An Overview of Total Quality Management, Luftig & Warren, International, 1992 Luftig, J. Dr. Deming’s Theory of Profound Knowledge as a Foundation for Strategic Planning and Policy Deployment, Luftig & Warren, International, 1997 Luftig, J. and Jordan, V. Design of Experiments in Quality Engineering, McGraw-Hill/Irwin Publishing Company, 1998 25