Establishing the Integrity
of Data:
Measurement Systems Analysis
prepared by
Jeffrey T. Luftig, Ph.D.
W. Edwards Deming Professor of Management
Lockheed Martin Engineering Management Program
University of Colorado at Boulder
Topics
 Measurement Scales and Types of
Data
 Establishing the
 Reliability and Validity of
Instrumentation, or
 Precision and Accuracy of
Instrumentation
Measurement Scales and Data
Measurement Scales and Data
Measurement as a Process
 As in any process, regardless of the nature
of data collected or generated,
measurement systems must demonstrate
Stability through time, or control
Minimal variation as a proportion of
specifications, or capability
Minimal variation as a proportion of
process variation
Measurement as a Process
Standard
Product or
Process to be
Measured
MEASUREMENT
Procedure
Equipment
PROCESS
Operator
Ambient
Environmental
Characteristics
Measurement
Definition of Terms
 Reference Value
The theoretically or agreed upon correct
value of the characteristic being measured,
traceable to some standard
 Resolution
The smallest increment, or unit of measure,
available from a measurement process
Generally at least 1/10th of the specification
range
Definition of Terms
 Precision
The degree of agreement (or variability)
between individual measurements or test
results from measuring the same
specimen(s)
 Accuracy (Bias)
The difference between the average of the
measurement error distribution and the
reference value of the specimen measured
Precision
Precision vs. Accuracy
Accuracy
Definition of Terms
 Repeatability
The variation in repeated measurements of the
same items with a single measurement system
Within appraiser/system variation
 Reproducibility
The variation in the average measurements by
different appraisers or systems measuring the
same items
Between appraiser/system variation
Measurement Error
Distribution of repeated
measures on a single
specimen or part
Precision
- Repeatability
- Reproducibility
Accuracy
(Bias)
Reference Value
Terms
 Linearity
The degree to which bias changes with
changes in the magnitude of the
characteristic measured
 Stability
The dependability, or consistency of the
measurement process over time
Measurement
Systems Capability
 The variability resulting from measurement
error must not exceed a significant
proportion of the intended specifications
said to be capable
 In addition, it is not desirable for
measurement error to exceed a significant
proportion of the total process variability
 Capability is not the same as acceptability,
acceptability must be determined on a case
by case basis
Measurement Systems Capability
Measurement
Error Distribution
LSL
2
2
 E   Rpt
  Rpd
5.15E
(USL - LSL)
5.15 E
%R &R 
100
USL  LSL
USL
Measurement System Studies
 Potential Studies
Assess potential of a measurement
system to be capable over the long term
10 parts measured 2–3 times by one or
more appraisers
A “quick and dirty” study to find out if you
are in the ballpark
Assesses repeatability and reproducibility
Often called an R&R study
Measurement System Studies
 Potential Studies
Error Through Time
Tests of Between-Subjects Effects
Dependent Variable: ADAS ADA within Period
Source
Time Period
Specimen
Res idual
Total
Type III Sum
of Squares
54.021
325.873
431.604
811.498
df
3
7
21
31
Mean Square
18.007
46.553
20.553
F
.876
Sig.
.469
F
.097
Sig.
.961
Bias Through Time
Tests of Between-Subjects Effects
Dependent Variable: MEASURE Gram Weight
Source
Time Period
Specimen
Res idual
Total
Type III Sum
of Squares
21.844
925.719
1571.406
2518.969
df
3
7
21
31
Mean Square
7.281
132.246
74.829
Measurement System Studies
Potential Studies (continued)
Multiple Comparisons
Dependent Variable: Gram Weight
Tukey HSD
(I) Time of Day Parts
Were Measured
1 Early Morning
2 Late Morning
3 Early Afternoon
464.0
4 Late Afternoon
463.5
Mean
(J) Time of Day Parts
Difference
Were Measured
(I-J)
Std. Error
2 Late Morning
-.75
4.33
3 Early Afternoon
1.50
4.33
4 Late Afternoon
-.13
4.33
1 Early Morning
.75
4.33
3 Early Afternoon
2.25
4.33
4 Late Afternoon
.63
4.33
1 Early Morning
-1.50
4.33
Gram
Weight Values Repeated
Through Time4.33
2Mean
Late
Morning
-2.25
4 Late Afternoon
-1.63
4.33
1 Early Morning
.13
4.33
2 Late Morning
-.63
4.33
3 Early Afternoon
1.63
4.33
Estimated Means
Bas ed on observed463.0
means.
462.5
462.0
461.5
Early Morning
Late Morning
Early Afternoon
Late Afternoon
Sig.
.998
.985
1.000
.998
.953
.999
.985
.953
.981
1.000
.999
.981
95% Confidence Interval
Lower Bound Upper Bound
-12.81
11.31
-10.56
13.56
-12.18
11.93
-11.31
12.81
-9.81
14.31
-11.43
12.68
-13.56
10.56
-14.31
9.81
Gram
Weight
-13.68
10.43
a, b
Tukey HSD
-11.93
12.18
Time
-12.68of Day Parts11.43
Were Measured
N
-10.43
13.68
3 Early Afternoon
1 Early Morning
4 Late Afternoon
2 Late Morning
Sig.
Means for groups in homogeneous s ubs ets are dis played.
Bas ed on Type III Sum of Squares
The error term is Mean Square(Error) = 74.829.
a. Uses Harmonic Mean Sample Size = 8.000.
b. Alpha = .05.
Time of Day Parts Were Measured
8
8
8
8
Subset
1
461.63
463.13
463.25
463.88
.953
Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
 This result of the
previous analysis
allows us to
calculate the
average variance
of the repeated
measures, which
when we take its
square root gives
us the estimate of
the standard
deviation due to
measurement
error:
2 = 66.39
 = 8.15
Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
 Using the estimate
of measurement
error, we can
calculate the
Precision-Tolerance
ratio, which in the
case of short-term
studies, should be
less than 10%.
Assuming the
engineering tolerance
for this process is
470 (USL) –
450(LSL) = 20:
2 = 66.39
 = 8.15
P/T = Precision-Tolerance Ratio =
= 6() / USL-LSL
= 6(8.15) / 470 – 450
= 2.44
= 244% > 10% Requirement (S-T)
Results & Conclusions: Evaluating the
Precision & Accuracy of the Measurement
System
 Likewise, we can
estimate the
Accuracy (amount of
Bias) in the scale by
calculating the
average of the
differences between
the Means of the
Repeated Measures
and the True Values
for the associated
specimens:
 = -3.09
Estimate Bias at 3.09 Grams; as
compared to the Precision estimate,
this is arguably an inconsequential
value.
Measurement System Studies
 Short-term Studies
25 parts measured 5-8 times by one or
more appraisers
A more thorough short-term assessment
 Long-term Studies
8-10 parts measured 25+ times by one or
more appraisers
Assesses through time stability
Measurement System Studies
 Long-term Studies
Measurement System Studies
 Long-term Studies
Measurement Systems
Requirements
 Summary
 Regardless of the type of data gathered by an
instrument, and the assessment methodology
employed, the instrument or device utilized to
obtain criterion data must meet three
requirements before the experiment should
proceed:
 The instrument must be precise or reliable;
 The instrument must be accurate or valid; and
 The instrument should be / must be operating in a state
of statistical control.
Sources and References
 The material used in the PowerPoint presentations associated with this course was drawn from a number
of sources. Specifically, much of the content included was adopted or adapted from the following
previously-published material:
 Luftig, J. A Quality Improvement Strategy for Critical Product and Process Characteristics. Luftig
& Associates, Inc. Farmington Hills, MI, 1991
 Spooner-Jordan, V. Understanding Variation. Luftig & Warren International, Southfield, MI 1996
 Luftig, J. and Petrovich, M. Quality with Confidence in Manufacturing. SPSS, Inc. Chicago, IL
1997
 Littlejohn, R., Ouellette, S., & Petrovich, M. Black Belt Business Improvement Specialist
Training, Luftig & Warren International, 2000
 Ouellette, S. Six Sigma Champion Training, ROI Alliance, LLC & Luftig & Warren, International,
Southfield, MI 2005
 Luftig, J. An Overview of Total Quality Management, Luftig & Warren, International, 1992
 Luftig, J. Dr. Deming’s Theory of Profound Knowledge as a Foundation for Strategic Planning
and Policy Deployment, Luftig & Warren, International, 1997
 Luftig, J. and Jordan, V. Design of Experiments in Quality Engineering, McGraw-Hill/Irwin
Publishing Company, 1998
25