Comparing Two Measurement
Devices
0011 0010 1010 1101 0001 0100 1011
Brian Novatny
2003
1
2
4
Background
• Each measurement made by an
instrument or measuring device consists
of the true, unknown level of the
characteristic or item measured plus an
error of measurement.
• In practice it is important to know whether
or not the variance in errors of
measurement of an instrument, or the
imprecision of measurement, is suitably
small as compared to the variance of the
characteristic or product measured, or the
product variability.
0011 0010 1010 1101 0001 0100 1011
1
2
4
Background
0011 0010 1010 1101 0001 0100 1011
• For efficiency of the measuring process, the
variance in errors of measurement should be
several or many times smaller than the variability
of the characteristic measured or product
variance.
1
– Power = 1 - Beta
– Beta is the Type II error
2
4
• High measurement error causes the power of
most statistical tests to decrease unless
compensated for by larger sample sizes
Measurements are indicated by
0011 0010 1010 1101 0001 0100 1011
• Device 1 = B1 + Xi + Ei1
• where B1 = bias for device 1
•
Xi = true value
•
Ei1 = random errors for device 1
• Device 2 = B2 + Xi + Ei2
1
2
4
Regression Approach
0011 0010 1010 1101 0001 0100 1011
• Y = b0 +b1*X
• Device 1 = Intercept + Slope * Device 2
1
2
• Intercept value should be zero, if not, it indicates
bias of the two devices
• Slope term should be around 1 indicating
“Similarity”
• Mean Square Error estimate “Precision”
• R-Square term estimates some measure of
strength
4
Problems with Regression
0011 0010 1010 1101 0001 0100 1011
• The X variable (independent variable) should be
measured without error
– this is never the case, but errors in the X variable should
be small, and it won’t be when comparing devices
– they will be on equal footing
1
2
• Asymmetry - specifically designate one device to
predict the other
4
– prefer symmetric approach where is doesn’t matter
which variable is the input and which is the output
• Inverse regression can pose several problems
when trying to resolve the asymmetry problem
Problems with Regression
0011 0010 1010 1101 0001 0100 1011
• What should the R-square value be?
– No objective justification
1
2
• How to handle the case when there are multiple
measuring devices?
– Pairwise comparisons
– multiple testing error problem
4
Some Solutions
0011 0010 1010 1101 0001 0100 1011
• Grubbs model
– uses sums and differences
• Pittman and Morgan along with Maloney and
Rastogi
1
– provide proofs and refinements to Grubbs model
• Blackwood and Bradley
– multivariate test on bias and precision
2
4
• Tan and Iglewicz modify the standard regression
approach based on Mandel’s work of Errors in
Variables
Grubbs Model
0011 0010 1010 1101 0001 0100 1011
• Involves calculating the Sums and Differences of
the two devices
• The differences will be used to estimate bias
– standard paired t-test with n-1 degrees of freedom
1
2
• Performing a correlation analysis analysis on the
sums and differences is used to estimate
precision of the two devices
4
– follow student’s t distribution with n-2 degrees of
freedom
• Provides two independent tests
– one for bias of the two measurement devices
– one for precision equivalency
Simultaneous Test for Precision and
Bias
0011 0010 1010 1101 0001 0100 1011
• Multivariate approach instead of
independent tests
• Differences are regressed on the Sums
1
– Difference = Intercept + Slope * Sums
• Model F-test uses the UNCORRECTED
SUMS OF SQUARES to get the correct
number for the df
2
4
– instead of the familiar corrected sums of
squares
– the overall Type I error (Alpha) rate is exact
Simultaneous Test for Precision and
Bias
0011 0010 1010 1101 0001 0100 1011
• If the Model F indicates significance, then
tests for the Bias and Precision are just
the individual F-tests
1
2
– the overall test is generally more powerful
– it can reject the equivalence assumption of the
two devices even though each individual test
does not
4
Simultaneous Test for Precision and
Bias
0011 0010 1010 1101 0001 0100 1011
• The Precision test is exact
1
• The Bias test is exact only when the
precision between the two devices is
equal
2
4
– use the paired difference t-test otherwise
– more powerful
– Uses UMVU (uniform minimum variance
unbiased) estimate of the variance
Advantage of Simultaneous Test
0011 0010 1010 1101 0001 0100 1011
• Type I Error Rate is exact
• Overall test could reject even though
individual test do not
– power of test
• Statistical modeling
– usual array of diagnostics
• residuals
1
2
4
Regression approach
0011 0010 1010 1101 0001 0100 1011
• Regression can still be used, but an
adjustment to the model must be made
• In Simple Linear Regression, the results
(Beta Hat) are achieved by minimizing the
sum of squared residuals in the direction
of the dependent variable
• The correction is to achieve Beta hat by
minimizing the sum of squared residuals
in the direction of -Lambda/Beta hat
1
2
4
Regression approach
0011 0010 1010 1101 0001 0100 1011
• Lambda is a calculated value to adjust the
slope calculations
– called the Precision Ratio
• ratio of the machines repeatability
• machine 1 / machine 2 or inverse
1
2
• Lambda is determined by performing the
standard Gauge RxR studies or taking
repeated values
4
Regression approach
0011 0010 1010 1101 0001 0100 1011
• The approach still uses regression and
Lambda value helps solve the asymmetry
problem
1
2
– as lambda approaches infinity, implies X
values approach 0, so Beta hat is Sxy/Sxx,
which is where X is the independent variable
– as lambda approaches 0, implies Y values
approach 0, so Beta hat is Syy/Sxy, which is
where Y is the independent variable
4
Regression approach
0011 0010 1010 1101 0001 0100 1011
• Approach handles precision but not bias
• Uses Polar coordinates for confidence
intervals
– Slope = TAN (Theta)
– Intercept = Tau/COS (Theta)
1
2
4
Example
0011 0010 1010 1101 0001 0100 1011
Device1
5.00
5.17
5.17
5.00
8.50
5.67
8.00
9.00
8.50
11.17
9.00
Device2
4.73
4.83
4.63
4.37
7.03
4.50
7.03
7.93
7.50
9.57
7.70
Sum
9.73
10.00
9.80
9.37
15.53
10.17
15.03
16.93
16.00
20.73
16.70
Diff
0.27
0.33
0.53
0.63
1.47
1.17
0.97
1.07
1.00
1.60
1.30
1
2
4
Standard Regression Analysis
0011 0010 1010 1101 0001 0100 1011
•
•
•
•
R-Square = 98.24%
SQRT MSE = 0.31
Device 2 = -0.28 + 1.2 * Device 1
95% Confidence Intervals
– Intercept (-1.07,0.52)
– Slope (1.07,1.31)
• Conclusion
– No Bias, but not similar
1
2
4
Simultaneous Test for Bias and
Precision
0011 0010 1010 1101 0001 0100 1011
• Regress Diff on Sums (Uncorrected SS)
• Model F = 73.22 ==> p-value = 0.0000027
• Therefore, devices are different relative to
their bias and precision
• Individual Precision Test
– F = 17.5 ==> p-value 0.0024
1
2
4
• Individual Bias Test (not exact)
– F = 128.93 ==> p-value = 0.0000012
• Paired t-test has a T value = 6.98 and pvalue less then 0.00001
Correct Conclusion
0011 0010 1010 1101 0001 0100 1011
• The two devices measure differently
• Strong Bias
1
• Strong lack of precision (repeatability)
2
4
Potential Problems
0011 0010 1010 1101 0001 0100 1011
• Methods do not account for a difference in
Gain, or slope of devices
1
2
• Devices might measure equally well or
poor at the low and high ends of the scale,
but the relationship is not constant
4
– collect data at one end of the data range
– power of the test could be compromised
Multiple Measuring Devices
0011 0010 1010 1101 0001 0100 1011
• Grubbs and others propose technique for
three measuring devices
– comparisons when one device is a “Standard”
– with three devices, get a more powerful test
1
2
4
• Multivariate methods lead to fuller choice
of sub-hypothesis and can be used
regardless of the number of measurement
devices
Multiple Measuring Devices
0011 0010 1010 1101 0001 0100 1011
• One method involves performing a
multivariate regression on q-1
measurement devices
1
– independent variable = mean of each part
– dependent variable = deviations from that
mean
2
4
• Independent variable is averaged each
part across all the measurement devices
• Dependent variable is calculated by the
differences of each value from that mean
Multiple Measuring Devices
0011 0010 1010 1101 0001 0100 1011
• Generally have to fit a Full model and a
Reduced model (intercepts only)
• Then compare the two models
1
– usually through some matrix manipulation
2
4
• Technique can be performed by most
software packages that can perform
MANOVA techniques
Authors Opinion
• As the title says, this is just my opinion
and not based on any concrete proof
0011 0010 1010 1101 0001 0100 1011
– such as simulation studies
• My preferred method of analysis would be
the Multivariate approach using
Blackwood and Bradley’s Regression with
the Uncorrected Sums of Squares
1
2
4
– this procedure seems to have a more powerful
test in finding differences
– eliminates the possibility of getting a negative
variance, which Grubbs method could get
Authors Opinion
0011 0010 1010 1101 0001 0100 1011
• With the Multivariate approach, there is a
natural extension to testing more than two
measuring devices
1
2
• Of course, there is no reason to try both
the multivariate approach and Grubbs
approach since they are easily computed
using standard data analysis techniques
4
Final Comments
0011 0010 1010 1101 0001 0100 1011
• These methods are not to replace Gage
RxR studies, but to evaluate two devices
against each other
1
2
• Each device should be tested for bias and
repeatability and linearity as desired
4
– corrective action should be taken as needed
Final Comments
• The test for Bias is only a test for
agreement between the two devices, not a
bias against a standard
0011 0010 1010 1101 0001 0100 1011
– both devices could be grossly off from the
standard (but in the same direction and
amount)
1
2
4
• If there is a claim that one device is
“superior” to another (better precision),
these methods could prove the validity of
the claim and provide the precision
estimates
References for Two Device
Comparison
0011 0010 1010 1101 0001 0100 1011
• Grubbs, F.E. (1973). “Errors of Measurement,
Precision, Accuracy and the Statistical
Comparison of Measuring Instruments” ,
Technometrics Vol. 15 pp. 53-66
• Bradley, E.L. and Blackwood, L.G (1991). “An
Omnibus Test for Comparing Two Measuring
Devices”, Journal of Quality Technology, Vol. 23
pp. 12-16
• Tan, C.Y. and Iglewicz, B. (1999). “MeasurementMethods Comparisons and Linear Statistical
Relationship”, Technometrics, Vol. 41 pp. 192-201
1
2
4
References for Multiple Device
Comparison
0011 0010 1010 1101 0001 0100 1011
• Christensen, R. and Blackwood, L.G (1993).
“Tests for Precision and Accuracy of Multiple
Measuring Devices”, Technometrics, Vol. 35 pp.
411-420
1
2
• Bedrick, E.J. (2001). “An Efficient Scores Test for
Comparing Several Measuring Devices”, Journal
of Quality Technology, Vol. 33 pp. 96-102
4
0011 0010 1010 1101 0001 0100 1011
brian.novatny@us.michelin.com
1
2
4