Chemometrics
Method comparison
Department of Chemical Pathology,
University of Pretoria,
Dr R Delport 2003
Chemometrics
Performance characteristics that are
taken into account:
Precision,
Accuracy,
Interference,
Working Range, and
Detection Limit.
http://www.westgard.com/lesson21.htm
Chemometrics
To carry out a good method validation study,
you need to do the following:
• Define a quality requirement for the test in the form of
the amount of error that is allowable, preferably an
allowable total error,
• Select appropriate experiments to reveal the expected
types of analytical errors,
• Collect the necessary experimental data,
• Perform statistical calculations on the data to estimate
the size of analytical errors,
• Compare the observed errors with the defined allowable
error, and
• Judge the acceptability of the observed method
performance.
http://www.westgard.com/lesson21.htm
Chemometrics
An experimental plan can be formulated by:
• Recognizing the types of errors that need to be
assessed for this test and method,
• Identifying the appropriate experiments and the
amount of data needed to estimate those types of
errors, then
• Organizing these experiments to perform the quick
and easy ones first and the ones taking more time
and effort last.
http://www.westgard.com/lesson21.htm
Chemometrics
Types of errors
• Imprecision or random errors,
• Inaccuracy, bias, or systematic errors, which
can be of two types
•Constant systematic error or
•Proportional systematic error.
http://www.westgard.com/lesson21.htm
Chemometrics
The dashed line in the middle of the figure
represents ideal method performance where
the test method and the comparative
method give exactly the same results.
The bottom line in the figure shows the
effect of a proportional systematic error,
where the magnitude of the error increases
as the test result gets higher.
The top line shows the effect of a constant
systematic error, where the whole line is
shifted up and all results are high by the
same amount.
Note that these results will also be subject to the random error of the
method, therefore the actual data points would scatter about the line as
illustrated in the figure. The range of this scatter above and below the line
provides some idea of the amount of random error that is present.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.
Replication experiment
• Provides information about random error
• Is performed by making measurements on a series of
aliquots of the same test samples within a specified
period of time, usually within an analytical run, within a
day, or over a period of a month.
• Preliminary experiment involves determining within-run
imprecision.
• Final experiment requires at least 20 working days to
provide estimate of the total imprecision, which includes
within and between run components.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.
Interference experiment
• Provides information about the constant systematic error
caused by the lack of specificity of the method.
• One test sample is prepared by adding the suspected
material to a sample containing the analyte.
• A second aliquot of the original sample is diluted by the
same amount with solvent, then both samples are
analyzed by the test method and the difference
determined.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.
Recovery experiment
• Provides information about the proportional systematic
error caused by a competitive reaction.
• Test sample is prepared by adding a standard solution of
the analyte being tested to an aliquot of a patient
specimen.
• A baseline sample is prepared by adding an equal
amount of the solvent used for the standard solution to a
second aliquot of the same patient specimen.
• The two samples are then analyzed by the test method
and the amount recovered is compared to the amount
http://www.westgard.com/lesson21.htm
added.
Chemometrics
Experiments for estimating analytical errors
Description of different experiments.
Comparison of methods experiment
• Is primarily used to estimate the average systematic
error observed with real patient samples.
• Can also reveal the constant or proportional nature of
that error.
• A series of patient specimens are collected and
analyzed by both the test method and a comparative
analytical method.
• The results are compared to determine the differences
between the methods, which are the analytical errors
between the methods.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Get the method working and establish an operating
protocol.
• Set up the instrument, prepare the reagents, calibrate
the methods, and obtain results from test samples.
• Check the standards and be sure the method is
properly calibrated, otherwise calibration errors will
show up throughout the experimental studies.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Determine the working range.
• The working range will vary from test to test and must be
defined as part of the specifications for the method.
• Check by analyzing a series of solutions, in duplicate or
triplicate, covering the concentrations range of interest.
• If detection limit is a critical characteristic, it may be
assessed at this time or in the next phase of preliminary
experiments.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Determine within-run imprecision, recovery, and
interference.
• The replication experiment might include 20 samples of
two or three materials whose concentrations closely match
the medical decision levels of interest for the tests.
• Interference experiments should test common problems
such as hemolysis, lipemia, and high bilirubin.
• Recovery experiments assess whether there are any
competitive reactions due to the matrix or other materials
in the native specimens.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
• Fresh patient specimens, and stored specimens.
• Minimum of 40 well-chosen patient samples should be
tested over a minimum of 5 working days.
• Distributed one-third in the low to low-normal range, onethird in the normal range, and one-third in the high
abnormal range.
• Method acceptability should be judged on the basis of the
sizes of the random, systematic, and total analytical
errors.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
Difference plot
• Used as initial graph if the two methods are
expected to show one-to-one agreement,
• Displays the difference between the test
minus comparative results on the y-axis
versus the comparative result on the x-axis,
such as shown in the accompanying figure.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
Difference plot
• Used as initial graph if the two methods are
expected to show one-to-one agreement,
• Displays the difference between the test
minus comparative results on the y-axis
versus the comparative result on the x-axis,
such as shown in the accompanying figure.
• Differences should scatter around the line
of zero differences, half being above and
half being below.
• Repeat measurements if indicated.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
Correlation coefficient
• The correlation coefficient is a statistic that
is almost always calculated and reported to
describe the results from a comparison of
methods study.
• The new or "test" method values are plotted
on y-axis and comparison
values on x-axis.
http://www.westgard.com/lesson21.htm
Chemometrics
Experiments for estimating analytical errors
Walking tour of the plan.
Comparison of methods experiment.
• A value of 1.000 indicates perfect
correlation between the results of two
methods.
• Other statistics (such as slope, intercept,
and standard deviation of the residuals) can
also be calculated from the same data to
estimate the size of errors occurring
between the methods.
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Random error, RE, or imprecision
• Can be either positive or negative.
• Direction and exact magnitude cannot be predicted.
Random error
• Imprecision is quantitated by calculating the standard
deviation (SD) from the results of a set of replicate
measurements.
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Random error, RE, or imprecision
• Can be either positive or negative.
• Direction and exact magnitude cannot be predicted.
Random error
• Imprecision is quantitated by calculating the standard
deviation (SD) from the results of a set of replicate
measurements.
• As the SD often increases as the concentration
increases, the coefficient of variation (CV) is
calculated to express the SD as a percentage of the
mean concentration from the replication study.
• Maximum size of a random error is commonly
expressed as a 2 SD or 3 SD estimate to help
understand the potential size of the error that might
occur.
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Systematic error, SE, or inaccuracy
• Is always in one direction.
• A systematic shift displaces the mean of the
distribution from its original value.
Systematic error
• In contrast to random errors that may be either
negative or positive and whose direction can not
be predicted, systematic errors are in one
direction and cause all the test results to be
either high or low.
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Systematic error, SE, or inaccuracy
Systematic error
• How high or how low can be described by the
bias, which is calculated as the average
difference, or the difference between averages,
between the value by the "test" method and a
"comparative" method in a comparison of
methods experiment.
• Alternatively, the expected systematic difference
may be predicted from the equation of the line
that best fits the graphical display of test method
values on the y-axis vs comparative method
values on the x-axis.
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Systematic error, SE, or inaccuracy
Systematic error
• How high or how low can be described by the
bias, which is calculated as the average
difference, or the difference between averages,
between the value by the "test" method and a
"comparative" method in a comparison of
methods experiment.
• Alternatively, the expected systematic difference
may be predicted from the equation of the line
that best fits the graphical display of test method
values on the y-axis vs comparative method
values on the x-axis.
• SE may stay the same over a range of
concentrations, in which case it can also be
called constant error, or it may change as
concentration changes, in which case it can be
called proportional error.
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Total Error, TE,
• It is the net or combined effect of random and
systematic errors, as shown in the
accompanying figure.
Total error
http://www.westgard.com/lesson15.htm
Chemometrics
Analytical errors
Total Error, TE,
• It is the net or combined effect of random and
systematic errors, as shown in the
accompanying figure.
Total error
• It represents a "worst-case" situation, or just
how far wrong a test result might be due to both
random and systematic errors.
• Because laboratories typically only make a
single measurement for each test, that
measurement can be in error by the expected
SE, or bias, plus 2 or 3 SD, depending on how
you quantitate the effect of RE.
http://www.westgard.com/lesson15.htm
Chemometrics
analysed with: A nalyse-It + Clinical Labo rato ry v1.40
Test Bias plots
NCCLS EP9-A example dataset
Method: Comparative v Test
Performed by Analyse-it Software, Ltd.
Identity line
A =B
300
Date
n
1 February 1999
40
Mean of Method - Test
250
Bias
95% CI
200
-0.1
-2.3 to 2.2
150
95% limits of agreement
Lower
Upper
100
50
0
0
100
200
Mean of Method - Com parative
300
-13.7
13.6
95% CI
-17.4 to -10.0
9.9 to 17.3
50
Chemometrics
0
0
100
200
300
Difference between methods
15
10
5
0
Zero bias
-5
-10
-15
0
100
200
Mean of Method - Com parative
300
0
5
10
15
Chemometrics
Difference between methods
(%)
15
10
5
0
Zero bias
-5
-10
-15
0
100
200
Mean of Method - Com parative
300