SMILE
Johns Hopkins University
Baltimore, MD USA
Author:
Validation Committee
Review History
Document Number:
Equ35-A-04
Effective (or Post) Date:
19 Dec 2008
Date of last review:
11 Aug 2010
Reviewed by:
Heidi Hanes
SMILE Comments: This document is provided as an example only. It must be revised to accurately reflect your lab’s
specific processes and/or specific protocol requirements. Users are directed to countercheck facts when considering
their use in other applications. If you have any questions contact your SMILE representative.
SMILE Accuracy Guidelines-Chemistry
SMILE Accuracy Guidelines-Chemistry
Document Number
210
Effective Date
19 Dec 2008
Page
1 of 1
Supercedes
New
Subject
Guidelines for performing accuracy testing on a
chemistry analyzer
Date of last review:
Review History
Reviewed by:
Version # [0.0]
Revision Date
[dd/mm/yy]
19 Dec 2008
Erin Gover
Description (notes)
Revision
History
106731738
1 of 7
30-Apr-09
SMILE Chemistry Accuracy Flowchart
Determine your comparison or
reference method
Test 20 samples in duplicate
and average the results
Evaluate data
following step III.
A in attached
guideline
No
Collect additional data
over larger range
R >0.975?
Yes
Evaluate Error indices
following step III. B in
attached guideline
(Two Instrument Comparison)
95% of error
indices between 1.0 and 1.0?
Plot the data following step III. D
in attached guideline
(Alternate Method Comparison)
No
Yes
Yes
Accuracy experiment
PASSES
95% of error
indices between 1.0 and 1.0?
Accuracy experiment
FAILS. Troubleshoot
before repeating accuracy
studies
Evaluate data following step
III. D. 6 in attached guideline
(Six Sigma)
Yes
Sigma Metric
>2.0?
No
Accuracy experiment
FAILS. Troubleshoot
before repeating accuracy
studies
Accuracy experiment
PASSES
106731738
No
2 of 7
30-Apr-09
SMILE Chemistry Accuracy Guidelines
ACCURACY is the true value of a substance being measured. Verification of accuracy is the
process of determining that the test system is producing true, valid results.
I.
Determine your comparison or reference method
A. The comparison method must be previously validated.
B. The comparison method must be currently performing successfully on EQA
C. The ideal comparison method is a similar instrument/method
D. Comparison to an in-house method is preferred if the in-house instrument meets the
above criteria.
E. Samples with known values, such as proficiency testing samples or commercial
standards, may be used as the reference method.
II. Sample Criteria
A. A minimum of 20 samples that cover the reportable range of the method and include
points near the Medical Decision Points.
B. Patient, quality control, and proficiency testing materials may be used.
C. Testing: Run each sample in duplicate on each instrument
1. Ideally samples should be run within 2 hours of each other.
2. Duplicates should be averaged.
3. Data should be plotted immediately to identify and correct any outliers.
III. Acceptability criteria—evaluate the data using one of the options below:
A. Plot the data in EP Evaluator Two
Instrument Comparison Module or a similar
regression analysis program.
1. Plot the reference method on the X axis
and the method being validated on the Y
axis.
2. Enter the Allowable Total Error (TEa)
concentration and percent. Refer to
SMILE Chemistry TE Limits table.
(Appendix 2)
3. Enter the results for the reference (X)
and the new (Y) methods.
4. Evaluate the statistics by clicking on Print
Preview or on the statistics tab.
5. The correlation coefficient (R) must be
>0.975
A. Plot the data in a regression analysis
program or spreadsheet. The SMILE
Accuracy Pack is included for this use
(Appendix 1).
1. Plot the reference method on the X axis
and the method being validated on the Y
axis.
2. Enter the Allowable Total Error (TEa)
concentration and percent. Refer to
SMILE Chemistry TE Limits table.
(Appendix 2)
3. Enter the results for the reference (X)
and the new (Y) methods.
4. Evaluate the statistics.
5. The correlation coefficient (R) must be
>0.975
If
Then
R <0.975
 Data does not extend over acceptable range.
 More data must be evaluated over larger range.
R >0.975
 Proceed with step B below to evaluate acceptability.
B. Using this method, 95% of the data points from the comparative method must be
within Total Allowable Error limit of the reference method (refer to SMILE
Chemistry TE Limits table, Appendix 2). Use one of the methods below to
evaluate.
106731738
3 of 7
30-Apr-09
1. Using a regression analysis program or
spreadsheet (such as the SMILE Accuracy
Pack, Appendix 1) calculate the “Error
Index” to measure the difference between
the two methods as a ratio of the Total
Allowable Error.
a. The Error Index can be calculated by
subtracting the reference method data
point (X) from the method being validated
data point (Y) and dividing by the Total
Allowable Error (TEa). The equation is:
(Y-X)/TEa.
b. Use the percent TEa or the minimum
detectable difference (absolute TEa),
whichever is greater.
c. The absolute TEa value is calculated by
converting the TEa percentage to a
decimal and multiplying by the reference
method data point.
d. The Error Index is measured for each XY pair, and must fall within -1 and 1. If
more than 5% of the specimens have an
Error Index of less than -1 or greater than
1, the accuracy experiment fails.
1. The EP Evaluator program uses a statistic
called the “Error Index” to measure the
difference between the two methods as a
ratio of the Total Allowable Error.
a. The Error Index can be calculated by
subtracting the reference method data
point (X) from the method being validated
data point (Y) and dividing by the Total
Allowable Error (TEa). The equation is:
(Y-X)/TEa.
b. EP Evaluator uses the percent TEa or
the minimum detectable difference
(absolute TEa), whichever is greater.
c. The absolute TEa value is calculated by
converting the TEa percentage to a
decimal and multiplying by the reference
method data point.
d. The Error Index is measured for each XY pair, and must fall within -1 and 1. If
more than 5% of the specimens have an
Error Index of less than -1 or greater than
1, the accuracy experiment fails. (See
Appendix 3 for an example EP report).
C. Evaluate results using the table below:
If
95% of Error Indices
are acceptable
<95% of Error Indices
are acceptable
Then
 Accuracy is acceptable
 Proceed with Linearity experiments

Proceed with step D below.
D. Plot the data in a regression analysis program such as the SMILE Accuracy
Pack (Appendix 1) or EP Evaluator Alternate (Quantitative) Method
Comparison module. Deming Regression is preferred, however if not
available regular regression is acceptable provided that the Correlation
Coefficient (R) is >0.975.
1. Plot the reference method on the X axis and the method being validated
on the Y axis.
2. Enter the Allowable Total Error (TEa) concentration and percent. Refer to
SMILE Chemistry TE Limits table (Appendix 2).
3. Enter Medical Decision Points
i.
A Medical Decision Point (MDP) is the concentration of the analyte at
which a medical decision is triggered. You may enter up to five values.
These values are plotted on the graph, and the program computes a
predicted Y value and confidence interval at each X decision point.
106731738
4 of 7
30-Apr-09
ii.
For many analytes, the medical decision points correspond to the
lower and upper limits of the normal range. It is advisable to also
include medical decision points at the DAIDS Toxicity limits and/or
laboratory established critical values.
4. Evaluate using one of the methods described below:
a. If using EP Evaluator program create the
report by selecting the Print Preview
button.
b. Review the Medical Decision Point
Analysis data on page 1 of the report. (See
Appendix 4 for an example EP report)
c. Subtract the X Method MDP from the low
and the high 95% confidence limits for
each Medical Decision Point.
d. Calculate Error Indices for each end of the
95% confidence limits for each Medical
Decision Point as follows: Divide the
difference (from step c above) by the
SMILE TE Limit absolute value.
ii. Use the percent TEa or the minimum
detectable difference, whichever is
greater.
iii. To calculate the absolute TEa value,
convert the TEa percentage to a
decimal and multiply by the reference
method data point.
iv. All error indices must fall within -1 and
1. An Excel spreadsheet is provided to
calculate Error Indices (See Appendix
5 SMILE Error Index Calculator).
a. If using a regression analysis program or
spreadsheet calculate and evaluate the
95% confidence limits and error indices.
The SMILE Accuracy Pack, (Appendix 1)
performs these calculations automatically.
i. Subtract the X Method MDP from the
low and the high 95% confidence limits
for each Medical Decision Point.
ii. Calculate Error Indices for each end of
the 95% confidence limits for each
Medical Decision Point as follows:
Divide the difference (from step i. above)
by the SMILE TE Limit absolute value.
1. Use the percent TEa or the minimum
detectable difference, whichever is
greater.
2. To calculate the absolute TEa value,
convert the TEa percentage to a
decimal and multiply by the reference
method data point.
i. All error indices must fall within -1
and 1. An Excel spreadsheet is
provided to calculate Error Indices
(See Appendix 5 SMILE Error Index
Calculator).
5. Evaluate acceptability using the table below:
If
All error indices fall
between -1 and 1
Any error indices fall
outside -1 and 1
Then
 Proceed with step 6 below.



Consider possible causes of inaccuracy.
Troubleshoot before repeating accuracy studies.
Consult SMILE for assistance.
6. Six Sigma method acceptability evaluation
a. Plot the data in a Six Sigma analysis program such as the SMILE
Accuracy Pack (Appendix 1) or EP Evaluator Six Sigma Metrics module.
b. Enter the Allowable Total Error (TEa) concentration and percent. Refer to
SMILE Chemistry TE Limits table.
c. Enter Medical Decision Points.
d. Enter the slope and intercept calculated in step D above.
i. If using EP Evaluator, use the Deming Regression slope and intercept
calculated by the program.
106731738
5 of 7
30-Apr-09
ii.
If EP Evaluator is not used, it is acceptable to use slope and intercept
calculated using regular regression.
e. Enter the Means and %CVs from your Between Day Precision
experiments.
i.
If using EP Evaluator create the report by selecting the Print Preview
button. (See Appendix 6 for an example EP report).
ii.
The SMILE Accuracy Pack and EP Evaluator calculate a Sigma
metric for each Medical Decision point.
iii.
Sigma metric=(TEa-Systematic Error)/Random Error
f. Evaluate acceptability at each MDP using the table below:
If
The Sigma
metric less
than 2.0
The Sigma
metric is
between 2.03.0
If the Sigma
metric is
between 3.04.0
If the Sigma
metric is
between 4.06.0
If the Sigma
metric is >6.0
Then
 The method has unacceptable performance and does not meet
your requirement for quality, even when the method is working
properly.
 It is not acceptable for routine operation.
 The method has marginal performance and provides the
necessary quality when everything is working correctly.
 This method will require:
o 4-8 controls per run
o well-trained operators
o reduced rotation of personnel
o more aggressive preventive maintenance
o careful monitoring of patient test results
o continual efforts to improve method performance
 The method has fair performance and meets your requirement for
quality and can be well-managed in routine operation.
 This method will require 2 control measurements per run using
standard Westgard QC rules.
 The method has good performance and is clearly acceptable and
can be well-managed in routine operation with only 2 control
measurements per run using standard Westgard QC rules.

The method has Six Sigma performance and is even easier to
manage and control.
IV. References
A. GCLP Workshop and Workbook18-20 May 2008, Verification of Performance
Specifications, pages 1-33.
B. Clinical and Laboratory Standards Institute (CLSI). User Verification of
Performance for Precision and Trueness: Approved Guideline-Second Edition.
CLSI document EP15-A2 (ISBN 1-56238-574-7). Clinical and Laboratory
Standards Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania
19098-1898 USA, 2005.
C. NCCLS. (Currently CLSI) Evaluation of Precision Performance of Quantitative
Measurement Methods; Approved Guideline—Second Edition. NCCLS document
EP5-A2 (ISBN 1-56238-542-9). NCCLS, 940 West Valley Road, Suite 1400,
Wayne, Pennsylvania 19087-1898 USA, 2004.
106731738
6 of 7
30-Apr-09
D. Clinical and Laboratory Standards Institute (CLSI). Method Comparison and Bias
Estimation Using Patient Samples: approved Guidelines- Second Edition. CLSI
document EP9-A2 (ISBN 1-56238-472-4). Clinical and Laboratory Standards
Institutes, 940 West Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898
USA, 2005.
E. Clinical and Laboratory Standards Institute (CLSI).Preliminary Evaluation of
Quantitative Clinical Laboratory Measurement Procedure: Approved Guidline –
Third Edition. CLSI document EP10-A3 (ISBN 1-56238-622-0). ). Clinical and
Laboratory Standards Institutes, 940 West Valley Road, Suite 100, Wayne,
Pennsylvania 19098-1898 USA, 2005.
F. Clinical and Laboratory Standards Institute (CLSI).Estimation of Total analytical
Error for Clinical Laboratory Methods. Approved 2003. CLSI document EP21-A
(ISBN 1-56238-502-xX. Clinical and Laboratory Standards Institutes, 940 West
Valley Road, Suite 100, Wayne, Pennsylvania 19098-1898 USA, 2005.
G. EP Evaluator Release 8, David G. Rhoads Associates Inc., www.dgrhoads.com.
H. James O. Westgard, Online Validation Training, Westgard QC, Inc.
www.westgard.com, Sections 11-Determining Bias,12- Estimating Trueness, and
13- Judging Method Acceptability.
106731738
7 of 7
30-Apr-09