SMILE
Johns Hopkins University
Baltimore, MD USA
Qualitative Validation Overview
Author: Validation Committee
Review History
Document Number:
Equ35-02
Effective (or Post)
Date:
4 March 2011
Date of last review:
NA
Reviewed by:
Heidi Hanes
SMILE Comments: This document is provided as an example only. It must be revised to accurately reflect your lab’s specific
processes and/or specific protocol requirements. Users are directed to countercheck facts when considering their use in other
applications. If you have any questions contact SMILE.
Qualitative Validation Overview
Document Number
1000
Effective Date
17 February 2010
Page
1 of 1
Supersedes
Version 1.1
Subject
Guidelines for validation of a qualitative method
Author(s)
Approved by
Name, Title
Date
Jo Shim
January 2010
Name, Title
Date
SMILE Validation Committee
Date of last review:
Review History
Reviewed by:
Version # [0.0]
Revision
History
308872332
Revision Date
[dd/mm/yy]
1.1
3/31/10
1.2
02/15/2011
January 2010
15 February 2011
Anne Sholander
Description (notes)
Update to accuracy requirements.
Updated terminology in Diagnostic Sensitivity and Specificity section
Page 1 of 4
SMILE
Johns Hopkins University
Baltimore, MD USA
Qualitative Validation Overview
Validation of a qualitative laboratory test consists of an established set of required experiments.
This overview is intended to be applicable to both CLIA Waived and non-waived methods. A Waived test
is one that has been granted Waived status by the US Clinical Laboratory Improvement Act of 1988 (CLIA
’88). As indicated in the document, additional validation testing may be required for non-FDA Approved
test methods. After completing all of the validation experiments, raw data and results should be compiled
and filed in an organized manner. All validation records should be retained indefinitely. A validation
summary should be prepared that contains the signature of the Laboratory Director, indicating the
validation has been reviewed and approved.
The following are the required components of validation:
1. Precision is reproducibility – the agreement of the measurement of replicate runs of the same
sample. It is the process of determining the range of random error. The precision is measured in
terms of coefficient of variation (CV). Precision testing is required for qualitative tests that meet
the following criteria:
i.
Qualitative results are derived from a quantitative value such as an OD.
ii.
Manufacturer’s package insert describes precision specifications for the assay.
For qualitative tests that do require precision testing follow the guidelines listed below:
a. Within run and between run reproducibility will be determined by running the negative
control and positive control as follows:
i. For within run, at least 20 replicates of negative control and at least 20 replicates
positive control will be tested in one run.
ii. For between run reproducibility, both negative and positive control will be tested
at least once per day but not more than 5 times per day to obtain a total of 20
replicates each.
b. Acceptability criteria: The results obtained from the negative and positive controls will be
used to calculate the CV and compared to the manufacturer’s claims for reproducibility.
The laboratory CV should be less than or equal to the manufacturer’s stated CV. In the
event that an assay does not perform as expected, the Laboratory Director will determine
acceptability.
2. Accuracy is the true value of a substance being measured. Verification of accuracy is the process of
determining that the test system is producing true, valid results.
a. Determine your comparison or reference method
i.
The comparison method must be previously validated.
ii. The comparison method must be currently performing successfully on EQA.
308872332
Page 2 of 4
SMILE
Johns Hopkins University
Baltimore, MD USA
iii. Comparison to a method currently in use in the laboratory is preferred if this
method meets the above criteria.
iv. Samples with known values, such as proficiency testing samples or commercial
standards, may be used as the reference method.
v. Examples of possible reference methods for qualitative tests include in-house or
external quantitative methods, use of a reference lab, correlation with another lab
using a previously validated method, use of EQA samples or other commercially
prepared reference material with known values.
b. Sample Criteria
i. A minimum of 10 samples for each expected result. For example, if a test
method gives results of “Positive/Negative”, the accuracy study must include 10
known positives and 10 known negatives.
c.
Testing: Two levels of quality control must be run each day that testing is performed, not
including controls internal to the kit cartridge/testing device.
d. Acceptability criteria:
The performance of qualitative tests is most commonly described in terms of sensitivity
and specificity. The table below is a contingency table that compares the results of a
qualitative test with the outcome of the diagnostic accuracy criteria. The entry in each
cell of the table represents the number of specimens corresponding to the labels in the
margins.
Diagnostic Sensitivity and Specificity
Method being
Validated
(Results from Comparison Study)
Total
Positive
Negative
Positive
# true positive (TP)
# false positive (FP)
TP+FP
Negative
# false negative (FN)
# true negative (TN)
FN+TN
Total
TP+FN
FP+TN
N
i.
Calculate the estimated Diagnostic Sensitivity(True positive rate) = 100 x
[TP/(TP+FN)]
ii.
Calculate the estimated Diagnostic Specificity(True negative rate) = 100 x
[TN/(FP+TN)]
iii.
Calculate the percent Positive Agreement (Positive Predictive Value)
=100 x TP/(TP+FP)
308872332
Page 3 of 4
SMILE
Johns Hopkins University
Baltimore, MD USA
iv.
Calculate the percent Negative Agreement (Negative Predictive Value)
=100 x TN/(TN+FN)
v.
Compare the results calculated above with the manufacturer’s stated claims for
Sensitivity, Specificity and Agreement found in the test kit package insert.
vi.
Results must be equal to, or greater than, the manufacturer’s claims for the method to
be considered accurate.
3. Linearity with AMR and CRR – Not applicable for qualitative tests.
4. Analytical Sensitivity is the lowest concentration of an analyte that can be measured. For an FDA
approved, unmodified method, the manufacturer’s stated sensitivity (cut-off value) will be used.
5. Analytical Specificity is the determination of the effect of interfering substances. For an FDA
approved, unmodified method, the manufacturer’s stated specificity will be used.
6. Reference Ranges – Can be determined by the laboratory with laboratory director approval.
Verification of manufacturer’s stated reference range is not required.
7. Summary and Approval. See SMILE Qualitative Testing Validation Summary template.
8. References
a. GCLP Workshop and Workbook18-20 May 2008, Verification of Performance Specifications,
pages 1-33.
b. Clinical and Laboratory Standards Institutes (CLSI), User Protocol for Evaluation of Qualitative
Test Performance; Approved Guideline--Second Edition. CLSI document EP12-A2 (ISBN 156238-654-9). Clinical and Laboratory Standards Institute, 940 West Valley Road, Suite 1400,
Wayne, PA 19087-1898 USA, 2008.
c. EP Evaluator Release 8, David G. Rhoads Associates Inc., www.dgrhoads.com.
d. James O. Westgard, Online Validation Training, Westgard QC, Inc. www.westgard.com,
Sections 11-Determining Bias,12- Estimating Trueness, and 13- Judging Method Acceptability.
308872332
Page 4 of 4