CAP Accreditation Requirements for Validation of
Laboratory Tests
AMT 75th Educational Program and National Meeting
Stephen J. Sarewitz, MD, FCAP
July 9, 2013
www.cap.org
v. 1.0
Biography
Stephen J. Sarewitz, MD, FCAP
• Board of Governors, College of American
Pathologists
• Member, Council on Accreditation, CAP
• Former Chair, Checklist Committee, CAP Laboratory
Accreditation Program
• Former Chair, Area Committee on General
Laboratory Practices, CLSI
© 2013 College of American Pathologists. All rights reserved.
3
Objectives
• Perform appropriate validation/verification studies
in accordance with CAP requirements prior to
implementing new tests, including FDA
cleared/approved tests and laboratory-developed
tests.
• Understand validation requirements for FDA
cleared/approved tests modified by the laboratory.
• Understand the current status of validation
requirements for laboratory-developed tests.
© 2013 College of American Pathologists. All rights reserved.
4
Topics
What this discussion covers:
• Validation/verification of the test method itself
What it doesn’t cover: other aspects of implementing new tests,
such as:
•Calibration or QC validation
•Reagent handling
•Documentation of test procedure
•Training of personnel and competency assessment
•Proficiency testing or alternative assessment
•Sample handling
© 2013 College of American Pathologists. All rights reserved.
5
Definitions Used in CAP Laboratory
Accreditation Program
• Validation: provision of objective evidence through
a defined process that a test performs as intended
[CLSI]
• Verification: an abbreviated process to
demonstrate that a test performs in substantial
compliance to previously established claims
• Sometimes the term “validation” is used to cover
both of the above—can be source of confusion
© 2013 College of American Pathologists. All rights reserved.
6
Types of Validation
•
Analytic validation: “…the process of assessing the assay and its
performance characteristics and determining the optimal conditions
that will generate a reliable, reproducible, and accurate…assay for
the intended application.” --Required by CLIA and CAP
Accreditation Checklists
•
Clinical validation: determining “the ability of a test to diagnose or
predict risk for a particular health condition, measured by clinical (or
diagnostic) sensitivity…, specificity, and predictive values.” --Not
addressed by CLIA; in CAP only in Molecular Pathology and
Microbiology (in Molecular section) Checklists (for nonwaived tests).
•
Clinical utility: “determining the net balance of health-related risks
and benefits associated with the test’s use in practice [to] patient,
family, healthcare organizations, and society.” --Subjective, complex,
involves areas outside of laboratory diagnosis. Not in CLIA.
Mentioned in CAP (MOL); no requirements.
© 2013 College of American Pathologists. All rights reserved.
7
Analytic Validation
Analytic Validation
• Required by CLIA and CAP for non-waived tests (No CLIA reqs.
for waived tests; CAP has very limited req. for waived tests)
• CLIA and CAP requirements similar
• Distinction between FDA-cleared/approved tests, and others:
o For tests cleared or approved by FDA—verification
required: ie, laboratory must show that it can obtain
performance specifications comparable to those
established by the manufacturer
o Requirements the same for both moderate and high
complexity tests approved/cleared by FDA
o For tests not approved/cleared by FDA: laboratory must
establish the performance specifications (= validating the
test)
© 2013 College of American Pathologists. All rights reserved.
9
CAP Definition of LDT
A laboratory-developed test (LDT) is defined as follows
- A test used in patient management that has all of
the following characteristics:
• The test is performed by the clinical laboratory in
which the test was developed
• The test is neither FDA-cleared nor FDA-approved,
or is an FDA-cleared/approved test modified by the
laboratory (sample types or the use of collection
devices not listed in manufacturer instructions
constitute modifications, by this definition)
• The test was first used for clinical testing after April
23, 2003
© 2013 College of American Pathologists. All rights reserved.
10
Analytic Validation, continued
• For FDA cleared/approved tests that have been modified by
the laboratory, lab must validate the modification(s) or show
that they don’t degrade test performance
o Sample types and collection devices not listed in mfg
labeling are considered modifications of the test (ex. body
fluids)
o Using a test in a patient population excluded in mfg
labeling is a modification (ex. glucometer in critically ill
patients)
o Adjusting the reference range is not considered a
modification
o By definition, all modified FDA cleared/approved tests, and
LDT’s*, are high complexity
*Unless specifically categorized by FDA/CLIA as waived or PPM
© 2013 College of American Pathologists. All rights reserved.
11
CAP Performance Specifications for
Analytic Validation
• Analytic accuracy (= bias) : systematic error
• Precision : random error
• Analytic sensitivity (limit of detection, LOD)*
• Analytic specificity (interferences)
• Reportable range
• Reference range
• In COM checklist (moved from GEN for 2011 edition)
*Literally means ability of assay to detect a change in concentration of the analyte; assays
with steeper calibration curves are more sensitive in this sense. However this is not the
meaning of the term for purposes of validation/verification—LOD is what is meant.
© 2013 College of American Pathologists. All rights reserved.
12
CAP Requirements…in COM Checklist
(in GEN before 2011)
Accuracy & Precision
(COM.40300, ph II)
Analytic sensitivity (LOD)
(COM.40400, ph II)
Analytic specificity
(interferences) (COM.40500, ph
II)
Reportable range
(AMR)(COM.40600, ph II)
Reference range (COM.50000,
ph II)
FDA approved/cleared
LDTs & modified FDA tests
Verify mfger’s results
Establish (= validation)
Verify: manufacturer or
literature documentation
OK
Reference literature or
manufacturer
documentation
Verify*
Establish
Verify **
Establish**
Establish; studies by
manufacturer or in literature
OK
Establish*
*Reportable range ( = AMR, generally) is the range of values that the method can directly
measure without dilution or concentration, while meeting specifications for accuracy & precision
--Details on establishing & validating AMR are in other checklists (ex. CHM, HEM, MOL)
**In some cases labs may use manufacturer or literature data when verification/establishment of
a reference range is not practical: ex. pediatric blood cell count / index parameters;
therapeutic drug levels.
© 2013 College of American Pathologists. All rights reserved.
© 2013 College of American Pathologists. All rights reserved.
13
Analytic Validation – Some New Provisions in July
2013 Edition of COM Checklist
• Validation studies must be performed in location where
patient testing will be done
o If instrument is moved, lab must verify that performance
specifications (eg, accuracy, precision, etc.) were not
affected by the move or new environment (change in
ambient temp, humidity). (Intro.)
• Multiple instruments of same make & model: each must be
validated separately (Intro.)
o Interpretation:
− accuracy could be verified for 2nd instrument by
comparison study with 1st instrument (15-20 samples)
− No separate reference range study needed for 2nd
instr., assuming comparison study showed absence of
significant bias
© 2013 College of American Pathologists. All rights reserved.
14
Analytic Validation – New Provisions in COM, continued
• Reports for LDT’s contain a description of the method,
statement that the test was developed by the laboratory, and
appropriate performance characteristics (COM.40630, ph I)
(similar item was in MOL & MIC; added to COM 2013)
o Not a regulatory requirement, unlike FDA-mandated
statement on reports of tests using ASR’s (see MOL, ANP,
CYG, MIC)
• Mfg instructions must be followed for FDA cleared/approved
tests, OR lab must document validation of any modifications
(COM.10600, ph II; moved from Procedure Manual section of
COM)
• For analytic interferences (COM.40500, II), lab has an
appropriate plan of action when they are present
© 2013 College of American Pathologists. All rights reserved.
15
CLIA --- Differences from CAP
– Analytic Validation
• For FDA approved/cleared tests:
o No provisions on analytic sensitivity or analytic
specificity in CLIA
• No requirements for waived tests
• No CLIA requirement for establishing or verifying
performance specifications for any test system used
prior to April 24, 2003.
o CAP does NOT have any grandfather date; all
non-waived tests subject to CAP analytic
validation / verification requirements
© 2013 College of American Pathologists. All rights reserved.
16
Waived Tests
•
No CLIA requirements
•
CAP: Verify reference range if
practical
POC.04525 (ph II): Reference
intervals…are established or
verified….
NOTE: If a…reference study is
not…practical….then the
POCT site should carefully
evaluate the use of published
data for its own reference
ranges…
If waived method is modified,
then all CAP/CLIA validation
requirements apply
© 2013 College of American Pathologists. All rights reserved.
17
Validation/Verification Specifics
•
Neither CLIA nor CAP list specifics of validation/verification—numbers
of samples, limits of acceptability, number of days required for
studies, statistical analysis
o Variability across laboratories in terms of environment, patient
populations, use of tests – specifics thus need to be determined by
laboratory director
o Validation = more extensive study than verification
•
Exceptions:
o Number of samples—HER2, estrogen receptor, progesterone receptor
tests in breast carcinoma tissue
o Concordance acceptability threshold – ER/PgR receptor in breast
carcinoma
o Number of data points for AMR (3 required)
- No requirement for linearity studies per se….but AMR is “linearity
lite”
© 2013 College of American Pathologists. All rights reserved.
18
Some Guidelines
Validation/Verification Guidelines
FDA cleared/approved LDT
Accuracy (bias)**
At least 40 samples across
AMR; could be > 100
Run study for 20 days
Reportable range (= linear
range) (=AMR)*
20-40 samples across
AMR
2-3 samples at clinical
decision points run
daily for 5 days
3 points near low end,
midpoint, and high end
Reference range
20 samples
40-60 samples; 120 or
more ideal
Precision (random error)
Same
[See CLSI Guidelines (references at end of presentation)]
*CAP checklist requirement [ex. CHM.13600 (II0]. AMR must be verified at least every 6 mos.
**Recommend duplicate measurements of each sample (not in consecutive order) to detect
errors, sample mix-ups, etc.
© 2013 College of American Pathologists. All rights reserved.
20
Validation Guidelines---Using Statistics
• Westgard: “Statistics are just tools for combining many
experimental results…and summarizing all that data in just a
few numbers…the statistics are used to make reliable estimates
of the errors from the data….”
• Precision (= random error) – actually, we measure imprecision
o If determined within a run = repeatability; across multiple
runs across multiple days = reproducibility. The latter is most
reflective of actual lab practice.
o Calculate standard deviation (SD) / coefficient of variation
(CV)
o Use F test to see if variance (=SD^2) of test method is
statistically different from old method, or claim of
manufacturer
© 2013 College of American Pathologists. All rights reserved.
21
Validation Guidelines, continued
• Accuracy / Bias (= systematic error):
o Run comparison of methods study (test method, vs.
reference method / lab’s previous method /
manufacturer’s results, etc)
− Line of best fit* (visually or using statistics program) gives
linear regression equation Y = a + bX
− Calculate correlation coefficient “r”
– If r is high (>=.99), use regression line to find bias at
analyte concentrations corresponding to critical
decision points (ex. glucose: 126 mg/dL)
*Line that minimizes the square of the distances of the data points to the line
© 2013 College of American Pathologists. All rights reserved.
22
Validation Guidelines, cont.
– If r < .975, regression equation not reliable; use
paired t-test to determine if a bias is present at the
mean of the data
− Analytes with wide range (cholesterol, glucose,
enzymes, etc) tend to have high r in comparison
studies; analytes with narrow range (electrolytes) tend
to have low r
− r should not be used to determine the acceptability of a
new method
© 2013 College of American Pathologists. All rights reserved.
23
250
Y: new method mg/dL
200
Y = -7 + 1.18 * X
150
r = .99
100
Comp.
method
50
50
0
0
50
100
150
200
250
New
method Bias
52
+2
100
111 +11
150
170 +20
X: comparative method mg/dL
High r……but significant proportional bias
© 2013 College of American Pathologists. All rights reserved.
24
Validation Guidelines, continued
•
For AMR, 5 points is closer to ideal but 3 is sufficient (CHM.13600, ph II)
o Separate linearity study not required: AMR is “linearity lite”
•
Reference range classically = central 95% of values for “normal” population. If
20 samples are run to verify mfg range, if <= 2 lie outside the mfg limits, then
the mfg reference range is considered verified
•
Decision limits For some tests, reference range = a cut-off point or decision
limit: ex., troponin, cholesterol. Studies for these tests would not be samples
from a reference population, but rather studies of samples with levels on either
side of the decision point, using clinical data or a comparative method.
•
Transference by calculation The lab may be able to”transfer” reference range
from a previous method or a reference (or mfger) method (Westgard; CLSI
guideline C28-A3).
o If new method = Y and old method = X, Y(upper) = a + bX(upper), etc.
o Limit transference to one change of methods, to prevent potentially
amplifying errors by multiple transfer procedures (Westgard).
o Perform 20-sample verification study if there is doubt re the reliability of the
transfer
© 2013 College of American Pathologists. All rights reserved.
25
Defining Allowable Total Error…
Accuracy & Precision
• FDA cleared/approved tests: = manufacturer specifications
• For LDT’s:
o Decision by Laboratory Director
o Ideally, laboratory defines allowable total error in advance
for the intended application of the test
• Possible ways to establish allowable total error include:
o CLIA acceptability limits for variability in proficiency testing
results (403.909 – 403.959)
− Ex. AST = +/- 20%, glucose +/- 6 mg/dL or 10% (greater),
Na +/- 4 mmol/L, HDL-chol +/- 30%
o Range of actual PT results for the analyte/method
o Clinical decision points—literature, input from clinicians
© 2013 College of American Pathologists. All rights reserved.
26
Allowable Error vs. Observed Error:
Using Statistics to Help Analyze Accuracy &
Precision (Systematic & Random Error)
Observed total error should be less than allowable
total error
Observed (actual) total error can be defined as: bias
+ [3 x (SD of method)] (Westgard)
© 2013 College of American Pathologists. All rights reserved.
27
Example – New Method for Serum Aspartate
Aminotransferase (AST)
• Total allowable error:
o Lab determines it should be <= CLIA proficiency testing limit
for AST, which is +/- 20% of target value
• Lab’s new AST method:
o Bias is + 2 U/L compared to reference method
o Imprecision = standard deviation = +/- 3 U/L
• Consider target value of 50 U/L:
o Total allowable error = +/- 20% x 50 = 10 U/L
o Observed total error = bias + (3 x SD) = 2 U/L + (3 x 3 U/L) =
11 U/L
• Observed total error of 11 exceeds total allowable error of 10
so test method is not acceptable
(Note: in this calculation, the absolute value of bias and SD are used)
© 2013 College of American Pathologists. All rights reserved.
28
AMR and Calibration Reqs. in Coagulation (HEM
chklst): Apply Only to Directly Measured Analytes
• Method must be calibrated (HEM.38008, ph II)
• There must be criteria for calibration verification
(HEM.38006, ph II):
o Changes of reagent lots for chemically / physically
active components, unless lab can show that lot
changer does not affect patient test results
o Unacceptable QC (not correctable by other means)
o Major maintenance or change of critical instrument
component
o When recommended by manufacturer
o At least every 6 mos.
• System must be recalibrated if cal ver fails to meet lab’s
criteria (HEM.38007, ph II)
© 2013 College of American Pathologists. All rights reserved.
29
AMR and Calibration in Coagulation (HEM Chklst):
Apply Only to Directly Measured Analytes, continued
• AMR is validated at low, mid- and high range (HEM.38009, ph II)
o If materials used for cal/cal ver include the above values,
then cal/cal ver also covers AMR validation
o Revalidate every 6 mos, or following change in major instr.
components or lots of critical reagents unless lab can show
that change of lots does not affect results
o AMR validation not required for methods that measure an
analyte quantitatively or semiquantitatively and report a
qualitative value based on a threshold
• For result greater or less than AMR, numeric result not reported
unless sample is diluted, concentrated or processed by a mixing
procedure to bring result into the AMR (HEM.38010, ph II)
© 2013 College of American Pathologists. All rights reserved.
30
Coagulation: Directly Measured Analytes
• Often immunoassays (EIA, immunoturbidity,
chromogenic)
o Ex., protein C ag, free & total protein S ag, von
Willebrand factor ag, LWM heparin, quant. ddimer
• May include tests reported in % if traceable to a
standard
• Whether or not test involves a reference/standard
curve not relevant
• Does not apply to analytes measured by functional
assay with results convertible to concentration by a
formula
© 2013 College of American Pathologists. All rights reserved.
31
Other Reqs. in COM
Other Requirements in COM/GEN
1. Lab director or designee qualified as director must approve
validation studies prior to reporting patient results (COM.40000, ph I).
2. Lab must list all LDT’s implemented in past 2 years for review by
inspector (COM.40200, ph I).
3. Lab’s test methods, including performance specifications and
supporting validation/verification data, must be available to clients
and inspection team (COM.40700, ph II). Lab may require clients to
treat information as confidential.
4. If method changes such that results may be significantly altered, the
change must be explained to clients (COM.40800, ph II).
5. Lab evaluates reference range and takes corrective action as
appropriate (COM.50100, ph II).
a. Change of method or patient population
6. GEN .20377 (II) Retain validation records while method is in use and
at least 2 years afterwards
© 2013 College of American Pathologists. All rights reserved.
33
Other Requirements in COM, continued
7. Intermittent testing (ex. seasonal testing for influenza):
o A test is considered to be taken out of production if 1)
patient testing not offered; and 2) PT / alternative
assessment is suspended
o When suspended test is put back on-line:
1. PT or alternative assessment must be performed within
30 days prior to re-starting patient testing
2. Performance specifications are verified as applicable
within 30 days prior to restarting patient testing
3. Competency is assessed for analysts within 12 mos. Prior
to restarting testing
(COM.40100, ph II)
(Note: for tests for which PT is required, if a PT challenge is not offered in the 30 days
prior to restarting testing, lab may do alternative assessment)
© 2013 College of American Pathologists. All rights reserved.
34
Validation Items in Other Checklists –
Examples (Full List at End of Presentation)
•
MOL.30900 (II) Validation studies include samples for each reportable
genotype
•
NOTE: Assays for genetic disorders with a limited number of possible
genotypes (e.g. hereditary hemochromatosis) should confirm the ability of the
assay to detect these genotypes. Assays for genetic disorders with
considerable allelic heterogeneity and/or significant numbers of private
mutations (e.g. cystic fibrosis or hereditary nonpolyposis colorectal cancer)
should confirm the accuracy of the methodology used to provide a high
degree of assurance that the assay will detect targeted genotypes. Various
sample types may affect…analytical performance....laboratories may need
to establish sample-specific analytical and clinical performance
characteristics. The number of samples depends on the intended use of the
test. …laboratories are encouraged to review the cited references for
guidance and provided confidence intervals to estimated performance
characteristics.
•
MOL.31015 (II) Validation studies include expected specimen types (frozen,
fresh, paraffin-embedded, blood, etc.)
© 2013 College of American Pathologists. All rights reserved.
35
Validation Items in Other Checklists –
Revision for 2013
In subsections “FDA cleared/approved non-amplification
methods” and “FDA cleared/approved target & signal
amplification methods & sequencing:
• MIC.64770 and 64815 (II): If the laboratory tests sample types
or uses collection devices other than those listed in the
package insert, the laboratory performs validation studies to
document adequate performance of the test.
o NOTE: Results from tests performed on sample types not
listed in the pkg. insert may be reported without complete
validation only if…1) validation studies are ongoing but
have not been completed; 2) the sample type is
encountered rarely, precluding an adequate number for
validation studies. Under these circumstances, the test
report must include a disclaimer stating that the sample
type has not been validated.
© 2013 College of American Pathologists. All rights reserved.
36
Complications – Qualitative Tests
•
Performance specifications must be established/verified as
applicable. Ex. reportable range may not be applicable. For tests
with a cut-off, precision is considered to be variation in the test result
near the cut-off point.
•
Can use kappa statistic to determine the degree to which
agreement between 2 comparative methods is result of chance:
kappa = [P(o) – P(e)]/[1-P(e)] where P(e) = agreement by chance and P(o) = observed agreement
P(e) = [%test1(pos) x %test2(pos)] + [%test1(neg) x %test2(neg)]
kappa values: 0 = chance agreement; 1 = perfect agreement
.41 - .60 = moderate agreement
.61 - .80 = substantial agreement
See Viera AJ, Garrett JM. Fam Med 2005;37(5):360-363.
© 2013 College of American Pathologists. All rights reserved.
37
Complications – “Gold Standard” Problem
• New method is considered superior in analytic sensitivity
to old established method—how to handle discordant
results in method comparison study.
o Troponin replacing CK-MB in dx of myocardial
infarction
o Molecular tests for microorganisms replacing culture
or immunologic tests—Chlamydia, Trichomonas
• Approaches:
o Use clinical data (chart review)
− Ex. clinical & EKG evidence of MI
o Use reference method if available—must test all
samples, not just discordant ones
© 2013 College of American Pathologists. All rights reserved.
38
Complications -- Body Fluids -- New Checklist
Item 2013
•
COM.40620 (ph II)
o Testing of body fluid specimens using methods intended for other
specimen types (e.g. blood….) has been validated…for
accuracy, precision, analytic sensitivity, analytic interferences,
and reportable range.
o NOTE: applies to B.F. testing offered as a routine, orderable test.:
− Written procedure required
− Lab director determines extent of validation studies
− Blood performance specs. may be used if matrix effects can
be excluded (by studies using mixtures of samples, spiking,
dilution)
− Reference range must be reported, unless result is reported as
comparison to blood level (ex. pleural fluid protein, LD,
cholesterol; ascitic amylase) – literature data OK
− Alternative assessment or PT required
© 2013 College of American Pathologists. All rights reserved.
39
New Body Fluid Validation Requirement, continued
• NOTE: for clinically unique tests, validation may not
be practical
o Add comment: ex., “The reference range and
other performance specifications have not been
established for this body fluid. The test result must
be integrated into the clinical context for
interpretation.”
© 2013 College of American Pathologists. All rights reserved.
40
Migration of Analytic Validation Requirements
from GEN to COM
Requirement
GEN no. of
citations
COM no. of
citations
Approval of validation studies by director
Accuracy/precision
Analytic sensitivity
Interferences
Reportable range
Data available to clients
94
18
9
9
16
1
176
65
29
24
33
23
Communicate significant method changes
Establish/verify reference intervals
Re-evaluate reference intervals as necessary
Total
0
14
3
164
2
30
13
395
--First seven months of 2011: GEN contains requirements
--First seven months of 2012: COM contains requirements
--395/164 = 140% increase in citations at on-site inspections
© 2013 College of American Pathologists. All rights reserved.
41
Clinical Validation
CAP Clinical Validation Requirements—Limited to MOL / MIC*
New text for 2013 in MOL, Intro. To Validation section:
•…clinical validity, which includes .…clinical sensitivity, clinical
specificity, positive and negative predictive values in defined
populations or likelihood ratios, and clinical utility should … be
considered, although individual laboratories may not be able to
assess these parameters within their own patient population,
especially for rare diseases. However, patients without disease
can typically be tested to assess clinical specificity. If clinical
validity cannot be established within a laboratory, it is
appropriate to cite scientific literature that established clinical
sensitivity and specificity.
(Clinical utility: determining the net balance of health-related risks and benefits
associated with the test’s use in practice [to] patient, family, healthcare
organizations, and society.)
*except for clinical claims
© 2013 College of American Pathologists. All rights reserved.
43
Clinical Validation Requirements in CAP, cont.
•
MOL.31590 (II) Clinical performance characteristics of each assay
are documented
o Diagnostic sensitivity & specificity, positive & negative predictive values,
likelihood ratios*, clinical utility*
o May depend on clinical setting, genotype/phenotype associations when
these vary with particular mutations or polymorphisms, &
genetic/environmental/epigenetic factors affecting clinical expression of
a genetic alteration
•
For FDA cleared/approved tests, literature or mfg data can be used
•
For laboratory-developed tests (LDT’s)—lab should perform studies in-house
o Use clinical, biopsy & radiologic findings, other lab results*
o Literature may be used for very rare conditions, or very common
conditions for which clinical validity is well-established in literature
o Establishing clinical validity may require extended studies…that go
beyond the purview…of the individual laboratory
(*New text, 2013)
© 2013 College of American Pathologists. All rights reserved.
44
CAP Clinical Validation Requirements, continued
•
COM.40640, for 2013 (was in MOL) (II): All clinical claims made by the
laboratory about an LDT are validated.
•
NOTE: Clinical claims may include statements about a test's
diagnostic sensitivity and specificity, ability to predict the risk of a
disease or condition, clinical usefulness, or cost-effectiveness. Clinical
claims may be found on the test report or in other information
distributed by the laboratory (websites, test catalogues, newsletters,
memoranda, advertisements, etc.). Laboratories are not required to
make clinical claims about a test, but any claims made by the
laboratory must be validated. In general, the laboratory should
validate claims through a clinical study, but for rare conditions or
well-accepted uses of a test, reference to published peer-reviewed
literature is acceptable.
© 2013 College of American Pathologists. All rights reserved.
45
CAP Clinical Validation Requirements in MIC
•
Introduction to subsection on LDTs in Molecular section of MIC:
•
Diagnostic sensitivity and specificity must be determined relative to
some "gold standard" (eg, biopsy findings, clinical findings, etc.). The
sensitivity of an assay equals [TP/(TP+FN)] X 100 and the specificity of
an assay equals [TN/(TN+FP)] X 100. (TP=true positive, TN=true
negative, FN=false negative, FP=false positive.) Determinations of
sensitivity and specificity should be done in a "blinded" fashion (ie,
without prior knowledge of the patient's disease status). For some
infections, it may not be possible to identify large numbers of positives
(ie, patients with the infection) to establish the diagnostic sensitivity of
the assay. In such instances, the laboratory should procure as many
positive cases as is reasonably possible for method validation and in
addition cite any publications that have investigated the diagnostic
sensitivity of the assay.
© 2013 College of American Pathologists. All rights reserved.
46
Further Issues with Clinical Validation
1. Why restrict clinical validation requirements to MOL and MIC?
a. The importance of clinical validation is mainly (but not
exclusively) in molecular/genetic LDT’s.
b. If applied to all lab areas, clinical validation requirements would
capture LDT’s used for—
i. Flow cytometry markers
ii. Coagulation factor assays
iii. Manual microbiology cultures
iv. Immunohistochemistry
v. Conventional pap smear
2. Future revisions in clinical validation requirements not clear --pending FDA decisions on regulation of LDT’s
a. Requirements will most likely be risk-based
b. CAP has proposed partnership with FDA
© 2013 College of American Pathologists. All rights reserved.
47
Future of LDT Validation
Potential Regulatory Change for LDT’s
• 7/19/2010: FDA announces intent to regulate
LDT’s
• FDA has asserted that laboratory tests are
medical devices, subject to its jurisdiction
• Medical Device Amendments (1976) to Food,
Drug & Cosmetic Act (1938) define a
medical device as:
© 2013 College of American Pathologists. All rights reserved.
49
Medical Device Amendments (1976)
• “instrument, apparatus, implement, machine,
contrivance, implant, in vitro reagent, or other
similar or related article….intended for use in the
diagnosis of disease or other conditions, or in the
cure, mitigation, treatment or prevention of disease
in man…”
• “Enforcement discretion”: FDA may choose not to
enforce applicable regulations
© 2013 College of American Pathologists. All rights reserved.
50
FDA’s Rationale for End of Enforcement Discretion
• Increase in type & number of LDT’s
• Many developed by commercial labs or biotech companies
• More like commercially-developed tests rather than tests used
in limited way by local laboratory
• LDT viewed as “route to market” driving venture capital
funding
• Some LDT’s aggressively marketed to clinicians or directly to
consumers
• Public needs assurances that diagnostics are sound & reliable
• Uneven playing field for IVD manufacturers because
distributed “test kits” had to go through FDA review prior to
marketing, while LDTs did not.
• No post-market reporting or recall requirements apply to LDTs.
© 2013 College of American Pathologists. All rights reserved.
51
Planned FDA LDT Regulation
• 3 guidance documents anticipated;
o Overall regulatory framework
o Registry Requirements
o Description of the Synergies Between CLIA
Regulations and FDA Quality System Regulation
• Initial timeline:
o proposed regulations mid – end 2013 with period
for comments
o final regulations 2014 - 2015
• Current status: delayed for uncertain period
© 2013 College of American Pathologists. All rights reserved.
52
CAP Approach
• Meetings with FDA
• CAP proposes public-private partnership with CMS
(CLIA) & FDA with 3-tier risk based system of
accreditation/regulation...initially proposed 2009
o Covers analytic and clinical validation
o Applies to all LDT’s
© 2013 College of American Pathologists. All rights reserved.
53
CAP Proposed Risk Classification Principles for LDT’s
• Laboratory assigns risk; submits analytic/clinical
validation & other info to CAP
• LDT’s put on-line before 4/24/2003 are
grandfathered.
• CAP confirms risk level & informs lab
o Appeal process offered by CAP
© 2013 College of American Pathologists. All rights reserved.
54
Classification Principles for LDT’s, continued
Classification
Principles
Oversight
Low
Test used with other
clinical information; not
used alone to
determine Rx
-Lab notifies CAP
& CAP confirms
risk class
-Lab performs
validation
-Accreditor
inspects
Incorrect result unlikely
to cause serious harm
Lab makes no claims
that test alone
determines prognosis
or Rx
© 2013 College of American Pathologists. All rights reserved.
55
Classification Principles for LDT’s, continued
Classification
Principles
Oversight
Moderate
Test often (but not
necessarily) used to
predict prognosis or
determine if patient can
receive a specific Rx
-Lab performs
validation
-CAP performs desk
review of validation
materials & must
approve before test
goes on-line; FDA
informed
-Accreditor inspects
Lab may make claims
about clinical accuracy
Incorrect result may
cause serious harm
Test method is well
understood and
independently verifiable
© 2013 College of American Pathologists. All rights reserved.
56
Classification Principles for LDT’s, continued
Classification
Principles
Oversight
High
Test predicts risk,
progression or patient
eligibility for a specific Rx
AND uses proprietary
algorithm; result cannot
be tied to methods used
or interlab comparisons
cannot be performed
-Lab performs
validation & submits
to FDA
-FDA reviews
validation before
test goes on-line
-Accreditor inspects
Incorrect result may
cause significant harm
AND test method not well
understood or not
independently verifiable
© 2013 College of American Pathologists. All rights reserved.
57
Elements of LDT Validation
Defining the
Disorder/Test/Clinical
Scenario
Analytical Validity
• Encompasses the development stage of
an LDT and serves to establish a testing
procedure and identify the intended use
of the test.
• The test’s ability to accurately and
reliably measure the analyte of interest
in the clinical laboratory, and in
specimens representative of the
population of interest.
• The ability of a test to diagnose or
Clinical Validity
© 2013 College of American Pathologists. All rights reserved.
predict risk for a particular health
condition, measured by clinical (or
diagnostic) sensitivity, clinical (or
diagnostic) specificity, and predictive
values.
58
FDA Proposed Risk Classification
• Low risk: Little potential for injury, adjunctive tests
identifying one of many characteristics of a tissue or
cell with little clinical impact
Potential non-serious injury,
relatively easy to detect false result—tests where
multiple findings used to direct Rx; disease
monitoring tests
• High risk: Serious injury if incorrect, difficult to detect
false result—companion diagnostics, cancer
diagnosis, serious communicable diseases
© 2013 College of American Pathologists. All rights reserved.
59
Questions? accred@cap.org