Assay Quality Considerations
Christopher N. Greene, PhD
Newborn Screening and Molecular Biology Branch,
Division of Laboratory Sciences
NCEH, CDC
Tuesday, 9th July 2013
National Center for Environmental Health
U.S. Centers for Disease Control and Prevention
Molecular Testing
Multiple molecular assays have been developed for
screening procedures. These assays vary on their
complexity, the target analyte, as well as the platform used
for analysis.
Testing Parameters
Examples
Target Analyte
Genetic Mutations, Gene Dosage
Platform
PCR, qPCR, bead arrays, Fluorescent Probe, DNA
Sequencers
Complexity
Single- or multi-plex of single nucleotides,
Laboratory Developed Test, ASRs, FDA-approved,
Gene Sequencing, Next-Gen Sequencing
Molecular Testing
Molecular tests require verification or validation of assay
results. Level of evidence required is related to the type of
assay, i.e. laboratory developed test, analyte specific
reagents or FDA approved assays.
When Adopting a New Test
Required Evidence
Test Development - LDT
Accuracy, precision, reproducibility,
sensitivity, specificity, robustness
Test Validation - ASR
Clinical and analytic performance and
test limitations
Test Verification – FDA approved
Identify samples or reference
materials of known quantitative or
qualitative values to verify test
performance
Laboratory Regulatory and
Accreditation Guidelines
 US Food and Drug Administration (FDA):
 approves kits and reagents for use in clinical testing
 Clinical Laboratory Improvement Amendments (CLIA):
 Regulations passed by Congress1988 to establish quality
standards for all laboratory testing to ensure the accuracy,
reliability and timeliness of patient test results regardless of
where the test was performed
 College of American Pathologists (CAP):
 Molecular Pathology checklist
 State Specific Regulations
Testing Parameters
 Accuracy
 Limit of Detection
 Sensitivity
 Range
 Specificity
 Reproducibility
 Precision
 Robustness
Definitions
• Accuracy: Agreement between test result and the
“true” result
– Comparison of results between new method and a
reference method
– Results of new method on certified reference materials
• Sensitivity: Ability to obtain positive results in
concordance with a positive result of a reference
method – can the test detect a positive result
Definitions
• Specificity: Ability to obtain a negative
result for a qualitative test in concordance
with reference method
• Precision: Measure of random error in
quantitative tests, the closeness of
agreement usually expressed as standard
deviation of CV
Definitions
• Limit of Detection: Lowest amount of
analyte that is distinguishable from
background or a negative control
• Range: Range of test values expected for
the test population. For quantitative, high
and low range of target, for qualitative,
possible detectable alleles of the assay
Definitions
• Reproducibility: Equivalent to precision for
qualitative and semi-quantitative tests.
Can be used to indicate with-in or between
run agreement of test results
• Robustness: Test precision, given small,
deliberate changes in test conditions (e.g.
incubation times, temperature variation)
Pre-Validation
Choose and evaluate assay methodology
Determining analytic performance of an assay involves:
Reviewing professional guidelines and relevant literature
Stringent design/analysis of primers and probes
Quality and quantity of extracted nucleic acid
Appropriate platform for the test
Availability of controls or calibrators
Optimization of amplification and detection
Verifying Assay Specifications
• Demonstrate that
laboratory can obtain
specifications
comparable to
manufacturer
• Applies to unmodified
FDA-approved tests
– Accuracy
– Precision
– Reportable Results
• May require matrixappropriate materials
during verification
• Verify appropriate test
results for population
Assay Validation
Establish that the assay fulfills its intended use
with the appropriate sample type and population
• Performance characteristics of laboratory
developed tests
• Reagents that are the components of a test that
is created without regulatory approval – ASR
• Use of FDA-approved tests that have been
modified for the testing procedure
Modified FDA Test
• Alteration of any assay step or analyte for
the test that may affect performance
• Changes could include:
– Type or concentration of specimen
– Storage specifications
– Extraction method
– Amplification parameters or platforms
– Use of test for other purposes
Assay Validation
 Required for:
 New testing methodology
 Assay modification – includes cross-checks for different
makes/models of instrumentation
 Applies to:
 Modified FDA assays
 Laboratory Developed methods
Controls for Each Run
Appropriate positive and negative controls
should be included for each run of
specimens being tested
Molecular Assay Controls
 Positive controls:
 Inhibitors
 Component failure
 Interpretation of results
 Sources:
Residual positive DBS
PT samples
QC materials through purchase or exchange
 Negative controls:
 Nucleic acid contamination
Common Molecular Assay
Problems and Trouble Shooting
 Temperature errors
 Bad dNTPs
 Template/Sequence
 Bad primers
 PCR inhibitors
 Bad enzyme
 Buffers
Critical Molecular Assay
Components
 Nucleic Acids: Prepare aliquots appropriate to workflow
to limit freeze-thaw cycles
 Primers and probes
 dNTPs
 Genomic DNA
4-8°C: Up to one year
-20°C: Up to seven years
 Enzymes
 Benchtop coolers recommended
 Fluorescent reporters
 Limit exposure to light
 Amber storage tubes or wrap in shielding (foil)
Positive Controls
Ideally should represent each target allele
used in each run
May not be feasible when:
 Highly multiplex genotypes possible
Systematic rotation of different alleles as positives
 Rare alleles
Heterozygous or compound heterozygous specimens
Positive Controls
 Assays based on presence or absence of product
 Internal positive amplification controls to distinguish true
negative from false due to failure of DNA extraction or PCR
amplification
 PCR amplification product of varying length
 Specimens representing short and long amplification products to
control for differential amplification
 Quantitative PCR
 Controls should represent more than one concentration
 Control copy levels should be set to analytic cut-offs
Quantitative PCR Controls
• Contamination controls
• Calibration control with reliable low limit of
quantification
• Measures for consistency, especially for
DBS
• Measure of DNA quality – internal control
False Negative: ADO
Allele drop-out (ADO): the failure of a molecular test to
amplify or detect one or more alleles
 Potential causes:
 DNA template concentration
• Incomplete cell lysis
• DNA degradation
 Non-optimized assay conditions
 Unknown polymorphisms in target sites
 Reagent component failure
 Major concern for screening laboratories
 Confirmation of mutation inheritance in families is not an option
DNA Degradation
Lane 1 + 7:
1kb size standard ladder
Lane 2:
100ng control genomic DNA
Lanes 3-5:
Crude cell lysates
Question?
How can you control for presence of
sufficient amount/quality of DNA for a
PCR based test in a NBS lab?
PCR Amplification Controls
• Allele-specific amplification
• Are there problems with this assay?
• What additional controls would be useful?
Allele 1 + 2
Allele 2
Allele 1
Reference
Negative
PCR with Internal Controls
Tetra-primer ARMS-PCR
Simultaneous amplification of:
 Positive amplification control
 Mutation allele
 Reference allele
Alternative to tetra-primer ARMS is to
include an additional primer set to
amplify a different control sequence
Allele Drop-out in PCR Testing
5’
Cgtgatgtacgaggttccat
ggacatgatGcactacatgctccaaggtagtggag
5’
cctgtactaCgtgatgtacgaggttccat
ggacatgatGcactacatgctccaaggtagtggag
Allele Drop-out in PCR Testing
5’
gatgtacgaggttccat
ggacatgatGcGctacatgctccaaggtagtggag
SNP in primer site
Cgtgatgtacgaggttccat
5’
ggacatgatGcGctacatgctccaaggtagtggag
False Negatives: Deletions
Forward Primer
A
Reverse Primer
Forward Primer
G
Reverse Primer
False Negatives: Deletions
Forward Primer
A
Reverse Primer
Forward Primer
G
Reverse Primer
Deletion
False Positives
 Potential causes:
 Non-optimized assay conditions
 Unknown polymorphisms in target sites
Gene duplications
Oligonucleotide mis-priming at related sequences
Psuedogenes or gene families
 Oligonucleotide concentrations too high
 Nucleic acid cross-contamination
How Many Samples?
It depends…
•
•
•
•
New test or to replace an old test?
How is the test to be used?
What performance criteria are most important?
How available are appropriate samples?
• For screening: daily load vs. weekly load vs.
monthly load – it depends
CLSI Evaluation Protocols recognized
by FDA as Consensus Standards
• EP5-A2: Evaluation of Precision Performance of
Quantitative Measurement Methods
• EP9-A2: Method Comparison and Bias Estimation Using
Patient Samples
• EP10-A2: Preliminary Evaluation of Quantitative Clinical
Methods
• EP12-A User Protocol for Evaluation of Qualitative Test
Performance
• EP15-A2: User Demonstration of Performance for
Precision and Accuracy
Molecular Assay Proficiency
Testing Material Sources
 CDC NSQAP
 SeraCare
 UKNEQS
 Corielle
 EuroGentest
 ECACC
 CAP
 In-house samples
 Maine Molecular
 Round-robin with other
NBS laboratories
Professional Guidelines
 American College of Medical Genetics (ACMG)
 Standards and Guidelines for Clinical Genetics Laboratories
 Clinical and Laboratory Standards Institute (CLSI)
 MM01-A2: Molecular Diagnostic Methods for Genetic Diseases
 MM13-A: Collection, Transport, Preparation, and Storage of
Specimens for Molecular Methods
 MM14-A: Proficiency Testing (External Quality Assessment) for
Molecular Methods
 MM17-A: Verification and Validation of Multiplex Nucleic Acid
Assays
 MM19-P: Establishing Molecular Testing in Clinical Laboratory
Environments
Additional Sources