# 508
Risk Assessment and Quality Plans for Next-Generation Sequencing (NGS)
assays in the context of evolving regulatory frameworks
Massimo Morra, MD PhD FACMG; Sherylynn Barker, MLS (ASCP); Bruce Leisz, BSME, MBA; Shujun Luo, MS; Sarah Garcia, PhD MS CGC;
Deanna Church, PhD; Martina Lefterova, MD PhD; Christian Haudenschild, PhD; Rich Chen, MD MS.
Personalis, Inc. | 1330 O’Brien Drive, Menlo Park, CA 94025
1. Background Information
Contact:
massimo.morra@personalis.com
4. Source of Errors Identification through Fish(-Like) Diagram
Next generation sequencing (NGS) technologies enable high-throughput sequencing and play an essential
role in the operational implementation of personalized or precision medicine approaches. NGS-based assays
are generally developed and offered by clinical laboratories as Laboratory Developed Test (LDTs). LDT
testing services and procedures are central to public health and to support medical decision-making. Clinical
laboratories offering LDTs operate under the Clinical Laboratory Improvement Amendments of 1988 (CLIA)
regulations, which require a laboratory to have quality control (QC) procedures to monitor the accuracy and
precision of the complete testing process. Modernization of the CLIA oversight framework in relation to high
complexity LDT services is currently under discussion and the U.S. Food and Drug Administration (FDA) is
re-evaluating its enforcement discretion approach to LDTs. The Centers for Medicare & Medicaid Services
(CMS), the College American of Pathologist (CAP), the American College of Medical Genetics (ACMG), the
American Medical Association (AMA), the Association for Molecular Pathology (AMP) and other organizations
are key stakeholders in this complex process of changing regulatory frameworks.
Here we present and discuss examples on applying risk assessment strategies to quantify and contain
risk in NGS platforms and assays, with the ultimate goal of improving patient testing safety and quality
systems frameworks.
Please Note: the Risk Assessment and Approach here presented and discussed is unrelated to Individualized Quality
Control Plan (IQCP) regulatory considerations. Specifically, an IQCP option is now available to provide an opportunity to
tailor QC plans to the clinical laboratory’s unique testing environment, although it is currently un-applicable to NGS clinical
laboratories (Refer to the 12/28/2015 clarification to CAP by CMS regarding the unavailability of the IQCP option for NGS
molecular-based testing).
Specimens
Testing
Personnel
Equipment
Regulatory
Compliance
Annotation
Test Systems
Incorrect
Test
Results
Reagents
Environment
Software/LIMS
NGS
Pipeline
Gene Variants
Communications
5. Risk Quantification and Prioritization (Based on CLSI EP23)
Severity
Probability of Harm
Negligible
Minor
Serious
Critical
Catastrophic
Frequent
unacceptable
unacceptable
unacceptable
unacceptable
unacceptable
Probable
acceptable
unacceptable
unacceptable
unacceptable
unacceptable
Occasional
acceptable
acceptable
acceptable
unacceptable
unacceptable
Remote
acceptable
acceptable
acceptable
unacceptable
unacceptable
Improbable
acceptable
acceptable
acceptable
acceptable
acceptable
2. Methodology
A hybrid approach based on CDC / CMS, CAP / ISO 15189 and Clinical Laboratory and Standards Institute
(CLSI) EP23 guidelines and worksheets is used. First, assay process maps are generated to identify risk points,
sources of potential failures, or errors in the testing process (Refer to Section 3a, 3b, 3c).
Key questions are to be answered at each processing stage and stages junctures:
• What can go wrong?
• What are the potential sources of errors?
This includes but is not limited to Risk Assessment in relation to specimens, test systems, reagents,
environment, testing personnel, software and LIMS, communications and regulatory compliance (Refer to 4).
Risk quantification consists of assessing and grading probability and severity (Refer to 5). Means of failure
detection are then charted through a CLSI EP23 modified document in relation to different types of controls
(Not Shown). Based on Risk Acceptability and Mitigation determinations, Corrective and / or Preventive CAPA
(Corrective And Preventive Action) actions are then initiated and logged into the CAPA Quality System (Refer
to 6). In addition, residual risk is placed in the context of a Quality Control (QC) plan and actions taken or
planned to mitigate risks. QC plans are evolved to include steps assuring accuracy and reliability of test results
in relation to patient care (Refer to 7). Processes and procedures are then modified to eliminate or reduce
identified risks at acceptable levels.
Ongoing Quality Assurance (QA) assessment is performed through Quality Monitors (QMs) and Indicators (QIs)
to assess quality based on prior risk assessments and implementations. QMs and QIs monitoring allows for
tuning of QC metrics and Standards Operating Procedures (SOPs) steps and new CAPA items are initiated as
applicable (Refer to 6).
3. Risk Points Identification and Process Maps (Simplified Versions)
3a. Product (Assay) Life Cycle Map (not comprehensive)
RnD
Validation
6. Risk Assessment, CAPAs and Quality Management Plans
Risk Assessment
CAPA
Risk Acceptable?
Yes / No
Corrective and
Preventive Actions
• Yes = No action
• No = open CAPA
QC Metrics
• Corrective
• Preventive
Risk Assessment and Mitigation, and related Quality Plans extend to all testing phases
(i.e. pre-analytical, analytical, post-analytical, non-phase specific)
Communication (not comprehensive list)
NGS Pipeline (not comprehensive list)
• Quality Goals and / or QC Metrics
• To provide an informed framework for Clinical Clients
(i.e. healthcare providers) regarding Clinical Test
availability, Specimen requirements and tests Methods
Performance Specifications
• Quality Goals and / or QC Metrics
• To define NGS pipeline specifications
• Means of Minimizing Risk
• SOPs; Document Control tools; Change Control Policies
Specimen tracking (not comprehensive list)
3b. Process (Assay) Map (not comprehensive)
Communications
with Clients
Test Performing
Test Ordering
and Specimen
Reception
• Method Performance
Specification documents
• Assays Data Sheets
• Marketing Materials
• Communications
• Specimen Tracking
• Specimen Accessioning
Results Reporting
• Wet-lab
• NGS Pipeline
• Gene Variants Processing
• Report generation
• Report transmission
• Results communication
• Follow up
3c. Test (Assay) Performing Map (not comprehensive)
• Feed back into
QC metrics or
SOPs steps tuning
7. Examples of Risk Assessment and Risk Mitigation in Relation to
Quality Control Plans
• Example of Risks
• Ordered tests are not clinically indicated
• Patients and Health Care Providers do not understand
test limitations
Production
CAPAs result in new
or improved
• QC step or metric
• SOP step
Quality Monitor
or Indicator
• Quality Goals and / or QC Metrics
• To provide a Specimen tracking and handling
controlled process
• Examples of Risks
• Lost specimens
• Delays in testing
• To define NGS pipeline operation
requirements to dedicated staff
• To define post-pipeline QC metrics
• To define Validation (major vs minor)
requirements
• Examples of Risks
• Lack of NGS Pipeline new releases
proper Verification and Validation
• Passing of operation sequencing runs
not meeting required QCs metrics
• Lack of / insufficient training and
competency for staff running the
NGS Pipeline
• Means of Minimizing Risks
• SOPs; LIMSs workflows; proper staff
training and competency
• Means of Minimizing Risk
• SOPs; communication; LIMSs workflows; tracking tools
Conclusions
Our work and experience suggests that integrated Risk Management approaches may assist in de-convoluting
LDTs NGS-based assays complexity.
Specimen
Accessioning
DNA
Extraction
Sample
Prep
NGS
Sequencing
NGS
Pipeline
Variants
Screening
Variants
Classification
References
• Clinical and Laboratory Standards Institute (CLSI) Guideline EP23 documentation
• College of American Pathologists (CAP) checklists
• COM.50300 Risk Assessment
• COM.50600 Quality Assessment Monitoring
• CAP ISO 15189 – “Risk Management Guideline for Laboratories” guideline document
Variants
Confirmation
Reporting
Risk Management encompasses Risk Identification and Assessment; Risk Quantification; Risk Control,
Mitigation and Acceptance; Risk Communication; and Risk Monitoring. Risk Management is operated through
dedicated teams of ad-hoc stakeholders.
This Risk Management integrated approach is expected to result in an improvement of NGS QC plans in
relation to key process steps and critical factors such as annotation, databases, software, testing personnel
competency, reagents and environment.
Integration of risk management strategies and QC / QA plans with everyday testing observations, Proficiency
Testing (PT) analysis, and assessment of new assay validations provides the basis to maintain, extend and
evolve dynamic QA frameworks.
100-860-A