New York State Guidelines
CLINICAL APPLICATION OF NEXT GENERATION SEQUENCING AND IMPLICATION OF
NY STATE GUIDELINES
Hannah Murfet (BSc, PCQI), Product Quality Manager, Horizon Discovery
This article aims to provide an overview of the clinical application of NGS and the implication of the recently
released NY State Guidelines. A summary of the validation requirements and use of reference materials is
below:
Validation Requirements:



Test for Accuracy: Sequence a well-characterized reference sample to determine error rate across all target
areas.
Test for Precision: This must be test on patient samples. Prior to final validation non-patient defined
mixtures of cell line DNAs (not plasmids) can be used to will help with initial optimisation and understanding of
precision for the detection of variants at the same codon.
Understand Analytical Sensitivity: Establish the sensitivity of variant calling for each type of variant. This can
initially be established with defined mixtures of cell line DNAs (not plasmids) followed by 3 – 5 patient samples.
Reference Standard/Control Use:



A positive/sensitivity control should be included in each NGS run.
This control should be a low positive DNA sample containing multiple known variants(mutations) at
challenging sensitivities to test the workflow and platform
A defined rotation schedule should be employed if not all variants can be covered in a single control sample.
The clinical applications of Next-Generation Sequencing (NGS) technologies are continuing to advance.
Sanger sequencing is still generally considered the gold standard but this comes with low throughput and
high cost. However advances in this Next Generation Sequencing are reducing the cost of sequencing,
whilst facilitating new mechanisms for disease prediction.
Next Generation sequencing is becoming an appealing option with much higher throughput and potential
to change the face of genetic medicine (American College of Medical Genetics and Genomics, 2013).
However care must be taken to ensure high levels of control are maintained, this can be achieved through
validation and reporting controls where specific NGS requirements are expected, in addition to those
mandated by clinical accreditation and certification programmes.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44
Clinical Context of NGS
Figure 1: Overview of the Clinical Context of NGS.
Meeting Requirements
Figure 2: Overview of Clinical NGS Requirement areas.
In the clinical context quality assurance consists of the maintenance of a desired level of quality for
laboratory services. Systematic implementation of a quality management system helps maintain
consistency and quality of services offered, whilst allowing the development of efficiency and definition of
authority.
Quality assurance is essential to ensure efficient and reproducible operations and document controls form
one essential part of quality assurance. While the requirements specifically identify Standard Operating
Procedures (SOPs) as a requirement, supporting documentation will help provide context.
Typically quality management systems take a three tier hierarchy. At the highest level the policies define
the organisation’s strategy and focus; here you will find organisational goals, expectations and
commitments. Procedures define and document instructions on performing business/quality management
or technical activities. In the case of New York State Department of Health guidelines, there is clear focus
on the requirement for SOPs.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44
Figure 3: Document hierarchy for management system documents.
SOPs can be broken into two levels. The first level represents the flow of information and useful for
demonstrating the sequence of event and associated responsibilities or authorities. The first level
procedures are best kept at a relatively high level and work well as flow diagrams supported by
supplementing text such as the example below.
Figure 4: Level one process example including steps, assigned responsibility/authority and sub steps. A level
one process may reference more specific and detailed level two processes.
An overview of the typical testing sequence is shown below; this may be incorporated into one or a few
level one processes depending on the complexity of the clinical laboratory’s operations.
Figure 5: Testing sequence overview.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44
Level two processes are best documented as clear ‘how to guides’, detailing all the responsibilities,
materials and procedures necessary to complete the activity. For laboratory focussed activities, these are
best used as inputs to and outputs from validation studies to establish consistent and clear protocols that
support good training and operation of the laboratory.
Records are an essential practice and should include which test instruments were used in each test and
documentation of all reagent lot numbers. Any deviations from standard procedures should be recorded,
including any corrective measures to report and resolve any deviations. Templates may be generated to
ensure consistency in output records for both testing and reporting.
In addition to documented processes, implementation of predetermined checkpoints or key performance
indicators should be included to permit the monitoring of quality assurance over time. Once established
these may act as a trigger for assay drift, operator variability or equipment issues.
Controls for reports and data must be implemented. In the US compliance to the HIPAA Act (Health
Insurance Portability and Accountability Act) must be implemented to ensure traceability and protection of
patient data. Many authorities mandate record retention periods, including CLIA who mandate that records
and test reports must be stored for at least 2 years. With strong implementation of quality assurance
systems clinical laboratories can assure reliability and continuity of results and may look to further
certification such as the implementation of ISO 15189, especially in countries where there are no formal
accreditation schemes.
Validation for clinical laboratories involves the assessment of various parameters including those used for
protocols, tests, materials and platforms. The aim of validation is to provide confidence to meet critical
requirements. The following data section (section 3) should be considered when determining the scope of
validation required.
Figure 6: Validation and assurance levels.
An overview of the specific validation requirements for Next Generation Sequencing is given below, this
may be used as a basic checklist of coverage, supplemental to more general accreditation or certification
requirements e.g. those required by CLIA or ISO 15189.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44
Figure 7: Overview of NGS Specific Validation Requirements.
Given the huge amount of data that may be generated through Next Generation Sequencing, accurate and
efficient systems are essential. Data is generally established through validation and then monitored through
the establishment of predetermined checkpoints or key performance indicators to ensure consistency and
accuracy of service provision.





Accuracy
Validation including a minimum of 50 patient samples with representation for material type (FFPE) and variants
across target areas and confirmed by an independent reference method. Minimum of 10 positive samples for
each type of variant.
A recomended approach is to sequence a well characterised reference sample to determine specificity.
If vigorous validation of reported variants has not been completed in the original studies, ongoing confirmation
by independent reference methods must be performed until at least 10 reference points have been
independently validated.
A disclaimer must be used where incidental finding of unknown significance are included where there is no
established confirmatory assay. The disclaimer must clearly state that the variant has not been verified.
Robustness
Robustness is the likelihood of assay success; adequate quality control measures must be in place to determine
success of techniques such as extraction, library preparation or sequencing.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44











Precision
Precision is related to within run control.
For each type of variant a minimum of 3 positive samples containing variants near the stated sensitivity of the
assay must be analysed in triplicate in the same run using different barcodes.
Renewable reference samples can be used to determine the analytical validity of the test. These can establish
baseline data to which future modifications can be compared.
Repeatability and Reproducibility
Repeatability and reproducibility is related to between run controls to determine ability to return identical
results under identical (repeatability) or changed (reproducibility) conditions.
For each type of variant a minimum of 3 positive samples containing variants near the stated sensitivity of the
assay must be analysed in three separate runs using different barcodes on different days by two different
technologists where possible.
If multiplexing samples with distinct barcodes, it must be verified that there is no cross talk and that all target
areas and variants are reproducible independent of which patient/barcode combination is used.
It is useful to consider instrument-instrument variability as well as interoperator variability. Parameters for
expected reproducibility should be established, typically around 95-98%.
Analytical Sensitivity and Specificity
Sensitivity and specificity refers to positive and negative percent respectively, in agreement of results when
compared to gold standard.
Interrogate all types of variants in three target areas with consistently poor coverage and three with
consistently good coverage. This can be established with defined mixtures of cell line DNA and not plasmids,
but needs to be verified with 3 – 5 patient samples.
The limit of detection should be established.
Confidence intervals for variant types must be determined.
Table 1: Overview of NGS Specific Data Requirements.
A minimum data set is expected to establish key performance characteristics:-
Figure 8: NGS minimum data set.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44
In contrast to quality assurance where the infrastructure for quality is established to maintain the right
service, quality control addresses testing and sampling to confirm outputs against requirements. Quality
control takes place across various areas from reagents used, software and in assay controls:





Overview of Quality Controls
Quality control of reagent lots is best implemented at the point of goods in inspection. A clear label should be
placed on the reagent under inspection then testing can be performed to validate/confirm analytical sensitivity.
Quality control of software updates can be handled through a version control and impact assessment process.
All re-validation must be clearly documented and demonstrate consistency in analytical sensitivity.
Sample identity is essential especially if samples are pooled. Proficiency testing protocols must be established
to allow for execution as required by clinical accreditation bodies (such as CLIA).
Quality control stops may be added to laboratory process before the sequencing run, to the run itself and at
the end before data analysis.
Use of control materials such as positive and negative controls. An overview of these is presented in the next
figure.
Table 2: Overview of quality controls.
Figure 9: Control Requirements.
Several QC requirements may be required and documentation of these requirements is best detailed in the
relevant SOP to ensure they are documented and available at point of use. The quality control measures
applied can vary depending of chosen methods and sequencing instrument but should include methods to
identify sample preparation failures and failed sequencing runs.
Both accreditation programmes such as CLIA and standards certifications such as ISO 15189 contain specific
requirements around the generation, approval, issue and re-issue of reports. The most essential
requirements related to Next Generation Sequencing have been listed below.
Overview of Quality Controls



The laboratory director is responsible for designing the advantages and limitations of their test offerings so that
healthcare providers can make an informed decision.
Turnaround times for reports should be prepared to ensure there are clear requirements for NGS test
prioritisation, these turnaround times should be clinically appropriate.
All detected somatic variants, identifying each variant’s significance.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44




Incidental findings including clinical relevance.
Identify limitations of the assay, including for which target areas the assay lacked sufficient coverage to
confidently determine mutational status.
Include information related to compare the level of Exome vs. Genome sequencing to an available disease
specific panel test.
Two references were used to generate this list: (American College of Medical Genetics and Genomics, 2013)
(New York State Department of Health, 2014).
Table 3: Overview of NGS report requirements.
Conclusions
The overall picture for Next Generation Sequencing is evolving with clearer requirements for everything
from quality assurance to reporting requirements. The key with Next Generation Sequencing, as with any
clinical technology is to establish validity and clearly document any assumptions or limitations of the
platforms and assays involved, especially when reporting to the requesting clinician. While there is not
complete understanding the clinical implications of some variants, there are clear prospects emerging for
this technology to improve the standard of care and further advance the development and clinical support
for companion diagnostics.
Horizon Diagnostics
Horizon Diagnostics is proud to support clinical laboratories with the provision of sustainable reference
materials for research use; applications include the validation of equipment and consumables offline from
patient testing. This report has been provided for information only.
Bibliograpy
 American College of Medical Genetics and Genomics. (2013, September). ACMG clinical laboratory
standards for next-generation sequencing.
 Horizon Discovery. (n.d.). ISO 15189: A Standard of Yin and Yang.
 New York State Department of Health. (2014, January). "Next Generation" Sequencing (NGS)
guidelines for somatic genetic variant detection.
t: +44 (0)1223 655580
f: +44 (0)1223 655581
e: [email protected]
w: www.horizondiscovery.com
Horizon Discovery Ltd., Building 7100, Cambridge Research Park, Waterbeach, Cambridge, CB25 9TL, United Kingdom
UK Registration No: 05363294 UK VAT No: GB 189 1993 44
Download

New_York_State_Guidelines