NGS IN THE CLINIC A review of its impact on test development, operations and result reporting Qiagen Webinar – January 19, 2016 Birgit Funke, PhD, FACMG Associate Professor of Pathology, Harvard Medical School Director Clinical R&D; Laboratory for Molecular Medicine 3 http://www.nature.com/nrg/journal/v11/n1/images/nrg2626-f2.jpg WHAT WILL BE COVERED 1. How is NGS used in the clinic today • Focus on inherited disorders 2. Bringing NGS in house: • Clinical development and validation • The role of bioinformatics 3. What NGS does not do well • Technical limitations 1. Time permitting: Real world challenges (case example) THE DISRUPTIVE NATURE OF NGS NGS (research) • NGS (clinical) Clinical NGS is being implemented in an increasing number of labs • Majority focus on gene panels • Implementation of exome/genome sequencing quickly increasing DETECTION RATE OVER TIME DCM 2011: ~37% 2007: ~10% NGS WHICH DISORDERS BENEFIT FROM NGS TESTING? EXAMPLE: INHERITED CARDIOMYOPATHIES HCM DCM ARVC OTHER (RARE) LVNC RCM 1:500 • • • • > 1:2,500 1:1,000 - 1:5,000 Collective incidence: > 1/500 Can lead to SCD Substantial genetic component Incentive for predictive testing DISEASE CHARACTERISTICS • Locus heterogeneity - 1 disease / many genes • Allelic heterogeneity - many disease causing variants/gene • Spectrum of pathogenic variation - not yet well understood • Phenotypic overlap - can complicate testing process LOCUS HETEROGENEITY DCM TNNT2 HCM MYBPC3 50% TTN DSP MYH7 34% MYH7 LMNA Melissa Kelly Ahmed Alfares ALLELIC HETEROGENEITY # of Variants Hypertrophic Cardiomyopathy: 10 years – 11 genes tested 63% of variants have been seen only once Need to sequence entire coding sequence of many genes for maximum clinical sensitivity MYBPC3 E258K MYBPC3 927-9G>A MYH7 R663H MYBPC3 W792fs R502W # of Probands Courtesy of Ahmed Alfares and Heidi Rehm CLINICAL HETEROGENEITY Traditional genetic testing : 1 gene panel for each diagnosis • Proband with clinical Dx + family history of DCM • DCM gene panel detects a variant of uncertain significance + + + + 3 • Variant did not segregate 3 PREVENTING DIAGNOSTIC ODYSSEYS _ • Patient was seen again , diagnosis was revised to ARVC + + • ARVC panel identified a likely pathogenic variant + + + + + 3 3 Traditional testing (disease centric) does not make sense for disorders with clinical and genetic overlap + MULTI-DISEASE GENE PANELS HCM test DCM test ARVC test MULTI-CARDIOMYOPATHY TEST HCM DCM ARVC • ~3% of DCM patients carry a pathogenic variant in an ARVC gene MULTI DISEASE GENE PANELS IMPROVE CLINCAL DIAGNOSIS GeneReviews (2012): “80-90% of patients with Costello syndrome carry a mutation in HRAS.” LMM broad referral population data: (n=23) Most patients would have received a NEGATIVE report if only HRAS had been tested MULTI DISEASE GENE PANELS IMPROVE CLINCAL DIAGNOSIS Phenotypic expansion • Original clinical definition based on most severe cases • Often too narrow, full range of clinical variability emerges over time Phenotypic overlap • Disorders present the same -> diagnostic “error” • Happens more often as genetic testing is moving out of specialty clinics to more general (genetics) care Now widely recognized • Many physicians are beginning to change workflow THE CHANGING GENOMIC TESTING WORKFLOW “SEQUENCE FIRST” Patient Clinical “sequence first” Hypothesis Diagnosis Genetic Test Sequence Variants negative Result Clinical Diagnosis Inter pretation • Fill in failed • Confirm variants THE CURRENT DEBATE: PANELS OR EXOME? GENE PANELS EXOME GENOME 10s – 100s of genes 22,000 genes Exome + Intergenic •Med-Low coverage •Completeness <100% •High coverage •Completeness Clinically well defined cases ? Complex phenotypes / diagnostic odysseys INCENTIVES • A large fraction of panels are NEGATIVE (often >50%) • Growing appreciation of “phenotypic expansion” – argument for hypothesis fee testing • Additional tests often end up being more expensive than WES • Always up to date (accelerating pace of gene discovery) • Easier to maintain for labs than growing # of gene panels BARRIERS BARRIERS • Cost (though gap is closing) • Incomplete coverage (suboptimal design) RISK • Loss of intimate /a priori knowledge on tested genes COMMUNITY EFFORTS TO DEVELOP STANDARDS ACMG + AMP (2015) New guideline for clinical grade variant classification (Mendelian disorders) ClinGen Unite medical genetics community by developing approaches to curate, centralize and share genetic data NEW! ASSESSMENT OF GENE-DISEASE RELATIONSHIPS • Traditionally not needed because old tests contained only well established disease genes • Many published claims for a gene-disease relationship do not withstand the rigor of clinical grade curation • Most novel variants in genes that are not strongly linked with disease cannot be interpreted USING CLINICAL VALIDITY TO CONFIGURE TESTS Definitive evidence Strong evidence Moderate evidence Limited evidence Disputed / no evidence Predictive Tests, Incidental Findings Diagnostic Panels Exome/Genome Research Courtesy of ClinGen (Heidi Rehm) CLINICAL TEST DEVELOPMENT AND VALIDATION (NGS) DEVELOPMENT AND VALIDATION OF AN (NGS) TEST DESIGN TEST DESIGN LIBRARY PREP http://www.cdc.gov/genomics/gtesti ng/ACCE/ VALIDATION OPTIMIZATION • • • • CAPTURE SEQ ANALYSIS Test each step String together Optimize Lock down protocol Test analytical performance using locked down protocol DESIGN - A THREE DIMENSIONAL EXERCISE Technology Content Gene selection • Clinical validity Gene characteristics • Homology, High GC/AT? • Pathogenic variant types • Mosaicism? X Sequencer • capacity/ln • Read length Bioinformatics pipeline • SNV • Indel • CNV • SV Techn. Reality Check = Operational • Target size • # samples/lane • Sample volume received • Test cost, TAT Diagnostic • % gene unsequenceable • Difficult variants prevalent • Ability to detect low AF Development + validation plan Clinical Reality Check Supplemental assay/s when: • NGS suboptimal but gene or variant type essential Validation (reference) samples • Specimen types • Variant Types • Reference material sources Fill-in + confirmation strategy (Sanger) Consult domain experts • Difficult gene critical? • Alternate assays offered elsewhere • Required analytical sensitivity*? • Required completeness * TP/TP+FN THERE IS NO “ONE SIZE FITS ALL” Two difficult to sequence genes – two solutions* (Laboratory for Molecular Medicine) Renal Panel (PKD1) Both present similar challenge Hearing Loss Panel (STRC) Essential gene (detection rate) YES (85%) YES (10%) Sequenceable via NGS Highly homologous (partial) Highly homologous (entire gene) Pathogenic variant spectrum SNVs, indels >>CNVs CNVs >>SNVs, indels Well sequenced gene YES NO Clinical specificity HIGH LOW Keep on panel NO - alternate test for ADPKD YES Supplemental assay N/A LR-PCR x 2 Operational factors N/A High failure LR-PCR #1 – DROP Impact on performance metric N/A Reduced analytical sensitivity OK Two very different solutions TEST DEVELOPMENT/OPTIMIZATION DESIGN VALIDATION OPTIMIZATION • Select test contents • Characterize contents • Clinical needs LIBRARY PREP CAPTURE SEQ ANALYSIS • Technical limitations • Supplemental assays? • Fill-in and variant confirmation ? • Establish development + validation plan • • • • • Test each step String together Optimize if need be Lock down protocol Develop Q/C stops Test analytical performance using locked down protocol NGS RUN PRIMARY ANALYSIS BASE CALLING FASTQ Sometimes on the Sequencer (newer machines, e.g. MiSeq) DEMULTIPLEXING SECONDARY ANALYSIS ALIGNMENT BAM output files VARIANT CALLING “bioinformatics pipeline” VCF ANNOTATION TERTIARY ANALYSIS PRIORITIZATION FILTRATION CLASSIFICATION CLINICAL REPORT Trend is to add more functionality to the sequencer THE ROLE OF BIOINFORMATICS (…not just for NGS) Research Clinical Dev Bioinformatics IT Clinical Ops BIOINFORMATICS STAFF (Partners Personalized Medicine) Molecular Dx Lab + Translational Genomics Core 2008 2009 2010 2011 7 1st NGS panel Bioinformatics FTEs 6 5 4 3 2 1 NGS work started 2012 2013 2014 COMMONLY USED DATA ANALYSIS FILTERS • • Secondary Analysis • Coverage (read depth) • Strand bias • Mapping quality (confidence in location of read) • Allelic fraction (% reads containing variant) • Quality by depth Tertiary Analysis • Variant frequency in control cohorts COVERAGE : HOW MUCH IS ENOUGH? Red = reads containing a variant call Courtesy of Karl Voelkerding • In theory, a heterozygous variant is present in 50% of the reads • In reality, this is not always the case (range can be broad) • To be confident to identify variants with a range of allelic fractions (e.g. >20%): Need ~20x coverage to be 99% confident to detect variant allele For details see: Schrijver 2012, J. Mol. Diagn. 14 REALITY CHECK Need ~20x coverage to be 99% confident to detect heterozygous variant * Example LMM (germline) – 46 genes (~250 kb) • Very high average coverage (300-400x) • >97% of bases >20x • 3% <20x = 7.5 kb • Reducing avg. coverage increases the % of the test with insufficient coverage! Need to sequence at high coverage to maximize completeness * Schrijver 2012, J. Mol. Diagn. 14 ANALYSIS THRESHOLDS – ALLELIC FRACTION 203 samples; n=1455 variants: NGS + Sanger TRUE POS FALSE POS HOM TAZ_EX10 c.718G>C TTN_EX37 [SB= -3536.45] c.8887A>C TTN_EX37 [SB= 17.08] c.8887A>C [SB= 14.07] TTN_EX37 c.8887A>C [SB= 8.04] • Need large “truth sets” • Until recently there was none labs could obtain pre-test development • Data shown here generated from launching NGS pipeline w/o any threshold (2011) • very high overhead for confirmatory testing till truth set established HET via Sanger confirmation • Now have some community resources: NIST’s NA12878 high confidence regions LMNA_EX10 c.*4C>T [SB= -106.6] TTN_EX157 c.31861A>G [SB= -855.52] TTN_EX45A c.15318T>C TTN_EX155 [SB= -146.03] c.31709T>C [SB= -10.01] ALLELIC FRACTION THRESHOLDS SNVs + IN/DELS STRAND BIAS READ 1 READ 2 A T C C G T G G A T T T C G T T A C A C G T T T T T T T See also: Allhoff et al. BMC Bioinformatics 2013 14 (Suppl 5):S1 doi:10.1186/1471-2105-14-S5-S1 ANALYSIS THRESHOLDS – QUALITY BY DEPTH QUALITY CONTROL STEPS DNA Extraction • OD 260/280 Library Generation • Fragment size (each step) Sequencing • Cluster Density, Error rate • Total reads, % reads pass filter Alignment, Variant Calling • % reads on target, avg coverage • Min cov/base, library complexity, • MQ, Strand Bias, GC/AT bias, etc Filtration, Prioritization Variant confirmation • Concordance w/ NGS result Variant + Gene assessment • Double review Interpretation+ Report • Double review of variants and interpretation • Q/C Approach? • Individual steps? • End-to-end? Both is best! • End-to-end critical measures • Completeness • Accuracy EXAMPLE Q/C STEPS - CLUSTER DENSITY (HiSeq) Difficult to prescribe standard – reduced quality at this step can be acceptable depending on the overall test design (fill in, confirmation) DETERMINE NEED FOR ORTHOGONAL CONFIRMATION OF VARIANTS Substitutions * In/dels FN 0/542 3/70 % FN 0% 4.29% SENS 100% 95.70% 95% CI 99.3-100 89.8-99.2 FP 1/1321 9/60 % FP 0.08% 15.00% When initial performance assessment reveals high FP rate one needs to plan for orthogonal confirmation Today, NGS is deemed of high enough accuracy for SNVs such that confirmation may not needed Laboratory for Molecular Medicine 2013, unpublished data (HiSeq2000, 50b PE) TEST VALIDATION DESIGN VALIDATION DEVELOPMENT • Select test contents • Characterize contents • Clinical needs LIBRARY PREP CAPTURE SEQ ANALYSIS • Technical limitations • Supplemental assays? • Fill-in and variant confirmation ? • Establish development + validation plan • • • • Test each step String together Optimize Lock down protocol Test analytical performance using locked down protocol WHY IS NGS DIFFERENT? • Genetic testing traditionally synonymous with genotyping (assessing whether known pathogenic variants are present) • Sequencing based tests predominantly used to screen the entire coding sequence enables novel variant detection • Core issues • Cannot validate every theoretically possible variant • The NGS testing process is highly complex (many steps) • Increased requirement for bioinformatic processing • Knowledge required for interpretation does not scale proportionally • These issues are not entirely new – but are an order of magnitude worse compared to traditional methods (e.g. Sanger sequencing) • NGS “pushed us over the edge” - catalyzed awareness and a push for novel approaches and standards ANALYTICAL PERFORMANCE METRICS DESCRIPTION EXAMPLE Quantitative Test Result can have any value between 2 limits) Heteroplasmy of mitochondrial variants Qualitative Test True quantitative signal can only have one of two possible values Sequencing (WT or variant allele) • See also: • Mattocks 2010: Eur J Hum Genet 18:1276 • CDC MMWR 2009 Vol 58 (www.cdc.gov/mmwr) • Jennings 2009 Arch Pathol Lab Med.133:743–755 SENS SPEC ACC PREC LOD Different approaches depending on type of test Sens: Spec: Acc: Prec: LOD: Sensitivity Specificity Accuracy Precision Limit of Detection ANALYTICAL SENSITIVITY Base line (“methods-based approach”) • Maximize # of variants tested to achieve high statistical confidence • Pathogenicity doesn’t matter • Determine separately by variant type (SNV, indel, CNV) • Critical importance of confidence interval!! VARIANT TYPE # FN SENSITIVITY All variants 415 2 99.52% Substitutions 350 0 100.00% Indels (all) 65 2 96.92% >10 bp 14 1 92.86% 95%CI 98.2-100 98.9-100 89.8-99.2 70.2-98.8 Disease/gene specific add-ons • Test samples with common disease causing mutations • Example: validate the mutations present on the ACMG 23 panel for CF if NGS test covers CFTR ANALYTICAL SPECIFICITY Definition • Ability to return a negative result when no variant is present • Reflects frequency of false positives: TN / TN + FP Common practice – count WT bases correctly identified as WT • 20 kb sequenced: 5 FP 19995 true negatives • Specificity = 19995 / 20000 = 99.98% Not very useful – other metrics are more meaningful • Technical Positive Predictive Value (likelihood that a variant call is FP): TP/ TP + FP • Number of FPs per TEST • Particularly important if lab doesn’t confirm variants! VALIDATION LAYERS EXAMPLE: ANALYTICAL SENSITIVITY TIER 1: BASE LINE (METHODS BASED, CUMULATIVE) NA12878 Test 1 (100 genes) Variants 600 Test 2 (70 genes) 300 Test 3 (30 genes) 100 Test 4 (200genes) 800 Analytical sensitivity (+CI) (=TP/TP+FN) TIER 2: DISEASE/GENE SPECIFIC CONSIDERATIONS Well character. gene + n samples (e.g. frequent pathogenic variants) In/del rich gene + n samples CNV rich gene + n samples Mosaicism + n samples Need for more reference materials! 25 bp deletion (intron 32, MYBPC3) Dhandapany et al. Nat Genet. 2009. 41(2):187-91 • c.3628-41_3628-17del (outside region interpreted by many labs) • MAF = 4-8% in SE Asia • Het: Mild/late onset HCM (high lifetime penetrance) – OR ~6 • Hom: Severe and early onset disease, often Heart Failure WHAT NGS DOES NOT DO WELL CURRENT NGS LIMIATIONS THE DEVIL IS IN THE DETAILS NGS HAS PROBLEMS WITH • Regions of high homology • Repeat expansions • Large in/dels, CNVs and other structural variants http://www.trizic.com/the-devilis-in-the-details-how-preparedare-you-for-a-presence-exam/ LESS PROBLEMATIC FOR GENE PANELS • Testing labs typically have clinical domain expertise • Contents typically highly curated + well understood • Add-on tests typically available for difficult genes KNOWLEDGE DOES NOT SCALE EASILY • Sequencing ≠ understanding SEQUENCING GENES WITH HOMOLOGY STRC (nonsyndromic hearing loss) • 99.6% identical across cds (98.9% incl. introns) • First half of gene 100% identical (incl. introns) 29 1 100% identical incl. introns maps to pseudogene Mandelker 2014, J. Mol. Diagn. # % # % # HOMOL. HOMOL. # HOMOL. HOMOL. GENE EXONS EXONS EXONS GENE EXONS EXONS EXONS HYDIN 86 78 91 RPS17 5 5 100 TTN 363 26 7 OPN1LW 6 5 83 NEB 183 24 13 OCLN 9 5 56 STRC 29 23 79 GBA 12 5 42 DDX11 27 recessive 18 67 PMS2 hearing15 33 STRC: 10% of nonsyndromic loss 5 RANBP2 29 15 52 FANCD2 44 5 11 DPY19L2 22 13 of congenital 59 FCGR3B 6 4 67 CYP21A2: Nearly all adrenal hyperplasia TNXB 44 12 27 RHD 10 4 40 NCF1 11 11 100 CEL 11 4 36 VWF: Von Willebrandt’s disease (common coagulopathy) ADAMTSL2 19 10 53 CATSPER2 13 4 31 SMN1 9 9 100 CHEK2 16 4 25 SMN2 9 9 100 CLCNKB 20 4 20 PMS2: Colorectal CA (on ACMG Incidental Findings gene list) ABCC6 31 9 29 NOTCH2 34 4 12 CYP21A2 10 8 80 TBX20 8 3 38 IKBKG 10 8 80 KRT81 9 3 33 OTOA 28 8 29 KRT86 9 3 33 Exome wide clinical grade shared via Get-RM CFC1 6 6 resource 100 to beSTAT5B 19 (in progress) 3 16 OPN1MW 6 6 100 PIK3CA 21 3 14 CHRNA7 10 6 60 HS6ST1 2 2 100 VWF 52 6 12 HBA1 3 2 67 Diana Mandelker, Himanshu Sharma, Mark Bowser NGS IN THE CLINIC REAL WORLD CHALLENGES A WORD OF CAUTION • THE NGS PROCESS IS HIGHLY COMPLEX • EVEN THE BEST INFORMATICS AND Q/C MEASURES DO NOT REPLACE A CRITICAL MIND • WATCH OUT FOR RESULTS THAT DON’T MAKE SENSE CASE EXAMPLE 9 year old boy with suspected Noonan syndrome TEST ORDERED Noonan Spectrum Panel (12 genes tested via NGS) NEGATIVE FAMILY HISTORY REVISITED • Physician ordered whole exome • Pathogenic missense variant in SOS1 • Gene WAS covered by our test… • What went wrong? some features • Variant in untested region of SOS1 or variant type not detectable? • Technical false negative? ….at this point the lab director received an unpleasant email Accession ID PM13-00632A Index/Barcode GCAT Amplicons for Sequencing Follow-up (1) PTPN11 01 DIGGING IN THE TRASH…. Failed Regions (1 entries, 1 amplicons)--> exons with 1 or more bases that are insufficiently covered (<20x) PTPN11 PTPN11_Exon_01 44/44 Variant calls Gene Exon Coverage A;B breakdown B frequency Mapping Quality Category Classification Predicted Zygosity Predicted Variant Predicted AA Strand bias All entries below are variant calls that were removed by the pipeline Common variants meeting criteria to be classified as "BENIGN" or "LIKELY BENIGN"(7 entries, 7 amplicons) Gene HRAS KRAS MAP2K2 MAP2K2 SPRED1 SPRED1 SPRED1 Exon Exon 02 Exon 05 Exon 06 Intron 07 Exon 03 Intron 04 Exon 07 Coverage A;B breakdown B frequency Mapping Quality Category 761 503;258 0.34 57.84 Variant 1000 0;1000 1 59.25 Variant 945 501;444 0.47 31.67 Variant 692 0;692 1 59.38 Variant 942 424;518 0.55 59.54 Variant 992 417;575 0.58 59.24 Variant 1000 410;590 0.59 59.22 Variant Classification Predicted Zygosity Predicted Variant Benign (Het) c.81T>C Benign (Hom) c.483G>A Benign (Het) c.660C>A Benign (Hom) c.919+12A>G Benign (Het) c.291G>A Benign (Het) c.424-8C>A Benign (Het) c.1044T>C Predicted AA Strand bias p.His27His -2494.88 p.Arg161Arg -13656.42 p.Ile220Ile -4434.96 -6098.73 p.Lys97Lys -8898.5 -6371.93 p.Val348Val -9031.77 FILTERED (REMOVED FROM ANALYSIS) SECTION Common False Positives (3 entries, 3 amplicons): Strand Bias >20 or Allelic fraction <0.2 Gene HRAS KRAS SOS1 • • • Exon Intron 02 Exon 06 Exon 14 Coverage A;B breakdown B frequency Mapping Quality Category Classification Predicted Zygosity Predicted Variant Predicted AA Strand bias 481 424;57 0.12 58.83 Low_AF Benign (Het) c.111+15G>A -222.02 961 856;105 0.11 59.34 Low_AF Benign (Het) c.519T>C p.Asp173Asp -118.12 985 631;354 0.36 59.44 Strand_bias (Het) c.2200A>C p.Thr734Pro 97.48 SOS1 Thr734Pro: strand bias = 97.48 Highest strand bias observed in technical validation = 1.78 Used threshold of 20 since TEST LIMITATIONS IN CLINICAL PRACTICE • 100% sensitive for SNVs • CI = 99.3-100% Technical false negatives will happen ACKNOWLEDGMENTS LMM Development • Heidi Rehm • Birgit Funke • Matt Lebo • Beth Duffy-Hynes • Sami Amr • Mark Bowser • Heather McLaughlin • Diana Mandelker • Heather Mason-Suares Fellows • Jun Shen • Ozge Birsoy Operations • Lisa Mahanta • Robin Hill Cook • Laura Hutchinson • Shangtao Liu • Ashley Frisella • Timothy Fiore • Jiyeon Kim • Maya Miatkowski • William Camara • Eli Pasackow • Yan Meng • Ahmad A. Tayoun • Jillian Buchan • Fahad Hakami • Hana Zouk Genetic Counselors • Amy Lovelette • Melissa Kelly • Mitch Dillon • Shana White • Andrea Muirhead • Danielle Metterville Marketing • Priscilla Cepeda Office • Peta-Gaye Brooks • Kim O’Brien • Lauren Salviati • Charlotte Mailly • Cassandra Novick Transl. Genomics Core Sami Amr • All team members IT Team • Sandy Aronson • Natalie Boutin • All team members ClinGen leadership and cardiovascular domain working group Bioinformatics • Matt Lebo • Rimma Shakhbatyan • Jason Evans • Himanshu Sharma • Peter Rossetti • Ellen Tsai • Chet Graham Leadership and Faculty • Scott Weiss • Meini Sumbada-Shin • Robert Green • Immaculata deVivo • Mason Freeman • John Iafrate • Mira Irons • Isaac Kohane • Raju Kucherlapati • Cynthia Morton • Soumya Raychaudhuri • Patrick Sluss THANKS! Title, Location, Date 62