CBI NGS Workshop Lesson 4 The Genome Analysis Toolkit (GATK) Liu Huan(刘欢) Center for Bioinformatics, Peking University 2011-05-30 Outline Basic Concepts Overview for Variant Discovery GATK Architecture Data Processing Pipeline of GATK for Variant Detection Basic Concepts Single-nucleotide polymorphism (SNP) - a DNA sequence variation occurring when a single nucleotide - A,T,C,G- in the genome differs between members of a biological species or paired chromosomes in an individual. - e.g. two DNA fragments from different individuals, AAGCCTA to AAGCTTA (two alleles) - Almost all common SNPs have only two alleles - Within a population, SNPs can be assigned a minor allele frequency Basic Concepts Indel - an insertion or a deletion - e.g. reference: AT GG indel 1: AT _G indel 2: ATCGG Basic Concepts copy number variation (CNV) - a form of structural variation - alterations of the DNA of a genome that results in the cell having an abnormal number of copies of one or more sections of the DNA - CNVs correspond to relatively large regions of the genome that have been deleted (fewer than the normal number) or duplicated (more than the normal number) on certain chromosomes - e.g. normal chromosome structure: A-B-C-D CNV 1: A-B-C-C-D (a duplication of "C") CNV 2: A-B-D (a deletion of "C") - This variation accounts for roughly 12% of human genomic DNA - each variation may range from about one kilobase (1,000 nucleotide bases) to several megabases in size Outline Basic Concepts Overview for Variant Discovery GATK Architecture Data Processing Pipeline of GATK for Variant Detection Framework for variation discovery and genotyping from next-generation DNA sequencing Phase 1: - raw read data with platform-dependent biases were transformed into a single, generic representation with wellcalibrated base error estimates, mapped to their correct genomic origin and aligned consistently with respect to one another. Mapping algorithms placed reads with an initial alignment on the reference genome, either generated in, or converted to, the technology-independent SAM reference file format. - molecular duplicates were eliminated - initial alignments were refined by local realignment and then an empirically accurate per-base error model was determined. Framework for variation discovery and genotyping from next-generation DNA sequencing Phase 2: - the analysis-ready SAM/BAM files were analyzed to discover all sites with statistical evidence for an alternate allele present among the samples including SNPs, short indels and copy number variations (CNVs) Framework for variation discovery and genotyping from next-generation DNA sequencing Phase 3: - technical covariates, known sites of variation, genotypes for individuals, linkage disequilibrium (LD), and family and population structure were integrated with the raw variant calls from phase 2 to separate true polymorphic sites from machine artifacts, and at these sites, high-quality genotypes were determined for all samples. Outline Basic Concepts Overview for Variant Discovery GATK Architecture Data Processing Pipeline of GATK for Variant Detection GATK architecture MapReduce MapReduce： - parallel computation - two steps: subdivide large problems into many discrete independent pieces, which are fed to the map function, followed by reduce function, joining the map results back into a final product - subdividing - load balance Example： - SNP discovery: map function - ChIP-seq (peak calling) : reduce function GATK architecture traversals - provide the division and preparation of data walkers - analysis module - provide the map and reduce methods that consume the data GATK can provide a nearly comprehensive set of traversal types that satisfy the data access needs of the majority of analysis tools Traversal Types in GATK “By each sequencer read” (read-based) and “by every read covering each single base position in a genome” (locus-based) - standard methods for accessing data for several analyses - e.g. counting reads, building base quality histograms, reporting average coverage of sequencer reads over the genome, calling SNP Read-based Traversals Read-based Traversal - presents the analysis walker with each read individually, passing each read once and only once to the walker’s map function. - along with the sequencer read, the walker is presented with the reference bases that the read overlaps - is useful for analyzing read quality scores, alignment scores, and merging reads from multiple bam files. Locus-based Traversals Locus-based Traversal - It presents the analysis walkers with all the associated genomic data, including all the reads that span the genomic location, all reference ordered data, and the reference base at the specific locus in the genome. - Each of these single-base loci are passed to the walker’s map function - e.g. depth of coverage calculation, variant analysis Depth of Coverage Walker in GATK Depth of Coverage: - important in CNV discovery, SNP calling, and other downstream analysis Depth of Coverage Walker in GATK: - at each site the walker receives a list of the reads covering the reference base and emits the size of the pileup - The end user can optionally exclude reads of low mapping quality, and other read filtering criteria. - can also be provided with a list of regions to calculate coverage, summing the average coverage over each region - can also be used to quantify sequencing results over complex or highly variable regions, e.g major histocompatibility complex (MHC) Depth of Coverage Walker in GATK Outline Basic Concepts Overview for Variant Discovery GATK Architecture Data Processing Pipeline of GATK for Variant Detection Data Processing Pipline of GATK initial mapping refinement of the initial reads multi-sample indel and SNP calling filtering of the raw SNP calls finally variant quality score recalibration. Reference Genome of GATK hg19 is not supported b37 is used - to keep up to date with dbSNP and the 1000 Genomes Project data files Resources Download: GSA FTP server: location: ftp.broadinstitute.org username: gsapubftp-anonymous password: <blank> Raw Data Processing raw fastq file NGS reads aligner For Illumina data: recommend BWA - accurate, fast, well-supported, opensource, and emits BAM files natively Raw BAM to realigned, recalibrated BAM Purpose of realignment - locally realign reads such that the number of mismatching bases is minimized across all the reads - In general, a large percent of regions requiring local realignment are due to the presence of an insertion or deletion (indels) in the individual’s genome with respect to the reference genome. Such alignment artifacts result in many bases mismatching the reference near the misalignment, which are easily mistaken as SNPs. Two steps of realignment - Step 1: Determining (small) suspicious intervals which are likely in need of realignment - Step 2: Running the realigner over those intervals Raw BAM to realigned, recalibrated BAM Two types of realignment - Realignment only at known sites very efficient can operate with little coverage can only realign reads at known indels - Fully local realignment uses mismatching bases to determine if a site should be realigned, and relies on sufficient coverage to discover the correct indel allele in the reads for alignment much slower (involves SW step) can discover new indel sites in the reads Raw BAM to realigned, recalibrated BAM Purpose of base quality recalibration - After recalibration, the quality scores in the QUAL field in each read in the output BAM are more accurate in that the reported quality score is closer to its actual probability of mismatching the reference genome - the recalibration tool attempts to correct for variation in quality with machine cycle and sequence contex - more accurate quality scores Base Quality Recalibration: analyzing the covariation among several features of a base. e.g. - Reported quality score - The position within the read - The preceding and current nucleotide observed by the sequencing machine - Probability of mismatching the reference genome these covariation recalibrate the quality scores of all reads in a BAM file recommendation: lane-level recalibration, sample-level realignment Initial variant discovery and genotyping Input BAMs for variant discovery and genotyping - already have a single realigned, recalibrated, dedupped BAM per sample, called sampleX.bam, for X from 1 to N samples in your cohort. Multi-sample SNP and indel calling - apply the Unified genotyper to identify sites among the cohort samples. This will produce a multi-sample VCF file, with sites discovered across samples and genotypes assigned to each sample in the cohort. - Note: by default the Unified Genotyper calls SNPs only. To enable the indel calling capabilities instead use the -glm DINDEL argument. Initial variant discovery and genotyping Selecting an appropriate quality score threshold - A common question is the confidence score threshold to use for variant detection. - Recommend: Deep (> 10x coverage per sample) data recommend a minimum confidence score threshold of Q30 with an emission threshold of Q10. These Q10-Q30 calls will be emitted filtered out as LowQual. Shallow (< 10x coverage per sample) data recommend a min. confidence score of Q4 and an emission threshold of Q3, since variants have by necessity lower quality with shallower coverage. Initial variant discovery and genotyping Protocol VCF (variant call format) - standarised format for storing the most prevalent types of sequence variation, including SNPs, indels and larger structural variants, together with rich annotations - usually stored in a compressed manner, and can be indexed for fast data retrieval of variants from a range of positions on the reference genome - VCFtools: a software suite that implements various utilities for processing VCF files, including validation, merging and comparing… - http://vcftools.sourceforge.net Initial variant discovery and genotyping VCF (variant call format) Initial variant discovery and genotyping Integrating analyses: getting the best call set possible Problems of raw VCF file - raw VCF will have many sites that aren't really genetic variants but are machine artifacts that make the site statistically non-reference - should separate out the FP machine artifacts from the TP genetic variants ! Tools: - VariantFiltrationWalker: apply hard filters - Variant quality score recalibration: build an adaptive error model using known variant sites and then apply this model to estimate the probability that each variant is a true genetic variant or a machine artifact. Recommend: Regardless of whether you'll ultimately apply hard filtering or adaptive error modeling to select your final calls, first apply some common SNP filters to avoid obvious misalignment and indel artifacts. Integrating analyses: getting the best call set possible Analysis read VCF protocol: Integrating analyses: getting the best call set possible Basic indel filtering: - purpose: remove alignment artifacts from the data - methods: flagging variants with high strand bias and in poorly mapped regions (HARD_TO_VALIDATE set) with more than 10% of the reads having mapping quality 0 - arguments for VariantFiltrationWalker: Integrating analyses: getting the best call set possible Basic SNP filtering: - purpose: remove alignment artifacts from the data - methods: flagging SNPs within clusters (3 SNPs with 10 bp of each other) and those in poorly mapped regions (HARD_TO_VALIDATE set) with more than 10% of the reads having mapping quality 0 - arguments for VariantFiltrationWalker: Integrating analyses: getting the best call set possible Filtering around indels - Purpose: It's possible that, despite even local realignment, misalignments around true and artifactual indels will result in some false SNP calls. These errors are quite common if you didn't do local realignment, didn't provide a set of known indels during local realignment, and around very large indels that can't be modeled properly by local realignment. - methods: perform indel calling, then you can filter your SNP calls around the raw indel calls from your data set - arguments for VariantFiltrationWalker: Integrating analyses: getting the best call set possible Making analysis ready calls SNP calls with hard filtering - GATK recommended hard filtering: arguments for VariantFiltrationWalker: Integrating analyses: getting the best call set possible Making analysis ready calls with variant quality score recalibration - newly developed - An alternative approach to hard filtering: Variant quality score recalibration - methods: assign a well-calibrated probability to each variant call in a call set. One can then create highly accurate call sets by filtering based on this single estimate for the accuracy of each call. Expected SNP call quality Using GATK walker : VariantEval - giving sensitivity, specificity, and Ti/Tv ratios for known and novel calls - Expected Ti/Tv ratios: evaluating the quality of SNP calls whole genome, or in the targeted whole exome (Agilent), or interested regions Reference Mark A et,al. A framework for variation discovery and genotyping using next-generation DNA sequencing data. Nature Genetics, 43: 491-498, 2011 Mark A et,al. The Genome Analysis Toolkit: a MapReduce framework for analyzing nextgeneration DNA sequencing data. Genome Res. 20:1297-303, 2010 Wiki: GATK http://www.broadinstitute.org/gsa/wiki/index.php/Main_Page Best Practice Variant Detection with the GATK v2 http://www.broadinstitute.org/gsa/wiki/index.php/Best_Practice_Variant_Detection_with_the_GATK_v2 1000 Genomes: A Deep Catalog of Human Genetic Variation http://www.1000genomes.org/wiki/Analysis/Variant%20Call%20Format/vcf-variant-call-format-version-40 VCF poster: The Variant Call Format and VCFtools, by Petr Danecek et. al. http://vcftools.sourceforge.net/VCF-poster.pdf VCFtools http://vcftools.sourceforge.net Wikipedia http://en.wikipedia.org/ Thanks for Attention !