Meta-analysis of neuroimaging data Tor D. Wager Columbia University
advertisement
SCANLab Meta-analysis of neuroimaging data What, Why, and How Tor D. Wager Columbia University http://www.columbia.edu/cu/psychology/tor/ Uses of meta-analysis in neuroimaging • Meta-analysis is an essential tool for summarizing the vast and growing neuroimaging literature Wager, Lindquist, & Hernandez, in press SCANLab http://www.columbia.edu/cu/psychology/tor/ Uses of meta-analysis in neuroimaging • Assess consistency of activation across laboratories and task variants • Compare across many types of tasks and evaluate the specificity of activated regions for particular psychological conditions • Identify and define boundaries of functional regions SCANLab • Co-activation: Develop models of functional systems and pathways Wager, Lindquist, & Kapan, 2007 http://www.columbia.edu/cu/psychology/tor/ Functional networks in meta-analysis • Use regions or distributed networks in a priori tests in future studies SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of cognitive control Authors Chein et al. Year Method 2002 Density (Gaussian) Clustering of peaks, Wager et al. 2003 chi-square Wager et al. 2004 KDA, Spatial MANOVA Buchsbaum 2005 ALE et al. Chein & 2005 Density (Gaussian) Schneider Laird et al. 2005 ALE Psychological focus Verbal working memory SCANLab Working memory Attention/task switching Wisconsin Card Sorting Practice effects in cognitive control Stroop interference Owen et al. 2005 ALE N-back working memory Neumann et ALE, Co-activation 2005 Stroop interference al. "replicator dynamics" Costafreda 2006 Spatial location Verbal fluency in left IFG et al. Gilbert et al. 2006 Spatial location/Chisquare/classifier Episodic memory, Multitasking, Mentalizing in BA 10 Nee et al. KDA, logistic regression Cognitive control/interference 2007 Van 2008 Snellenberg MKDA/KDA * & Wager Cognitive control and memory http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of emotion & motivation SCANLab Authors Year Method Chi-square within Phan et al. 2002 regions Murphy et Spatial location (K-S 2003 al. Test) Wager et al. 2003 KDA, Chi-square Kringelbach 2004 Spatial location et al. Psychological focus Phan et al. 2004 Qualitative Emotion Baas et al. Northoff et al. 2004 Chi-square Amygdala lateralization 2005 Clustering of peaks Self-referential processes Krain et al. 2006 ALE Decision-making Wager et al. 2008 MKDA, Chi-square Kober et al. 2008 MKDA, Co-activation * Emotion Emotion Emotion Reinforcers in OFC Emotion Emotion http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of disorders Authors Zakzanis et al. Zakzanis et al. Whiteside et al. Glahn et al. Year Method Psychological focus SCANLab 2000 Effect sizes Schizophrenia 2003 Effect sizes Alzheimer's disease 2004 Effect sizes Obsessive-compulsive disorder 2005 ALE Working memory in schizophrenia Fitzgerald et 2006 ALE al. Dickstein et 2006 ALE al. Van Snellenberg 2006 Effect sizes et al. Spatial location Steele et al. 2007 ("unwarped") Depression, DLPFC ADHD Schizophrenia and working memory Depression, frontal cortex Valera et al. 2007 Effect sizes Brain structure in ADHD Etkin & Wager 2007 MKDA, Co-activation Anxiety disorders Hoekert et al. 2007 Effect sizes Emotional prosody in schizophrenia http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of language SCANLab Authors Turkeltaub et al. Year Method 2002 ALE Psychological focus Single-word reading Jobard et al. 2003 Clustering of peaks Word reading Brown et al. 2005 ALE Vigneau et 2006 Clustering of peaks al. ALE, Co-activation Ferstl et al. 2008 "replicator dynamics" Turkeltaub 2002 ALE et al. Jobard et al. 2003 Clustering of peaks Speech production Language, left cortical hemisphere Brown et al. 2005 ALE Speech production Vigneau et al. 2006 Clustering of peaks Language, left cortical hemisphere Ferstl et al. 2008 Text comprehension Single-word reading Word reading ALE, Co-activation Text comprehension "replicator dynamics" http://www.columbia.edu/cu/psychology/tor/ Meta-analyses of other stuff SCANLab Authors Joseph Grezes & Decety Kosslyn & Thompson Year Method 2001 Spatial location 2001 Qualitative Action 2003 Logistic regression Visual imagery Nielsen et al. 2004 Gottfried & Zald Nickel & Seitz Petacchi et al. Psychological focus Object recognition: category specificity Kernel density/multivariate Cognitive function 2005 Spatial location Olfaction in OFC 2005 Clustering of peaks Parietal cortex 2005 ALE Auditory function, cerebellum Average maps in CARET Lewis 2006 Tool use Postuma & Dagher 2006 Co-activation Basal ganglia Zacks 2008 Mental rotation http://www.columbia.edu/cu/psychology/tor/ SCANLab Using meta-analysis to evaluate consistency: Why? http://www.columbia.edu/cu/psychology/tor/ Locating emotion-responsive regions 164 PET/fMRI studies, 437 activation maps, 2478 coordinates SCANLab http://www.columbia.edu/cu/psychology/tor/ Why identify consistent areas? • Making statistic maps in neuroimaging studies involves many tests (~100,000 per brain map) • Many studies use uncorrected or improperly corrected p-values SCANLab Long-term Memory # of Maps P-value thresholds used Uncorrected Corr. How many false positives? A rough estimate: 663 peaks, 17% of reported activations Wager, Lindquist, & Kaplan, 2007 http://www.columbia.edu/cu/psychology/tor/ Consistency SCANLab Emotion: 163 studies Consistently Activated Reported peaks regions 163 studies http://www.columbia.edu/cu/psychology/tor/ Ventral surface Lateral surface (R) Medial surface (L) SCANLab vmPFC Gyrus rectus Central sulcus dmPFC pOFC BF rdACC Pre SMA pgACC dmPFC PCC OCC CM, MD sgACC mTC sTC aINS latOFC lFG Kober et al., in press, NI Deep nuclei TC vmPFC http://www.columbia.edu/cu/psychology/tor/ SCANLab Using meta-analysis to evaluate specificity: Why? http://www.columbia.edu/cu/psychology/tor/ Disgust responses: Specificity in insula? Insula SCANLab http://www.columbia.edu/cu/psychology/tor/ Disgust responses: Specificity in insula? SCANLab Search Area: Insula Feldman-Barrett & Wager, 2005; Phan, Wager, Taylor, & Liberzon, 2002; Phan, Wager, Liberzon & Taylor, 2004 http://www.columbia.edu/cu/psychology/tor/ Meta-analysis plays a unique role in answering… SCANLab The Neural Correlates of Task X • Is it reliable? – Would each activated region replicate in future studies? – Would activation be insensitive to minor variations in task design? • Is it task-specific? – Predictive of a particular psychological state or task type? – Diagnostic value? http://www.columbia.edu/cu/psychology/tor/ SCANLab Using meta-analysis to evaluate consistency: How? http://www.columbia.edu/cu/psychology/tor/ Meta-analysis: Multilevel kernel density estimate (MKDE) Monte Carlo: Expected maximum proportion Under the null hypothesis Peak coordinate locations (437 maps) Permute blobs within study maps … Damasio, 2000 Liberzon, 2000 Kernel convolution SCANLab Wicker, 2003 Apply threshold E Significant regions Weighted average … Comparison indicator maps Wager, Lindquist, & Kaplan, 2007; Etkin & Wager, in press Proportion of activated Comparisons map (from 437 comparisons) http://www.columbia.edu/cu/psychology/tor/ MKDA: Key points SCANLab • Statistic reflects consistency across studies. Study comparison map is treated as a random effect. Peaks from one study cannot dominate. • Studies are weighted by quality (see additional info on handouts for rationale) • Spatial covariance is preserved in Monte Carlo. Less sensitive to arbitrary standards for how many peaks to report. http://www.columbia.edu/cu/psychology/tor/ Whether and how to weight studies/peaks MKDA analysis weights by sqrt(sample size) and study quality (including fixed/random effects) Study quality weight Weighted proportion of P CIM c N c c Sample size for activating c c N c map c c studies SCANLab Weighted average Activation indicator (1 or 0) for map c c 1 Fixed effects c 0.75 Random effects http://www.columbia.edu/cu/psychology/tor/ Monte Carlo Simulation • Simulation vs. theory (e.g. Poisson process) • Simulation allows: SCANLab – Non-stationary spatial distribution of peaks (clumps) under null hypothesis; randomize blob locations – Family-wise error rate control with irregular (brain-shaped) search volume – Cluster size inference, given primary threshold Monte Carlo: E(max(P|H0)) http://www.columbia.edu/cu/psychology/tor/ Compare with Activation Likelihood Estimate (ALE), Kernel Density Analysis (KDA) Peak coordinates Combined across studies Kernel convolution Density kernel OR ALE kernel SCANLab Apply significance threshold Peak density or Significant results ALE map Ignores the fact that some studies report more peaks than others! Density kernel: Chein, 1998; Phan et al., 2002; Wager et al., 2003, 2004, 2007, in press Gaussian density kernel + ALE: Turkeltaub et al., 2002; Laird et al., 2005; others http://www.columbia.edu/cu/psychology/tor/ Comparison with other methods MKDA KDA/ALE SCANLab • Statistic reflects consistency • Peaks are lumped together, study across studies. Study comparison is fixed effect. Peaks from one map is treated as a random study can dominate, studies that effect. Peaks from one study report more peaks dominate. cannot dominate. • Studies are weighted by quality • No weighting, or z-score weighting (problematic) • Spatial covariance is preserved in • Spatial covariance is not Monte Carlo. Less sensitive to preserved in Monte Carlo. arbitrary standards for how many Effects of reporting standards peaks to report. large. See handouts for more comparison points http://www.columbia.edu/cu/psychology/tor/ ALE approach • Treats points as if they were Gaussian probability distributions. • Summarize the union of probabilities at each voxel: probability of any peak “truly” lying in that voxel SCANLab P(X1 X 2 ... X n ) 1 P(X) 1 P(X1) * P(X 2 ) * ...P(X n ) P(Xi ) is the probability that peak Xi lies in a given voxel The bar indicates the complement operator Null hypothesis: P(X) 0 No peaks lie in voxel Alt hypothesis: voxel P(X) 0 At least one peak lies in http://www.columbia.edu/cu/psychology/tor/ ALE meta-analysis • Analyst chooses smoothing kernel • ALE analysis with zero smoothing: SCANLab – Every voxel reported in any study is significant in the meta-analysis • Test case: 3-peak meta analysis, one peak activates in voxel: P(X1) 1,P(X2 ) 0,P(X3 ) 0 ALE statistic: Highest 1 Pr(X ) 1 (0)* (1)* (1) 1 possible value! • In practice: 10 – 15 mm FWHM kernel http://www.columbia.edu/cu/psychology/tor/ Comparison across methods: Inference Property KDA ALE Multilevel KDA Kernel Spherical Gaussian Spherical Interpretation of statistic Null hypothesis Num nearby peaks Interpretation of significant result Assumptions Generalize to SCANLab Prob. that at least one peak nearby Peaks are not No peaks truly spatially consistent activate More peaks lie near One or more peaks voxel than lies at this voxel expected by chance Num. study maps activating nearby Study maps are not spatially consistent A higher proportion of studies activate near voxel than expected by chance 1. Study is fixed 1. Study is fixed Activation ‘blobs’ effect (homogenous effect (homogenous are spatially sample of studies) sample of studies) independent under 2. Peaks are 2. Peaks are the null hypothesis spatially spatially independent under independent under the null hypothesis the null hypothesis New peaks from New peaks from New study maps same studies same studies http://www.columbia.edu/cu/psychology/tor/ Comparison: Correction and Weighting SCANLab Property KDA ALE Multilevel KDA Multiple comparisons FWER FDR Weighting None, or weight peaks by z-score None FWER (recommended) or FDR Weight studies by sample size, fixed/random effects, quality http://www.columbia.edu/cu/psychology/tor/ Density analysis: Summary SCANLab Working memory Executive WM Long-term memory Memory Inhibition Task switching Response selection Wager et al., 2004; Nee, Wager, & Jonides, 2007; Wager et al., in press; Van Snellenberg & Wager, in press http://www.columbia.edu/cu/psychology/tor/ SCANLab Using meta-analysis to evaluate specificity: How? http://www.columbia.edu/cu/psychology/tor/ Specificity • Task-related differences in relative activation frequency across the brain: SCANLab – MKDA difference maps (e.g., Wager et al., 2008) • Task-related differences in absolute activation frequency – Nonparametric chi-square maps (Wager, Lindquist, & Kaplan, 2007) • Classifier systems to predict task type from distributed patterns of peaks (e.g., Gilbert) http://www.columbia.edu/cu/psychology/tor/ MKDA Difference maps: Emotion example SCANLab Experienced Perceived • Approach: – Calculate density maps for two conditions, subtract to get difference maps – Monte Carlo: Randomize blob locations within each study, recalculate density difference maps and save max – Repeat for many (e.g., 10,000) iterations to get max distribution http://www.columbia.edu/cu/psychology/tor/ – Threshold based on Monte Carlo simulation Emotion example: Selective regions Experience > Perception Perception > Experience OFC SCANLab Amy TP IFG Hy Midb OFC aIns TP OFC vaIns dmPFC Amy mOFC Hy vaIns IFG pgACC PAG Hy Midb PAG TP TP CB Amy OFC CB aIns Amy Wager et al., in press, Handbook of Emotion http://www.columbia.edu/cu/psychology/tor/ Task-brain activity associations in meta-analysis SCANLab Study contrast map Region/V Task oxel 1 condition Study 1 1 Disgust Study 2 0 Fear Study 3 1 Disgust Study 4 1 Happiness Study 5 0 Anger … … … Study N 0 Sadness Measures of association: Chi-square • But requires high expected counts (> 5) in each cell. Not appropriate for map-wise testing over many voxels Fisher’s exact test (2 categories only) Multinomial exact test • Computationally impractical! Nonparametric chi-square • Approximation to exact test • OK for low expected counts http://www.columbia.edu/cu/psychology/tor/ Nonparametric chi-square: Details SCANLab Study contrast map Region/V Task oxel 1 condition Study 1 1 Disgust Study 2 0 Fear Study 3 1 Disgust Study 4 1 Happiness Study 5 0 Anger … … … Study N 0 Sadness Idea of exact test: • Conditionalize on marginal counts for activation and task conditions. • Null hypothesis: no systematic association between activation and task • P-value is proportion of nullhypothesis possible arrangements that can produce distribution across task conditions as large as observed or larger. http://www.columbia.edu/cu/psychology/tor/ Nonparametric chi-square: Details SCANLab Study contrast map Region/V Task oxel 1 condition Study 1 0 1 Study 2 Study 3 Study 4 Study 5 … Study N 0 1 0 … 1 Disgust Fear Disgust Happiness Anger … Sadness Permutation test: • Permute activation indicator vector, creating null-hypothesis data (no systematic association) • Marginal counts are preserved. • Test 5,000 or more samples and calculate P-value based on observed null-hypothesis distribution http://www.columbia.edu/cu/psychology/tor/ Density difference vs. Chi-square • Relative vs. absolute differences SCANLab Experience Perception Voxels (one-dimensional brain) Chi-square Density http://www.columbia.edu/cu/psychology/tor/ Can we predict the emotion from the pattern of brain activity? Experienced SCANLab Perceived • Approach: predict studies based on their pattern of reported peaks (e.g., Gilbert, 2006) • Use naïve Bayesian classifier (see work by Laconte; Tong; Norman; Haxby). Cross-validate: predict emotion type for new studies that are not part of training set. http://www.columbia.edu/cu/psychology/tor/ Classifying experienced emotion vs. perceived emotion: 80% accurate SCANLab Experience EXP vs. PER DMPFC vs. Pre-SMA PAG vs. Ant. thalamus Perception Deep cerebellar nuc. vs. Lat. cerebellum http://www.columbia.edu/cu/psychology/tor/ Outline: Why and How… • Consistency: Replicability across studies SCANLab – Consistency in single-region results: MKDA – Consistency in functional networks: MKDA + Co-activation • Specificity and “reverse inference” – Brain-activity – psychological category mappings for individual brain regions: MKDA difference maps; Nonparametric Chi-square – Brain-activity – psychological category mappings for distributed networks Applying classifier systems to meta-analytic data http://www.columbia.edu/cu/psychology/tor/ Extending meta-analysis to connectivity SCANLab Study contrast map Region/V oxel 1 Region/V oxel 2 Study 1 1 0 Study 2 0 0 Study 3 1 1 Study 4 1 1 Study 5 0 0 … … … Study N 0 1 N = 45 Region 1 No Region 1 Yes Region 2 Yes 6 23 Region 2 No 12 4 Co-activation: If a study (contrast map) activates within k mm of voxel 1, is it more likely to also activate within k mm of voxel 2? Measures of association: Kendall’s Tau-b Fisher’s exact test Nonparametric chi-square Others… http://www.columbia.edu/cu/psychology/tor/ Kendall’s Tau: Details • Ordinal “nonparametric” association between two variables, x and y • Uses ranks; no assumption of linearity or normal distribution (Kendall, 1938, Biometrika) • Values between [-1 to 1], like Pearson’s correlation SCANLab N 1 4 min( rank(x), rank(y)) i 1 i1 N(N 1) Tau is proportion of concordant pairs of observations sign(x diff. between pairs)= sign(y diff. between pairs) Tau = (# concordant pairs - # discordant pairs) / total # pairs http://www.columbia.edu/cu/psychology/tor/ Meta-analysis functional networks: Examples • Emotion: Kober et al. (in press), 437 maps SCANLab http://www.columbia.edu/cu/psychology/tor/ Meta-analysis of emotion Acknowledgements Lisa Feldman Barrett Statistics Martin Lindquist Meta-analysis of cognitive function SCANLab Ed Smith Tom Nichols Luan Phan Steve Taylor Israel Liberzon Students Hedy Kober Lauren Kaplan Jason Buhle Jared Van Snellenberg Derek Nee John Jonides Ed Smith Funding agencies: National Science Foundation National Institute of Mental Health http://www.columbia.edu/cu/psychology/tor/ SCANLab Weighting http://www.columbia.edu/cu/psychology/tor/ Whether and how to weight studies/peaks • Studies (and peaks) differ in sample size, methodology, analysis type, smoothness, etc. • Advantageous to give more weight to more reliable studies/peaks SCANLab • Z-score weighting – Advantages: Weights nominally more reliable peaks more heavily – Disadvantages: Small studies can produce variable results. Reporting bias: High z-score peaks are high partially due to error; “capitalizing on chance” • Must convert to common Z-score metric across different analysis types in different studies http://www.columbia.edu/cu/psychology/tor/ Whether and how to weight studies/peaks • Alternative: Sample-size weighting SCANLab – Advantages: • Weights studies by the quality of information their peaks are likely to reflect • Avoids overweighting peaks reported due to “capitalizing on chance” – Disadvantages: Ignores relative reliability of various peaks within studies http://www.columbia.edu/cu/psychology/tor/ SCANLab MKDA vs. KDA vs. ALE: Comparison chart http://www.columbia.edu/cu/psychology/tor/ SCANLab More details on reverse inference http://www.columbia.edu/cu/psychology/tor/ Is brain activity diagnostic of a particular psychological state? SCANLab Pleasure? Punishing wrongdoers Forward inference Reverse inference Brain activity P(Brain | Psy) Given a psychological We observe state brain activity P(Psy | Brain) Can we infer Given brain psychological pleasure? activity ‘Forward’ and ‘reverse’ inference are not the same! Reverse inference requires comparing across many psychological states! http://www.columbia.edu/cu/psychology/tor/ The predictive value problem: Worked example For a brain region to be used as a marker of pleasure SCANLab – The brain region must respond consistently to pleasure – The brain region must respond specifically to pleasure (not activated by other things) Non-pleasure Pleasure P(Brain|no pleasure) = .4 1-Specificity Ventral caudate P(pleasure) = .1 Prior P(Brain|Pleasure) = .9 Forward inference; Sensitivity Caculate reverse inference: P(Pleasure|Brain) = .2 http://www.columbia.edu/cu/psychology/tor/ SCANLab More details on connectivity http://www.columbia.edu/cu/psychology/tor/ SCANLab More details on MKDA difference maps and nonparametric chi-square maps http://www.columbia.edu/cu/psychology/tor/
