http://theveryclosedcircle.blogspot.com/2012/04/360-meta-analysis.html
Special Topics in Biological Data
Analysis:
Meta-analysis
Jarrett Byrnes
UMass Boston Spring 2014
IS ONE STUDY EVER DEFINITIVE?
Tilman et al. 1997 Science
Huston et al. 2000 Nature
HOW CAN WE DETERMINE THE
ANSWER?
Narrative Reviews
• Author searches and summarizes evidence
from their perspective
• Can be biased by author’s own opinion or
searching abilities (if they did a search)
“In a strictly defined sample of competition studies using controlled field experiments, covering
215 species and 527 experiments competition was found in most of the studies, in somewhat
more than half of the species, and in about two-fifths of the experiments.”
Connell 1983
“Rare until recently, field-experimental studies of interspecific competition now
number well over 150. Competition was found in 90% of the studies and 76% of
their species, indicating its pervasive importance in ecological systems.”
Schoener 1983
Systematic Review
• Defined repeatable criteria for searching
• Clear inclusion/exclusion strategy
• Systematic coding/grading of results from
studies
• Not often done outside of medicine
• But see http://www.environmentalevidencejournal.org/
http://www.campbellcollaboration.org/
http://www.cochrane.org/
Vote-Counting
“Rare until recently, field-experimental studies of interspecific
competition now number well over 150. Competition was found in 90% of
the studies and 76% of their species, indicating its pervasive importance
in ecological systems.”
Schoener 1983
• Says nothing about strength of effect
• Weak evidence = no evidence, biased by
experiments with large and small sample sizes
One Giant P-Value
• One study contains p(data|Hypothesis) for
one data set
• Why not combine them?
For example:
p=1-pnorm(Szi, m=0, s=1)
• Unequal sample sizes, methodologies,
assumptions, etc……
Meta-analysis!
An analysis of analyses!
Systematic combination of information
from published studies to derive the
weight of evidence to a particular question
The First Meta-Analysis
Pearson, K. (1904). Report on certain enteric
fever inoculation statistics. British Medical
Journal, 3, 1243-1246.
1970's: Dawn of Modern Meta-analysis
• First, medical: Anti-coagulants as a treatment
for heart attacks (Chalmers 1977)
• Psychotherapy (Smith and Glass 1977)
• Education: Class size and outcomes (Glass and
Smith 1979)
• Opposition: "an exercise in Mega-Silliness"
Why would you object to metaanalysis as a tool?
Characteristics of Meta-Analysis
• Systematic search for studies
• Strict criteria for inclusion
• Common effect size
D=
Example: Hedge's D
Treatment - Control
Jm
SDPooled
SDPooled =
(n t -1)sdt + (n c -1)sdc
nt + nc - 2
• Analysis that accounts for difference in power
of different studies
Why did it take so long to get to Ecology?
eigenfactor.org
Example 1: Strength of Competition
D=
Treatment - Control
JM
SDPooled
SDPooled =
(n t -1)sdt + (n c -1)sdc
nt + nc - 2
Gurevitch et al. 1992
Example 2: Does Diversity Influence
Ecosystem Function
Cardinale et al., Nature 2006
93 studies  175 experiments
LRAve = ln(yp / ym)
LRMax = ln(yp / ymax)
Standing stock
or
Resource depletion
Monoculture, m
Most diverse
polyculture, p
Species richness
Standing stock of t
Effect size
2
A.
Resource depletion by t
B.
Trophic group t (resource)
Plants (nutrients or water)
Herbivores (live plant tissue)
Predators (live prey)
Detritivores (dead organic matter)
0
-2
Effect size
1
C.
D.
Aquatic
Terrestrial
0
34 41
38 32
34
35
28
39
-1
LRm
LRm
LRAve LRMax
LRm
LRm
LRAve LRMax
Y=
4
Standing stock
(YS / Ym)
3
Ymax S
K S
3
2
2
1
1
0
Ymax
K
0
4
3
Resource depletion
3
(YS / Ym)
2
2
1
1
0
Ymax
0
0
10
20
30
40
Species richness, S
80
K
Example 3: Are Seagrass Beds Changing?
Waycott et al. 2009 PNAS
Example 3: Are Seagrass Beds Changing?
Waycott et al. 2009 PNAS
For LOTS more in EEB
• Ecology Volume 80 Issue 4 in 1999
• Ecology Volume 91 Issue 9 in 2010: Is
meta-analysis a mega-mistake when
it comes do diversity and
productivity?
SO WHAT ARE WE DOING HERE?
Course Goals
1. Learn how to do a meta-analysis
2. DO a meta-analysis
– and publish it!
3. Learn how to teach yourself new statistical
skills
4. Learn how to participate in a working group
type of environment
Course Outline
• Now-March: Gathering and
Assessing Data
• March-April: Analytic Techniques
• April-May: (not so) Unique Challenges
Class Structure
• 1-2 or 2:15 You lecture on the topic of the
week
• 2:15-2:45 Code or other demo (if needed)
• 2:45-4: Working group
I’m going to lecture?!
• You will encounter many new analytic
techniques in the future
• This class provides an introduction to learning
a new topic
• Meet with me at 1pm on Tuesday to go over
your lecture for the week
• I’ll be lecturing on some of the hairy stuff &
all of the code
a Requirement
•
•
•
•
We will use R for analysis
metafor package for meta-analysis
nlme and lmer for mixed models
MCMCglmm and JAGS for Bayesian methods
http://www.metafor-project.org/
http://swirlstats.com/
By March 1st!
http://www.nceas.ucsb.edu/handbook_of_meta-analysis
Other Books!
• Introduction to Meta-Analysis. 2009. Michael
Borenstein, Larry Hedges, Julian Higgins, and
Hannah R. Rothstein. Wiley.
• The Handbook of Research and Synthesis and
Meta-Analysis, 2nd Edition. Edited by Harris
Cooper, Larry Hedges, and Jeffry Valentine.
Russel Sage Foundation.
What will you do in this course?
Choose a Question
Search the Literature, with a Protocol
http://agris.fao.org/
(Re)Choose a Question
What will you do in this course?
Harvest Data
http://arohatgi.info/WebPlotDigitizer/app/
Design a Database
Generate Strict Meta-data
Go back to the beginning…
• As you begin to search, you will learn about
what information is out there
• Starting over (sometimes more than once) is
normal!
Be Circumspect: Assess Bias, Power,
Sensitivity, etc.
Analysis!
http://www.metafor-project.org/
Publish?
Is species diversity decreasing at
local scales in the sea?
Is species diversity decreasing at
local scales in the sea?
Problems with Vellend et al.
• Sampling bias
• Low power of test misses trend
• Didn't test fundamental assumptions of
conclusions
• Extrapolation from data they had to
conclusions about things not measured
• Terrestrial only
Is species diversity decreasing at
local scales in the sea?