C2 Training: May 9 – 10, 2011
Introduction to meta-analysis
The Campbell Collaboration
www.campbellcollaboration.org
Pooled effect sizes
• Average across studies
• Calculated using inverse variance weights
– Studies with more precise estimates (larger N, smaller sd)
contribute more to overall average than those with less precise
estimates
• Choice between fixed and random effects models (and
mixed models): 1. a priori expectations, 2. statistical tests
– Do we expect studies to estimate a single population parameter?
– If Yes, use fixed effect model and test homogeneity assumption
– Usually use random effect model or mixed model
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Combining effect sizes across studies
• Compute effect sizes within each study
• Create a set of independent effect sizes
• Compute weighted mean and variance of effect sizes
• Compute 95% confidence interval for weighted mean
effect size
• Test for the homogeneity of effect sizes
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Create a set of independent effect sizes
• Likely to have multiple effect sizes per study
• Effect sizes within a study can be:
– Multiple measures of study participants
– Measures of independent groups of participants
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
What to do with multiple effect sizes per study?
• Use only independent groups in each analysis
– Use only one measure of the study participants in each analysis
– Use results within studies that are derived from independent
groups of study participants
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Compute weighted mean and variance of effect sizes
• Compute individual study effect sizes
• Correct effect sizes for any biases, e.g., Hedges correction
• Compute study effect size variance
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
(75  1)5.742  (65  1)4.162
sp 
 25.69  5.06
(75  65  2)
32.55  32.40
ESsm 
 0.03
5.06
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
p. 62 of MST pdf
75  65
(0.03)
1
SE sm 

 0.17, w i 
 34.60
2
75*65 2(75  65)
(0.17)
2
95% CI :(0.03  1.96*0.17)  [ 0.30, 0.36]
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Use weighted mean for effect sizes
• Use weighted means because each effect size has a
different variance that depends on the study’s sample size
and effect size value
• Weight all analyses by the inverse of the effect size variance
or
1
wi 
2
(SE ES )
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Weighted Mean Effect Size
k is the number of effect sizes
k
w
ES =
i
ESi
i=1
k
w
i
i=1
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
ESsm  0.14
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Standard error of weighted mean effect size
SEES =
1
k
w
 i
i=1
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Confidence Interval for Mean Effect Size
α = Significance level, z = Critical value from
standard normal distribution
ESL  ES - z (1-α) (SE ES )
ESU  ES  z (1-α) (SE ES )
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
SEES  0.10
ES L  0.14 1.96*0.10   0.06
ESU  0.14  1.96*0.10  0.34
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Test of Homogeneity
• Statistical test that addresses whether the k effect sizes that
are averaged into a mean value all estimate the same
population effect size
• In a homogeneous distribution, effect sizes differ from
population mean only by sampling error
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Assessment and adjustments for bias
• Small sample bias – Hedges’ g correction for SMD
• Range restriction – Hunter & Schmidt
• Missing data – sensitivity analysis, concerns about integrity
of intent-to-treat analysis (missing cases), outcome reporting
bias (missing data)
• Publication bias – missing studies (and missing data for
available studies)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Assessing risk of publication bias
1.
Funnel plots – plot study effect sizes by their standard
errors
– “interoccular analysis” of funnel plots is unreliable
2. Trim and fill analysis (need ~ 10+ studies)
3. Statistical tests (Egger’s test and others)
Do NOT use Failsafe N (Becker, 2005)
See Rothstein, Sutton & Bornstein (2005)
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Homogeneity/heterogeneity tests
Statistical tests of homogeneity of results across studies
• Are all studies estimating a common population parameter
with
• Differences between studies due to chance (sampling error)
alone?
Homogeneity tests performed under the fixed effect model
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Form of the homogeneity test
k
Q=
 w (ES - ES)
i
2
i
i=1
When we reject the null hypothesis of
homogeneity, the variability of the effect
sizes is more than would be expected from
sampling error
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Computational form


w
ES

i
i

k


2
Q =   w i ESi  -  i=1 k
 i=1

 wi
k
2
i=1
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
To Test Homogeneity
• Compare Q to the (1 – α) critical value of the chi-square
distribution with k-1 degrees of freedom
• Significant Q = heterogeneity
• Non-significant Q = homogeneity
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
 (3)  3.52, p  0.32, n.s.
2
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Preliminary analysis
• Graph effect sizes and 95% CI
• Compute overall mean effect size and 95% CI
• Compute homogeneity test
• Interpret findings
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Forest plots
• Easy to produce from raw data
• Cochrane’s RevMan software
free, downloadable at:
http://www.cc-ims.net/revman
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
p. 59 of MST pdf
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Moderator models: ANOVA
• When we find that a group of studies are heterogeneous, we
can explore whether moderator variables explain this
variation
• When we have continuous moderators, we use regression
models
• When we have categorical moderators, we use ANOVA
• Caution needed:
– Moderator analysis is correlational
– Moderators may be confounded within studies
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Example from Sirin paper
Sirin, S. R. (2005). Socioeconomic status and academic achievement: A
meta-analytic review of research. Review of Educational Research, 75,
417-453.
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org
Group exercise
• Using one of the studies provided from the MST review,
code the elements in Level 4. Choose at least 2 outcomes
from the study.
C2 Training Materials – Oslo – May 2011
www.campbellcollaboration.org