Computing Random Effects Models
in Meta-analysis
Terri Pigott, C2 Methods Editor & co-Chair
Professor, Loyola University Chicago
tpigott@luc.edu
The Campbell Collaboration
www.campbellcollaboration.org
As I am going through the beginning slides:
• Please download RevMan if you do not already have it:
– http://ims.cochrane.org/revman/download
• Data is here:
https://my.vanderbilt.edu/emilytannersmith/training-materials/
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Fixed and Random Effects Models in Meta-analysis
• How do we choose among fixed and random effects models
when conducting a meta-analysis?
• Common question asked by reviewers working on
systematic reviews that include a meta-analysis
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Our goal today
• Provide a brief description of fixed and of random effects
•
•
•
•
models
Discuss how to estimate key parameters in the random
effects model
Do sample computations to illustrate how to obtain the
variance component when computing the mean effect size
Illustrate how to compute the random effects model in
RevMan
Share resources for these methods for reviewers
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
An advertisement
• Much of my own thinking on these issues has been clarified
by the following sources:
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H.
R. (2010). A basic introduction to fixed effect and random
effects models for meta-analysis. Research Synthesis
Methods, 1, 97-111.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H.
R. (2009). Introduction to meta-analysis. West Sussex, UK:
John Wiley.
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Our choice between the two models depends on:
• Our assumption about how the effect sizes vary in our meta-
analysis
• The two models are based on different assumptions about
the nature of the variation among effect sizes in our research
synthesis
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Fixed-Effect model
• Borenstein et al. adopt the wording of fixed effect (no “s”)
here because in a fixed effect model, we assume that the
effect sizes in our meta-analysis differ only because of
sampling error and they all share a common mean
• Our effect sizes differ from each other because each study
used a different sample of participants – and that is the only
reason for the differences among our estimates
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Picture from Borenstein et al. (2010)
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
In a fixed-effect model
• Note that the effect size from each study estimate a single
common mean – the fixed-effect
• We know that each study will give us a different effect size,
but each effect size is an estimate of a common mean,
designated in the prior picture as θ
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
In a random effects model
• We assume two components of variation:
– Sampling variation as in our fixed-effect model assumption
– Random variation because the effect sizes themselves are
sampled from a population of effect sizes
• In a random effects model, we know that our effect sizes will
differ because they are sampled from an unknown
distribution
• Our goal in the analysis will be to estimate the mean and
variance of the underlying population of effect sizes
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Another picture from Borenstein et al. (2010)
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
In a random effects model
• We see in the picture that each distribution has its own mean
that is sampled from the underlying population distribution of
effect sizes
• That underlying population distribution also has its own
variance, τ2 , commonly called the variance component
• Thus, each effect size has two components of variation, one
due to sampling error, and one from the underlying
distribution
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Some notation for shorthand
vi2 is the sampling variance for our effect size
 is the variance component,
2
the variance of our effect size distribution
Ti is our effect size (SMD, odds-ratio, correlation, etc)
 is the mean of the underlying effect size distribution
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
In fixed effects, we can write our model as
Ti    ei , where
2
i
ei ~ N (0, v )
So, each effect size estimates a single mean
effect, θ, and differs from this mean effect
by sampling error
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
In random effects, we can write our model as
Ti    ei  i , where
ei ~ N (0, v ) and i ~ N (0, )
2
i
2
Each effect size differs from the underlying
population mean, μ, due to both sampling
error and the underlying population
variance
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Given the prior slides, how do we choose?
• The fixed-effect model assumes only sampling error as the
source of variation among effect sizes
• This assumption is plausible when our studies are close
replications of one another, using the same procedures,
measures, etc.
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Choosing a model (continued)
• In the random effects model, we assume that our effect
sizes are sampled from an underlying population of effect
sizes
• We already assume that our studies will differ not only
because there are different participants, but also because of
differences in the way studies were conducted
• Thus, we often choose random effects models because we
anticipate variation among our studies for a number of
reasons
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Implications of the choice
• Recall that all our analyses in a meta-analysis are weighted
by the inverse of the variance of the effect size, i.e., by the
precision of the effect size estimate
• For a random effects model, the variance for each effect size
is equal to vi + τ2
• In other words, in a fixed effect model, we will more heavily
weight larger studies. In a random effects model, the larger
studies will not be weighted as heavily
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Example of a fixed effect analysis from RevMan
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Same data, random effects analysis
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
So, where are we now?
• We have discussed the underlying assumptions of fixed
effect and random effects models
• We have also talked about how we choose between the two
models
• Now we will talk about how to estimate the parameters of a
random effects model
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Review of steps for fixed effect basic meta-analysis
• Compute the effect size and its variance for each study
• Compute the study weight – the inverse of the variance
(1/var)
• Compute the weighted mean effect size and its standard
error (inverse of the sum of the weights)
• Compute the 95% confidence interval for the weighted mean
• Compute the test of homogeneity
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Goals in a basic meta-analysis using a random
effects model
• We want to estimate a mean effect size, μ, the mean effect
size from the underlying population
• We also want to estimate the variance of the underlying
effect size distribution, τ2
• Biggest difficulty: how to estimate τ2
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Estimating the variance component, τ2
• There are two main methods for computing τ2
– Variously called the method of moments, the DerSimonian/Laird
estimate
– Restricted maximum likelihood
• The method of moments estimator is easy to compute and is
based on the value of Q, the homogeneity statistic
• Restricted maximum likelihood requires an iterative solution
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Method of moments estimator
Q  (k  1)
 
, where
c
Q is the homogeneity statistic,
k is the number of effect sizes, and
2
c is based on the fixed effect weights, wi  1/ vi , and is
c
w
i

2
w
 i
w
i
This is the estimator used in many meta-analysis
programs such as RevMan and CMA
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
An example
• So before we do anything in our basic random effects meta-
analysis, we need to compute Q and the fixed effects
weights, wi = 1/vi , to get the variance component, τ2,.
• Once we have our τ2 , we can continue with our computation
of the mean effect size and its variance.
• We will use an example of a meta-analysis of gender
differences in transformational leadership
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Eagly, Johannesen-Schmidt & van Engen (2003)
• This synthesis examines the standardized mean difference
estimated in primary studies for the difference between men
and women in their use of transformational leadership.
• Transformational leadership involves “establishing onself as
a role model by gaining the trust and confidence of followers”
(Eagly et al. 2003, p. 570).
• The sample data is a subset of the studies in the full metaanalysis, a set of 24 studies that compare men and women
in their use of transformational leadership
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Data
Study name
Bass 1996-3
Bono & Judge 2000
Boomer 1994
Carless 1998
Church & Waclawski 1998
Church & Waclawski 1999
Church et al 1996
Cuadrado 2002
Daughtry & Finch 1997
Davidson 1996
Gillespie & Mann 2000
Hill 2000
Jantzi & Leithwood 1996
Judge & Bono 2000
Komives 1991
Kuchinke 1999
Leithwood & Jantzi 1997
Manning 2000
McGrattan 1997
Rosen 1993
Rozier & Hersh-C 1996
Sosik & Megerian 1999
Spreitzer et al. 2000
Wipf 1999
Campbell Collaboration Colloquium – May 2012
Effect size
-0.09
-0.1
-0.62
-0.17
0.61
0.2
-0.22
-0.17
-0.25
-0.15
-0.1
-0.36
-0.29
-0.13
0.31
0.09
-0.35
-0.08
-0.72
-0.44
-0.36
0
-0.21
-0.87
SE eff size N men
0.066436
420
0.148128
112
0.262187
30
0.083113
368
0.091648
456
0.150202
1236
0.117768
209
0.185405
65
0.147431
130
0.280935
27
0.252084
92
0.356387
29
0.104781
288
0.117224
134
0.077847
296
0.214466
73
0.067508
965
0.199267
44
0.241996
42
0.224536
29
0.087466
229
0.283473
112
0.089504
326
0.239102
29
N women
493
77
31
240
50
132
111
53
72
24
19
11
135
160
383
31
288
59
32
67
316
14
204
55
www.campbellcollaboration.org
Back to our Method of moments estimator
Q  (k  1)
 
, where
c
Q is the homogeneity statistic, where
2
Q
2
2
w
T

(
w
T
)
 i i  i /  wi
k is the number of effect sizes, and
c is based on the fixed effect weights, wi , and is
c
w
i

2
w
 i
w
i
So we need c, and Q
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
To get c:
∑ wt - ∑ wt2/ ∑ wt =
1695.922 – (223238.2)/1695.922 =
1596.29
To get Q:
167.151-(-186.795)2 /1695.922 =
146.58
To get τ2 :
(Q – (k-1))/c =
[146.58 – (24-1)]/1596.29 = 0.079
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Now that we have τ2 , we need to get the mean and
its variance
• We first need to add τ2 to each study’s variance to get the
random effects variance
• Then we get the new weights to use for the computation of
the mean effect size and its variance
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Data
The random effects
mean is found from
(-35.76)/223.00 = -0.16
The se for the mean is
found by
1/sqrt(223.00) = 0.067
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
RevMan for random effects models
• We can also get the results in RevMan without having to do
the computations.
• Open the file called “Gender differences for transformational
leadership.rm5” – click on open a review from a file
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Click on data and analyses (I already put in data)
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Click on 1.3 Transformational leadership score
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Click on Forest plot
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
These
are the
fixed
effects
CI’s for
each
study.
Here is our tau and
test that tau is zero.
Campbell Collaboration Colloquium – May 2012
This is the random effects
mean and CI
www.campbellcollaboration.org
Once we have the random effects mean and CI
• We will want to test whether the variance component is
different from zero.
• Testing whether the variance component is different from
zero is equivalent to asking whether the variation among
studies is greater than what we would expect with only
sampling error
• The test: the fixed effects test of homogeneity
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Restricted maximum likelihood estimator
• Many statisticians do not like the method of moments
estimator though RevMan and Comprehensive Metaanalysis use this estimator
• Can estimate τ2 using HLM, SAS Proc Mixed, or R
• I can give you sample programs for these estimators
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Reporting guidelines for random effects models
• These are the reporting guidelines according to me
• You should report the following when using a random effects
model for estimating a mean effect size:
– Rationale for choosing the random effects model
– The method for computing the variance component, τ2 (and it’s
okay to say you used RevMan or CMA, for example)
– The estimate of the variance component
– The mean effect size and its variance or standard error
– The confidence interval around the effect size
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Some concluding remarks
• Given the anticipated differences among studies in a typical
social science systematic review, we usually choose random
effects models
• From the C2 Methods Editor (me): make sure to provide a
rationale for your choice in the protocol, based on your
substantive knowledge of the area of the review
• Many other issues not touched on in this talk and see the
references following this slide
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
References
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R.
(2010). A basic introduction to fixed effect and random effects models
for meta-analysis. Research Synthesis Methods, 1, 97-111.
Borenstein, M., Hedges, L. V., Higgins, J. P. T., & Rothstein, H. R.
(2009). Introduction to meta-analysis. West Sussex, UK: John Wiley.
Eagly, A. H., Johannesen-Schmidt, M. C. & van Engen, M. L. (2003).
Transformational, transactional, and laissez-faire leadership styles: A
meta-analysis comparing women and men. Psychological Bulletin,
129, 569-591.
Raudenbush, S. W. (2009). Analyzing effect sizes: Random- effects
models. In Cooper, H., Hedges, L. V. & Valentine, J. C. (Eds.). The
handbook of research synthesis and meta-analysis (2nd ed). New
York: Russell Sage Foundation.
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass
0130 Oslo, Norway
E-mail: info@c2admin.org
http://www.campbellcollaboration.org
The Campbell Collaboration
www.campbellcollaboration.org