Moderator analyses: Categorical models and
Meta-regression
Terri Pigott, C2 Methods Editor & co-Chair
Professor, Loyola University Chicago
tpigott@luc.edu
The Campbell Collaboration
www.campbellcollaboration.org
Moderator analyses in meta-analysis
• We often want to test our hypotheses about whether
variation among studies in effect size is associated with
differences in study methods or participants
• We have these ideas a priori, incorporating these
characteristics of studies into our coding forms
• Two major forms of moderator analyses in meta-analysis:
categorical models analogous to ANOVA, and metaregression
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Assumptions for this session
• We will focus on random effects models as these are the
most common in Campbell reviews
• I will assume that we have computed the random effects
variance component (as you did if you were in my session
yesterday - though you may feel like this right now)
• We will use two software packages:
– RevMan – available here:
http://ims.cochrane.org/revman/download
– Comprehensive Meta-analysis – available for free download
and limited trial here:
http://www.meta-analysis.com/pages/demo.html
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Categorical moderators
• When the moderator variable is categorical, we can estimate
models analogous to ANOVA
• Typically, we are interested in comparing the group mean
effect sizes for 2 or more groups
• For example, we will look at a meta-analysis where we
compare the mean effect size for studies published in three
different sources: journals, dissertations, and unpublished
studies
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Categorical moderator models
• With a one-way random effects ANOVA model, recall that we
will compute
– A mean effect size and standard error for each group, and then
test whether these means are significantly different from one
another
– The mean effect size and standard error require an estimate of
the variance component
– QUESTION: Will we assume that each group has the same
variance component? Or, will we assume that each group has
its own variance component?
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
What are our assumptions if we decide to use separate
estimates within subgroups?
• We believe that the variation among studies is different
between groups.
• For example, if we are testing out an intervention and we
have studies that use either a low-income and a high-income
group of students, we might believe that there will be more
variation in effectiveness among studies that have mostly
low-income participants
• Another example: the effectiveness of an intervention for
juvenile delinquents will vary more for the group that had a
prior arrest than for those that do not have a prior arrest
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
What are our assumptions when we use a pooled
estimate?
• We believe that the variation among effect sizes are the
same no matter the group.
• For an intervention review, we may assume that the variation
among studies does not differ within the groups of interest
• Caveat: We might have to use a pooled estimate if we have
small sample sizes within subgroups. We need at least 5
cases (in general) to be able to estimate a separate variance
component for each subgroup
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Flowchart from Borenstein
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Steps for a random effects ANOVA
• Make a decision about the use of a pooled or a separate
estimate of the variance component
• Compute the group mean effect sizes, and their standard
errors
• Compare the group mean effect sizes to see if they are
statistically different from one another
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Eagly, Johannesen-Schmidt & van Engen (2003)
• This synthesis examines the standardized mean difference
estimated in primary studies for the difference between men and
women in their use of transformational leadership.
• Transformational leadership involves “establishing onself as a
role model by gaining the trust and confidence of followers”
(Eagly et al. 2003, p. 570).
• The sample data is a subset of the studies in the full metaanalysis, a set of 24 studies that compare men and women in
their use of transformational leadership
• Positive effect sizes indicate males score higher than females
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
To follow along:
• Open RevMan
• Open a review from a file
• Open the file named:
Gender_differences_for_transformational_leadership.rm5
• Go to Data and analyses on the left-hand menu
• Double-click on 1.1 Transformational leadership
• NOTE: RevMan uses the assumption that each group has a
different variance component
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Summary of results – separate variance estimates for
each group
Group
Journals
k
Mean
95% CI
τ2
p
13
-0.05
[-0.24, 0.12]
0.09
<0.001
Dissertations
7
-0.47
[-0.69,-0.26]
0.02
0.22
Unpublished
4
-0.16
[-0.30,-0.03]
0.00
0.87
24
-0.16
[-0.29, -0.03]
0.08
<0.001
TOTAL
•Journals have a significant variance component, and the mean is not
different from zero
•Dissertations and unpublished studies both have a non-significant
variance component, but both find that women score higher on
transformational leadership
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Summary of results – separate variance estimates for
each group (continued)
Group
Journals
k
Mean
95% CI
τ2
p
13
-0.05
[-0.24, 0.12]
0.09
<0.001
Dissertations
7
-0.47
[-0.69,-0.26]
0.02
0.22
Unpublished
4
-0.16
[-0.30,-0.03]
0.00
0.87
24
-0.16
[-0.29, -0.03]
0.08
<0.001
TOTAL
•The test of the variance component as different from zero is exactly the
fixed effects test of homogeneity.
•To get this test, we compute the test of homogeneity within each group
of studies.
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Test of between group differences
• To test between group differences in a random effects
model, we test whether the variance component for the
variation among the random effects means is equal to zero
• There are several ways to obtain this value
• We will use a test of homogeneity of the three means – we
will treat the three group means as a meta-analysis
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Test of between-group differences
• We will compute a test of homogeneity using our three
means as if this is a meta-analysis
• We will use the means and their estimated variances to
compute the sums we need to compute the homogeneity test
• These computations are all done “behind the scenes” by
RevMan
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Computation of Q between groups
Source
Mean
Var
Wt
Wt*Mean
Wt*Mean2
Journals
-0.05
0.008
122.53
-6.13
0.031
Dissertations
-0.47
0.012
86.46
-40.64
19.10
Unpublished
-0.16
0.005
211.32
-33.82
5.41
420.31
-80.59
24.54
SUM
(80.59* 80.59)
Q  24.54 
420.31
 9.09
Compare 9.09 to a chi-square
with df=3-1=2. p-value is
0.011
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
What happens if we use the same variance
component for all groups?
• We will need to try this in Comprehensive Meta-analysis
• Open your trial version of Comprehensive Meta-analysis
• Check that you will run the trial
• Open the file called: leaderage.cma
• Data is here:
https://my.vanderbilt.edu/emilytannersmith/training-materials/
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Run analyses
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Meta Analysis
Group by
pub source
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
1.00
3.00
3.00
3.00
3.00
3.00
3.00
3.00
3.00
4.00
4.00
4.00
4.00
4.00
Overall
Study name
Statistics for each study
Hedges's
g
B23
CA
CH
CW1
CW2
DF
JB
JL
KO
KUU
LJ
RH
SM
BO
CU
DA
HI
MCG
RO
WI
BJ
GM
MA
SP
-0.090
-0.170
-0.220
0.610
0.200
-0.250
-0.130
-0.290
0.310
0.090
-0.350
-0.360
0.000
-0.055
-0.620
-0.170
-0.150
-0.360
-0.720
-0.440
-0.870
-0.477
-0.100
-0.100
-0.080
-0.210
-0.130
-0.159
Standard
error
0.066
0.083
0.118
0.092
0.150
0.147
0.117
0.105
0.078
0.214
0.068
0.087
0.283
0.084
0.262
0.185
0.281
0.356
0.242
0.225
0.239
0.141
0.148
0.252
0.199
0.090
0.162
0.066
Variance
0.004
0.007
0.014
0.008
0.023
0.022
0.014
0.011
0.006
0.046
0.005
0.008
0.080
0.007
0.069
0.034
0.079
0.127
0.059
0.050
0.057
0.020
0.022
0.064
0.040
0.008
0.026
0.004
Lower
limit
-0.220
-0.333
-0.451
0.430
-0.095
-0.539
-0.360
-0.495
0.157
-0.330
-0.482
-0.531
-0.556
-0.219
-1.134
-0.533
-0.701
-1.059
-1.194
-0.880
-1.339
-0.752
-0.390
-0.594
-0.471
-0.385
-0.447
-0.288
Upper
limit
0.040
-0.007
0.011
0.790
0.495
0.039
0.100
-0.085
0.463
0.510
-0.218
-0.189
0.556
0.108
-0.106
0.193
0.401
0.339
-0.246
0.000
-0.401
-0.201
0.190
0.394
0.311
-0.035
0.187
-0.031
Hedges's g and 95% CI
Z-Value
p-Value
-1.355
-2.045
-1.868
6.656
1.330
-1.696
-1.109
-2.768
3.982
0.420
-5.185
-4.116
0.000
-0.662
-2.365
-0.917
-0.534
-1.010
-2.975
-1.960
-3.639
-3.389
-0.675
-0.397
-0.401
-2.346
-0.805
-2.429
0.176
0.041
0.062
0.000
0.183
0.090
0.267
0.006
0.000
0.675
0.000
0.000
1.000
0.508
0.018
0.359
0.593
0.312
0.003
0.050
0.000
0.001
0.500
0.692
0.688
0.019
0.421
0.015
-1.00
-0.50
Favours A
0.00
0.50
1.00
Favours B
Meta Analysis
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Notes about the CMA forest plot
• Like RevMan, the confidence intervals around each study
are the fixed effects confidence intervals (they use the
within-study fixed effects variance)
• The group means are the random effects means computed
using random effects weights. Their confidence intervals are
also use random effects.
• In this example, we are using the same variance component
for all groups
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Summary from CMA
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Notes about CMA results
• We assumed that the variance component was 0.08 for all
studies
• Compared to our separate variance estimates, this value is
smaller than the separate variance estimate for journals, but
larger than the separate estimate for dissertations and
unpublished articles
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Reporting results from a random effects
categorical analysis
• The assumption made about the
•
•
•
•
random effects variance: separate
estimate for each group, or the same
estimate for all groups.
Rationale for the choice of variance
component
The random effects mean and CI
The value of the variance
components (or variance
component)
The test of the between-group
differences, and its significance
Campbell Collaboration Colloquium – May 2012
Group
RE mean
RE CI
τ3
Q for between-group means
df and p-value
www.campbellcollaboration.org
What is meta-regression?
• Meta-regression is a statistical technique used in a meta –
analysis to examine how characteristics of studies are
related to variation in effect sizes across studies
• Meta-regression is analogous to regression analysis but
using effect sizes as our outcomes, and information
extracted from studies as moderators/predictors
• NOTE: We can conduct a meta-regression in any statistical
program. Here we will use CMA. BUT, note that using other
standard programs may necessitate some adjustments to the
results since they don’t produce exactly what we want.
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Meta-regression used to examine heterogeneity
• When we have a heterogeneous set of effect sizes, we can
use statistical techniques to examine the association among
characteristics of the study and variation among effect sizes
• We have a plan for these analyses a priori – based on our
understanding of the literature, and a logic model or
framework
• Meta-regression used when we have more than one
predictor or moderator (either continuous or categorical)
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Form of the meta-regression model
ES i  B0  B1 xi1  ...  B p xip
ESi is our generic effect size for study i
xi1 , xi 2 ,... xip are the values for the predictor variables for study i
B0 , B1 ,..., B p are the unknown regression coefficients
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Recall that the variance of the effect size
• Depends on the sample size for all types of effects we have
talked about
• Thus, the precision of each study’s effect size depends on
sample size
• This is different from our typical application of regression
where we assume every person has the same “weight”
• Thus, we need to use weighted least squares regression to
account for the fact that the precision of each effect size
depends on sample size
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Random effects meta-regression
• As in the categorical analysis discussion, we will need an
estimate of the random effects variance for our studies that
will be used as our weights in the regression
• There are many ways to compute the variance component in
a random effects meta-regression
• For now, let’s assume a single variance component for all
studies.
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Test for the fit of the meta-regression model
• As in a standard regression model, we can use the
regression ANOVA table for diagnostics about the fit of a
meta-regression
• Recall that in a standard regression analysis, we would get
the following regression ANOVA table:
ANOVAb,c
Model
1
Reg ression
Residual
Total
Sum of
Squares
556.483
923.856
1480.338
df
2
23
25
Mean Square
278.241
40.168
F
6.927
Sig .
.004a
a. Predictors: (Constant), grade, percmin
b. Dependent Variable: ztrans
c. Weighted Least Squares Reg ression - Weighted by wt
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Test of Model Fit in Meta-regression
• In meta-regression, we use the ANOVA table to get two
different Q statistics:
• QM – model sum of squares, compare to chi-square
distribution with p – 1 df (p is number of predictors in the
model)
• QR – residual sum of squares, compare to chi-square
distribution with k - p – 1 df (k is the number of studies)
• See Lipsey & Wilson. 2001. Practical Meta-analysis.
Thousand Oaks, CA: Sage. pp. 122-124
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
QM , the model sum of squares
• Qmodel is the test of whether at least one of the regression
coefficients (not including the intercept) is different from zero
• We compare QM to a chi-square distribution with p – 1
degrees of freedom with p = # of predictors in model
• If QM is significant, then at least one of the regression
coefficients is different from zero
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
QR , the error or residual sum of squares
• QR is the test of whether there is more residual variation than
we would expect IF the model “fits” the data
• We compare QR to a chi-square distribution with k - p – 1
degrees of freedom with k = # of studies/effect sizes, and p =
# of predictors in model
• If QR is significant, then we have more error or residual
variation to explain, or that is not accounted for by the
variables we have in the model
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Testing significance of individual regression coefficients
in meta-regression
In a standard regression analysis, we find the t-tests on the printout to
see which regression coefficients are significantly different from zero
• Those significant regression coefficients indicate that these predictors
are associated with the outcome
• We will use CMA which gives us the z-tests for the regression
coefficients
• NOTE: When doing meta-regression in a standard program like
SPSS, we have to make some adjustments since these programs do
not compute the weighted regression in the way we need for metaanalysis
•
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
To conduct a meta-regression in CMA: run an
analysis to get to the table below
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Under Analyses, choose meta-regression
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
On the next page, choose the continuous outcome,
averageage
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
By default, the analysis will be fixed effects.
Choose method of moments under Computational
options
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Plot of points and regression line
Regression of averageage on Hedges's g
0.80
0.62
Hedges's g
0.44
0.26
0.08
-0.10
-0.28
-0.46
-0.64
-0.82
-1.00
24.90
28.62
32.34
36.06
39.78
43.50
47.22
50.94
54.66
58.38
62.10
averageage
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Results
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Example for meta-regression
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Objective of the review
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Interventions
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Example
from
Wilson &
Lipsey
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
What to report in a random effects metaregression?
• The software and/or method used to compute the results
• The method used to compute the random effects variance
component
• The goodness of fit tests: Qmodel , and QResidual
• The regression coefficients and their test of significance
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
Final notes
• Software may be a problem in meta-regression as only CMA
computes meta-regression. RevMan does not have the
capacity for meta-regression
• CMA only allows one predictor in the meta-regression
• To conduct the analyses as seen in the Wilson & Lipsey
example, you need to use other general statistical programs
like SPSS, or STATA
• There are R programs available to conduct meta-analyses
as well
Campbell Collaboration Colloquium – May 2012
www.campbellcollaboration.org
P.O. Box 7004 St. Olavs plass
0130 Oslo, Norway
E-mail: info@c2admin.org
http://www.campbellcollaboration.org
The Campbell Collaboration
www.campbellcollaboration.org