Practical Meta-Analysis for the
Social Sciences
Evan J. Ringquist
School of Public and Environmental Affairs
Indiana University
Workshop in Methods Presentation
January 11, 2013
Bloomington, IN 47405
1
“Meta-Analysis is not a fad. It is rooted in the
fundamental values of the scientific enterprise:
replicability, quantification, causal and correlational
analysis. Valuable information is needlessly
scattered in individual studies. The ability of social
scientists to deliver generalizable answers to basic
questions of policy is too serious a concern to allow
us to treat research integration lightly. The potential
benefits of meta-analysis method seem enormous.”
(Bangert-Drowns 1986: 398)
2
Meta-Analysis Defined
• Meta-analysis is a systematic, quantitative,
replicable process of synthesizing
numerous and sometimes conflicting
results from a body of original studies.
• Meta-analysis, then, provides a powerful
set of tools for aggregating knowledge.
3
Number of Articles Published Each Year in the
“Meta-Analysis” Topic Category as Referenced by
the Social Sciences Citation Index
1400
1200
800
600
400
200
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
1995
1994
1993
1992
1991
1990
1989
1988
1987
1986
1985
1984
1983
1982
1981
0
1980
articles per year
1000
4
Outline of Presentation
1. Motivation for Using Meta-Analysis in the
Social Sciences
2. Quick Introduction to Meta-Analysis
3. Meta-Regression Models: The Basics
4. Meta-Regression Analysis for the Social
Sciences
– CRVE, GEE, and if we have time, HLM
5
Section I: Motivation
6
Cumulative Knowledge and the
Scientific Enterprise
• A Central Goal of Science is the
Accumulation and Aggregation of
Knowledge. Yet the Social Sciences in
General and Researchers in Public
Management and Policy in Particular Have
Found Cumulative Knowledge to be an
Elusive Goal.
7
“Cumulative Knowledge” in
Environmental Justice
• “Overwhelming evidence” of significant
environmental inequities (Goldman 1993;
see also Mohai and Bryant 1994)
• “even a reasonably generous reading of
the empirical research alleging
environment inequity . . . must leave room
for profound skepticism” (Foreman 1998;
see also Bowen 2001)
8
“Cumulative Knowledge” in
Education Policy
• “Extensive research has been conducted on the academic success
of students enrolled in school choice program nationwide. Rigorous
studies show strong gains for voucher and scholarship tax credit
recipients . . .” (Alliance for School Choice, 2010)
• “The reason that vouchers had subsided as a point of advocacy is
because they don’t work.” (Randi Weingarten, President, American
Federation of Teachers; Associated Press 2011).
• “Few contemporary questions in American education have
produced such wide-spread controversy as that regarding the
potential of school vouchers to reduce inequality in student
outcomes. . . these efforts notwithstanding, answers to the voucher
question still appear uncertain.” (Cowen 2008)
9
Cumulative Knowledge in Public
Management
• the more knowledge we create and diffuse, “the
less we know, because our research is not
aggregated or accumulated into substantial
bodies of knowledge” (Van Slyke, O’Leary, and
Kim 2010: 290.
• The first step in making Public Administration a
stronger and more robust field is the need to
aggregate knowledge in the sense of making it
cumulative. (Rosenbloom 2010)
10
Cumulative Knowledge in Policy
Studies
• “political scientists have spent (almost
literally) countless articles and books
proposing something like ‘laws’ or theories
that, taken collectively . . . have produced
infinitely more confusion than clarity”
(DeLeon 1998: 150).
11
Cumulative Knowledge in Policy
Evaluation
•
Social scientists identify new and important questions having policy
relevance. Initial studies examining these questions provide clear answers
regarding, for example, the effectiveness of social policy interventions.
Subsequent studies cast doubt on these initial conclusions, however, and
with the proliferation of studies comes a proliferation of conclusions. The
best efforts of social and behavioral researchers generate confusion and
uncertainty rather than clarity. In the end, researchers conclude that the
phenomena being studied are ‘hopelessly complex’ and move on to other
questions. After several repetitions of this cycle, social and behavioral
scientists themselves become cynical about their own work and express
doubts about whether behavioral and social science in general is capable of
generating cumulative knowledge or answers to socially important questions
(Hunter and Schmidt 1996: 325-6).
12
Why Do Social Scientists Struggle
with Knowledge Accumulation?
1. Epistemological Differences: Social Sciences
ought not aim to generate generalizable
“scientific” knowledge ala Chemistry.
2. Subject Matter Differences: Social scientists
study organizations and institutions devised
and populated by strategic actors, not
molecules or organizms without agency.
3. Social Scientists Use The Wrong Tools for
Aggregating Knowledge.
13
Section II: Introduction to
Meta-Analysis
14
Meta-Analysis Defined (again)
• Meta-analysis is a systematic, quantitative,
replicable process of synthesizing
numerous and sometimes conflicting
results from a body of original studies.
• Meta-analysis, then, provides a powerful
set of tools for aggregating knowledge.
15
The Language of Meta-Analysis
• Original Study: A piece of original
research, published or unpublished, that
aims to test a hypothesis and/or estimate
a quantity of interest.
• Focal Predictor: The independent variable
in an original study that measures the key
exogenous concept associated with the
research question of interest.
16
The Language of Meta-Analysis
• Effect Size: A standardized measure of the
relationship between the focal predictor
and the dependent variable in an original
study. Effect sizes are the unit of analysis
and the quantity of interest in a metaanalysis, often designated Θi. Without
effect sizes there can be no meta-analysis.
17
The Language of Meta-Analysis
• Effect Size Variance: A measure of the
uncertainty associated with a particular
effect size.
• Fixed and Random Effects Models:
Methods of weighting effect sizes that
embody different assumptions about the
effect size variance.
18
How Do We Calculate Effect
Sizes?
• Three Families of Effect Sizes
1. D-based effect sizes (standardized mean
differences, common in education and
psychology)
2. Odds-based effect sizes (e.g. log odds of an
event, common in medicine)
3. R-based effect sizes (partial correlation
coefficients, unusual in the literature but
19
most useful for social scientists)
Calculating R-based Effect Sizes
from Original Studies
• r = √[t2 / (t2 + df)]
• r = √[Z2 / n]
• r = √[Χ21 / n]
[1]
[2]
[3]
• V[r] = (1-r2)2 / (n-1)
[4]
20
Fisher’s Corrections for R-based
Effect Sizes and Variances
• Zr = 0.5 ln[(1+r) / (1-r)]
[5]
• V[Zr] = 1/(n-3)
[6]
21
Example: Are Pollution Emissions
Higher in Black Neighborhoods?
• Ringquist 1997: t=3.06, df=29202
– r=.02
– Zr=.02
– V[Zr] = .000034
• Downey 1998: t=3.38, df=112
– r=.30
– Zr=.31
– V[Zr] = .0072
22
What Can We Do With Effect Sizes
Statistically?
1. Communicate results from original research in
a more meaningful fashion (not meta-analysis).
2. Combine (average) effect sizes to estimate the
population effect size
3. Test null hypothesis that population effect size
equals zero
4. Explain variation in effect sizes across original
studies. We use “Meta-Regression” to
23
account for this variation.
Combining Effect Sizes
• We calculate average effect sizes, or our
estimate of the population effect size, by
calculating a weighted average of all effect
sizes from original studies where the
weights are inverse variances
• Θbar= ΣwiΘi / Σwi
[7]
• Wi = 1/vi
24
Combining Effect Sizes:
Fixed Effects Models
• Fixed Effects models assume that effect
sizes vary across original studies only due
to sampling error:
– Θi = Θ + ei
– E[Θi]= Θ
– ei ~ N(0,vi)
[8], so
[9], and
[10]
25
Combining Effect Sizes:
Fixed Effects Models
• Fixed effects meta-regression assumes
that effect sizes conditional upon
moderator variables differ only due to
sample size
• Fixed effects estimates apply only to
sample of original studies in hand
26
Combining Effect Sizes:
Random Effects Models
• Random Effects models assume that
effect sizes are normally distributed
random variables
– Θ ~ N(µΘ, τ2)
[11]
– Θi = µΘ + ei
[12]
– E[Θi] = µΘ
[13], and
– ei ~ N(0,vi + τ2)
[14]
– Always use random effects in the social
sciences!
27
Estimating the Random Effects
Variance Component
• Using Restricted Maximum Likelihood
– Le(τ2) = -.5 * Σ [ln(vi + τ2) + ((ei2 / (vi + τ2) - .5 * ln |X’v-1X| [15]
• Using Method of Moments (MOM)
– tr(M) = Σvi-1 – tr [(Σvi-1 XX’)-1 (Σvi-2 XX’)]
[16]
• Using MOM Approximation
– τ2 = [SSEols / (m-k-1)] – vbar
[17]
28
What Can We Do With Effect
Sizes Substantively?
• Estimate severity of problems
– E.g., environmental inequities
– E.g., effects of climate change
• Measure important quantities of interest
– E.g., statistical value of a life
– E.g., hedonic pricing of environmental
amenities
29
What Can We Do With Effect
Sizes Substantively?
• Program Evaluation
– E.g., effectiveness of educational vouchers
– E.g., effectiveness of job training programs
• Theory Testing and Development
– E.g., Top-down vs. bottom-up implementation
– E.g., effects of negative campaign ads
– E.g., Ricardian equivalence
30
Section III: Introduction to
Meta-Regression
31
“exploration of the pattern of variation in
effect sizes among studies is a far more
important goal of meta-analysis than the
construction of powerful tests of null
hypotheses.” (Osenberg et al. 1999: 1105)
32
Moving Meta-Analysis to the
Social Sciences
• Techniques for calculating and combining
effect sizes developed for synthesizing the
results from experiments. Can these same
techniques really be used to synthesize
results from the non-experimental
multivariate models common in the social
sciences?
33
Three Critiques of MetaAnalysis in the Social Sciences
1. Parameter estimates from the general
linear model are not comparable.
2. Effect sizes cannot be calculated from
multivariate models.
3. Parameter estimates from different
models estimate different population
parameters, and therefore cannot be
combined.
34
Response to Critique #1:
• While regression parameters are not
comparable, meta-analysis does not
combine parameter estimates. Rather,
meta-analysis combines effect sizes which
are standardized with respect to scale.
This is not a valid critique.
35
Response to Critique #2
• In fact, the same formulas that can be
used to calculate r-based effect sizes from
experimental studies can be used to
calculate partial correlations from
regression models, probit models, etc.
(see Greene 1993: 180)
36
Response to Critique #2
• Example 1: Panel regression model,
Y=continuous, K=33, N=307,538
– Pcorr = .0099, Zr = .0096
• Example 2: Probit model, Y=dichotomous, K=8,
N=14131
– Pcorr = .1081, Zr = .1062
• Example 3: OLS regression model,
Y=continuous, K=24, N=285
– Pcorr = .1619, Zr = .1565
37
Response to Critique #3
• E.g., Study 1
– Y = b0 + b1X1 + b2X2 + b3X3 + e
• E.g., Study 2
– Y = b0 + b1X1 + b2X2 + b4X4 + e
• E[b11] ≠ E[b12], β11 ≠ β12, and Θ1 ≠ Θ2
• Therefore dependent variable Θi measures
fundamentally different quantities
38
Response to Critique #3
• Critique #3 is a valid critique, and this is
why calculating average effect sizes is of
little value in the social sciences – these
average effect sizes combine estimates of
different population parameters, and
therefore are not valid estimates of any
useful quantity of interest.
• Thankfully, we can address Critique #3
39
using Meta-Regression.
What Do We Need to Conduct a
Meta-Regression?
• Measure of effect size that is comparable
across original studies employing different
measures, models, and samples
• Measure of (un)certainty or effect size
variance
• Moderator variables that account for
variability in effect sizes
40
Introduction to Meta-Regression
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)
– Θi is the effect size
– X represents moderator variables accounting for
differences in effect sizes within and across studies
• Scientifically Interesting Moderators (Rubin 1992)
– e.g., differences attributable to target characteristics
– e.g., differences attributable to program or policy design
• Scientifically Uninteresting Moderators
– e.g., differences attributable to estimation technique or
research design
41
Number of Publications Using Phrase “MetaRegression” in Google Scholar, 1990-2011
42
Example: Synthesizing Research
on Educational Vouchers
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)
– Θi is the estimated point biserial correlation (effect
size Zr) between the use of an educational voucher
and student standardized test scores from a particular
statistical model in a particular original study.
43
Example: Educational Vouchers
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)
– X1 moderator variable identifying effect sizes from
models limiting sample to black children
– X2 moderator variable identifying effect sizes from
voucher programs available to religious schools
– X3 moderator variable identifying effect sizes from
models not using random assignment to treatment
and control groups.
44
Example: Educational Vouchers
• Θi = b0 + b1X1i + b2X2i + b3X3 + ei: ei ~ N (0, vi)
– b0 average effect of vouchers on student test scores
for all students in experimental studies where
vouchers use is limited to secular schools
– b1 differential effect of vouchers for black students
– b2 differential effect from voucher programs that
include religious schools
– b3 differential voucher effect from quasi-experimental
studies
45
Independent Variables
Black Student
Hispanic Student
Religious Schools
Matched Control Groups
Convenience Control Groups
Pretest Included
Intercept (Baseline Effect Size)
Adjusted R2a
Psuedo-R2a
I2a
Sample Size
CRVE
Parameter
Estimates
(St.Err.)
0.0296**
(0.0130)
-0.0470***
(0.0103)
-0.0312
(0.0187)
-0.0097
(0.0064)
-0.0073
(0.0085)
-0.0326***
(0.0115)
0.1097***
(0.0257)
.44
CRVE 95% CI
0.0030
0.0562
-0.0680
-0.0259
-0.0693
0.0069
-0.0226
0.0033
-0.0246
0.0101
-0.0560
-0.0092
0.0573
0.1620
GEE
Parameter
Estimates
(St.Err.)
0.0370***
(0.0116)
-0.0413***
(0.0084)
0.0180
(0.0201)
-0.0069*
(0.0040)
-0.0082
(0.0082)
-0.0228*
(0.0120)
0.0830***
(0.0299)
na
.68
na
25.01%
na
611
611
GEE 95% CI
0.0143
0.0598
-0.0577
-0.0248
-0.0574
0.0213
-0.0148
0.0009
-0.0243
0.0078
-0.0464
0.0008
0.0243
0.1416
46
Obtaining Estimates for
Meta-Regression Models
• OLS Estimator of Meta-Regression Model
– E[b] = β = (X’X)-1 X’Θ
– var[b] = σ2(X’X)-1
[18]
[19]
• But OLS is incorrect for two reasons:
1. Θi is heteroskedastic (recall assumptions)
2. OLS weights all observations equally
(unequal variances means unequal
certainty)
47
Generalized Least Squares in
Meta-Regression
• In OLS, ee’ = σ2I
• In GLS, ee’ = σ2Ω
– where Omega is a diagonal matrix with
proportional error variances on main diagonal
• GLS estimates then are:
– E[bgls] = βgls = (X’Ω-1X)-1 X’Ω-1Θ
– var[bgls] = σ2 (X’Ω-1X)-1
[20]
[21]
48
Weighted Least Squares in
Meta-Regression
• Find a weight matrix W so that W’W = Ω-1
• Then the WLS meta-regression estimates
become:
– E[bwls] = βwls = (X’W’WX)-1 X’W’WΘ
– var[bwls] = σ2 (X’W’WX)-1
[22]
[23]
49
Weighted Least Squares in
Meta-Regression (cont.)
• The good news is that we have a handy
estimate of the diagonal elements of the
weight matrix W: the effect size variances
(or more properly, their square roots)
• The bad news is that fixed effects
variance estimates are almost certainly
wrong, so we need to use random effect
variances.
50
RE Variance Component τ2
• Cannot include moderator variables for all
independent variables or all design
elements in original studies
• Random Effects variance component τ2
can be conceived of as reflecting the
impact of unobserved variables on Θi
51
Weights in Random Effects
Meta-Regression
(v1 + τ2) 0
0
0
0
(v2 + τ2) 0
0
ee’remr = σ2
.
.
.
.
0
0
0
(vn + τ2)
• With weights as the square roots of the
diagonal elements of this matrix.
52
Estimating the Random Effects
Meta-Regression Model in Practice
• Option 1: Use the WLS function in Stata,
SAS, or some other statistical package.
– Don’t do this. All WLS algorithms treat weights
as proportional variance weights rather than
as absolute variance weights. This means
that meta-regression parameter standard
errors will be incorrect.
53
Estimating the Random Effects
Meta-Regression Model in Practice
• Option 2: Use the VWLS command in
Stata.
– This will work as long as you import the
proper random effects variance weights
(actually, their standard deviations). These
must be calculated outside of the VWLS
model. This works because VWLS treats the
imported variances as known and absolute
rather than as estimated and proportional. 54
Estimating the Random Effects
Meta-Regression Model in Practice
• Option 3: Use the “Metareg” Command in
Stata
– While Stata does not include commands for
meta-analysis, the Stata User Group (SUG)
has produced a number of good metaanalysis ADO files, including a random effects
meta-regression algorithm. These commands
can be downloaded from the Stata website.
55
Estimating the Random Effects
Meta-Regression Model in Practice
• Option 4: Use “Manually Weighted” Least
Squares
– Estimate the REMR model by first manually
weighting each observation by the square root
of the proper random effects variance, then
using OLS on the transformed data. While this
is more work than using the VWLS or Metareg
commands, it is necessary when estimating
more advanced meta-regression models.
56
Comparison of REMR Results in
Environmental Justice (n=680)
Intercept
Pollution
Census Tracts
Select on DV
Metareg
(MLE)
VWLS
(MOM)
MWLS
(MLE)
.068
(.006)
-.020
(.008)
.011
(.006)
-.078
(.019)
.073
(.008)
-.020
(.011)
.009
(.008)
-.093
(.025)
.068
(.006)
-.020
(.008)
.011
(.006)
-.078
(.019)
57
Consequences of Inadequately
Addressing Effect Size Heterogeneity
• Inadequately addressing heterogeneity in effect
sizes means you have a misspecified REMR
model.
• Consequences if r(Me,Mk) = 0?
• Consequences if r(Me,Mk) ≠ 0?
• Consequences of remaining effect size
heterogeneity in REMR model are akin to
specification error in OLS regression
13
Validity Check: Can MetaRegression Really Recover the
Quantity of Interest from
Original Studies?
59
Simulation Strategy
• Two-level Monte-Carlo simulation
– Assume Population Regression Model
+ β2 X2 + β3 X3 + ε.
Y=β0 + β1 X1
• First level simulate 10,800 original studies by
• Generate X from distribution N(0,1,Cxx), generate
Y using β1 = .2, .7 (X1 is focal predictor), generate
ε ~ N (0, 30)
• Allow measurement of X and Y to vary across
studies [X, X*1000, and ln(X)]
• Estimate sample regression models that employ
60
combinations of X1, X2, and X3
Simulation Strategy (cont.)
• Second level simulate 1000 meta-analyses of
sample size 500
• Estimate meta-regression model
– Y=β0 + β1 M1 + β2 M2 + ε
– M represent moderator variables
• Key Question: Can REMR recover partial
correlation parameter from population
regression model in original studies?
61
Simulation Results
Table 7.10. Ability of REMR Model Recover Population Parameter of Interest in the Presence of
Effect Size Heterogeneity and Variation in the Measurement of X and Y in Original Studies
Corr Population
(Xj,Xk) Parameter
of Interest
Estimate of
b0
Parameter Parameter
Estimate
Estimate
for
for
Moderator Moderator
Variable X2 Variable X3
Predicted
Value of
Population
Parameter
of Interest
.60
.3234
.5008
-.1648
-.0253
.3107
.60
.0950
.1644
-.0552
-.0065
.1027
62
Section IV:
Advanced Meta-Regression
63
“Meta-Analytic techniques developed to date
have not addressed the problem of
combining information on second-order
effects, such as partial regression
coefficients. Consequently, meta-analysis
has not been very useful in the synthesis of
evidence from causal/explanatory research."
(Becker 1992: 342)
64
Remaining Problems in
Meta-Regression
1. Non-Independence of Observations
2. Unequal Numbers of Effect Sizes per
Original Study
65
Problem 1:
Non-Independence of Observations
• Reasons for Correlated Observations
1. Common data sets across studies
2. Common research teams across studies
3. Single original study produces multiple
effect sizes (generally incorporates both 1
and 2)
• Consequences of correlated observations
for REMR: incorrect parameter st. errors 66
Addressing Problem 1:
The Approach in Meta-Analysis
• Traditional Solutions in Meta-Analysis
1. Select one effect size per study to use in
meta-analysis
2. Calculate average effect size per study and
use this in meta-analysis
67
Addressing Problem 1:
The Approach in Meta-Analysis
• Why the Traditional Solutions are Really
Bad Ideas:
1. Assumes each additional effect size from a
particular study contains no information
2. Cannot estimate effect of moderators within
studies, only across studies
3. Dramatically reduces sample size and
statistical power
68
Addressing Problem 1:
The Approach in Econometrics
• The Traditional Solution in Econometrics:
Cluster Robust Variance Estimation
(CRVE)
– i.e., Huber-White Standard Errors as adjusted
for clusters by Liang and Zeger (1986) and
Arellano (1987).
– Effect sizes Θig clustered by original study G
69
Addressing Problem 1:
The Approach in Econometrics
• In Matrix Form:
V[bREMRg] = (M’W’WM)-1 (Σ Mg’W’(egeg’)WMg)
(M’W’WM)-1
[24]
70
(τ2+v
11 ρ1
2
+σ 1)
0
ρ1
(τ2+v
.
+σ21)21
0
.
.
.
.
=
.
.
.
.
0
.
σ22** ρ2
ρ2
ρ2
.
ρ2
.
.
ρ2
σ22** ρ2
.
ρ2
ρ2
σ22** ρ2
.
.
ρ2
ρ2
ρ2
.
σ22**
.
ee’
.
.
.
.
.
.
.
.
0
.
σ2g** ρg
.
ρg
σ2g** ρg
ρg
ρg
0
.
.
.
.
.
.
0
ρg
σ2g**
71
Addressing Problem 1 Using
Example REMR Model
Intercept
Pollution
Census Tracts
Select on DV
REMR
CRVE
REMR
.068
(.006)
-.020
(.008)
.011
(.006)
-.078
(.019)
.068
(.011)
-.020
(.015)
.011
(.010)
-.078
(.032)
72
Addressing Problem 1 Using CRVE
• CRVE REMR model should be standard
practice in the Social Sciences -- BUT
• Only three examples of published CRVE
CRVE models. Why?
• Researchers developing statistics of metaanalysis unfamiliar with developments in
econometrics, and vice versa.
73
Extensions to the
CRVE REMR Model
1. Small number of clusters?
– Use the “Wild Cluster Bootstrap”
– Cameron, Gelbach, and Miller 2008
2. Two-way clustering? E.g., effect sizes
clustered by study and by data set?
– Use “multi-level clustering”
– Cameron, Gelbach, and Miller (2011)
74
Problem 2: Unequal Numbers of
Effect Sizes Per Study
• Reasons for Unequal Numbers of Effect
Sizes Per Study
– Varying numbers of subgroup analyses
– Varying sensitivity tests
• Consequences of Unequal Numbers of
Effect Sizes Per Study
– Few studies may dominate data set and
REMR results, threatening external validity
75
Addressing Problem 2:
The Approach in Meta-Analysis
• Traditional Solutions in Meta-Analysis
1. Select one effect size per study to use in
meta-analysis
2. Calculate average effect size per study and
use this in meta-analysis
• As we have seen, this is really bad advice
76
Addressing Problem 2:
The Approach in Econometrics
• The Econometric approach to Problem 3
generally takes one of two forms.
1. Relative “importance weights” attached to
observations (e.g., weight = 1/g)
2. Generalized Estimating Equations (GEE)
77
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE
– E[Θi] = h (Mb)
[25]
• Where h is a link function (identity in regression)
– V[Θi|Mi] = σ2i = g([Mbi] / φ
[26]
• i.e., the variance is a function of the expectation
• φ (phi) is a scale parameter that we will ignore
today
78
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE (cont.)
– Assume we have three effect sizes from two
studies: [Θ11 , Θ21 , Θ22]
• Θ = (1/3) Θ11 + (1/3) Θ21 + (1/3) Θ22
• Θ = (1/2) Θ11 + (1/2) Θ21
• Θ = (1/2) Θ11 + (1/4) Θ21 + (1/4) Θ22
[27]
[28]
[29]
– All are unbiased estimates of population effect
size Θ, but with different variances
79
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE (cont.)
– GEE estimates Θ using an optimal weighting
scheme employing the correlation among
observations from the same cluster (ρ). In this
simple example:
Θ = [((1+ρ) / (3+ρ)) Θ11] + [(1 / (3+ρ)) Θ21] +
[(1 / (3+ρ)) Θ22]
[30]
80
Addressing Problem 2 Using GEE
• A Simple Introduction to GEE (cont.)
– We still need an estimate of the conditional
variance of the sample effect size Θi. GEE
estimates this variance (σ2) using:
• σ2 = (A)1/2 R (A)1/2
[31]
• A has E[Θi ] = h Mb on the diagonal
• R is the “working correlation matrix”
• For REMR, the exchangeable working correlation
81
matrix is most appropriate
Addressing Problem 2 Using GEE
• Obtaining GEE Estimates
• Estimate a naïve model for Θ assuming independent
observations to obtain starting values for b and e.
• Use e to estimate the elements of the working correlation
matrix R.
• Re-estimate the structural model with weights derived
from the estimated values of ρ in R.
• Iterate steps 2 and 3 to convergence.
82
Addressing Problem 2 Using GEE
A Bonus from Using GEE to Estimate the
REMR Model: If we use the “empirical”
matrix ee’ rather than the “model” matrix ee’,
GEE also controls for Problem 2
– GEE calculates parameter estimates
controlling for clustering
– Empirical GEE calculates parameter variance
estimates controlling for clustering
83
Addressing Problem 4 Using
Example REMR Model
Intercept
Pollution
Census Tracts
Select on DV
REMR
CRVE
REMR
GEE
REMR
.068
(.006)
-.020
(.008)
.011
(.006)
-.078
(.019)
.068
(.011)
-.020
(.015)
.011
(.010)
-.078
(.032)
.067
(.011)
-.015
(.014)
.008
(.009)
-.072
(.021)
84
Addressing Problem 2 Using GEE
• GEE is an excellent choice for estimating
the REMR model.
• GEE REMR is quite rare – fewer than half
a dozen in the published literature
• Preferred over CRVE when number of
effect sizes are highly unequal across
studies. CRVE preferred in balanced
85
samples.
Estimating Advanced REMR
Models in Stata
• Stata cannot estimate directly either the
CRVE REMR or the GEE REMR model.
• Neither VWLS nor Metareg can estimate
these models.
• Models must be estimated using MWLS
data in conjunction with “robust cluster” or
“xtgee” commands in Stata.
86
The Take Aways
1. Meta-regression is a powerful and underutilized tool for synthesizing research in the
social sciences.
2. When estimating meta-regression models,
social scientists should use the CRVE REMR
or the GEE REMR model.
3. Social scientists should not use the standard
REMR model or the HLM REMR model.
87
“(Meta-analysis) is going to
revolutionize how the sciences . . .
handle data. And it is going to be the
way many arguments will end.”
T. Chalmers, quoted in Mann 1990: 480
88
Evan Ringquist
eringqui@indiana.edu
89