A Quarter Century on,
Where are we?
(MRA) is at once a framework in which to organize and
interpret exact and inexact replications, to review more
objectively the literature and explain its disparities, and to
engage in the self-analysis of investigating the
socioeconomic phenomenon of social scientific research
itself– Stanley & Jarrell (1989, p. 168).
Stanley, T.D. and S.B. Jarrell (1989) Metaregression analysis: A quantitative method
of literature surveys. Journal of Economic
Surveys, 3: 161-70
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Tom Stanley, Hendrix College
Where are we going?
Exponential
@
18%/year
 More objective reviews of economic research
 Explanation of disparate research findings
¿ Investigation of the socio-economics of
economics research?
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Has MRA’s promise been realized?
• Female economists find less wage discrimination
against women than do male researchers (S&J,
1998; J&S, 2004, Weichselbaumer & Winter-Ebmer, 2005)
• Publication bias is the result of professional
incentives and the pressure to publish.
• Researcher ideology affects reported results
(Doucouliagos and Paldam, 2006).
• The Research Cycle: Reported findings generally
confirm a novel hypothesis; later, the incentive for
rejection increases (S, J & D, 2008).
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Investigation of the socio-economics of
economics research?
 More objective reviews of economic research
 Explanation of disparate research findings
 Investigation of the socio-economics of
economics research, . . . but much more to do.
¿ Framework to organize and interpret exact and
inexact replications?
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Has MRA’s promise been realized?
• Deakin University’s (Chris Doucouliagos)
meta-data repository.
• Bob Reed, Maren Duvendack, and
others have expressed interest in
organizing something more formal.
• Piers Steel and metaBUS.
• Needless to say, . . . .
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Framework to organize and interpret
exact and inexact replications?
How about S&J’s (1989) exact MRA model?
Does anyone still use it?
(3)
Where: “bi is the reported estimate of b from the ith study. . .
Zik the meta-independent variable which measures relevant
characteristics of an empirical study and explains its
systematic variation” (p. 164)
•
Zik might include:
1.
2.
3.
4.
5.
Dummy variables for omitted variables.
Specification variables
Sample Size
Author characteristics
Data characteristics
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
bi = b + SbkZik +ei
Due to obvious Heteroskedasticity, S&J 89
recommended the WLS version of (3):
ti= bi/Sb = b (1/Sb ) + Sbk (Zik/Sb ) +ei /Sb (4)
i
i
i
• t-value, ti, is the dependent variable and
precision (1/Sb ) is an independent variable.
i
• WLS is neither fixed- nor random-effects, in
practical application, WLS is better than both
(Stanley and Doucouliagos, 2013a&b, Deakin SWP).
• We never wanted to use fixed- or random-effects,
in spite of citing Hedges and Olkin (1985).
• Had we included the intercept, we would have
fully anticipated current practice {FAT-PET-MRA}.
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
i
• Fixed-effect MRA: same as our WLS regression,
but divides SEs by square root of MSE (H&O, 85)
• causes SEs & CIs to be too small & too narrow.
• assumes that policy makers wish to make
inferences to a population that is identical,in every
respect, to past research. Like that happens!
• So why divide by root MSE???
• WLS already correct the SEs for both excess
heterogeneity and heteroskedasticity
• Fixed-effect MRA is never relevant in economics.
T.D. Stanley, He n drix College
MAER-Net September 12, 2014
Neither Fixed
Nor Random
ni , to the conventional meta-regression model,
bi = b + SbkZik +ni + ei
(1)
Where ni is assumed to be normal and independent
of the sampling errors, ei, and the moderators, Zik.
• Problems:
• In economics, excess heterogeneity is systematic!
• Typically, ni will be the result of omitting relevant
variables; thus, it introduces bias.
• In the 1980s, we saw no reason to use FE or RE
• Now, we know that we should never use FE or RE!
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
• Random-effects MRA adds a second error term,
Problems and Issues—2014
• Should we divide meta-regression SEs by the
square root of MSE? No! (Hedges and Olkin, 1985)
• Does Random-Effects MRA become biased with
publication bias? YES! (Stanley and Doucouliagos, 2013b)
Meta-analysis (MA)—Weighted Averages
• Fixed-Effects Estimator (FEE): confidence intervals
are too narrow if there is heterogeneity.
• Random-Effects Estimator (REE): can be very
biased with publication bias {already widely established}
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Multiple Meta-Regression (MRA)
• In two recent simulation papers, Chris and I show:
• WLS is as good as Fixed or Random-Effects
under the best conditions for FE and RE.
• If we are making inferences to policy settings or
to future research, WLS is much better than FE.
• When there is publication bias, WLS is much
better than RE.
• Worse still: all tests of either heterogeneity or
publication bias are known have low power.
• Therefore, in practice, we should never use
either FE or RE. . . . Never!
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Neither Fixed nor Random ̶ 2014
Stanley, T.D., Doucouliagos H(C). 2013a.
Neither Fixed nor Random: Weighted least
squares meta-regression analysis. SWP,
Economics Series 2013-1, Deakin University.
http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2013_1.pdf
Stanley, T.D., Doucouliagos H(C). 2013b.
Better than Random: Weighted least squares
meta-regression analysis. SWP, Economics
Series 2013-2, Deakin University.
http://www.deakin.edu.au/buslaw/aef/workingpapers/papers/2013_2.pdf
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Our two working papers:
Weighted Least Squares
• MRAs should never be estimated by OLS,
because there is much variation among the
reported SEs of bi or effecti
ˆ =(MtW-1M)-1MtW-1b
• Enter WLS: β
Where:
2

W=
(2)
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
(WLS-MRA)
The Gauss-Markov Theorem
• As long as W in (2) is known up to some
unknown proportion, 2, WLS { βˆ } is the Best
{Minimum Variance} Linear Unbiased Est.
• Invariance to proportional excess
heterogeneity is a robustness property of the
Gauss-Markov Theorem and WLS.
• It is not an assumption.
T.D. Stanley, Hendrix College SRSM
July 2, 2014
{Proportional Heterogeneity Invariance}
Traditional, Unrestricted WLS
replaces Wwith:
0
0
SE22
 
0
0



0


  SEL2
0

(5)
and S 2 is estimated by the WLS residuals, automatically
T.D. Stanley, Hendrix College SRSM
July 2, 2014
2
ˆ
W= S
SE12
Simulation Design
• Generate Yj and estimate b from:
(6)
• Half the studies omit the relevant variable X2i
2
• a3i ~N(0,  h ) adds excess random heterogeneity
by always omitting relevant variable X3i
• When b = 1, r between Y and X1 is .27
• n= {62, 125, 250, 500, 1000} in primary regressions
• X1j , X2j, X3j are generated randomly, but X2j & X3j are
forced to be correlated with X1j .
• Fixed- or random-effects model is always true.
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Yj = 100 + b X1j +a2 X2j +a3i X3j + ej
• Experiment 1: 10,000 WLS, RE and FE-MRAs
are calculated with one moderator variable,
Mi = {0,1}, reflecting whether the original study
omitted X2i, or not, (Tables 1 & 2)
bi = b0 + b1 Mi + ui
(7)
Where: bi is the ith primary study’s estimate of b
• Experiment 2: Experiment 1 plus 50% of the
studies select statistically significant results,
and either MRA (7), above, or a multiple FATPET-MRA is used. (Tables 3 & 4)
bi = b0 + b1 Mi + b2 SEi + ui
(8)
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Simulation Design—Cont.
MRA
n
h
True
Effect
I2
FE-MRA
RE-MRA
WLSMRA
20
20
20
20
20
20
20
80
80
80
80
80
80
80
20
20
20
20
20
20
20
80
80
80
80
80
80
80
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
.0948
.2433
.6014
.8503
.9465
.9761
.9858
.0936
.2469
.6011
.8493
.9465
.9761
.9858
.0593
.3186
.6465
.8687
.9517
.9777
.9863
.0589
.3179
.6471
.8683
.9517
.9777
.9863
.9489
.8769
.7067
.4740
.3088
.2277
.1909
.9495
.8741
.7007
.4769
.3173
.2384
.2047
.9545
.8738
.7070
.4688
.3125
.2301
.1851
.9532
.8704
.7040
.4765
.3153
.2364
.1947
.9544
.9218
.9082
.9191
.9254
.9265
.9233
.9553
.9429
.9371
.9495
.9433
.9460
.9472
.9603
.9187
.8996
.9183
.9220
.9227
.9252
.9568
.9382
.9444
.9460
.9427
.9468
.9436
.9505
.9350
.9079
.9000
.9110
.9339
.9464
.9525
.9350
.9058
.9079
.9167
.9440
.9528
.9531
.9278
.9064
.8996
.9119
.9378
.9455
.9532
.9282
.9138
.9049
.9240
.9393
.9566
.5349
.9352
.9286
Average
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Table 1: Coverage Percentages
MRA n
h
True
Effect
I2
20
20
20
20
20
20
20
80
80
80
80
80
80
80
20
20
20
20
20
20
20
80
80
80
80
80
80
80
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
.0948
.2433
.6014
.8503
.9465
.9761
.9858
.0936
.2469
.6011
.8493
.9465
.9761
.9858
.0593
.3186
.6465
.8687
.9517
.9777
.9863
.0589
.3179
.6471
.8683
.9517
.9777
.9863
Average
RE-MRA WLS-MRA
|Bias|
|Bias|
RE-MRA
MSE
WLS-MRA
MSE
.00059
.00105
.00091
.00085
.00087
.00157
.01341
.00048
.00059
.00029
.00030
.00023
.00012
.00240
.00046
.00186
.00147
.00068
.00118
.00075
.01035
.00067
.00013
.00068
.00009
.00012
.00163
.00195
.00041
.00124
.00157
.00031
.00282
.00014
.00148
.00051
.00040
.00021
.00077
.00031
.00099
.00203
.00042
.00172
.00164
.00107
.00358
.00247
.01111
.00067
.00013
.00060
.00048
.00063
.00005
.00040
.00554
.00829
.01498
.03555
.11340
.40591
1.6279
.00110
.00173
.00331
.00833
.02669
.09887
.38644
.00564
.00825
.01487
.03607
.11352
.39659
1.6164
.00110
.00172
.00333
.00822
.02720
.09808
.38633
.00549
.00845
.01687
.04661
.13435
.35193
.88102
.00109
.00179
.00386
.01066
.02919
.06928
.15535
.00558
.00837
.01706
.04736
.13372
.33989
.83945
.00109
.00177
.00389
.01035
.02953
.06986
.15414
.00163
.00136
.19483
.12064
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Table 2: Bias and MSE
Simulation Results I:
“A house divided by itself cannot stand”—A. Lincoln
applications. Thus, Do Not Divide by root MSE!
• When there is no heterogeneity and FE-MRA is
true, WLS-MRA is equivalent to FE-MRA.
• RE-MRA: When RE-MRA’s model is true,
WLS-MRA provides acceptable and comparable
coverage, and its bias and MSE are a bit better.
• Irony: WLS-MRA dominates RE-MRA in those exact
circumstances for which RE-MRA is designed—
large additive, excess random heterogeneity
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
• FE-MRA: unacceptable SEs in most actual
MRA
n
h
True
Effect
I2
20
20
20
20
20
20
20
80
80
80
80
80
80
80
20
20
20
20
20
20
20
80
80
80
80
80
80
80
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
.1689
.3241
.5697
.8102
.9264
.9670
.9809
.1551
.3589
.6184
.8372
.9362
.9701
.9818
.0825
.2358
.5325
.8083
.9255
.9666
.9806
.0450
.2591
.5926
.8364
.9349
.9695
.9817
Average
RE-MRA WLS-MRA
|Bias|
|Bias|
RE-MRA
MSE
WLS-MRA
MSE
.0348
.0581
.1140
.2367
.4510
.8138
1.5212
.0361
.0668
.1322
.2659
.4900
.8939
1.6566
.0135
.0168
.0350
.0916
.2415
.5566
1.2326
.0101
.0158
.0314
.0940
.2564
.6142
1.3571
.0328
.0510
.0957
.1964
.3470
.5692
.8595
.0345
.0593
.1148
.2250
.3891
.6092
.8880
.0128
.0129
.0221
.0583
.1669
.3541
.6554
.0096
.0115
.0172
.0570
.1740
.3756
.6591
.0151
.0218
.0414
.1084
.3259
1.0391
3.6524
.0039
.0085
.0237
.0824
.2687
.8868
3.0617
.0056
.0083
.0155
.0412
.1567
.6540
2.8299
.0012
.0020
.0042
.0163
.0886
.4553
2.1485
.0147
.0209
.0397
.1035
.2677
.6824
1.6393
.0037
.0075
.0200
.0643
.1815
.4365
.9305
.0056
.0085
.0171
.0477
.1483
.4317
1.1672
.0012
.0019
.0041
.0133
.0559
.1978
.5624
.4049
.2521
.5703
.2527
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Table 3: Bias and MSE with 50% Publication Bias
MRA
n
20
20
20
20
20
20
20
80
80
80
80
80
80
80
20
20
20
20
20
20
20
80
80
80
80
80
80
80
h
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
0
0.125
0.25
0.5
1.0
2.0
4.0
True
Effect
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1
1
1
1
1
1
1
1
1
1
1
1
1
1
Average
I2
.0948
.2433
.6014
.8503
.9465
.9761
.9858
.0936
.2469
.6011
.8493
.9465
.9761
.9858
.0593
.3186
.6465
.8687
.9517
.9777
.9863
.0589
.3179
.6471
.8683
.9517
.9777
.9863
RE-MRA
|Bias|
WLS-MRA
|Bias|
RE-MRA
MSE
WLS-MRA
MSE
.16575
.15454
.10523
.01049
.08977
.11570
.19965
.14279
.12187
.06794
.04014
.14829
.21051
.38580
.02702
.03110
.03162
.02446
.01027
.01052
.05311
.02505
.02933
.03398
.02473
.01618
.04481
.21000
.16364
.14634
.08881
.00049
.07918
.06047
.03766
.14135
.11082
.04757
.05133
.13834
.13721
.06001
.02664
.02868
.02711
.02144
.00171
.04896
.16689
.02471
.02690
.02921
.02216
.00175
.03268
.11280
.05218
.06086
.07024
.11731
.32996
1.0056
3.0241
.02492
.02212
.01648
.02536
.08918
.26245
.84128
.01722
.02553
.04590
.10850
.30811
.97668
2.8764
.00391
.00602
.01039
.02287
.06726
.22621
.70017
.05157
.05960
.07462
.14752
.35647
.84226
1.8469
.02454
.01996
.01601
.03112
.08054
.14630
.25204
.01720
.02607
.05264
.13508
.33235
.75452
1.6098
.00390
.00611
.01168
.02753
.06377
.12240
.23085
.09038
.06553
.40490
.26226
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Table 4: Bias, MSE (50% Pub’bias) FAT-PET-MRA
Simulation Results II
When there is publication bias:
• WLS-MRA always has less bias and, on
average, substantially lower MSE
• Irony: WLS dominates RE in those exact
circumstances for which RE-MRA is designed.
• When RE is better, it is not much better, and
those cases cannot be identified, in practice.
• Thus, there is No Reason to use RandomEffects Meta-Regression. . . . . . Ever!
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
• WLS-MRA dominates RE-MRA.
Why WLS works so well when
there is high heterogeneity?
8
7
6
SE-squared
For very large
heterogeneity, the
random heterogeneity
2
term, n j , will
dominate sampling
error,  e2j , and its
variation, making the
overall variance of the
2
estimate,  j , roughly
2
proportional to n j .
9
5
4
3
2
1
0
0
50
100
150
Heterogeneity Variance
200
250
Unlike S & J, S & J’s MRA Model
Still works after all these Years
• Should we divide MRA SEs by √MSE? Never!
• Is RE-MRA biased with publication bias? Yes!
• WLS-MRA dominates RE-MRA with or
without Correcting for Publication Bias.
• WLS also dominates REE and FEE
weighted averages when combining
Cohen’s d from RCTs.
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
Multiple MRA
• We need more realistic simulation studies
• Alternative modeling strategies {generalto-specific; Bayesian modeling}
• Unbalanced panel MRA models.
• We need to continue to raise the quality
of MRA applications, making them more
robust, comprehensive and rigorous.
• Wish us luck at ASSA in Boston.
T.D. Stanley, Hendrix College
MAER-Net September 12, 2014
The Task Ahead
Thank You!