Funded through the ESRC’s Researcher
Development Initiative
Session 1.3 – Equations
Prof. Herb Marsh
Ms. Alison O’Mara
Dr. Lars-Erik Malmberg
Department of Education,
University of Oxford
Session 1.3 – Equations
Establish
research
question
Define
relevant
studies
Develop code
materials
Data entry
and effect size
calculation
Pilot coding;
coding
Locate and
collate studies
Main analyses
Supplementary
analyses
3
In this and following formulae, we will
use the symbols d and δ to refer to
any measure for the observed and the
true effect size, which is not necessarily
the standardized mean difference.
Formula for the observed effect size in a fixed
effects model
d j    ej
Where
dj is the observed effect size in study j
δ is the ‘true’ population effect
and ej is the residual due to sampling variance
in study j
 To calculate the overall mean observed effect size
(dj in the fixed effects equation)
dj
d j    ej
( w d )


w
i
i
i
 where wi = weight for the individual effect size, and
di = the individual effect size.
The effect sizes are weighted by the inverse of the
variance to give more weight to effects based on
large sample sizes
1
wi 
vi
The standard error of each effect size is given by
the square root of the sampling variance
SE =  vi
The variances are calculated differently for each
type of effect size.
6
Variance for standardised mean difference effect
size is calculated as
di2
(n1 n 2 )
vi 

(n1 n 2 )
2(n1 n 2 )
 Where n1 = sample size of group 1, n2 is the sample size of group
2, and di = the effect size for study i.
Variance for correlation effect size is calculated as
1
vi 
ni  3
 Where ni is the total sample size of the study
7
 Expand the general model to include predictors
s
d j   0    s X sj  e j
s 1
 Where
 βs is the regression coefficient (regression slope)
for the explanatory variable.
 Xsj is the study characteristic (s) of study j.
Example: Gender as a predictor of
achievement
achievement j   0    genderj  e j
Formula for the observed effect size in a
random effects model
d j    u j  ej
Where
dj is the observed effect size in study j
δ is the mean ‘true’ population effect size
uj is the deviation of the true study effect size
from the mean true effect size
and ej is the residual due to sampling
variance in study j
 To calculate the overall mean observed effect size
(dj in the random effects equation)
dj
d j    u j  ej
( w d )


w
i
i
i
 where wi = weight for the individual effect size, and
di = the individual effect size.
Random effects differs from fixed effects in the
calculation of the weighting (wi)
The weight includes 2 variance components: withinstudy variance (vi) and between-study variance (vθ)
The new weighting for the random effects model
(wiRE) is given by the formula:
wiRE
1

vi  v
Recall the weighting
1
wi 
formula for fixed
vi
effects model:
vi is calculated the same as in the fixed effects
models.
12
vθ is calculated using the following formula
v 
Q  ( k  1)
w 
i
 wi 2
 wi
Where Q = Q-statistic (measure of whether effect
sizes all come from the same population)
k = number of studies included in sample
wi = effect size weight, calculated based on fixed
effects models.
13
Thus, larger studies receive proportionally less
weight in RE model than in FE model.
This is because a constant is added to the
denominator, so the relative effect of sample
size will be smaller in RE model
14
 If the homogeneity test is rejected (it almost always
will be), it suggests that there are larger differences
than can be explained by chance variation (at the
individual participant level). There is more than one
“population” in the set of different studies.
 The random effects model determines how much of
this between-study variation can be explained by
study characteristics that we have coded.
 Expand the general model to include predictors
s
d j   0    s X sj  u j  e j
s 1
 Where
 βs is the regression coefficient (regression slope)
for the explanatory variable.
 Xsj is the study characteristic (s) of study j.
Example: Gender as a predictor of
achievement
achievement j  0    genderj  e j
Formula for the observed effect size in a multilevel
model
d j   0  u j  ej
Where
dj is the observed effect size in study j
0 is the mean ‘true’ population effect size
uj is the deviation of the true study effect size from the
mean true effect size
and ej is the residual due to sampling variance in
study j
Note: This model treats the moderator effects as fixed
and the ujs as random effects.
s
d j   0   s X sj  u j  e j
s 1
In this equation, predictors are included in the
model.
s is the regression coefficient (regression slope)
for the explanatory variable. (Equivalent to β in
multiple regression.)
Xsj is the study characteristic (s) of study j.
Example: Gender as a predictor of
achievement
achievement j   0    genderj  u j  e j
s
d j   0   s X sj  u j  e j
s 1
 If between-study variance = 0, the multilevel model
simplifies to the fixed effects regression model
s
d j   0   s X sj  e j
s 1
 If no predictors are included the model simplifies to
random effects model
d j    u j  ej
 If the level 2 variance = 0 , the model simplifies to
the fixed effects model
d j    ej
 Many meta-analysts use an adaptive (or
“conditional”) approach
IF between-study variance is found in the
homogeneity test
THEN use random effects model
OTHERWISE use fixed effects model
 Fixed effects models are very common, even
though the assumption of homogeneity is
“implausible” (Noortgate & Onghena, 2003)
 There is a considerable lag in the uptake of new
methods by applied meta-analysts
 Meta-analysts need to stay on top of these
developments by
 Attending courses
 Wide reading across disciplines
24
Usually start with a Q-test to determine the overall
mean effect size and the homogeneity of the effect
sizes (MeanES.sps macro)
If there is significant homogeneity, then:
 1) should probably conduct random effects analyses
instead
 2) model moderators of the effect sizes (determine the
source/s of variance)
ES i
The homogeneity (Q) test asks whether the different effect sizes
are likely to have all come from the same population (an
assumption of the fixed effects model). Are the differences
among the effect sizes no bigger than might be expected by
chance?

di
Q   wi d id

2
= effect size for each study (i = 1 to k)
= mean effect size
= a weight for each study based on the sample size
However, this (chi-square) test is heavily dependent on sample size. It is
almost always significant unless the numbers (studies and people in
each study) are VERY small. This means that the fixed effect model will
almost always be rejected in favour of a random effects model.
Significant heterogeneity in the
effect sizes therefore random
effects more appropriate and/or
moderators need to be modelled
27
The analogue to the ANOVA homogeneity analysis
is appropriate for categorical variables
 Looks for systematic differences between groups of
responses within a variable
 Easy to implement using MetaF.sps macro
 MetaF ES = d /W = Weight /GROUP = TXTYPE /MODEL =
FE.
Multiple regression homogeneity analysis is more
appropriate for continuous variables and/or when
there are multiple variables to be analysed
 Tests the ability of groups within each variable to predict
the effect size
 Can include categorical variables in multiple regression
as dummy variables
 Easy to implement using MetaReg.sps macro
 MetaReg ES = d /W = Weight /IVS = IV1 IV2 /MODEL = FE.
 Like the FE model, RE uses ANOVA and multiple
regression to model potential
moderators/predictors of the effect sizes, if the Qtest reveals significant heterogeneity
 Easy to implement using MetaF.sps macro (ANOVA)
or MetaReg.sps (multiple regression).
 MetaF ES = d /W = Weight /GROUP = TXTYPE /MODEL =
ML.
 MetaReg ES = d /W = Weight /IVS = IV1 IV2 /MODEL = ML.
Significant heterogeneity in
the effect sizes therefore
need to model moderators
v 
Q  ( k  1)
w 
i
 wi 2
 wi
31
Similar to multiple regression, but corrects the
standard errors for the nesting of the data
Start with an intercept-only (no predictors) model,
which incorporates both the outcome-level and the
study-level components
This tells us the overall mean effect size
Is similar to a random effects model
Then expand the model to include predictor
variables, to explain systematic variance between
the study effect sizes
32
d j   0  u j  ej
 (MLwiN screenshot)
s
d j   0   s X sj  u j  e j
s 1
 Using the same simulated data set with n = 15
 The random effects is better than the fixed effects
approach in almost all conceivable cases
 “The results of the simulation study suggest that
the maximum likelihood multilevel approach is in
general superior to the fixed-effects approaches,
unless only a small number of studies is available.
For models without moderators, the results of the
multilevel approach, however, are not substantially
different from the results of the traditional randomeffects approaches” (p. 765)
 Multilevel models:
 build on the fixed and random effects models
 account for between-study variance (like random effects)
 Are similar to multiple regression, but correct the
standard errors for the nesting of the data. Improved
modelling of the nesting of levels within studies
increases the accuracy of the estimation of standard
errors on parameter estimates and the assessment of the
significance of explanatory variables (Bateman and
Jones, 2003).
 Multilevel modelling is more precise when there is
greater between-study heterogeneity
 Also allows flexibility in modelling the data when
one has multiple moderator variables (Raudenbush
& Bryk, 2002)
 Multilevel modelling has the promise of being able
to include multivariate data – still being developed
 Easy to implement in MLwiN (once you know how!)
 See worked examples for HLM, MLwiN, SAS, & Stata at
http://www.ats.ucla.edu/stat/examples/ma_hox/default.htm
 Lipsey, M. W., & Wilson, D. B. (2001). Practical
meta-analysis. Thousand Oaks, CA: Sage
Publications.
 Van den Noortgate, W., & Onghena, P. (2003).
Multilevel meta-analysis: A comparison with
traditional meta-analytical procedures. Educational
and Psychological Measurement, 63, 765-790.
 Wilson’s “meta-analysis stuff” website:
http://mason.gmu.edu/~dwilsonb/ma.html
 Raudenbush, S.W.
and Bryk, A.S. (2002). Hierarchical
nd
Linear Models (2 Ed.).Thousand Oaks: Sage
Publications.