Dependent effect sizes in Meta-Analysis: Incorporating the Degree of
Interdependence
Context: Occasionally, a sample may contributed more than one effect size. For the
same category of relations, some studies may have one effect size for one sample,
where others may have two or more effect sizes for one sample. Most commonly
used meta-analysis procedures assume that the effect size being meta-analyzed are
independent, we have to decide how to handle multiple effect sizes from a single
sample.
The case of Independent Effect Sizes
Suppose there are K studies in a meta-analysis, the sample correlations (Peason r)
given by the ith sample is denoted by ri. It was assumed that the “true” correlation
underlying the K samples are heterogeneous and follow an arbitrary distribution with
a mean of . and a variance of 𝜎𝜌2 . Suppose only two sources of variation for each
sample correlation, the “true” variation 𝜎𝜌2 and the sampling error 𝜎𝑒2 .
Hunter-Schmidt (1990) approach: assumes a random effects model which does not
assume the population correlations are homogeneous across studies. It is expressed
as the weighted average of sample correlation:
∑(𝑛𝑖 − 1)𝑟𝑖 ∑(𝑛𝑖 − 1)𝑟𝑖
𝑟̅ =
=
∑(𝑛𝑖 − 1)
𝑁−𝐾
where N is the total sample size of all K samples. In the case of independent
corrlations, Hunter and Schmidt provided formulae to estimate the degree of
Heterogeneity.
But it is biased shown by Martinussen & Bjornstad (1999). They provided a revised
formula to estimate the degree of heterogeneity
where 𝑆𝑟2 is the weighted variance of 𝑟𝑖 s
The case of dependent effect sizes
Two types of dependence:
(1) A study may report the correlations between two variables across several
conditions for a single sample.
(2) A study may report the correlations between a criterion and several different
predictors. (itercorrelations among all the variables are available)
Two problems need to be solved:
(1) Whether the within-sample heterogeneity is the same as the between-samples
heterogeneity (the simpler case that the within-sample correlations are
homogeneous in study 2004)
(2) If the within-sample heterogeneity is different from the between-sample
heterogeneity, we need to find a way to incorporate this fact in the overall
meta-analysis.
𝜌𝑟𝑟𝑖 is an index of intercorrelations which is the correlation between any two sample
correlations 𝑟𝑖𝑘 and 𝑟𝑖𝑙 given by the ith sample. There are two parameters 𝜌𝑥𝑥𝑖
and 𝜌𝑒𝑒𝑖 which lead to 𝜌𝑟𝑟𝑖 . 𝜌𝑥𝑥𝑖 represents a common factor among the x
variables, 𝜌𝑒𝑒𝑖 represents another common factor among the unexplained portion.
𝜌𝑟𝑟𝑖 = 0: pi sample correlations are actually independent, hence, the only source of
variance of correlations is the sampling error
𝜌𝑟𝑟𝑖 = 1: pi sample correlations are identical in value, hence the variance of the
correlations from a sample is zero
Samplewise Procedure
compute a weighted average of the dependent correlations contributed by a sample:
where wk is the weight associated with the kth correlations in the ith sample.
If the correlations are obtained from the same sample, then
Two ways to use the sample size to weight the within-sample average:
(1) samplewise-N procedure: The original sample size is used as the weight
(regardless of the number of effect sized this sample contributes)
(2) Samplewise-PN procedure (rarely used)
use the pini (the sample size multiplied by the number of correlations) is used as the
weight
The samplewise procedure, to the extent that the correlations used to compute the
average are independent, the sampling error variance for the average correlation is
overestimated and hence 𝜎𝜌2 , the degree of heterogeneity or the variance of the
population correlations is underestimated. The problems arises from the use of
incorrect sample size in weighting the within-sample average.
A more accurate weight for the within-sample average
Solution: compute the weight, taking into account the degree of interdependence.
Step 1: compute the weight
2
𝜌̂𝑟𝑟𝑖
𝐶𝑖 =
1+(𝑝𝑖 −1)𝜌𝑟𝑟𝑖
𝑛∗𝑖 =
𝑝𝑖
𝑛𝑖 −1
𝐶𝑖
2
∑(𝑟𝑖𝑗 − 𝑟̅𝑖 ) /(𝑝𝑖 − 1)
𝑆𝑟𝑖
≈1− 2 ≈1−
(1 − 𝑟̅ 2 )2 /(𝑛𝑖 − 1)
𝑆𝑒𝑖
, (correction facror)
+ 1, (adjusted sample size)
the adjusted-individual procedure: 𝑟𝑟𝑟𝑖 is computed for each study with multiple
effect sizes, and the sample size for each of these studies is adjusted by the
estimated degree of dependence.
̅̅̅̅
the adjusted-weighted procedure: 𝜌
̂𝑟𝑟 is used to adjusted the sample size of all
studies with multiple effect sizes.