Publication/Availability Bias
Problems with bad data
Problem of missing studies
• Missing at random is okay
• Nonrandom is a problem
• Sources of nonrandom samples of
studies
– Publication bias – significant, small N
– Grad student work; other filedrawer
– Deliberate misrepresentation for financial
reasons
Forest Plot
Drift to the right by
precision?
Source: Borenstein,
Hedges, Higgins &
Rothstein, 2009, p. 282
Funnel plot
Assymetry?
Esp lower right.
These appear
biased and
heterogeneous to
me.
Source: Sutton (2009). In Cooper, Hedges, & Valentine (Eds)
Handbook fo Research Synthesis Methods p. 501
Fail-safe N
• How many studies do we need to make
the result n.s.? (Rosenthal)
• How many studies do we need to make
the summary effect small enough to
ignore? (Orwin)
kobt (d obt  d c )
k fs 
d c  d fs
Corwin, R. G. (1983). A fail-safe
N for effect size in meta-analysis.
Journal of Educational Statistics,
8, 157-159.
kfs = failsafe studies
kobt = studies in meta
dobt = summary ES
dc = desired lower boutnd
dfs= studies with this (eg 0)
size needed to lower ES
Trim & Fill
Creates symmetry;
Adjusts summary ES
Source: Borenstein, Hedges,
Higgins & Rothstein, 2009, p. 287
Cumulative Forest
Source: Borenstein, Hedges, Higgins &
Rothstein, 2009, p. 288
Egger’s Regression
Ti = effect size; vi=sampling variance of ES
zi  Ti / vi
zi  0  1 (1/ vi )   i
Should be flat (centered)
if no bias. This shows
small studies have higher
values.
Source: Sutton (2009). In Cooper,
Hedges, & Valentine (Eds) Handbook
fo Research Synthesis Methods p. 441
Sensitivity Analysis
Outliers. Run twice. Pray.
Source: Greenhouse & Iyengar (2009).
In Cooper, Hedges, & Valentine (Eds)
Handbook fo Research Synthesis
Methods p. 422
Sensitivity Analysis
Varying levels of tau-squared
Source: Louis & Zelterman (1994).
In Cooper, Hedges (Eds) Handbook of
Research Synthesis p. 418