Practical Meta-Analysis
These lectures are based on the book
Practical Meta-Analysis
by David B. Wilson & Mark W. Lipsey
1
Outline of Meta-Analysis
Topics covered will include:
What is meta-analysis ?
Meta-analysis issues:
•
•
•
•
•
•
•
•
•
•
Problem definition and basic concepts
Finding eligible studies, criterion for inclusion
Coding issues, key variables, etc.
Effect sizes (ES) and computational issues
Combining effect sizes using meta-analysis
Publication bias
Interpretation of results
Evaluating the quality of a meta-analysis
Outline of Meta-Analysis
Topics covered will include:
•
•
Effect Sizes (ES) – these are combined using
meta-analysis
Conducting a meta-analysis – how do we
combine effect sizes from multiple studies to
obtain an aggregate effect size.
What is Meta-analysis?
• Meta-analysis focuses on the aggregation
the findings of from multiple research
studies.
• These studies must produce quantitative
findings from empirical research.
• In order combine the study results we
need a standard measure of effect size (ES)
that can be calculated for each study and
then combined.
What is Meta-analysis?
• Meta-analysis essentially takes a weighted
average of the effect sizes from each of the
studies.
• The weights take into account sample sizes
and variability of the results from each study.
• We also may need to account for random
differences between studies due to variations
in procedures, settings, populations sampled,
etc.
Example: Effectiveness of Correctional
Boot-Camps for Prisoners
• 32 unique studies were found, that reported on
43 independent boot-camp/comparison samples
looking recidivism as the response.
• For each study, the odds ratio for recidivism was
calculated looking at the “benefit” of the
prisoner boot-camps.
• Random differences between the differences in
the study designs, prisoner populations,
protocols, etc. were accounted for.
Forest Plot from a Meta-Analysis oft
Correctional Boot-Camps (Wilson et al.)
F
a
vors C
om
pa
riso
n
H
a
rer &K
lein
-S
a
ffra
n, 19
96
Jo
ne
s &R
o
ss, 1
9
97
F
l. D
ep
t. of JJ (S
tua
rt C
o.), 19
97
F
l. D
ep
t. of JJ (P
olk C
o
., B
o
ys), 1
99
7
Jo
ne
s (F
Y
97
), 1
99
8
Jo
ne
s (F
Y
94
-95
), 1
99
8
M
acke
nzie&S
o
urya
l (Illin
ois), 1
99
4
M
acke
nzie&S
o
urya
l (Lo
uisian
a), 19
94
Jo
ne
s (F
Y
91
-93
), 1
99
8
M
acke
nzie&S
o
urya
l (F
lorida
), 1
99
4
Jo
ne
s (F
Y
96
), 1
99
8
M
arcus-M
en
do
za(M
en
), 1
99
5
M
acke
nzie
,e
ta
l. 1
99
7
P
en
n
.D
e
pt. o
fC
orrectio
ns, 2
00
1
F
lo
w
ers, C
a
rr, &R
ub
ack 19
91
B
ure
auo
fD
ataa
ndR
e
sea
rch, 19
9
6
M
acke
nzie&S
o
urya
l (O
kla
ho
m
a
), 1
99
4
T
3A
ssocia
te
s, 2
00
0
M
acke
nzie&S
o
urya
l (N
ewY
ork), 1
99
4
P
eters, 19
96
b
C
a
m
p&S
an
dh
u, 1
99
5
M
acke
nzie&S
o
urya
l (S
.C
., N
ew
), 19
9
4
Jo
ne
s, 1
9
96
N
YD
C
S(88
-9
6R
e
le
ase
s), 2
00
0
M
arcus-M
en
do
za(W
o
m
e
n), 19
95
A
ustin
, Jon
es, &B
o
lyard
,1
99
3
B
urn
s &V
ito, 19
95
P
eters, 19
96
a
F
l. D
ep
t. of JJ (B
ay C
o.), 19
9
7
N
YD
C
S(96
-9
7R
e
le
ase
s), 2
00
0
N
YD
C
S(97
-9
8R
e
le
ase
s), 2
00
0
F
l. D
ep
t. of JJ (P
in
ellas C
o
.), 1
99
6
F
l. D
ep
t. of JJ (M
a
na
te
eC
o.), 1
99
6
C
AD
ep
t. of theY
ou
thA
utho
rity, 1
99
7
B
oyle
s, B
o
ken
kam
p
, &M
ad
ura
,1
99
6
M
acke
nzie&S
o
urya
l (S
.C
., O
ld
), 1
99
4
F
l. D
ep
t. of JJ (P
olk C
o
., G
irls), 1
99
7
Jo
ne
s, 1
9
97
T
h
om
as &P
e
te
rs, 19
9
6
W
righ
t &M
ays, 19
98
M
acke
nzie&S
o
urya
l (G
e
o
rg
ia), 19
94
O
ve
ra
ll M
e
anO
dd
s-R
atio
F
a
vors B
o
otcam
p
The Great Debate & Historical Background





1952: Hans J. Eysenck concluded that there
were no favorable effects of psychotherapy,
starting a raging debate.
20 years of evaluation research and hundreds
of studies failed to resolve the debate
1978: To prove Eysenck wrong, Gene V. Glass
statistically aggregated the findings of 375
psychotherapy outcome studies.
Glass (and colleague Smith) concluded that
psychotherapy did indeed work
Glass called his method “meta-analysis”
The Emergence of Meta-Analysis
•
The ideas behind meta-analysis predate
Glass’ work by several decades
Karl Pearson (1904)
•
•
averaged correlations for studies of the effectiveness of
inoculation for typhoid fever
R. A. Fisher (1944)
•
•
•
“When a number of quite independent tests of significance
have been made, it sometimes happens that although few or
none can be claimed individually as significant, yet the
aggregate gives an impression that the probabilities are on the
whole lower than would often have been obtained by chance”.
Source of the idea of cumulating probability values
The Emergence of Meta-analysis
•
Ideas behind meta-analysis predate Glass’
work by several decades.
W. G. Cochran (1953)
•
•
•
Discusses a method of averaging means across
independent studies
Laid-out much of the statistical foundation that
modern meta-analysis is built upon (e.g., Inverse
variance weighting and homogeneity testing)
The Logic of Meta-analysis
•
•
Traditional methods of review focus on statistical
significance testing
Significance testing is not well suited to this task
Highly dependent on sample size
Null finding does not carry the same “weight” as a
significant finding
•
•
•
•
•
significant effect is a strong conclusion
non-significant effect is a weak conclusion
Meta-analysis focuses on the direction and magnitude
of the effects across studies, not statistical significance
•
•
Isn’t this what we are interested in anyway?
Direction and magnitude are represented by the effect size
When Can You Do Meta-Analysis?
•
Meta-analysis is applicable to collections of
research that
•
•
•
•
•
Are empirical, rather than theoretical
Produce quantitative results, rather than
qualitative findings
Examine the same constructs and relationships
Have findings that can be configured in a
comparable statistical form (e.g., as effect sizes,
correlation coefficients, odds-ratios, proportions)
Are “comparable” given the question at hand
Forms of Research Findings
Suitable to Meta-analysis
•
Central tendency research
Prevalence rates
•
•
Pre-post contrasts
Growth rates
•
•
Group contrasts
Experimentally created groups
•
•
Comparison of outcomes between treatment and
comparison groups
Naturally occurring groups
•
•
•
Comparison of spatial abilities between boys and girls
Rates of morbidity among high and low risk groups
Forms of Research Findings
Suitable to Meta-analysis
•
Association between variables
Measurement research
•
•
E.g. Validity generalization
Individual differences research
•
•
E.g. Correlation between personality constructs
Effect Size: The Key to Meta-Analysis
•
The effect size (ES) makes meta-analysis
possible
•
•
It is the “dependent variable” in a metaanalysis
It standardizes findings across studies such
that they can be directly compared and
aggregated.
Effect Size: The Key to Meta-Analysis
•
Any standardized index can be an “effect
size” (e.g., standardized mean difference,
correlation coefficient, odds-ratio) as long as
it meets the following:
•
•
•
•
It is comparable across studies (generally
requires standardization)
Represents the magnitude and direction of the
relationship of interest
It is independent of sample size
Different meta-analyses may use different
effect size indices.
The Replication Continuum
Pure
Replications
Conceptual
Replications
You must be able to argue that the collection of studies you are
meta-analyzing examine the same relationship. This may be at
a broad level of abstraction, such as the relationship between a
risk factor and disease incidence or between a medical therapy
and outcome of interest. Alternatively it may be at a narrow
level of abstraction and represent pure replications.
The closer to pure replications your collection of studies, the
easier it is to argue comparability.
Which Studies to Include?
•
It is critical to have an explicit inclusion and
exclusion criteria. The broader the research
domain, the more detailed they tend to become.
•
•
May need to refine criteria as you interact with the
literature.
Components of a detailed criteria for inclusion:
•
•
•
•
•
•
•
distinguishing features
research respondents
key variables
research methods
cultural and linguistic range
time frame
publication types
Methodological Quality Dilemma
•
Include or exclude low quality studies?
•
•
•
•
•
The findings of all studies are potentially in error
(methodological quality is a continuum, not a
dichotomy, i.e. bad vs. good)
Being too restrictive may restrict ability to generalize
Being too inclusive may weaken the confidence that
can be placed in the findings.
Methodological quality is often in the “eye-of-thebeholder”.
You must strike a balance that is appropriate to your
research question.
Searching Far and Wide
•
•
•
The “we only included published studies
because they have been peer-reviewed”
argument.
Significant findings are more likely to be
published than non-significant findings.
This is a major source of bias!
Critical to try to identify and retrieve all
studies that meet your eligibility criteria
Searching Far and Wide
•
Potential sources for identification of documents:
•
Computerized bibliographic databases
(WSU Databases – PubMed, Proquest, OVID, CINAHL, etc.)
•
•
•
•
•
•
•
“Google” internet search engine
Authors working in the research domain (email a
relevant Listserv?)
Conference programs and proceedings
Dissertations
Review articles
Hand searching relevant journals
Government reports, bibliographies, clearinghouses
A Note About Computerized
Bibliographies
•
•
•
•
•
Rapidly changing area
Get to know your local librarian!
Searching one or two databases is
generally inadequate.
Use “wild cards” (e.g., random? will find
random, randomization, and randomize)
Throw a wide net; filter down with a
manual reading of the abstracts
Strengths of Meta-Analysis
•
•
•
•
•
Imposes a discipline on the process of summing
up research findings
Represents findings in a more differentiated and
sophisticated manner than conventional reviews
Capable of finding relationships across studies
that are obscured in other approaches
Protects against over-interpreting differences
across studies
Can handle a large numbers of studies (this
would overwhelm traditional approaches to
review)
Weaknesses of Meta-Analysis
•
•
•
•
Requires a good deal of effort
Mechanical aspects don’t lend themselves
to capturing more qualitative distinctions
between studies
“Apples and oranges” criticism
Most meta-analyses include “blemished”
studies to one degree or another (e.g., a
randomized design with attrition)
Weaknesses of Meta-Analysis
•
Selection bias poses a continual threat
• Negative and null finding studies that
you were unable to find.
•
Outcomes for which there were
negative or null findings that were not
reported.