Quantitative Synthesis I
Prepared for:
The Agency for Healthcare Research and Quality (AHRQ)
Training Modules for Systematic Reviews Methods Guide
www.ahrq.gov
Systematic Review Process Overview
Example: Meta
Meta-Analysis
Analysis Data Set
BetaBeta
-Blockers after Myocardial Infarction - Secondary Prevention
N
===
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Study
============
Reynolds
Wilhelmsson
Ahlmark
Multctr. Int
Baber
Rehnqvist
Norweg.Multr
Taylor
BHAT
Julian
Hansteen
Manger Cats
Rehnqvist
q
ASPS
EIS
LITRG
Herlitz
Year
====
1972
1974
1974
1977
1980
1980
1981
1982
1982
98
1982
1982
1983
1983
1983
1984
1987
1988
Experiment
Control
Obs
Tot
Obs
Tot
====== ====== ====== ======
3
38
3
39
7
114
14
116
5
69
11
93
102
1533
127
1520
28
355
27
365
4
59
6
52
98
945
152
939
60
632
48
471
138
38
1916
9 6
188
88
1921
9
64
873
52
583
25
278
37
282
9
291
16
293
25
154
31
147
45
263
47
266
57
858
45
883
86
1195
93
1200
169
698
179
697
Odds
Ratio
=====
1.03
0.48
0.58
0.78
1.07
0.56
0.60
0.92
0.
0.72
0.81
0.65
0.55
0.73
0.96
1.33
0.92
0.92
95%
CI
Low
High
===== =====
0.19
5.45
0.18
1.23
0.19
1.76
0.60
1.03
0.62
1.86
0.15
2.10
0.46
0.79
0.62
1.38
0.5
0.57
0.90
0.55
1.18
0.38
1.12
0.24
1.27
0.40
1.30
0.61
1.51
0.89
1.98
0.68
1.25
0.73
1.18
Combining Effect Estimates
What is the average (overall) treatment‐control diff
difference in blood pressure?
i bl d
?
Study
N
Mean difference
(mm Hg)
A
554
‐6.2
‐6.9 to ‐5.5
B
304
‐7.7
‐10.2 to ‐5.2
C
39
‐0.1
‐6.5 to 6.3
95% Confidence Interval
Simple Average
What is the average (overall) treatment‐control diff
difference in blood pressure?
i bl d
?
(‐6.2) 6.2) + ((‐
(‐7.7) 7.7) + ((‐
(‐0.1)
4 7 mm Hg
= ‐4.7 mm Hg
3
Study
y
N
M
Mean difference
mmHg
g
A
554
‐6.2
‐6.9 to ‐5.5
B
304
‐7.7
‐10.2 to ‐5.2
C
39
‐0.1
‐6.5 to 6.3
95% CI
Weighted Average
What is the average (overall) treatment‐control diff
difference in blood pressure?
i bl d
?
(554 x -6.2)
(554 x 6.2) 6.2) + (304 x (304 x -7.7)
7.7)
7.7) + (39 x (39 x -0.1)
0.1)
6.4 mm Hg
6 4 mm Hg
= -6.4 mm Hg
554 + 304 + 39
Study
y
N
M
Mean difference
mmHg
g
A
554
‐6.2
‐6.9 to ‐5.5
B
304
‐7.7
77
‐10.2 to ‐5.2
10 2 t 5 2
C
39
‐0.1
k
95% CI
‐6.5 to 6.3
X 
w x
i 1
k
i i
w
i 1
i
General Formula:
Weighted Average Effect Size (d+)
k
d 

i 1
k
wid i

i 1
Where:
di = effect size of the ith study
wi = weight of the ith study
k = number of studies
wi
Calculation of Weights
Generallyy is the inverse of the variance of
treatment effect (that captures both study size
and precision))
Different formula for odds ratio, risk ratio, and
risk difference
Readily available in books and software
Heterogeneity (Diversity)
Is it reasonable?
Are the characteristics and effects of studies sufficiently similar to
estimate an average effect?
Types of heterogeneity:
Clinical diversity
Methodological diversity
Statistical heterogeneity
Clinical Diversity
Are the studies of similar treatments,
populations, settings, design, et cetera, such that
an average effect would be clinically meaningful?
Example: A Meta-analysis With a
Large Degree of Clinical Diversity
25 randomized controlled trials compared
endoscopic hemostasis with standard therapy for
bleeding peptic ulcer.
5 different types of treatment were used: monopolar
electrode, bipolar electrode, argon laser,
neodymium-YAG laser, and sclerosant injection.
4 different conditions were treated: active bleeding,
g,
a nonspurting blood vessel, no blood vessels seen,
and undesignated.
3 different outcomes were assessed: emergency
g y overall mortality,
y and recurrent bleeding.
g
surgery,
Sacks HS, et al. JAMA 1990;264:494-9.
Methodological Diversity
Are the studies of similar design
g and conduct
such that an average effect would be clinically
meaningful?
Statistical Heterogeneity
Is the observed variabilityy of effects g
greater than
that expected by chance alone?
Two statistical measures are commonly used to
assess statistical heterogeneity:
Cochran’ss Q
Cochran
Q-statistics
statistics
I2 index
Example: A Fixed Effect Model
Suppose that we have a
container
t i
with
ith a very llarge
number of black and white
balls.
The ratio of white to black
balls is predetermined and
fi d
fixed.
We wish to estimate this
ratio.
ratio
Now, imagine that the
container represents a
clinical condition and the
balls represent outcomes.
Random Sampling From a Container With a Fixed Number
of White and Black Balls (Equal Sample Size)
Random Sampling From a Container With a Fixed Number
of Black and White Balls (Different Sample Size)
Different Containers With Different Proportions of Black
and White Balls (Random Effects Model)
Random Sampling From Containers To Get an Overall
Estimate of the Proportion of Black and White Balls
Statistical Models of Combining 2x2 Tables
Fixed effect model: assumes a common treatment effect.
For inverse variance weighted method, the precision of the
estimate determines the importance of the study.
The Peto and Mantel-Haenzel methods are noninverse variance
weighted fixed effect models.
Random effects model: in contrast to the fixed
fi ed effect
model, accounts for within-study variation.
The most popular random effects model in use is the
DerSimonian and Laird inverse variance weighted method,
which calculates the sum of the within-study variation and the
among-study variation.
Random effects model can also be implemented with Bayesian
methods.
methods
Example Meta-analysis Where Fixed and the
Random Effects Models Yield Identical Results
Example Meta-analysis Where Results from
Fixed and Random Effects Models Will Differ
Gross PA, et al. Inn Intern Med 1995;123:518-27. Reprinted with permission from the American College of Physicians.
Weights of the Fixed Effect
and Random Effects Models
Fixed Effect Weight
R d
Random
Eff
Effects
t W
Weight
i ht
1
wi 
vi
1
w 
vi  v *
*
i
where: vi = within studyy variance
v* = between study variance
Commonly Used Statistical Methods
for Combining 2x2 Tables
Odds Ratio
Risk Ratio
Risk Difference
Fixed Effect • Mante
• Mantel‐Haenszel • Inverse variance Model
• l‐Haenszel Peto
• Inverse variance weighted
g
weighted
• Exact
E t
• Inverse variance weighted
Random • DerSimonian • DerSimonian • DerSimonian Effects and Laird
and Laird
and Laird
Model
Dealing With Heterogeneity
Lau J,
L
J ett al.
l Ann
A Intern
I t
Med
M d 1997;127:820-6.
1997 127 820 6 Reprinted
R i t d with
ith permission
i i from
f
the
th American
A
i
College
C ll
off
Physicians.
Summary:
Statistical Models of Combining 2x2 Tables
Most meta-analyses of clinical trials combine
treatment effects (risk ratio,
ratio odds ratio
ratio, risk
difference) across studies to produce a common
estimate, by using either a fixed effect or random
effects
ff
model.
d l
In practice, the results from using these two models
are similar when there is little or no heterogeneity.
heterogeneity
When heterogeneity is present, the random effects
model generally produces a more conservative result
(smaller Z-score) with a similar estimate but also a
wider confidence interval; however, there are rare
exceptions of extreme heterogeneity where the
random effects model may yield counterintuitive
results.
Caveats
Manyy assumptions
p
are made in meta-analyses,
y
so
care is needed in the conduct and interpretation.
Most meta
meta-analyses
analyses are retrospective exercises,
exercises
suffering from all the problems of being an
observational design.
design
Researchers cannot make up missing information
or fix poorly collected,
collected analyzed
analyzed, or reported
data.
Key Messages
Basic meta-analyses can be easily carried out with
readily
dil available
il bl statistical
i i l software.
f
Relative measures are more likely to be
h
homogeneous
across studies
t di and
d are generally
ll
preferred.
The random effects model is the appropriate
statistical model in most instances.
The decision to conduct a meta-analysis
meta analysis should be
based on:
a
a well
well-formulated
formulated question,
appreciation of the heterogeneity of the data, and
understanding of how the results will be used.