Handout H6
The scope of meta-analysis:
Meta-analysis of observational studies
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Objectives
• Understand the importance of systematic
reviews of observational studies
• Understand the limitations of meta-analysis
in observational studies
• Understand the difficulties of avoiding
publication bias in observational studies
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Why do we need observational
studies?
• Randomisation may be
– impossible
– unnecessary
– inappropriate
Black, BMJ 1996
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Potentials of systematic reviews
• More objective appraisal of the evidence than traditional
narrative reviews applies equally to OS & RCT
• May resolve uncertainty when original research, reviews
and editorials disagree applies equally to OS & RCT
• May generate promising research questions to be
addressed in future studies applies equally to OS & RCT
• Meta-analysis will enhance the precision of effect
estimates, leading to reduced probability of false negative
results BUT in OS may be a precise biased result
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Meta-analysis
• A statistical analysis which combines the
results of several independent studies
considered by the analyst to be
‘combinable’
Huque 1988
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Assumptions in meta-analysis
• “Fixed-effects model”: Underlying effect is
the same value (fixed) in each study. The
differences between study results are solely
due to the play of chance.
• “Random-effects model”: Treatment effect
for the individual studies are assumed to
vary around some overall central effect
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Fundamental difference in assumptions & how
they apply to MA of RCT or observational studies
• In meta-analysis of observational studies
confounding, residual confounding and bias:
– May introduce heterogeneity
– May lead to misleading (albeit very precise) estimates
• In well-conducted RCT there should not be
confounding
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Trial (Year)
Barber (1967)
Reynolds (1972)
Wilhelmsson (1974)
Ahlmark (1974)
Multicentre International (1975)
Yusuf (1979)
Andersen (1979)
Rehnqvist (1980)
Baber (1980)
Wilcox Atenolol (1980)
Wilcox Propanolol (1980)
Hjalmarson (1981)
Norwegian Multicentre (1981)
Hansteen (1982)
Julian (1982)
BHAT (1982)
Taylor (1982)
Manger Cats (1983)
Rehnqvist (1983)
Australian-Swedish (1983)
Mazur (1984)
EIS (1984)
Salathia (1985)
Roque (1987)
LIT 91987)
Kaul (1988)
Boissel (1990)
Schwartz low risk (1992)
Schwartz high risk (1992)
SSSD (1993)
Darasz (1995)
Basu (1997)
Aronow (1997)
Mortality results from 33
trials of beta-blockers in
secondary prevention after
myocardial infarction
Adapted from Freemantle et al BMJ 1999
0.80 (0.74 - 0.86)
Overall (95% CI)
0.1
0.2
0.5
1
2
5
10
Relative risk
(95% confidence interval)
Results from 29 studies examining the association between intact foreskin
and the risk of HIV infection in men
Study
Allen
Barongo
Bollinger
Bwayo
Bwayo
Cameron
Carael
Chao
Chiasson
Diallo
Greenblatt
Grosskurth
Hira
Hunter
Konde-Luc
Kreiss
Malamba
Mehendal
Moss
Nasio
Pepin
Quigley
Sassan
Sedlin
Seed
Simonsen
Tyndall
Urassa 1
Urassa 2
Urassa 3
Urassa 4
Urassa 5
Van de Perre
0.2
0.5
1
2
Relative risk
(95% confidence interval)
5
10
Adapted from Van Howe Int J STD AIDS 1999
Formaldehyde exposure and lung
cancer
SMR (95% CI)
150
100
50
0
Anatomists, Funeral Directors
Pathologists Embalmers
(3 Cohorts)
(7 Cohorts)
Industrial
Workers
(14 Cohorts)
Blair et al Scan J Work Environ Health 1990
Dietary fat and breast cancer
Relative Risk (95% CI)
1.8
1.6
1.4
1.2
1.0
0.8
0.6
12 Case-Control
Studies
6 Cohort Studies
Boyd et al Br J Cancer 1993
Intermittent sunlight exposure and
melanoma
Odds Ratio (95% CI)
3
2
1
0
7 Case-Control
Studies with
Blinding
9 Case-Control
Studies without
Blinding
Nelemans et al J Clin Epidemiol 1995
Test of homogeneity
• Examines the possibility of excess variability
between the results of the different studies
• Has low power if the number of studies is
small
• Can get a set of homogeneous but spurious
findings
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Beta-carotene and cardiovascular
mortality
Cohort
Country
Male health workers
USA
Social insurance, men
Finland
Social insurance, women
Finland
Male chemical workers
Switzerland
Hyperlipidaemic men
USA
Nursing home residents
USA
Cohorts combined
0.1
0.5
0.75
1
1.25
1.5
Relative risk (95% CI)
Jah et al Ann Intern Med 1995
Beta carotene and cardiovascular disease
Cohorts
Male health workers
USA
Social insurance, men
Finland
Social insurance, women
Finland
Male chemical workers
Switzerland
Hyperlipidaemic men
USA
Nursing home residents
USA
Trials
Cohorts combined
Male smokers
Finland
Skin cancer patients
USA
(Ex)-smokers, asbestos workers
USA
Male physicians
USA
Trials combined
0.1
0.5
0.75
1
1.25
Relative risk (95% CI)
Egger et al. BMJ 1998;316:140-4
1.5
1.75
“Well, so much for antioxidants.”
Smoking and suicide
No of cigarettes
MRFIT screenees
1-14
14-24
25+
Whitehall I
1-14
14-24
25+
North Karelia men
1-14
14-24
25+
Kuopio men
1-14
14-24
25+
Meta-analysis
1-14
14-24
25+
0.2
1
2
5
10
25
Relative rate (95% CI)
Davey Smith et al Lancet 1992
Smoking and homicide
• Non-smoker 1.00
• 1-2 packs/day 1.71 (1.29-2.28)
• 2+ packs/day 2.04 (1.32-3.15)
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Fundamental difference in
assumptions
• In meta-analysis of observational studies
confounding, residual confounding and bias:
– May introduce heterogeneity
– May lead to misleading (albeit very precise)
estimates
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
What is the appropriate
weighting factor?
Inverse of variance?
Case-control studies of Helicobacter
pylori infection and CHD
Study
Cases
+ve
Controls
+ve
Cases
-ve
Controls ve
Crude
OR
Adjusted
OR
Weight
Danesh
1999a
472
272
650
850
2.3
1.9
44%
Danesh
1999b
134
294
112
348
1.4
1.3
17%
Patel
1995
56
135
27
170
2.6
2.8
6%
Murray
1995
102
1117
33
863
2.4
1.5
9%
McDonagh
1997
315
625
134
353
1.3
0.9 m
1.0 f
25%
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Two case-control studies of Helicobacter pylori
infection and CHD
Danesh et al 1999 a
Danesh et al 1999 b
Numbers in study
2224
888
Response rate controls
<20%
60%
Response rate cases
~60%
56%
Adjustment for social
position
+
+++
Other adjustments
+
+++
Representative cases?
No
Fairly representative
Representative controls?
No
Fairly representative
OR sex/age adjusted
2.3
1.4
OR fully adjusted
1.9
1.3
Weight in meta-analysis
44%
17%
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Prospective and nested case-control studies of
Helicobacter pylori infection and CHD
Study
Cases
+ve
Controls
+ve
Cases
-ve
Controls
-ve
Crude
OR
Strachan
1998
204
1061
82
449
1.05
1.02
17%
Wald 1997
308
595
340
701
1.07
1.06
37%
Aromaa 1998
229
411
47
116
1.38
-
9%
Folsom 1998
111
257
106
241
0.98
0.97
13%
Ossewaarde
1998
39
84
15
24
0.74
-
2%
Whincup
2000
401
740
104
285
1.48
1.30
20%
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Adjusted Weight
OR
Erasmus Summer Programme
In RCT meta-analyses the appropriate
study weights should relate to precision
of effect estimates (e.g. inverse of
variance).
In observational meta-analyses this may
not generally be the case.
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Inverse of variance weighting
• Can lead to magnification of pooled effect
estimates when confounding and bias
involved (e.g. H pylori)
• Can lead to under-estimation of effect
estimates when measurement error is
important
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Huxley, Lancet 2002
Publication bias in MA / SR of
observational studies
Reporting bias in observational research
Data collected
(e.g cohort study)
Data analysed
Report written
Paper published
“I regret to inform you that the Journal of xxx will not be
able to use your manuscript … We think the study is welldesigned, with a fair follow-up and appropriate statistical
analysis, but the negative results found can only be published
as a Letter to the editor …”
Rejection of ‘negative’ prospective cohort study finding of
association of d-dimer with CHD. May 2006
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Genetic meta-analysis may be
an exception …
To the homogeneity and “spurious
precision” problems…
But may be particularly prone to
publication bias
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Funnel plot of meta–analysis of ACE I/D
and CHD
Standard error
0.0
0.1
0.2
0.3
0.4
0.33
0.5 0.66
1.0
1.5
Odds ratio
2.0
3.0
Conclusions
• The principles of systematic reviews are applicable to
any research design
• Reviews of observational studies should always be
systematic
• Much attention should be given to exploring possible
sources of heterogeneity
• HOWEVER: Meta-analysis of observational studies
will often produce misleading and spuriously precise
estimates
• Trial registers should solve much of the problems of
publication bias in RCT, but trying to solve publication
bias in observational studies impossible?
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme
Future work
• We need to define optimal search strategies to
identify epidemiological studies in the literature
• We need validated instruments to assess the study
quality at the design, conduct and analysis level
• We need to improve the quality of reporting of
epidemiological studies
• We need to facilitate individual patient data
analyses
• We need to better define the place of meta-analysis
in systematic reviews of epidemiological studies
11 August 2010
Topics in Meta-Analysis (Matthias Egger)
Erasmus Summer Programme