How to Use Systematic
Reviews
Primary Care Conference
June 27, 2007
David Feldstein, MD
Objectives
Systematically analyze validity of a
meta-analysis
 Discuss results of a meta-analysis
 Apply evidence from a meta-analysis to
patient care decisions

Systematic Review vs.
Meta-Analysis

Systematic Review
– Critical evaluation of research to answer a
focused question where an attempt is
made to evaluate all available research
– Also known as an overview

Meta-analysis
– Quantitative strategy to combine the
results of multiple studies into a single
pooled estimate
Case





A 55 y.o. male with a history of an MI in 1995
presents to clinic for an annual physical.
He has been feeling well, but is concerned
about the risk of future MI’s and death.
He is on ASA, statin, ACE and B-blocker
His brother who also has CAD told him that
he should take vitamin E
Should I recommend vitamin E?
Question

In patients with CAD does vitamin E
supplementation decrease the risk of
death?
Search

PubMed
– Clinical Queries
» Find Systematic Reviews
» “Vitamin E and cardiovascular disease”
Articles


Eidelman RS. Hollar D. Hebert PR. Lamas
GA. Hennekens CH. Randomized trials of
vitamin E in the treatment and prevention of
cardiovascular disease. Archives of Internal
Medicine. 164(14):1552-6, 2004 Jul 26.
Shekelle PG. Morton SC. Jungvig LK. Udani
J. Spar M. Tu W. J Suttorp M. Coulter I.
Newberry SJ. Hardy M. Effect of
supplemental vitamin E for the prevention
and treatment of cardiovascular disease.
Journal of General Internal Medicine.
19(4):380-9, 2004 Apr.
Steps in Critical Appraisal
Is the study valid?
 What are the results?
 Can I apply the results to my patient?

Critical Appraisal Worksheet Validity
1.
2.
3.
4.
Did the overview explicitly address a
sensible clinical question?
Was the search for relevant studies detailed
and exhaustive?
Were the primary studies of high
methodologic quality?
Were assessments of studies reproducible?
Guyatt G, Rennie D. Users' guides to the medical literature : a manual for
evidence-based clinical practice. Chicago, IL: AMA Press; 2002.
Critical Appraisal
First read the abstract
 Each article has sections that are
labeled
 Look at the labeled areas to answer the
corresponding validity questions for
your assigned article
 Feel free to look at the other article if
time permits

Validity

Did the overview address a sensible
clinical question?
Validity

Basic Principles - Sensible question
– Is it reasonable to expect similar results
across trials?
– Consider:
» Population
» Intervention
» Outcomes
– Take into account inclusion and exclusion
criteria
Validity

Was the search for relevant studies
detailed and exhaustive?
Validity

Basic Principles – Search
– Sensitive search to find all published data
» Multiple databases
– Avoid language bias
» Non-English speaking authors more likely to
publish positive studies in English journals
– Avoid publication bias
» Larger studies or positive results more often
published
» Need to search for unpublished data
FP
Validity

Exhaustive search:
– MEDLINE, CINAHL
– Cochrane controlled trials registry
(CENTRAL)
– Foreign language literature (EMBASE and
LILACS)
– References cited in primary sources
– Conference proceedings
– Unpublished data from experts in field,
authors, drug manufacturers
Validity

Were the primary studies of high
methodologic quality?
Jadad Score
Jadad AR, Moore RA, Carroll D, et al. Assessing the quality of reports of randomized clinical trials: is blinding
necessary? Control Clin Trials. 1996
Validity

Basic Principles - Quality of individual
studies
– Should look at criteria similar to User’s
Guides for individual studies
– Often use predefined scoring systems, but
is not necessary
– Lower quality trials overestimate treatment
benefit
» Allocation Concealment
» Blinding
Jadad Scores (Shekelle)
Trial
Jadad Score
CHAOS
3
GISSI
3
Haeger
0
HOPE
3
SPACE
4
Validity

Were assessments of studies
reproducible?
Validity

Basic Principles – Assessment
reproducible
– 2 people independently
»Evaluate studies for inclusion
»Abstract data
»Assess validity
– To prevent bias or random error
– Check agreement between reviewers
Validity Conclusions
Useless
Perfect

0
1
X
2
3
 = Archives
X = JGIM
4
5
6
7
8
9
10
Results

Where the results similar from study to
study?
– Heterogeneity
Heterogeneity



Are differences between studies due to chance
alone
Check forest plot
Often statistically checked using x2 test
– not sensitive
– p<0.1 implies heterogeneity


2
I test newer method for evaluating
Authors should provide reasons for
heterogeneity a priori
Favors Treatment
1
Favors Control
Favors Treatment
1
Favors Control
Forest Plots
Forest Plots
Forest Plots
Forest Plots
Forest Plots
Results (JGIM)

What is the magnitude of the treatment
effect?
– Mortality Vit E v. placebo (secondary
prevention)
– Look at the top of Figure 2 pg 384
Results (JGIM)
Relative Risk for death with Vit E = 0.96
Results (JGIM)

How precise is the treatment effect?
»RR death = 0.96
»95% confidence interval = 0.84-1.10
Applying Results

Is our patient so different from those in the
study that its results can not apply?
– No reason to think that our patient would not have
similar results

Is the intervention feasible in our setting?
– Certainly

What are our patients values and
preferences?
– Prevention of MI and Death
Patient Conclusion
What would you do with this patient?
 Well done meta-analysis of reasonable
quality individual trials
 No evidence of decrease in death
 Fairly tight confidence intervals
 I would not recommend vitamin E

Objectives/ Wrap-up
Systematically analyze validity of a
meta-analysis
 Discuss results of a meta-analysis
 Apply evidence from a meta-analysis to
patient care decisions

Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? Int J Epidemiol. 2002;31(1):1-5.
Egger M, Ebrahim S, Smith GD. Where now for meta-analysis? Int J Epidemiol. 2002;31(1):1-5.