Clinical Epidemiology Boot Camp:
Systematic Reviews
Selina Liu MD MSc FRCPC
Research Fellow
Resident Research Career Development Program
September 19, 2012
Outline

Introduction – Evidence-Based Medicine (EBM)


Levels of evidence
To discuss the definition of a systematic review

vs. traditional/narrative reviews

The process of conducting a systematic review
Strengths & limitations of systematic reviews


To describe how to critically appraise a systematic review

Example of a systematic review
Evidence-Based Medicine

What is Evidence-Based Medicine?

“…the conscientious, explicit and judicious use of current best
evidence in making decisions about the care of individual
patients”

“ It’s about integrating individual clinical expertise and the best
external evidence”

philosophical origins – date back to mid-19th century Paris (or
possibly earlier)
Sackett DL et al. BMJ. 1996;312(7023):71-2
Evidence-Based Medicine

Five Steps of Evidence-Based Medicine

1. Asking Focused Questions


2. Finding the Evidence


Testing evidence for validity, clinical relevance, and applicability
4. Making a Decision


Systematic retrieval of the best evidence available
3. Critical Appraisal


Translation of uncertainty to an answerable question
Application of results in practice
5. Evaluating Performance

Auditing evidence-based decisions
Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net
Evidence-Based Medicine

Why Evidence-Based Medicine?

clinical decision making is complex!
Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91
Evidence-Based Medicine

How do we practice Evidence-Based Medicine?
Can be difficult:

“information
overload”  difficult for clinicians to “keep up” with all of
the latest evidence

often there are multiple studies examining the same or similar questions
 may be of variable quality, generalizability

estimated time required for reading (general medicine):
 19 articles per day, 365 days per year
Davidoff F et al. BMJ. 1995;310(6987):1085-6
Evidence-Based Medicine

Weighing the evidence - “Levels of Evidence”
OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”,
Oxford Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653
Evidence-Based Medicine
of RCTs
Systematic reviews of
cohort studies
Systematic Reviews

What is a Systematic Review?

the application of strategies that limit bias in the assembly,
critical appraisal, and synthesis of all relevant studies on a
specific topic

use rigorous, standardized methods for selecting & assessing articles
Oxford Centre for Evidence-Based Medicine www.cebm.net/?o=1116
OR

a report that summarizes all evidence that can be drawn from
research (or other sources), that is relevant to a specific clinical
question
Systematic Reviews

Systematic Reviews vs. Traditional Review Articles

traditional review articles




written by senior expert in the field, summarizes evidence and
recommendations
usually address broad areas/questions (i.e. “management of T2DM”)
often lack structure
may include personal experience/anecdotal evidence
Fletcher RH & Fletcher SW 2005. Clinical Epidemiology: The Essentials
Systematic Reviews

Systematic Review vs. Traditional/Narrative Review
Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380
Systematic Reviews
Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)
Systematic Reviews

Process of Conducting a Systematic Review
1. Define the question
2. Conduct literature search
3. Apply inclusion and exclusion criteria
4. Create data abstraction
5. Conduct analysis
Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)
Systematic Reviews - Process

1. Define the Question

single, focused question


i.e. What is the effect of cinnamon on glycemic control in diabetes?
specify inclusion and exclusion criteria

Population, Intervention or Exposure, Outcome, Methodology

For the systematic review to be useful:


strong studies of the question should be available, but their results should
not be so much in agreement that the question is already answered!
there should not be so few studies of the question that each individual study
could be fully critiqued directly
Systematic Reviews - Process

2. Conduct literature search

need to ensure that all of the appropriate studies are included


decide on information sources


NOT just a biased sample of studies
i.e. MEDLINE, recent reviews, textbooks, experts in the field, articles
cited by references already found by other approaches, databases of
articles, clinical trial registries etc.
identify titles and abstracts
Systematic Reviews - Process

3. Apply inclusion and exclusion criteria

Apply inclusion and exclusion criteria to titles and abstracts


obtain full articles for eligible titles and abstracts
Apply inclusion and exclusion criteria to full articles

Select final eligible articles

Assess agreement on study selection

of the initial titles and abstracts retrieved, usually only a small
proportion of articles are selected
Systematic Reviews - Process

4. Create data abstraction



Assess methodologic quality of each article
Assess agreement on validity assessment
Data abstraction




Participants
Interventions and Comparison Interventions
Study Design
Results
Systematic Reviews - Process

5. Conduct analysis

Summarize data

If appropriate: meta-analysis – statistical technique to combine
quantitative data


Describe results – often graphically




usually combine studies vs. combine patients
Forest Plot – shows point estimate and confidence interval (for RCTs,
observational studies)
Summary Receiver-Operator Curves (for studies of diagnostic tests)
Explore heterogeneity, conduct subgroup analysis (if appropriate)
Explore possibility of publication bias (and other biases)
Systematic Reviews - Process

How to decide if appropriate to perform a meta-analysis?

Two general approaches:

1. statistical test for homogeneity

BUT – even if fail to reject H0 (i.e. no evidence of a statistically significant
difference between studies), usually have high risk of false-negative (saying
studies are homogeneous when they really are not)


Limited power - meta-analyses are usually of few number of studies,
- affected also by number of patients/study, distribution of patients
among studies
2. informed judgement
Systematic Reviews - Process

Meta-analysis – mathematical models:

Fixed-Effect Model



Assumes that studies are of exactly the same question, so results differ
only by chance
Confidence intervals calculated may imply more precision (i.e. are
narrower) than in reality (since studies usually differ somewhat)
Random-Effects Model



Assumes that the studies address somewhat different questions, but
that they form a closely related family of studies of a similar question
Studies taken to be a random sample of all studies bearing on the
question
Produces WIDER confidence intervals (more “realistic”)
Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4 th Edition)
Systematic Reviews – Forest Plot
Impact of Treatment on Mortality
Study name
Kelly, 1964
Hedrin, 1980
Leigh, 1962
Novak, 1992
Saint, 1998
Pilbean, 1936
Day, 1960
Kelly, 1966
Singh, 2000
Stewart, 1994
Statistics for each study
Odds
ratio
Lower
limit
0.590
0.464
0.394
0.490
1.250
0.129
0.313
0.429
0.718
0.143
0.328
0.096
0.201
0.076
0.088
0.479
0.027
0.054
0.070
0.237
0.082
0.233
Odds ratio and 95% CI
Upper
limit
3.634
1.074
2.055
2.737
3.261
0.605
1.805
2.620
2.179
0.250
0.462
0.01
0.1
Favours Tx
Meta Analysis
1
10
Favours Pbo
100
Systematic Reviews - Bias

Several types of bias:

publication bias


language bias


i.e. if only English-language articles are selected
size bias


published studies may be systematically different than unpublished studies
(“positive” studies vs. “negative” studies?)
large studies that result in several publications may be more readily noticed
than smaller studies
bias related to funding?

industry-sponsored studies
How to detect publication bias?

Funnel plots – plot effect vs. study size/precision
symmetrical,
peaked distribution
(inverted funnel)
Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)
How to detect publication bias?

Funnel plots
asymmetrical
distribution
Guyatt G et al. 2008. Users’ Guide to the Medical Literature: A Manual for Evidence-Based Clinical Practice (2nd Edition)
Systematic Reviews - Strengths

provide an efficient way to become familiar with the best
available research evidence for a focused clinical question

can establish whether results are consistent, generalizable
across populations/settings, treatment variations, and whether
findings vary by certain subgroups

can extend the available literature (if the review team has
obtained unpublished information from the primary authors)

meta-analyses – may provide a more precise estimate of the
underlying “true effect” than any individual study
Garg AX, Hackam D, Tonelli M. 2008. Clin J Am Soc Nephrol. 3(1):253-60.
Systematic Reviews - Limitations

summarized results are limited by the quality of the primary
studies


garbage out”
results dependent on selection of included articles


“garbage in
quality threshold, publication bias, language bias etc.
meta-analyses - may be inappropriate to mathematically
combine primary study results if the primary studies differ in
design, quality, population, intervention etc.


subjectivity involved in deciding whether to pool or not
subjectivity in interpretation of summarized results (“overinterpretation)
Garg AX, Hackam D, Tonelli M. 2008. Clin J Am Soc Nephrol. 3(1):253-60.
Systematic Reviews – Critical Appraisal
Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA. 1994;272(17):1367-71
Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33
Critical Appraisal – Tools
Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157
Critical Appraisal – Tools
Oxford Centre for Evidence-Based Medicine http://www.cebm.net/index.aspx?o=1157
Critical Appraisal – Tools

AMSTAR – 2007

Assessment of Multiple SysTemAtic Reviews


Shea BJ, Grimshaw JM, Wells GA et al.
11 item tool

developed via exploratory factor analysis of a 37-item assessment
tool
Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10
Critical Appraisal - Tools
Shea BJ, Grimshaw JM, Wells GA et al. BMC Med Res Methodol. 2007;7:10
Reporting Systematic Reviews - Tools

The PRISMA Statement – 2009

Preferred Reporting Items for Systematic reviews and MetaAnalyses

Moher D, Liberati A, Tetzlaff J, Altman DG, The PRISMA Group






PLoS Medicine
Annals of Internal Medicine
BMJ
Journal of Clinical Epidemiology
Open Medicine
International Journal of Surgery
The PRISMA Statement - 2009

Aim – to help authors improve the reporting of
systematic reviews and meta-analyses

**NOT intended to be a quality assessment tool


27 item checklist
four-phase flow diagram

update and expansion of prior QUOROM statement

QUality Of Reporting Of Meta-analyses - 1996

focused on reporting of meta-analyses of randomized controlled trials
Table 1. Checklist of items to include when reporting a systematic review or meta-analysis.
Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses:
The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097
Figure 1. Flow of information through the different phases of a systematic review.
Moher D, Liberati A, Tetzlaff J, Altman DG, et al. (2009) Preferred Reporting Items for Systematic Reviews and Meta-Analyses:
The PRISMA Statement. PLoS Med 6(7): e1000097. doi:10.1371/journal.pmed.1000097
Example of a Systematic Review
Cochrane Database of Systematic Reviews 2012, Issue 9:CD007170
Cinnamon for Diabetes Mellitus

Objective – to evaluate the effects of cinnamon in patients with
diabetes mellitus

Selection criteria – all RCTs comparing the effects of orally
administered monopreparations of cinnamon (Cinnamomum spp.) to
placebo, active medication or no treatment in persons with either type
1 or type 2 diabetes mellitus

Primary outcomes – fasting blood glucose, post-prandial glucose,
adverse events

Secondary outcomes – HbA1c, serum insulin, HOMA-IR, HRQoL,
morbidity, costs
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Risk of Bias Summary

Review authors’ judgements about each “risk of bias” item for each
included study
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Risk of Bias Graph

Review authors’ judgements about each “risk of bias” item presented as
percentages across all included studies
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Results – Cinnamon vs. Placebo

Primary outcome – fasting blood glucose
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Results – Cinnamon vs. Placebo

Primary outcome – adverse events
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Results – Cinnamon vs. Placebo

Secondary outcome – HbA1c
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Results

Other primary outcomes:
 Post-prandial glucose – 1trial

Other secondary outcomes:
 Serum insulin – 2 trials
 HOMA-IR – 2 trials
 HRQoL, morbidity, costs – no trials
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Conclusions

Insufficient evidence to support the use of cinnamon for type 1
or type 2 diabetes

Further trials required:
 To address methodologic issues in current studies


i.e. allocation concealment, blinding
To include other important endpoints

HRQOL, diabetes complications, cost
Leach MJ & Kumar S. Cochrane Database of Syst Rev 2012, Issue 9:CD007170
Useful Resources

Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to
the Medical Literature: A Manual for Evidence-Based Clinical
Practice (2nd Edition). New York NY, McGraw-Hill


Oxford Centre for Evidence-Based Medicine (CEBM)


available online via Western Libraries
www.cebm.net
Cochrane Database of Systematic Reviews

www.thecochranelibrary.com
References

Sackett DL, Rosenberg WMC, Muir Gray JA, Haynes RB, Richardson WS. BMJ. 1996;312(7023):71-2

Mulrow CD, Cook DJ, Davidoff F. Ann Intern Med. 1997;126(5):389-91

Davidoff F, Haynes B, Sackett D, Smith R. BMJ. 1995;310(6987):1085-6

Oxford Centre for Evidence-Based Medicine (CEBM) www.cebm.net

OCEBM Levels of Evidence Working Group. “The Oxford 2011 Levels of Evidence”, Oxford
Centre for Evidence-Based Medicine. www.cebm.net/index.aspx?o=5653

Fletcher RH & Fletcher SW. 2005. Clinical Epidemiology: the Essentials (4th Edition). Baltimore MD,
Lippincott Williams & Wilkins

Guyatt G, Rennie D, Meade MO, Cook DJ. 2008. Users’ Guide to the Medical Literature: A Manual
for Evidence-Based Clinical Practice (2nd Edition). New York NY, McGraw-Hill

Cook DJ, Mulrow CD, Haynes RB. Ann Intern Med. 1997;126(5):376-380

Garg AX, Hackam D, Tonelli M. 2008. Clin J Am Soc Nephrol. 3(1):253-60.

Oxman AD, Cook DJ, Guyatt GH, Evidence-Based Medicine Working Group. JAMA.
1994;272(17):1367-71

Tonelli M, Hackam D, Garg AX. Methods Mol Biol. 2009;473:217-33

Cochrane Database of Systematic Reviews www.thecochranelibrary.com