PHPC2017
Brief Overview of Systematic Review
Prof Joey Yang
Division of Epidemiology
JC School of Public Health and Primary Care
The Chinese University of Hong Kong
Outline
§ Intro: review & systematic review
§ Procedures of SR
§ Characteristics and advantages of SR
§ Bias in SR
§ Tutorial
EBM: emphasis on research evidence
qEpidemiological researches are conducted to provide
evidence about the distribution and determinants
(including risk factors and effective interventions) of
health status/ events
qEvidence-based medicine (EBM) emphasizes that
clinical practice should integrate best research
evidence with clinical expertise and patient values.
CLINICAL QUESTION:
Is nicotine replacement gum effective
for helping smokers stop smoking?
NICOTINE REPLACEMENT THERAPY (NRT)
o Literature search for relevant studies
o Jarvis, BMJ 1982; 285: 537-540
o Randomized, blind, placebo controlled,
clinical trial
o 58 on NRT, 58 on placebo
o 1 year follow-up, objective outcome
o 1 lost to follow-up, I.T.T. analysis
RESULTS OF THE NRT TRIAL
Stopped Smoking
% of
Yes
No
Total
Quitters
Active Gum
27
31
58
47%
Placebo Gum
12
46
58
21%
RD = 26%
95% CI: 9%, 43%
P<0.01
RESUTLS OF 28 RANDOMIZED CONTROLLED
TRIALS OF NRT
First Author
Archeeon
B.T.S.
Cambell
Clavel
Fagerstrome(1)
Fagerstrome(2)
Fee
Gillbert
Hall
Harackiewicz
Hjalmarson
Hughes(1)
Hughes(2)
Jamrozik
Rate Diff. (%)
20
1
2
9
19
16
4
2
21
-1
11
22
3
2
First author
Jarvik
Jarvis
Jensen
Killen(1)
Killen(2)
Malcolm
Ocken
Page
Pirie
Russell
Sutton(1)
Sutton(2)
Segnan
Tonensen
Rate Diff. (%)
11
26
0
20
4
15
5
-3
13
5
6
8
2
16
Need for summary of evidence
Does NRT work?
How much does it work?
Intro: what is a review?
l A critical, constructive analysis of the literature in
a specific field through summary, classification,
analysis, comparison.
l A scientific text relying on previously published
literature or data. New data from the author’s
experiments are not presented (with exceptions:
some reviews contain new data).
Intro: traditional narrative review
l Define a topic and audience
l Search the literature
l Take notes while reading
l Organize, evaluate, analyze, and compare literatures
l Find a logical structure
l Write a draft and revise it
Intro: traditional narrative review
l Selected studies are compared and summarized on
the basis of the author’s experience, existing theories
and models
l Results are based on a qualitative rather than a
quantitative level
Drawback: Every step is subjective and prone to bias
RESUTLS OF 28 RANDOMIZED CONTROLLED
TRIALS OF NRT
First Author
Archeeon
B.T.S.
Cambell
Clavel
Fagerstrome(1)
Fagerstrome(2)
Fee
Gillbert
Hall
Harackiewicz
Hjalmarson
Hughes(1)
Hughes(2)
Jamrozik
Rate Diff. (%)
20
1
2
9
19
16
4
2
21
-1
11
22
3
2
First author
Jarvik
Jarvis
Jensen
Killen(1)
Killen(2)
Malcolm
Ocken
Page
Pirie
Russell
Sutton(1)
Sutton(2)
Segnan
Tonensen
Rate Diff. (%)
11
26
0
20
4
15
5
-3
13
5
6
8
2
16
Definition of systematic review
A systematic review attempts to collate all empirical evidence
that fits pre-specified eligibility criteria in order to answer a
specific research question. It uses explicit, systematic methods
that are selected with a view to minimizing bias, thus providing
more reliable findings from which conclusions can be drawn and
decisions made.
Epidemiological study designs
u Case report/case series
u Correlational (ecological) study
u Cross-sectional study
u Case-control study
u Cohort study
u Randomized controlled trial
u Systematic review
Procedures of SR
Research question
Literature search
Study selection
Data extraction
Procedures of
systematic review
Quality
assessment
Data analysis
Result
interpretation
Conclusion
Report writing
Procedures of SR: ask a structured research question
Intervention/Exposure
Patient/
population
Outcome: present
Outcome: absent
Time
Outcome: present
Control/Non-Exposure
Setting (not
necessary)
Outcome: absent
Procedures of SR: ask a structured research question
Setting: the facilities/environment where the research or practice
is done.
§ The effectiveness and complications of a surgical operation
delivered in a tertiary hospital vs. a community clinic
§ The effectiveness of a health education program delivered in a
factory vs. a general community
§ The effectiveness of artemisinin in an area of high quinineresistance
§ The effectiveness of an HIV/AIDS health education program in a
resource-limited country
Procedures of SR: ask a structured research question
Example: Compared with chemotherapy alone, is chemotherapy
with cetuximab better in improving survival of chemotherapynaive advanced non-small cell lung cancer (NSCLC)?
•
P: chemotherapy-naïve patients with advanced NSCLC
•
I: chemotherapy with cetuximab
•
C: chemotherapy alone
•
O: survival (the time from start of treatment to death)
•
S: no requirement
Procedures of SR
Research question
Literature search
Study selection
Data extraction
Procedures of
systematic review
Quality
assessment
Data analysis
Result
interpretation
Conclusion
Report writing
Procedures of SR: comprehensive literature search
Consequence of biased literature search in systematic review
Procedures of SR: comprehensive literature search
Electronic databases
Data
sources
Clinical trials registry
Reference lists of relevant documents
Un-indexed, important journals in this field
Experts, pharmaceutical companies, etc.
Supplementary search
Conference proceedings
Procedures of SR: comprehensive literature search
1.
PubMed/MEDLINE: Medical Literature Analysis and Retrieval System Online
2.
EMBASE: Excerpta Medica dataBASE
3.
CENTRAL: The Central Register of Controlled Trials
4.
Databases of other languages: e.g. CBM, Wanfang and CNKI in China
5.
Databases for specific research areas: e.g. International Pharmaceutical Abstracts
6.
Databases for degree dissertation: e.g. Index to Theses in Great Britain and
Ireland
7.
Databases for gray literature: e.g. System for Information on Grey Literature
8.
……
The 3 databases must be searched in systematic reviews of RCTs
Procedures of SR: comprehensive literature search
Research question: Compared with chemotherapy alone, is
chemotherapy with cetuximab better in improving survival
of chemotherapy-naive advanced non-small cell lung cancer
(NSCLC)?
Search databases using the terms and their variations
about the key components of the research question
• NSCLC
• Chemotherapy
• Cetuximab
• Randomized controlled trial
Procedures of SR: study selection
v To identify eligible studies from the
retrieved records
v “Filter”: eligibility criteria, which
are developed mainly based on the
research question (PICOS)
Procedures of SR
Research question
Literature search
Study selection
Data extraction
Procedures of
systematic review
Quality
assessment
Data analysis
Result
interpretation
Conclusion
Report writing
Procedures of SR: data extraction
Questionnaire survey
in primary studies
Procedures of SR: data extraction
Data extraction in
systematic reviews
Data
Procedures of SR: data extraction
What data to extract?
1. “Which studies are included?”
2. “What are they like?”
3. “Are they credible? ”
4. “What are the results?”
Procedures of SR: data extraction
1. “Which studies are included?” – bibliographic information
§ First author’s surname
§ Publication year
§ Journal
§ Title of article
§ …….
Procedures of SR: data extraction
2. “What are they like?” – details of PICO(S)
§ Participants: mean age, male%, smoker%, severity of disease, etc.
§ Intervention & control: comparison (e.g., A vs B or A+B vs A?), dosage,
frequency, adherence rate, etc.
§ Outcome: name (e.g., incident stroke), ascertainment method (e.g., selfreported, medical records), timing (e.g., 6 mo, 12 mo), unit (e.g., mmol/L
or mg/dL), etc.
§ Setting (not necessary): tertiary hospital, primary care center, etc.
Procedures of SR: data extraction
3. “Are they credible?” – data relating to study quality
§ Evaluate the methodological quality/ risk of bias of studies
against a set of “quality criteria” (items that correspond to
various methodological aspects of a study)
The Cochrane Collaboration’s tool
for assessing risk of bias in randomized controlled trials
Higgins JPT. BMJ 2011;343:d5928
Procedures of SR: data extraction
4. “What are the results?” – effect estimate and 95%
CI, or the data for calculating them (e.g. 2x2 table data)
 Dichotomous outcome: e.g., presence/absence of disease
p Risk ratio, odds ratio, risk difference
‚ Continuous outcome: e.g., blood pressure, scores
p Mean difference
ƒ Time-to-event outcome: e.g., overall survival time
p Hazard ratio
Procedures of SR
Research question
Literature search
Study selection
Data extraction
Procedures of
systematic review
Quality
assessment
Data analysis
Result
interpretation
Conclusion
Report writing
Distribution of the effects if all studies are found
Number of subjects of studies
0.5
0.8 1 1.25
Risk Ratio
2
3
Procedures of SR: data analysis
— Meta-analysis: the use of statistical methods to summarize
the results of individual studies.
— By combining data from all relevant studies, meta-analysis
can provide more precise estimates of the effects of health
care than those derived from the individual studies
included within a review.
— In theory, it should be done only when all studies have
the same underlying effects
Procedures of SR: data analysis
Research question: Compared with chemotherapy alone, is
chemotherapy with cetuximab better in improving response rate
of chemotherapy-naive advanced non-small cell lung cancer
(NSCLC)?
— P: non-small cell lung cancer
— I: chemotherapy plus cetuximab
— C: chemotherapy alone
— O: objective response
— 4 eligible RCTs are included
Procedures of SR: data analysis
Forest plot
# of events &
participants of
cetuximab group
Study ID
Total # of events
& participants of
cetuximab group
# of events &
participants of
control group
Total # of events
& participants of
control group
Weight given
to each study
Total weight
(100%)
Point estimate and 95% CI in
each study, both as blocks
and lines and as text
The area of the block indicates the
weight assigned to that study in the
meta-analysis while the horizontal
line depicts the 95% CI.
Procedures of SR: data analysis
Forest plot
The vertical line (RR=1, line of null effect)
indicates null effect, i.e. no difference
between the two groups.
l In this example, a point estimate located on the right side of
the line of null effect favors cetuximab.
l A horizontal line that does not cross the vertical line
indicates statistical significance (P≤0.05).
l In this example, cetuximab is superior to control. The
difference is statistically significant.
Overall estimate and 95% CI, both
as blocks and lines and as text.
The ends of diamond block depict
the 95% CI of the overall estimate.
Procedures of SR: data analysis
Conventionally, in meta-analysis, heterogeneity refers to the
variability (difference) in the intervention effects being
evaluated in the different studies, which is known
as statistical heterogeneity.
Procedures of SR: data analysis
(sources of heterogeneity)
1. The play of chance
— Random error
— Not a problem: meta-analysis aims to reduce random error
2. Clinical heterogeneity
— Variation in the patients characteristics, treatment regimen.
3. Methodological heterogeneity
— Variation in study design, outcomes, length of follow-up, statistical
methods, etc.
Procedures of SR: data analysis
(test for heterogeneity)
— Assesses whether observed differences in results are
compatible with chance alone.
— When the observed intervention effects are more different
from each other than one would expect due to random
error (chance) alone, there exists significant/substantial
statistical heterogeneity, which could be due to clinical
heterogeneity, methodological heterogeneity, or both.
— I2 statistic and/or Cochrane’s Q test
Procedures of SR: data analysis
(test for heterogeneity)
— 0% ~100%, describes the percentage of the variability in effect estimates that is
due to clinical/methodological heterogeneity rather than sampling error
(chance).
— I2=0%: the heterogeneity is mainly due to the play of chance; no important
clinical/methodological heterogeneity
— I2>50%: significant methodological/clinical heterogeneity
Procedures of SR: data analysis
(test for heterogeneity)
— Cochran's Q test (χ2 test): measures the possibility that the variability in results
solely arises from the play of chance
— P≤0.10: the heterogeneity is NOT mainly due to chance; clinical/methodological
heterogeneity is very likely to exist. The smaller the P value, the larger the
possibility that clinical/methodological heterogeneity exists
RESUTLS OF 28 RANDOMIZED CONTROLLED
TRIALS OF NRT
First Author
Archeeon
B.T.S.
Cambell
Clavel
Fagerstrome(1)
Fagerstrome(2)
Fee
Gillbert
Hall
Harackiewicz
Hjalmarson
Hughes(1)
Hughes(2)
Jamrozik
Rate Diff. (%)
20
1
2
9
19
16
4
2
21
-1
11
22
3
2
First author
Jarvik
Jarvis
Jensen
Killen(1)
Killen(2)
Malcolm
Ocken
Page
Pirie
Russell
Sutton(1)
Sutton(2)
Segnan
Tonensen
Rate Diff. (%)
11
26
0
20
4
15
5
-3
13
5
6
8
2
16
Procedures of SR: data analysis
(test for heterogeneity)
X27-Observed2 = 57 vs X27-Max2 = 37
P=0.0006
What does this result mean?
How to address this issue?
DIFFERENT TRUE UNDERLYING
EFFECTS SHOULD NOT BE COMBINED
Number of subjects of studies
Combined Effect That
Does Not Make Sense
1
2
3
4
5
6
7
8
9
10
11
12
13
Value of Effect (e.g., RR)
14
15
16
17
18
RANDOMIZED TRIALS OF NRT: VOLUNTEER SMOKERS
First
Author
Rate
Difference
(%)
Community volunteers
Malcolm
15
Jarvik
11
Killen
20
Clavel
9
Hall
21
Archeeon
20
Hughes
22
Killen
4
Pirie
13
First
Author
Rate
Difference
(%)
Antismoking clinics
Jarvis
26
Fee
4
Fagerstrome
16
Hjalmarson
11
RANDOMIZED TRIALS OF NRT: INVITED SMOKERS
First
Author
Rate
Difference
(%)
General practice
Russell
5
Fagerstrome 19
Jamrozik
2
Page
-3
Cambell
2
Sutton
6
Sutton
8
Gillbert
2
First
Author
Rate
Difference
(%)
Hughes
Ocken
Segnan
Harackiewicz
3
5
2
-1
Hospitals
B.T.S.
Tonensen
Jensen
1
16
0
SUBGROUP ANALYSIS: THE EFFICACY OF NRT
ACCORDING TO THE TYPE OF PATIENTS
Type of patients
No. of
Trials
No. of
Subjects
Combined
R.D. (95% CI)
Community volunteers
Antismoking clinics
All volunteered smokers
9
4
13
2687
774
3460
11%
12%
11% (7%, 15%)
General practice
Hopitals
All invited smokers
12
3
15
5341
1832
7146
4%
2%
3% (2%, 5%)
28
10606
6% (4%, 8%)
All combined
The type of patients is a factor that causes the huge
heterogeneity between studies
SUBGROUP ANALYSIS: INTERPRETATION
• If the results of two subgroups are very different: the
subgroup factor is the cause of heterogeneity; report the
two subgroup results separately; don’t combine all studies
(the same as stratified analysis for EM)
• If there is significant heterogeneity between studies (i.e.
their results are very different) and the results of the two
subgroups are similar: the subgroup factor cannot explain/
is not the cause of the heterogeneity; try other methods of
meta-analysis, or simply don’t do any meta-analysis
Procedures of SR
Research question
Literature search
Study selection
Data extraction
Procedures of
systematic review
Quality
assessment
Data analysis
Result
interpretation
Conclusion
Report writing
Reporting guidelines:
PRISMA statement and its extensions
(www.prisma-statement.org)
Summary:
the key characteristics of a systematic review
1.
A clearly stated set of objectives with pre-defined eligibility
criteria for studies
2.
An explicit, reproducible methodology
3.
A systematic search that attempts to identify all studies that
would meet the eligibility criteria
4.
An assessment of the validity of the findings of the included
studies, for example through the assessment of risk of bias
5.
A systematic presentation, and synthesis, of the characteristics
and findings of the included studies
What is the minimal number of studies to be included in a systematic review?
Summary: systematic review vs. meta-analysis
All reviews
Systematic
reviews
Systematic review with
meta-analysis
Other reviews
Systematic review may or may not include a meta-analysis
(depending on objective; availability of data; between-study heterogeneity)
Summary: advantages of systematic reviews over
narrative reviews
1. Standardized methods
2. Less selection bias
3. More objective (more reproducible)
4. Qualitative results (significance test)
5. Quantitative results (confidence interval)
6. Exploration of heterogeneity
7. Easy to be updated
Selection bias in systematic reviews
Estimated
Effect
True Effect
Estimated
Effect
Poor literature search may lead to studies missing in any part of
the funnel plot (unpredictable)
Selection bias in systematic reviews
— Publication bias: a special type of selection bias seen in
systematic reviews. It refers to the selective publication of
studies based on the nature and direction of the results (i.e.
whether results are “positive” or not).
— Specifically, small studies are more likely to remain
unpublished if their results are non-significant or
unfavorable, whereas larger studies get published regardless.
— In this situation, a systematic review of the published studies
could identify a spurious beneficial (over-estimated)
intervention effect.
Funnel plot for detecting publication bias
A simple scatter plot of the intervention effect estimates
from individual studies against some measure of each
study’s size or precision (e.g., sample size, standard
error).
All studies are published (symmetric funnel plot) à no evidence of publication bias
Funnel plot for detecting publication bias
Overestimated Effect
True Effect
Selective non-publication of small studies favoring the control group à the
overall RR becomes further away from 1 (asymmetric funnel plot) à
overestimating the effect (publication bias)
TUTORIAL:
Systematic Review
QUESTIONS FOR REVISION
1. Why do primary studies differ in their
results?
2. What is publication bias? What problem it
may cause if a systematic review fails to
include some small negative studies?
3. What is the objective of stratified
(subgroup) analysis in a systematic review?
4. Describe the advantages of systematic
reviews over traditional narrative reviews.
1. Individual studies may differ from each other
in their results because of
a.
Random error
b.
Biases or design features
c.
Clinical features
d.
All of the above
2. Clinical heterogeneity means that
a.
The effect differs according to study design
b.
The effect differs by clinical features, such as
the type of patients
c.
The effect differs according to sample size
d.
The effect differs due to biases
3. Regarding publication bias, which of
the following statements is correct?
a.
Large positive studies are most likely to be
unpublished
b.
Good search of MEDLINE will reduce publication
bias
c.
The overall effect is normally under-estimated
d.
The overall effect is normally over-estimated
4. Stratified analysis in a meta-analysis is
normally used to examine the presence of
a.
Interaction
b. Publication bias
c.
Confounding
d. Random error
5. Regarding systematic reviews, which of
the following statements is INCORRECT?
a.
It may not have a single study in it
b.
It may not have a meta-analysis in it
c.
The increased precision is the only gain
d.
It must exhaust the search for primary studies