Critical appraisal Systematic Review
กิตติพนั ธุ์ ฤกษ์ เกษม
ภาควิชาศัลยศาสตร์
มหาวิทยาลัยเชียงใหม่
Systematic review vs Meta-analysis
• Systematic review: a systematic approach to
minimising bias and error
• Meta-analysis: a statistical analysis, which
aim to produce a single estimate of a
treatment effect
• Systematic review may or may not include
Meta-analysis
Why do we need systematic review?
• The early 1980s uncomplicated MI
“Should pt receive a beta-blocker for
secondary prevention before discharge?”
Library: 4 randomised controlled trials (RCT)
Beta-blocker vs placebo
RCT 1.Mortality and hospital readmission is not
different
RCT 2. Not conclusive
RCT 3. Beta-blocker not shown benefit
RCT 4. Long term beta-blocker reduces the
mortality and rate of re-infarction
A review in BMJ 1981
• There is no clear evidence that betablocker improves long tem survival after
MI despite almost 20 yrs of clinical trials
• Good enough!!
Another review in European
Heart Journal 1981
“it seems perfectly reasonable to treat
patients who have survived an infarction
with beta-blocker”
Limitation of a single study
• Too small sample size
false negative
Problem of Conventional review
• Prone to bias and error
– Select only evidence support the author’s view
– Not specify methodological quality of studies
– Finally choose most vote ignore sample size,
and design
Meta-analysis = combining all available data
• Attractive alternative to such large,
expensive and problematic study
• Weight average of the result
large > small trial
Meta-analysis Beta-blocker trials - MI
Beta-blocker better
Placebo better
Cumulative meta-analysis of
beta-blocker trials
Cumulative meta-analysis
Significant effect from 1980 onwards (OR not across 1
Meta-analysis Beta-blocker trials - MI
Maybe
Unnecessary
trials
Beta-blocker better
Placebo better
Benefit
• Estimate the overall effect
• Examine different result between studies
(heterogeneity)
• Identified insufficient data
Cochrane collaboration
• International organisation of
health care profession
• Promoting accessibility of
systematic review
• Foster development of
systematic review
• 50 collaborative review
groups
www.cochrane.org/cochrane/ccweb.htm
Potentials of systematic review
• Good
• Bad
Systematic review
Basic structure and types
กิตติพนั ธุ์ ฤกษ์ เกษม
ภาควิชาศัลยศาสตร์
มหาวิทยาลัยเชียงใหม่
1. Basic structure
Like primary research
• Why- Introduction, background
• How-method
• What we found-result
• What it mean-discussion
Basic structure
• Abstract
• Introduction
– Background
– Objectives
• Method “treat a paper like a patient in 1
reseach”
–
–
–
–
–
–
–
Type of studies
Inclusion criteria type of participants
Exclusion criteria
Type of intervention
Type of outcome measures
Search strategy for identification of study
Method of analysis
Basic structure
• Result (special diagram)
• Conclusion
• Reference
Forest plot
Odd s ra tio
(95% CI)
Stroke rate ‘LA vs GA in
carotid sx trial’ 1966-2001
% We igh t
Foresell et al. 1989
0.32 (0.01,8.06)
23.2
Sbarigia et al. 1999
0.13 (0.01,2.53)
55.1
Kasprzak et al. 1999
0.37 (0.01,9.17)
21.8
Pluskwa et al. 1989
Not estimable
0.0
Prough et al. 1989
Not estimable
0.0
Binder et al. 1999
Not estimable
0.0
Overall (95% CI )
0.22 (0.04,1.33)
LA better
GA better
.05
.5
1
2
Odds ratio
Black square = OR, horizontal line = 95% confidence interval
Area of black square = weight, diamond = combined OR with 95% CI
2. types
Types
• Systematic review of primary research
– Observational studies
– Diagnostic screening
– RCT
3. Method
“treat a paper like a patient in
1 reseach”
The process (1)
•
•
•
•
•
Research question
Writing protocol
Searching
Article retrieval
Literature review
The process (2)
•
•
•
•
Inclusion/ exclusion criteria
Validity and quality of articles
Data extraction/ synthesis
Interpretation
The question
• Is local anesthesia is better than
general anesthesia during carotid
endarterectomy?
Writing the protocol
•
•
•
•
•
•
•
•
•
•
•
Background
Objectives
Type of studies
Inclusion criteria
Type of participants
Exclusion criteria
Type of intervention
Type of outcome measures
Search strategy for identification of study
Method of analysis
Reference
Searching
•
•
•
•
Medline
Other database
Hand searching the literature
Writing to people
Getting the article
•
•
•
•
Which ones to get?
It takes time
Libraries
Inter-library Loans
Literature review
• You don’t have to read the whole paper yet!
• Translation
Validity and quality of articles
• Do read the paper and see what the
author thought was wrong
• Unequal intervention/control size
• Hidden loss to follow up
Data extraction
• Read method carefully
• Design a form
Synthesis/ Interpretation
• Estimates and confidence intervals
pool effect make by statistic method e.g. Peto
method (fix method) give more weight effect
for large study than small study (P value)
• Difference between studies (Heterogeneity)
Chi-squared test (P value)
Small RCTs show LA is marginal
lower mortality than GA
Local
Study or Subgroup
General
Peto Odds Ratio
Events Total Events Total Weight
Peto, Fixed, 95% CI
Binder 1999
0
27
0
19
Forssell 1989
0
56
1
55
14.5%
0.13 [0.00, 6.70]
Kasprzak 2006
0
91
1
95
14.5%
0.14 [0.00, 7.12]
McCarthy 2001a
1
34
1
33
28.5%
0.97 [0.06, 15.85]
Pluskwa 1989
0
10
0
10
Not estimable
Prough 1989
0
13
0
10
Not estimable
Sbarigia 1999
0
55
3
52
Total (95% CI)
Total events
286
1
Peto, Fixed, 95% CI
Not estimable
42.6%
0.12 [0.01, 1.21]
274 100.0%
0.23 [0.05, 1.01]
6
Heterogeneity: Chi² = 1.44, df = 3 (P = 0.70); I² = 0%
Test for overall effect: Z = 1.94 (P = 0.05)
Peto Odds Ratio
0.1 0.2
0.5
1
2
5 10
Local better General better
Critical Appraisal
1. Are the result valid?
2. What are the results?
1. Are the result valid?
• Did this review address a sensible
clinical question
• Was the search for relavant studies
detailed and exhaustive?
• Were selection and assessment of
studies reproducible?
• Were the primary studies of high
methodological quality?
Publication bias
“A (significant) beneficial treatment effect
are published, but an equal result remain
unpublished”
• In general medical journal and public
heath journal reported statistically
significant 85.4%
• In psychological journal 95.6%
Time lag bias
• “Positive result will dominate the literature
for several year until the negative will
report later”
• HIV trial in USA, median time to publish of
positive result 4.2 years, but negative
result 6.4 years
Duplicate publication bias
• “ one study presents and reports several
times” “ include this lead to
overestimation of treatment effect”
• Ondersetron to prevent postoperative
nausea vomitting 16 studies
3 duplicated papers
• Sometimes difficult to say, since not
share single common authors !!!!!!
Language bias
• “Authors tend to report positive result in
international papers, English language
journal, but if negative result are
published in local journal”
Outcome reporting bias
• In trials many outcome is recorded but
only favorable finding will be reported
• Clinical trials by drug companies,
unpublished trials gave information on
adverse effect > published trials
Selection Bias
• Tend to happen in non Randomised
controlled trials (non RCT)
• For example select low risk group to
new treatment group
2. What are the result?
• Were the results similar from study to
study? If yes, the credit of single
estimates is OK.
– Point estimates similar?
– Overlapping confidence interval
– Test for heterogeneity? (Chi square test)
– Percentage of variability (I2 ) good < 20%,
concern 20-50%,serious concern > 50%
Local
Study or Subgroup
General
Events Total Events
Peto Odds Ratio
Total Weight
Agrifoglio 1987
0
29
2
28
1.5%
Allen 1994
2
274
4
310
4.4%
0.58 [0.12, 2.89]
Andersen 1980
6
179
1
152
5.1%
3.66 [0.82, 16.41]
Bartoloni 1991
0
69
0
75
Becquemin 1991
1
132
6
220
4.8%
0.37 [0.08, 1.71]
Bowyer 2000
2
272
2
228
3.0%
0.84 [0.12, 6.02]
Brown 1999
0
157
0
132
Calligaro 2001
3
185
4
216
5.1%
0.87 [0.20, 3.91]
Corson 1987
1
145
3
223
2.8%
0.54 [0.07, 4.08]
ECST 1998
0
58
17
1681
1.6%
0.35 [0.02, 5.03]
Gabelman 1983
0
48
1
40
0.7%
0.11 [0.00, 5.68]
Ghali 1997
0
33
0
57
Not estimable
Godin 1989
0
50
0
50
Not estimable
Gurer 2003
1
200
2
165
2.2%
0.42 [0.04, 4.09]
Harbaugh 2000
2
632
0
171
1.0%
3.57 [0.12, 105.59]
2
116
1
118
2.2%
1.99 [0.21, 19.35]
47 3382
12
593
22.2%
0.65 [0.32, 1.33]
Hartsell 1999
Imparato 1998
Not estimable
Not estimable
0
50
0
30
Kalko 2007
2
306
3
127
3.1%
0.22 [0.03, 1.55]
Kraiss 1995
0
18
1
178
0.3%
0.33 [0.00, 294.58]
Kucey 1998
0
185
23
1096
8.4%
0.30 [0.09, 0.98]
Love 2000
0
200
3
243
2.2%
0.16 [0.02, 1.56]
Marrocco 2004
0
50
0
63
McCarthy 2001b
2
100
0
140
1.5%
11.13 [0.67, 186.25]
McCleary 1996
1
32
0
33
0.8%
7.62 [0.15, 384.38]
Mertens 2000
0
22
0
72
Mofidi 2006
1
192
3
179
3.0%
0.34 [0.05, 2.43]
Muskett 1986
0
30
0
45
NASCET 1998
0
104
14
1310
2.8%
0.34 [0.04, 2.53]
Ombrellaro 1996
0
140
0
126
Palmer 1989
2
184
0
37
0.8%
3.34 [0.08, 137.98]
Quigley 2000
0
126
1
78
0.7%
0.07 [0.00, 4.13]
Reina 1998
3
116
3
188
4.2%
1.67 [0.32, 8.77]
Rignano 1999
0
76
1
169
0.6%
0.23 [0.00, 16.24]
Santamaria 2004
0
203
2
56
1.0%
0.01 [0.00, 0.28]
Schwartz 1988
0
149
2
162
1.5%
0.15 [0.01, 2.35]
Shah 1994
5
654
3
219
4.5%
0.51 [0.10, 2.56]
Sternbach 2002
2
226
1
324
2.2%
2.89 [0.29, 28.92]
Stone 2000
0
67
0
61
Stoughton 1998
0
150
0
58
Syrek 1999
1
83
0
26
0.5%
3.72 [0.04, 369.48]
Taylor 1999
2
140
16
1152
5.2%
1.03 [0.23, 4.60]
10631 100.0%
0.63 [0.45, 0.89]
Total events
9564
88
Look
overlapping
confidence
interval
Not estimable
Not estimable
Not estimable
Not estimable
Not estimable
Not estimable
Not estimable
Rerkasem Cochrane Database
Syst Rev 2008; (4):CD000126.
131
Heterogeneity: Chi² = 33.76, df = 30 (P = 0.29); I² = 11%
Test for overall effect: Z = 2.65 (P = 0.008)
Peto, Fixed, 95% CI
0.13 [0.01, 2.06]
Jopling 1983
Total (95% CI)
Peto Odds Ratio
Peto, Fixed, 95% CI
0.1 0.2
0.5
1
2
5
Local better General better
10
Small RCTs show LA is marginal
lower mortality than GA
Local
Study or Subgroup
General
Peto Odds Ratio
Events Total Events Total Weight
Peto, Fixed, 95% CI
Binder 1999
0
27
0
19
Forssell 1989
0
56
1
55
14.5%
0.13 [0.00, 6.70]
Kasprzak 2006
0
91
1
95
14.5%
0.14 [0.00, 7.12]
McCarthy 2001a
1
34
1
33
28.5%
0.97 [0.06, 15.85]
Pluskwa 1989
0
10
0
10
Not estimable
Prough 1989
0
13
0
10
Not estimable
Sbarigia 1999
0
55
3
52
Total (95% CI)
Total events
286
1
Peto, Fixed, 95% CI
Not estimable
42.6%
0.12 [0.01, 1.21]
274 100.0%
0.23 [0.05, 1.01]
6
Heterogeneity: Chi² = 1.44, df = 3 (P = 0.70); I² = 0%
Test for overall effect: Z = 1.94 (P = 0.05)
Peto Odds Ratio
0.1 0.2
0.5
1
2
5 10
Local better General better
Rerkasem Cochrane Database Syst Rev 2008 (4):CD000126.
2. What are the result?
• Were the results similar from study to
study?
• What are the overall results of the
reviews?
• How precise were the results?