Medical evidence increasing at epidemic rates:
we all need EBP skills to keep up-to-date
Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)
Medical evidence increasing at epidemic rates:
we all need EBP skills to keep up-to-date
approx 75 new
trials published
every day
Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)
Medical evidence increasing at epidemic rates:
we all need EBP skills to keep up-to-date
MEDLINE 2010
2,000 articles / day
approx 75 new
trials published
every day
Bastian, Glasziou, Chalmers (2010) 75 Trials and 11Systematic Reviews a Day: How Will We Ever Keep Up? PLoS Med 7(9)
About 10% of
published evidence
is worth reading
About 1/3 of
worthwhile
evidence is
eventually refuted
or attenuated
About 1/2 of
relevant evidence is
not implemented
Rapid critical appraisal
using GATE
Rod Jackson
University of Auckland, NZ
August 2011
6
Graphic Appraisal Tool for Epidemiological studies
Graphic Approach To Evidence Based Practice
Graphic Approach To Epidemiology
the GATE frame
the shape of every epidemiological study 8
8
British doctors
smoking status measured
smokers
Lung cancer
non-smokers
yes
no
5 years
Longitudinal (cohort) study
9
British doctors
smoking status measured
smokers
Lung function
non-smokers
normal
abnormal
Cross-sectional study
10
British doctors
Randomised to aspirin or placebo
aspirin
placebo
yes
Myocardial infarction
no
5 years
RCT
11
Middle-aged US women
Test applied
Mammogram positive
Mammogram negative
yes
Breast cancer
no
Clinical use of a diagnostic test
12
Middle-aged US women
Breast cancer
no Breast cancer
positive
mammogram
negative
Diagnostic test accuracy study
13
GATE: Graphic Appraisal Tool for
Epidemiological studies
1 picture, 2 formulas & 3 acronyms
14
14
One picture: the GATE frame
every epidemiological study hangs on the GATE frame
15
The 1st acronym = PECOT : the 5 parts
of every epidemiological study
P
Participants
Exposure Group
E
C
O
Time
Comparison Group
Outcomes
T
16
Lewis RT et al. Should antibiotic prophylaxis be
used routinely in clean surgical procedures: A
tentative yes. Surgery 1995;118:742-7.
17
Background. The incidence of surgical site infection (SSI)
after clean surgical procedure is regarded as too low for
routine antibiotic prophylaxis. But risk of SSI can be as high as
20%. We assessed the value of prophylactic cefotaxime in
patients stratified for risk of SSI in a double-blind RCT.
Methods. Patients having clean elective operations were
stratified for risk & randomized to receive IV cefotaxime 2 gm
or placebo before operation & followed for 4-6 weeks for SSI.
Results. The 378 of 775 patients who received cefotaxime
had 70% fewer SSIs than those who did not --Mantel-Haenszel
risk ratio (MH-RR) 0.31; 95 % CI 0.11 to 0.83. Benefit was
clear in the 616 low risk patients--0.97% versus 3.9% SSI
(MH-RR 0.25, CI 0.07 to 0.87, p = 0.018), but only a trend was
seen in 136 high risk patients--2.8% versus 6.1% SSI (MH-RR
0.48, CI 0.09 to 2.5).
Conclusions. The results indicate clear benefit for routine
antibiotic prophylaxis in clean surgical procedures. High risk
18
patients need more study.
19
1st critical appraisal task: describe study’s design by
hanging on GATE frame using PECOT acronym
P
E
C
O
T
20
Participants
Study Setting
Eligible Participants
P
Participants
21
Lewis Trial
Participants
Study Setting: patients admitted to QE Hospital,
Montreal, Canada (1992-5?)
Eligible Participants: undergoing clean surgery or
simple cholecystectomy
P
Participants: 633 (?consecutive eligible patients)
22
Exposure & Comparison Groups
Exposure or
Intervention Group
(EG)
EG
CG
Comparison or
Control Group
(CG)
23
Exposure & Comparison Groups:
low risk group
633
316
Exposure or
Intervention Group
(EG):
2g cefotaxime IV preop
317
Comparison or
Control Group
(CG):
Identical placebo
IV
24
Exposure & Comparison Groups:
low risk group
633
316
317
Exposure or
Comparison or
Intervention Group
Control Group
(EG):
(CG):
308* 308*
2mg cefotaxime IV
Identical placebo
pre-op
IV
* With complete follow-up
25
Outcomes (O)
yes
a
b
O
‘Dis-ease’
no
c
Outcomes (O)
d
26
633
Outcomes (O)
316
Surgical site
infection (SSI)
yes
317
a= 3 b=
12
O
no
c
d
Primary Outcome
(O)
27
Time (T)
incidence
T
prevalence
28
Time (T)
316
317
Outcome: SSI
incidence
T= time from
initiation of treatment
to end of follow-up
29
Study design: GATE frame & PECOT
Participants
P
Exposure Group
E
C
O
Time
T
Comparison Group
Outcomes
30
Lewis
Setting: QE Hospital, Montreal
Eligible: clean surgery or cholecystectomy
Participants
633
316
Exposure Group:
IV cefotaxime
Time:
Up to 6 wks post-op
317
308 308
3
12
Comparison Group:
IV placebo
Outcomes:
SSI
31
Questions?
32
The 1st formula: study analyses
Occurrence (risk) of disease
= Numerator ÷ Denominator
D
N
33
All epidemiological studies involve measuring
the OCCURRENCE of ‘outcomes’
D
Denominator (Participants)
N
Numerator (Outcomes)
Occ = N÷D
34
All epidemiological studies involve measuring
the OCCURRENCE of ‘outcomes’
D
Denominator (Participants)
T
During what period of time (T) was N
measured? (incidence)
N
Numerator (Outcomes)
Occ = N÷D (T?)
35
All epidemiological studies involve measuring
the OCCURRENCE of ‘outcomes’
Denominator (Participants)
D
T
At what point in time (T) was N measured?
(prevalence)
N
Numerator (Outcomes)
Occ = N÷D (T?)
36
The 1st formula:
Occurrence (risk) = Numerator ÷ Denominator
P
Exposed
Group
Comparison
Group
DE DC
T
NE
NC
O
T
37
2nd appraisal task: describe analyses by hanging
numbers on the GATE frame and calculating
occurrences in exposure & comparison groups
P
Denominator 1:
Exposure Group
(EG)
Numerator 1:
a
EG CG
a
c
O
b
Denominator 2:
Comparison Group
(CG)
Numerator 2: b
d
38
Occurrence = N ÷ D
P
Denominator 1:
Exposure Group
EG
Numerator 1:
a
EG CG
a
c
Exposure Group Occurrence:
EGO = a ÷ EG
O
b
d
Denominator 2:
Comparison Group
CG
Numerator 2:
b
Comparison Group Occurrence:
CGO = b ÷ CG
39
Occurrence = N ÷ D
P
Denominator 1:
Exposure Group
EG
Numerator 1:
a
EG CG
a
c
Exposure Group Occurrence:
EGO = a ÷ EG
O
b
d
Denominator 2:
Comparison Group
CG
Numerator 2:
b
Comparison Group Occurrence:
CGO = b ÷ CG
40
Occurrence = N ÷ D
P
Denominator 1:
Exposure Group
EG
Numerator 1:
a
EG CG
a
c
Exposure Group Occurrence:
EGO = a ÷ EG
O
b
d
Denominator 2:
Comparison Group
CG
Numerator 2:
b
Comparison Group Occurrence:
CGO = b ÷ CG
41
Calculate EGO & CGO for SSI in low risk group
P
Denominator 1:
Exposure Group
EG = 316
Numerator 1:
a=3
EG CG
a
c
O
b
d
Denominator 2:
Comparison Group
CG = 317
Numerator 2:
b = 12
EGO = 3/316= 9.5/1000 at 6
CGO = 12/317 = 37.9/1000
weeks
at 6 weeks
42
ITT (intention to treat) analysis
Calculate EGO & CGO for SSI in low risk group
P
Denominator 1:
Exposure Group
EG = 308
Numerator 1:
a=3
EG CG
a
c
O
b
d
Denominator 2:
Comparison Group
CG = 308
Numerator 2:
b = 12
EGO = 3/308= 9.7/1000 at 6
CGO = 12/308 =39/1000 at
weeks
6 weeks
43
OT (on treatment) or per-protocol analysis
Describing differences between
occurrences
Relative difference or Relative Risk = EGO ÷ CGO
Absolute Difference or Risk Difference = EGO - CGO
Number Needed To Treat (NNT) = 1 ÷ RD
44
Describing differences between
occurrences (SSI in low risk patients)
Relative difference or Relative Risk = EGO ÷ CGO
= 9.5/1000 ÷ 37.9/1000 = 0.25
Absolute Difference or Risk Difference = EGO - CGO
= 9.5/1000 - 37.9/1000 = -28.4/1000
Number Needed To Treat (NNT) = 1 ÷ RD
= 1 ÷ (- 28.4 /1000) = - 1000/28.4 = 35
‘if 35 patients were given IV cefotaxime pre-op, there would be 1 fewer SSI up
to 6 weeks post-op’
45
Study analyses
it’s all about EGO & CGO
46
Questions?
47
The 2nd acronym = RAMBO* : assessing bias
‘strength of study’
P
P
Recruitment
Allocation
E
E
C
C
Maintenance
O
Blind or
O
T
T
Objective outcomes
measurement
48
48
* Paul Glasziou
The 2nd acronym = RAMBO* : assessing non
random error (i.e. bias)
P
P
Recruitment
Allocation
E
E
C
C
Maintenance
O
Blind or
O
T
T
Objective outcomes
measurement
49
* Paul Glasziou
49
3rd appraisal task: assess the degree of bias by
applying the RAMBO acronym
P
P
Recruitment
Allocation
E
E
C
C
Maintenance
O
Blind or
O
T
T
Objective outcomes
measurement
50
Study setting
RAMBO
Eligible people
P
P
E
C
O
were Recruitment processes
appropriate to study goals?
• Study setting & eligibility criteria well
described?
e.g. Recruit random/representative sample
OR consecutive eligibles OR volunteers
from advertisements
• Participants representative of eligibles?
• Prognostic/risk profile appropriate to
study question?
T
51
RAMBO: A is for Allocation
Was Allocation to
EG & CG
successful?
RCT: Allocate randomly by
Cohort: Allocate by
investigators (e.g drugs) measurement (e.g. smoking)
EG CG
were EG & CG
similar at
baseline?
O
T
EG CG
O
T
52
P
EG CG
O
RAMBO
were Participants Maintained as
allocated?
did most participants remain in
allocated groups (EG & CG)
Participants &/or investigators blind to
exposure (and comparison exposure)?
Compliance high & similar in EG & CG?
Contamination low & similar in EG & CG?
Co-interventions low & similar in EG & CG?
T
Completeness of follow-up high & similar in EG
& CG?
53
P
RAMBO
Were outcomes measured
Blind or Objectively?
EG CG
O
T
If outcome measurements not
Objective (eg. automated or definitive)
were investigators Blind to exposure
(and comparison exposure)
54
The 4 (GATE) study biases
P
Recruitment bias
Allocation bias
E
T
C
O
Maintenance bias
Outcomes
Measurement bias
55
Questions?
56
The 2nd formula: assessing random error
Random error = 95% Confidence Interval(1.96
x Standard Error)
57
4th appraisal task: assess degree of random
error in study findings using the 2nd formula
Random error = 95% Confidence Interval
For the Outcome SSI (low risk group) :
EGO = 9.5/1000; (95% CI = 3.2 to 27.5)
CGO = 37.9/1000; (95% CI = 21.8 to 65)
EGO÷CGO = 0.25 (0.07 to 0.88)
EGO-CGO = -28.4 (-52 to -4.8)
NNT = -35 (-19 to -211)
58
Excel CATs & paper Gate-lites
There is a GATE for every study design
www.epiq.co.nz
59
59
Final appraisal task: search for & appraise
SRs / meta-analyses using 3rd acronym
(FAITH)
• Find appropriate studies?
• Appraise selected studies?
• Include only valid studies?
• Total-up (synthesise) appropriately?
• Heterogeneity adequately addressed?
60
Systematic Reviews
There are 4 Cochrane SRs on this topic
and the findings are not consistent
61
Using GATE
as a framework for evidence
based practice
The first 4 steps of EBP
1. Ask a focused question.
2. Access (systematically search for)
epidemiological evidence to help answer question.
3. Appraise evidence found for its validity, effect
size, precision (ideally all the relevant evidence)
4. Apply the evidence:
a. amalgamate the valid evidence with other
relevant information (patient/community values,
clinical/health issues, & policy context) and make an
evidence-based decision; and
b. act (implement) the decision in practice
EBP Step 1: Ask- turn your question
into a 5-part PECOT question
Participants (the patient problem)
Exposure (e.g. a therapy)
Comparison (there is always an alternative! another therapy or no treatment…
Outcome (e.g. a disease you want to prevent or
manage)
Time frame (over which you expect a result)
EBP Step 2: Access the evidence – use
PECOT to choose search terms
Participants (the patient problem)
Exposure (e.g. a therapy)
Comparison (there is always an alternative! another therapy or no treatment…
Outcome (e.g. a disease you want to prevent or
manage)
Time frame (over which you expect a result)
65
EBP Step 3: Appraise the
evidence
‘using the best evidence from
epidemiology to help inform decisions’
more critically (using GATE)
more systematically (using FAITH)
EBP Step 4: APPLY the evidence by: a.
AMALGAMATING the relevant information & making
an evidence-based decision:’ the X-factor
©
X-factor: making evidence-based decisions
Evidence
Clinical / health
considerations
Patient / community
preferences
Policy issues
Xpertise: ‘putting it all together’ the art of
practice