Levels of Evidence
• Why?
• What?
• How?
1
Time-poor clinician suffering from
Information Overload
2
Evidence-Based Medicine
• EBM is ...”the conscientious, explicit and
judicious use of current best evidence in
making decisions about the care of an
individual patient. It means integrating
individual clinical expertise with the best
available external clinical evidence from
systematic research”
(Sackett, D. BMJ 1996;312:71-72).
3
4
Steps in EBM
Defining
the
question
or problem
Auditing
the
outcome
Searching
for the
evidence
Applying
the results
Critically
appraising
the
literature
5
The Evidence Pyramid is a guideline to
the hierarchy of study design
6
Type of question = type of study
design
Type of question
Suggested study
Therapy
RCT > prospective cohort
Diagnosis
Prospective, blind comparison to a
gold standard
Etiology/Harm
RCT > cohort > case control > case
series
Prognosis
Cohort study > case control > Case
series
Prevention
RCT > cohort study >case control >
case series
Cost
Economic analysis
7
Level Intervention
Diagnostic accuracy
Prognosis
Aetiology
Screening Intervention
I4
A systematic review of level II
studies
A study of test accuracy with: an
independent, blinded comparison
with a valid reference standard,5
among consecutive persons with a
defined clinical presentation6
A study of test accuracy with: an
independent, blinded comparison
with a valid reference standard,5
among non-consecutive persons
with a defined clinical presentation6
A comparison with reference
standard that does not meet the
criteria required for Level II and III-1
evidence
A systematic review
of level II studies
A prospective
cohort study
A systematic review of
level II studies
A prospective cohort
study
A systematic review of
level II studies
A randomised controlled
trial
All or none
All or none
A pseudorandomised
controlled trial (i.e.
alternate allocation or
some other method)
II
A systematic review of level
II studies
A randomised controlled trial
III-1
A pseudorandomised
controlled trial (i.e. alternate
allocation or some other
method)
III-2
A comparative study with
concurrent controls: ▪ Nonrandomised, experimental
trial9 ▪ Cohort study ▪ Casecontrol study ▪ Interrupted
time series with a control
group
III-3
A comparative study without Diagnostic case-control study6
concurrent controls: ▪
Historical control study ▪ Two
or more single arm study10 ▪
Interrupted time series
without a parallel control
group
Case series with either post- Study of diagnostic yield (no
test or pre-test/post-test
reference standard)
outcomes
IV
Analysis of
A retrospective cohort
prognostic factors
study
amongst persons in
a single arm of a
randomised
controlled trial
A comparative study with
concurrent controls: ▪
Non-randomised,
experimental trial ▪ Cohort
study ▪ Case-control
study
A retrospective
cohort study
A case-control study
A comparative study
without concurrent
controls: ▪ Historical
control study ▪ Two or
more single arm study
Case series, or
cohort study of
persons at different
stages of disease
A cross-sectional study Case series
or case series
NHMRC Levels of Evidence
8
NHMRC Assessment of study quality –
Grades of Recommendations
1. The evidence base, in terms of the number of studies, level of evidence and
quality of studies (risk of bias).
2. The consistency of the study results.
3. The potential clinical impact of the proposed recommendation.
4. The generalisability of the body of evidence to the target population for the
guideline.
5. The applicability of the body of evidence to the Australian healthcare context.
9
Checklist for appraising the quality of studies of interventions (Cochrane handbook)
1. Method of treatment assignment
a. Correct, blinded randomisation method described
OR randomised, double-blind method stated
AND group similarity documented
b. Blinding and randomisation stated but method not described
OR suspect technique (eg allocation by drawing from an envelope)
c. Randomisation claimed but not described and investigator not blinded
d. Randomisation not mentioned
2. Control of selection bias after treatment assignment
a. Intention to treat analysis AND full follow-up
b. Intention to treat analysis AND <15% loss to follow-up
c. Analysis by treatment received only OR no mention of withdrawals
d. Analysis by treatment received
AND no mention of withdrawals
OR more than 15% withdrawals/loss-to-follow-up/post-randomisation exclusions
3. Blinding
a. Blinding of outcome assessor
AND patient and care giver
b. Blinding of outcome assessor
OR patient and care giver
c. Blinding not done
4. Outcome assessment (if blinding was not possible)
a. All patients had standardised assessment
b. No standardised assessment OR not mentioned
Source: National Health and Medical Research Council (NHMRC). How to review the evidence:
systematic identification and review of the scientific literature. Canberra: NHMRC, 1999: p.45
10
NHMRC Grades of Recommendations
Grade of recommendation
Description
A
Body of evidence can be trusted to
guide practice
B
Body of evidence can be trusted to
guide practice in most situations
C
Body of evidence provides some
support for recommendation(s) but
care should be taken in its application
D
Body of evidence is weak and
recommendation must be applied
with caution
11
Clinical Guidelines for Stroke Management 2010. National Stroke Foundation
12
Class of evidence
Study design
Quality Criteria
I
Good quality randomized controlled trial
(RCT)
Adequate random assignment method
Allocation concealment
Groups similar at baseline
Outcome assessors blinded
Adequate sample size
Intention-to-treat analysis
Follow-up rate 85%
No differential loss to follow-up
Maintenance of comparable groups
II
Moderate quality RCT
Violation of one or more of the criteria for a good quality RCT
II
Good quality cohort
Blind or independent assessment in a prospective study, or use
cohort of reliable data in a retrospective study
Non-biased selection
Follow-up rate 85%
Adequate sample size
Statistical analysis of potential confounders
II
Good quality case-control
Accurate ascertainment of cases
Nonbiased selection of cases/controls with exclusion criteria
applied equally to both
Adequate response rate
Appropriate attention to potential confounding variables
III
Poor quality RCT
Major violations of the criteria for a good or moderate quality RCT
III
Moderate or poor quality cohort
Violation of one or more criteria for a good quality cohort
III
Moderate or poor quality case-control
Violation of one or more criteria for a good quality case-control
III
Case series, databases or registries
Brain Trauma, F., S. American Association of Neurological, et al. (2007). "Guidelines for the management of severe traumatic
brain injury." Journal of Neurotrauma 24 Suppl 1.
13
Table 1. Applying Classification of Recommendations and Level of Evidence
Morgenstern, L. B., J. C. Hemphill, 3rd, et al. (2010). "Guidelines for the management of
spontaneous intracerebral haemorrhage: a guideline for healthcare professionals from
the American Heart Association/American Stroke Association." Stroke 41(9): 2108-29.
14
Is Evidence-Based Surgery an oxymoron?
What if there is no level I evidence?
Surgical RCTs have well-recognized disadvantages:
high costs, administrative complexity, prolonged time
to completion, recruitment difficulty, blinding,
randomization technique standardization, poor
generalizability or external validity, patient
compliance, underpowered studies, crossovers and
drop outs, multiple surgical options, technological
advancement, patient complexity, variability and
preference and selection bias ....
15
• It is the well-defined research question that dictates the study
design, not that every study should be a RCT because it’s the
gold standard”
• “The proper use of evidence-based information is not the strict
adherence to only RCTs, but more accurately, the informed and
effective use of all types of evidence … Large, prospective
cohort studies in a surgical setting are often thought to be on a
par with RCTs and provide superior generalizability”
Fisher, C. G. and K. B. Wood (2007). "Introduction to and techniques of
evidence-based medicine." Spine 32(19 Suppl): S66-72.
16
ISAT (Lancet, 2002; Lancet, 2005)
•
•
•
•
•
•
•
9559 eligible patients, 2143 randomised
– 43 Participating centres; enrolled 1-44% of eligible patients
Equipoise did not exist in over 75%
– 3615 underwent surgery
– 2737 underwent endovascular treatment
– 1064 unknown treatment
Differences between groups
– Cross-overs, time to treatment,
Small but significant difference in time between randomisation and first
procedure
– Coiling 1.1 days
Surgery 1.7 days
Cross-overs
– Coiling  surgery 9 patients
– Surgery  coiling 38 patients
Different experience of INR and surgeons
Unknown differences between centres
17
Using the evidence....
• Know which levels and grades of recommendations
are being used and quote/reference them
• Stay up to date with developments or changes to
levels and grades
• Use NHMRC levels and grades where possible
• Look for other levels of evidence when RCTs or level
1 studies are not possible
• Make evidence-based decisions
• Become a lifelong learner of EBM
http://libguides.mq.edu.au/content.php?pid=167579&
sid=1412023
18