Chicago 2014
Incorporating lower levels of
evidence
Peter Morley, Eddy Lang
E3, GRADE expert
Chicago 2014
COI #380
Commercial/industry
Evidence Evaluation Expert
(E3: ILCOR/AHA)
Potential intellectual conflicts
Acting Chair Australian Resuscitation
Council (ARC)
ANZCOR delegate on International
Liaison Committee on Resuscitation
(ILCOR)
Chicago 2014
EBP is to consider the best
available evidence
Chicago 2014
Issues
EBP is to consider the best available
evidence
GRADE and other approaches to
systematic reviews only incorporate RCTs
and cohort studies
Chicago 2014
Unfortunately in Resuscitation and
First Aid science
RCTs are lacking
Good quality non-RCTs are also lacking
Sometimes only case series, case reports
or animal data (previous LOE 4 and 5)
Chicago 2014
GRADE allocation of quality of
evidence across each outcome
RCTs start as high
can be downgraded
Non-RCTs start as low
Can be downgraded
Can occasionally be upgraded
Chicago 2014
So where do Case Series fit?
Case series are observational studies
Same bias assessment tools as other
observational studies
Recommended only to include if 4 or
more cases are included in a
manuscript
Chicago 2014
Bias assessment for Case Series
Were appropriate eligibility criteria developed
and applied to both the cohort of interest and
the comparison cohort? NO
Was measurement of both exposure and
outcome appropriate and consistently applied
to both the cohort of interest and the
comparison cohort? NO
Was confounding adequately controlled for?
Could be YES
Was follow up complete? Could be YES
Determinants of quality
Chicago 2014
• RCTs start high
• Observational studies start low
• Case series start very low
What lowers quality of evidence?
5 factors:
Methodological Inconsistency
limitations
of results
Indirectness
of evidence
Imprecision
of results
Publication
bias
Chicago 2014
Still go on to complete a GRADE
Evidence Profile table
Enter all studies you include in your
final analysis for each outcome
Chicago 2014
www.guidelinedevelopment.org
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Then populate the fields using the
drop-down menus
No serious limitations: Most
information is from studies at low
risk of bias.
Serious limitations: Most
information is from studies at
moderate risk of bias.
Very serious limitations: Most
information is from studies at high
risk of bias.
Chicago 2014
And don’t forget to add a
comment to the footnotes each
time you are downgrading
based on your assessment
Chicago 2014
Implications of the assessment of
limitations
No serious limitations: Most information is from
studies at low risk of bias
No downgrading of evidence
Serious limitations: Most information is from
studies at moderate risk of bias
Evidence would be normally downgraded 1 level
(ie. for RCTs from high to moderate)
Very serious limitations: Most information is from
studies at high risk of bias
Evidence would be normally downgraded 2
levels
(ie. for RCTs from high to low)
Chicago 2014
Then populate the fields using the
drop-down menus
Reviewers should document
limitations when (1) point estimates
vary widely across studies, (2)
confidence intervals show minimal
or no overlap (ie. studies appear to
have different effects), or
(3) statistical tests of heterogeneity
are suggestive
Chicago 2014
Concerns about directness therefore
arise when there are differences in:
• The Population (eg. patients in
cardiac arrest vs not in cardiac
arrest),
• Intervention (eg. standard CPR
using 2010 guidelines vs
standard CPR using 2005
guidelines), or
• Outcomes (surrogates, eg.
ROSC vs termination of VF for 5
seconds), or where
• no head to head Comparisons
between interventions
Chicago 2014
Other concerns about indirectness
Evidence from animal studies, manikins or other
models would generally be rated as having very
serious limitations (but this would be dependent on
the key outcomes listed).
Interventions may be delivered/implemented differently
in different settings (eg. therapeutic hypothermia).
Limitations should therefore be considered if the
intervention cannot be implemented with the
same rigor or technical sophistication as in the
studies from which the data come.
Limitations in more than one type of directness may
suggest a need to rate the studies as having very serious
limitations.
Chicago 2014
Concerns about imprecision arise
when:
• Cis overlap with a clinical
decision threshold, eg. 1%
absolute difference
• Trials stopped early
• Early publications (particularly
if small)
• Optimal Information Size (eg.
<400 patients for continuous
data)
• Outcomes: importance, the
adverse effects, the burden to the
patient, resource use, and the
difficulty of introducing a
measure into practice
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Publication bias
Reviewers should allocate strongly
suspected when the evidence consists
of a number of small studies,
especially if these are industry
sponsored or if the investigators share
another conflict of interest.
The risk of publication bias in
observational studies is probably
larger than RCTs
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Magnitude of effect for non-RCTs
A large magnitude effect would be considered as justification to
increase the rating by one level (eg. low to moderate) if:
relative risk [RR] = 2-5 or RR 0.5-0.2 with no plausible
confounders; or
RR very large >5 or <0.2 and no serious problems with risk of bias or
precision (sufficiently narrow confidence intervals);
The reviewer would be more likely to rate up if the above size of effects
were rapid and out of keeping with prior trajectory; they are usually
supported by indirect evidence.
If above criteria are all met, and the RR is very large (eg. >5-10),
rating up by two levels (from low to high) could be considered, but
obviously not if other concerns were present (such as risk of bias,
imprecision, inconsistency, indirectness and publication bias).
Chicago 2014
Then populate the fields using the
drop-down menus
Chicago 2014
Plausible confounding for non-RCTs
If:
all plausible prognostic factors are
accurately measured in
observational studies, and if
all the observed residual
confounders and biases would
diminish the observed effect,
then the effect estimate would be
strengthened. In this setting, the rating
up by one level could be considered.
Chicago 2014
Chicago 2014
Dose-response gradient for non-RCTs
A dose-response gradient (such as
increased effect with increased dose, or
decreased time to intervention) increases
the confidence in the findings of
observational studies.
In this setting, the rating up by one level
could be considered
Chicago 2014
Outcomes
For dichotomous (yes/no)
variables, for a single study, enter
number with outcome, and total
number of patients.
Then click “apply”
Chicago 2014
No control group
For dichotomous (yes/no)
variables, for a single
study, enter number with
outcome, and total
number of patients.
Then click “apply”
Chicago 2014
No relative effects
Choose desired variable: usually Relative Risk (RR): single
or multiple studies if meta-analysis performed
Chicago 2014
When and how to perform a meta-analysis
(interventional studies)
When
More than one study, of
Good methodological quality, addressing
Near identical individual components of PICO
Population, Intervention, Comparison,
Outcome
Rarely with non-RCTs
How
Send email and data for listed outcomes for
studies to Peter or Eddy
Chicago 2014
GRADE evidence profile
(preferred)
Chicago 2014
Chicago 2014
Chicago 2014
Chicago 2014
Cost and resource implications
Peter Morley, Eddy Lang, Laurie Morrison
Methods Committee
Chicago 2014
COI #380
Commercial/industry
Evidence Evaluation Expert (E3:
ILCOR/AHA)
Potential intellectual conflicts
Acting Chair Australian Resuscitation
Council (ARC)
ANZCOR delegate on International
Liaison Committee on Resuscitation
(ILCOR)
Chicago 2014
ILCOR Methods Committee
Guidance on the Consideration
of Resources in the Creation of
Treatment Recommendations
as part of the CoSTR
Chicago 2014
Issues
the requirement of significant
methodological expertise and rigour
around health economic evaluations
the variability in cost and resource
considerations across a myriad of
settings
Chicago 2014
2 Cost/Resource Questions
Are the resources required small?
Is the incremental cost small relative to
the net benefits?
Chicago 2014
Overarching principles
The greater the resource requirements
(cost, including but not limited to
equipment price and training costs), the
less likely it is that an option will be a
priority.
Resource costs, and the implications of
these to the Councils implementing the
guidelines, are likely to vary substantially
across jurisdictions.
Chicago 2014
1st question
“Are the resources required small?”
Chicago 2014
“Are the resources required small?”
No*
Probably no
Uncertain
Probably yes
Yes*
* Need to be supported by formal cost
analysis
Chicago 2014
“Are the resources required small?”
No*
Probably no
Uncertain
Probably yes
Yes*
* Need to be supported by formal cost
analysis
Chicago 2014
“Are the resources required small?”
No
supported by formal cost analysis in
relevant contexts
Chicago 2014
“Are the resources required small?”
Probably no
Significant resources are probably required, but no
formal cost analysis identified.
Incremental implementation costs are significant
and likely to impact on the introduction of the new
therapy/diagnostic technique.
These could include significant drug, equipment or
diagnostic test costs, significant educational
requirements, or additional significant costs for
employing more staff or purchase of monitoring
devices.
Please elaborate on the reasons you chose this
option.
Chicago 2014
“Are the resources required small?”
Uncertain
Resource estimates vary considerably and
no reliable source of information can be
identified
Chicago 2014
“Are the resources required small?”
Probably yes
Resources probably not altered, but no
formal cost analysis identified.
Incremental implementation costs are
small and unlikely to impact on the
introduction of the new therapy/diagnostic
technique.
Chicago 2014
“Are the resources required small?”
Yes
supported by formal cost analysis
Chicago 2014
“Are the resources required small?”
No*
Probably no
Uncertain
Probably yes
Yes*
* Need to be supported by formal cost
analysis
Chicago 2014
2nd question
“Is the incremental cost small relative
to the net benefits?”
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
No*
Probably no
Uncertain
Probably yes
Yes*
* Need to be supported by formal
cost-effectiveness analysis
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
No*
Probably no
Uncertain
Probably yes
Yes*
* Need to be supported by formal
cost-effectiveness analysis
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
No
A formal cost-effectiveness analysis
suggests that the intervention is not
justified in terms of the resource allocated
for benefit obtained.
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
Probably no
Although no formal cost-effectiveness
analysis was conducted, the resources
required to implement the intervention or
diagnostic strategy appear to be large or
resource intensive relative to the
estimate of the net benefit.
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
Uncertain
The resource considerations are
difficult to estimate and the benefits
and risks/harms are uncertain so no
reasonable consideration of costeffectiveness is possible at this time.
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
Probably yes
Although no formal cost analysis was
conducted and efficacy estimates are
uncertain, the resources required are
deemed small enough that
implementation might be considered
reasonable from a cost effectiveness
perspective.
Potential example - ASA in suspected ACS.
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
Yes
A formal cost-effectiveness analysis
supports the view that the costs are
redeemed by improved outcomes.
Chicago 2014
“Is the incremental cost small
relative to the net benefits?”
No*
Probably no
Uncertain
Probably yes
Yes*
* Need to be supported by formal
cost-effectiveness analysis
Chicago 2014
A couple of examples
Chicago 2014
Example 1
Cardiogel
Chicago 2014
Cardiogel
systematic review identifying moderate
quality evidence from 3 randomized
controlled trials (Lang 2010 234, Morrison
2012 245, Morley 2013 341) downgraded
for risk of bias demonstrating that in
patients with cardiac arrest, “cardiogel” a
hypothetical free radical scavenger improves
survival to hospital discharge in comparison
to placebo.
The RR for this and other critical outcomes is
0.8 95% (CI 0.7-0.9)
Chicago 2014
Cardiogel
cardiogel is perceived by committee
members and the TFQO as expensive
“Probably no” to “cost small” Q
“Uncertain” to cost-effectiveness Q
(as no formal analysis was conducted
or identified)
Chicago 2014
Cardiogel: ILCOR guidance
The treatment recommendation should
not address resource considerations
State “no formal economic analysis based
on a systematic review was completed or
identified” in the “research evidence”
column to provide justification
Furthermore the question of resource
consumption should not weigh in the
development of the treatment
recommendation.
Chicago 2014
Example 2
“A novel bodywrap device”
Chicago 2014
Bodywrap
systematic review identifying very low
quality evidence (Lang 2010 234,
Morrison 2012 245, Morley 2013 341)
downgraded for imprecision, risk of
bias and indirectness suggesting that
a novel bodywrap device to assist CPR
fails to improve any critical or
important outcome in comparison to
standard CPR; RR 1.0 95% CI 0.4 – 1.6
no evidence of harm.
Chicago 2014
Bodywrap
the device is perceived by the committee members
and the TFQO to be significantly more expensive
in all countries represented by the Taskforce than
standard CPR and/or its implementation would
consume significant resources in most health care
settings
“Probably no” to “cost small” Q
an economic analysis based on a systematic
review of addressing the incremental costeffectiveness of CPR bodywrap was not done
“Uncertain” to cost-effectiveness Q
(as no formal analysis was conducted or
identified)
Chicago 2014
Bodywrap: ILCOR guidance (1)
In this setting it would be appropriate for
the TFQO and EvRevs to incorporate
consensus-based statements pertaining
to resource considerations into the
development of the treatment
recommendation for this PICO.
Specifically it would be appropriate to
suggest against the use of bodywrap over
standard CPR – weak recommendation
Chicago 2014
Bodywrap: ILCOR guidance (2)
“In making this recommendation we place
greatest value on not directing health care
resources that were perceived by the
committee to be high towards an unproven
treatment except under experimental
conditions.”
Chicago 2014
If bodywrap were:
relatively inexpensive and/or already
readily available to resuscitation teams
in most or all settings
the Task Force could consider making
a weak recommendation in favour of
CPR bodywrap over standard CPR
“In making this recommendation we
place a higher value on a potential
clinical benefit in a setting with a high
rate of poor outcomes over other
considerations”.
Additional comments
Chicago 2014
when a useless or harmful therapy is in
wide use, developing recommendations
against a management approach is
also appropriate as would
recommendations to use
interventions only in research.
Task Forces may also choose to divide
their recommendations into ones that
address high and low resource
settings.
Chicago 2014