Holger Schünemann, MD, PhD
Professor and Chair, Dept. of Clinical Epidemiology & Biostatistics
Professor of Medicine
Michael Gent Chair in Healthcare Research
McMaster University, Hamilton, Canada
HOW GRADE COULD HELP TO
IMPLEMENT THE EVIDENCE
Content
Pulmonary rehabilitation compared to usual community care for COPD with recent exacerbation
Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane
Database of Systematic Reviews 2010, Issue 11.
Outcomes
No of Participants
(studies)
Follow up
Quality of the
evidence
(GRADE)
Relative effect
(95% CI)
Hospital
admission
250
(6 studies)
3-18 months
⊕⊕⊕⊕
HIGH
OR 0.22
(0.08 to 0.58)
Mortality
110
(3 studies)
3-48 months
⊕⊕⊕⊝
MODERATE2
due to imprecision
OR 0.28
(0.1 to 0.84)
Anticipated absolute effects
Risk with Usual
community care
Risk difference with
Pulmonary rehabilitation
(95% CI)
405 per 1000
275 fewer per 1000
(from 122 fewer to 353
fewer)
Low1
100 per 1000
70 fewer per 1000
(from 15 fewer to 89
fewer)
High1
Systematic
Reviews & GRADE
500 per 1000
281 fewer per 1000
(from 43 fewer to 409
fewer)
Quality of life
(CRQ) dyspnea
Chronic
Respiratory
Questionnaire3.
Scale from: 1 to
7.
258
(5 studies)
12 and 76 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
3.1
The mean quality of
life (crq) dyspnea in
the intervention groups
was
0.97 higher
(0.35 to 1.58 higher)
Quality of life
(SGRQ) total
St George's
Respiratory
Questionnaire5.
Scale from: 0 to
100.
127
(3 studies)
12 and 26 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
50
The mean quality of
life (sgrq) total in the
intervention groups
was
9.88 lower
(5.37 to 14.4 lower)
Ambulation (as
measured by 6
min walking
distance)
distance in
meters 6
299
(6 studies)
1 - 208 weeks7
⊕⊕⊕⊝
MODERATE4,8
due to imprecision
Resource use not reported
-
-
The mean ambulation
(as measured by 6 min
walking distance) in
the intervention groups
was
77.7 higher
(12.21 to 143.2 higher)
-
See footnote
Evidence &
judgments
See footnote
Recommendation,
health policy &
implementation
Content
Examples and summary from
leading/co-leading 15 guideline
projects
• 11 World Health
Organization
• World Allergy Organization,
Allergic Rhinitis in Asthma,
American Thoracic
Society
Systematic
Evidence &
Recommendation,
(2)
Reviews & GRADE
judgments
health policy &
Pulmonary rehabilitation compared to usual community care for COPD with recent exacerbation
Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane
Database of Systematic Reviews 2010, Issue 11.
Outcomes
No of Participants
(studies)
Follow up
Quality of the
evidence
(GRADE)
Relative effect
(95% CI)
Hospital
admission
250
(6 studies)
3-18 months
⊕⊕⊕⊕
HIGH
OR 0.22
(0.08 to 0.58)
Mortality
110
(3 studies)
3-48 months
⊕⊕⊕⊝
MODERATE2
due to imprecision
OR 0.28
(0.1 to 0.84)
Anticipated absolute effects
Risk with Usual
community care
Risk difference with
Pulmonary rehabilitation
(95% CI)
405 per 1000
275 fewer per 1000
(from 122 fewer to 353
fewer)
Low1
100 per 1000
70 fewer per 1000
(from 15 fewer to 89
fewer)
High1
500 per 1000
281 fewer per 1000
(from 43 fewer to 409
fewer)
Quality of life
(CRQ) dyspnea
Chronic
Respiratory
Questionnaire3.
Scale from: 1 to
7.
258
(5 studies)
12 and 76 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
3.1
The mean quality of
life (crq) dyspnea in
the intervention groups
was
0.97 higher
(0.35 to 1.58 higher)
Quality of life
(SGRQ) total
St George's
Respiratory
Questionnaire5.
Scale from: 0 to
100.
127
(3 studies)
12 and 26 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
50
The mean quality of
life (sgrq) total in the
intervention groups
was
9.88 lower
(5.37 to 14.4 lower)
Ambulation (as
measured by 6
min walking
distance)
distance in
meters 6
299
(6 studies)
1 - 208 weeks7
⊕⊕⊕⊝
MODERATE4,8
due to imprecision
Resource use not reported
-
-
The mean ambulation
(as measured by 6 min
walking distance) in
the intervention groups
was
77.7 higher
(12.21 to 143.2 higher)
-
See footnote
See footnote
implementation
Guideline development Process
Critical
Outcome
Critical
Outcome
Important
Outcome
Not
High
Moderate
Low
Very low
Summary of findings
& estimate of effect
for each outcome
Systematic review
Grade down
Outcome
Grade up
P
I
C
O
Randomization
increases initial
quality
1. Risk of bias
2. Inconsistency
3. Indirectness
4. Imprecision
5. Publication
bias
1. Large effect
2. Dose
response
3. Confounders
Guideline development
Formulate recommendations:
• For or against (direction)
• Strong or conditional/weak
(strength)
By considering:
 Quality of evidence
 Balance benefits/harms
 Values and preferences
Revise if necessary by considering:
 Resource use (cost)
Grade
overall quality of evidence
across outcomes based on
lowest quality
of critical outcomes
•
•
•
•
“We recommend using…”
“We suggest using…”
“We recommend against using…”
“We suggest against using…”
Outcome generation and selection
• Multidisciplinary panels
– Researchers, epidemiologists, public health
officers, methodologists, patient representatives…
• Delphi process
– 3 rounds
1. List of possible outcomes from literature
– Panel members review and add
2. List of all outcomes grouped by theme
– Panel members rate importance
3. Final agreement and results
Outcomes for screening on a scale of 1 (not important) to 9 (critical)
Mortality from cervical cancer
Cervical cancer Incidence
Detected CIN 2,3
Major Infections (requiring hospital admission and antibiotics, e.g. PID)
Maternal bleeding
Premature delivery
Fertility
Identification of STIs (benefit)
Minor infections (requiring outpatient treatment only)
8.2
8.3
7. 9
6.0
5.8
5.7
5.4
5.0
3.8
Challenges and advantages of this
approach
• Often starting with
many outcomes
• Experts initially focused
on what they know
from research studies
• Requires detailed
explanations
• ↓ Participation of panel
members
• Perspective taken
• Complete
• Everyone involved
– Independent ratings
• Numerical estimates
• Well documented and
kept record
• Transparent
• Reduces work
WHO influenza guidelines
• New guideline on pharmacological
management of influenza
– Previously few randomized trials
• Low quality evidence for many outcomes (imprecision)
• Industry sponsored – publication bias
• Not all outcomes
• Review of observational studies
– To inform guidelines
Methods
• Standard systematic review
– MEDLINE, EMBASE, CENTRAL, CINAHL, SIGLE, the Chinese
Biomedical Literature Database, Panteleimon and LILACS
for relevant studies up to November 2010
– contacted pharmaceutical companies and international
agencies
– RevMan 5.1
• 10 PICO → recommendations approach
– Outcomes determined through Delphi process previously
• QoE according to GRADE approach
– GRADEpro (www.gradeworkinggroup.org)
– Risk of bias using modified Ottawa Newcastle scale
Results
Question: Should oseltamivir vs. no antiviral treatment be used for influenza (follow-up: 30 days)?
Quality assessment
Participants Risk of bias Inconsistency Indirectness Imprecision Publication
bias
(studies)
Summary of Findings
Overall quality of
evidence
Study event rates (%)
With no
antiviral
treatment
Relative effect Anticipated absolute effects
With oseltamivir (95% CI)
Risk with no antiviral treatment
Absolute effect with Oseltamivir
(95% CI)
Mortality
681
(3 studies)
no serious no serious
no serious no serious undetected1 ⊕⊕⊝⊝
risk of bias inconsistency indirectness imprecision
LOW1
1557
(9 studies)
serious2
59/242
(24.4%)
no serious
no serious no serious undetected1 ⊕⊝⊝⊝
61/320
inconsistency indirectness imprecision
VERY LOW1,2
(19.1%)
due to risk of bias
31/439
(7.1%)
adj OR 0.23
(0.13 to 0.43)
240 deaths per 1000
172 fewer deaths per 1000
(from 120 to 201 fewer)
228/1237
(18.4%)
OR 0.51
(0.23 to 1.14)3
240 deaths per 1000
101 fewer deaths per 1000
(from 172 fewer to 25 more)
Hospitalisation
150710
(5 studies)
no serious no serious
no serious no serious undetected4 ⊕⊕⊝⊝
risk of bias inconsistency indirectness imprecision
LOW4
242762
(6 studies)
serious2
1238/100585 431/50125
(1.2%)
(0.86%)
adj OR 0.75
(0.66 to 0.89)
12 hospitalisations per 1000
3 fewer hospitalisations per 1000
(from 1 to 4 fewer)
1086/96352
(1.1%)
OR 0.75
(0.66 to 0.86)
12 hospitalisations per 1000
3 fewer hospitalisations per 1000
(from 2 to 4 fewer)
no serious no serious undetected1 ⊕⊝⊝⊝
indirectness imprecision
VERY LOW1,6
due to risk of bias,
inconsistency
200/1032
(19.4%)
-
-
647/50017
(1.3%)
adj OR 0.83
(0.59 to 1.16)
21 pneumonias per 1000
4 fewer pneumonias per 1000
(from 9 fewer to 3 more)
1273/99020
(1.3%)
OR 0.64
(0.46 to 0.88)
20 pneumonias per 1000
7 fewer pneumonias per 1000
(from 2 to 10 fewer)
no serious
no serious no serious undetected4 ⊕⊝⊝⊝
1738/
inconsistency indirectness imprecision
VERY LOW2,4
146410
due to risk of bias (1.2%)
ICU admissions/mechanical ventilation/respiratory failure
1032
Serious5
(6 studies5)
serious6
Complications - Pneumonia
150466
(3 studies)
no serious serious6
risk of bias
no serious no serious undetected4 ⊕⊝⊝⊝
indirectness imprecision
VERY LOW4,6
due to
inconsistency
265276
(6 studies)
serious2
no serious no serious undetected4 ⊕⊝⊝⊝
3244/
indirectness imprecision
VERY LOW2,4,6
166256
due to risk of bias, (2%)
inconsistency
serious6
2111/
100449
(2.1%)
Content
Pulmonary rehabilitation compared to usual community care for COPD with recent exacerbation
Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane
Database of Systematic Reviews 2010, Issue 11.
Outcomes
No of Participants
(studies)
Follow up
Quality of the
evidence
(GRADE)
Relative effect
(95% CI)
Hospital
admission
250
(6 studies)
3-18 months
⊕⊕⊕⊕
HIGH
OR 0.22
(0.08 to 0.58)
Mortality
110
(3 studies)
3-48 months
⊕⊕⊕⊝
MODERATE2
due to imprecision
OR 0.28
(0.1 to 0.84)
Anticipated absolute effects
Risk with Usual
community care
Risk difference with
Pulmonary rehabilitation
(95% CI)
405 per 1000
275 fewer per 1000
(from 122 fewer to 353
fewer)
Low1
100 per 1000
70 fewer per 1000
(from 15 fewer to 89
fewer)
High1
Systematic
Reviews & GRADE
500 per 1000
281 fewer per 1000
(from 43 fewer to 409
fewer)
Quality of life
(CRQ) dyspnea
Chronic
Respiratory
Questionnaire3.
Scale from: 1 to
7.
258
(5 studies)
12 and 76 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
3.1
The mean quality of
life (crq) dyspnea in
the intervention groups
was
0.97 higher
(0.35 to 1.58 higher)
Quality of life
(SGRQ) total
St George's
Respiratory
Questionnaire5.
Scale from: 0 to
100.
127
(3 studies)
12 and 26 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
50
The mean quality of
life (sgrq) total in the
intervention groups
was
9.88 lower
(5.37 to 14.4 lower)
Ambulation (as
measured by 6
min walking
distance)
distance in
meters 6
299
(6 studies)
1 - 208 weeks7
⊕⊕⊕⊝
MODERATE4,8
due to imprecision
Resource use not reported
-
-
The mean ambulation
(as measured by 6 min
walking distance) in
the intervention groups
was
77.7 higher
(12.21 to 143.2 higher)
-
See footnote
Evidence &
judgments
See footnote
• Systematic and
transparent
approach
• Transparently lay
out rationale for
recommendations
• Manage COI
Recommendation,
health policy &
implementation
Judgments/explanations
1
Although we did not downgrade, publication bias
cannot be excluded and is of concern.
2 Studies not adjusted for potential confounding factors.
3 Significant differences in effect for pandemic versus
seasonal influenza (see subgroup analyses table).
4 Publication bias a concern since large studies had forprofit funding and weighted heavily in analyses.
5 No independent comparison group.
6 High heterogeneity among studies.
7 Measured in select patients in trials.
Getting from evidence to
recommendations - GRADE
Recommendations are based on judgments:
– Quality of evidence (confidence in estimates of
effect)
– Balance between benefits and downsides
– Values and preferences
– Resource use
But judgments need to be based on the best
available evidence and transparent
Balancing desirable and undesirable
consequences
↑ herd
immunity
Conditional
Strong
↓
Morbidity
↓ Death
↑ QoL
For
↑ Resources
↑ Allergic
reactions
↑ Nausea
↑ Local skin
reactions
Against
Balancing desirable and undesirable
consequences
Conditional
Strong
For
Against
Balancing desirable and undesirable
consequences
Conditional
Strong
For
Against
Balancing desirable and undesirable
consequences
Conditional
Strong
For
Against
Balancing desirable and undesirable
consequences
Conditional
Strong
For
Against
Content
Pulmonary rehabilitation compared to usual community care for COPD with recent exacerbation
Bibliography: Puhan M, et al. Pulmonary rehabilitation following exacerbations of chronic obstructive pulmonary disease. Cochrane
Database of Systematic Reviews 2010, Issue 11.
Outcomes
No of Participants
(studies)
Follow up
Quality of the
evidence
(GRADE)
Relative effect
(95% CI)
Hospital
admission
250
(6 studies)
3-18 months
⊕⊕⊕⊕
HIGH
OR 0.22
(0.08 to 0.58)
Mortality
110
(3 studies)
3-48 months
⊕⊕⊕⊝
MODERATE2
due to imprecision
OR 0.28
(0.1 to 0.84)
Anticipated absolute effects
Risk with Usual
community care
Risk difference with
Pulmonary rehabilitation
(95% CI)
405 per 1000
275 fewer per 1000
(from 122 fewer to 353
fewer)
Low1
100 per 1000
70 fewer per 1000
(from 15 fewer to 89
fewer)
High1
Systematic
Reviews & GRADE
500 per 1000
281 fewer per 1000
(from 43 fewer to 409
fewer)
Quality of life
(CRQ) dyspnea
Chronic
Respiratory
Questionnaire3.
Scale from: 1 to
7.
258
(5 studies)
12 and 76 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
3.1
The mean quality of
life (crq) dyspnea in
the intervention groups
was
0.97 higher
(0.35 to 1.58 higher)
Quality of life
(SGRQ) total
St George's
Respiratory
Questionnaire5.
Scale from: 0 to
100.
127
(3 studies)
12 and 26 weeks
⊕⊕⊕⊝
MODERATE4
due to imprecision
The mean quality of
life (sgrq) total in the
control groups was
50
The mean quality of
life (sgrq) total in the
intervention groups
was
9.88 lower
(5.37 to 14.4 lower)
Ambulation (as
measured by 6
min walking
distance)
distance in
meters 6
299
(6 studies)
1 - 208 weeks7
⊕⊕⊕⊝
MODERATE4,8
due to imprecision
Resource use not reported
-
-
The mean ambulation
(as measured by 6 min
walking distance) in
the intervention groups
was
77.7 higher
(12.21 to 143.2 higher)
-
See footnote
Evidence &
judgments
See footnote
Recommendations,
health policy &
implementation
Question/Recommendation: Should contacts of new or recurrent TB cases be investigated?
Population: In people (at normal risk) who had contact with new or recurrent cases of TB (does)
Intervention: contact investigation
Setting (if relevant): high income countries
Decision domain:
Quality of evidence (QoE)
Is there high or moderate quality
evidence?
The higher the quality of
evidence, the more likely is a
strong recommendation
Comparison: no investigation
Judgment
Yes

No

Summary of reason for judgment
Explanation

There is very low quality
evidence from observational
studies that are moderate risk
of bias for the critical
outcomes.
Balance of benefits versus
harms and burdens
Are you confident that the
benefits outweigh the harms
and burden or vice versa?
The larger the difference
between the benefits and harms
and the certainty around that
difference, the more likely is a
strong recommendation. The
smaller the net benefit or net
harm and the lower the
certainty for that net effect, the
more likely is a
conditional/weak
recommendation.
Yes

No

There is considerable benefit while
little clinical harm or downsides are
expected
 The yield for all tuberculosis
(bacteriologically-confirmed
and clinically diagnosed)
was 4.5% of contacts
investigated.
 Latent tuberculosis
infection was found in
51.4% of contacts
investigated.
 The yield for all tuberculosis
(bacteriologically-confirmed
and clinically diagnosed)
was 7.0 % of pediatric
contacts investigated.
 Latent tuberculosis
infection was found in
40.4% % of pediatric
contacts investigated.
Subdomains influencing judgment
QoE for benefits:
Very low
QoE for harms:
Harms not explicitly evaluated
QoE for resource use:
Resource use not explicitly evaluated
Key reasons for down- or upgrading?
Risk of bias was a reason for downgrading for
most critical outcomes
All critical outcomes measured?
Harms and resources not explicitly evaluated
Baseline risk for benefits and harm and
burden?
 Is the baseline risk similar across
subgroups?
 Should there be separate
recommendations for subgroups?
Relative risk for benefits and harms:
 Are the relative benefits large?
Yes, the relative benefits are probably large.
 Are the relative harms large?
No, the relative harms are probably small.
Recommendations for other groups are made
separately, pediatric and adult index cases
were considered together.
Requirement for modeling:
 Is there a lot of extrapolation and
modeling required for these
outcomes?
Yes, modeling is required.
Values and preferences
Are you confident about the
assumed or identified relative
values and are they similar
across the target population?
Yes

No

Benefits much higher valued than
expected minor harms.
The more certainty or similarity
in values and preferences, the
more likely a strong
recommendation.
Resource implications
Are the resources worth the
expected net benefit from
following the recommendation?
The lower the cost of an
intervention compared to the
alternative, and other costs
related to the decision – that is,
the fewer resources consumed –
the more likely is a strong
recommendation in favour of
that intervention.
Overall strength of
recommendation
Remarks
Yes

A high value was placed on
avoiding consequences of TB,
dissemination of TB and
mortality.
A low value was placed on
possible adverse events.
There is likely little variability
and panel is quite certain
No

Resources required are worth the
net benefit considering the benefit
on mortality and new TB cases.
?Strong/con
ditional?
There are resources required to
conduct contact investigation
but these resources are worth
the expected benefits and
downstream treatment costs.
Perspective taken:
Patients or public
Source of values:
Guideline panels assessment
Source of variability if any:
Not a lot of variability
Method for determining values satisfactory for
this recommendation:
Yes, given the expected small variability and
difference between guideline panel and
patients.
Depending on contact investigation
strategy used, resource utilization
and are
implications
vary.unit?
What
the cost perwill
resource
Opportunity cost may be high.
Feasibility:
Opportunity
Feasibilitycost:
is dependent on existing
Differences
across
settings:
and well functioning
programs.
Resources worth in smear positive
index cases
The guideline panel recommends that contacts of patients with TB who are at normal risk be investigated. (NOTE: this is
a hypothetical recommendation developed for this article and not intended for clinical decision making.)
This recommendation places a high value on the benefits that can be expected (mortality reduction, reduction in TB dissemination) and
a relatively low value on the required resources and side effects of treatment.
Should cryotherapy versus LEEP be used in women with histologically
confirmed cervical intraepithelial neoplasia?
Quality assessment
No. of
studies
Design
Limitations
Inconsistency
No of patients
Indirectness
Imprecision
Other
Effect
Cryotherapy
LEEP
Relative
(95% CI)
Absolute effect at 1 year
(95% CI)
Quality
Importance
12/161
(7.5%)
4/168
(2.4%)
OR 3.3
(1.04 to 10.46)
51 more per 1000
(from 1 to 179 more)
O
CRITICAL
OO
CRITICAL
Recurrence CIN2–3 (follow-up 12 months randomized trials; 3–85 months observational studies)
1
randomized
trials
no serious
limitations
no serious
inconsistency
no serious indirectness seriousa,b
none
3
observational
studies
no serious
limitations
no serious
inconsistency
no serious indirectness no serious
imprecision
none
2227/14 387
(15.5%)
319/7454
(4.3%)
—
OR 2.66
(1.89 to 3.75)
2.4%c
37 more per 1000
(from 20 to 60 more)
Cervical cancer (follow-up 12 months randomized trials; 3–85 months to 26 years observational studies)
1
randomized
trials
no serious
limitations
no serious
inconsistency
no serious indirectness very seriousa
none
0/200
(0%)
0/200
(0%)
—
0 fewer per 1000d
OO
CRITICAL
2
observational
studies
no serious
limitations
no serious
inconsistency
no serious indirectness no serious
imprecision
none
2/679
(0.3%)
3/3350
(0.1%)
—
0 fewer per 1000e
OO
CRITICAL
none
15/170
(8.8%)
8/186
(4.3%)
OR 2.15
(0.89 to 5.22)
45 more per 1000
(from 5 fewer to 147 more)
OO
CRITICAL
3/300
(1%)
2/298f
(0.67%)
—
0.4 more per 1000
(from 8 fewer to 9 more)
OO
CRITICAL
136
480
OO
CRITICAL
OOO
CRITICAL
Treatment unacceptable to women (follow-up 2 weeks; acceptability question)
1
randomized
trials
no serious
limitations
no serious
inconsistency
no serious indirectness very seriousf
All severe adverse events (follow-up mean 12–16 months; stenosis and PID)
2
randomized
trials
no serious
limitations
no serious
inconsistency
no serious indirectness very seriousf
none
All severe adverse events (follow-up 33 months; PID, plug syndrome, stenosis, blood transfusion)
5
randomized
trials
no serious
limitations
no serious
inconsistency
serioush
serioush
none
4%i
—
OR 0.53
(0.1 to 2.88)
18 fewer per 1000
(from 36 fewer to 67 more)
All severe adverse events (follow-up 12 months; PID, stenosis, major bleeding)
9
observational
studies
serious
limitationsj
no serious
inconsistency
seriousi
seriousf
none
1/2233
(0%)
38/960
(4%)a
—
10 fewer per 1000
(from 20 fewer to 0)
Recommendation
• In settings where LEEP is available and accessible, and
women present with CIN lesions extending into the
cervical canal, the expert panel suggests treatment
with LEEP over cryotherapy (conditional
recommendation, OO quality evidence)
• Remarks: The benefits of LEEP were greater than those
of cryotherapy, and the harms were fewer in these
women. However, since there are greater resource
implications for LEEP than cryotherapy, and thus LEEP
is not available in all settings, a conditional
recommendation was made.
Implications of
a conditional/weak recommendation
• Patients: The majority of people in this situation
would want the recommended course of action, but
many would not
• Clinicians: Be more prepared to help patients to
make a decision that is consistent with their own
values/decision aids and shared decision making
• Policy makers: There is a need for substantial
debate and involvement of stakeholders
Implications of
a strong recommendation
• Patients: Most people in this situation would want
the recommended course of action and only a small
proportion would not
• Clinicians: Most patients should receive the
recommended course of action
• Policy makers: The recommendation can be adapted
as a policy in most situations , can be used as quality
indicator/performance measure
Only two of six performance
measures seemed reasonable
WHO evaluation & feedback
• WHO staff & guideline review
committee members (GRC) invited
to feedback (Jan 2011) about using
GRADE approach
– what worked
– what did not work
• Group discussion (NGT), 11 + 2
Summary of feedback
• Transparency of the GRADE process helps
• Requirement for a good Chair
– methods knowhow to move the process
• For observational studies we need better/
¿different? summaries (e.g. narrative versions)
• Integration and elicitation of values and
preferences was frequently challenging
• For global guidelines: issues around the
description of baseline risks and applicability
across countries require work
Summary of feedback
• Variability in baseline risk → weak or conditional
recommendations should follow
• A description of the modifying factors and the
layout of the evidence could be a great benefit
and will facilitate implementation
• Measures should be taken to streamline the
timing of the development of guidelines
• Working with centers, training, and capacity
building of these centers who collaborate with
WHO a priority for implementing GRADE
• Impact evaluation (of current process for
development of guidelines) should take place
Conclusions
• (WHO) guidelines should be based on the best
available evidence to be evidence based
• GRADE not avoid judgments but provides
framework
• combines what is known in health research methodology
and provides an approach to improve communication
• GRADE process works – is it better?
• Change in culture towards the use of evidence
• Transparency in decision making and judgments is
key
Thanks
• Nancy Santesso, Andy Oxman, Suzanne Hill
• WHO staff who participated in providing
feedback
Results - PRISMA
Records identified through
database searching (all
study designs)
EMBASE, MEDLINE = 9873
SIGLE = 7
CINAHL = 1062
LILACS = 19
COCHRANE = 301
Chinese Biomedical
Literature Database = 914
Panteleimon = 12
(Total n = 12176)
Records after duplicates removed
(n = 7456)
Records screened
(n = 7483)
Studies awaiting
assessment
(n = 6)
•Studies awaiting
translation (1)
•Papers could not
obtain in full (5)
Full-text articles
assessed for eligibility
(n = 920 )
Studies included
N = 89
Question
•51 + 5 studies
•7 studies
•6 studies
•0 studies
•8
•0 studies
•16
•0 studies
•1 study
•2 studies
Note: one study may
be relevant to
multiple questions
Additional records
identified through
other sources
Pharmaceutical
companies
(n = 12)
Reference lists of
relevant papers
(n=15)
Records excluded
(n = 6563)
Full-text articles
excluded
(n = 825)
Excluded for
•Not influenza or
influenza like illness
•Fewer than 25 people
•Randomised controlled
trial, or not an
observational study
•Not antiviral agent
•Antiviral agents
analysed together
•Prophylaxis
•No outcomes reported
Results
Should oseltamivir versus no treatment be used to treat influenza?
Mortality (adjusted)
oseltamivir no treatment
Study or Subgroup
log[Odds Ratio]
Odds Ratio
Total
Hanshaoworakul 2009
-2.040221 0.58739416
315
130
28.5%
0.13 [0.04, 0.41]
Liem 2009 (1)
-0.941609 0.75113239
55
12
17.5%
0.39 [0.09, 1.70]
McGeer 2009 (2)
-1.309333
69
100
54.0%
0.27 [0.12, 0.62]
242 100.0%
0.23 [0.13, 0.43]
0.4270348
Total (95% CI)
Total Weight
Odds Ratio
SE
439
IV, Random, 95% CI
Heterogeneity: Tau² = 0.00; Chi² = 1.58, df = 2 (P = 0.45); I² = 0%
IV, Random, 95% CI
0.1 0.2
0.5
1
2
5
10
Favours oseltamivir Favours no treatment
Test for overall effect: Z = 4.63 (P < 0.00001)
(1) Adjusted for neutropenia and hospital admission
(2) Does not specify what was adjusted for
Mortality (unadjusted)
Oseltamivir
Study or Subgroup
Chemaly 2007
Events
No treatment
Total Events
Odds Ratio
Total Weight
M-H, Random, 95% CI
0
25
3
8
5.2%
0.03 [0.00, 0.69]
150
328
5
8
13.4%
0.51 [0.12, 2.15]
Hien 2009
5
25
2
4
8.6%
0.25 [0.03, 2.24]
Huang 2009
2
17
1
57
7.3%
7.47 [0.63, 88.02]
0
118
0
27
18
55
8
12
14.4%
0.24 [0.06, 0.92]
Estenssoro 2010
Li 2010
Liem 2009
McGeer 2009
Not estimable
8
68
34
100
18.9%
0.26 [0.11, 0.60]
Siston 2010 (1)
21
476
5
74
17.3%
0.64 [0.23, 1.74]
Xi 2009
24
125
3
30
14.9%
2.14 [0.60, 7.64]
320 100.0%
0.51 [0.23, 1.14]
Total (95% CI)
Total events
1237
228
61
Heterogeneity: Tau² = 0.70; Chi² = 16.76, df = 7 (P = 0.02); I² = 58%
Test for overall effect: Z = 1.64 (P = 0.10)
(1) Pregnant women
Odds Ratio
M-H, Random, 95% CI
0.001
0.1
1
10
1000
Favours oseltamivir Favours no treatment
Recommendation
-
The Guidelines Group recommends that TB
programs/clinicians use/do not use
fluoroquinolones in the treatment of all
patients with MDR (Strong (conditional)
recommendation/ low (very low, low, moderate,
high) grade of evidence)
Example: Oseltamivir for Avian Flu
Recommendation: In patients with confirmed or
strongly suspected infection with avian influenza A
(H5N1) virus, clinicians should administer
oseltamivir treatment as soon as possible (strong
recommendation, very low quality evidence).
Remarks: This recommendation places a high value
on the prevention of death in an illness with a high
case fatality. It places relatively low values on
adverse reactions, the development of resistance
and costs of treatment.
Schunemann et al. The Lancet ID, 2007
Other explanations
Remarks: Despite the lack of controlled treatment
data for H5N1, this is a strong recommendation, in
part, because there is a lack of known effective
alternative pharmacological interventions at this
time.
The panel voted on whether this recommendation
should be strong or weak and there was one
abstention and one dissenting vote.
Recommendation: In settings where LEEP/LLETZ is available and accessible, the expert panel suggests treatment with LEEP/LLETZ over
cryotherapy
Population: Women with histologically confirmed CIN
Intervention: Cryotherapy versus LEEP
Decision
Factor
Explanation
High or moderate evidence
There is moderate-quality evidence from both randomized and
(is there high- or moderate-quality
observational controlled studies for recurrence rates. However, there
evidence?)
is low-quality evidence for other outcomes that were considered
critical and important for decision-making (e.g. severe adverse
 Yes
OO
The higher the quality of evidence, the
events, cervical cancer). There is uncertainty for fertility and other
 No
more likely is a strong
obstetric outcomes, and HIV acquisition/transmission was not
recommendation.
measured.

Certainty about the balance of
benefits versus harms and burdens
(is there certainty?)
The larger the difference between the
desirable and undesirable
consequences and the certainty
around that difference, the more
likely is a strong recommendation. The
smaller the net benefit and the lower
the certainty for that benefit, the
more likely is a conditional/ weak
recommendation.
 Yes
 No

Recurrence rates of CIN1, CIN2–3 and all CINs are probably
greater with cryotherapy:
o
CIN2–3, odds ratio (OR) 3.3 (CI 1.04 to 10.46)
o
CIN1, OR 2.74 (CI 0.62 to 12.07)
o
All CIN, OR 2.14 (CI 1.05 to 4.33).
Cryotherapy may be less acceptable to patients than LEEP.
There may be little difference in serious adverse events
between cryotherapy and LEEP, but there may be fewer minor
adverse events (such as pain) with cryotherapy.
It is unclear whether there is a difference in fertility/obstetric
outcomes.
High value was placed on CIN recurrence, serious adverse
events and acceptability to the patient.
Low value was placed on minor adverse events.


There is not a lot of variability
The panel felt secure in assuming the populations value


Need for more skilled providers to perform LEEP
Need for more or expensive equipment/supplies for LEEP;
electricity supply for LEEP
Need for local anaesthesia with LEEP

Benefits of LEEP
were greater, and 
harms were fewer
or similar


Certainty in or similar values (is there
certainty or similarity?)
The more certainty or similarity in
values and preferences, the more
likely is a strong recommendation.
Yes
 No
Similar values
across women
Yes
 No
More resources
required for LEEP
Resource implications
(are resources worth expected
benefits?)
The lower the cost of an intervention
compared to the alternative that is
considered and other costs related to
the decision – that is, fewer resources
consumed – the more likely is a strong
recommendation.
Overall strength of recommendation

Conditional
Methods
Types of participants
• We included studies in all populations with influenza or influenza like-illness.
Types of intervention
• Oseltamivir, zanamivir, amantadine or rimantadine in any dose or by any route.
Type of outcome measures
• We determined a priori to report on the following outcomes because they were
judged to be important or critical for decision making:
• Mortality, Hospitalisation, ICU Admission, mechanical ventilation and respiratory
failure, Duration of hospitalization, Time to alleviation of symptoms, Time to
return to normal activity, Complications
• Critical adverse events (e.g. major psychotic disorders, encephalitis, stroke and
seizure),
• Important adverse events (e.g. pain in extremities, clonic twitching, body
weakness, dermatological changes such as uticaria and rash)
• Viral shedding and Resistance
GRADE Uptake


















World Health Organization
Allergic Rhinitis in Asthma Guidelines (ARIA)
American Thoracic Society
American College of Physicians
European Respiratory Society
European Society of Thoracic Surgeons
British Medical Journal
Infectious Disease Society of America
American College of Chest Physicians
UpToDate®
National Institutes of Health and Clinical Excellence (NICE)
Scottish Intercollegiate Guideline Network (SIGN)
Cochrane Collaboration
Infectious Disease Society of America
Clinical Evidence
Agency for Health Care Research and Quality (AHRQ)
Partner of GIN
Over 60 major organizations
GRADE Uptake


















World Health Organization
Allergic Rhinitis in Asthma Guidelines (ARIA)
American Thoracic Society
American College of Physicians
European Respiratory Society
European Society of Thoracic Surgeons
British Medical Journal
Infectious Disease Society of America
American College of Chest Physicians
UpToDate®
National Institutes of Health and Clinical Excellence (NICE)
Scottish Intercollegiate Guideline Network (SIGN)
Cochrane Collaboration
Infectious Disease Society of America
Clinical Evidence
Agency for Health Care Research and Quality (AHRQ)
Partner of GIN
Over 60 major organizations
Recommendation 1a
• The panel recommends that people who had
household contact with smear positive or
M/XDR TB index cases be investigated for
active TB (strong recommendation, very low
quality evidence).
Recommendation 1b
• The panel recommends that people who had
household contact with TB index cases who
are children younger than 5 years of age be
investigated for active TB (strong
recommendation, very low quality evidence).