Finding What Works in Health Care: Standards for Systematic Revi…
Finding What Works in Health Care: Standards for Systematic Reviews
(2011)
Healthcare decision makers in search of reliable
information that compares health interventions
Paperback
| $61.00
Chapter: 4 Standards
for Buy
Synthesizing
the Body
of Evidence
×
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
Visit NAP.edu/10766 to get more information about this book,but
to buy it in print, or to download it as a free PDF.
Buy Ebook | $48.99 separate studies, and can help clarify what is
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
services. Systematic reviews can be helpful for
MyNAP members save
clinicians who want to integrate research findings into
10% online.
their daily practices, for patients to make well-informed
Login or Register to
choices about their own care, for professional medical
societies and other organizations that develop clinical
save!
practice guidelines.

4
Standards for Synthesizing the Body of
Download Free PDF Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
Evidence
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
therigor
qualitative
and quantitative
of the process.
Abstract: This chapter addresses
synthesis (meta-analysis) of the body of evidence. The committee recIn Finding What Works in Health Care the Institute of
ommends four related standards. The
systematic
review (SR)
should for
Medicine
(IOM) recommends
21 standards
developing high-quality
systematic
reviews
of
use prespecified methods; include a qualitative
synthesis
based
on escomparative effectiveness research. The standards
sential characteristics of study quality
(risk of bias, consistency, preaddress the entire systematic review process from the
initial
of formulating the
topic and
building the
cision, directness, reporting bias, and
forsteps
observational
studies,
dose–
review team to producing a detailed final report that
response association, plausible confounding that would change an
synthesizes what the evidence shows and where
observed effect, and strength of association);
andremain.
make an explicit
knowledge gaps
judgment of whether a meta-analysis is appropriate. If conducting
Finding What Works in Health Care also proposes a
meta-analyses, expert methodologists
shouldfordevelop,
framework
improvingexecute,
the quality and
of the science
systematic reviews. This book will serve
peer review the meta-analyses. The underpinning
meta-analyses
should address
as a vital resource for both sponsors and producers of
heterogeneity among study effects, accompany
all of
estimates
with
systematic reviews
comparative
effectiveness
research.
measures of statistical uncertainty, and assess the sensitivity of conclusions to changes in the protocol, assumptions, and study selection
(sensitivity analysis). An SR that uses rigorous and transparent methods will enable patients, clinicians, and other decision makers to discern what is known and not known about an intervention’s effectiveness and how the evidence applies to particular population groups
and clinical situations.
More than a century ago, Nobel prize-winning physicist J. W. Strutt Lord
Rayleigh
“the Care:
work which
deserves
…
Finding
Whatobserved
Works inthat
Health
Standards
for Systematic
Revi…
×
the most credit is that in which discovery
and explanation go hand in hand,
Healthcare decision makers in search of reliable
information
that compares
health
in which not only
facts presented,
but their
relation
to interventions
old ones is
Buynew
Paperback
| $61.00
 are
increasingly turn to systematic reviews for the best
pointed out” (Rayleigh, 1884). In other words,
the contribution of any singusummary of the evidence. Systematic reviews identify,
lar piece of research draws not only from
itsassess,
ownand
unique
discoveries,
but
select,
synthesize
the findings of
similar
but separate studies, and can help clarify what is
also from its relationship
to previous
research (Glasziou et al., 2004; Mul| $48.99
 Buy Ebook
known and not known about the potential benefits
row and Lohr, 2001). Thus, the synthesis
and
assessment
of and
a body
eviand
harms
of drugs, devices,
other of
healthcare
services.
Systematic
reviews can beeffectivehelpful for
dence is at the
heartmembers
of a systematic
review
(SR)
of comparative
MyNAP
save
clinicians who want to integrate research findings into
ness research10%
(CER).
online.
their daily practices, for patients to make well-informed
Login
or
Register
to
choices about theirchallenges
own care, for professional
The previous chapter described the considerable
involved medical
in
societies and other organizations that develop clinical
save!
assembling all the individual studies that
comprise current knowledge on
practice guidelines.
the effectiveness of a healthcare intervention: the “body of evidence.” This
Too often systematic reviews are of uncertain or poor
Free PDF that
chapter begins withDownload
the assumption
the body of evidence was identiquality. There are no universally accepted standards
developing
leading study
to variability
fied in an optimal manner and that the for
risk
of biassystematic
in eachreviews
individual
in how conflicts of interest and biases are handled,
was assessed appropriately—both according
to the committee’s standards.
how evidence is appraised, and the overall scientific
This chapter addresses the synthesis and
rigorassessment
of the process. of the collected evidence, focusing on those aspects that are
most salient to setting standards.
In Finding What Works in Health Care the Institute of
The science of SR is rapidly evolving; much
has
yetrecommends
to be learned.
The for
purMedicine
(IOM)
21 standards
developing high-quality systematic reviews of
pose of standards for evidence synthesis and assessment—as in other SR
comparative effectiveness research. The standards
methods—is to set performance expectations
and
tosystematic
promote
accountabiliaddress the
entire
review
process from the
initial stifling
steps of formulating
the topic
and building the
ty for meeting those expectations without
innovation
in methods.
review team to producing a detailed final report that
Thus, the emphasis is not on specifyingsynthesizes
preferred
technical
but
what
the evidencemethods,
shows and where
knowledge
gaps remain.
rather the building blocks that help ensure
objectivity,
transparency, and
scientific rigor.
Finding What Works in Health Care also proposes a
for improving
the quality
the science
As it did elsewhere in this report, theframework
committee
developed
thisof chapunderpinning systematic reviews. This book will serve
ter’s standards and elements of performance
based on available evidence
as a vital resource for both sponsors and producers of
systematic
reviews ofResearch
comparativeand
effectiveness
and expert guidance from the Agency for
Healthcare
Quality
research.
(AHRQ) Effective Health Care Program, the Centre for Reviews and Dissemination (CRD, part of University of York, UK), and the Cochrane Collaboration (Chou et al., 2010; CRD, 2009; Deeks et al., 2008; Fu et al., 2010;
Lefebvre et al., 2008; Owens et al., 2010). Guidance on assessing quality of
evidence from the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) Working Group was another key source of
information (Guyatt et al. 2010; Schünemann et al., 2009). See Appendix F
×
for a detailed summary of AHRQ, CRD, and Cochrane guidance for the assessment
synthesis
of aCare:
body of
evidence.for Systematic Revi…
Finding
What and
Works
in Health
Standards
The committee had several opportunities for learning the perspectives
of stakeholders on issues related to this chapter. SR experts and representatives from medical specialty associations,
payers,
and
consumer
groups
Healthcare
decision
makers
in search of
reliable
information
that compares
health and
interventions
provided both
written
to|the
committee’s
questions
oral
Buyresponses
Paperback
$61.00
increasingly turn to systematic reviews for the best
testimony in a public workshop (see Appendix
C).
summary of
the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Buy Ebook | $48.99
known and not known about the potential benefits
In addition, staff conducted informal, structured
interviews
with
other
key
and harms of drugs,
devices, and
other
healthcare
services. Systematic reviews can be helpful for
stakeholders.MyNAP members save
clinicians who want to integrate research findings into
The committee
recommends four standards
for theforassessment
and
10% online.
their daily practices,
patients to make
well-informed
Login
or
Register
to
choices
about
their
own
care,
for
professional
qualitative and quantitative synthesis of an SR’s body of evidence. Each medical
societies and other organizations that develop clinical
save! of two parts: first, a brief
standard consists
statement describing the relatpractice guidelines.

ed SR step and, second, one or more elements of performance that are
Too often systematic reviews are of uncertain or poor
Download
PDF Box
fundamental to carrying
out Free
the step.
4-1 lists all of the chapter’s recquality. There are no universally accepted standards
for developing
systematic reviews
leading
to variability
ommended standards. This chapter provides
the background
and
rationale
in how conflicts of interest and biases are handled,
for the recommended standards and elements
of performance, first outlinhow evidence is appraised, and the overall scientific
ing the key considerations in assessingrigor
a body
evidence, and followed by
of theof
process.
sections on the fundamental components
of qualitative and quantitative
In Finding What Works in Health Care the Institute of
synthesis. The order of the chapter’s standards
and
the presentation
Medicine (IOM)
recommends
21 standardsof
forthe
developing high-quality systematic reviews of
discussion do not necessarily indicate the sequence in which the various
comparative effectiveness research. The standards
steps should be conducted. Although an
SR synthesis
shouldreview
always
in- from the
address
the entire systematic
process
initial steps
the topic
and building the
clude a qualitative component, the feasibility
ofofaformulating
quantitative
synthesis
review team to producing a detailed final report that
(meta-analysis) depends on the available
data. Ifwhat
a metaanalysis
is conductsynthesizes
the evidence shows
and where
knowledge
remain. synthesis. Moreed, its interpretation should be included
in the gaps
qualitative
over, the overall assessment of the body
of evidence
be also
done
untila
Finding
What Works cannot
in Health Care
proposes
framework for improving the quality of the science
the syntheses are complete.
underpinning systematic reviews. This book will serve
In the context of CER, SRs are produced
to help consumers, clinicians,
as a vital resource for both sponsors and producers of
systematic
reviews of and
comparative
developers of clinical practice guidelines,
purchasers,
policyeffectiveness
makers to
research.
make informed healthcare decisions (Federal Coordinating Council for
Comparative Effectiveness Research, 2009; IOM, 2009). Thus, the assessment and synthesis of a body of evidence in the SR should be approached
with the decision makers in mind. An SR using rigorous and transparent
methods allows decision makers to discern what is known and not known
about an intervention’s effectiveness and how the evidence applies to particular population groups and clinical situations (Helfand, 2005). Making
evidence-based decisions—such as when a guideline developer recommends
what
should
should
notStandards
be done infor
specific
clinicalRevi…
circumFinding
What
Works
in and
Health
Care:
Systematic
stances—is a distinct and separate process from the SR and is outside the
scope of this report. It is the focus of a companion IOM study on develop-
×
1 in search of reliable
Healthcare guidelines.
decision makers
ing standards for trustworthy clinical practice
information that compares health interventions
 Buy Paperback | $61.00
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Buy Ebook | $48.99
and notof
known
about the
potential benefits
The SR field lacks an agreed-on lexiconknown
for some
its most
fundamental
and harms of drugs, devices, and other healthcare
terms and concepts, including what actually
services.constitutes
Systematic reviews can be helpful for
MyNAP members save
clinicians who want to integrate research findings into
1 The IOM report,
10% online.
Clinical Practice Guidelines
We Can
, is
their daily practices,
forTrust
patients
to available
make well-informed
Login
or
Register
to
choices
about
their
own
care,
for
professional
medical
at the National Academies Press website: http://www.nap.edu/.
societies and other organizations that develop clinical
save!
practice guidelines.
A NOTE ON TERMINOLOGY

Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
BOX 4-1
how evidence is appraised, and the overall scientific
for Synthesizing
the Body of Evidence
rigor of the process.
Download Free PDF
Recommended Standards
In Findingto
What
Works in Health
theof
Institute of
Standard 4.1 Use a prespecified method
evaluate
the Care
body
Medicine (IOM) recommends 21 standards for
evidence
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
Required elements:
initial steps of formulating the topic and building the
review team to producing a detailed final report that
4.1.1 For each outcome, systematically
assess the following
synthesizes what the evidence shows and where
gaps remain.
characteristics of the bodyknowledge
of evidence:
Risk of bias
Consistency
Precision
Directness
Reporting bias
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
research.
4.1.2 For bodies of evidence that include observational research, also systematically assess the following characteristics for each outcome:
Dose–response association
Plausible confounding that would change the observed
effect
Finding What Works in
Health
Care: Standards for Systematic Revi…
Strength of association
×
4.1.3 For each outcome specified
in the protocol, use conHealthcare decision makers in search of reliable
sistent
language
to characterize
thethat
level
of confiinformation
compares
health interventions
Paperback
| $61.00
 Buy
increasingly turn to systematic reviews for the best
dence in the estimates of the effect of an intervention
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
Standard 4.2 Conduct
a
qualitative
synthesis
but
separate studies, and can help clarify what is
Buy Ebook | $48.99
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
Required elements:
services. Systematic reviews can be helpful for
MyNAP members save
clinicians who want to integrate research findings into
10% online.
4.2.1 Describe
the clinical and methodological
characteristheir daily practices, for
patients to make well-informed
Login
or
Register
to
choices
about
their
own
care, for
professional medical
tics of the included studies, including their size,
inclusocieties and other organizations that develop clinical
save!
sion or exclusion of important
subgroups,
timeliness,
practice
guidelines.

and other relevant factors
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts
of interest andconfusion.
biases are handled,
the quality of a body of evidence. This leads
to considerable
Behow evidence is appraised, and the overall scientific
cause this report focuses on SRs for therigor
purposes
of CER and clinical deciof the process.
Download Free PDF
sion making, the committee uses the term “quality of the body of evidence”
In Finding What Works in Health Care the Institute of
to describe the extent to which one canMedicine
be confident
that the estimate of
(IOM) recommends 21 standards for
an intervention’s effectiveness is correct.
This terminology
is designed
developing
high-quality systematic
reviews of to
The standards
support clinical decision making and iscomparative
similar toeffectiveness
that usedresearch.
by GRADE
and
address the entire systematic review process from the
adopted by the Cochrane Collaborationinitial
andsteps
other
organizations
forbuilding
the the
of formulating
the topic and
review team to
a detailed
final report that
same purpose (Guyatt et al., 2010; Schünemann
etproducing
al., 2008,
2009).
synthesizes what the evidence shows and where
Quality encompasses summary assessments
of aremain.
number of characterisknowledge gaps
tics of a body of evidence, such as within-study bias (methodological qualiFinding What Works in Health Care also proposes a
ty), consistency, precision, directness or
applicability
of the
evidence,
and
framework
for improving
the quality
of the science
others (Schünemann et al., 2009). Syn- underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
research.
4.2.2 Describe the strengths and limitations of individual
studies and patterns across studies
4.2.3 Describe, in plain terms, how flaws in the design or
execution of the study (or groups of studies) could
bias the results, explaining the reasoning behind
theseinjudgments
Finding What Works
Health Care: Standards for Systematic Revi…
4.2.4 Describe the relationships between the characteristics of the individual studies and their reported findHealthcare decision makers in search of reliable
ings and patterns across studies
information that compares health interventions
| $61.00
 BuythePaperback
4.2.5 Discuss
relevance of
individual studies to the pop-
×
increasingly turn to systematic reviews for the best
summary of the evidence.
Systematic
ulations, comparisons, cointerventions,
settings,
andreviews identify,
select, assess, and synthesize the findings of similar
outcomes or measures of but
interest
Buy Ebook | $48.99 separate studies, and can help clarify what is
known and not known about the potential benefits
of drugs, devices,
and other
healthcare
Standard 4.3 Decide if, in additionand
toharms
a qualitative
analysis,
the
services. Systematic reviews can be helpful for
MyNAP
members
save
systematic
review
will include
a quantitative
analysis (metaclinicians who want to integrate research findings into
analysis)10% online.
their daily practices, for patients to make well-informed
Login or Register to
choices about their own care, for professional medical
societies and other organizations that develop clinical
save!element:
Required
practice guidelines.

Too often
systematic
reviews to
are deof uncertain or poor
4.3.1 Explain
why a pooled
estimate
might
be useful
Download
Free PDF
quality. There are no universally accepted standards
cision makers
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
Standard 4.4 If conducting a meta-analysis,
do the
how evidence isthen
appraised,
and the overall scientific
rigor of the process.
following:
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
Use expert methodologists
to develop, execute, and
address the entire systematic review process from the
peer review the meta-analyses
initial steps of formulating the topic and building the
review team to producing a detailed final report that
Address the heterogeneitysynthesizes
among what
study
effects
the evidence shows and where
knowledge
gaps remain.
Accompany all estimates with
measures
of statistical
Required elements:
4.2.1
4.2.2
4.2.3
uncertainty
4.2.4
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
Assess the sensitivity of conclusions
to changes in the
underpinning systematic reviews. This book will serve
protocol, assumptions, and
selection
(sensitivity
as study
a vital resource
for both
sponsors and producers of
systematic reviews of comparative effectiveness
analysis)
research.
NOTE: The order of the standards does not indicate the sequence
in which they are carried out.
thesis is the collation, combination, and summary of the findings of a body
of evidence (CRD, 2009). In an SR, the synthesis of the body of evidence
should always include a qualitative component and, if the data permit, a
quantitative synthesis (meta-analysis).
The following section presents the background and rationale for the
committee’s
recommended
standard
and performance
elements
for preFinding
What Works
in Health Care:
Standards
for Systematic
Revi…
specifying the assessment methods.
×
decision makers in search of reliable
A Need for Clarity Healthcare
and Consistency
information that compares health interventions
Buy Paperback | $61.00
 evidence
increasingly
turnexperts
to systematic
reviews for the
Neither empirical
nor agreement
among
is available
tobest
summary of the evidence. Systematic reviews identify,
support the committee’s endorsement of a specific approach for assessing
select, assess, and synthesize the findings of similar
and describing the quality of a body of but
 Buy Ebook | $48.99evi-separate studies, and can help clarify what is
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
services. Systematic reviews can be helpful for
MyNAP members save
who want
to integrate
research findings into
dence. Medical specialty societies, U.S.clinicians
and other
national
government
10% online.
their daily practices, for patients to make well-informed
agencies, private
research groups, and others have created a multitude of
Login or Register to
choices about their own care, for professional medical
systems for assessing
and
characterizing
the quality
a body of
societies
and other of
organizations
thatevidence
develop clinical
save!
practice
guidelines.
(AAN, 2004; ACCF/AHA, 2009; ACCP, 2009; CEBM, 2009; Chalmers et al.,
1990; Ebell et al., 2004;
Faraday
al., 2009;
Guirguis-Blake
2007; or poor
Too often
systematic reviewset
areal.,
of uncertain
Download
FreeetPDF
quality. There
are no
universally
accepted
standards
Guyatt et al., 2004; ICSI, 2003; NCCN, 2008;
NZGG,
2007;
Owens
et al.,
for developing systematic reviews leading to variability
2010; Schünemann et al., 2009; SIGN, 2009;
USPSTF,
2008).
The
various
in how conflicts
of interest
and
biases
are handled,
how
evidence
is
appraised,
and
the
overall
scientific
systems share common features, but employ conflicting evidence hierarrigor of the process.
chies; emphasize different factors in assessing the quality of research; and
Finding
What Works
Health Care
the Institute of
use a confusing array of letters, codes, Inand
symbols
to inconvey
investigaMedicine (IOM) recommends 21 standards for
tors’ conclusions about the overall quality
of a body
of evidence
(Atkins
developing
high-quality
systematic reviews
of et
comparative
research.
standards
al., 2004a, 2004b; Schünemann et al., 2003;
Westeffectiveness
et al., 2002).
TheThereader
address the entire systematic review process from the
cannot make sense of the differences (Table
4-1). Through public testimony
initial steps of formulating the topic and building the
review
team to producing
a detailedand
final users
report that
and interviews, the committee heard that
numerous
producers
synthesizes what the evidence shows and where
of SRs were frustrated by the number, variation, complexity, and lack of
knowledge gaps remain.
transparency in existing systems.
Finding What Works in Health Care also proposes a
One comprehensive review documented
40 different systems for grading
framework for improving the quality of the science
the strength of a body of evidence (West
et al., 2002).
Another
underpinning
systematic
reviews.review,
This book conwill serve
as
a
vital
resource
for
both
sponsors
and
producers
ducted several years later, found that more than 50 evidence-grading sys- of
systematic reviews of comparative effectiveness
tems and 230 quality assessment instruments
research. were in use (COMPUS,
2005).
Early systems for evaluating the quality of a body of evidence used simple hierarchies of study design to judge the internal validity (risk of bias) of
a body of evidence (Guyatt et al., 1995). For example, a body of evidence
that included two or more randomized controlled trials (RCTs) was assumed to be “high-quality,” “level 1,” or “grade A” evidence whether or not
the trials met scientific standards. Quasi-experimental research, observa-
×
tional studies, case series, and other qualitative research designs were automatically
considered
lower
quality
evidence.for
AsSystematic
research documented
Finding
What Works
in Health
Care:
Standards
Revi…
the variable quality of trials and widespread reporting bias in the publication of trial findings, it became clear that such hierarchies are too simplistic because they do not assess the extent
to which
themakers
design
and ofimpleHealthcare
decision
in search
reliable
information
that
compares
health
interventions
Paperback
| $61.00
mentation of RCTs
(or other
study designs)
avoid
biases
that
may
reduce
 Buy
increasingly turn to systematic reviews for the best
confidence in the measures of effectiveness
(Atkins
et al., Systematic
2004b; Coleman
summary
of the evidence.
reviews identify,
select, assess, and synthesize the findings of similar
et al., 2009; Harris et al., 2001).
but separate studies, and can help clarify what is
| $48.99
The early hierarchies
produced
conflicting
conclusions about effective Buy Ebook
known and not known about the potential benefits
harms of drugs,
other healthcare
ness. A study by Ferreira and colleaguesandanalyzed
the devices,
effect and
of applying
difservices. Systematic reviews can be helpful for
members
save to the conclusions of six Cochrane SRs
ferent “levelsMyNAP
of evidence”
systems
clinicians who want to integrate research findings into
10%for
online.
of interventions
low back pain (Ferreira
et al.,
2002).
They found
their daily
practices,
for patients
to make that
well-informed
Login or Register to
choices about their own care, for professional medical
the conclusions of the reviews were highly dependent on the system used
societies and other organizations that develop clinical
save!
to evaluate the evidence
practice guidelines.
Download Free PDF
TABLE 4-1 Examples of Approaches to
Therapeutic Interventions*
System
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts
of interest
biases are handled,
Assessing
the
Body and
of Evidence
for
how evidence is appraised, and the overall scientific
rigor of the process.
System for Assessing the Body of Evidence
In Finding What Works in Health Care the Institute of
Agency for Healthcare High High confidence
that
the evidence
reflects
Medicine (IOM)
recommends
21 standards
for
developing
high-quality
systematic
reviews
of
Research and Quality
the true effect. Further research is very uncomparative effectiveness research. The standards
likely to change
confidence
of the
estiaddress theour
entire
systematic review
process
from the
initial steps of formulating the topic and building the
mate of effect.
review team to producing a detailed final report that
Mod- Moderate
confidence
that
the evidence
resynthesizes
what the
evidence
shows and where
gaps remain.
erate flects theknowledge
true effect.
Further research may
change our
confidence
theCare
estimate
of efFinding
What Works inin
Health
also proposes
a
framework
for
improving
the
quality
of
the
science
fect and may change the estimate.
Low
underpinning systematic reviews. This book will serve
Low confidence
that the
evidence
as a vital resource
for both
sponsorsreflects
and producers of
reviews of research
comparative is
effectiveness
the true systematic
effect. Further
likely to
research.
change the confidence in the estimate of effect and is likely to change the estimate.
InEvidence either is unavailable or does not
suffi- permit a conclusion.
cient
American College of
Chest Phsicians
High Randomized controlled trials (RCTs) without
important limitations or overwhelming evi-
dence from observational studies.
Finding What Works in Health
Care:
Standards
for Systematic
Revi…
ModRCTs
with important
limitations
(inconsiserate tent results, methodological flaws, indirect,
or imprecise) or exceptionally strong eviHealthcare
decision makers
in search of reliable
dence from
observational
studies.
information that compares health interventions
Paperback
| $61.00
 Buy Low
Observational
studies or case series.
×
increasingly turn to systematic reviews for the best
of the evidence. Systematic reviews identify,
American Heart Asso- A
Multiple summary
RCTs or
meta-analyses.
select, assess, and synthesize the findings of similar
ciation/American
but separate
studies, and can help
clarify what is
RCT,
or nonrandomized
studies.
Buy B
Ebook Single
| $48.99
known
and
not
known
about
the
potential
benefits
College of Cardiology C
Consensus
opinion
of
experts,
case
studies,
and harms of drugs, devices, and other healthcare
services.
Systematic reviews can be helpful for
or standard
of care.
MyNAP members save
clinicians who want to integrate research findings into
10% online. Starting points for
Grading of Recomlevel:
theirevaluating
daily practices, quality
for patients
to make well-informed
Login
or
Register
to
choices
about
their
own
care,
for
professional medical
mendations Assesssave!
RCTs startsocieties
high. and other organizations that develop clinical
ment, Development
practice guidelines.

and Evaluation
(GRADE)
Observational studies start low.
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
Factors that mayfordecrease
increase
theleading
quality
developing or
systematic
reviews
to variability
how conflicts of interest and biases are handled,
level of a body ofinevidence:
how evidence is appraised, and the overall scientific
rigor of the process.
Download Free PDF
Decrease: Study limitations, inconsistency of
In Finding Whatof
Works
in Health Care
the Institute of
results, indirectness
evidence,
imprecision
Medicine (IOM) recommends 21 standards for
of results, developing
and highhigh-quality
risk of publication
bias.
systematic reviews
of
comparative
effectiveness
The standards
Increase: Large
magnitude
of research.
effect, dose–readdress the entire systematic review process from the
sponse gradient,
allofplausible
biases
would
re-the
initial steps
formulating the
topic and
building
review teameffect.
to producing a detailed final report that
duce the observed
synthesizes what the evidence shows and where
knowledge gaps remain.
System
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
System for Assessing the Body
of Evidence
systematic reviews of comparative effectiveness
research.
High Further research is very
unlikely to change our confi-
dence in the estimate of effect.
Mod- Further research is likely to have an important impact
erate on our confidence in the estimate of effect and may
change the estimate.
Low
Further research is very likely to have an important impact on our confidence in the estimate of effect and is
likely to change the estimate.
Finding What Works
Health
Care:
foruncertain.
Systematic Revi…
Very in
Any
estimate
ofStandards
effect is very
low
×
National High High-powered RCTs or meta-analysis.
Healthcare decision makers in search of reliable
Comprecompares health interventions
Buy Paperback
| $61.00
Lower
from Phase
IIinformation
Trials tothat
large
cohort studies to
 Ranges
hensive
increasingly turn to systematic reviews for the best
case series to individual
practitioner
experience.
summary
of the evidence.
Systematic reviews identify,
Cancer
select, assess, and synthesize the findings of similar
Network
but separate studies, and can help clarify what is
Buy Ebook | $48.99

known
andmay
not known
about thedown
potential
benefits
Oxford
Varies with type of question.
Level
be graded
on
the
and harms of drugs, devices, and other healthcare
Centre for basis of study quality, imprecision,
indirectness,
inconsistency
services. Systematic
reviews can
be helpful for
MyNAP members save
who want to
integrate
research
findings into
Evidence- between studies, or becauseclinicians
the absolute
effect
size
is very
10% online.
their daily practices, for patients to make well-informed
Based
small.
may be
if there is a large or very large
LoginLevel
or Register
to graded up
choices about their own care, for professional medical
societies and other organizations that develop clinical
Medicine effect
save! size.
practice guidelines.
Level Systematic review (SR) of randomized trials or n-of-1
1
trial.
Download Free PDF Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
For rare harms: SR offorcase-control
studies,
orleading
studies
redeveloping systematic
reviews
to variability
in how conflicts of interest and biases are handled,
vealing dramatic effects.
how evidence is appraised, and the overall scientific
Level SR of nested case-control
or process.
dramatic effect.
rigor of the
2
For rare harms: Randomized trial or (exceptionally) obIn Finding What Works in Health Care the Institute of
servational study with
dramatic
effect. 21 standards for
Medicine
(IOM) recommends
developing high-quality systematic reviews of
Level Nonrandomized controlled
cohort/follow-up study.
comparative effectiveness research. The standards
3
address the entire systematic review process from the
initial steps of formulating the topic and building the
Level Case-control studies,review
historically
controlled studies.
team to producing a detailed final report that
synthesizes what the evidence shows and where
4
knowledge gaps remain.
Level Opinion without explicit critical appraisal, based on limWhat Worksor
in Health
5
ited/undocumentedFinding
experience,
basedCare
onalso proposes a
framework for improving the quality of the science
mechanisms.
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
research.
Scottish In- 1++ High-quality meta-analyses, SRs of RCTs, or RCTs with a
tercollegiate
very low risk of bias.
Guidelines 1+ Well-conducted meta-analyses, SRs, or RCTs with a low
Network
risk of bias.
1− Meta-analyses, SRs, or RCTs with a high risk of bias.
×
2++ High-quality SRs of case control or cohort studies. Highquality
case
control
or cohort
with Revi…
a very low
Finding What Works in
Health
Care:
Standards
forstudies
Systematic
risk of confounding or bias and a high probability that
the relationship is causal.
Healthcare
decision
in search
of reliable
2− Case control or cohort
studies
withmakers
a high
risk of
coninformation that compares health interventions
Buy Paperback
| $61.00
or bias and
a significant risk that the relation founding
increasingly turn to systematic reviews for the best
ship is not causal. summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
3
Nonanalytic studies, but
e.g., case reports, case series.
Buy Ebook | $48.99 separate studies, and can help clarify what is

known and not known about the potential benefits
4 Expert opinion.
and harms of drugs, devices, and other healthcare
* Some systems use different grading schemes
depending
typeforof
services. Systematic
reviewson
can the
be helpful
MyNAP members save
clinicians who want to integrate research findings into
intervention 10%
(e.g.,online.
preventive service, diagnostic
tests, and therapies). This
their daily practices, for patients to make well-informed
table includes
systems
for therapeutic
interventions.
Login
or Register
to
choices
about their own care, for professional medical
societiesCEBM
and other
organizations
that(2008);
develop clinical
save!
SOURCES: ACCF/AHA
(2009); ACCP (2009);
(2009);
NCCN
practice guidelines.
Owens et al. (2010); Schünemann et al. (2009); SIGN (2009).
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
the how
number
and
qualityand
of the
trials
reevidence
is appraised,
overall
scientific
rigor of the process.
Download Free PDF
primarily because of differences in
quired for a particular level of evidence. In many cases, the differences in
the conclusions were so substantial that
they could
lead
to contradictory
In Finding
What Works
in Health
Care the Institute of
Medicine (IOM) recommends 21 standards for
clinical advice. For example, for one intervention,
“back school,”2 the condeveloping high-quality systematic reviews of
research.
The standards
clusions ranged from “strong evidence comparative
that backeffectiveness
schools are
effective”
to
address the entire systematic review process from the
“no evidence” on the effectiveness of back
schools.
initial steps of formulating the topic and building the
One reason for these discrepancies was
failure
distinguish
review
team to to
producing
a detailedbetween
final report that
synthesizes what the evidence shows and where
the quality of the evidence and the magnitude of net benefit. For example,
knowledge gaps remain.
an SR and meta-analysis might highlight a dramatic effect size regardless
Finding What Works in Health Care also proposes a
of the risk of bias in the body of evidence.
Conversely, use of a rigid hierarframework for improving the quality of the science
chy gave the impression that any effectunderpinning
based on systematic
randomized
evidence
reviews.trial
This book
will serve
as
a
vital
resource
for
both
sponsors
and
producers
of
was clinically important, regardless of the size of the effect. In 2001, the
systematic reviews of comparative effectiveness
U.S. Preventive Services Task Force broke
new ground when it updated its
research.
review methods, separating its assessment of the quality of evidence from
its assessment of the magnitude of effect (Harris et al., 2001).
What Are the Characteristics of Quality for a Body
of Evidence?
×
Experts in SR methodology agree on the conceptual underpinnings for the
systematic
assessment
of a body
evidence. for
TheSystematic
committee Revi…
identified
Finding
What Works
in Health
Care:ofStandards
eight basic characteristics of quality, described below, that are integral to
assessing and characterizing the quality of a body of evidence. These characteristics—risk of bias, consistency, precision,
directness,
reporting
Healthcare decision
makers and
in search
of reliable
information that compares
health interventions
bias, and for observational
studies, dose–response
association,
plausible
| $61.00
 Buy Paperback
increasingly turn to systematic reviews for the best
confounding that would change an observed
and strength
associsummaryeffect,
of the evidence.
Systematic of
reviews
identify,
select,
assess, and synthesize
the findings
of similar
ation—are used by GRADE; the Cochrane
Collaboration,
which
has adopted
but separate studies, and can help clarify what is
Ebook
| $48.99
the GRADE approach;
the AHRQ
Effective
Health Care Program, which
 Buyand
known and not known about the potential benefits
and harms
of drugs,
devices, and
healthcare
adopted a modified version of the GRADE
approach
(Owens
et other
al., 2010;
services. Systematic reviews can be helpful for
Balshem et al.,
2011; members
Falck-Ytter
et al., 2010;
Schünemann et al., 2008). AlMyNAP
save
clinicians who want to integrate research findings into
online. varies somewhat,
though their 10%
terminology
Falck-Ytter
histoGRADE
coltheir daily
practices, for and
patients
make well-informed
Login
or
Register
to
choices about their own care, for professional medical
leagues describe any differences between the GRADE and AHRQ quality
societies and other organizations that develop clinical
save!
characteristics as essentially semantic (Falck-Ytter
et al., 2010). Owens and
practice guidelines.
his AHRQ colleagues appear
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
2 Back schools are educational programs
designed to teach patients how
for developing systematic reviews leading to variability
to manage chronic low back pain to prevent
future
episodes.
The
in how conflicts
of interest
and biases
arecurhandled,
how
evidence
is
appraised,
and
the
overall
scientific
riculums typically include the natural history, anatomy, and physiology
rigor of the process.
Download Free PDF
of back pain as well as a home exercise program (Hsieh et al., 2002).
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
initial steps of formulating the topic and building the
BOX 4-2
review team to producing a detailed final report that
synthesizes
what the
showsthe
and where
GRADE
Approach
toevidence
Assessing
knowledge gaps remain.
Key Concepts Used in the
Quality of a Body of Evidence
Finding What Works in Health Care also proposes a
framework
for improving
the quality of the science
The Grading of Recommendations
Assessment,
Development,
underpinning systematic reviews. This book will serve
and Evaluation (GRADE) Working Group
uses a point system to
as a vital resource for both sponsors and producers of
of comparative
effectiveness
upgrade or downgrade the ratings systematic
for eachreviews
quality
characteristic.
research.
A grade of high, moderate, low, or very low is assigned to the body
of evidence for each outcome. Eight characteristics of the quality
of evidence are assessed for each outcome.
Five characteristics can lower the quality rating for the body of
evidence:
Limitations in study design and conduct
×
Inconsistent results across studies
of evidence
with respect
the study design,
Finding WhatIndirectness
Works in Health
Care: Standards
fortoSystematic
Revi…
populations, interventions, comparisons, or outcomes
Imprecision of the estimates of effect
Publication bias
Healthcare decision makers in search of reliable
information that compares health interventions
Buy Paperback | $61.00

increasingly
turn to
systematic
reviews
the best
Three factors can increase the quality
rating
for
the body
of for
evisummary of the evidence. Systematic reviews identify,
dence because they raise confidence
inassess,
the certainty
of the
estimates
select,
and synthesize
findings of similar
but separate studies, and can help clarify what is
(particularly for
observational
studies):
Buy Ebook | $48.99

Large magnitude of effect
MyNAP members save
Plausible
confounding that
10% online.
effect
Login or Register to
Dose–response
gradient
save!
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
services. Systematic reviews can be helpful for
clinicians
who want
to integrate
research findings into
would
reduce
the
demonstrated
their daily practices, for patients to make well-informed
choices about their own care, for professional medical
societies and other organizations that develop clinical
practice guidelines.
SOURCES: Atkins et al. (2004a); Balshem et al. (2011); Falck-Ytter et
Download Free PDF Too often systematic reviews are of uncertain or poor
al. (2010); Schünemann et al. (2009).
quality. There are no universally accepted standards
to agree (Owens et al., 2010). As Boxes
proaches are quite similar.3
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
4-2
and 4-3 indicate, the two aphow evidence is appraised, and the overall scientific
rigor of the process.
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address
the entire
systematic
process
In the context of a body of evidence, risk
of bias
refers
to thereview
extent
to from the
initial steps of formulating the topic and building the
which flaws in the design and execution
of a collection of studies could
review team to producing a detailed final report that
bias the estimate of effect for each outcome
under
synthesizes
what study.
the evidence shows and where
knowledge gaps remain.
Risk of Bias
3 For detailed descriptions of the AHRQ and GRADE methods, see the
Finding What Works in Health Care also proposes a
GRADE Handbook for Grading Quality
of Evidence and Strength of Recframework for improving the quality of the science
ommendations (Schünemann et al., 2009)
and “Grading
the Strength
ofserve
underpinning
systematic reviews.
This book will
a vital resource
for both sponsors and producers of
a Body of Evidence When ComparingasMedical
Interventions—AHRQ
systematic reviews of comparative effectiveness
and the Effective Health Care Program”
(Owens et al., 2010).
research.
BOX 4-3
Key Concepts Used in the AHRQ Approach to Assessing the
Quality of a Body of Evidence
×
The Agency for Healthcare Research and Quality (AHRQ) Effective Health
Program
refers
to the evidence
evaluation
Finding What
WorksCare
in Health
Care:
Standards
for Systematic
Revi…
process as grading the “strength” of a body of evidence. It requires
that the body of evidence for each major outcome and comparison
of interest be assessed according to
the concepts
listedin below.
Healthcare
decision makers
search of Afreliable
information
that compares
health review
interventions
Buy Paperback
| $61.00
ter a global
of the concepts,
AHRQ
systematic
assessment
increasingly turn to systematic reviews for the best
teams assign a grade of high, moderate,
low,
orevidence.
insufficient
toreviews
the identify,
summary
of the
Systematic
select, assess, and synthesize the findings of similar
body of evidence for each outcome.
separate studies, and can help clarify what is
 Buy Ebook | $48.99but
known and not known about the potential benefits
Evaluation components in all systematic
reviews:
and harms of
drugs, devices, and other healthcare
services. Systematic reviews can be helpful for
MyNAP members save
who
to integrate research findings into
Risk10%
of bias
in the design and clinicians
conduct
ofwant
studies
online.
their daily practices, for patients to make well-informed
Consistency
in the to
estimates of
effect
across
studies
Login or Register
choices
about
their own
care, for professional medical
other organizations
develop clinical
Directness
of the evidence in societies
linkingand
interventions
to that
health
save!
practice guidelines.
outcomes
often systematic
reviews are
uncertain or poor
Precision Download
or degreeFree
of certainty
an estimate
of of
effect
PDF Too about
quality. There are no universally accepted standards
for an outcome
for developing systematic reviews leading to variability
in how
conflictscontexts
of interest and
Applicability of the evidence to
specific
andbiases are handled,
how evidence is appraised, and the overall scientific
populations
rigor of the process.
In Finding
Works to
in Health
Care the Institute of
Other considerations (particularly
with What
respect
observational
Medicine (IOM) recommends 21 standards for
studies):
developing high-quality systematic reviews of
comparative effectiveness research. The standards
Dose–response association address the entire systematic review process from the
initial steps of formulating the topic and building the
Publication bias
review team to producing a detailed final report that
Presence of confounders thatsynthesizes
would diminish
an observed
what the evidence
shows and where
knowledge gaps remain.
effect
Strength of association (magnitude
of effect)
Finding What
Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
SOURCE: Owens et al. (2010).
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
Chapter 3 describes the factors relatedresearch.
to the design and conduct of ran-
domized trials and observational studies that may influence the magnitude
and direction of bias for a particular outcome (e.g., sequence generation,
allocation concealment, blinding, incomplete data, selective reporting of
outcomes, confounding, etc.),4 as well as
4 Sequence generation refers to the method used to generate the ran-
dom assignment of study participants in a trial. A trial is “blind” if par-
ticipants are not told to which arm of the trial they have been asconcealment
is a methodfor
used
to preventRevi…
selection
Findingsigned.
What Allocation
Works in Health
Care: Standards
Systematic
bias in clinical trials by concealing the allocation sequence from those
assigning participants to intervention groups. Allocation concealment
prevents researchers from (unconsciously
ordecision
otherwise)
Healthcare
makers influencing
in search of reliable
information
that compares
health interventions
Paperback
| $61.00
the intervention
group
to which each
participant
is assigned.
 Buy
×
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Buy Ebook
$48.99
available tools for assessing
risk| of
biasknown
in individual
studies.
Assessing
risk
and not known
about the
potential benefits
harms of drugs, devices, and other healthcare
of bias for a body of evidence requires aand
cumulative
assessment of the risk
services. Systematic reviews can be helpful for
members
save for each specific outcome of interest.
of bias acrossMyNAP
all individual
studies
clinicians who want to integrate research findings into
10% online.
daily
practices, for
patients to
Study biases are outcome dependent intheir
that
potential
sources
ofmake
biaswell-informed
imLogin or Register to
choices about their own care, for professional medical
pact differentsave!
outcomes in different ways;
forand
example,
blindingthat
ofdevelop
out- clinical
societies
other organizations
come assessment to a treatment grouppractice
mightguidelines.
be less important for a study

of the effect of an intervention on mortality
than for a study measuring
Download Free PDF Too often systematic reviews are of uncertain or poor
There are no estimate
universally accepted
standards
pain relief. The degree of confidence inquality.
the summary
of effect
will
for developing systematic reviews leading to variability
depend on the extent to which specificinbiases
in the included studies affect
how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
a specific outcome.
rigor of the process.
In Finding What Works in Health Care the Institute of
Consistency
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
For the appraisal of a body of evidence,comparative
consistency
refersresearch.
to the The
degree
of
effectiveness
standards
address the entire systematic review process from the
similarity in the direction and estimated size of an intervention’s effect on
initial steps of formulating the topic and building the
review can
team provide
to producing
a detailed
report that
specific outcomes.5 SRs and meta-analyses
clear
andfinal
convincsynthesizes what the evidence shows and where
ing evidence of a treatment’s effect when the individual studies in the body
knowledge gaps remain.
of evidence show consistent, clinically important effects of similar magniFinding What Works in Health Care also proposes a
tude (Higgins et al., 2003). Often, however,
the results differ in the included
framework for improving the quality of the science
studies. Large and unexplained differences
(inconsistency)
are of
underpinning
systematic reviews.
Thisconcern
book will serve
as
a
vital
resource
for
both
sponsors
and
producers of
especially when some studies suggest substantial benefit, but other studies
systematic reviews of comparative effectiveness
indicate no effect or possible harm (Guyatt
et al., 2010).
research.
However, inconsistency across studies may be due to true differences in
a treatment’s effect related to variability in the included studies’ populations (e.g., differences in health status), interventions (e.g., differences in
drug doses, cointerventions, or comparison interventions), and health outcomes (e.g., diminishing treatment effect with time). Examples of inconsistency in a body of evidence include statistically significant effects in opposite directions, confidence intervals that are wide or fail to overlap, and
×
clinical or statistical heterogeneity that cannot be explained. When differences
in estimates
true for
differences
in aRevi…
treatment’s
Finding
What
Works inacross
Healthstudies
Care: reflect
Standards
Systematic
effect, then inconsistency provides the opportunity to understand and
characterize those differences, which may have important implications for
clinical practice. If the inconsistency results
from
biases
in instudy
Healthcare
decision
makers
searchdesign
of reliableor
information
that compares
Buy Paperback
| $61.00
improper study
then a thorough
assessment
of health
theseinterventions
differexecution,
increasingly turn to systematic reviews for the best
ences may inform future study design. summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
5 In analyses involving indirect comparisons,
network
meta-analyses,
oris
but separate
studies, and
can help clarify what
 Buy Ebook | $48.99knownconsistency
and not known about
the to
potential
benefits
mixed-treatment comparisons, the term
refers
the deand harms of drugs, devices, and other healthcare
gree to which the direct comparisonsservices.
(head-to-head
comparisons)
and
Systematic reviews
can be helpful for
MyNAP members save
who want
to integrate
research
findings into
the indirect comparisons agree with clinicians
each other
with
respect
to the
10% online.
their daily practices, for patients to make well-informed
magnitudeLogin
of the
interest.
ortreatment
Register to effect ofchoices
about their own care, for professional medical
save!
societies and other organizations that develop clinical
practice guidelines.
Download Free PDF
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
errors
inprocess.
the estimates of effect,
rigor of the
Precision
A measure of the likelihood of random
precision refers to the degree of certainty about the estimates for specific
In Finding What Works in Health Care the Institute of
outcomes. Confidence intervals about the
estimate
of effect21
from
eachfor
Medicine
(IOM) recommends
standards
developing
systematic
reviews of instudy are one way of expressing precision,
withhigh-quality
a narrower
confidence
comparative effectiveness research. The standards
terval meaning more precision.
address the entire systematic review process from the
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
Directness
The concept of directness has two dimensions, depending on the context:
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
When interventions are compared,underpinning
directness
refersreviews.
to theThis
extent
toserve
systematic
book will
a vital resource
for both sponsors
andbeproducers of
which the individual studies were as
designed
to address
the link
systematic reviews of comparative effectiveness
tween the healthcare interventionresearch.
and a specific health outcome. A
body of evidence is considered indirect if the included studies only
address surrogate or biological outcomes or if head-to-head (direct)
comparisons of interventions are not available (e.g., intervention A is
compared to intervention C, and intervention B is compared to C,
when comparisons of A vs. B studies are of primary interest, but not
available).
×
The other dimension of “directness” is applicability (also re­ferred to as
6 A body
Finding What
Works in Health
Care:validity).
Standards
for Systematic
generalizability
or external
of evidenceRevi…
is applicable
if it focuses on the specific condition, patient population, intervention, comparators, and health outcomes that are the focus of the SR’s
research protocol. SRs should assess
the applicability
the evidence
Healthcare
decision makers of
in search
of reliable
information that compares health interventions
Buy
| $61.00
to pa­tients
inPaperback
everyday clinical
settings. This is especially imporseen
increasingly turn to systematic reviews for the best
tant because numerous clinically relevant
dis­tSystematic
inguishreviews
clinical
summary offactors
the evidence.
identify,
select,
assess,
synthesize
the findings
of similar
trial participants from most patients,
such
asand
health
status
and cobut separate studies, and can help clarify what is
$48.99race, and ethnicity (Pham et al.,
asBuy
morbidities
wellEbook
as age,|gender,
known and not known about the potential benefits
of drugs, devices, and other healthcare
2007; Slone Survey, 2006; Vogeli etand
al.,harms
2007).
services. Systematic reviews can be helpful for
MyNAP members save
clinicians who want to integrate research findings into
6 As noted in10%
Chapter
onedaily
of practices,
seven criteria
thewell-informed
online.1, applicability istheir
for patientsthat
to make
to selection
choices
about
their own care,
for professional
committeeLogin
usedortoRegister
guide its
of SR
standards.
In that
context,medical
societies and other organizations that develop clinical
save!
applicability
relates to the aim of CER, that is, to help consumers, clinipractice guidelines.
cians, purchasers, and policy makers to make informed decisions that
Too often systematic reviews are of uncertain or poor
Download
PDF
will improve health
care atFree
both
the individual
and popula­tion levels.
quality. There are no universally accepted standards
for developing systematic
reviews
leading to variability
The other criteria are acceptability/credibility,
efficiency
of conducthow conflicts of interest and biases are handled,
ing the review, patient-centeredness,in
scientific
rigor, timeliness, and
how evidence is appraised, and the overall scientific
transparency.
rigor of the process.
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes
the evidence
shows and
where
reporting
biaswhat
in the
biomedical
literaknowledge gaps remain.
Reporting Bias
Chapter 3 describes the extent of
ture. Depending on the nature and direction of a study’s results, research
Finding
What
Worksselectively
in Health Care also
proposes a
findings or findings for specific outcomes
are
often
published
framework for improving the quality of the science
(publication bias and outcome reporting
bias), published in a particular
underpinning systematic reviews. This book will serve
language (language bias), or released inasjournals
withfordifferent
ease
ac- of
a vital resource
both sponsors
andof
producers
systematic reviews of comparative effectiveness
cess (location bias) (Dickersin, 1990; Dwan
et al., 2008; Gluud, 2006;
research.
Hopewell et al., 2008, 2009; Kirkham et al., 2010; Song et al., 2009, 2010;
Turner et al., 2008). Thus, for each outcome, the SR should assess the
probability of a biased subset of studies comprising the collected body of
evidence.
Dose–Response Association
×
When findings from similar studies suggest a dose–response relationship
across
studies,
it may
increase
confidence
in the
bodyRevi…
of evidence.
Finding
What
Works
in Health
Care:
Standards
for overall
Systematic
“Dose–response association” is defined as a consistent association across
similar studies of a larger effect with greater exposure to the intervention.
For a drug, a dose–response relationship
mightdecision
be observed
treatHealthcare
makers in with
search the
of reliable
information
that
health interventions
ment dosage, intensity,
or duration.|The
concept
ofcompares
dose–response
also
$61.00
 Buy Paperback
increasingly turn to systematic reviews for the best
applies to non-drug exposures. For example,
SR of nutritional
counsummaryin
of an
the evidence.
Systematic reviews
identify,
and synthesize
thenumber
findings of similar
seling to encourage a healthy diet, doseselect,
wasassess,
measured
as “the
and
but separate studies, and can help clarify what is
Ebook the
| $48.99
length of counseling
contacts,
magnitude
and complexity of education Buy
known and not known about the potential benefits
and harms of drugs,
devices, and other
healthcare
al materials provided, and the use of supplemental
intervention
elements,
services. Systematic reviews can be helpful for
such as support
groups
sessions
MyNAP
members
saveor cooking classes” (Ammerman et al.,
clinicians who want to integrate research findings into
10%needs
online.to be exercised their
2002, p. 6). Care
in the
oftodose–redailyinterpretation
practices, for patients
make well-informed
Login
or
Register
to
choices about their own care, for professional medical
sponse relationships that are defined across, rather than within, studies.
societies and other organizations that develop clinical
save!
Cross-study comparisons of different “doses”
may reflect other differences
practice guidelines.
among studies, in addition to dose, that is, dose may be confounded with
Download Free PDF Too often systematic reviews are of uncertain or poor
other study characteristics, populations
included,
ornoother
aspects
ofstandards
the
quality.
There are
universally
accepted
for developing systematic reviews leading to variability
intervention.
in how conflicts of interest and biases are handled,
The absence of a dose–response effect,
the observed
of doses,
how in
evidence
is appraised, range
and the overall
scientific
rigor
of
the
process.
does not rule out a true causal relationship. For example, drugs are not always available in a wide range of doses.InIn
some
instances,
any
dose
aboveof
Finding
What
Works in Health
Care
the Institute
(IOM) recommends 21 standards for
a particular threshold may be sufficientMedicine
for effectiveness.
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
Plausible Confounding That Would Change an
Observed Effect
Although controlled trials generally minimize confounding by randomizing
Finding What
Works in Health Care also proposes a
subjects to intervention and control groups,
obser-
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic
reviews
of comparative
effectiveness
to
selection
bias,
especially
when
research.
vational studies are particularly prone
there is little or no adjustment for potential confounding factors among
comparison groups (Norris et al., 2010). This characteristic of quality refers
to the extent to which systematic differences in baseline characteristics,
prognostic factors, or co-occurring interventions among comparison
groups may reduce or increase an observed effect. Generally, confounding
results in effect sizes that are overestimated. However, sometimes, particularly in observational studies, confounding factors may lead to an under-
×
estimation of the effect of an intervention. If the confounding variables
wereWhat
not present,
the
measured
would for
haveSystematic
been even Revi…
larger. The
Finding
Works in
Health
Care:effect
Standards
AHRQ and GRADE systems use the term “plausible confounding that would
decrease observed effect” to describe such situations. The GRADE Handbook provides the following examples (Schünemann
al., 2009,
p.of125):
Healthcare decisionet
makers
in search
reliable
information that compares health interventions
Buy Paperback | $61.00

increasingly turn to
systematic
reviews for a
the
best
A rigorous systematic review of observational
studies
including
total
summary of the evidence. Systematic reviews identify,
of 38 million patients demonstrated
higher
rates the
in private
select,
assess,death
and synthesize
findings offorsimilar
buthospitals
separate studies,
and can help
clarify
what is
profit versus private
not-for-profit
(Devereaux
et
al.,
2004).
 Buy Ebook | $48.99known and not known about the potential benefits
One possible bias relates to different
disease severity in patients in
and harms of drugs, devices, and other healthcare
the two MyNAP
hospital
types. It is likely, services.
however,
that patients
notSystematic
reviews can in
be the
helpful
for
members save
clinicians who want to integrate research findings into
for-profit
hospitals
those in the for-profit hospitals.
10%
online. were sicker than
their daily practices, for patients to make well-informed
Thus, toLogin
the extent
thattoresidual confounding
existed,
would
biasmedical
or Register
choices about their
own care,itfor
professional
societies and other
that develop
clinical
save! the not-for-profit hospitals.
results against
The organizations
second likely
bias was
practice guidelines.
the possibility that higher numbers of patients with excellent private
often systematic
reviews
areresources
of uncertain or poor
Download
Free
PDFto aToo
insurance coverage
could
lead
hospital
having
more
quality. There are no universally accepted standards
and a spill-over effect that would benefit
those
without
such
coverfor developing
systematic
reviews
leading
to variability
how conflicts
of interest
and biases
are handled,
age. Since for-profit hospitals are in[more]
likely
to admit
a larger
prohow evidence is appraised, and the overall scientific
portion of such well-insured patients
than
not-for-profit hospitals,
rigor of
the process.
the bias is once again against the not-for-profit hospitals. Because the
In Finding What Works in Health Care the Institute of
plausible biases would all diminishMedicine
the demonstrated
intervention
(IOM) recommends
21 standards forefdeveloping
high-quality
systematic reviewsstudof
fect, one might consider the evidence
from
these observational
comparative effectiveness research. The standards
ies as moderate rather than low quality.
address the entire systematic review process from the
A parallel situation exists when observational
studiesthe
have
to the
initial steps of formulating
topicfailed
and building
review team to producing a detailed final report that
demonstrate an association but all plausible biases would have insynthesizes what the evidence shows and where
creased an intervention effect. This
situation
usually arise in the
knowledge
gaps will
remain.
exploration of apparent harmful effects. For example, because the hyFinding What Works in Health Care also proposes a
poglycemic drug phenformin causes
lacticforacidosis,
thequality
related
framework
improving the
of theagent
science
underpinning
systematic Nevertheless,
reviews. This book will
serve
metformin is under suspicion for the
same toxicity.
very
as a vital resource for both sponsors and producers of
large observational studies have failed
to demonstrate
an association
systematic
reviews of comparative
effectiveness
research.
(Salpeter et al., 2004). Given the likelihood that
clinicians would be more alert to lactic acidosis in the presence of the
agent and overreport its occurrence, one might consider this moder-
ate, or even high-quality evidence refuting a causal relationship betweenWorks
typicalintherapeutic
doses
of metformin
and lactic Revi…
acidosis.
Finding What
Health Care:
Standards
for Systematic
Strength of Association
×
Healthcare decision makers in search of reliable
information
that confounding
compares health interventions
Because observational
are subject
to many
factors
Buystudies
Paperback
| $61.00
increasingly turn to systematic reviews for the best
(e.g., patients’ health status, demographic
characteristics)
and greater
risk
summary
of the evidence. Systematic
reviews
identify,
select,
assess, execution,
and synthesize and
the findings
of similar
of bias compared to controlled trials, the
design,
statistical
but separate studies, and can help clarify what is
Buyshould
Ebookbe
| $48.99
analyses in each study
assessed
carefully to determine the influknown and not known about the potential benefits
andthe
harms
of drugs, devices,
andStrength
other healthcare
ence of potential confounding factors on
observed
effect.
of
services. Systematic reviews can be helpful for
association refers
tomembers
the likelihood
MyNAP
save that a large observed effect in an obserclinicians who want to integrate research findings into
online.
vational study10%
is not
due to bias from potential
confounding
factors.
their daily practices,
for patients
to make well-informed
Login or Register to
choices about their own care, for professional medical
societies and other organizations that develop clinical
save!
practice guidelines.


Evidence on Assessment Methods Is Elusive
Too often
systematic
reviews
are of uncertain
or poor
Applying the aboveDownload
conceptsFree
in aPDF
systematic
way
across
multiple
intervenThere are no universally accepted standards
tions and numerous outcomes is clearlyquality.
challenging.
Although many SR exfor developing systematic reviews leading to variability
perts agree on the concepts that should
underpin
the
assessment
of handled,
the
in how
conflicts of
interest
and biases are
how evidence is appraised, and the overall scientific
quality of body of evidence, the committee
did not find any research to
rigor of the process.
support existing methods for using these basic concepts in a systematic
Finding What Works in Health Care the Institute of
method such as the GRADE and AHRQ Inapproaches.
The GRADE Working
Medicine (IOM) recommends 21 standards for
Group reports that 50 organizations have
eitherhigh-quality
endorsed
or arereviews
using
developing
systematic
of an
comparative
effectiveness
research.However,
The standards
adapted version of their system (GRADE
Working
Group, 2010).
address the entire systematic review process from the
the reliability and validity of the GRADEinitial
andsteps
AHRQ
methods
haveand
not
beenthe
of formulating
the topic
building
review team
to producing
a detailed
final report that
evaluated, and not much literature assesses
other
approaches.
Furthersynthesizes what the evidence shows and where
more, many GRADE users are apparently
selecting aspects of the system to
knowledge gaps remain.
suit their needs rather than adopting the entire method. The AHRQ
Finding What Works in Health Care also proposes a
method is one adaptation.
framework for improving the quality of the science
underpinningevidence
systematic reviews.
This bookthat
will serve
The committee heard considerable anecdotal
suggesting
as a vital resource for both sponsors and producers of
many SR producers and users had difficulty
using GRADE. Some organizasystematic reviews of comparative effectiveness
tions seem reluctant to adopt a new, more
complex system that has not
research.
been sufficiently evaluated. Others are concerned that GRADE is too time
consuming and difficult to implement. There are also complaints about the
method’s subjectivity. GRADE advocates acknowledge that the system does
not eliminate subjectivity, but argue that a strength of the system is that,
unlike other approaches, it makes transparent any judgments or disagreements about evidence (Brozek et al., 2009).
Finding What Works in Health Care: Standards for Systematic Revi…
×
RECOMMENDED STANDARD FOR
ASSESSING AND DESCRIBING
THE
Healthcare decision makers in search of reliable
information
compares health interventions
QUALITY
OF A BODY
OFthat
EVIDENCE
| $61.00
 Buy Paperback
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
The committee recommends the following
for assessing
andof deselect,standard
assess, and synthesize
the findings
similar
but
separate
studies,
and
can
help
clarify
what
is
scribing the quality
of aEbook
body of| evidence.
$48.99 As noted earlier, this overall as Buy
known and not known about the potential benefits
sessment should be done once the qualitative
quantitative
syntheses
and harmsand
of drugs,
devices, and other
healthcare
services.
Systematic
reviews
can
be
helpful
for
are completed
(see Standards
4.2–4.4 below). The order of this chapter’s
MyNAP
members save
clinicians who want to integrate research findings into
standards does
not
indicate the sequence
in which the various steps
10%
online.
their daily practices, for patients to make well-informed
Login or Register
to 4.1 is presented
choices about
theirto
own
care, forthe
professional
medical
should be conducted.
Standard
first
reflect
commitsocieties and other organizations that develop clinical
save!
tee’s recommendation
that the SR specifies
its methods a priori in the repractice guidelines.
search protocol.7
Download Free PDF
Standard 4.1—Use a prespecified
evidence
Required elements:
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
method
to evaluate the body of
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
rigor of the process.
In Finding
Whatthe
Worksfollowing
in Health Care
the Institute of
4.1.1 For each outcome, systematically
assess
characMedicine (IOM) recommends 21 standards for
teristics of the body of evidence:
developing high-quality systematic reviews of
Risk of bias
Consistency
Precision
Directness
Reporting bias
4.1.2
comparative effectiveness research. The standards
address the entire systematic review process from the
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
For bodies of evidence that include
research,
as a vitalobservational
resource for both sponsors
and producers of
systematic reviews
of comparative effectiveness
also systematically assess the following
characteristics
for
research.
each outcome:
Dose-response association
Plausible confounding that would change the observed
effect
Strength of association
4.1.3 For each outcome specified in the protocol, use consistent
language to characterize the level of confidence in the estiof in
theHealth
effect Care:
of an intervention
Finding Whatmates
Works
Standards for Systematic Revi…
Rationale
×
Healthcare decision makers in search of reliable
information that compares health interventions
Buy Paperback
| $61.00
If an SR is to be objective,
it should use
prespecified, analytic methods. If
increasingly turn to systematic reviews for the best
the SR’s assessment of the quality of a body
ofofevidence
isSystematic
to be credible
summary
the evidence.
reviews identify,
select,
andon
synthesize
the findings
of similar
and true to scientific principles, it should
beassess,
based
agreed-on
concepts
but separate studies, and can help clarify what is
BuySR
Ebook
| $48.99
of study quality. If the
is to be
comprehensible,
it should
usepotential
unambiguknown and not known
about the
benefits
and harms of drugs, devices, and other healthcare
ous language, free from jarservices. Systematic reviews can be helpful for
MyNAP members save
clinicians who wantstandards
to integrate research
findings into
7 See Chapter 2 for the committee’s recommended
for devel10% online.
their daily practices, for patients to make well-informed
oping the SR
research
protocol.
Login
or Register
to
choices about their own care, for professional medical
societies and other organizations that develop clinical
save!
practice guidelines.


systematic reviews are of uncertain or poor
Free
PDF Too
gon, to describe theDownload
quality of
evidence
foroften
each
outcome. Decision makers
quality. There are no universally accepted standards
—whether clinicians, patients, or others—should
havereviews
to decipher
for developingnot
systematic
leading tounvariability
howsymbols
conflicts of in
interest
andto
biases
are handled,
defined and possibly conflicting terms in
and
order
understand
how evidence is appraised, and the overall scientific
the methods and findings of SRs.
rigor of the process.
Clearly, the assessment of the quality of a body of evidence—for each
In Finding What Works in Health Care the Institute of
outcome in the SR—must incorporate multiple
dimensions
quality.
Medicine (IOM)
recommendsof
21study
standards
for
developing
high-quality
systematic
reviews
of
Without a sound conceptual framework for scrutinizing the body of evicomparative effectiveness research. The standards
dence, the SR can lead to the wrong conclusions
about
an intervention’s
address the entire
systematic
review process from the
initial steps of formulating
the topic
and building the
effectiveness, with potentially serious implications
for clinical
practice.
review team to producing a detailed final report that
The lack of an evidence-based system
for assessing and characterizing
synthesizes what the evidence shows and where
the quality of a body of evidence is clearly
problematic.
knowledge
gaps remain.A plethora of systems are in use, none have been evaluated,
and all have their proponents
Finding What Works in Health Care also proposes a
and critics. The committee’s recommended
quality
characteristics
arescience
framework
for improving
the quality of the
underpinning systematic reviews. This book will serve
well-established concepts for evaluating
quality; however, the SR field
as a vital resource for both sponsors and producers of
needs unambiguous, jargon-free language
for systematically
applying
these
systematic
reviews of comparative
effectiveness
concepts. GRADE merits consideration,research.
but should be rigorously evaluated
before it becomes a required component of SRs in the United States. Until
a well-validated standard language is developed, SR authors should use
their chosen lexicon and provide clear definitions of their terms.
QUALITATIVE SYNTHESIS OF THE BODY
OF EVIDENCE
Finding What Works in Health Care: Standards for Systematic Revi…
×
As noted earlier, the term “synthesis” refers to the collation, combination,
Healthcare decision makers in search of reliable
and summary of the results of an SR. The
committee uses the term “qualiinformation that compares health interventions
Buy Paperback | $61.00

tative synthesis” to refer to an assessment
of theturn
body
of evidence
that
increasingly
to systematic
reviews for
the best
summary
of
the
evidence.
Systematic
reviews
identify,
goes beyond factual descriptions or tables that, for example, simply detail
select, assess, and synthesize the findings of similar
how many studies were assessed, the reasons
forstudies,
excluding
studies,
but separate
and can other
help clarify
what is
Buy Ebook | $48.99

known
and
not
known
about
the
potential
benefits
the range of study sizes and treatments compared, or quality scores of
and harms of drugs, devices, and other healthcare
each study as measured by a risk of biasservices.
tool. Systematic
While anreviews
accurate
can be descriphelpful for
MyNAP members save
clinicians
who sufficient
want to integrate
research
findings into
tion of the body of evidence is essential,
it is not
(Atkins,
2007;
10% online.
their daily practices, for patients to make well-informed
Mulrow and Lohr,
2001).
Login or
Register to
choices about their own care, for professional medical
The primary
focus of the qualitative synthesis
be to develop
and
societies and should
other organizations
that develop
clinical
save!
practice guidelines.
to convey a deeper understanding of how an intervention works, for whom,
and under what circumstances.
identified
nine
key
purposToo often systematic
reviews
are of
uncertain
or poor
Download FreeThe
PDFcommittee
quality. There are no universally accepted standards
es of the qualitative synthesis (Table 4-2).
for developing systematic reviews leading to variability
If crafted to inform clinicians, patients,
and
other
decision
the
in how
conflicts
of interest
and makers,
biases are handled,
how evidence
is appraised,
and the overalland
scientific
qualitative synthesis would enable the reader
to judge
the relevance
rigor of the process.
validity of the body of evidence for specific clinical
TABLE 4-2
Purpose
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
Key Purposes of the Qualitative
Synthesis
address
the entire systematic review process from the
initial steps of formulating the topic and building the
Relevant Content in the Systematic
Review (SR)
review team to producing a detailed final report that
synthesizes
what the evidence
shows
and
where
A description of the clinical
environment
in which
the
reknowledge gaps remain.
To orient
the reader
to the clinical
landscape
search was conducted. It should enable the reader to grasp
What Works
Health Carepatients
also proposes
a
the relevance of the body Finding
of evidence
toinspecific
and
framework for improving the quality of the science
clinical circumstances. It should
describe
the
settings
in will serve
underpinning
systematic
reviews.
This book
a vital
resource
for both sponsors
and producers of
which care was provided, as
how
the
intervention
was delivered,
systematic reviews of comparative effectiveness
by whom, and to whom. research.
To describe A description of the actual care and experience of the study
what actu- participants (in contrast with the original study protocol).
ally happened to
subjects
during the
course of
the What
studiesWorks in Health Care: Standards for Systematic Revi…
Finding
To critique
the
strengths
and weaknesses of
the body of
evidence
×
A description of the strengths and weaknesses of the individual studies’ design and execution, including their common
makers
in search of reliable
features and differences. ItHealthcare
shoulddecision
highlight
well-designed
information that compares health interventions
Buy Paperback
| $61.00
and
studies, contrasting
them with others, and inexecuted
increasingly turn to systematic reviews for the best
clude an assessment of the
extentof to
risk of
biasidentify,
afsummary
the which
evidence.the
Systematic
reviews
select, assess, and synthesize the findings of similar
fects summary estimates of
the intervention’s effect. It should
but separate studies, and can help clarify what is
Buy Ebook
| $48.99
include
also
a succinct
summary
issues
lead to
the
known andof
notthe
known
aboutthat
the potential
benefits
harms
of drugs,“fair,”
devices,
and other
healthcare
use of particular adjectivesand(e.g.,
“poor,”
“low
quality,”
services. Systematic reviews can be helpful for
MyNAP
members
save
“high risk
of bias,”
etc.) in describing
thetoquality
the findings into
clinicians who want
integrateof
research
10%
online.
their daily practices, for patients to make well-informed
evidence.
Login or Register to
choices about their own care, for professional medical
To identify save!
An examination of how heterogeneity
inorganizations
the treatment’s
ef-clinical
societies and other
that develop
practice
guidelines. in the study populadifferences fects may be due to clinical
differences
in the de- tion (e.g.,
demographics, coexisting conditions, or treatments)
Download Free PDF Too often systematic reviews are of uncertain or poor
There arein
nothe
universally
accepted
standards
sign and
as well as methodological quality.
differences
studies’
designs.
for developing systematic reviews leading to variability
execution
in how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
of the indirigor of the process.
vidual studIn Finding What Works in Health Care the Institute of
ies that exMedicine (IOM) recommends 21 standards for
plain why
developing high-quality systematic reviews of
comparative effectiveness research. The standards
their results
address the entire systematic review process from the
differ
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
Finding What Works in Health Care also proposes a
To describe
how the design
and execution
of the individual studies affect their relevance to realworld clinical
settings
A description of the applicability
of the studies’
framework for improving
the quality health
of the science
underpinningintervention,
systematic reviews.
This book will serve
conditions, patient population,
comparators,
as a vital resource for both sponsors and producers of
and health outcomes to
the SRreviews
research
question.
It
systematic
of comparative
effectiveness
research.
should also address how
adherence of patients and
providers may limit the applicability of the results. For example, the use of prescribed medications, as directed,
may differ substantially between patients in the community compared with study participants.
To integrate
For each important outcome, an overview of the nested
×
the general
subgroup analyses, as well as a presentation of the overall
summary
the summary
assessment
of for
the Systematic
evidence. Revi…
Finding
What of
Works
in Healthand
Care:
Standards
evidence and
the subgroup
analyses based
Healthcare decision makers in search of reliable
information that compares health interventions
on setting and Buy Paperback | $61.00
increasingly turn to systematic reviews for the best
patient
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
populations
but separate studies, and can help clarify what is
Buy Ebook of
| $48.99
To call atten- A description
important patient subgroups (e.g., by co-
known and not known about the potential benefits
harmsor
of ethnicity)
drugs, devices,that
and other
tion to patient morbidity, age, gender,and
race,
are healthcare
unadservices. Systematic reviews can be helpful for
members
save
populations MyNAP
dressed
in the
body ofclinicians
evidence.
who want to integrate research findings into
10% online.
their daily practices, for patients to make well-informed
that have been
Login or Register to
choices about their own care, for professional medical
inadequately
societies and other organizations that develop clinical
save!
practice guidelines.
studied or for
whom results
differ
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
A clear synthesis of thein evidence
goes
beyond
how conflicts that
of interest
and
biases areprehandled,
how
evidence
is
appraised,
and
the
overall
scientific
sentation of summary statistics. The summary statistics
rigor of the process.
Download Free PDF
To interpret
and assess the
robustness of should not dominate the discussion; instead, the synthesis
Finding
What Works
in Health Care using
the Institute
the metaof the evidence shouldInbe
carefully
articulated,
theof
Medicine (IOM) recommends 21 standards for
analysis results summary statistics to developing
support high-quality
the key conclusions.
systematic reviews of
To describe
how the SR
findings contrast with conventional
wisdom
effectiveness
The standards
Sometimes commonlycomparative
held notions
aboutresearch.
an intervention
address the entire systematic review process from the
or a type of study design
not
supported
by the
initialare
steps
of formulating
the topic
and body
buildingof
the
review
team
to
producing
a
detailed
final
report
that
evidence. If this occurs, the qualitative synthesis should
synthesizes what the evidence shows and where
clearly explain how theknowledge
SR findings
differ from the congaps remain.
ventional wisdom.
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
research.
decisions and circumstances. Guidance from the Editors of Annals of Internal Medicine is noteworthy:
We are disappointed when a systematic review simply lists the
characteristics and findings of a series of single studies without attempting, in a sophisticated and clinically meaningful manner, to
discover the pattern in a body of evidence. Although we greatly val-
ue meta-analyses, we look askance if they seem to be mechanisticonsideration
of the appropriateFinding cally
Whatproduced
Works inwithout
Health careful
Care: Standards
for Systematic
Revi…
ness of pooling results or little attempt to integrate the finds into
the contextual background. We want all reviews, including metaanalyses to include rich qualitative Healthcare
synthesis.
(Editors,
p.of reliable
decision
makers2005,
in search
information that compares health interventions
1019)
 Buy Paperback | $61.00
×
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Buy Ebook | $48.99
known
and not should
known about
potential
benefits
Although the qualitative synthesis of CER
studies
be the
based
in sysand harms of drugs, devices, and other healthcare
tematic and scientifically rigorous methods,
nonetheless
involves
numerservices.itSystematic
reviews can
be helpful
for
MyNAP members save
clinicians wholegitimacy,
want to integrate
research
findings into
ous judgments—judgments
about the relevance,
and
relative
10% online.
their daily practices, for patients to make well-informed
uncertainty of
some
of the evidence;
the implications
missingmedical
Login
or aspects
Register to
choices about
their own care, forof
professional
and other organizations
that develop
evidence (a commonplace
occurrence);societies
the soundness
of technical
meth-clinical
save!
practice guidelines.
Judgments and Transparency Are Key

ods; and the appropriateness of conducting a meta-analysis (Mulrow et al.,
often systematic
reviews
of uncertain
1997). Such judgments
may be
inherently
but
theyareare
alwaysor poor
Download
Free
PDF Toosubjective,
quality. There are no universally accepted standards
valuable and essential to the SR process.
If the SR team approaches the litfor developing systematic reviews leading to variability
erature from an open-minded perspective,
members
arebiases
uniquely
poin howteam
conflicts
of interest and
are handled,
how evidence is appraised, and the overall scientific
sitioned to discover and describe patterns in a body of evidence that can
rigor of the process.
yield a deeper understanding of the underlying science and help readers to
In Finding What Works in Health Care the Institute of
interpret the findings of the quantitative
synthesis (if conducted). However,
Medicine (IOM) recommends 21 standards for
the SR team should exercise extreme care
to keep
such systematic
discussions
approdeveloping
high-quality
reviews
of
comparative
research. Thedata.
standards
priately balanced and, whenever possible,
driveneffectiveness
by the underlying
address the entire systematic review process from the
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
RECOMMENDED STANDARDS FOR
QUALITATIVE SYNTHESIS
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
The committee recommends the following
standard
and reviews.
elements
of perunderpinning
systematic
This book
will serve
as
a
vital
resource
for
both
sponsors
and
producers
of
formance for conducting the qualitative synthesis.
systematic reviews of comparative effectiveness
research.
Standard 4.2—Conduct a qualitative synthesis
Required elements:
4.2.1 Describe the clinical and methodological characteristics of the
included studies, including their size, inclusion or exclusion of
important subgroups, timeliness, and other relevant factors
Finding 4.2.2
WhatDescribe
Works inthe
Health
Care:and
Standards
forofSystematic
Revi…
strengths
limitations
individual studies
and patterns across studies
×
4.2.3 Describe, in plain terms, how flaws in the design or execution
Healthcare decision makers in search of reliable
of the study (or groups of studies)
could bias the results, exinformation that compares health interventions
Buy Paperback | $61.00

plaining the reasoning behindincreasingly
these judgments
turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
4.2.4 Describe the relationships between
the characteristics of the
select, assess, and synthesize the findings of similar
but separate
studies, and
canpatterns
help clarify what is
individualBuy
and|their
reported
findings
and
Ebook
$48.99
 studies
known and not known about the potential benefits
across studies
4.2.5
and harms of drugs, devices, and other healthcare
services.studies
Systematic
can be helpful for
Discuss
themembers
relevance
of individual
toreviews
the populations,
MyNAP
save
clinicians who want to integrate research findings into
comparisons,
settings,
and outcomes or
10% online.cointerventions,their
daily practices, for patients to make well-informed
Login orofRegister
to
choices about their own care, for professional medical
measures
interest
societies and other organizations that develop clinical
save!
practice guidelines.
Rationale
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic
reviews
to variability
The qualitative synthesis is an often undervalued
component
ofleading
an SR.
in how conflicts of interest and biases are handled,
Many SRs lack a qualitative synthesis altogether
or simply provide a nonanhow evidence is appraised, and the overall scientific
alytic recitation of the facts (Atkins, 2007).
Patients,
clinicians, and others
rigor of
the process.
Download Free PDF
should feel confident that SRs accurately reflect what is known and not
In Finding What Works in Health Care the Institute of
known about the effects of a healthcareMedicine
intervention.
To give21readers
a
(IOM) recommends
standards for
developing
high-quality
systematic
reviews
of
clear understanding of how the evidence applies to real-world clinical circomparative effectiveness research. The standards
cumstances and specific patient populations,
SRs
should
describe—in
easyaddress the
entire
systematic
review process
from the
initialmethodological
steps of formulating the
topic and building of
the
to-understand language—the clinical and
characteristics
review team to producing a detailed final report that
the individual studies, including their strengths
andthe
weaknesses
synthesizes what
evidence showsand
and their
where
remain.
relevance to particular populations andknowledge
clinicalgaps
settings.
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
the
background and rationale for the
research.
META-ANALYSIS
This section of the chapter presents
committee’s recommended standards for conducting a meta-analysis: first,
considering the issues that determine whether a meta-analysis is appropriate, and second, exploring the fundamental considerations in undertaking a meta-analysis. A detailed description of meta-analysis methodology is
beyond the scope of this report; however, excellent reference texts are
available (Borenstein, 2009; Cooper et al., 2009; Egger et al., 2001; Rothstein et al., 2005; Sutton et al., 2000). This discussion draws from these
×
sources as well as guidance from the AHRQ Effective Health Care Program,
CRD,What
and the
Cochrane
Collaboration
(CRD, 2009;
Deeks et al.,
2008; Fu et
Finding
Works
in Health
Care: Standards
for Systematic
Revi…
al., 2010).
Meta-analysis is the statistical combination of results from multiple individual studies. Meta-analytic techniques
have been
Healthcare
decision makers in search of reliable
information that compares health interventions
 Buy Paperback | $61.00
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
used for more than a century for a variety
ofassess,
purposes
(Sutton
Higgins,
select,
and synthesize
the and
findings
of similar
but separate studies, and can help clarify what is
2008). The nomenclature
for SRs
and meta-analysis has evolved over time.
Buy Ebook
| $48.99
known and not known about the potential benefits
Although often used as a synonym for SR
theofpast,
has
andin
harms
drugs, meta-analysis
devices, and other healthcare
Systematic
reviews
be helpful
for
come to mean
the quantitative
analysisservices.
of data
in an SR.
As can
noted
earlier,
MyNAP
members save
clinicians who want to integrate research findings into
the committee
views
“metaanalysis” as their
a broad
term that
encompasses
a
10%
online.
daily practices,
for patients
to make well-informed
Login
or Register to approaches
choiceswhose
about their
owniscare,
professional medical
wide variety of
methodological
goal
to for
quantitatively
societies and other organizations that develop clinical
save!
synthesize and
summarize data across practice
a set of
studies. In the context of
guidelines.

CER, meta-analyses are undertaken to combine and summarize existing
Too often systematic reviews are of uncertain or poor
Download
Free PDF of
evidence comparing
the effectiveness
multiple healthcare interventions
quality. There are no universally accepted standards
systematic
leadingthe
to variability
(Fu et al., 2010). Typically, the objective for
ofdeveloping
the analysis
is toreviews
increase
in how conflicts of interest and biases are handled,
precision and power of the overall estimated
effect of an intervention by
how evidence is appraised, and the overall scientific
producing a single pooled estimate, such
odds ratio. In CER, large
rigoras
of an
the process.
numbers are often required to detect what
may be modest or even small
In Finding What Works in Health Care the Institute of
treatment effects. Many studies are themselves
too
small to 21
yield
concluMedicine (IOM)
recommends
standards
for
developing high-quality systematic reviews of
sive results. By combining the results of multiple studies in a meta-analycomparative effectiveness research. The standards
sis, the increased number of study participants
error,
address the can
entirereduce
systematicrandom
review process
from the
initial stepsof
of formulating
building the
improve precision, and increase the likelihood
detectingtheatopic
realand
effect
review team to producing a detailed final report that
(CRD, 2009).
synthesizes what the evidence shows and where
knowledge
gaps remain.
Fundamentally, a meta-analysis provides
a weighted
average of treatment effects from the studies in the SR.Finding
While
varying
in details,
the
What
Works in Health
Care also
proposes a
framework
for
improving
the
quality
of
the science
weights are set up so that the most informative studies have the greatest
underpinning systematic reviews. This book will serve
impact on the average. While the term as
“most
informative” is vague, it is
a vital resource for both sponsors and producers of
systematic
reviews
of comparative
effectiveness
usually expressed in terms of the sample
size and
precision
of the
study.
research.
The largest and most precisely estimated studies receive the greatest
weights. In addition to an estimate of the average effect, a measure of the
uncertainty of this estimate that reflects random variation is necessary for
a proper summary.
In many circumstances, CER meta-analyses focus on the average effect
of the difference between two treatments across all studies, reflecting the
common practice in RCTs of providing a single number summary. While a
×
meta-analysis is itself a nonrandomized study, even if the individual studies
in the
SR are
themselves
randomized,
it can fill
or an exFinding
What
Works
in Health
Care: Standards
fora confirmatory
Systematic Revi…
ploratory role (Anello and Fleiss, 1995). Although it has been underused for
this purpose, meta-analysis is a valuable tool for assessing the pattern of
results across studies and for identifying
the need
formakers
primary
research
Healthcare
decision
in search
of reliable
information that compares health interventions
| $61.00
(CRD, 2009; Sutton
and Paperback
Higgins, 2008).
 Buy
increasingly turn to systematic reviews for the best
In other circumstances, individual studies
inofSRs
of more
than two
treatsummary
the evidence.
Systematic
reviews
identify,
select, assess,
synthesize
findings of similar
ments evaluate different subsets of treatments
soand
that
direct,thehead-tobut separate studies, and can help clarify what is
Buy Ebook
$48.99
head comparisons
two| treatments
of interest, for example, are
 between
known and not known about the potential benefits
and harms
of drugs, devices,
and otherthe
healthcare
limited. Treatment networks allow indirect
comparisons
in which
two
services. Systematic reviews can be helpful for
MyNAP
save
treatments are
eachmembers
compared
to a common
clinicians who want to integrate research findings into
10% online.
Login or Register to
save!
their daily practices, for patients to make well-informed
choices about their own care, for professional medical
societies and other organizations that develop clinical
indirect
treatment estimate then conpractice guidelines.
third treatment (e.g., a placebo). The
sists of the difference between the two comparisons with the common
Too often systematic reviews are of uncertain or poor
Download
Free
treatment. The network
is said
toPDF
be consistent
if the indirect estimates
quality. There are no universally accepted standards
developing
reviews leading to is
variability
are the same as the direct estimates (Luforand
Ades,systematic
2004). Consistency
in how conflicts of interest and biases are handled,
most easily tested when some studies test
all three treatments. Finding
how evidence is appraised, and the overall scientific
consistency increases confidence that the
estimated
rigor of
the process.effects are valid. Inconsistency suggests a bias in either orInboth
of the indirect or direct estiFinding What Works in Health Care the Institute of
mates. While the direct estimate is often
preferred,
bias in the
design for
of
Medicine
(IOM) recommends
21 standards
developing high-quality systematic reviews of
the direct comparison studies may suggest that the indirect estimate is
comparative effectiveness research. The standards
better (Salanti et al., 2010). Proper consideration
of indirect
readdress the entire
systematicevidence
review process
from the
initial steps
formulating the
topic and building the
quires that the full network be considered.
Thisoffacilitates
determining
review team to producing a detailed final report that
which treatments work best for which synthesizes
reportedwhat
outcomes.
the evidence shows and where
knowledge
remain.
Many clinical readers view meta-analyses
as gaps
confirmatory
summaries
that resolve conflicting evidence from previous
In this
role,
all thea
Finding Whatstudies.
Works in Health
Care also
proposes
framework
for improving
the quality
the science
potential decision-making errors in clinical
trials
(e.g., Type
1 andofType
2
underpinning systematic reviews. This book will serve
errors or excessive subgroup analyses)8asapply
to meta-analyses
asand
well.
a vital resource
for both sponsors
producers of
systematic reviews
of comparative
effectiveness
However, in an exploratory role, meta-analysis
may be
more useful
as a
research.
means to explore heterogeneity among study findings, recognize types of
patients who might differentially benefit from (or be harmed by) treatment
or treatment protocols that may work more effectively, identify gaps in
knowledge, and suggest new avenues for research (Lau et al., 1998). Many
of the methodological developments in meta-analysis in recent years have
been motivated by the desire to use the information available from a metaanalysis for multiple purposes.
When Is Meta-Analysis Appropriate?
×
Finding What Works in Health Care: Standards for Systematic Revi…
Meta-analysis has the potential to inform and explain, but it also has the
potential to mislead if, for example, the individual studies are not similar,
are biased, or publication or reporting biases are large (Deeks et al., 2008).
Healthcare decision makers in search of reliable
A meta-analysis should not be assumedinformation
to always be an appropriate step in
Buy Paperback | $61.00 that compares health interventions

an SR. The decision to conduct a meta-analysis
is neither
purely
analytical
increasingly turn
to systematic
reviews
for the best
summary of the evidence. Systematic reviews identify,
nor statistical in nature. It will depend on a number of factors, such as the
select, assess, and synthesize the findings of similar
availability of suitable data and the likelihood
that
the analysis
could
inform
but separate
studies,
and can help
clarify what
is
Buy Ebook | $48.99

known
and
not
known
about
the
potential
benefits
clinical decision making. Ultimately, it is a subjective judgment that should
and harms of drugs, devices, and other healthcare
be made in consultation with the entireservices.
SR team,
including
both
clinical
Systematic
reviews can
be helpful
for
MyNAP members save
clinicians whoof
want
to integrate research findings into
and methodological
perspectives. For purposes
transpar10% online.
their daily practices, for patients to make well-informed
Loginisor
to
choices
about their
own care,
professional medical
8 A Type 1 error
a Register
false-positive
result.
A Type
2 error
is a for
false-negasocieties and other organizations that develop clinical
save!
tive result.
practice guidelines.
Download Free PDF
ency, the review team should clearly
tive determination (Fu et al., 2010).
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
explain
the rationale for each subjecin how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
rigor of the process.
In Finding What Works in Health Care the Institute of
Data Considerations
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
Conceptually a meta-analysis may make
sense, and
the studies
may
appear
comparative
effectiveness
research.
The standards
address
entire
systematic
from the
sufficiently similar, but without unbiased
datathethat
are
in (or review
may process
be transinitial steps of formulating the topic and building the
formed into) similar metrics, the meta-analysis
may
not befinal
feasible.
review teamsimply
to producing
a detailed
report that
synthesizes
what
the
evidence
shows
and
where
There is no agreed-on definition of “similarity” with respect to CER data.
knowledge gaps remain.
Experts agree that similarity should be judged across three dimensions
What studies
Works in Health
Care alsosimilar,
proposes a
(Deeks et al., 2008; Fu et al., 2010): First,Finding
are the
clinically
framework for improving the quality of the science
with comparable study population characteristics,
interventions, and outunderpinning systematic reviews. This book will serve
as aalike
vital resource
for both sponsorsin
and
producers of
come measures? Second, are the studies
methodologically
study
reviews of comparative effectiveness
design, conduct, and quality? Third, aresystematic
the observed
treatment effects
research.
statistically similar? All three of these questions should be considered before deciding a meta-analysis is appropriate.
Many meta-analyses use aggregate summary data for the comparison
groups in each trial. Meta-analysis can be much more powerful when outcome, treatment, and patient data—individual patient data (IPD)—are available from individual patients. IPD, the raw data for each study participant,
permit data cleaning and harmonization of variable definitions across
×
studies as well as reanalysis of primary studies so that they are more readily combined
(e.g., in
clinical
measurement
reported
at a common
time). IPD
Finding
What Works
Health
Care: Standards
for Systematic
Revi…
also allow valid analyses for effect modification by factors that change at
the patient level, such as age and gender, for which use of aggregate data
are susceptible to ecological bias (Berlin
et al., 2002;
et al.,of2004).
Healthcare
decisionSchmid
makers in search
reliable
information
compares
health interventions
Buy Paperback
$61.00
By permitting 
individual
modeling in| each
study,that
IPD
also focus
attention
increasingly turn to systematic reviews for the best
on study-level differences that may contribute
treatsummary ofto
theheterogeneity
evidence. Systematicof
reviews
identify,
assess,
and synthesize
findings
of similar
ment effects across studies. When IPD select,
are not
available
fromtheeach
study
in
but separate studies, and can help clarify what is
$48.99 together with summary data from
the meta-analysis,
they Ebook
can be |analyzed
 Buy
known and not known about the potential benefits
harmsThe
of drugs,
and the
otherindividhealthcare
the other studies (Riley and Steyerberg,and
2010).
IPDdevices,
inform
services. Systematic reviews can be helpful for
MyNAP
saveof data inform the study-level effects. The
ual-level effects
andmembers
both types
clinicians who want to integrate research findings into
10% online.
increasing availability
of data repositories
registries
may make
hytheir and
daily practices,
for patients
to makethis
well-informed
Login or Register to
choices about their own care, for professional medical
brid modeling the norm in the future.
societies and other organizations that develop clinical
save!
Advances in health information technology,
such as electronic health
practice guidelines.
records (EHRs) and disease registries, promise new sources of evidence on
Download Free PDF Too often systematic reviews are of uncertain or poor
the effectiveness of health interventions.
As There
thesearedata
sources
become
quality.
no universally
accepted
standards
for
developing
systematic
reviews
leading
to
variability
more readily accessible to investigators, they are likely to supplement or
in how conflicts of interest and biases are handled,
even replace clinical trials data in SRs of
how evidence is appraised, and the overall scientific
rigor of the process.
In Finding What Works in Health Care the Institute of
CER. Furthermore, as with other data sources,
therecommends
potential 21
for
bias and
Medicine (IOM)
standards
for
developing high-quality systematic reviews of
confounding will need to be addressed.
comparative effectiveness research. The standards
The Food and Drug Administration Sentinel
Initiative
and related
activiaddress the
entire systematic
review process
from the
initial steps
of formulating
the topic
andimporbuilding the
ties (e.g., Observational Medical Outcomes
Partnership)
may
be an
review team to producing a detailed final report that
tant new data source for future SRs. When
operational,
the Sentinel
synthesizes
what the evidence
shows andInitiawhere
knowledge
gaps remain.
tive will be a national, integrated, electronic
database
built on EHRs and
claims records databases for as many as
100 What
million
(HHS,
2010;
Finding
Worksindividuals
in Health Care also
proposes
a
framework
for
improving
the
quality
of
the
science
Platt et al., 2009). Although the principal objective of the system is to deunderpinning systematic reviews. This book will serve
tect adverse effects of drugs and otherasmedical
products, it may also be
a vital resource for both sponsors and producers of
systematic reviews
of comparative
effectiveness
useful for SRs of CER questions. A “Mini-Sentinel”
pilot
is currently
under
research.
development at Harvard Pilgrim Health Care (Platt, 2010). The system will
be a distributed network, meaning that separate data holders will contribute to the network, but the data will never be put into one common
repository. Instead, all database holders will convert their data into a common data model and retain control over their own data. This allows a single
“program” to be run (e.g., a statistical analysis in SAS) on all the disparate
datasets, generating an estimated relative risk (or other measure) from
eachWhat
database.
These
then can
be Standards
viewed as afor
type
of meta-analysis.
Finding
Works
in Health
Care:
Systematic
Revi…
Will the Findings Be Useful?
×
Healthcare decision makers in search of reliable
information
that compares
interventions
The fact that available
are conducive
to pooling
is nothealth
in itself
suffiBuydata
Paperback
| $61.00
increasingly turn to systematic reviews for the best
cient reason to conduct a meta-analysis
(Fu etofal.,
meta-analysis
summary
the 2010).
evidence.The
Systematic
reviews identify,
select, assess, results
and synthesize
findings
of similar
should not be undertaken unless the anticipated
are the
likely
to probut separate studies, and can help clarify what is
Buy Ebook
| $48.99
duce meaningful answers
that are
useful
to patients, clinicians, or other
known and not known about the potential benefits
and harms
of drugs,are
devices,
and other differhealthcare
decision makers. For example, if the same
outcomes
measured
services. Systematic reviews can be helpful for
ently in the individual
studiessave
and the measures
cannot be converted to a
MyNAP members
clinicians who want to integrate research findings into
10%
online.
common scale,
doing
a meta-analysis may
not practices,
be appropriate
(Cummings,
their daily
for patients to
make well-informed
Login
or
Register
to
choices about their own care, for professional medical
2004). This situation may occur in studies comparing the effect of an intersocieties and other organizations that develop clinical
save!
vention on a variety of important patient
outcomes
practice
guidelines.such as pain, mental


health status, or pulmonary function.
Download Free PDF
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
rigor of the process.
Conducting the Meta-Analysis
Addressing Heterogeneity
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
Good statistical analyses quantify the amount
of variability in the data in
developing high-quality systematic reviews of
order to obtain estimates of the precision
with which
estimates
be
comparative
effectiveness
research.may
The standards
address
theconfidence
entire systematic
review
processare
from the
made. Large amounts of variability reduce
our
that
effects
initial steps of formulating the topic and building the
accurately measured. In meta-analysis,review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
variability arises from three sources—clinical
diversity,
methodological
di-a
Finding What
Works in Health
Care also proposes
framework for improving the quality of the science
versity, and statistical heterogeneity—which should be separately considunderpinning systematic reviews. This book will serve
ered in presentation and discussion (Fuaset
al.,resource
2010). for
Clinical
diversity
de- of
a vital
both sponsors
and producers
systematic reviews of interventions,
comparative effectiveness
scribes variability in study population characteristics,
and
research.
outcome ascertainments. Methodological diversity encompasses variability
in study design, conduct, and quality, such as blinding and concealment of
allocation. Statistical heterogeneity, relating to the variability in observed
treatment effects across studies, may occur because of random chance,
but may also arise from real clinical and methodological diversity and bias.
Assessing the amount of variability is fundamental to determining the
relevance of the individual studies to the SR’s research questions. It is also
×
key to choosing which statistical model to use in the quantitative synthesis.
Large
amounts
ofin
variability
may suggest
a poorly
formulatedRevi…
question or
Finding
What
Works
Health Care:
Standards
for Systematic
many sources of uncertainty that can influence effects. As noted above, if
the individual studies are so diverse in terms of populations, interventions,
comparators, outcomes, time lines, and/or
settings,
dataof will
not
Healthcare
decisionsummary
makers in search
reliable
information
that
compares
health
interventions
yield clinically
meaningful
conclusions
about the
effect
of an
intervention
Buy Paperback
| $61.00
increasingly turn to systematic reviews for the best
for important subgroups of the population
(West
al., 2010).
summary
of theet
evidence.
Systematic reviews identify,
assess,
and synthesize
the findings
similar
In general, quantifying heterogeneityselect,
helps
determine
whether
andofhow
but separate studies, and can help clarify what is
Buy Ebook
$48.99tests of the presence of heterothe data may be
but|specific
combined,
known and not known about the potential benefits
and harms
of drugs,
devices,of
andtheir
other poor
healthcare
geneity can be misleading and should not
be used
because
services. Systematic reviews can be helpful for
statistical properties
and because
MyNAP members
save an assumption of complete homogeneity
clinicians who want to integrate research findings into
10%unrealistic
online.
is nearly always
(Higgins et al.,
Graphical
representations
their2003).
daily practices,
for patients
to make well-informed
Login
or
Register
to
choices about their own care, for professional medical
of among-study variation such as forest plots can be informative (Figure 4societies and other organizations that develop clinical
save!
1) (Anzures-Cabrera and Higgins, 2010).practice guidelines.
When pooling is feasible, investigators typically use one of two statistical
Download Free PDF Too often systematic reviews are of uncertain or poor
techniques—fixed-effects or random-effects
models—to
analyze
and standards
intequality. There
are no universally
accepted
systematic reviews
variability
grate the data, depending on the extentforofdeveloping
heterogeneity.
Each leading
modeltohas
in how conflicts of interest and biases are handled,
strengths and limitations. A fixed-effects
theoverall
treathowmodel
evidenceassumes
is appraised,that
and the
scientific
rigor
of
the
process.
ment effect is the same for each study. A random-effects model assumes
that some heterogeneity is present andInacceptable,
andin the
can
be of
Finding What Works
Healthdata
Care the
Institute
(IOM) recommends
21 be
standards
pooled. Exploring the potential sourcesMedicine
of heterogeneity
may
moreforimdeveloping high-quality systematic reviews of
portant than a decision about the use of
fixed- or random-effects models.
comparative effectiveness research. The standards
address
entire
systematic
review process
from the
Although the committee does not believe
thattheany
single
statistical
techinitial steps of formulating the topic and building the
nique should be a methodological standard,
it is essential that the SR team
review team to producing a detailed final report that
clearly explain and justify the reasons why
it chose
theevidence
technique
actually
synthesizes
what the
shows and
where
knowledge gaps remain.
used.
Finding What Works in Health Care also proposes a
framework for improving the quality of the science
underpinning systematic reviews. This book will serve
as a vital resource for both sponsors and producers of
systematic reviews of comparative effectiveness
research.
Finding What Works in Health Care: Standards for Systematic Revi…
×
Healthcare decision makers in search of reliable
information that compares health interventions
Buy Paperback | $61.00
increasingly turn to systematic reviews for the best
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Buy Ebook | $48.99
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
services. Systematic reviews can be helpful for
MyNAP members save
clinicians who want to integrate research findings into
10% online.
their daily practices, for patients to make well-informed
Login or Register to
choices about their own care, for professional medical
societies and other organizations that develop clinical
save!
practice guidelines.


Download Free PDF
FIGURE 4-1 Forest plot.
SOURCE: Schriger et al. (2010).
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
in how conflicts of interest and biases are handled,
how evidence is appraised, and the overall scientific
rigor of the process.
In Finding What Works in Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
initial
steps
of formulating the topic
and building
In meta-analyses, the amount of withinand
between-study
variation
de-the
review team to producing a detailed final report that
termines how precisely study and aggregate
treatment
effects
areand
estimatsynthesizes
what the evidence
shows
where
knowledge gaps
remain. of their uncered. Estimates of effects without accompanying
measures
Statistical Uncertainty
tainty, such as confidence intervals, cannot
correctly
interpreted.
A forFindingbe
What
Works in Health
Care also proposes
a
framework
for
improving
the
quality
of
the
science
est plot can provide a succinct representation of the size and precision of
underpinning systematic reviews. This book will serve
individual study effects and aggregatedaseffects.
When effects are heterogea vital resource for both sponsors and producers of
neous, more than one summary effect may
be necessary
to fullyeffectiveness
describe
systematic
reviews of comparative
research.
the data. Measures of uncertainty should also be presented for estimates of
heterogeneity and for statistics that quantify relationships between treatment effects and sources of heterogeneity.
Between-study heterogeneity is common in meta-analysis because studies differ in their protocols, target populations, settings, and ages of included subjects. This type of heterogeneity provides evidence about potential variability in treatment effects. Therefore, heterogeneity is not a nui-
×
sance or an undesirable feature, but rather an important source of information
to Works
be carefully
analyzed
(Lau
et al., 1998).
of eliminating
Finding
What
in Health
Care:
Standards
for Instead
Systematic
Revi…
heterogeneity by restricting study inclusion criteria or scope, which can
limit the utility of the review, heterogeneity of effect sizes can be quantified, and related to aspects of study populations
or design
Healthcare decision
makersfeatures
in search ofthrough
reliable
information that
compares
health interventions
statistical techniques
as meta-regression,
which
associates
the size of
Paperback
| $61.00
 Buysuch
increasingly turn to systematic reviews for the best
treatment effects with effect modifiers.summary
Meta-regression
most useful
in
of the evidence.is
Systematic
reviews identify,
select, assess,
synthesize
the findings
of similar
explaining variation that occurs from sources
thatand
have
no effect
within
but separate studies, and can help clarify what is
Buy among
Ebook studies
| $48.99
studies, but big
effects
(e.g.,
use of randomization or dose
known and not known about the potential benefits
and harms of drugs,
devices,are
and exploratory,
other healthcare
employed). Except in rare cases, meta-regression
analyses
services. Systematic reviews can be helpful for
motivated byMyNAP
the need
to explain
members
save heterogeneity, and not by prespecificaclinicians who want to integrate research findings into
10% online.
tion in the protocol.
Meta-regression istheir
observational
inpatients
nature,
and well-informed
if the
daily practices, for
to make
Login
or
Register
to
choices about their own care, for professional medical
results of meta-regression are to be considered valid, they should be clinisocieties and other organizations that develop clinical
save!
cally plausible and supported by other practice
external
evidence. Because the
guidelines.
number of studies in a meta-regression is often small, the technique has
Download Free PDF Too often systematic reviews are of uncertain or poor
low power. The technique is subject to quality.
spurious
because
many
poTherefindings
are no universally
accepted
standards
developing systematic
reviews
to variability
tential covariates may be available, andfor
adjustments
to levels
of leading
signifiin how conflicts of interest and biases are handled,
cance may be necessary (Higgins and Thompson,
Users
should
also
how evidence is2004).
appraised,
and the
overall scientific
rigor of the in
process.
be careful of relationships driven by anomalies
one or two studies. Such
influential data do not provide solid evidence
strong
In Finding of
What
Works inrelationships.
Health Care the Institute of
Medicine (IOM) recommends 21 standards for
developing high-quality systematic reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
initial steps of formulating the topic and building the
previously, a detailed discussion
of meta-analysis methodolreview team to producing a detailed final report that
the scope of this report. synthesizes
There are
many
what
the evidence shows and where
knowledge gaps remain.
Research Trends in Meta-Analysis
As mentioned
ogy is beyond
Finding What Works in Health Care also proposes a
framework methods.
for improving Fortunately,
the quality of the metascience
unresolved questions regarding meta-analysis
underpinning systematic reviews. This book will serve
analysis methodological research is vibrant
and ongoing. Box 4-4 describes
as a vital resource for both sponsors and producers of
systematicand
reviews
of comparative
effectiveness
some of the research trends in meta-analysis
provides
relevant
referresearch.
ences for the interested reader.
Sensitivity of Conclusions
Meta-analysis entails combining information from different studies; thus,
the data may come from very different study designs. A small number of
studies in conjunction with a variety of study designs contribute to hetero-
×
geneity in results. Consequently, verifying that conclusions are robust to
small
changes
in the
data and
to changes
in modeling
assumptions
Finding
What
Works
in Health
Care:
Standards
for Systematic
Revi…solidifies the belief that they are robust to new information that could appear.
Without a sensitivity analysis, the credibility of the meta-analysis is
reduced.
Healthcare decision makers in search of reliable
information
health interventions
| $61.00
Results are considered
robust if small
changesthat
incompares
the metaanalytic
proto Buy Paperback
increasingly turn to systematic reviews for the best
col, in modeling assumptions, and in study
selection
do not
affect reviews
the consummary
of the evidence.
Systematic
identify,
select, assess,
synthesize
the findingsSensiof similar
clusions. Robust estimates increase confidence
inand
the
SR’s findings.
but separate studies, and can help clarify what is
Buyconclusions
Ebook | $48.99
tivity analyses
subject
to such
tests by perturbing these charknown and not known about the potential benefits
and harms of drugs, devices, and other healthcare
acteristics in various ways.
services. Systematic reviews can be helpful for
MyNAP
members
savefor example, assess whether the results
The sensitivity
analysis
could,
clinicians who want to integrate research findings into
change when10%
the online.
meta-analysis is rerun
leaving
one study
outtoatmake
a time.
their
daily practices,
for patients
well-informed
Login or Register to
choices about their own care, for professional medical
One statistical test for stability is to check that the predictive distribution
societies and other organizations that develop clinical
save!
of a new study from a meta-analysis with
oneguidelines.
of the studies omitted would
practice
include the results of the omitted study (Deeks et al., 2008). Failure to meet
Download Free PDF Too often systematic reviews are of uncertain or poor
this criterion implies that the result of quality.
the omitted
is unexpected
There are study
no universally
accepted standards
for
developing
systematic
reviews
leading to variability
given the remaining studies. Another common criterion is to determine
in how conflicts of interest and biases are handled,
whether the estimated average treatment
effect changes
how evidence
is appraised,substantially
and the overall scientific
rigor
of
the
process.
upon omission of one of the studies. A common definition of substantial
involves change in the determination ofInstatistical
significance
ofthe
the
sum-of
Finding What Works
in Health Care
Institute
(IOM) recommends
standards for
mary effect, although this definition is Medicine
problematic
because 21
a significance
developing high-quality systematic reviews of
threshold may be crossed with an unimportant
change in the magnitude or
comparative effectiveness research. The standards
addresssignificance
the entire systematic
process
from the
precision of the effect (i.e., loss of statistical
mayreview
result
from
initial steps of formulating the topic and building the
omission of a large study that reduces the
precision, but not the magnireview team to producing a detailed final report that
tude, of the effect).
synthesizes what the evidence shows and where
gaps remain.
In addition to checking sensitivity toknowledge
inclusion
of single studies, it is important to evaluate the effect of changes
in the
that
may
alter the
Finding
Whatprotocol
Works in Health
Care
also proposes
a
framework for improving the quality of the science
composition of the studies in the meta-analysis. Changes to the inclusion
underpinning systematic reviews. This book will serve
and exclusion criteria—such as the inclusion
non-English
literature
or
as a vital of
resource
for both sponsors
and producers
of
systematic
reviews ofnot
comparative
the exclusion of studies that enroll some
participants
in theeffectiveness
target
research.
population or the focus on studies with low risk of bias—may all modify results sufficiently to question robustness of inferences.
BOX 4-4
Research Trends in Meta-Analysis
Meta-analytic research is a dynamic and rapidly changing field.
The following
key areas
of research
with recommended
Finding What
Works indescribes
Health Care:
Standards
for Systematic
Revi…
citations for additional reading:
×
Prospective meta-analysis—In this approach, studies are
Healthcare decision makers in search of reliable
identified and
evaluated
prior
to
the results
of any health
individinformation
that compares
interventions
Buy Paperback | $61.00

increasingly
turn
to
systematic
reviews
ual studies being known. Prospective meta-analysis (PMA) for the best
summary of the evidence. Systematic reviews identify,
allows selection criteria and hypotheses
defined
pri- of similar
select, assess,to
andbe
synthesize
theafindings
butPMA
separate
studies,
and can help
clarify what is
ori to the trials
can
implement
stanEbookconcluded.
| $48.99
 Buy being
known and not known about the potential benefits
dardization across studies so that
heterogeneity is deand harms of drugs, devices, and other healthcare
creased.
In addition, small studies
thatSystematic
lack statistical
services.
reviews canpowbe helpful for
MyNAP members save
clinicians who want to integrate research findings into
er individually
if large studies are not
10% online.can be conducted
their daily practices, for patients to make well-informed
feasible.
See
example:
Berlinchoices
and Ghersi,
2005;
Login
or for
Register
to
about their2004,
own care,
for professional medical
societies
and other organizations
save!
Ghersi
et al., 2008; The Cochrane
Collaboration,
2010. that develop clinical
practice guidelines.
Too often
systematic
reviews are
uncertain or poor
Meta-regression—In
thisPDF
method,
potential
sources
of ofhetDownload Free
quality. There are no universally accepted standards
erogeneity are represented as predictors
in a regression
for developing systematic reviews leading to variability
model, thereby enabling estimation
theirofrelationship
in how of
conflicts
interest and biases are handled,
how evidence is appraised, and the overall scientific
with treatment effects. Such analyses are exploratory in the
rigor of the process.
majority of cases, motivated by the need to explain heteroIn Finding What Works in Health Care the Institute of
geneity. See for example: Schmid
et al., 2004; Smith et al.,
Medicine (IOM) recommends 21 standards for
1997; Sterne et al., 2002; Thompson
andhigh-quality
Higgins,systematic
2002. reviews of
developing
comparative effectiveness research. The standards
address the entire systematic review process from the
Bayesian methods in meta-analysis—In
these approaches,
initial steps of formulating the topic and building the
as in Bayesian approaches in other
both the
datafinal report that
reviewsettings,
team to producing
a detailed
synthesizes what the evidence shows and where
and parameters in the meta-analytic model are considered
knowledge gaps remain.
random variables. This approach allows the incorporation
Finding What Works in Health Care also proposes a
of prior information into subsequent
analyses, and may be
framework for improving the quality of the science
more flexible in complex situations
than standard
methodunderpinning
systematic reviews.
This book will serve
as
a
vital
resource
for
both
sponsors
ologies. See for example: Berry et al., 2010; O’Rourke and and producers of
systematic reviews of comparative effectiveness
Altman, 2005; Schmid, 2001; Smith
et al., 1995; Sutton and
research.
Abrams, 2001; Warn et al., 2002.
Meta-analysis of multiple treatments—In this setting, direct treatment comparisons are not available, but an indirect comparison through a common comparator is. Multiple treatment models, also called mixed comparison models
or network meta-analysis, may be used to more efficiently
model treatment comparisons of interest. See for example:
Cooper
etin
al.,Health
2009; Care:
Dias etStandards
al., 2010; Salanti
et al., 2009.
Finding What
Works
for Systematic
Revi…
×
Individual participant data meta-analysis—In some cases,
study data may include outcomes,
treatments, and characHealthcare decision makers in search of reliable
information
that compares
healthsuch
interventions
teristics
individual
participants.
Meta-analysis
with
Paperback
| $61.00
of Buy
increasingly turn to systematic reviews for the best
individual participant data (IPD)summary
offersofmany
advantages
the evidence. Systematic reviews identify,
over meta-analysis of aggregateselect,
studylevel
data.
See the
forfindings
ex- of similar
assess, and
synthesize
but separate studies, and can help clarify what is
ample:
Berlin
et Ebook
al., 2002;
Simmonds
et al., 2005; Smith et
Buy
| $48.99
known and not known about the potential benefits
al., 1997; Sterne et al., 2002; Stewart,
1995;
Thompson
and
and harms
of drugs,
devices, and
other healthcare
services.
Systematic
reviews
can
be
helpful for
Higgins,
2002;
Tierney
et al., 2000.
MyNAP
members
save
10% online.
Login or Register to
save!
clinicians who want to integrate research findings into
their daily practices, for patients to make well-informed
choices about their own care, for professional medical
societies and other organizations that develop clinical
practice guidelines.
Another good practice is to evaluate sensitivity
to choices about outcome
Download Free PDF Too often systematic reviews are of uncertain or poor
quality.
There are
universally
accepted
metrics and statistical models. While one
metric
andnoone
model
may standards
in the
for developing systematic reviews leading to variability
end be chosen as best for scientific reasons,
results that are highly model
in how conflicts of interest and biases are handled,
how evidence
appraised,
and theprone
overall scientific
dependent require more trust in the modeler
andismay
be more
to
rigor of the process.
being overturned with new data. In any case, support for the metrics and
In Finding What Works in Health Care the Institute of
models chosen should be provided.
Medicine (IOM) recommends 21 standards for
Meta-analyses are also frequently sensitive
to assumptions about missdeveloping high-quality systematic reviews of
ing data. In meta-analysis, missing datacomparative
include not
only missing
outcomes
effectiveness
research. The
standards
address the entire systematic review process from the
or predictors, but also missing variances and correlations needed when
initial steps of formulating the topic and building the
constructing weights based on study precision.
with any
statistical
review team As
to producing
a detailed
final report that
synthesizes
what
the
evidence
shows
and where
analysis, missing data pose two threats: reduced power and bias. Because
knowledge gaps remain.
the number of studies is often small, loss of even a single study’s data can
Finding What
Works in Health
Careaalso
proposes a
seriously affect the ability to draw conclusive
inferences
from
metaframework for improving the quality of the science
analysis. Bias poses an even more dangerous
problem.
Seemingly
underpinning
systematic
reviews. Thisconclubook will serve
as
a
vital
resource
for
both
sponsors
and
producers of
sive analyses may give the wrong answer if studies that were excluded—
systematic reviews of comparative effectiveness
because of missing data—differ from the
studies that supplied the data. The
research.
conclusion that the treatment improved one outcome, but not another,
may result solely from the different studies used. Interpreting such results
requires care and caution.
RECOMMENDED STANDARDS FOR METAANALYSIS
×
The committee recommends the following standards and elements of performance
conducting
theCare:
quantitative
synthesis.
Finding
What for
Works
in Health
Standards
for Systematic Revi…
Standard 4.3—Decide if, in addition to a qualitative analysis, the
systematic review will include a quantitative analysis (metaHealthcare decision makers in search of reliable
analysis)
information that compares health interventions
Buy Paperback | $61.00

increasingly turn to systematic reviews for the best
Required element:
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
4.3.1 Explain why a pooled estimatebut
might be useful to decision
Buy Ebook | $48.99 separate studies, and can help clarify what is
known and not known about the potential benefits
makers
and harms of drugs, devices, and other healthcare
services. Systematic reviews can be helpful for
savea meta-analysis, then do the
StandardMyNAP
4.4—Ifmembers
conducting
clinicians who want to integrate research findings into
their daily practices, for patients to make well-informed
following:10% online.
Login or Register to
choices about their own care, for professional medical
Required elements:
societies and other organizations that develop clinical
save!
practice guidelines.

4.4.1 Use expert methodologists to develop, execute, and peer reDownload Free PDF Too often systematic reviews are of uncertain or poor
view the meta-analyses
quality. There are no universally accepted standards
developing
systematic reviews leading to variability
4.4.2 Address heterogeneity amongforstudy
effects
in how conflicts of interest and biases are handled,
4.4.3 Accompany all estimates withhow
measures
statistical
evidence isof
appraised,
and the overall scientific
rigor of the process.
uncertainty
In Finding What
Works in Health
Careprotothe Institute of
4.4.4 Assess the sensitivity of conclusions
to changes
in the
Medicine (IOM) recommends 21 standards for
col, assumptions, and study selection
(sensitivity
analysis)
developing high-quality
systematic
reviews of
comparative effectiveness research. The standards
address the entire systematic review process from the
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge gaps remain.
Rationale
Finding What Works in Health Care also proposes a
framework
for improving
the qualityreproof the science
A meta-analysis is usually desirable in an
SR because
it provides
underpinning systematic reviews. This book will serve
ducible summaries of the individual study
results
and
offer of
as a vital
resource
forhas
both potential
sponsors andto
producers
systematic
reviewsstudies.
of comparative
effectiveness
valuable insights into the patterns of results
across
However,
many
research.
published analyses have important methodological shortcomings and lack
scientific rigor (Bailar, 1997; Gerber et al., 2007; Mullen and Ramirez, 2006).
One must always look beyond the simple fact that an SR contains a metaanalysis to examine the details of how it was planned and conducted. A
strong meta-analysis emanates from a well-conducted SR and features and
clearly describes its subjective components, scrutinizes the individual
studies for sources of heterogeneity, and tests the sensitivity of the find-
ings to changes in the assumptions and set of studies (Greenland, 1994;
Walker
et al.,
2008).
Finding
What
Works
in Health Care: Standards for Systematic Revi…
REFERENCES
Healthcare decision makers in search of reliable
×
information that compares health interventions
Buy Paperback | $61.00
AAN (American
Academy of Neurology).increasingly
2004. Clinical
practice
guidelines
turn to systematic
reviews
for the best
summary
of
the
evidence.
Systematic
reviews
process manual. http://www.aan.com/globals/axon/assets/3749.pdfidentify,
select, assess, and synthesize the findings of similar
(accessed February 1, 2011).
but separate studies, and can help clarify what is
Buy Ebook | $48.99

and not knownguideline
about the potential
benefits
ACCF/AHA. 2009. Methodology manualknown
for ACCF/AHA
writing
and harms of drugs, devices, and other healthcare
committees.
services. Systematic reviews can be helpful for
MyNAP members save
clinicians who want to integrate research findings into
http://www.americanheart.org/downloadable/heart/12378388766452009M
10% online.
their daily practices, for patients to make well-informed
(accessed July
29,
Login
or2009).
Register to
choices about their own care, for professional medical
ACCP (American
College
of
Chest
Physicians).
2009.
ACCP grading
syssocieties and
otherThe
organizations
that develop
clinical
save!
practice guidelines.
tem for guideline recommendations. http://www.chestnet.org/education/hsp/gradingSystem.php
February
1, 2011).
systematic
reviews are of uncertain or poor
Download Free(accessed
PDF Too often
quality. There are no universally accepted standards
Ammerman, A., M. Pignone, L. Fernandez,
K. Lohr, A. D. Jacobs, C. Nester, T.
for developing systematic reviews leading to variability
Orleans, N. Pender, S. Woolf, S. F. Sutton,
J. Lux,
and L.and
Whitener.
2002.
in how L.
conflicts
of interest
biases are handled,
Counseling to promote a healthy diet. how evidence is appraised, and the overall scientific
rigor of the process.
http://www.ahrq.gov/downloads/pub/prevent/pdfser/dietser.pdf (acIn Finding What Works in Health Care the Institute of
cessed September 26, 2010).
Medicine (IOM) recommends 21 standards for
Anello, C., and J. L. Fleiss. 1995. Exploratory
or analytic
developing
high-qualitymeta-analysis:
systematic reviews of
comparative
The standards
Should we distinguish between them?
Journal effectiveness
of Clinicalresearch.
Epidemiology
address the entire systematic review process from the
48(1):109–116.
initial steps of formulating the topic and building the
team
to producing
a detailed for
final metareport that
Anzures-Cabrera, J., and J. P. T. Higgins.review
2010.
Graphical
displays
synthesizes what the evidence shows and where
analysis: An overview with suggestions
for practice. Research Synthesis
knowledge gaps remain.
Methods 1(1):66–89.
Finding What Works in Health Care also proposes a
Atkins, D. 2007. Creating and synthesizing
evidence with decision makers
framework for improving the quality of the science
in mind: Integrating evidence from clinical
trials
and other
study
designs.
underpinning
systematic
reviews.
This book
will serve
as a vital resource for both sponsors and producers of
Medical Care 45(10 Suppl 2):S16–S22.
systematic reviews of comparative effectiveness
Atkins, D., D. Best, P. A. Briss, M. Eccles,research.
Y. Falck-Ytter, S. Flottorp, and
GRADE Working Group. 2004a. Grading quality of evidence and strength
of recommendations. BMJ 328(7454):1490–1497.
Atkins, D., M. Eccles, S. Flottorp, G. Guyatt, D. Henry, S. Hill, A. Liberati, D.
O’Connell, A. D. Oxman, B. Phillips, H. Schünemann, T. T. Edejer, G. Vist, J.
Williams, and the GRADE Working Group. 2004b. Systems for grading
the quality of evidence and the strength of recommendations I: Critical
appraisal of existing approaches. BMC Health Services Research 4(1):38.
×
Finding What Works in Health Care: Standards for Systematic Revi…
Bailar, J. C., III. 1997. The promise and problems of meta-analysis. New England Journal of Medicine 337(8):559–561.
Balshem, H., M. Helfand, H. J. Schünemann, A. D. Oxman, R. Kunz, J. Brozek,
Healthcare decision makers in search of reliable
G. E. Vist, Y. Falck-Ytter, J. Meerpohl,information
S. Norris, and G. H. Guyatt. 2011.
Buy Paperback | $61.00 that compares health interventions

GRADE guidelines: 3. Rating the quality
of evidence.
Journalreviews
of Clinical
increasingly
turn to systematic
for the best
summary of the evidence. Systematic reviews identify,
Epidemiology (In press).
select, assess, and synthesize the findings of similar
Berlin, J. A., J. Santanna, C. H. Schmid, L.butA.separate
Szczech,
H.and
I. Feldman,
and
the
studies,
can help clarify
what
is
Buy Ebook | $48.99

known
and
not
known
about
the
potential
benefits
Anti-lymphocyte Antibody Induction Therapy Study Group. 2002. Indiand harms of drugs, devices, and other healthcare
vidual patient versus group-level dataservices.
meta-regressions
the
investiSystematic reviewsfor
can be
helpful
for
MyNAP members save
who want
integrate
findings into
gation of treatment
effect modifiers: clinicians
Ecological
biasto rears
itsresearch
ugly head.
10% online.
their daily practices, for patients to make well-informed
Statistics inLogin
Medicine
21(3):371–387.
or Register
to
choices about their own care, for professional medical
societies
and other organizations
that develop
clinical
Berlin, J., andsave!
D. Ghersi. 2004. Prospective
meta-analysis
in dentistry.
The
practice guidelines.
Journal of Evidence-Based Dental Practice 4(1):59–64.
TooRegistries
often systematic
reviews
are of uncertain
or poor
———. 2005. Preventing
publication
and
prospective
metaDownload
Free PDFbias:
quality. There are no universally accepted standards
analysis. Publication bias in meta-analysis:
Prevention, assessment and
for developing systematic reviews leading to variability
adjustments, edited by H. R. Rothstein,
A. J.conflicts
Sutton,
and M.
in how
of interest
andBorenstein,
biases are handled,
how evidence is appraised, and the overall scientific
pp. 35–48.
rigor of the process.
Berry, S., K. Ishak, B. Luce, and D. Berry. 2010. Bayesian meta-analyses for
In Finding What Works in Health Care the Institute of
comparative effectiveness and informing
coverage decisions. Medical
Medicine (IOM) recommends 21 standards for
Care 48(6):S137.
developing high-quality systematic reviews of
comparative effectiveness
research. The
standards
Borenstein, M. 2009. Introduction to meta-analysis
. West Sussex,
U.K.:
John
address the entire systematic review process from the
Wiley & Sons.
initial steps of formulating the topic and building the
team to
a detailed
report that
Brozek, J. L., E. A. Aki, P. Alonso-Coelle,review
D. Lang,
R.producing
Jaeschke,
J. W.final
Williams,
synthesizes what the evidence shows and where
B. Phillips, M. Lelgemann, A. Lethaby,knowledge
J. Bousquet,
G. Guyatt, H. J.
gaps remain.
Schünemann, and the GRADE Working Group. 2009. Grading quality of
Finding What Works in Health Care also proposes a
evidence and strength of recommendations
guideframeworkin
forclinical
improving practice
the quality of
the science
underpinning
systematicand
reviews.
This book
will serve
lines: Part 1 of 3. An overview of the GRADE
approach
grading
quality
as a vital resource for both sponsors and producers of
of evidence about interventions. Allergy
64(5):669–677.
systematic reviews of comparative effectiveness
CEBM (Centre for Evidence-based Medicine).
research. 2009. Oxford Centre for Evidence-based Medicine—Levels of evidence (March 2009). http://www.cebm.net/index. aspx?o=1025 (accessed February 1, 2011).
Chalmers, I., M. Adams, K. Dickersin, J. Hetherington, W. Tarnow-Mordi, C.
Meinert, S. Tonascia, and T. C. Chalmers. 1990. A cohort study of summary reports of controlled trials. JAMA 263(10):1401–1405.
Chou, R., N. Aronson, D. Atkins, A. S. Ismaila, P. Santaguida, D. H. Smith, E.
Whitlock, T. J. Wilt, and D. Moher. 2010. AHRQ series paper 4: Assessing
×
harms when comparing medical interventions: AHRQ and the Effective
Health
Program.
Journal
Clinical Epidemiology
63(5):502–512.
Finding
WhatCare
Works
in Health
Care:ofStandards
for Systematic
Revi…
Cochrane Collaboration. 2010. Cochrane prospective meta-analysis methods
group. http://pma.cochrane.org/ (accessed January 27, 2011).
Coleman, C. I., R. Talati, and C. M. White.
2009. decision
A clinician’s
on
Healthcare
makers inperspective
search of reliable
information that
compares
health interventions
rating the strength
evidence in|a$61.00
systematic
review.
Pharmacotherapy
 BuyofPaperback
increasingly turn to systematic reviews for the best
29(9):1017–1029.
summary of the evidence. Systematic reviews identify,
select,
assess, and synthesize
the findingsSerof similar
COMPUS (Canadian Optimal Medication
Prescribing
and Utilization
but separate studies, and can help clarify what is
| $48.99
vice). 2005. Evaluation
tools for
Canadian
Optimal Medication Prescribing
 Buy Ebook
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
and Utilization Service.
services. Systematic reviews can be helpful for
MyNAP members save
http://www.cadth.ca/media/compus/pdf/COMPUS_Evaluaclinicians who want to integrate research findings into
10% online.
tion_Methodology_final_e.pdf
(accessed
September
2010).
their daily
practices, for 6,
patients
to make well-informed
Login or Register to
choices about their own care, for professional medical
Cooper, H. M., L. V. Hedges, and J. C. Valentine. 2009. The handbook of resocieties and other organizations that develop clinical
save!
search synthesis and meta-analysis, 2nd
ed.guidelines.
New York: Russell Sage
practice
Foundation.
Download Free PDF Too often systematic reviews are of uncertain or poor
Cooper, N., A. Sutton, D. Morris, A. Ades,
andThere
N. Welton.
2009.accepted
Addressing
quality.
are no universally
standards
for
developing
systematic
reviews
leading
to
between-study heterogeneity and inconsistency in mixed treatment variability
in how conflicts of interest and biases are handled,
comparisons: Application to stroke prevention
treatments
how evidence is
appraised, andin
theindividuals
overall scientific
rigor
of
the
process.
with non-rheumatic atrial fibrillation. Statistics in Medicine 28(14):1861–
1881.
In Finding What Works in Health Care the Institute of
Medicine 2009.
(IOM) recommends
21 standards
for
CRD (Centre for Reviews and Dissemination).
Systematic
reviews:
developing high-quality systematic reviews of
CRD’s guidance for undertaking reviews
in health care. York, U.K.: York
comparative effectiveness research. The standards
address the entire systematic review process from the
Publishing Services, Ltd.
initial steps of formulating the topic and building the
review team to producing a detailed final report that
synthesizes what the evidence shows and where
knowledge
gaps remain. effects is unrelibased
on standardized
Cummings, P. 2004. Meta-analysis
able. Archives of Pediatrics & Adolescent
Medicine
Finding
What Works158(6):595–597.
in Health Care also proposes a
framework
for
improving
quality of the
science
Deeks, J., J. Higgins, and D. Altman, eds. 2008. Chapter 9: the
Analysing
data
underpinning systematic reviews. This book will serve
and undertaking meta-anayses. In Cochrane
handbook for systematic reas a vital resource for both sponsors and producers of
reviews
comparative
effectiveness
views of interventions, edited by J. P. systematic
T. Higgins
andofS.
Green. Chichester,
research.
UK: John Wiley & Sons.
Devereaux, P. J., D. Heels-Ansdell, C. Lacchetti, T. Haines, K. E. Burns, D. J.
Cook, N. Ravindran, S. D. Walter, H. McDonald, S. B. Stone, R. Patel, M.
Bhandari, H. J. Schünemann, P. T. Choi, A. M. Bayoumi, J. N. Lavis, T. Sullivan, G. Stoddart, and G. H. Guyatt. 2004. Payments for care at private
for-profit and private not-for-profit hospitals: A systematic review and
meta-analysis. Canadian Medical Association Journal 170(12):1817–1824.
×
Dias, S., N. Welton, D. Caldwell, and A. Ades. 2010. Checking consistency in
mixed
meta
analysis.for
Statistics
in Medicine
Finding
Whattreatment
Works incomparison
Health Care:
Standards
Systematic
Revi… 29(7
8):932–944.
Dickersin, K. 1990. The existence of publication bias and risk factors for its
occurrence. JAMA 263(10):1385–1389.Healthcare decision makers in search of reliable
information
that Chan,
compares
interventions
Dwan, K., D. G.
Altman,
J. A. Arnaiz, |
J. Bloom,
A.-W.
E.health
Cronin,
E.
Buy Paperback
$61.00
increasingly turn to systematic reviews for the best
Decullier, P. J. Easterbrook, E. Von Elm,
C. Gamble,
D. Ghersi,
J. P.
A. Ioansummary
of the evidence.
Systematic
reviews
identify,
select, assess,
and synthesize
the findings
similar
nidis, J. Simes, and P. R. Williamson. 2008.
Systematic
review
of theofembut separate studies, and can help clarify what is
Ebook
| $48.99bias and outcome reporting bias.
pirical evidence
study
publication
 ofBuy
known and not known about the potential benefits
and harms of drugs, devices, and other healthcare
PLoS ONE 3(8):e3081.
services. Systematic reviews can be helpful for
Ebell, M. H., J.MyNAP
Siwek,members
B. D. Weiss,
save S. H. Woolf, J. Susman, B. Ewigman, and
clinicians who want to integrate research findings into
10%
online.
M. Bowman.
2004.
Strength of recommendation
taxonomy
A patheir daily practices,
for patients(SORT):
to make well-informed
Login
or
Register
to
choices about their own care, for professional medical
tient-centered approach to grading evidence in medical literature.
societies and other organizations that develop clinical
save!
American Family Physician 69(3):548–556.
practice guidelines.
Editors. 2005. Reviews: Making sense of an often tangled skein of evidence.
Download Free PDF Too often systematic reviews are of uncertain or poor
Annals of Internal Medicine 142(12 Pt quality.
1):1019–1020.
There are no universally accepted standards
for
developing
systematic reviews
leading
to variability
Egger, M., G. D. Smith, and D. G. Altman. 2001. Systematic
reviews
in health
in how conflicts of interest and biases are handled,
care: Meta-analysis in context. London,
BMJ
Publishing
Group.
howU.K.:
evidence
is appraised,
and the
overall scientific
rigor
of
the
process.
Falck-Ytter, Y., H. Schünemann, and G. Guyatt. 2010. AHRQ series commentary 1: Rating the evidence in comparative
reviews.
In Finding Whateffectiveness
Works in Health Care
the Institute of
Medicine
(IOM)
recommends
21
standards
for
Journal of Clinical Epidemiology 63(5):474–475.
developing high-quality systematic reviews of
Faraday, M., H. Hubbard, B. Kosiak, andcomparative
R. Dmochowski.
2009. Staying at
effectiveness research. The standards
address
the entire systematic
review
process
from the
the cutting edge: A review and analysis
of evidence
reporting
and
gradinitial steps of formulating the topic and building the
ing; The recommendations of the American
Urological Association. BJU
review team to producing a detailed final report that
International 104(3): 294–297.
synthesizes what the evidence shows and where
knowledge gaps remain.
Federal Coordinating Council for Comparative
Effectiveness Research.
2009. Report to the President and the Finding
Congress
Available
What .Works
in Healthfrom
Care also proposes a
framework for improving the quality of the science
http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf.
underpinning systematic reviews. This book will serve
Ferreira, P. H., M. L. Ferreira, C. G. Maher,
R. sponsors
D. Herbert,
and of
as a K.
vitalRefshauge,
resource for both
and producers
systematic “levels
reviews ofof
comparative
effectiveness
J. Latimer. 2002. Effect of applying different
evidence”
criteria
research.
on conclusions of Cochrane reviews of interventions for low back pain.
Journal of Clinical Epidemiology 55(11):1126–1129.
Fu, R., G. Gartlehner, M. Grant, T. Shamliyan, A. Sedrakyan, T. J. Wilt, L.
Griffith, M. Oremus, P. Raina, A. Ismaila, P. Santaguida, J. Lau, and T. A.
Trikalinos. 2010. Conducting quantitative synthesis when comparing
medical interventions: AHRQ and the Effective Health Care Program. In
Methods guide for comparative effectiveness reviews, edited by Agency for
×
Healthcare Research and Quality. http://www.effectivehealthcare.ahrq.gov/index.cfm/search-for-guidesreviews-and-reports/?
Finding
What Works in Health Care: Standards for Systematic Revi…
pageaction=displayProduct&productID=554 (accessed January 19, 2011).
Gerber, S., D. Tallon, S. Trelle, M. Schneider, P. Jüni, and M. Egger. 2007.
Bibliographic study showed improving
methodology
of meta-analyses
Healthcare
decision makers
in search of reliable
information
that compares
healthEpidemiology
interventions
published in
leading
1993–2002.
Journal
of Clinical
Buyjournals:
Paperback
| $61.00
increasingly turn to systematic reviews for the best
60(8):773–780.
summary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Buy Ebook | $48.99
known and not known about the potential benefits
Ghersi, D., J. Berlin, and L. Askie, eds. 2008.
Chapter
19:devices,
Prospective
and harms
of drugs,
and other metahealthcare
services.
Systematic
reviews
can
be
helpful
for
analysis. Edited
J. Higgins
and S. Green, Cochrane handbook for sysMyNAPbymembers
save
clinicians who want to integrate research findings into
tematic reviews
of interventions. Chichester,
UK: John
Wiley to&make
Sons.
10% online.
their daily practices,
for patients
well-informed
Login
or
Register
to
choices
about
their
own
care,
for
professional
medical
Glasziou, P., J. Vandenbroucke, and I. Chalmers. 2004. Assessing the quality
societies and other organizations that develop clinical
of research.save!
BMJ 328(7430):39–41.
practice guidelines.

Gluud, L. L. 2006. Bias in clinical intervention research. American Journal of
Download Free PDF Too often systematic reviews are of uncertain or poor
Epidemiology 163(6):493–501.
quality. There are no universally accepted standards
for developing
systematic
reviews leading
to variability
GRADE Working Group. 2010. Organizations
that have
endorsed
or that
are
in how conflicts of interest and biases are handled,
using GRADE. http://www.gradeworkinggroup.org/society/index.htm
how evidence is appraised, and the overall scientific
(accessed September 20, 2010).
rigor of the process.
Greenland, S. 1994. Invited commentary:
A critical look at some popular
In Finding What Works in Health Care the Institute of
meta-analytic methods. American Journal
of(IOM)
Epidemiology
140(3):290–
Medicine
recommends 21
standards for
developing high-quality systematic reviews of
296.
comparative effectiveness research. The standards
Guirguis-Blake, J., N. Calonge, T. Miller,address
A. Siu,theS.entire
Teutsch,
E. Whitlock,
systematic
review processand
from the
steps
of formulating
the processes
topic and building
for the U.S. Preventive Services Task initial
Force.
2007.
Current
of the
review team to producing a detailed final report that
the U.S. Preventive Services Task Force:
Refining
recomsynthesizes
what evidence-based
the evidence shows and
where
knowledgeMedicine
gaps remain.147:117–122.
mendation development. Annals of Internal
Guyatt, G. H., D. L. Sackett, J. C. Sinclair,
R. Hayward,
D.Health
J. Cook,
and
R. J. a
Finding
What Works in
Care also
proposes
framework
for improving
the quality
of the
science
Cook. 1995. Users’ guides to the medical
literature:
A method
for
grading
underpinning systematic reviews. This book will serve
health care recommendations. JAMA as
274(22):1800–1804.
a vital resource for both sponsors and producers of
reviewsand
of comparative
effectiveness
Guyatt, G., H. J. Schünemann, D. Cook, systematic
R. Jaeschke,
S. Pauker.
2004.
research.
Applying the grades of recommendation for antithrombotic and thrombolytic therapy: The seventh ACCP conference on antithrombotic and
thrombolytic therapy. Chest 126(3 Suppl):179S–187S.
Guyatt, G., A. D. Oxman, E.A. Akl, R. Kunz, G. Vist, J. Brozek, S. Norris, Y. Falck-Ytter, P. Glasziou, H. deBeer, R. Jaeschke, D. Rind, J. Meerpohl, P.
Dahm, and H. J. Schünemann. 2010. GRADE guidelines 1. Introduction—
×
GRADE evidence profiles and summary of findings tables. Journal of
Clinical
Epidemiology
(In press).
Finding
What Works
in Health
Care: Standards for Systematic Revi…
Harris, R. P., M. Helfand, S. H. Woolf, K. N. Lohr, C. D. Mulrow, S. M.
Teutsch, D. Atkins, and the Methods Work Group Third U. S. Preventive
Services Task Force. 2001. Current methods
the U.S.
Preventive
SerHealthcareof
decision
makers
in search of reliable
information
that compares
health
vices Task Force:
A review
of the process.
American
Journal
ofinterventions
Preventive
Paperback
| $61.00
 Buy
increasingly turn to systematic reviews for the best
Medicine 20(3 Suppl):21–35.
summary of the evidence. Systematic reviews identify,
select,
assess, andand
synthesize
the findings
similar
Helfand, M. 2005. Using evidence reports:
Progress
challenges
inofevibut separate studies, and can help clarify what is
Buy Ebook
| $48.99
dence-based
decision
making.
Healthknown
Affairs
24(1):123–127.
and not known about the potential benefits
and harms
of drugs, devices,
other
healthcare
HHS (U.S. Department of Health and Human
Services).
2010.and
The
Sentinel
services. Systematic reviews can be helpful for
Initiative: AMyNAP
national
strategy
for monitoring medical product safety. Availmembers
save
clinicians who want to integrate research findings into
able from 10% online.
their daily practices, for patients to make well-informed
Login
or
Register
to
choices about their own care, for professional medical
http://www.fda.gov/Safety/FDAsSentinelInitiative/ucm089474.htm.
societies and other organizations that develop clinical
save!
Higgins, J. P. T., and S. G. Thompson. 2004.
Controlling
the risk of spurious
practice
guidelines.
findings from meta-regression. Statistics in Medicine 23(11):1663–1682.
Download Free PDF Too often systematic reviews are of uncertain or poor
Higgins, J. P. T., S. G. Thompson, J. J. Deeks,
quality.and
ThereD.
areG.
noAltman.
universally2003.
acceptedMeastandards
for
developing
systematic
reviews
leading
to
variability
suring inconsistency in meta-analyses. BMJ 327(7414):557–560.
in how conflicts of interest and biases are handled,
Hopewell, S., K. Loudon, M. J. Clarke, A.how
D. evidence
Oxman,is appraised,
and K. Dickersin.
2009.
and the overall
scientific
rigor
of
the
process.
Publication bias in clinical trials due to statistical significance or direction of trial results (Review). Cochrane
Database
of Systematic
In Finding
What Works
in Health CareReviews
the Institute of
Medicine
(IOM)
recommends
21
standards
for
1:MR000006.
developing high-quality systematic reviews of
Hopewell, S., J. Clarke Mike, L. Stewart,comparative
and J. Tierney.
2008. Time to publieffectiveness research. The standards
address the
entire systematic
reviewof
process
from the
cation for results of clinical trials (Review).
Cochrane
Database
Systeminitial steps of formulating the topic and building the
atic Reviews (2).
review team to producing a detailed final report that
Hsieh, C., A. H. Adams, J. Tobis, C. Hong,synthesizes
C. Danielson,
K. Platt,
F. Hoehler,
what the evidence
shows
and where S.
knowledge gaps remain.
Reinsch, and A. Rubel. 2002. Effectiveness
of four conservative treatments for subacute low back pain: A randomized
clinical
Spine
Finding What Works
in Health trial.
Care also
proposes a
framework for improving the quality of the science
27(11):1142–1148.
underpinning systematic reviews. This book will serve
ICSI (Institute for Clinical Systems Improvement).
2003.
Evidence
grading
as a vital resource
for both
sponsors and
producers of
systematic reviews of comparative effectiveness
system. http://www.icsi.org/evidence_grading_system_6/evidence_research.
grading_system_pdf_.html (accessed September 8, 2009).
IOM (Institute of Medicine). 2009. Initial national priorities for comparative
effectiveness research. Washington, DC: The National Academies Press.
Kirkham, J. J., K. M. Dwan, D. G. Altman, C. Gamble, S. Dodd, R. Smyth, and
P. R. Williamson. 2010. The impact of outcome reporting bias in ran-
×
domised controlled trials on a cohort of systematic reviews. BMJ
340:c365.
Finding
What Works in Health Care: Standards for Systematic Revi…
Lau, J., J. P. A. Ioannidis, and C. H. Schmid. 1998. Summing up evidence: One
answer is not always enough. Lancet 351(9096):123–127.
Lefebvre, C., E. Manheimer, and J. Glanville.
2008.
Chapter
Searching
for
Healthcare
decision
makers 6:
in search
of reliable
informationreviews
that compares
health interventions
Paperback
$61.00
studies. In Cochrane
for|systematic
of interventions
,
 Buyhandbook
increasingly turn to systematic reviews for the best
edited by J. P. T. Higgins and S. Green.
Chichester,
UK: John
Wiley
& identify,
summary
of the evidence.
Systematic
reviews
select, assess, and synthesize the findings of similar
Sons.
but separate studies, and can help clarify what is
Buy2004.
Ebook
| $48.99
Lu, G., and A. E.
Combination
of direct and indirect evidence in
Ades.
known and not known about the potential benefits
and harms
of drugs, devices,
and other healthcare
mixed treatment comparisons. Statistics
in Medicine
23(20):3105–3124.
services. Systematic reviews can be helpful for
Mullen, P. D.,MyNAP
and G.members
Ramirez.save
2006. Theclinicians
promise
and pitfalls of systematic
who want to integrate research findings into
10% online.
reviews. Annual
Review of Public Health
their27:81–102.
daily practices, for patients to make well-informed
Login or Register to
choices about their own care, for professional medical
Mulrow, C. D., and K. N. Lohr. 2001. Proof and policy from medical research
societies and other organizations that develop clinical
save!
evidence. Journal of Health Politics Policy
and
Law 26(2):249–266.
practice
guidelines.
Mulrow, C., P. Langhorne, and J. Grimshaw. 1997. Integrating heterogeneous
Download Free PDF Too often systematic reviews are of uncertain or poor
pieces of evidence in systematic reviews.
Medicine
quality.Annals
There areof
noInternal
universally accepted
standards
for
developing
systematic
reviews
leading
to
variability
127(11):989–995.
in how conflicts of interest and biases are handled,
NCCN (National Comprehensive Cancer
Network).
2008. About
NCCN
how
evidence is appraised,
and thethe
overall
scientific
rigor
of
the
process.
clinical practice guidelines in oncology. http://www.nccn.org/professionals/physician_gls/about.asp (accessed
September
8, 2009).
In Finding What
Works in Health
Care the Institute of
Medicine
(IOM)
recommends
21
for
Norris, S., D. Atkins, W. Bruening, S. Fox, E. Johnson, R. Kane, S.standards
C. Morton,
developing high-quality systematic reviews of
M. Oremus, M. Ospina, G. Randhawa,comparative
K. Schoelles,
P. Shekelle, and M.
effectiveness research. The standards
address the
entire systematic
review process
from the
Viswanathan. 2010. Selecting observational
studies
for comparing
medinitial steps of formulating the topic and building the
ical interventions. In Methods guide for
comparative effectiveness reviews,
review team to producing a detailed final report that
edited by Agency for Healthcare Research
andwhat
Quality.
http:/
/www.efsynthesizes
the evidence
shows
and where
knowledge gaps remain.
fectivehealthcare.ahrq.gov/index.cfm/search-for-guides-reviews-andreports/?pageaction=displayProduct&productID=454
(accessed
Januarya
Finding What Works in Health
Care also proposes
framework for improving the quality of the science
19, 2011).
underpinning systematic reviews. This book will serve
NZGG (New Zealand Guidelines Group).as2007.
Handbook
forsponsors
the preparation
a vital resource
for both
and producers of
systematic
reviews of comparative
effectiveness
of explicit evidence-based clinical practice
guidelines.
http://www.nzgresearch.
g.org.nz/download/files/nzgg_guideline_handbook.pdf (accessed February 1, 2011).
O’Rourke, K., and D. Altman. 2005. Bayesian random effects meta-analysis
of trials with binary outcomes: Methods for the absolute risk difference
and relative risk scales Statistics in Medicine 24(17):2733–2742.
Owens, D. K., K. N. Lohr, D. Atkins, J. R. Treadwell, J. T. Reston, E. B. Bass, S.
Chang, and M. Helfand. 2010. Grading the strength of a body of evidence
×
when comparing medical interventions: AHRQ and the Effective Health
Care
Program.
of Clinical
Epidemiology
63(5):513–523.
Finding
What
Works Journal
in Health
Care: Standards
for Systematic
Revi…
Pham, H. H., D. Schrag, A. S. O’Malley, B. Wu, and P. B. Bach. 2007. Care patterns in Medicare and their implications for pay for performance. New
England Journal of Medicine 356(11):1130–1139.
Healthcare decision makers in search of reliable
information that compares health interventions
Platt, R. 2010. FDA’s
Mini-Sentinel
Paperbackprogram.
| $61.00
 Buy
increasingly turn to systematic reviews for the best
http://www.brookings.edu/~/media/Files/events/2010/0111_sensummary of the evidence. Systematic reviews identify,
select, assess, and synthesize the findings of similar
tinel_workshop/06%20Sentinel%20Initiative%20Platt%20Brookings%2020
but separate studies, and can help clarify what is
(accessed October
2010). | $48.99
 Buy25,Ebook
known and not known about the potential benefits
and harms
of drugs, devices, and
and other
healthcare
Platt, R., M. Wilson, K. A. Chan, J. S. Benner,
J. Marchibroda,
M. McClelservices. Systematic reviews can be helpful for
lan. 2009. The
new
Sentinelsave
Network:clinicians
Improving
the evidence of medMyNAP
members
who want to integrate research findings into
10%
online.
ical-product
safety.
New England Journal
of Medicine
their daily
practices, for 361(7):645–647.
patients to make well-informed
Login
or
Register
to
choices about their own care, for professional medical
Rayleigh, J. W. 1884. Address by the Rt. Hon. Lord Rayleigh. In Report of the
societies and other organizations that develop clinical
save!
fifty-fourth meeting of the British Association
for the Advancement of Scipractice guidelines.
ence, edited by Murray J. Montreal.
Too often systematic reviews are of uncertain or poor
quality. There are no universally accepted standards
for developing systematic reviews leading to variability
how conflicts of interest and biases are handled,
Riley, R. D., and E. W. Steyerberg. 2010. in
Meta-analysis
of a binary outcome
how evidence is appraised, and the overall scientific
using individual participant data and rigor
aggregate
data. Research Synthesis
of the process.
Download Free PDF
Methods 1(1):2–19.
In Finding What Works in Health Care the Institute of
Rothstein, H. R., A. J. Sutton, and M. Borenstein,
editors.
2005.
Publication
Medicine (IOM)
recommends
21 standards
for
developing high-quality systematic reviews of
bias in metaanalysis: Prevention, assessment and adjustments. Chichester,
comparative effectiveness research. The standards
U.K.: Wiley.
address the entire systematic review process from the
initial steps
of formulating
themultiple-treattopic and building the
Salanti, G., V. Marinho, and J. Higgins. 2009.
A case
study of
review team to producing a detailed final report that
ments meta-analysis demonstrates that
covariates
considered.
synthesizes
what theshould
evidence be
shows
and where
knowledge gaps remain.
Journal of Clinical Epidemiology 62(8):857–864.
Salanti, G., S. Dias, N. J. Welton, A. Ades,Finding
V. Golfinopoulos,
M.Care
Kyrgiou,
D. a
What Works in Health
also proposes
framework for
improving
theeffects
quality of in
themultiscience
Mauri, and J. P. A. Ioannidis. 2010. Evaluating
novel
agent
underpinning systematic reviews. This book will serve
ple-treatments meta-regression. Statistics
in Medicine 29(23):2369–2383.
as a vital resource for both sponsors and producers of
systematic
reviews of2004.
comparative
Salpeter, S., E. Greyber, G. Pasternak, and
E. Salpeter.
Riskeffectiveness
of fatal
research.
and nonfatal lactic association with metformin use in type 2 diabetes
mellitus. Cochrane Database of Systematic Reviews 4:CD002967.
Schmid, C. 2001. Using bayesian inference to perform meta-analysis. Evaluation & the Health Professions 24(2):165.
Schmid, C. H., P. C. Stark, J. A. Berlin, P. Landais, and J. Lau. 2004. Meta-regression detected associations between heterogeneous treatment ef-
×
fects and study-level, but not patient-level, factors. Journal of Clinical
Epidemiology
Finding
What Works57(7):683–697.
in Health Care: Standards for Systematic Revi…
Schriger, D. L., D. G. Altman, J. A. Vetter, T. Heafner, and D. Moher. 2010.
Forest plots in reports of systematic reviews: A cross-sectional study reviewing current practice. International
Journal
of Epidemiology
Healthcare
decision
makers in search39(2):421–
of reliable
information that compares health interventions
429.
 Buy Paperback | $61.00
increasingly turn to systematic reviews for the best
Schünemann, H., D. Best, G. Vist, and A.summary
D. Oxman.
2003. Letters,
of the evidence.
Systematic numbers,
reviews identify,
select, assess,
andof
synthesize
the findings
of similar
symbols and words: How to communicate
grades
evidence
and recbut separate studies, and can help clarify what is
Buy Ebook
| $48.99
ommendations.
Medical
Association
Journal 169(7):677–680.
 Canadian
known and not known about the potential benefits
harms of drugs,
devices,
other healthcare
Schünemann, H., A. D. Oxman, G. Vist, J.andHiggins,
J. Deeks,
P.and
Glasziou,
and
services. Systematic reviews can be helpful for
G. Guyatt. 2008.
Interpreting
results and drawing concluMyNAPChapter
members12:
save
clinicians who want to integrate research findings into
10% online.
sions. In Cochrane
handbook for systematic
, edittheir dailyreviews
practices, of
for interventions
patients to make well-informed
Login
or
Register
to
choices about their own care, for professional medical
ed by J. P. T. Higgins and S. Green. Chichester, UK: John Wiley & Sons.
societies and other organizations that develop clinical
save!
Schünemann, H. J., J. Brożek, and A. D. Oxman.
2009. GRADE handbook for
practice guidelines.
grading quality of evidence and strength of recommendations. Version 3.2
Download Free PDF Too often systematic reviews are of uncertain or poor
[updated March 2009]. http://www.cc-ims.net/gradepro
(accessed
Noquality. There are no universally
accepted standards
for developing systematic reviews leading to variability
vember 10, 2010).
in how conflicts of interest and biases are handled,
SIGN (Scottish Intercollegiate Guidelines
2009. SIGN
A scientific
howNetwork).
evidence is appraised,
and the 50:
overall
of the process.
guideline developer’s handbook. http:/rigor
/www.sign.ac.uk/guidelines/fulltext/50/index.html (accessed Februray
1, 2011).
In Finding
What Works in Health Care the Institute of
Medicine
(IOM)
recommends protocols
21 standards for
Silagy, C.A., P. Middelton, and S. Hopewell. 2002. Publishing
of
developing high-quality systematic reviews of
systematic reviews: Comparing whatcomparative
was doneeffectiveness
to what was
planned.
research. The standards
address the entire systematic review process from the
JAMA 287:2831–2834.
steps of formulating the topic and building the
Simmonds, M., J. Higginsa, L. Stewartb,initial
J. Tierneyb,
M. Clarke, and S.
review team to producing a detailed final report that
Thompson. 2005. Meta-analysis of individual
patient
data from
synthesizes what
the evidence
shows randomand where
gaps remain.
ized trials: A review of methods usedknowledge
in practice.
Clinical Trials 2(3):209.
Slone Survey. 2006. Patterns of medication
use
inWorks
the in
United
States,
2006: A
Finding
What
Health Care
also proposes
a
framework for improving the quality of the science
report from the Slone Survey.
underpinning systematic reviews. This book will serve
http://www.bu.edu/slone/SloneSurvey/AnnualRpt/SloneSurveyWebas a vital resource for both sponsors and producers of
Report2006.pdf (accessed February 1,systematic
2011). reviews of comparative effectiveness
research.
Smith, G., M. Egger, and A. Phillips. 1997. Meta-analysis: Beyond the grand
mean? BMJ 315(7122):1610.
Smith, T., D. Spiegelhalter, and A. Thomas. 1995. Bayesian approaches to
random-effects meta-analysis: A comparative study. Statistics in Medicine 14(24):2685–2699.
Song, F., S. Parekh-Bhurke, L. Hooper, Y. Loke, J. Ryder, A.J. Sutton, C.B.
Hing, and I. Harvey. 2009. Extent of publication bias in different cate-
gories of research cohorts: a meta-analysis of empirical studies. BMC
Medical
Research
Finding
What Works
in Methodology
Health Care:9:79.
Standards for Systematic Revi…
×
Song, F., S. Parekh, L. Hooper, Y. K. Loke,
J. Ryder, A. J. Sutton, C. Hing, C. S.
Healthcare decision makers in search of reliable
information that compares
health interventions
Kwok, C. Pang,
I. Harvey.
2010.
and publication
of reBuy
Paperback
| Dissemination
$61.00
and
increasingly turn to systematic reviews for the best
search findings: an updated review ofsummary
related
biases. Health Technology
of the evidence. Systematic reviews identify,
Assessment 14(8).
select, assess, and synthesize the findings of similar
but separate studies, and can help clarify what is
Sterne, J., P. Jüni,
Schulz,
D. Altman,
C.
Bartlett, and M. Egger. 2002. StaEbook
| $48.99
K.Buy
known and not known about the potential benefits
tistical methods for assessing the influence
characteristics
on
and harmsof
of study
drugs, devices,
and other healthcare
services. Systematic
reviews
can be helpful
for
treatment effects
in ‘meta epidemiological’
research.
Statistics
in MediMyNAP members
save
clinicians who want to integrate research findings into
10% online.
cine 21(11):1513–1524.
their daily practices, for patients to make well-informed
Login
or
Register
to
about their own care,
for professional
medical
Stewart, L. 1995. Practical methodologychoices
of meta-analyses
(overviews)
using
societies and other organizations that develop clinical
save!
updated individual
patient data. Statistics
in Medicine 14(19):2057–2079.
practice guidelines.
Sutton, A., and K. Abrams. 2001. Bayesian methods in meta-analysis and evoften systematic reviews are of uncertain or poor
Free
PDF Too
idence synthesis.Download
Statistical
Methods
in
Medical
Research 10(4):277.
quality. There are no universally accepted standards
for developing
systematic reviews
leading to variability
Sutton, A. J., and J. P. Higgins. 2008. Recent
developments
in meta-analysis.
in how conflicts of interest and biases are handled,
Statistics in Medicine 27(5):625–650.
how evidence is appraised, and the overall scientific
Sutton, A. J., K. R. Abams (Q: Abrams?), D.
T. A. Sheldon, and F.
rigorR.ofJones,
the process.
Song. 2000. Methods for meta-analysis
in medical research, Wiley series
In Finding What Works in Health Care the Institute of
in probability and statistics. Chichester,
U.K.:(IOM)
John
Wiley &21Sons.
Medicine
recommends
standards for
developing high-quality systematic reviews of
Thompson, S., and J. Higgins. 2002. How should meta-regression analyses
comparative effectiveness research. The standards
be undertaken and interpreted? Statistics
21(11):1559–1573.
addressin
theMedicine
entire systematic
review process from the
initial steps
of formulating
topic
and building the
Tierney, J., M. Clarke, and L. Stewart. 2000.
Is there
bias inthe
the
publication
review team to producing a detailed final report that
of individual patient data meta-analyses?
International
Journal
Techsynthesizes
what the evidence
showsof
and
where
knowledge gaps remain.
nology Assessment in Health Care 16(02):657–667.
Turner, E. H., A. M. Matthews, E. Linardatos,
R. A.Works
Tell,in and
Rosenthal.
Finding What
HealthR.
Care
also proposes a
framework
for
improving
the
quality
of the science
2008. Selective publication of antidepressant trials and its influence
on
underpinning systematic reviews. This book will serve
apparent efficacy. New England Journal
of Medicine 358(3):252–260
as a vital resource for both sponsors and producers of
reviews Grade
of comparative
effectiveness
USPSTF (U.S. Preventive Services Task systematic
Force). 2008.
definitions.
research.
http://www.ahrq.gov/clinic/uspstf/grades.htm (accessed January 6,
2010).
Vogeli, C., A. Shields, T. Lee, T. Gibson, W. Marder, K. Weiss, and D. Blumenthal. 2007. Multiple chronic conditions: Prevalence, health consequences,
and implications for quality, care management, and costs. Journal of General Internal Medicine 22(Suppl. 3):391–395.
Walker, E., A. V. Hernandez, and M. W. Kattan. 2008. Meta-analysis: Its
strengths and limitations. Cleveland Clinic Journal of Medicine 75(6):431–
439.
Warn, D., S. Thompson, and D. Spiegelhalter. 2002. Bayesian random effects metaanalysis of trials with binary outcomes: Methods for the absolute risk difference and relative risk scales. Statistics in Medicine
21(11):1601–1623.
West, S., V. King, T. S. Carey, K. N. Lohr, N. McKoy, S. F. Sutton, and L. Lux.
2002. Systems to rate the strength of scientific evidence. Evidence Report/Technology Assessment No. 47 (prepared by the Research Triangle
Institute–University of North Carolina Evidence-based Practice Center
under Contract No. 290-97-0011). AHRQ Publication No. 02-E016:64–88.
West, S. L., G. Gartlehner, A. J. Mansfield, C. Poole, E. Tant, N. Lenfestey, L.
J. Lux, J. Amoozegar, S. C. Morton, T. C. Carey, M. Viswanathan, and K. N.
Lohr. 2010. Comparative effectiveness review methods: Clinical heterogeneity.
http://www.effectivehealthcare.ahrq.gov/ehc/products/93/533/Clinical_Heteogeneity_Revised_Report_FINAL%209-24-10.pdf (accessed
September 28, 2010).
The National Academies of Sciences, Engineering, and Medicine
500 Fifth St., NW | Washington, DC 20001
© 2021 National Academy of Sciences. All rights reserved.