Systems of systematic
reviews, execution and
implications for quality
Key
Messages
“Evaluating
“E
l ti
th
the E
Evidence”
id
” and
d
“Making Recommendations” are
TWO Separate activities
Esther F Myers, PhD RD, FADA
Chief Science Officer
American Dietetic Association
emyers@eatright.org
Grading or Rating is used in
describing BOTH the strength of
the evidence and strength of the
recommendation
Overview

Background on systems to conduct
evidence analysis/systematic reviews
•
•
•
•

Principles
Domains of systems/grading methodologies
Steps in process
Heirarchy of evidence
Systematic reviews are intended to
sythensize and describe the “state
of the science”
Research to Practice


Clinical Practice Guidelines
Generalized Population Guidelines
Examples of
• Evidence Grading Methods
• Recommendation Rating Methods
• American Dietetic Association Evidence
Analysis Library
Research
Guidelines
Practice
Application of Research to
Practice Guidelines
Two Basic Principles of
Evidence Based Medicine
(EBM) are:
• Identify a specific problem/area of uncertainty
• Formulate problem as question/analytic framework
• A hierarchy of strength of
evidence behind
recommendations
• Search for and find evidence
• Evaluate the reports and grading the evidence
• The judgment of the clinician
when weighing alternative
management strategies,
considering patient values and
preferences, and societal values
• Form recommendation/make decisions
• Summarize and disseminate findings
1
AHRQ Domains of Systems used to
Evaluate Research
GRADE Domains used to Grade the
Strength of the Evidence
Quality
Quantity
Study
Design
Directness
Consistency
Strength of
evidence /
Grading
system
Study
Quality
Consistency
Agency for Healthcare Research & Quality
2002 - www.ahrq.gov
http://www.gradeworkinggroup.org// (2004)
http://www.gradeworkinggroup.org
GRADE - Additional factors to
consider



Domains used to Grade the
Strength of the Evidence
Strength of association (50 fold vs 3
fold increase in risk)
Evidence of dose
dose--response gradient
E ti
Estimates
t off effect
ff t off plausible
l
ibl
residual confounding factors on
observed effect
Risk of Bias
Directness
Consistency
Precision
Grading the Body of Evidence when comparing Medical
Interventions = Methods Guide Chapter - AHRQ 2009 www.ahrq.gov
Risk of Bias

Assessment of risk at individual
study level (e.g. research
publication)
Consistency

• Direction
• Effect size
• Quality Criteria Checklist





High risk (low quality score,
Medium (moderate quality score)
Low (high quality score)
Aggregated within study design type
and for body of evidence
Greater weight given to those with
lower risk of bias
Degree of similarity among studies

NOTE: Evaluation of consistency
ideally has evidence base with more
than one study replicating findings—
findings—
not a single study evidence base!
• One study = unknown consistency
2
Precision
Directness


Research has a single, direct link
(intervention to outcome)
Several questions (bodies of
evidence) may be necessary to
document the causal chain

• Statistical significance
• Clinical significance

• Not all steps in causal chain may be
“equal” in importance.





Interventions may not be compared
head to head
Dose-response association
DoseExistence of confounders
Strength of association (magnitude
off effect)
ff t)
Publication bias
IOM Standards for Systematic Reviews
for CER of Med/Surg Interventions

NOTE these first three apply more
often to observational studies, e.g.
epidemiological studies
Overview of Evidence
Analysis Methods
Common elements
 Expert group
 Questions developed
 Evidence search conducted
 Research to be summarized selected
 Selected research studies analyzed
• Quality of research (Risk of Bias)
• Results that apply to question


Conclusion
statements/Recommendations
developed
Strength of research support
determined (Grades)
Initiating a systematic review
• Teams, conflict of interest, user and stakeholder input,
formulating the topic, developing protocol, peer review
of protocol, and public availability of protocol

Finding and assessing individual studies
• Search for evidence,
evidence addressing potentially biased
reporting of research results, screening and selecting
studies, documenting search, manage data collection,
critically appraise each study


Influenced by “importance
importance of
outcome”
• Harm vs benefit
• Relative impact or size of harm/benefit
Intermediate or surrogate outcomes (e.g.
cholesterol levels vs mortality)
Other considerations –
secondary constructs

Degree of certainty of each specified
outcome
Synthesizing the body of evidence
• Pre
Pre--specified method to evaluate body of evidence,
qualitative synthesis, quantitative analysis as
appropriate, metameta-analysis process,

Reporting systematic reviews
• Prepare final report using structured format, peer review
draft report, publish in a manner that ensures free
public acccess
Hierarchy of Strength of Evidence
Type and Description of Evidence
 Primary Studies • Selecting studies that are both:
 Highly relevant
 Study designs minimize bias
• High strength of inference
Guyatt et al, JAMA.2000;284(10):1290-1296
3
Which research design?

Hierarchy of Evidence
Type of primary study preferred
varies by use:
Diagnosis
• CrossSectional
Studies
Treatment
• Randomized
Controlled
Trials
Prognosis
• Cohort
Studies (over
time with
outcomes)
Etiology/Harm
• Cohort/Case
Control (2
groups over
time)
Cochrane Reviews/Protocols
Cochrane Reviews

Considered as gold standard by
many
Protocol developed for review

Review includes



• Preterm infant feeding (2), Bone marrow
transplant, Pressure ulcers, HIV, Hip fractures,
COPD, TB, Acute kidney injury, Critically ill
children, head injury, Children on chemotherapy,
Prevention of nutritional rickets, Cerebral palsy,
Supplementation (Ca, Fe Iodine, D, A), Dietary
advice--Gestational DM, Type 2 DM, Cardiovascular,
advice
Schizophrenia,
• RCT (occasionally some quasi RCT)
• Summary
• Tables of evidence

Central Database of RCTs and
CCTs
• 350,000 citations searchable by
keywords

TYPICALLY NO
EPIDEMIOLOGICAL STUDIES
Nutrition intervention = 82 reviews
~57 focused on nutrition

25 as part of another topic
• Childhood obesity, Ceraberal palsy, Cystic fibrosis,
Preganancy, Colorectal cancer, Upper GI, Hepato
Billiary, Cardiovascular
http://www.cochrane.org
August 2011
Example – Dietary Advice for
reducing cardiovascular risk

38 trials (randomized studies, no
more than 20% loss to followfollow-up,
more than 3 months, healthy adults,
comparing dietary advice vs no or
minimal advice
WHO Report –Model for
Establishing Upper Levels for
Nutrients and Related Substances

Confirms importance of:
• A Priori Search Strategies
• Method for evaluating quality of
research
• Summary of research in tables

Used observational studies
WHO = World Health Organization
Http://www.who.int/ipcs/highlights/nutrientproject_may18/en/index.html
4
WHO: Useful characteristics to identify
high--quality observational studies
high











Unbiased selection of cohort
Adequate description of cohort
Use of validated dietary assessment method
Quantification of type & amount of nutrient intake
Use of validated method for endpoints
Documentation of drugs
Low number and random distribution of dropdrop-outs
Adequate followfollow-up period
Complete follow
follow--up
Appropriate analysis and reporting results
No known prepre-existing illness
WHO Report – Model for
Establishing Upper Levels

• Questions using results from studies not
designed to test safety to demonstrate
safety
Two processes

Confirms concept that absence of
evidence to refute safety is not
necessarily “safe”
safe
Guideline Methodologies
Overview of Methodologies (EA & Recommendations)
Evaluate and grade the strength of
the evidence
http://www.cebm.net/
http://www.ahrq.gov/clinic/uspstf/grades.htm

Make
M k and
d rate
t the
th strength
t
th off the
th
recommendations
http://www.nhlbi.nih.gov/
http://www.diabetes.org/
http://www.icsi.org/
http://www.gradeworkinggroup.org/publications/index.htm
Center for Evidence
Based Medicine


www.cebm.net
References
• Evidence
Evidence--Based Medicine: How to
practice and teach EBM (Sackett et al)
Levels of Evidence available at: http://www.cebm.net/levels_of_evidence.asp
Levels of Evidence and
Grades of Recommendations


Ranks the validity of evidence about
the value of preventative maneuvers
The levels and grades speak to the
y of evidence about:
validity
• prevention
• diagnosis
• therapy
• harm
5
CEBM Levels of Evidence
and
Grades of Recommendations

Level of Evidence/Type of Study
• 1a,1b
• 2a,2b,2c
• 3a,3b
•4
•5

Grade of Recommendation
Level 1 Evidence for
Prevention/Therapy
1a
Systematic Review with homogeneity of
Randomized Controlled Trials
1b
Individual Randomized Controlled Trials with
narrow Confidence Interval
1c
All or none (Patients died before Rx, now some
survive)
• A, B, C, D (A = Highest)
Level 2 & 3 Evidence for
Prevention/Therapy
2a
Systematic Review with homogeneity of cohort
studies
2b
Individual cohort study (including low quality
RCT
e.g. <80% f/up)
2c
Level 3 Evidence for
Prevention/Therapy
3a
Systematic reviews with homogeneity of
case--control
case
studies
3b
Individual case
case--control studies
Outcomes research/Ecological research
Level 4 and 5 for
Prevention Therapy


4 – Case series (poor quality cohort
and case control studies)
5 – Expert
E
t opinion
i i
without
ith t critical
iti l
appraisal
Grades of
Recommendation
A
B
C
D
Consistent level 1 studies
Consistent level 2 or 3 studies or
Extrapolations from level 1 studies
Level 4 studies or
Extrapolations from level 2 or 3 studies
Level 5 evidence or
Troublingly inconsistent or inconclusive
studies of any level
6
Guide to Community Preventive
Services
Quality of Overall Evidence
• Good
 Consistent results from well designed & conducted
studies
 Representative populations
 Directly assess effects on health outcomes
• Fair
 Sufficient
S ffi i
to determine
d
i
effects
ff
 Limited by number, quality, or consistency of individual
studies, generalizability, or indirect evidence on health
outcomes
• Poor
 Insufficient to assess effects on health outcome
 Limited number or power of studies, important design
flaws, gaps in chain of evidence
(Does not use hierarchy of evidence in same way)
Guide to Community Preventive Services
A - Recommend
(High certainty net benefit is substantial)
B - Recommend
(High certainty net benefit is moderate)
C – Recommend against routinely providing
service
i
(may be considerations that support service,
moderate certainty net benefit is small)
D - Recommend against
(Moderate or high certainty that service has no
net benefit or harms outweigh the benefits)
I - Insufficient Evidence to Recommend For
or Against
http://www.ahrq.gov/clinic/uspstf/grades.htm
National Heart, Blood,
and Lung Institute
A
NHLBI Evidence Model
Randomized controlled trials
(rich body of data)
B
Randomized controlled trials
(limited body of data)
C
Nonrandomized trials/
Observational
studies
D
Panel consensus judgement
http://www.nhbli.nih.gov/guidelines/obesity/e_txtbk/appndx/apndx1
a1.htm
See full page handout
American Diabetes
Association
A – Clear evidence



Well-conducted multi
Wellmulti--center trial
Meta--analysis
Meta
Non--experimental evidence
Non
evidence—
—CEBM all or none rule
B – Supportive evidence from well conducted
cohort studies

Prospective cohort studies from registry or metametaanalysis
GRADE
Levels of Evidence
High
Moderate Further resarch is likely to have an important impact on
our confidence in the estimate of effect and may change
the estimate
Low

Observational studies with high potential for bias
Conflicting evidence with weight of evidence
supporting recommendation
E – Expert consensus or clinical experience
Further resarch is very likely to have an important
impact our confidence in estaimte of effect and likely to
change the estimate
C – Poorly controlled or nonnon-controlled trials

Futher research is unlikely to change our confidence in
the estimate of effect
Very
Any estimate of effect is very uncertain
Low
http://www.gradeworkinggroup.org/
7
GRADE approach
GRADE Recommendations

(Body of evidence for each outcome)
Categories of recommendations
Five quality characteristics lower rating
•
•
•
•
•
• Do it
 or Don’t Do it
• Probably do it
 Or Probably don’t do it

Limitations in study design and conduct
Inconsistent results across studies
Indirectness
Imprecision
Publication bias
Three factors increase rating
• Large magnitude of effect
• Adequate accounting for plausible confounders
• Strength of association
Institute of Clinical
Systems Improvement
(ICSI) Methodology


Classes of Research
Report
Classes of Research Reports
Report Quality Categories
Conclusion Grades (I,II,III,IV)
(I II III IV)

ICSI modified to Conclusion Grades I,II,
III and Grade Not Assignable
ICSI Evidence Grading System. Available at:
http://www.icsi.org/evidence_grading_system_6/evidence_grading_system__pdf_.ht
ml.. Accessed August 2007
ml
As of Feb 2009
• 51.6% were clinical trials
• 39% were observational studies
• 9.4%
9 4% were review (narrative and
systematic reviews
A RCT
M MetaMeta-analysis,
systematic review,
decision analysis,
c/b or c/e study
C Nonrandomized trial
concurrent/ historical
controls
Case control study
Population based
descriptive study
R Narrative review,
Consensus
statement/report
X Medical opinion
D Cross sectional, Case
series, Case report
Articles in EAL

SYTHESIZED
B Cohort study

Greer et al, Jr Comm J Qual Improv
Improv.. 2000;26(12):700
2000;26(12):700--712

PRIMARY
Conclusion Grades
Grade I
Studies of strong design to answer questions
Clinically important and consistent
Adequate statistical power

Grade II
Studies of strong design to answer questions
Some uncertainty in conclusion/inconsistent
results, doubt about statistical or clinical
significance or generalizability
Separate studies of weaker design, all
consistent

8
Rating
Conclusion Grades
Definition
• benefits clearly exceed the harms (or harms clearly exceed
the benefits for a negative recommendation)
Strong
(Continued)
Grade III
Strong design, substantial uncertainty/
inconsistent results/
results/generalizability
generalizability unclear
Limited studies of weaker design or small effect
size or intermediate or surrogate outcome
• the quality of the supporting evidence is excellent/good
(grade I or II)

Fair
benefits exceed the harms (or harms clearly exceed the
benefits for a negative recommendation)
•
quality of evidence is not as strong (grade II or III)
Weak
• quality of evidence that exists is suspect
Grade IV
Expert opinion or consensus statement

Grade Not Assignable
Unsubstantiated by research studies
Informed medical commentators based on
clinical
•
• or that well-done studies (grade I, II, or III)* show little clear
advantage to one approach versus another
Consensus
• Expert opinion (grade IV) supports the guideline
recommendation

Insufficient
Evidence
• both a lack of pertinent evidence (grade V)* and/or an
unclear balance between benefits and harms
Published Topics /Published Topics Under Revision (updating)/ New Topics
Diseases & Conditions
• Adult Weight Management
• Aging
• Athletic Performance
• Bariatric Surgery
• Breastfeeding
• Celiac Disease
• Childhood Overweight
• Chronic Kidney Disease
• Chronic Obstructive Pulmonary Disease
• Critical Illness
C iti l Ill
• Diabetes 1 & 2
• Disorders of Lipid Metabolism
• Gestational Diabetes
• Heart Failure
• HIV/AIDS
• Hydration
• Hypertension
• Oncology
• Pediatric Weight Management
• Pre‐Diabetes
• Spinal Cord Injury
• Unintended Weight Loss
• Wound Care
Nutrients
• Dietary Fatty Acids
• Fiber
• Fluoride
• Nutrient Supplementation
• Sodium
• Topics from other projects cross‐linked • e.g., Athletic Performance & Macronutrient Intake, Vitamin C and Oncology, Omega‐3 and Disorders of Lipid Metabolism
Foods
• Advanced Food Production & Sustainable Agriculture
• Nutritive and non‐
nutritive sweeteners
• ‐Aspartame
• ‐Non‐nutritive sweeteners
• Vegetarian Nutrition
• Topics from other projects cross‐linked • e.g., Nuts and Disorders of Lipid Metabolism, Oats and Gluten Intolerance, Dairy and Childhood Overweight
EAL Content/Growth
Nutrition Care Process
• Energy Expenditure: Measurement vs. Estimation
• Health Disparities
• Medical Nutrition Therapy Effectiveness
• ‐Telenutrition
• Nutrition Counseling
• Nutrition Screening
• Topics from other projects cross‐linked • e.g., Diabetes and MNT Effectiveness, Critical Illness and cost‐effectiveness
How Much Content is on the EAL®?
Abstracted Articles/Worksheets
ADA Evidence-Based Nutrition Practice Guidelines
Published on EAL®:
2005
• Disorders of Lipid Metabolism
2009
(update 2011)
2006
• Adult Weight Management
• Critical Illness (update 2011)
(update 2011)
2007
• Celiac Disease
• Spinal Cord Injury
• Unintended Weight Loss
2010
• Pediatric Weight Management
2012)
• Oncology
(update
(update 2012)
• Chronic Kidney Disease
• HIV/AIDS
2008
• Diabetes
• Hypertension
• Heart Failure
• Chronic Obstructive Pulmonary Disease
• Gestational Diabetes
In Development
• Vegetarian Nutrition
• Pre- Diabetes
• Wound Care
EAL Usage
EAL® Page Views by Calendar Year
Overall Total
Year
Total
9,853,683 page views
2004
41,332
4,592
2005
378,026
31,502
Sept 2004 – Dec 31 2010
115
Avg /
Month
2006
1,068,758
89,063
2007
1,544,119
,
,
128,677
,
2008
2,020,497
168,375
2009
2,321,594
193,466
2010
2,457,410
204,784
On January 23, 2011, The ADA Evidence Analysis Library reached 10 million page views.
9
Dissemination of EAL®
Worldwide

Users from 206 different countries
Examples from EAL
10
Example from ADA
Evidence Analysis Library®
Protein Intake and Pre-Diabetes
What is the relationship
between amount of
protein intake and
metabolic outcomes in
persons with
prediabetes?
(Metabolic Syndrome, Impaired
glucose tolerance, Incidence of
diabetes)
What is the
relationship between
type of protein
intake and metabolic
outcomes in persons
with prediabetes?
Example: Conclusion Statement
What is the relationship between
type of protein intake and
metabolic outcomes in persons
with prediabetes?
11 studies (14 publications) were evaluated regarding the
relationship between type of protein intake and metabolic outcomes
in persons with prediabetes. Seven studies regarding consumption of
protein from animal sources report a positive association between
red meat intake and increased risk of the metabolic syndrome and
diabetes, while four studies regarding consumption of protein from
plant sources report reduced risk of metabolic syndrome and
diabetes. Three studies report sex differences. Further research is
needed regarding the relationship between type of protein intake and
metabolic outcomes in persons with prediabetes.
Grade II
Date of Literature Review: January 2009
http://www.adaevidencelibrary.com
http://
www.adaevidencelibrary.com
11
Example: Evidence
Summary
Types of Studies
Type of protein and prediabetes

11 studies (14 publications) were evaluated regarding the relationship between
type of protein intake and metabolic outcomes in persons with prediabetes.
11 studies
• RCT
• Prospective Cohort
• Cross Sectional
Neutral Positive –
Positive –
2
4
1
Neutral
4
-
Research Regarding Consumption of Protein from Animal Sources
In a neutral-quality cross-sectional study, Alvarez-Leon et al, 2006 analyzed the
relationship between the Mediterranean diet and the prevalence of metabolic
syndrome in participants from the Canarian Nutrition Survey (ENCA 1997
19971998). Adherence to the Mediterranean diet was assessed through food
frequency questionnaire. Of 1,747 subjects participating in the survey, a total
of 578 adults (249 men, 329 women, aged 18-75 years) were included in the
analysis and 24.4% were diagnosed with the metabolic syndrome. After
adjustment for confounding variables, adherence to the Mediterranean diet was
not related to prevalence of the metabolic syndrome. However, red meat intake
was associated with higher prevalence of blood pressure criteria (elevated
blood pressure or anti-hypertensive treatment, P for tendency=0.028).
In a positive-quality prospective cohort study, Damiao et al, 2006 examined the
association between dietary factors and the metabolic syndrome in a sevenyear follow-up of a cohort of Japanese-Brazilians. Of 647 subjects included at
baseline in 1993, 394 subjects participated in a second survey in 2000, and 151
Continues….
Conclusion
Create Research
Use EAL® to identify areas for future
research

• Lower level Conclusion Statements
(III, IV, V) and Recommendations
(Weak, Consensus, Insufficient Evidence) are
clearly areas for future research
ADA Outcomes Research
• Epidemiological research preferred for
questions about harm, prognosis and
diagnoses
d ag oses
• RCT preferred for treatment questions

• Dietetics PracticePractice-Based Research Network



Feasibility and Field Test MNT Guides
Nutrition Quality of Life (Validation)
CTIMEE (Critical Thinking in Measuring Energy
Expenditure)
Key
Messages
Systematic reviews are intended to
sythensize and describe the “state
of the science”
Value of using all types of research in the
body of evidence based on type of
question being answered


Clearly demonstrate which research
studies were evaluated in tables of
evidence
EAL contains ~ 39% observational types
of research
Questions??
“Evaluating
“E
l ti
th
the E
Evidence”
id
” and
d
“Making Recommendations” are
TWO Separate activities
Grading or Rating is used in
describing BOTH the strength of
the evidence and strength of the
recommendation
12