Systems of systematic reviews, execution and implications for quality Key Messages “Evaluating “E l ti th the E Evidence” id ” and d “Making Recommendations” are TWO Separate activities Esther F Myers, PhD RD, FADA Chief Science Officer American Dietetic Association emyers@eatright.org Grading or Rating is used in describing BOTH the strength of the evidence and strength of the recommendation Overview Background on systems to conduct evidence analysis/systematic reviews • • • • Principles Domains of systems/grading methodologies Steps in process Heirarchy of evidence Systematic reviews are intended to sythensize and describe the “state of the science” Research to Practice Clinical Practice Guidelines Generalized Population Guidelines Examples of • Evidence Grading Methods • Recommendation Rating Methods • American Dietetic Association Evidence Analysis Library Research Guidelines Practice Application of Research to Practice Guidelines Two Basic Principles of Evidence Based Medicine (EBM) are: • Identify a specific problem/area of uncertainty • Formulate problem as question/analytic framework • A hierarchy of strength of evidence behind recommendations • Search for and find evidence • Evaluate the reports and grading the evidence • The judgment of the clinician when weighing alternative management strategies, considering patient values and preferences, and societal values • Form recommendation/make decisions • Summarize and disseminate findings 1 AHRQ Domains of Systems used to Evaluate Research GRADE Domains used to Grade the Strength of the Evidence Quality Quantity Study Design Directness Consistency Strength of evidence / Grading system Study Quality Consistency Agency for Healthcare Research & Quality 2002 - www.ahrq.gov http://www.gradeworkinggroup.org// (2004) http://www.gradeworkinggroup.org GRADE - Additional factors to consider Domains used to Grade the Strength of the Evidence Strength of association (50 fold vs 3 fold increase in risk) Evidence of dose dose--response gradient E ti Estimates t off effect ff t off plausible l ibl residual confounding factors on observed effect Risk of Bias Directness Consistency Precision Grading the Body of Evidence when comparing Medical Interventions = Methods Guide Chapter - AHRQ 2009 www.ahrq.gov Risk of Bias Assessment of risk at individual study level (e.g. research publication) Consistency • Direction • Effect size • Quality Criteria Checklist High risk (low quality score, Medium (moderate quality score) Low (high quality score) Aggregated within study design type and for body of evidence Greater weight given to those with lower risk of bias Degree of similarity among studies NOTE: Evaluation of consistency ideally has evidence base with more than one study replicating findings— findings— not a single study evidence base! • One study = unknown consistency 2 Precision Directness Research has a single, direct link (intervention to outcome) Several questions (bodies of evidence) may be necessary to document the causal chain • Statistical significance • Clinical significance • Not all steps in causal chain may be “equal” in importance. Interventions may not be compared head to head Dose-response association DoseExistence of confounders Strength of association (magnitude off effect) ff t) Publication bias IOM Standards for Systematic Reviews for CER of Med/Surg Interventions NOTE these first three apply more often to observational studies, e.g. epidemiological studies Overview of Evidence Analysis Methods Common elements Expert group Questions developed Evidence search conducted Research to be summarized selected Selected research studies analyzed • Quality of research (Risk of Bias) • Results that apply to question Conclusion statements/Recommendations developed Strength of research support determined (Grades) Initiating a systematic review • Teams, conflict of interest, user and stakeholder input, formulating the topic, developing protocol, peer review of protocol, and public availability of protocol Finding and assessing individual studies • Search for evidence, evidence addressing potentially biased reporting of research results, screening and selecting studies, documenting search, manage data collection, critically appraise each study Influenced by “importance importance of outcome” • Harm vs benefit • Relative impact or size of harm/benefit Intermediate or surrogate outcomes (e.g. cholesterol levels vs mortality) Other considerations – secondary constructs Degree of certainty of each specified outcome Synthesizing the body of evidence • Pre Pre--specified method to evaluate body of evidence, qualitative synthesis, quantitative analysis as appropriate, metameta-analysis process, Reporting systematic reviews • Prepare final report using structured format, peer review draft report, publish in a manner that ensures free public acccess Hierarchy of Strength of Evidence Type and Description of Evidence Primary Studies • Selecting studies that are both: Highly relevant Study designs minimize bias • High strength of inference Guyatt et al, JAMA.2000;284(10):1290-1296 3 Which research design? Hierarchy of Evidence Type of primary study preferred varies by use: Diagnosis • CrossSectional Studies Treatment • Randomized Controlled Trials Prognosis • Cohort Studies (over time with outcomes) Etiology/Harm • Cohort/Case Control (2 groups over time) Cochrane Reviews/Protocols Cochrane Reviews Considered as gold standard by many Protocol developed for review Review includes • Preterm infant feeding (2), Bone marrow transplant, Pressure ulcers, HIV, Hip fractures, COPD, TB, Acute kidney injury, Critically ill children, head injury, Children on chemotherapy, Prevention of nutritional rickets, Cerebral palsy, Supplementation (Ca, Fe Iodine, D, A), Dietary advice--Gestational DM, Type 2 DM, Cardiovascular, advice Schizophrenia, • RCT (occasionally some quasi RCT) • Summary • Tables of evidence Central Database of RCTs and CCTs • 350,000 citations searchable by keywords TYPICALLY NO EPIDEMIOLOGICAL STUDIES Nutrition intervention = 82 reviews ~57 focused on nutrition 25 as part of another topic • Childhood obesity, Ceraberal palsy, Cystic fibrosis, Preganancy, Colorectal cancer, Upper GI, Hepato Billiary, Cardiovascular http://www.cochrane.org August 2011 Example – Dietary Advice for reducing cardiovascular risk 38 trials (randomized studies, no more than 20% loss to followfollow-up, more than 3 months, healthy adults, comparing dietary advice vs no or minimal advice WHO Report –Model for Establishing Upper Levels for Nutrients and Related Substances Confirms importance of: • A Priori Search Strategies • Method for evaluating quality of research • Summary of research in tables Used observational studies WHO = World Health Organization Http://www.who.int/ipcs/highlights/nutrientproject_may18/en/index.html 4 WHO: Useful characteristics to identify high--quality observational studies high Unbiased selection of cohort Adequate description of cohort Use of validated dietary assessment method Quantification of type & amount of nutrient intake Use of validated method for endpoints Documentation of drugs Low number and random distribution of dropdrop-outs Adequate followfollow-up period Complete follow follow--up Appropriate analysis and reporting results No known prepre-existing illness WHO Report – Model for Establishing Upper Levels • Questions using results from studies not designed to test safety to demonstrate safety Two processes Confirms concept that absence of evidence to refute safety is not necessarily “safe” safe Guideline Methodologies Overview of Methodologies (EA & Recommendations) Evaluate and grade the strength of the evidence http://www.cebm.net/ http://www.ahrq.gov/clinic/uspstf/grades.htm Make M k and d rate t the th strength t th off the th recommendations http://www.nhlbi.nih.gov/ http://www.diabetes.org/ http://www.icsi.org/ http://www.gradeworkinggroup.org/publications/index.htm Center for Evidence Based Medicine www.cebm.net References • Evidence Evidence--Based Medicine: How to practice and teach EBM (Sackett et al) Levels of Evidence available at: http://www.cebm.net/levels_of_evidence.asp Levels of Evidence and Grades of Recommendations Ranks the validity of evidence about the value of preventative maneuvers The levels and grades speak to the y of evidence about: validity • prevention • diagnosis • therapy • harm 5 CEBM Levels of Evidence and Grades of Recommendations Level of Evidence/Type of Study • 1a,1b • 2a,2b,2c • 3a,3b •4 •5 Grade of Recommendation Level 1 Evidence for Prevention/Therapy 1a Systematic Review with homogeneity of Randomized Controlled Trials 1b Individual Randomized Controlled Trials with narrow Confidence Interval 1c All or none (Patients died before Rx, now some survive) • A, B, C, D (A = Highest) Level 2 & 3 Evidence for Prevention/Therapy 2a Systematic Review with homogeneity of cohort studies 2b Individual cohort study (including low quality RCT e.g. <80% f/up) 2c Level 3 Evidence for Prevention/Therapy 3a Systematic reviews with homogeneity of case--control case studies 3b Individual case case--control studies Outcomes research/Ecological research Level 4 and 5 for Prevention Therapy 4 – Case series (poor quality cohort and case control studies) 5 – Expert E t opinion i i without ith t critical iti l appraisal Grades of Recommendation A B C D Consistent level 1 studies Consistent level 2 or 3 studies or Extrapolations from level 1 studies Level 4 studies or Extrapolations from level 2 or 3 studies Level 5 evidence or Troublingly inconsistent or inconclusive studies of any level 6 Guide to Community Preventive Services Quality of Overall Evidence • Good Consistent results from well designed & conducted studies Representative populations Directly assess effects on health outcomes • Fair Sufficient S ffi i to determine d i effects ff Limited by number, quality, or consistency of individual studies, generalizability, or indirect evidence on health outcomes • Poor Insufficient to assess effects on health outcome Limited number or power of studies, important design flaws, gaps in chain of evidence (Does not use hierarchy of evidence in same way) Guide to Community Preventive Services A - Recommend (High certainty net benefit is substantial) B - Recommend (High certainty net benefit is moderate) C – Recommend against routinely providing service i (may be considerations that support service, moderate certainty net benefit is small) D - Recommend against (Moderate or high certainty that service has no net benefit or harms outweigh the benefits) I - Insufficient Evidence to Recommend For or Against http://www.ahrq.gov/clinic/uspstf/grades.htm National Heart, Blood, and Lung Institute A NHLBI Evidence Model Randomized controlled trials (rich body of data) B Randomized controlled trials (limited body of data) C Nonrandomized trials/ Observational studies D Panel consensus judgement http://www.nhbli.nih.gov/guidelines/obesity/e_txtbk/appndx/apndx1 a1.htm See full page handout American Diabetes Association A – Clear evidence Well-conducted multi Wellmulti--center trial Meta--analysis Meta Non--experimental evidence Non evidence— —CEBM all or none rule B – Supportive evidence from well conducted cohort studies Prospective cohort studies from registry or metametaanalysis GRADE Levels of Evidence High Moderate Further resarch is likely to have an important impact on our confidence in the estimate of effect and may change the estimate Low Observational studies with high potential for bias Conflicting evidence with weight of evidence supporting recommendation E – Expert consensus or clinical experience Further resarch is very likely to have an important impact our confidence in estaimte of effect and likely to change the estimate C – Poorly controlled or nonnon-controlled trials Futher research is unlikely to change our confidence in the estimate of effect Very Any estimate of effect is very uncertain Low http://www.gradeworkinggroup.org/ 7 GRADE approach GRADE Recommendations (Body of evidence for each outcome) Categories of recommendations Five quality characteristics lower rating • • • • • • Do it or Don’t Do it • Probably do it Or Probably don’t do it Limitations in study design and conduct Inconsistent results across studies Indirectness Imprecision Publication bias Three factors increase rating • Large magnitude of effect • Adequate accounting for plausible confounders • Strength of association Institute of Clinical Systems Improvement (ICSI) Methodology Classes of Research Report Classes of Research Reports Report Quality Categories Conclusion Grades (I,II,III,IV) (I II III IV) ICSI modified to Conclusion Grades I,II, III and Grade Not Assignable ICSI Evidence Grading System. Available at: http://www.icsi.org/evidence_grading_system_6/evidence_grading_system__pdf_.ht ml.. Accessed August 2007 ml As of Feb 2009 • 51.6% were clinical trials • 39% were observational studies • 9.4% 9 4% were review (narrative and systematic reviews A RCT M MetaMeta-analysis, systematic review, decision analysis, c/b or c/e study C Nonrandomized trial concurrent/ historical controls Case control study Population based descriptive study R Narrative review, Consensus statement/report X Medical opinion D Cross sectional, Case series, Case report Articles in EAL SYTHESIZED B Cohort study Greer et al, Jr Comm J Qual Improv Improv.. 2000;26(12):700 2000;26(12):700--712 PRIMARY Conclusion Grades Grade I Studies of strong design to answer questions Clinically important and consistent Adequate statistical power Grade II Studies of strong design to answer questions Some uncertainty in conclusion/inconsistent results, doubt about statistical or clinical significance or generalizability Separate studies of weaker design, all consistent 8 Rating Conclusion Grades Definition • benefits clearly exceed the harms (or harms clearly exceed the benefits for a negative recommendation) Strong (Continued) Grade III Strong design, substantial uncertainty/ inconsistent results/ results/generalizability generalizability unclear Limited studies of weaker design or small effect size or intermediate or surrogate outcome • the quality of the supporting evidence is excellent/good (grade I or II) Fair benefits exceed the harms (or harms clearly exceed the benefits for a negative recommendation) • quality of evidence is not as strong (grade II or III) Weak • quality of evidence that exists is suspect Grade IV Expert opinion or consensus statement Grade Not Assignable Unsubstantiated by research studies Informed medical commentators based on clinical • • or that well-done studies (grade I, II, or III)* show little clear advantage to one approach versus another Consensus • Expert opinion (grade IV) supports the guideline recommendation Insufficient Evidence • both a lack of pertinent evidence (grade V)* and/or an unclear balance between benefits and harms Published Topics /Published Topics Under Revision (updating)/ New Topics Diseases & Conditions • Adult Weight Management • Aging • Athletic Performance • Bariatric Surgery • Breastfeeding • Celiac Disease • Childhood Overweight • Chronic Kidney Disease • Chronic Obstructive Pulmonary Disease • Critical Illness C iti l Ill • Diabetes 1 & 2 • Disorders of Lipid Metabolism • Gestational Diabetes • Heart Failure • HIV/AIDS • Hydration • Hypertension • Oncology • Pediatric Weight Management • Pre‐Diabetes • Spinal Cord Injury • Unintended Weight Loss • Wound Care Nutrients • Dietary Fatty Acids • Fiber • Fluoride • Nutrient Supplementation • Sodium • Topics from other projects cross‐linked • e.g., Athletic Performance & Macronutrient Intake, Vitamin C and Oncology, Omega‐3 and Disorders of Lipid Metabolism Foods • Advanced Food Production & Sustainable Agriculture • Nutritive and non‐ nutritive sweeteners • ‐Aspartame • ‐Non‐nutritive sweeteners • Vegetarian Nutrition • Topics from other projects cross‐linked • e.g., Nuts and Disorders of Lipid Metabolism, Oats and Gluten Intolerance, Dairy and Childhood Overweight EAL Content/Growth Nutrition Care Process • Energy Expenditure: Measurement vs. Estimation • Health Disparities • Medical Nutrition Therapy Effectiveness • ‐Telenutrition • Nutrition Counseling • Nutrition Screening • Topics from other projects cross‐linked • e.g., Diabetes and MNT Effectiveness, Critical Illness and cost‐effectiveness How Much Content is on the EAL®? Abstracted Articles/Worksheets ADA Evidence-Based Nutrition Practice Guidelines Published on EAL®: 2005 • Disorders of Lipid Metabolism 2009 (update 2011) 2006 • Adult Weight Management • Critical Illness (update 2011) (update 2011) 2007 • Celiac Disease • Spinal Cord Injury • Unintended Weight Loss 2010 • Pediatric Weight Management 2012) • Oncology (update (update 2012) • Chronic Kidney Disease • HIV/AIDS 2008 • Diabetes • Hypertension • Heart Failure • Chronic Obstructive Pulmonary Disease • Gestational Diabetes In Development • Vegetarian Nutrition • Pre- Diabetes • Wound Care EAL Usage EAL® Page Views by Calendar Year Overall Total Year Total 9,853,683 page views 2004 41,332 4,592 2005 378,026 31,502 Sept 2004 – Dec 31 2010 115 Avg / Month 2006 1,068,758 89,063 2007 1,544,119 , , 128,677 , 2008 2,020,497 168,375 2009 2,321,594 193,466 2010 2,457,410 204,784 On January 23, 2011, The ADA Evidence Analysis Library reached 10 million page views. 9 Dissemination of EAL® Worldwide Users from 206 different countries Examples from EAL 10 Example from ADA Evidence Analysis Library® Protein Intake and Pre-Diabetes What is the relationship between amount of protein intake and metabolic outcomes in persons with prediabetes? (Metabolic Syndrome, Impaired glucose tolerance, Incidence of diabetes) What is the relationship between type of protein intake and metabolic outcomes in persons with prediabetes? Example: Conclusion Statement What is the relationship between type of protein intake and metabolic outcomes in persons with prediabetes? 11 studies (14 publications) were evaluated regarding the relationship between type of protein intake and metabolic outcomes in persons with prediabetes. Seven studies regarding consumption of protein from animal sources report a positive association between red meat intake and increased risk of the metabolic syndrome and diabetes, while four studies regarding consumption of protein from plant sources report reduced risk of metabolic syndrome and diabetes. Three studies report sex differences. Further research is needed regarding the relationship between type of protein intake and metabolic outcomes in persons with prediabetes. Grade II Date of Literature Review: January 2009 http://www.adaevidencelibrary.com http:// www.adaevidencelibrary.com 11 Example: Evidence Summary Types of Studies Type of protein and prediabetes 11 studies (14 publications) were evaluated regarding the relationship between type of protein intake and metabolic outcomes in persons with prediabetes. 11 studies • RCT • Prospective Cohort • Cross Sectional Neutral Positive – Positive – 2 4 1 Neutral 4 - Research Regarding Consumption of Protein from Animal Sources In a neutral-quality cross-sectional study, Alvarez-Leon et al, 2006 analyzed the relationship between the Mediterranean diet and the prevalence of metabolic syndrome in participants from the Canarian Nutrition Survey (ENCA 1997 19971998). Adherence to the Mediterranean diet was assessed through food frequency questionnaire. Of 1,747 subjects participating in the survey, a total of 578 adults (249 men, 329 women, aged 18-75 years) were included in the analysis and 24.4% were diagnosed with the metabolic syndrome. After adjustment for confounding variables, adherence to the Mediterranean diet was not related to prevalence of the metabolic syndrome. However, red meat intake was associated with higher prevalence of blood pressure criteria (elevated blood pressure or anti-hypertensive treatment, P for tendency=0.028). In a positive-quality prospective cohort study, Damiao et al, 2006 examined the association between dietary factors and the metabolic syndrome in a sevenyear follow-up of a cohort of Japanese-Brazilians. Of 647 subjects included at baseline in 1993, 394 subjects participated in a second survey in 2000, and 151 Continues…. Conclusion Create Research Use EAL® to identify areas for future research • Lower level Conclusion Statements (III, IV, V) and Recommendations (Weak, Consensus, Insufficient Evidence) are clearly areas for future research ADA Outcomes Research • Epidemiological research preferred for questions about harm, prognosis and diagnoses d ag oses • RCT preferred for treatment questions • Dietetics PracticePractice-Based Research Network Feasibility and Field Test MNT Guides Nutrition Quality of Life (Validation) CTIMEE (Critical Thinking in Measuring Energy Expenditure) Key Messages Systematic reviews are intended to sythensize and describe the “state of the science” Value of using all types of research in the body of evidence based on type of question being answered Clearly demonstrate which research studies were evaluated in tables of evidence EAL contains ~ 39% observational types of research Questions?? “Evaluating “E l ti th the E Evidence” id ” and d “Making Recommendations” are TWO Separate activities Grading or Rating is used in describing BOTH the strength of the evidence and strength of the recommendation 12