2010 Annual Research Meeting Methods to Evaluate Quality Improvement Methods to Evaluate Quality Improvement Initiatives “Evaluation of Quality Improvement across “Evaluation of Quality Improvement across a number of sites” Gareth Parry • If a quality improvement initiative is found effective in one setting, would the same effects be found elsewhere? l h ? • How can we improve the generalization of quality i improvement research evaluations? h l i ? Additional acknowledgements: Additional acknowledgements: • John Ovretveit h Director of Research, Professor of Health Innovation Implementation and Evaluation, Medical Management Centre (MMC), Karolinska Institutet, Stockholm, Sweden. kh l d • Laura Leviton Special Advisor for Evaluation, The Robert Wood p , Johnson Foundation, New Jersey, USA. Overview • Where such problems occur in current practice. h h bl i i • A practical research design that aims to help providers choose whether to adopt a quality improvement change (QIC) that was tested elsewhere. • A programme of research that is useful to practitioners, yet can accumulate knowledge useful to more general decision‐makers about which results could be expected in which situations from a QIC. Background External Validity? Internal Validity Sample Selection Sample Measurement ?? Confounding Chance C l i Conclusion Clinical epidemiology Fletcher, Fletcher, Wagner It is impossible to step in the same river twice Heraclitus , BC Internal & External Validityy • Internal validity of a study is the validity of a claim that an intervention caused a change in outcome d h (Campbell & Stanley, 1963). • External validity about the outcomes of an intervention E t l lidit b t th t f i t ti (Shadish, Cook & Campbell, 2001): – Generalisation about a class of interventions: Validity of the Generalisation about a class of interventions: Validity of the claim that the causal relationship holds across variation in g ,p p p , p , organizations, patient populations, time periods, individual activities and measurement of outcomes – Generalisation to a new intervention: Degree of certainty about whether an intervention will produce similar outcomes for their organizations, patients, and situation. • Generalisability describes the range of situations or units of study to which findings or methods from elsewhere can be applied. Example • Hospital‐wide Code Rates and Mortality Before and After Implementation of a Rapid p p Response Team (Chan et al, JAMA 2008) • Conclusion: – “In this large single‐institution study, rapid response team implementation was not associated with reductions in hospital‐wide code rates or mortality.” Small theories of improvement Small theories of improvement • M Many quality improvements are not detailed lit i t t d t il d prescriptions but concepts and ideas or objectives (“small objectives ( small theories of improvement theories of improvement”)) – which provide latitude to implementers to adapt them. • Few studies describe well the details of the adaptation of content or details of implementation • Many change during the implementation, or are changed as part of an adaptation using a PDSA h d f d i i PDSA cycle Example • The Central Line Blood Stream Infection ( (CLBSI) bundle contains an element related to ) proper hand hygiene. • Small theory of improvement: Small theory of improvement: – Having hand cleaning material placed in positions where clinical staff will see them in the course of their standard line insertion and monitoring work will lead to their greater use, increased hand hygiene and lower rates of CLBSI. Example • Amendments: d g y g • An organization may test a number of settings for the cleaning material: 1) Near the entrances to the care area 1) Near the entrances to the care area 2) Near universal precautions equipment used in the course of line insertion. course of line insertion. Etc... • The The usefulness of these placements are likely usefulness of these placements are likely to be specific to the local context. A Research Design? A Research Design? • Better understanding and description of: – The small theory of improvement y f p – What was done to amend implementation of the small theory of improvement small theory of improvement – The context that modify or mediate the effect – What can be generalizable h b li bl – Whether results of single or multiple‐site studies effect the small theory of improvement. Generalizable Knowledge Site Implementation A. Adaptation d Small Theory of Improvement B. Adaptation Case Description Explanatory Theory C. Adaptation Test and Outcomes Refine Improved? Amendment 1 Amendment 2 Amendment 2 Amendment 3 Amendment 4 Yes Amendment 1 Amendment 1 Amendment 2 Amendment 3 Amendment 4 Yes Amendment 1 Amendment 2 Amendment 3 Amendment 4 No Etc. Prior Research Generalizable Specific to Context Summaryy • Theory‐driven research design. • Research Funding? • An entity similar to the Cochrane Collaborative may help provide leadership Collaborative may help provide leadership and expert synthesis of the results of quality improvement evaluation research li i l i h Questions?