2010 Annual Research Meeting
Methods to Evaluate Quality Improvement Methods
to Evaluate Quality Improvement
Initiatives “Evaluation of Quality Improvement across
“Evaluation of Quality Improvement across a number of sites”
Gareth Parry • If a quality improvement initiative is found effective in one setting, would the same effects be found elsewhere?
l
h ?
• How can we improve the generalization of quality i
improvement research evaluations?
h
l i ?
Additional acknowledgements:
Additional acknowledgements:
• John Ovretveit
h
Director of Research, Professor of Health Innovation Implementation and Evaluation, Medical Management Centre (MMC), Karolinska
Institutet, Stockholm, Sweden.
kh l
d
• Laura Leviton
Special Advisor for Evaluation, The Robert Wood p
,
Johnson Foundation, New Jersey, USA.
Overview
• Where such problems occur in current practice.
h
h
bl
i
i
• A practical research design that aims to help providers choose whether to adopt a quality improvement change (QIC) that was tested elsewhere.
• A programme of research that is useful to practitioners, yet can accumulate knowledge useful to more general decision‐makers about which results could be expected in which situations from a QIC.
Background
External Validity?
Internal Validity
Sample
Selection
Sample
Measurement
??
Confounding
Chance
C l i
Conclusion
Clinical epidemiology
Fletcher, Fletcher, Wagner
It is impossible to step in the same river twice Heraclitus , BC
Internal & External Validityy
• Internal validity of a study is the validity of a claim that an intervention caused a change in outcome d h
(Campbell & Stanley, 1963).
• External validity about the outcomes of an intervention E t
l lidit b t th
t
f i t
ti
(Shadish, Cook & Campbell, 2001):
– Generalisation about a class of interventions: Validity of the Generalisation about a class of interventions: Validity of the
claim that the causal relationship holds across variation in g
,p
p p
,
p
,
organizations, patient populations, time periods, individual activities and measurement of outcomes – Generalisation to a new intervention: Degree of certainty about whether an intervention will produce similar outcomes for their organizations, patients, and situation.
• Generalisability describes the range of situations or units of study to which findings or methods from elsewhere can be applied.
Example
• Hospital‐wide Code Rates and Mortality Before and After Implementation of a Rapid p
p
Response Team (Chan et al, JAMA 2008)
• Conclusion:
– “In this large single‐institution study, rapid
response team implementation was not associated with reductions in hospital‐wide code rates or mortality.”
Small theories of improvement
Small theories of improvement
• M
Many quality improvements are not detailed lit i
t
t d t il d
prescriptions but concepts and ideas or objectives (“small
objectives (
small theories of improvement
theories of improvement”))
– which provide latitude to implementers to adapt them. • Few studies describe well the details of the adaptation of content or details of implementation
• Many change during the implementation, or are changed as part of an adaptation using a PDSA h
d
f
d
i
i
PDSA
cycle Example
• The Central Line Blood Stream Infection (
(CLBSI) bundle contains an element related to )
proper hand hygiene.
• Small theory of improvement: Small theory of improvement:
– Having hand cleaning material placed in positions where clinical staff will see them in the course of their standard line insertion and monitoring work will lead to their greater use, increased hand hygiene and lower rates of CLBSI. Example
• Amendments: d
g
y
g
• An organization may test a number of settings for the cleaning material:
1) Near the entrances to the care area 1)
Near the entrances to the care area
2) Near universal precautions equipment used in the course of line insertion.
course of line insertion. Etc...
• The
The usefulness of these placements are likely usefulness of these placements are likely
to be specific to the local context.
A Research Design?
A Research Design?
• Better understanding and description of:
– The small theory of improvement
y f p
– What was done to amend implementation of the small theory of improvement
small theory of improvement
– The context that modify or mediate the effect
– What can be generalizable
h
b
li bl
– Whether results of single or multiple‐site studies effect the small theory of improvement.
Generalizable
Knowledge
Site Implementation
A. Adaptation
d
Small Theory of Improvement
B. Adaptation
Case Description
Explanatory Theory
C. Adaptation
Test and Outcomes Refine
Improved?
Amendment 1
Amendment 2
Amendment 2
Amendment 3
Amendment 4
Yes
Amendment 1
Amendment
1
Amendment 2
Amendment 3
Amendment 4
Yes
Amendment 1
Amendment 2
Amendment 3
Amendment 4
No
Etc.
Prior Research
Generalizable
Specific to Context
Summaryy
• Theory‐driven research design.
• Research Funding? • An entity similar to the Cochrane Collaborative may help provide leadership
Collaborative may help provide leadership and expert synthesis of the results of quality improvement evaluation research
li i
l i
h
Questions?