Health Care Q
Qualityy Improvement
p
(QI)
(Q )
“A
A broad range of activities of varying degrees of
complexity and methodological and statistical rigor
through which health care providers develop
develop,
implement, and assess small-scale interventions and
identify those that work well and implement them
more broadly in order to improve clinical practice.”
* The Ethics of Improving Health Care Quality & Safety: A Hastings
Center/AHRQ Project, Mary Ann Baily, PhD, Associate for Ethics &
Health Policy, The Hastings Center, Garrison, New York, October, 2004
Characteristics of Health Care QI
– Contextual factors (background variables or confounders in
research) are usually a major focus
– The initial intervention (changes to the system) are adapted and
modified as study progresses
– Measurement is done over time (improvement is temporal)
– Graphical analysis and presentation (Statistical Process Control, SPC)
– Involvement of local expertise in conducting project
– Multiple experimental cycles for rapid feedback and learning
Multi-factor
factor experiments to learn from complex systems with
– Multi
non-linear and dynamic cause and effect relationships
– Building reliability of the interventions can be a major part of the
effort
ff
– Sustainability is a consideration from the beginning of the project
Gold Standard for Building
g Evidence
Clinical Research:
Of all research designs, the randomized controlled
trial with adequate numbers of patients,
patients blinding of
therapists, patients and researchers, and carefully
standardized methods of measurement and analysis is
the best evidence for cause-effect relationships.
Quality Improvement:
Satisfactory prediction of the results of tests conducted
over a wide range
ove
ange of conditions.
Minimum Standard for Reporting in QI Projects:
Annotated Time Series
QI Project to Reduce Length of Stay in ED
320
300
Quick-look
x-rays
Work-up done on floor
Bed
ahead
LOS (minutes)
L
280
260
Individual responsible
for bed control
240
220
G l
Goal
200
180
160
2/16/98
3/16
4/13
Week
5/11
6/8
Time Series Chart vs. Before/After Summary
Average of multiple points
before and after change
80
90
70
Cycle Time (min.)
100
80
70
60
50
60
50
40
30
20
40
10
30
0
D ec
N ov
Oct
Sep
A ug
Avg Before
Change
Jul
M ar
Feb
Jan
0
A pr
10
Jun
20
M ay
Change
Made
date
Cyycle Tim e (m in.)
Cli i A
Clinic
Measures of Visit Cycle time (minutes)
Avg After
Change
Time Series Chart vs. Before/After Summary
Average of multiple points
before and after change
Clinic B
80
70
Cycle Time (min.)
100
80
70
60
60
50
40
30
50
20
40
10
30
0
20
Avg Before
Change
Dec
Nov
Oct
Sep
Aug
Jul
Jun
Apr
Mar
Feb
0
Jan
10
May
Change
Made
date
Cyclle Time (m
min.)
90
Measures of Visit Cycle time (minutes)
Avg After
Change
Time Series Chart vs. Before/After Summary
Average of multiple points
before and after change
Cli i C
Clinic
80
100
70
Cycle Time (min.)
80
70
60
50
60
50
40
30
20
40
10
30
0
20
Avg Before
Change
Dec
Nov
Oct
Sep
Aug
Jul
Jun
Mar
Feb
Jan
0
Apr
10
May
Change
Made
date
Cyccle Time (m
min.)
90
Measures of Visit Cycle time (minutes)
Avg After
Change
100
80
Cycle Tim
me (min.)
Clinic A
70
60
50
40
30
Dec
Nov
Oct
Sep
Aug
Jul
May
Apr
Mar
Jan
0
Feb
10
Jun
Change
Made
20
date
Time Series
Ch t vs.
Chart
Before/After
Summary
90
100
90
60
50
40
30
20
Dec
Nov
Oct
Sep
Aug
Jul
May
Apr
Feb
Jan
Mar
60
date
0
Jun
Change
Made
10
70
100
50
90
40
30
20
Dec
Nov
Apr
Mar
0
Feb
Avg After
Change
Jan
Avg Before
Change
Oct
Change
Made
10
Sep
0
50
Aug
10
60
Jul
20
Clinic B
70
Jun
30
80
May
35
40
Cycle Tim
me (min.)
C y cle T im e (m iin .)
70
Clinic
U itB2
Unit
70
date
80
Cycle Time (min
C
n.)
Cycle Time Results
for Clinics A, B, C
80
The Foundation of the Science of Improvement:
D i ’ S
Deming’s
System
t
off P
Profound
f
dK
Knowledge
l d
Appreciation
of a system
Theory of
Knowledge
g
y
gy
Psychology
Understanding
Variation
“The aim
“Th
i off this
hi chapter
h
is
i to provide
id an outside
id view
i – a lens
l
– that
h I call
ll a system off
profound Knowledge. It provides a map of theory by which to understand the
organizations that we work in.”
The Science of Improvement:
Increased capability to make improvements
Subject Matter
Knowledge
Learn to combine subject matter
knowledge and profound
knowledge in creative ways to
develop effective changes for
improvement
Appreciation
off a System
S t
Increased
capability to
Make
Improvements
Profound Knowledge
Theory of
Knowledge
Psychology
Understanding
Variation
Framework or Roadmap for
Improvement Project
Act
- What changes
are to be
made?
- Next cycle?
St d
Study
- Complete the analysis
of the data
- Compare data to
predictions
- Summarize what
was learned
Plan
- Objective
- Questions and
predictions (Why?)
- Plan to carry out
th cycle
the
l
(who, what, where, when)
- Plan for Data collection
Do
- Carry out the plan
- Document problems
and unexpected
observations
- Begin analysis
of the data
PDSA – The Continuous
Application of the Scientific
Method for Improvement
11
Repeated
p
Use of the PDSA Cycle
y
Model for Improvement
What are we trying to
accomplish?
How will we know that a
change is an improvement?
Changes That Result in
Improvement
A P
What change can we make that
will result in improvement?
S D
Implementation of
Change
Wide-Scale Tests
of Change
A P
S D
Hunches
Theories Ideas
Follow-up
Tests
Very Small
Scale Test
12
The Beginning of the
Science of Experimental
Design (1920’s)
Sir Ronald Fisher
Rothamsted Experimental Station
13
Internal Validity
Sampling
Sample
Sample
Selection
Measurement
??
g
Confounding
Chance
E t
External
l Validity
V lidit
Conclusion
(generalizability)
Clinical epidemiology
Fletcher, Fletcher, Wagner
W E
W.
E. Deming’s
Deming s Two Types of Studies
The aim of any experiment is to
provide a rational basis for
action
Enumerative study: an
experiment in which action will
be taken on the universe.
Chapter 7 from
Some Theory of Sampling, 1950
(First discussion in JASA, 1942)
Analytic study: an experiment in
which action will be taken on a
cause system to improve
performance in the future.
Environment for Enumerative Study
Internal Validity
Sampling
Random
Sample
Sample
Selection
Measurement
??
g
Confounding
Chance
Conclusion
External Validity
(generalizability)
Clinical Epidemiology
Fletcher, Fletcher, Wagner
Environment for Analytic Study (QI Studies)
Internal Validity
Sample
Selection
Sample
Measurement
??
Confounding
Chance
C
Conclusion
l i
Clinical epidemiology
It is
i impossible
i
ibl tto step
t iin th
the same river
i
ttwice
i
Heraclitus , BC
Fletcher, Fletcher, Wagner
Important Aspects of Enumerative and Analytic Studies
Type of Study
Aspects of Study
Enumerative
Analytic
Aim
Estimation
Prediction
Method of Access
Frame & Sample
Models of product
or process
o
p ocess
Major Source of
uncertainty
Sampling Error
Extrapolation to the
future
Major Source of
uncertainty
quantifiable?
Yes
No
Environment of
the study
Static
Dynamic
Role of the
statistician
Assess important
effects
Support subject
matter expert
Role of the subject
matter expert
Define the universe
approve the frame
Identify variables, levels:
assess conditions in the
future; assess degree of belief
Planned Experimentation, 1999
18
Estimation & Prediction in Various Types of Studies
Leverage
For
Improvement
Low
Estimation
Application
Examples
Acceptance sampling
Value inventory
Census
Theory to support
use of the standard
error of prediction
Probability distribution
in combination with a
frame and sampling by
random numbers
Role of subject
matter expert
Approvall off the
A
th
frame and definition
of the complete
coverage
Exit Poll of voters
Stable Process - no expected changes
Stable process - minor changes
St bl process – major
Stable
j changes
h
Shewhart
Shewhart
Weaker justification
Shewhart
Weaker justification
Poll 2-4 months before election
None
Prediction that
conditions will be
relatively
l ti l unchanged
h
d
in the future.
Prediction that
The change will not
interact with future
conditions.
Reasonable simulation
of the real thing
Pilot p
plant studies
Prototype testing
Accelerated life tests
Low dose extrapolation
High
Prediction
Planned Experimentation, 1999
19
Understanding Variation
The Shewhart chart is a
statistical tool used to
distinguish between
variation in a measure due
t common causes and
to
d
variation due to special
causes
Walter Shewhart
(1891 – 1967)
W. Edwards Deming
((1900 - 1993))
Chart Title
100
Upper Lim it
Measu
ure
90
Center Line
80
Low er Lim it
70
60
1
2
3
4
5
6
7
8
9
10
11
12
13
Run Order
14
15
16
17
18
19
20
21
22
23
24
Using Shewhart Chart to Provide Evidence of Improvement
Evidence of Improvement
Percent Adm witth Harm
45
40
35
30
25
20
15
10
5
0
1
3
5
7
9
11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47
Months
Figure 4-12: Shewhart Chart Revealing Process or System Improvement
Data Guide p. 4‐13
Using Shewhart Chart to Provide Evidence of
Sustainability of Improvement
45
Holding the Gain
Perce
ent Admissiions with Ha
arm
40
35
30
25
20
15
10
5
0
4-15: Shewhart Charts Depicting a Process or System “Holding the Gain”
Data Guide p. 4‐15
Experimental Patterns for QI Studies
Speroff and O’Connor, Study Designs for PDSA Quality Improvement
Research, Q .Manage Health Care, Vol 13, No.1, 2004
Multiple Time Series Design
Speroff and O’Connor, Study
Designs for PDSA Quality
Improvement Research, 2004
Study Design for IHI/GCHS Under 5
Mortality Project
Multiple Baseline,
Baseline Interrupted Time-Series Design
Health Center A
O O X X X X X X
M
M
M M
M
M
M
M
Health Center B
O O
O
X
X X X X
M
M
M M
M
M
M
M
Health Center C
O O
O
O
O X X X
X
M
M M M
Health Center D
O O
O
O
O O X X
X
X
M
M M M M M
Health Center E
O O
O
O
O
O O O O X
X
X
X
M
M
M
Health Center F
O O
O
O
O
O O O O X
X
X
M
M
M
M
Health Center G
O O
O
O
O
O O O O X
X
X M M M M
Health Center H
O O O
O
O
O O O O
X X
Month
1 2
3
Wave One
O – baseline data
4
O
5 6 7 8 9
X
M
M
M
M
M
M
10 11 12 13 14 15
Wave Two
X –intervention period
M – maintenance period
25
Improving Care in a Pediatric Diabetes Clinic
Percent of Diabetes Patients whose HgbA1c measurement has reached CCHMC goal
Latest HgbA1c result in preceding 13 months was used for each patient
All Therapies Included
40%
Control limits: 3 sigma
g n = 1308 range
g 1205 - 1413
average
35%
Protected
Phone
Time
CNPs
25%
Tighter Blood
Glucose
Targets
20%
15%
10%
Choice of
Th
Therapy
att
Onset
Ease in
Changing
NPH to
BBT
Consensus
Sick Day
Guidelines
Insulin Pens in Pharmacy
Care
M
Management
t for
f
DKA
Heightened A1c Goal
G
Awareness
5%
Short-staffed CNPs
Care Management for A1c
0%
Ja
nFe 03
b0
M 3
ar
-0
Ap 3
rM 03
ay
-0
Ju 3
n0
Ju 3
l-0
Au 3
gS e 03
p0
O 3
ct
N 03
ov
-0
D 3
ec
-0
Ja 3
nFe 04
b0
M 4
ar
-0
Ap 4
rM 04
ay
-0
Ju 4
n0
Ju 4
l-0
Au 4
gS e 04
p0
O 4
ct
N 04
ov
-0
D 4
ec
-0
Ja 4
nFe 05
b0
M 5
ar
-0
Ap 5
rM 05
ay
-0
Ju 5
n0
Ju 5
l-0
5
Percentt of Patients
30%
Month
Control Limits
Percent
Center Line
Last update: 06-23-05 by H. Atherton, Data source: Disease Management Database
Special Cause
26
Factorial Design for Improvement in Diabetes Clinic
Response Variables:
• HbA1C level
• ED visits
Care Management
g p of
Tests either subgroups
patients or time periods
Therapy
Th
1
Therapy
Th
2
No C
N
Care
Management
Th
Therapy
Th
Therapy
1
2
Manage
to easy
Glucose
G
ucose
Targets
No Sick day
guidelines
Test 12
Test 7
Test 3
Test 14
Sick day
guidelines
Test 2
Test 16
Test 11
Test 6
Manage
to Tight
Glucose
Targets
No Sick day
guidelines
Test 9
Test 13
Test 1
Test 15
Sick day
guidelines
Test 10
Test 4
Test 8
Test 5
27
2^4-1 Factorial for Diabetes Clinics
Standard Therapy
Self Management
Visit Freq:
std
No
Screening
g
Preventive
Screening
Education
Extensive
Optional Therapies
Self Management
Education
Test 3
Test 7
accell
T t8
Test
T t4
Test
std
Test 1
Test 6
accel
Extensive
Test 5
Randomize order of tests and/or sub-sets of patients populations
Test 2
Analysis of Data from Quality
Improvement Studies
The aim of the analysis is to give the experts in the subject
matter the best possible chance to take the right action.
Fi steps:
Five
t
1. Show all the data
2. Plot the data in the order in which the tests were conducted.
3. Use graphical displays to assess how much of the variation in the
d can be
data
b explained
l i d by
b factors
f
that
h were deliberately
d lib
l changed.
h
d
4. Rearrange this plot to study other sources of variation
5. Summarize
i the
h results
l off the
h study
d with
i h appropriate
i graphical
hi l
displays.
29
Some References for Analytic QI Studies
•
•
•
•
•
•
•
•
•
•
“Classification Problems of Statistical Inference” W. E. Deming, Journal of the American
Statistical Association, June 1942.
Some Theory of Sampling
Sampling, W.
W Edwards Deming (1950),
(1950) Dover Publications,
Publications NY.
NY
“On Probability as a Basis for Action”, The American Statistician, Vol. 29, No. 4, (1975),
pp 146-152.
Out of the Crisis
Crisis, W.
W Edwards Deming (1982),
(1982) MIT CAES
CAES, Cambridge.
Cambridge
Quality Improvement Through Planned Experimentation, Ron Moen, Thomas Nolan, Lloyd
Provost (1999), McGraw-Hill, NY.
ssu pt o s for
o Stat
Statistical
st ca Inference”,
e e ce , The
he American
me ican Statistician, Vo
Vol 47,
7, No. 1,, Ge
Gerald
ad
“Assumptions
Hahn, William Meeker (1993), pp 1-11.
“A Primer for Enumerative Vs. Analytic Studies: Using Caution in Statistical Inferences,”
ASQ Statistics Division Newsletter, Vol 16, No. 3 Eileen Beachell, Marilyn Monda (1996),
pp 6-10.
Quality Improvement through Planned Experimentation, 2nd edition Ron Moen, Thomas
Nolan, Lloyd Provost (1999), McGraw-Hill, NY.
The
h New Economics for
f Industry,
d
Government, Education.
d
2ndd Edition.
d
W. Edwards
d
d
Deming, Cambridge: MIT Press, 2000
The Data Guide: Learning from Data to Improve Health Care, L. Provost and S. Murray,
Associates in Process Improvement,
Improvement 2010.
2010