Using Data for Decision Making
in Health Care
“Plot the dots.”
Greg Ogrinc, MD, MS
Dartmouth Medical School
27 June 2009
1
Objectives
 Investigate
how healthcare systems are
measured and the effect of this
measurement
 Identify
how understanding and
measuring variation is integral to the
model for improvement
 Apply
the rules of analysis for
evaluating data over time from a system
2
Agenda
 Storytime!
 P4P
(10 min)
with buzzgroups and discussion (10
min)
 Evidence,
improvement, and measurement
(30 min)
 Data, group work, and discussion
 Introduction
to variation and data over time
(time ordered data) (40 min)
 Data, group work, and discussion
3
Storytime…
4
Percent Normotensive (<140/<90) (Avg=74.39, UCL=83.66, LCL=65.11 for subgroups Jun-07-May-08)
100%
90%
80%
UCL
Target
Avg
% patients with normal BP
70%
60%
LCL
50%
40%
30%
20%
10%
0%
May-05
Jun-05
Jul-05
Aug-05 Nov-05
Jan-06 May-06
Jul-06
Sep-06
Oct-06
Date
Jan-07
Mar-07
Jun-07
Jul-07
Sep-07 Nov-07 Jan-08
Mar-08 May-08
5
HTN letter
High blood pressure can lead to heart disease and stroke if not well
controlled. We review the risks and benefits of changing their
blood pressure treatment regimen on a regular basis. Checks of
your Blood Pressure show that it is higher than 140/90, so we need
to get this under better control to reduce your risk of heart disease
and stroke.
You can do the following things to help keep your blood pressure
under control:
- Reach or maintain a normal body weight
- Eat a healthy diet without too much salt.
- Eat plenty of fruits and vegetables
- Limit the amount of caffeine
- Include regular physical activity in your schedule
- Do not drink more then 2 ounces of liquor or 2 glasses of beer or
wine in one day.
6
HTN letter
In order to help you get better control of your blood
pressure, I recommend the following:
(1) Discontinue HYDROCHLOROTHIAZIDE.
(2) Start CHLORTHALIDONE 100MG each morning.
I have made the changes in the computer and the
meds will be mailed to you.
(3) Please come to the WHITE MOUNTAIN FIRM for
a blood pressure check in 2 weeks. You do not
need an appointment.
(4) Stop by the laboratory for blood work on the same
day you come for the blood pressure check.
7
P4P in NHS
Doran et al, NEJM, 2006
 Evaluated
first year of P4P for >8000
family practices in England
Measured proportion of patients eligible for
whom the indicator was met
 Diabetes – % blood pressure <145/<85
 Hypothyroid – % blood check within 15 mos
 Stroke – % cholesterol < 193
Able to request exceptions for some patients
8
9
P4P in NHS
Doran et al, NEJM, 2006
 Median
 Level
level of achievement was 83%
of achievement was affected by
 Age and socioeconomic of patients
 Size of practice, # patients per practitioner, age of
practitioner, whether practitioner was trained in the
UK
 Exception reporting (1% increase in exception
reporting increased achievement by 0.6%)
 Small
number of practices achieved high
scores by excluding large number of patients
 More common in small practices
10
Buzz Groups

Turn to the person next to you
Use the stories you just heard as well as your
knowledge about pay for performance
(P4P) – news, articles, journals, internet.
1.
What are the benefits of this approach to
improving quality?
2.
What might be some drawbacks to this
approach?
11
Evidence-based Improvement
Generalizable
Scientific Evidence
+
Particular
Patient
Measured
Performance
Improvement
Batalden, 2003
12
Evidence-based Improvement
Generalizable
Scientific Evidence
+
Particular
Context
Measured
Performance
Improvement
Batalden, 2003
13
Evidence-based Improvement
executing
locally
choosing
best
plan
Generalizable
Scientific evidence
• control for context
• generalize across
contexts
• experimental
design
• statistics
+
Particular
Context
• understand system
“particularities”
• learn structures,
processes,
patterns
• culture and context
of changes
Measured
Performance
Improvement
• balanced
measures
• clinical
• functional
• satisfaction
• costs
142003
Batalden,
Individual Patient Care versus
Systems of Care
Individual patient
Initial work-up History, physical
exam, chart
review
Further work- Labs, xrays,
up
ultrasound,
functional tests
Therapy
Surgery,
medications,
watchful waiting
System
Your experience
within system,
discuss with others
Process & causeeffect diagrams,
outcomes data
model for
improvement,
PDSA, root cause
analysis
15
“quality improvement”
The combined and unceasing efforts of
everyone – health care professionals,
patients and their families, researchers,
administrators, payers, planners, educators
– to make changes that will lead to better
patient outcome, better system
performance, and better professional
development.
Batalden and Davidoff, QSHC, 2007
16
Better
patient (population)
outcome
Better
professional
development
Everyone
Better
system
performance
17
Batalden and Davidoff, QSHC, 2007
Better
patient (population)
outcome
Better
professional
development
Everyone
Better
system
performance
18
Batalden and Davidoff, QSHC, 2007
Competence?
Accreditation / certification / licensure?
Faculty / curricula development?
Professional school admission / selection?
Interprofessional cooperation?
Joy / creativity / pride?
Measures?
Patient knowledge?
Variation?
Causes?
Better
patient /
Hiring / orientation? population
outcome
Supervision?
Accountability?
Participation /
commitment?
Recognition /
reward?
Better
professional
development
Linking / leadership?
Org. development?
Governance?
Financing?
Everyone
Better
system
performance
Measures?
Options / methods?
Reliability / failure?
What
Standards?
Linking / leadership /
supervisory development?
Leadership performance
review?
theRecognition / reward?
might be
foci of inquiry?
20
A Model for Improvement
What are we trying to accomplish?
How will we know that a change is an improvement?
What change can we make that will result in improvement?
ACT
PLAN
STUDY
DO
Langley et al. , The
Improvement Guide, 1996
21
Assumptions about Measurement for QI
 Measurement
has become “popular” and arises in
many venues (e.g., P4P, country comparisons, etc)
 Measurement
and measures fit into the model for
improvement
 Part of the cycle
 Necessary element
 Understanding
the measures means
understanding the level of the system to which the
measures relate
 How
you display and data can have a significant
effect on how it is interpreted and used
 Know how the measures relate to the system
22
What are the Challenges
of Measurement?

Time consuming/added work.

Threatening, especially when it is used against you.

Making sure the data are accurate and consistent.

Too many indicators; not the appropriate indicators.

Using the data you collect to actually take action.

Manual versus automated data collection systems.

Results don’t match management’s view of reality.

The indicators were given to me by my manager and I had
no input.

Ownership of the data collection process and the results.

Lack of training in data collection methods and analysis.
23
23
What are the Benefits of
Measurement?

Helps you make decisions; build confidence.

Allows you to keep tabs on what is going on.

Sets the stage for improvement/identifies problem areas.

Provides a common frame of reference for staff and
management.

Identify patterns and trends in the data.

See how well performance matches our goals.

Helps us focus on what is important.

Helps you “sell” your ideas to management.

Understand interrelationships between departments and units.

Moves you away from anecdotes and one person’s view.
24
24
The Quality Measurement Journey
AIM
(Why are you measuring?)
Concept
Measure
Operational Definitions
Data Collection Plan
Data Collection
Analysis
ACTION
Source: Lloyd, R. Quality Health Care. Jones and Bartlett Publishers, Inc., 2004:
62-64.
25
25
The Quality Measurement Journey
AIM – freedom from harm for hospitalized patients
Concept – reduce patient falls
Measure – falls rate (falls per 1000 patient days)
Operational Definitions - # falls/inpatient days
Data Collection Plan – monthly; no sampling; all IP
units
Data Collection – unit submits data to RM; RM
assembles and send to QM for analysis
Tests of
Change
Analysis – control chart
26
26
Relating the Three Faces of
Performance Measurement to your work
Not mutually exclusive silos
All three areas must be understood
as a system – interdependent.
Individuals need to build skills in all
three areas.
Organizations need translators who
are able to speak the language of
each approach.
Individuals often identify with one of
these approaches and dismiss the
value of the other two.
27
27
Primer on Heart Attacks

Often called “MI” or “AMI”

Symptoms include chest pain or pressure, shortness
of breath, sweating, nausea and may come on with
exertion
 Individuals often have specific risk factors (genetics, smoking,
diabetes, high blood pressure or cholesterol…)

Treatment aimed at opening up blocked artery to
restore blood flow to part of the heart and reducing the
work the heart has to do

Treatments may include
 Catheter with stent to open artery or medication to dissolve
clot
 Meds to thin blood (reduce chance new clot will form) like
aspirin and others
 Meds to reduce the work of the heart like beta-blockers and28
ACE inhibitors
What Happens to People
Having Heart Attacks?
Patient
Symptoms
Emergency
System
Emergency
Department
Cardiac
Care Unit
Where should we give aspirin?
29
Opportunities To Give Aspirin
Aspirin
Aspirin
Aspirin
Aspirin
Patient
Symptoms
Emergency
System
Emergency
Department
Cardiac
Care Unit
•Take when
calls
•Family
education
•Before
symptoms
start
•EMT gives
•MD orders
by radio
•MD orders
•Nurse
protocol
•Pre-print
admit
orders
30
31
Dartmouth Atlas
CMS Technical Process Quality Measures Scores
(0-100; low–hi)
Hospital
Composite
AMI
CHF
Pneumonia
California Pacific
Med Ctr
85.5
92.8
93.5
72.3
Hartford Hospital
80.1
90.2
84.0
60.7
BIDMC
92.5
96.4
94.5
86.5
Catholic Med Ctr
93.2
98.8
93.0
86.3
DHMC
88.3
96.0
88.5
78.5
Fletcher Allen
85.7
92.0
84.5
78.5
Medicare claims data, 2005
32
Dartmouth Atlas
CMS Technical Process Quality Measures Scores
(0-100; low–hi)
State
Composite
AMI
CHF
Pneumonia
California
85.4
92.3
88.1
73.5
Connecticut
88.6
92.7
88.6
81.6
Massachusetts
89.2
94.7
89.2
80.0
New Hampshire
90.7
95.3
90.8
83.1
Vermont
87.9
91.3
84.3
84.5
Medicare claims data, 2005
33
Group Work, Part 1
 Review
the data table from the first
hand-out
 Discuss
the questions in your group
 Be
prepared to discuss in the large
group setting
34
“If I had to reduce
my message for
management to
just a few words,
I’d say it all had to
do with reducing
variation.”
W. Edwards Deming
35
35
If you don’t understand the variation that
lives in your data, you will be tempted to ...

Deny the data (It doesn’t fit my view of reality!)

See trends where there are no trends

Try to explain natural variation as special
events

Blame and give credit to people for things
over which they have no control

Distort the process that produced the data

Kill the messenger!
36
Demonstrating Variation: Example #1
The figure below shows how two individuals, Mary and Bill,
placed ten shots on a target. Which individual is the better
shot?
Mary
37
Bill
37
Demonstrating Variation: Example #1
• Mary shots are clustered
• Bill hit the bulls eye, but his pattern is erratic
• Mary merely has to adjust her sights down and to the left. Then, her shots
should all cluster near the center of the target.
• People take the most current data point and assume that this represents the
process's performance
• If you are really sincere about understanding where your processes have been,
where they are now, and where they can be in the future, you must become
knowledgeable about the types of variation and how to depict them.
38
Mary
Bill
38
Demonstrating Variation: Example #2
Using whatever method you have for telling time (watch,
phone, Blackberry, PC, position of the sun, etc.) record the
exact time right now and decide what time it is at your
table!
39
39
The Problem
Aggregated data presented in
tabular formats or with summary
statistics will not help you
measure the impact of process
improvement/redesign efforts.
Aggregated data leads to
judgment, not to improvement.
40
40
Types of Variation
Common Cause Variation

Is inherent in the design of the
process

Is due to regular, natural or
ordinary causes

Affects all the outcomes of a
process

Results in a “stable” process
that is predictable

Also known as random or
unassignable variation
Special Cause Variation

Is due to irregular or
unnatural causes that are
not inherent in the design of
the process

Affect some, but not
necessarily all aspects of the
process

Results in an “unstable”
process that is not
predictable

Also known as non-random
or assignable variation
41
Key Point …
Common Cause does NOT mean “Good Variation.”
• It only means that the process is stable and predictable.
• If a patient’s systolic blood pressure averaged around
165 and was usually between 160 and 170 mmHg, this
might be stable and predictable but completely
unacceptable.
Special Cause variation does NOT mean “Bad Variation”
• A special cause may represent a very good result (e.g.,
a low turnaround time), which you would want to
emulate.
•Special Cause merely means that the process is
unstable and unpredictable.
42
Balancing under and over
acting to minimize net loss
Action
taken
Action
not taken
Action
needed
X
loss
Action not
needed
loss
X
43
Measure
Elements of a Run Chart
Time ordered
observations
The centerline
(CL) on a Run
Chart is the
Median
X (CL)
Four simple tests are used to
determine if special cause
variation is present
44
Elements of a statistical
process control (SPC) chart
Natural Process Limits
Mean
Measured
value
(“x”)
Time-ordered observations
(1
n)
45
Tests to Identify Special
Causes on a Run Chart
 Test
#1: Too few or too many runs
 Test
#2: A shift in the process
 7 data points on one side of the median
 Test
#3: A trend
 7 data points constantly going up or down depending
on how many data points you have on the chart)
 Test
#4: A saw toothed pattern
46
Total data points = 29
Run Chart: Medical Waste
Data points on the Median = 2
Number of “useful
observations” = 27
6.00
The number of runs = 14
5.75
5.50
Pounds of Red Bag Waste
5.25
5.00
4.75
Median=4.610
4.50
4.25
4.00
Points on the
Median (don’t
count these as
“useful
observations”)
3.75
3.50
3.25
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
Point Number
47
Does this chart allow us to understand variation?
48
How about this one?
49
Is this one any better?
Isn’t this a downward trend?
50
This must surely be an upward trend!
51
Maybe this one is a better chart?
If so, why?
Is this the
Median?
52
Chronic Obstructive Pulmonary
Disease Run Chart
COPD Length of Stay
1.
Find the Median
2.
Determine the “useful
observations”
3.
Apply the run test rules
spell los
14.0
12.0
8.0
6.0
4.0
33 data points with 2 on the median
2.0
We have 31 useful observations
de
c
oc
t
au
g
ju
n
04
Ap
r-
fe
b
de
c
oc
t
au
g
ju
n
03
Ap
r-
fe
b
de
c
oc
t
au
g
ju
n
02
0.0
Ap
r-
Days
10.0
Month
53
Chronic Obstructive Pulmonary
Disease Run Chart
COPD Length of Stay
1.
Find the Median
2.
Determine the “useful
observations”
3.
Apply the run test rules
spell los
14.0
12.0
8.0
6.0
4.0
12
runs (should be between 11 and 21 runs)
2.0
de
c
oc
t
au
g
04
Ap
r-
fe
b
de
c
oc
t
au
g
ju
n
03
Ap
r-
fe
b
de
c
oc
t
au
g
ju
n
02
Are there 7 data points constantly increasing (trend)?
ju
n
Are
there more than 8 points in a run above or below the median (shift) ?
0.0
Ap
r-
Days
10.0
Month
54
Group Work, Part 2
 Review
the run charts on the hand-out
 Discuss
the questions in your group
 Be
prepared to discuss in the large
group setting
55
56
57
58
Summary
 How
measures are used have a strong influence
on the care that is delivered (P4P)
 To
be most effective, the measures must be
clearly connected to an aim/goal, the process,
and changes that are tried
 Measuring
data over time (and using run charts
or statistical process control charts) creates
useful measurement that provides insight into
the process
 Measurement for improvement and decision-making,
not just for judgment and research
59