Fundamental Problems with a
Class of Quality and Safety Measures
The Perils of Exposure‐Time
M i c h a e l D. H o we l l , M D M P H
D i r e c t o r, C r i t i c a l C a r e Q u a l i t y
A s s o c i a t e D i r e c t o r, M e d i c a l C r i t i c a l C a r e
D i r e c t o r, R e s e a r c h C o r e f o r I n S I G H T
From the
Silverman Institute for Health Care Quality & Safety
Beth Israel Deaconess Medical Center
Harvard Medical School
Disclosures
E
Employment
l
t
 Beth Israel Deaconess Medical Center
G
Grants
 Robert Wood Johnson Foundation
Grant # 65121 (PI): Advancing the Science of Quality Improvement
G
# 6
(PI) Ad
i h S i
f Q li I
 Grant # 66350 (PI): Physician Faculty Scholars Program
Center for Integration of Medicine and Innovative Technology (CIMIT)
 Clinical Systems Innovation Grant 2010 (Co‐investigator)


 Other (Commercial / Financial / Etc.)
 None
An Important Category of
Quality and Safety Measures
“EXPOSURE‐TIME” METRICS
A Major Category of Measures in QI
j
g y
Q
Risk =
Risk # of bad events
Length of time patients are eligible to have event
Ventilator‐Associated Pneumonia (VAP)
(
)
VAP risk = 1000 *
VAP risk 1000 # of VAPs
Ventilator‐days
Falls
Fall risk = 1000 *
Fall risk 1000 # of falls
Patient‐days
Central Line Infections
Central
line = 1000 *
infections
# of line infections
Central line days
How prevalent are these measures?
Recommended, Required, or Being Tested By
Metric
Falls or falls with injury per 1,000 patient‐days
Pneumonias per 1,000 ventilator‐days
Bloodstream infections per 1,000 central‐line‐days
l d
f
ll
d
Infections per 1,000 urinary‐catheter days
Institute for Healthcare I
Improvement
t
The Joint C
Commission
i i
2,3,4,8
10








National National Quality State‐level Healthcare S f t N t
Safety Network
k
F
Forum
public reporting
bli
ti
1
9, 10
5,6,7,11,12











Public reporting
p
g
BIDMC rate = 3.44
Conclusion: BIDMC medical floors are 21% safer than Hospital X’s medical floors. Right?
g
Hospital X’s rate = 4.35
A Research Question Naturally Arises
Q
y
Do these types of metrics provide the “right”
right
answer to
• Researchers?
• Quality improvement teams?
• Patients
P ti t
• Payors?
• Policymakers?
Pragmatic Decision: Focus on Falls
Why focus on falls?
y
 Cli
Clinical and policy case
i l d li  Common
 Lethal
L th l
 Expensive
 Major national priority
 Research case
 High‐quality data available to allow hybrid methods approach
High q alit data a ailable to allo h brid methods approach
We’re going to talk about
three things.
three things
Three things
g
Build an existence proof with a thought experiment
B
ild i t
f ith th ght i
t
2. Use empirical data to get plausible estimates of
1.
How much fall risk varies across the exposure window (LOS)?

How much the exposure window (LOS) varies
(
)
 Among hospitals
 Within one hospital over time

3.
Use simulation studies to determine how much completely artifactural error these variation cause in the exposure‐time metric (falls per 1,000 patient‐
days).
)
Step 1: Thought Experiment
The Case of
Arbitrary Regional General Hospital
ARGH QI: 2010
Q
 ARGH decides to work on improving falls.
ARGH d id t k i
i g f ll
 They contract with their insurer to improve falls.
 If they succeed, insurer agrees to pay for one extra day of LOS.
 They want to use Teflon©‐coated floors.
 Management agrees: “It’s not the fall, it’s the friction!”
ARGH QI: 2010
Q
 Admit 100 patients per year.
Ad it ti t  Risk of falling:
 HD#1:
 HD#2:
 HD#3:
 HD#4:
 HD#5:
5%
0%
0%
0%
0%
ARGH QI: 2011
Q
 IImplementation
l
t ti
 Takes a whole year
 It’s hard to install Teflon®! That stuff is slippery.
It’ h d t i t ll T fl ®! Th t t ff i li
ARGH QI: 2012 (Evaluation Year)
Q
(
)
 Admit 100 patients per year (unchanged)
Ad it ti t (
h g d)
 Median LOS increases to from 2 to 3 days.
 Risk of falling:
Post
HD#1: 8% (increased by
y 4060%)
35
30
HD#2: 0%
25
20
0
HD#3: 0%
15
10
HD#4: 0%
5
0
 HD#5: 0%
1
2
1
2
3
4
5
50 
45
40
35 
30
25

20
15
10 
5
0
Number o
of Paitents
Number o
of Paitents
Pre
Length of Stay (days)
3
4
Length of Stay (days)
5
What does ARGH conclude?
Admissions and Length of Stay
Number of admissions
Median Length of Stay (days)
2009
(baseline)
2011
(post‐intervention)
100
100
2
3
Truth: Teflon® flooring results in a 45
5
40
10
60% increase in falls.
in falls
10
35
g
g
Patients with a given length of stay
One day
y
Two days
Three days
Four days
y
Five days
4
1
30
20
Conclusion: Falls and Fall Risk
Teflon® flooring results in a Risk of Falling
p
y
5%
8%
Hospital Day #1
20% reduction
in falls
in falls.
Hospital Day #2
0%
0%
Patient‐days
Hospital Day #3
Hospital Day #4
176
350
0%
0%
0%
0%
How could ARGH have made H
ld ARGH h d such a catastrophic error?
p
Step 2: Empirical data
Real‐world variation in fall risk f
across the exposure window
Results: Inpatient Fall Distribution
p
Dates evaluated
Patients
Patient‐days
Falls
Falls per 1 000 patient days
Falls per 1,000 patient days
Result
2005 ‐ 2008
121,865
583,786
1,513
26
2.6
Challenge
g
W
We have very few VERY long stay patients.
h f VERY l g t ti t
 Median LOS: 3 days
 95% of patients with LOS ≤ 14 days
9 % f ti t ith LOS 1 d
 Therefore lots of noise as we get to long lengths of stay.
Risk of Falling on a Given Hospital Day
Dailyy Fall Risk
0.4%
0.3%
0 2%
0.2%
0.1%
0.0%
Hospital Day
Risk of Falling on a Given Hospital Day
Dailyy Fall Risk
0.4%
0.3%
0 2%
0.2%
0.1%
y = 0.0006ln(x) + 0.0018
r² = 0.6408
0.0%
Hospital Day
Real‐world variation in
length of stay
Methods: Data
N
National Inpatient Sample 2001 & 2006
ti
l I
ti t S
l & 6
 Largest all‐payor database of U.S. discharges
 Representative, stratified random probability sample
R
t ti t tifi d d b bilit l
 n ~ 8,000,000 records  ~ 40 million discharges (weighted)
 n = 1,045 hospitals
n = 1 045 hospitals
Perce
ent of U..S. Hosp
pitals LOS variability
among hospitals in one year
LOS Variability 70%
59.5% 60%
50%
40%
27.0% 30%
20%
6.8% 10%
4.7% 0.9% 0.2% 0.1% 0.1% 0.1% 0.7% 0%
1
2
3
4
5
6
7
8
9 ≥ 10 Median Length of Stay (days) Length of Stay Histogram Among Hospitals with Median LOS of 4 days
g
y
g
g
p
y
50%
45%
40%
35%
30%
25%
20%
15%
10%
5%
0%
Length of Stay (Days)
Within‐Hospital Variations in Length of Stay Over Time
8
Length of Stay Varies in the Same Hospital
g
y
p
Median Length of Stay (Days)
7
6
5
Number of Hospitals
Total n = 244
4
1
1
1
1
1
1
2
3
10
11
13
15
16
32
134
3
2
1
NIS 2001
NIS 2006
Conclusions: Empirical data
Conclusions from empirical data
p
 Daily fall risk varies by more than two‐fold across the LOS.
D il f ll i k i b th t
f ld th LOS
 LOS varies among hospitals in the same year.
 LOS varies within the same hospital over time.
Step 3: Simulation Studies
DOES THIS RESULT IN MEANINGFUL VARIATION IN THE “FALLS/1,000 PATIENT‐DAYS” MEASURE?
Methods: Hybrid Approach with Real Data
y
pp
Approach: A
h  Create experimental conditions where:


We know the true underlying, time‐varying fall risk.
We impose an identical time‐varying fall risk on every hospital.
 Use real data to provide fall risk distributions and LOS variability measures.
Risk of Falling on a Given Hospital Day
Daily Fall Risk
0.4%
0.3%
0.2%
0.1%
0.0%
Hospital Day
Yes
Fall
0.0018
Fall
Yess
0.9982
0 0022
0.0022
Fall?
No
Ye
es
Fall?
No
Fall
0.0053
0 9978
0.9978
N
No
Repeat x 800 million.
Fall?
0.9947
No Fall
HD#1
HD#2
HD#3
HD#365
In this experiment,
every patient in every U.S. hospital
is exposed to an
id i l fall risk distribution.
identical
f ll i k di ib i
Risk of Falling on a Given Hospital Day
D
Daily Fall Risk
0.4%
0.3%
0.2%
0.1%
0.0%
Hospital Day
What would these hospitals report?
Misleading metric variation
among hospitals
at one point in time
i i i
Percen
nt of U.S. Hospitaals Artifactual Variation in Falls per 1,000 Patient‐Days
Patient
Days 30%
25.0% 25%
20.5% 20.5% 20%
15%
9.9% 10%
8.2% 4.5% 5%
2.9% 0.4% 0.2% 0.1% 0.2% 0.4% 1.3% 0.5% 0.7% 1.7% 2.7% 0.4% 0%
1.5
1.7
1.9
2.1
2.3
2.5
2.7
Falls per 1,000 Patient‐Days 2.9
3.1
Is this meaningful?
4
Artifactual Variability ‐‐ California Hospitals How would this affect California?
Falls p
per 1,000 Patient Days Remember: all of these actually had the SAME fall risk distribution
fall risk distribution.
3
2
1
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85
Misleading metric variation in individual hospitals over time
Within‐Hospital Artifactual Variation Over Time Nu
umber off Hospitaals (of 24
44) 60
50
40
30
20
10
0
Percent change in "Falls Per 1,000 Patient Days" Percent
change in "Falls Per 1 000 Patient Days"
in the same hospital over a 5‐year period Conclusions
Conclusions
 When true fall rates are held constant
Wh t f ll t h ld t t…
 The “falls per 1,000 patient days” metric (exposure‐
time metric) will cause policymakers to mistakenly
) ll
l
k
i k l
believe that



The “best” and “worst” performers in a single state may have rates that vary by more than 50%.
18% of hospitals in a given year have substantial variation from the average hospital
35% of hospitals
35
p
improve or worsen over time by more than 30%
p
y
3
Next steps
p
 Conduct hybrid empirical / simulation studies on C d t h b id i i l / i l ti t di exposure‐time metrics other than falls
 Test a potential solution to the errors introduced by the l l
h
d
db h
mathematical construction of this metric class
Particular thanks to …
 Robert Wood Johnson Foundation
R b t W d J h
F
d ti
 Collaborators
 Meghan Dierks, MD  Long Ngo, PhD
 D. Paul Phillips, PhD (Dept of Mathematics, U. of Dallas)
l hilli
h (
f
h
i
f ll )
 All of the dry erase markers who gave their lives in pursuit of this work
i f hi k
Thank you.
you