Misleading Advice about
AMI Mortality from Medicare
Jeffrey H. Silber, M.D., Ph.D.
Center for Outcomes Research
The Children’s Hospital of Philadelphia
and
The Leonard Davis Institute of Health
Economics
The University of Pennsylvania
AcademyHealth, Chicago 2009
Co-Authors:
Paul R. Rosenbaum, Ph.D.
Tanguy J. Brachet, Ph.D.
Richard N. Ross, M.S.
Laura J. Bressler, B.A.
Orit Even-Shoshan, M.S.
Scott A. Lorch, M.D., M.S.C.E.
Kevin G. Volpp, M.D., Ph.D.
Background
• Medicare’s web-based “Hospital Compare”
(HC) is considered the gold standard of
public reporting, and many papers have
now been based on the HC methodology
• For AMI mortality the present HC model
suggests that of 4,311 hospitals, none
were worse than average and 9 were
better than average. We ask, are these
evaluations sound?
Overview of this talk
• Medicare’s HC states all (but 9) hospitals are about the
same—that there is no volume outcomes relationship
• Hospitals can’t all be the same if there is a significant
Volume Outcomes relationship
• But there is an OBVIOUS Volume-Outcomes relationship
in AMI, yet using the Medicare HC model, this VO
relationship disappears (it is actually assumed away
through the specification of the HC Random Effects
Model)
• In a simulated world where we purposely construct a VO
relationship, fitting the Medicare HC model takes away
the VO relationship
• We conclude that the Medicare HC model is providing
mistaken advice about small hospitals whenever there
truly is a volume-outcomes relationship—such as with
AMI
Medicare’s Motivation for using
a Random Effects model
Chassin et al. HSR
1989
The Small Numbers “Solution” as
discussed in the HC Website
• The [Medicare] hierarchical regression
model adjusts mortality rates results for …
hospitals with few heart attack … cases in
a given year….
• This reduces the chance that such
hospitals’ performance will fluctuate wildly
from year to year or that they will be
wrongly classified as either a worse or
better performer….
The Random Effects model utilized by Hospital Compare
(
)
Logit P (Y=
1)= α i + β Zij
ij
Yij = 1 if patient j treated at hospital i had the event (e.g. death)
α i =µ + ωi =the hospital effect for hospital i, where ωi is modeled as N ( 0,τ 2 )
µ = typical hospital effect for all hospitals in the data set
τ 2 = the between hospital variance component
Zij = covariates in the random effects model. In the present Hospital Compare
model these are only patient characteristics.
Note: As the sample size at hospital i falls, the random effects model shrinks ωi
towards 0. As the hospital size increases, there is less shrinkage.
How the Hospital Compare Random Effects (RE) Model “Shrinks”
Predictions Based on Hospital Size
Hospital Size
SMALL
BIG
Outcomes Before
Shrinkage
If High Death Rate above mean:
If Low Death Rate below mean:
If High Death Rate above mean:
If Low Death Rate below mean:
Predicted Outcomes
after RE Model
Shrinkage
Shrinks toward
the mean of all
hospitals
Little or No
Shrinkage
Observed/Expected and Predicted/Expected Outcomes
The Hospital Compare Adjusted Death Rate
 PREDICTED death rate 

 * National Rate =
 EXPECTED death rate 
Hospital Compare Adjusted Death Rate
RE MODEL (with patient factors and hospital outcomes)
RE MODEL (with just patient factors)
The HC Adjusted Death Rate is not based
on O/E, it is P/E!
The volume-outcomes relationship
in AMI—the literature
•
•
•
•
•
•
•
•
•
Shortell and LoGerfo 1981
Luft, Hunt, and Maerki 1987
Farley and Ozminkowski 1992
Casale, Jones, Wolf, Pei, and Marlin 1998
Thiemann, Coresh, Oetgen, and Powe 1999
Tu, Austin, and Chan 2001
Halm, Lee, and Chassin 2002
Gandjour, Bannenberg, and Lauterbach 2003
Lin, Chu, and Lee 2007
The volume-outcomes relationship
in AMI—the data
• 208,157 Medicare patients
• Admitted for AMI to 3629 hospitals between
July 1, 2004 to June 30,2005
• Logistic Regression Model includes 27
variables describing patient characteristics
as included in Krumholz et al. 2006
Logistic regression mortality model to
estimate mortality
Variable
Odds Ratio
95% CI
P-Value
Anterolateral Wall Infarction
(ICD-9 410.20-410.69)
1.468
1.419 to 1.520
<.0001
Unstable Angina (HCC 82)
1.034
0.974 to 1.097
0.2698
Chronic Atherosclerosis (HCC 83, 84)
0.492
0.480 to 0.505
<.0001
Cardiopulmonary-Respiratory Failure and
Shock (HCC 79)
1.231
1.152 to 1.314
<.0001
Valvular Heart Disease (HCC 86)
0.942
0.914 to 0.970
<.0001
Hypertension (HCC 89, 91)
0.655
0.639 to 0.672
<.0001
Stroke (HCC 95, 96)
2.160
2.056 to 2.270
<.0001
Cerebral Vascular Dissease (HCC 97-99,
103)
0.906
0.862 to 0.953
0.001
Renal Failure (HCC 131)
1.433
1.386 to 1.482
<.0001
Logistic regression mortality model to
estimate mortality
Variable
Odds Ratio
95% CI
P-Value
COPD (HCC 108)
1.179
1.147 to 1.212
<.0001
Pneumonia (HCC 111, 112)
1.255
1.200 to 1.313
<.0001
Diabetes (HCC 15-20, 120)
1.012
0.985 to 1.040
0.3924
Malnutrition (HCC 21)
1.664
1.575 to 1.758
<.0001
Dementia (HCC 49, 50)
1.469
1.421 to 1.519
<.0001
Hemiplegia/Paraplegia (HCC 68, 69, 100102, 177, 178)
1.247
1.181 to 1.316
<.0001
Peripheral Vascular Dis. (HCC 104, 105)
1.209
1.169 to 1.251
<.0001
Metastatic Cancer (HCC 7, 8)
2.491
2.345 to 2.645
<.0001
Trauma (HCC 154-156, 158-162)
1.080
1.036 to 1.127
0.0004
Major Psychiatric Disorders (HCC 54-56)
1.070
0.993 to 1.152
0.0741
Chronic Liver Disease (HCC 25-27)
1.847
1.631 to 2.091
<0.001
Logistic regression mortality
model to estimate mortality
Variable
Odds Ratio
95% CI
P-Value
Age
1.056
1.054 to 1.058
<.0001
Male Sex
1.079
1.052 to 1.106
<.0001
Hx PTCA (ICD-9 36.01, 36.02, 36.05)
0.845
0.722 to 0.989
0.0356
Hx CABG (ICD-9 36.03, 36.04, 36.06,
36.07, 36,09, 36.11-36.17)
0.921
0.829 to 1.022
0.1203
Hx HF (HCC 80)
1.300
1.262 to 1.340
<.0001
Hx MI (HCC 81)
1.041
0.987 to 1.098
0.1360
Anterolateral Wall Infarction
(ICD-9 410.00-410.19)
1.726
1.667 to 1.788
<.0001
Hospital Volume (69 pts / yr increase)
0.955
0.948 to 0.963
<.0001
Relationship between the hospital volume quintile
and the odds of mortality
Hospital Volume
by Quintile
Number of
Hospitals
Logit Model
Random Effects
Model
Quintile 1
(Volume < 8 per year)
734
1.54
(1.41, 1.68)
1.53
(1.50, 1.56)
Quintile 2
(Volume 8-21 per year)
753
1.29
(1.23, 1.36)
1.29
(1.26, 1.32)
696
1.13
(1.10, 1.18)
1.13
(1.11, 1.16)
Quintile 4
(Volume 48-95 per year)
732
1.08
(1.05, 1.11)
1.07
(1.04, 1.10)
Quintile 5
(Volume 96-740 per year)
714
1
reference
1
reference
Quintile 3
(Volume 22-47 per year)
Mortality Ratio by AMI Volume for 3629 Hospitals
We can all agree there is a volumeoutcome relationship in AMI
•
•
•
•
Present in the literature
Present in the data
Present in the logit model
Present even in the Random Effects model
• BUT HC SUGGESTS THERE IS NO VOLUMEOUTCOMES RELATIONSHIP—They state
small hospitals are doing average, and almost
all large hospitals are average too. Remember,
of 4311 hospitals, 9 were better than average,
none were worse than average….WHY?
Mortality Ratio by AMI Volume for 3629 Hospitals
SUPPOSE WE CREATE AN
ARTIFICIAL WORLD WHERE THERE
DEFINITELY IS A VOLUMEOUTCOMES RELATIONSHIP
WILL MEDICARE STILL TELL US
EVERY HOSPITAL IS AVERAGE??
Comparing hospital volume and the HC compare model evaluations using simulated data
Simulated Hospital Evaluations
from the CMS Hospital Compare Model
Quintile 1
Worst Predicted
Mortality
Quintile
2
Quintile
3
Quintile
4
Quintile 5
Best Predicted
Mortality
Quintile 1 (Vol. < 8)
“True” Worst
N Hospitals
(Row %)
30
(4.1)
187
(25.4)
308
(42.0)
202
(27.5)
7
(1.0)
734
Quintile 2 (Vol. 8-21)
N Hospitals
(Row %)
104
(13.8)
184
(24.4)
187
(24.8)
199
(26.4)
79
(10.5)
753
Quintile 3 (Vol. 22-47)
N Hospitals
(Row %)
173
(24.9)
148
(21.3)
105
(15.1)
136
(19.5)
134
(19.3)
696
Quintile 4 (Vol. 48-95)
N Hospitals
(Row %)
215
(29.4)
127
(17.4)
81
(11.1)
95
(13.0)
214
(29.2)
732
203
(28.4)
80
(11.2)
45
(6.3)
94
(13.2)
292
(40.9)
714
725
726
726
726
726
3629
Quintile 5 (Vol. 96-740)
“True” Best
N Hospitals
(Row %)
N Hospitals
N
Hosp
IN THE SIMULATED WORLD WHERE
THERE TRULY IS A V-O RELATIONSHIP,
HC ELIMINATES THIS RELATIONSHIP
IN THE REAL WORLD WHERE THERE
APPEARS TO BE A V-O RELATIONSHIP, HC
ALSO ELIMINATES THIS RELATIONSHIP
How mistaken is the HC Advice?
• Very mistaken!
• Individual small hospital death rates are
being systematically reported as average
when as a group, they are performing far
worse than groups of hospitals with larger
size
• The Random Effects model is appropriate,
but Medicare’s use of the RE model is not
appropriate, and people are being given
incorrect advice
Conclusion
• Patients and policymakers would be well
advised to stop using the HC model in its
present form