Ranking hospitals on surgical mortality:
The importance of reliability adjustment
Justin B. Dimick, MD, MPH
Assistant Professor of Surgery
University of Michigan
AcademyHealth Annual Research Meeting
July 28th, 2009
Surgical quality measurement
Outcomes – “Gold standard”
• Patients, payers, providers agree they
are important
• Better than other alternatives
• Hospital volume, process measures
• Strong track record in cardiac surgery
– NY, PA, CA report cards
Big problems with small samples
A technique for dealing
with statistical “noise”
• Reliability adjustment: Adjust point estimate of an
outcome for uncertainty by “shrinking” it back towards the
population average
• Empirical Bayes techniques
• An application of hierarchical modeling (estimating
random effects)
• Increasingly applied in measuring performance in
education and health care
Adjusting mortality rates for reliability:
Reliability
adjusted
mortality rate
Reliability
=
=
Hospital
mortality
Reliability
Signal
Signal + Noise
6/29/2009
=
+
Overall
mortality
1-Reliability
Variance (µi)
Variance (µi) + Variance (εij)/n
What actually happens:
For hospitals with large number of cases, more
weight is afforded to the hospital’s mortality rate and
less to overall rate
For hospitals with few cases, less weight is afforded
to the hospital’s mortality rate and more to the overall
rate
Research question:
Are hospital rankings based on reliabilityadjusted mortality better at predicting future
performance?
6/29/2009
Data Source, Study Population, and
Primary Outcome
• Data Sources:
National Medicare data (2003-2006)
• Procedures:
3 high-risk operations; focus of quality assessment
activities; common to uncommon.
– Coronary artery bypass
– Abdominal aortic aneurysm repair
– Pancreatic resection
• Primary outcome:
– Death within 30-days or prior to discharge
6/29/2009
Analytic approach:
• Risk-adjustment:
– Patient age, gender, race, admission acuity,
coexisting diseases using Elixhauser index
• Reliability-adjustment:
– Hierarchical modeling to estimate random
effects for each hospital (xtmelogit in
STATA)
– Estimated using empirical Bayes techniques
6/29/2009
Analytic approach:
2003-04
2005-06
1
Risk-adjusted
mortality
2
NOT adjusted
for reliability
4
Risk-adjusted
mortality
Adjusted for
reliability
3
Rank hospitals,
create 5 groups
(quintiles)
5
1
2
3
4
5
6/29/2009
Risk-adjusted
mortality
Risk-adjusted
mortality
Number of patients and hospital
caseloads, 2003-04
Operation
Medicare cases
Hospital caseloads,
mean (SD)
Coronary artery bypass
303,132
267 (256)
Abdominal aortic aneurysm
repair
70,863
33 (49)
Pancreatic resection
6,192
6 (13)
Pancreatic Resection
40
30
20
10
0
Risk-adjusted
Mortality (%)
50
60
70
80
90 100
Impact of reliability adjustment
Before reliability adjustment
After reliability adjustment
Risk-adjusted mortality rates
20 randomly sampled hospitals
Are reliability adjusted mortality rates better at
predicting future performance?
Pancreatic cancer resection
12.0%
10.8%
10.5%
10.0%
9.0%
Riskadjusted
mortality, %
(2005-06)
8.0%
7.6%
7.4%
5.9%
6.0%
4.5%
4.0%
3.4% 3.2%
2.7%
2.0%
0.0%
1
2
3
4
5
Not adjusted for reliablity
1
2
3
4
5
Adjusted for reliability
Hospital Rankings (Quintiles)
(2003-04)
Are reliability adjusted mortality rates better at
predicting future performance?
Abdominal aortic aneurysm repair
6.0%
4.9%
5.0%
Risk-adjusted
mortality
(2005-06)
4.0%
4.9%
4.0%
3.9% 4.0%
3.7%
3.1%
3.4%
3.2% 3.3%
3.0%
2.0%
1.0%
0.0%
1
2
3
4
5
Not adjusted for reliablity
1
2
3
4
Adjusted for reliability
Quintiles of Hospital Rankings
(2003-04)
5
Are reliability adjusted mortality rates better at
predicting future performance?
Coronary artery bypass
6.0%
5.4%
5.3%
5.0%
4.5%
Risk-adjusted
mortality
(2005-06)
4.4%
4.0%
4.0%
4.0%
3.4%
3.0%
3.3%
3.0%
3.0%
2.0%
1.0%
0.0%
1
2
3
4
Not adjusted for reliablity
5
1
2
3
4
Adjusted for reliability
Quintiles of Hospital Rankings
(2003-04)
5
Summary:
• Reliability adjusted mortality better
forecasts future performance for high-risk
surgery
• Most important for uncommon operations
– Pancreatectomy > AAA >> CABG
6/29/2009
Implications:
• Reliability adjustment should become the
standard for all efforts aimed at tracking surgical
outcomes
• Applications:
– Massachusetts cardiac surgery report card
– Blueprint for a new ACS-NSQIP
– The Leapfrog Group’s composite measure
6/29/2009
Acknowledgements
Douglas Staiger, PhD
Collaborator, Dartmouth Medical School
6/29/2009