Enhanced Risk Adjustment from
Adding Laboratory Test Measures
to Administrative Data in the VA
Amresh Hanchate, PhD
Boston University &
VA Bedford / CHQOER
Collaborators
Ann Borzecki, VA Bedford / Boston Univ.
Michael Shwartz, Boston Univ. / VA
Boston
Arlene Ash, Boston University
Amy Rosen, VA Bedford / Boston Univ,
Funded by VA HSR&D
Borzecki (PI)
Background
Accurate risk adjustment is critical for quality
assessment and provider comparisons
Although ideal, chart-based clinical data are
costly and time-consuming to collect
Risk adjustment continues to be based on
administrative data, despite known limitations
– CMS Hospital Compare
Adding Automated Clinical Data
In recent years, common for hospitals to have automated
clinical information
– Laboratory test results
– Vital sign results
AHRQ initiative across different states to pool
standardized laboratory test data from participating
community hospitals
Early studies of enhanced risk adjustment indicate
improved prediction of inpatient or 30-day mortality
– Pine et al (JAMA 2007) - 188 PA hospitals
– Tabak et al (Med Care 2008) – 266 participating hospitals
– Escobar et al (Med Care 2008) – Northern California Kaiser
Permanente
VA Automated Clinical Data
Largest integrated health care system in the US
with considerable automated clinical data
covering all hospitals
Better standardization of data collection and
aggregation
Render et al (Critical Care 2005; 2008) – ICU
mortality
Objectives
To examine how adding laboratory test data to
a standard administrative data-based risk
adjustment model
improves prediction of inpatient mortality
changes relative rankings at hospital-level
Table 1. Study Cohorts
Fiscal Years 2004-2007
#
admissions
# hospitals
AMI
30,119
141
CHF
87,755
145
COPD
62,339
149
GI Hemorrhage
38,448
144
Cirrhosis & Alc. Hepatitis
14,328
143
Hip Fracture
8,745
133
Pneumonia
74,148
151
Renal Failure
25,859
147
Acute Stroke
25,582
145
Admission Cohort
Methods
Outcome: In-hospital Mortality
Administrative data-based risk adjustment
(Admin) model:
– Age, sex
– Admission secondary diagnosis ICD-9 codes grouped
into AHRQ Comorbidity categories
Enhanced risk adjustment (ERA) model:
– Adds following 6 laboratory test results to the
administrative data-based model
– sodium, BUN, WBC, creatinine, bilirubin and
hematocrit
– measured within 24 hours of admission time
– dichotomous indicators (abnormal test range)
– dichotomous indicator for missing test measure
Methods
Outcome: In-hospital Mortality
Administrative data-based risk adjustment
(Admin) model:
– Age, sex
– Admission secondary diagnosis ICD-9 codes grouped
into AHRQ Comorbidity categories
Enhanced risk adjustment (ERA) model:
– Adds following 6 laboratory test results to the
administrative data-based model
– sodium, BUN, WBC, creatinine, bilirubin and
hematocrit
– measured within 24 hours of admission time
– dichotomous indicators (abnormal test range)
– dichotomous indicator for missing test measure
Methods - 2
Each cohort split evenly into estimation and
validation samples
Estimated hierarchical logistic regression of
administrative data-based and ERA models
using the estimation sample
Estimates applied to validation sample to
measure prediction accuracy of the two models
Hospital-level risk adjusted rates calculated
using CMS Hospital Compare method
Table 2. Comparison of Risk Adjustment Models for
Inpatient Mortality: Discrimination and Calibration
Admission
Cohort
AMI
CHF
COPD
GI Hemorrhage
Cirrhosis & Hep.
Hip Fracture
Pneumonia
Renal Failure
Acute Stroke
Obs. Mort.
Obs. Mort.
Rate (%) in
Rate (%) in
Top Decile Bottom Decile
Observed
C-statistic
In-hospital
Mortality
Admin ERA Admin ERA Admin ERA
Rate (%)
Model Model Model Model Model Model
6.3
3.6
2.5
2.8
8.8
6.2
6.4
6.2
6.3
0.75
0.72
0.71
0.77
0.66
0.72
0.74
0.74
0.71
0.80
0.78
0.75
0.82
0.78
0.73
0.77
0.79
0.78
1.0
0.8
0.5
0.3
3.0
1.4
0.9
1.3
2.0
0.8
0.4
0.5
0.1
0.8
0.7
0.8
0.6
0.8
19.8
11.2
8.0
10.4
18.7
15.4
19.6
17.9
18.2
25.0
14.3
9.3
13.0
34.4
17.7
21.7
23.2
22.6
Table 2. Comparison of Risk Adjustment Models for
Inpatient Mortality: Discrimination and Calibration
Admission
Cohort
AMI
CHF
COPD
GI Hemorrhage
Cirrhosis & Hep.
Hip Fracture
Pneumonia
Renal Failure
Acute Stroke
Obs. Mort.
Obs. Mort.
Rate (%) in
Rate (%) in
Top Decile Bottom Decile
Observed
C-statistic
In-hospital
Mortality
Admin ERA Admin ERA Admin ERA
Rate (%)
Model Model Model Model Model Model
6.3
3.6
2.5
2.8
8.8
6.2
6.4
6.2
6.3
0.75
0.72
0.71
0.77
0.66
0.72
0.74
0.74
0.71
0.80
0.78
0.75
0.82
0.78
0.73
0.77
0.79
0.78
1.0
0.8
0.5
0.3
3.0
1.4
0.9
1.3
2.0
0.8
0.4
0.5
0.1
0.8
0.7
0.8
0.6
0.8
19.8
11.2
8.0
10.4
18.7
15.4
19.6
17.9
18.2
25.0
14.3
9.3
13.0
34.4
17.7
21.7
23.2
22.6
Table 2. Comparison of Risk Adjustment Models for
Inpatient Mortality: Discrimination and Calibration
Admission
Cohort
AMI
CHF
COPD
GI Hemorrhage
Cirrhosis & Hep.
Hip Fracture
Pneumonia
Renal Failure
Acute Stroke
Obs. Mort.
Obs. Mort.
Rate (%) in
Rate (%) in
Top Decile Bottom Decile
Observed
C-statistic
In-hospital
Mortality
Admin ERA Admin ERA Admin ERA
Rate (%)
Model Model Model Model Model Model
6.3
3.6
2.5
2.8
8.8
6.2
6.4
6.2
6.3
0.75
0.72
0.71
0.77
0.66
0.72
0.74
0.74
0.71
0.80
0.78
0.75
0.82
0.78
0.73
0.77
0.79
0.78
1.0
0.8
0.5
0.3
3.0
1.4
0.9
1.3
2.0
0.8
0.4
0.5
0.1
0.8
0.7
0.8
0.6
0.8
19.8
11.2
8.0
10.4
18.7
15.4
19.6
17.9
18.2
25.0
14.3
9.3
13.0
34.4
17.7
21.7
23.2
22.6
Table 2. Comparison of Risk Adjustment Models for
Inpatient Mortality: Discrimination and Calibration
Admission
Cohort
AMI
CHF
COPD
GI Hemorrhage
Cirrhosis & Hep.
Hip Fracture
Pneumonia
Renal Failure
Acute Stroke
Obs. Mort.
Obs. Mort.
Rate (%) in
Rate (%) in
Top Decile Bottom Decile
Observed
C-statistic
In-hospital
Mortality
Admin ERA Admin ERA Admin ERA
Rate (%)
Model Model Model Model Model Model
6.3
3.6
2.5
2.8
8.8
6.2
6.4
6.2
6.3
0.75
0.72
0.71
0.77
0.66
0.72
0.74
0.74
0.71
0.80
0.78
0.75
0.82
0.78
0.73
0.77
0.79
0.78
1.0
0.8
0.5
0.3
3.0
1.4
0.9
1.3
2.0
0.8
0.4
0.5
0.1
0.8
0.7
0.8
0.6
0.8
19.8
11.2
8.0
10.4
18.7
15.4
19.6
17.9
18.2
25.0
14.3
9.3
13.0
34.4
17.7
21.7
23.2
22.6
Table 2. Comparison of Risk Adjustment Models for
Inpatient Mortality: Discrimination and Calibration
Admission
Cohort
AMI
CHF
COPD
GI Hemorrhage
Cirrhosis & Hep.
Hip Fracture
Pneumonia
Renal Failure
Acute Stroke
Obs. Mort.
Obs. Mort.
Rate (%) in
Rate (%) in
Top Decile Bottom Decile
Observed
C-statistic
In-hospital
Mortality
Admin ERA Admin ERA Admin ERA
Rate (%)
Model Model Model Model Model Model
6.3
3.6
2.5
2.8
8.8
6.2
6.4
6.2
6.3
0.75
0.72
0.71
0.77
0.66
0.72
0.74
0.74
0.71
0.80
0.78
0.75
0.82
0.78
0.73
0.77
0.79
0.78
1.0
0.8
0.5
0.3
3.0
1.4
0.9
1.3
2.0
0.8
0.4
0.5
0.1
0.8
0.7
0.8
0.6
0.8
19.8
11.2
8.0
10.4
18.7
15.4
19.6
17.9
18.2
25.0
14.3
9.3
13.0
34.4
17.7
21.7
23.2
22.6
Table 3. Comparing Facility Performance using Risk Adj.
Mortality (RAM) from Admin. and ERA Models
Admission
Cohort
Concordance: # common Discordance: Average rank
facilities out of 10
difference between Admin.
lowest/highest RAM
and Lab. models for facilities
facilities from each model
without concordance
Lowest RAMHighest RAMLowest RAM Highest RAM
AMI
CHF
COPD
GI Hem.
Cirrhosis
Hip Fracture
Pneumonia
Renal Fail
Acute Stroke
5
4
5
5
2
4
1
2
5
7
3
4
7
5
5
3
3
5
13
21
39
31
15
16
32
20
42
15
25
47
10
11
10
21
30
28
Note: Only facilities with a minimum of 50 admissions for each admission
cohort included.
Table 3. Comparing Facility Performance using Risk Adj.
Mortality (RAM) from Admin. and ERA Models
Admission
Cohort
Concordance: # common Discordance: Average rank
facilities out of 10
difference between Admin.
lowest/highest RAM
and Lab. models for facilities
facilities from each model
without concordance
Lowest RAMHighest RAMLowest RAM Highest RAM
AMI
CHF
COPD
GI Hem.
Cirrhosis
Hip Fracture
Pneumonia
Renal Fail
Acute Stroke
5
4
5
5
2
4
1
2
5
7
3
4
7
5
5
3
3
5
13
21
39
31
15
16
32
20
42
15
25
47
10
11
10
21
30
28
Note: Only facilities with a minimum of 50 admissions for each admission
cohort included.
Table 3. Comparing Facility Performance using Risk Adj.
Mortality (RAM) from Admin. and ERA Models
Admission
Cohort
Concordance: # common Discordance: Average rank
facilities out of 10
difference between Admin.
lowest/highest RAM
and Lab. models for facilities
facilities from each model
without concordance
Lowest RAMHighest RAMLowest RAM Highest RAM
AMI
CHF
COPD
GI Hem.
Cirrhosis
Hip Fracture
Pneumonia
Renal Fail
Acute Stroke
5
4
5
5
2
4
1
2
5
7
3
4
7
5
5
3
3
5
13
21
39
31
15
16
32
20
42
15
25
47
10
11
10
21
30
28
Note: Only facilities with a minimum of 50 admissions for each admission
cohort included.
Table 3. Comparing Facility Performance using Risk Adj.
Mortality (RAM) from Admin. and ERA Models
Admission
Cohort
Concordance: # common Discordance: Average rank
facilities out of 10
difference between Admin.
lowest/highest RAM
and Lab. models for facilities
facilities from each model
without concordance
Lowest RAMHighest RAMLowest RAM Highest RAM
AMI
CHF
COPD
GI Hem.
Cirrhosis
Hip Fracture
Pneumonia
Renal Fail
Acute Stroke
5
4
5
5
2
4
1
2
5
7
3
4
7
5
5
3
3
5
13
21
39
31
15
16
32
20
42
15
25
47
10
11
10
21
30
28
Note: Only facilities with a minimum of 50 admissions for each admission
cohort included.
Summary
Adding laboratory test measures to administrative
data-based risk-adjustment models
improved the ability to identify patients at
higher risk for inpatient death, and
significantly changed relative hospital rankings
Limitations
Missing rates of laboratory test results sizable
(up to 28%) and varied significantly by hospitals
Abnormal ranges of laboratory test results may
vary for admission cohort
Robustness of hospital performance changes to
be measured