Reframed Question…
Quality Measurement: Is the
Information Sound Enough to be
Used by Decision Makers?
ƒ How good is good enough?
ƒ For use by whom for what purposes?
ƒ Purchasers--changes in plan design to
reward higher quality, more efficient
providers, steer enrollment (premiums, out
of pocket costs)
ƒ Plans--incentive payments, tiering, narrow
networks, channeling or centers of
excellence
ƒ Consumers--to guide treatment choices
ƒ Providers--quality improvement
Cheryl L. Damberg, Ph.D., Director of Research
Pacific Business Group on Health
Academy Health: June 8, 2004
2
How Good is Good Enough?
Reality Check!
ƒ We don’t know what the right standard is
ƒ What information?
ƒ Measures exist--few implemented routinely or universally
ƒ Most providers have no clue what their performance is
ƒ Should standards apply in same way to all end users?
ƒ What are the dangers of “noisy” information?
ƒ “I’m following guidelines, it is someone else who isn’t”
ƒ Demming Toyota studies (Six Sigma) showed that when
gave back noisy information on performance
ƒ Is the current information better than no information?
ƒ Absent information—choice is like a flip of the coin (50:50)
ƒ Increased variation, decreased quality
ƒ Disorienting; lost natural instinct for how to improve
ƒ Decisions will still be made with no information or
poor information
ƒ How do we make optimal decision in the face of
uncertainty?
ƒ Default position is to base decisions solely on price
ƒ Consequences differ
ƒ Decision theory analysis could help to inform these
questions
ƒ Need research in this area
3
© Pacific Business Group on Health, 2004
ƒ Patient—inconvenience for little gain in outcome
ƒ Provider—ruin reputation, livelihood
4
What’s Currently Going On Out There
in Measurement?
M ea sures
ƒ Using administrative data, often with poor case
mix adjustment
ƒ omitted variables that can lead to biased results
ƒ handling of missing data
ƒ rank ordering problems that lead enduser to incorrect
decision
9 Link to
outcom es
9 Im portance
9 Valid
9 Reliable
ƒ Research
- level work
ƒ Doing shrinking estimates to address noise
problem without thinking about issues of
underlying data quality
© Pacific Business Group on Health, 2004
© Pacific Business Group on Health, 2004
Where in the Measurement Process
Can Things Go Wrong?
ƒ Two ends of the extreme…examples
ƒ Commercial vendors
5
© Pacific Business Group on Health, 2004
6
Im plem entatio n
9 Poor data
9 Sm all “n”
D isplay
R e porting
9 W ill
enduser
draw
correct
conclusion
based on
how
reported?
© Pacific Business Group on Health, 2004
1
Data: The Next Generation…..
Underlying Problem of Data Quality
ƒ One of the greatest threats to validity of
performance results are the data that “feed”
the measures
ƒ Even if quality measure is good (i.e., reliable,
valid), can still produce bad (“biased”) result if the
data used to score performance are flawed or if
the source of data omits key variables important
in predicting the outcome.
7
© Pacific Business Group on Health, 2004
8
© Pacific Business Group on Health, 2004
Example 1: Risk-Adjusted Hospital
Outcome for Bypass Surgery
ƒ CA CABG Mortality Reporting Program
Table 1: Comparison of Audited Data and CCMRP Submissions
for Acuity, All Hospitals, 1999 Data
Audited Data
Elective Urgent Emergent Salvage Total
ƒ 70 hospitals submitted data in 1999
ƒ Concern about comparability across hospitals in
coding
CCMRP Elective
Data
Urgent
ƒ Potential impact on hospital scores
ƒ Importance of “getting it right” given public reporting
ƒ 38 hospitals selected for audit
ƒ Focused on outliers or near outliers, with random
selection in the middle; over sampled high risk cases
ƒ 2408 cases audited
447
431
7
1
886
140
911
53
4
1,108
Emergent
16
117
199
3
335
Salvage
1
18
29
4
52
604
1,477
288
12
2,381
Total
ƒ Inter-rater reliability 97.6% (range: 95-99%: Cohen’s
Kappa)
9
© Pacific Business Group on Health, 2004
10
Results of Audit
© Pacific Business Group on Health, 2004
Table 1: Agreement Statistics, All Hospitals, 1999 Data
Variable
ƒ Revealed downcoding and upcoding
problems
ƒ Worst agreement: acuity (65.6%), angina
type (65.4%), angina class (45.8%), MI
(68.3%), and ejection fraction (78.0%)
Missing data: incorrect classification of risk
based on policy of replacing with lowest risk
Acuity
ƒ Ejection fraction (15.8%), MI (38.1%)
100.00
65.56
64.36
0
NA
65.37
34.73
2,408
0
NA
86.21
42.47
CCS Angina Class
2,408
105
79.05
45.76
53.19
Congestive Heart Failure
2,408
31
38.71
82.23
32.94
COPD
2,408
6
0.00
86.34
73.25
Creatinine (mg/dl)
2,408
556
3.96
93.31
56.37
Cerebrovascular Disease
2,408
3
0.00
87.67
45.79
Dialysis
2,408
91
0.00
98.13
86.67
Diabetes
2,408
3
0.00
94.73
45.67
Ejection Fraction (%)
2,408
228
15.79
78.95
60.27
Method of measuring ejection fraction 2,408
Left Main Stenosis
12
2
2,408
Angina (Yes/No)
Time from PTCA to surgery
© Pacific Business Group on Health, 2004
2,408
Angina Type (Stable/Unstable)
Hypertension
11
% Missing Values
% Lower Triangle
Records Missing
that Would be
Severity Weighted
Audited Values Incorrectly Classified % Agreement Disagreement
406
0.00
74.34
Not Calculated
2,408
7
85.71
84.39
40.43
125
45
42.22
78.40
12.50
2,408
388
7.22
85.96
51.46
© Pacific Business Group on Health, 2004
2
Results of Audit
Impact on Fitted Model Characteristics when Replacing
Audited Records with Information from Audit, 1999 Data
ƒ Classification of some hospitals as outliers
may be a result of coding deficiencies
ƒ When model was re
- run, saw changes in
statistical significance and/or risk differential
ƒ Death (outcome variable)—small levels of
disagreement can change hospital rating
ƒ Change in rankings
-7.74
0.00
Creatinine (mg/dl)
0.18
0.00
Congestive Heart Failure
0.38
Hypertension
0.14
Dialysis
0.39
0.04
Acuity
0.19
Elective
Estimate
p-value
-9.11
0.00
0.01
0.15
OR
0.55
0.00
1.73
0.04
1.25
1.20
0.00
1.46
0.18
1.15
0.23
0.18
1.47
1.24
0.00
3.45
1.21
0.25
0.01
1.29
Reference Group
1.01
Reference Group
Urgent
0.26
0.02
1.29
0.33
0.00
Emergent
1.24
0.00
3.46
1.33
0.00
3.77
Salvage
2.46
0.00
11.71
3.11
0.00
22.46
2
R
c-statistic
Hosmer-Lemeshow χ (p-value)
2
1.39
0.202
0.818
0.833
9.303 (0.317)
23.068 (0.003)
© Pacific Business Group on Health, 2004
Example 2: Pay for Performance
ƒ Audit
ƒ Data cross validation
ƒ Training on coding of variables; support to
hospital coders
ƒ Display of confidence intervals
ƒ Plan payouts to medical groups based on
rewarding those groups that rank at 75th
percentile or higher
ƒ Rank ordering problems
ƒ Medical groups with estimates based on small “n”
(i.e., noisy) more likely to fall in top or bottom part
of distribution
ƒ Straight ranking ignores uncertainty in estimates
ƒ Potential for rewarding wrong players
ƒ Small hospital with zero deaths (CI: 0.0%-10.0%)
ƒ Combine data over multiple years
ƒ Generate more stable estimates for small volume
hospitals
© Pacific Business Group on Health, 2004
0.188
14
Steps Taken to Safeguard Against
Getting it Wrong
ƒ Rewarding noise, not signal
16
© Pacific Business Group on Health, 2004
Example 3: Individual Physician
Performance Measurement
Can We Proceed?
ƒ Small “n” problem
ƒ OK to start with Version 1.0 of the measures
ƒ Physician lacks enough events (e.g., diabetics) to
score him/her at the level of the individual
indicator
ƒ Estimates at indicator level are noisy (large SEs)
ƒ Means of soliciting feedback
ƒ Help drive improvement in measurement
ƒ Won’t get it perfect on first attempt
ƒ Important to safeguard against possible
mistakes in classifying
ƒ Need to pool more information on physician’s
performance across conditions to improve
the signal to noise ratio
ƒ Check validity of data (audit, cross validate)
ƒ Assess extent of disagreement
ƒ Perform sensitivity analyses
ƒ Create summary scores (e.g., RAND QA Tools)
17
Intercept
OR*
Fit Statistics:
© Pacific Business Group on Health, 2004
15
p-value
Diabetes
ƒ 1 (no different Æ better than)
ƒ 6 (worse than Æ no different)
ƒ 1 (no different Æ worse than)
13
Model: CCMRP Data and Audited Data
Where Record was Audited
Model: CCMRP Data
Estimate
© Pacific Business Group on Health, 2004
18
© Pacific Business Group on Health, 2004
3
Hedging Against Uncertainty
Measurement Issues Remain
ƒ Conservative ways of reporting so don’t
mislead (level of certainty in estimate)
ƒ Existing measures
ƒ OK, but difficult to implement (many rely on chart review)
ƒ Hospital performance
ƒ Rank ordering—small groups may rank either in
the highest/lowest part of the distribution, yet we
are most uncertain of their true performance
ƒ Complexity of what to measure (service line vs. overall)
ƒ Physician performance
ƒ Cruder binning (categorization)
ƒ Small “n” problem; challenges of pooling data
ƒ When faced with more uncertainty or
consequences are higher
ƒ Comprehensive assessment important, but too much
information will overwhelm endusers
ƒ Use measures as a tool to identify bottom
performers, then send out teams to find out
what is going as a way to validate
19
© Pacific Business Group on Health, 2004
ƒ Need for summary measures
ƒ Need to improve data systems
20
© Pacific Business Group on Health, 2004
Why Do We Need to Fill the Gaps?
ƒ Lack of information and transparency
ƒ Hard to improve if you don’t know where the
problem is
ƒ Continue rewarding status quo
ƒ Need to increase competition to improve
quality and contain costs
ƒ Information is vital for competitive markets to
operate
21
© Pacific Business Group on Health, 2004
4