The measurement problem
Measurement theory and provider
profiling
construct
indicator
Quality
εi(jk)
Timothy P. Hofer MD
Dept. Medicine,
University of Michigan
VA Ann Arbor HSR&D
Center of Excellence
Levels of care
Implications of the measurement model
Site
(clinic, hospital)
„
The indicator is a fallible measure of the
construct
„
„
Provider
(physician)
„
„
Some indicators are less precise than others
Quality indicators are very imprecise for a variety
of reasons
You need to account for the measurement error
The location of the construct variability can
suggest different causes, interventions and
measurement procedures
Patient
i
i i
Indicators
Intra-class correlation(=reliability)
„
„
Ability to distinguish between physicians
(or sites)
single observation under a specified set
of conditions of measurement.
2
⎛
⎞
σ
physician
⎟
Rx = ⎜ 2
2
2
⎜
⎟
+
+
⎝ σ site σ physician σ patient ⎠
Vol. 281 No. 22, pp. 2065-2160, June 9, 1999
1
Table 3: Amount of variation in hospitalization and outpatient visit rates attributable
MD laboratory utilization profiles
to a physician practice style effect
Adjusted for reliability
Unadjusted for reliability
Age-Sex Adjusted
Casemix Adjusted
13%
10%
7%
4%
0.51
0.40
Unadjusted for reliability (R2) *
8%
8%
Reliability adjusted (ICC) †
2%
1%
0.24
0.17
Deviations of MD profiles from mean laboratory utilization ($)
Variable
Visits
% of variation associated w/ physician
Unadjusted for reliability(R2) *
60
Reliability adjusted (ICC) †
Reliability‡
20
Hospitalizations
% of variation associated w/ physician
-20
-60
Reliability‡
Resources available
„
VA Network 11 Diabetes Care Project
VA Diabetes Registry Project (1998-2001)
„
Automated Clinical Databases
„
„
Data warehouse (VA Healthcare and analysis group)
Database Components
„
„
„
„
„
„
„
Cohort identification procedure
Data quality and measure validation
„
Selected Measures:
Resource Use
„
„
„
Encounter records (OPC/PTF )
Outpatient Pharmacy
Lab
primary care provider database (PCMM ()
Vitals
Kerr EA , et al. Journal on Quality Improvement 2002;
28(10):555-65.
Selected Measures :
Intermediate Outcomes
Cost of hypoglycemic medications
Cost of home glucose monitoring for
patients not on insulin
Cost of calcium channel blockers
„
„
„
„
Last A1c value
A1c ≥ 9.5%
Last LDL value
LDL ≥ 3.6 mmol/L (140mg/dl)
Quality
Outcomes
Intermediate
Outcomes
Processes
Quality
Outcomes
Intermediate
Outcomes
Processes
2
Selected Measures:
Process Measures
„
„
„
Selected Measures:
Mixed or Linked Measure
Hemoglobin A1c obtained
LDL-C obtained
Lipid profile obtained
„
LDL ≥ 3.6 mmol/L (140mg/dl)
or
on a statin
Quality
Outcomes
Intermediate
Outcomes
Processes
Are there differences between
physicians?
„
Outcomes
What are the sources of variation?
„
„
„
„
„
Diabetes Care Indicator
Noise
Unmeasured differences
Physician effects
Noise
Clinic or group effects
Physician
Health System/payor effects
Clinic
Intermediate outcomes
Diabetes Care Indicator
Intermediate Outcomes
Last HbA1c value
Team
PCP
1%
0
2%
Cost of home glucose monitoring
for patients not on insulin
18%
3%
3%
Cost of home glucose monitoring
for patients on insulin
8%
2%
1%
Cost of Calcium Channel
Blockers
1%
0
0
Resource Use
Cost of hypoglycemic
medications
System/Payor
Level of Care
Facility
Process measures
Level of Care
Diabetes Care Indicator
Facility
Team
PCP
12%
0
1%
Last HbA1c value ≥ 9.5%
16%
0
0
Last LDL-C value‡
7%
–
1%
Last LDL-C value ≥ 3.6 mmol/L
(140mg/dL)‡
2%
1%
1%
Last LDL-C value < 3.6 mmol/L
(140 mg/dL) or on a statin (%)‡
2%
2%
5%
Process Measures†
Hemoglobin A1c (HbA1c)
obtained
Low Density Lipoprotein
Cholesterol (LDL-C) obtained‡
Lipid profile obtained
Level of Care
Facility
Team
PCP
-
1%
8%
–
2%
8%
7%
2%
9%
3
Physician effect size
Physician effect size
Negligible
PCP Effect
200
Last LDL-C
Value (1%)
Panel size
150
100
Small
Moderate
Cost of home
glucose
monitoring for
patients not on
Insulin
Last LDL-C
value <3.6
mmol/L or
on a statin (5%)
Hemoglobin A1c
obtained (8%)
50
Median PCP Panel
size in study sample
0
.02
.04
.08
.10
Variance attributable to level of care
Implicit chart review – site level
„
„
How many reviews are needed?
1
(to detect differences between sites)
Re liability of site level quality score
.2
.4
.6
.8
„
Trained physician
reviewers
621 records
26 clinical sites
Site level
reliability
„
HTN 0.17
DM 0.10
COPD 0.11
„
Measurement models are fundamentally
important to measuring and profiling
quality.
There is often little reason or capability
to profile at the physician level.
Profiles that ignore measurement error
„
0
„
Conclusions
0
10
20
30
40
50
cases
„
Misrepresent the variability in quality
Are difficult (or impossible) to validate
Example – the imprecise thermometer
„
Budget cuts inspire innovation in the
clinic
4
Observed vs. true temperature
85
85
90
90
Body temperature(F)
95
100
Body temperature(F)
95
100
105
105
Observed temperature
true
observed
Scale transformation
0
85
.2
90
Body temperature(F)
95
100
Probability of prevention
.4
.6
.8
105
1
Strength in numbers
observed
true
observed
average
1
Reliability
2
3
4
Rating of preventability
5
6
Effect of gaming
Outlier Physicians (1991)
“A person with two watches is never quite
sure”
0.4
Deviations of MD profiles from mean Hgb A1c levels
“A person with one watch knows what
time it is”
0.2
0.0
These physicians eliminate from their 1992
panels the patients who in 1991 had HgbA1c levels
-0.2
in the top 5%.
-0.4
Non-outlier Physicians (1991)
0.4
0.2
These physicians have the same patient panels in 1992
as in 1991.
0.0
-0.2
-0.4
1991
1992
5
6