Measurement theory and provider
profiling
Timothy P. Hofer MD
Dept. Medicine,
University of Michigan
VA Ann Arbor HSR&D
Center of Excellence
The measurement problem
construct
indicator
Quality
ei(jk)
Levels of care
Site
(clinic, hospital)
Provider
(physician)
Patient
i
i i
Indicators
Implications of the measurement model

The indicator is a fallible measure of the
construct




Some indicators are less precise than others
Quality indicators are very imprecise for a variety
of reasons
You need to account for the measurement error
The location of the construct variability can
suggest different causes, interventions and
measurement procedures
Intra-class correlation(=reliability)


Ability to distinguish between physicians
(or sites)
single observation under a specified set
of conditions of measurement.



physician


Rx 
2
 2  2


  site  physician  patient 
2
Vol. 281 No. 22, pp. 2065-2160, June 9, 1999
MD laboratory utilization profiles
Deviations of MD profiles from mean laboratory utilization ($)
Unadjusted for reliability
60
20
-20
-60
Adjusted for reliability
Table 3: Amount of variation in hospitalization and outpatient visit rates attributable
to a physician practice style effect
Age-Sex Adjusted
Casemix Adjusted
13%
10%
7%
4%
0.51
0.40
Unadjusted for reliability (R2) *
8%
8%
Reliability adjusted (ICC) †
2%
1%
0.24
0.17
Variable
Visits
% of variation associated w/ physician
Unadjusted for reliability(R2) *
Reliability adjusted (ICC) †
Reliability‡
Hospitalizations
% of variation associated w/ physician
Reliability‡
VA Network 11 Diabetes Care Project
Resources available

VA Diabetes Registry Project (1998-2001)

Automated Clinical Databases


Data warehouse (VA Healthcare and analysis group)
Database Components







Encounter records (OPC/PTF )
Outpatient Pharmacy
Lab
primary care provider database (PCMM ()
Vitals
Cohort identification procedure
Data quality and measure validation

Kerr EA , et al. Journal on Quality Improvement 2002;
28(10):555-65.
Selected Measures:
Resource Use



Cost of hypoglycemic medications
Cost of home glucose monitoring for
patients not on insulin
Cost of calcium channel blockers
Quality
Outcomes
Intermediate
Outcomes
Processes
Selected Measures :
Intermediate Outcomes




Last A1c value
A1c  9.5%
Last LDL value
LDL  3.6 mmol/L (140mg/dl)
Quality
Outcomes
Intermediate
Outcomes
Processes
Selected Measures:
Process Measures



Hemoglobin A1c obtained
LDL-C obtained
Lipid profile obtained
Quality
Outcomes
Intermediate
Outcomes
Processes
Selected Measures:
Mixed or Linked Measure

LDL  3.6 mmol/L (140mg/dl)
or
on a statin
Are there differences between
physicians?

What are the sources of variation?





Noise
Unmeasured differences
Physician effects
Noise
Clinic or group effects
Physician
Health System/payor effects
Clinic
System/Payor
Outcomes
Diabetes Care Indicator
Level of Care
Facility
Team
PCP
1%
0
2%
Cost of home glucose monitoring
for patients not on insulin
18%
3%
3%
Cost of home glucose monitoring
for patients on insulin
8%
2%
1%
Cost of Calcium Channel
Blockers
1%
0
0
Resource Use
Cost of hypoglycemic
medications
Intermediate outcomes
Diabetes Care Indicator
Level of Care
Facility
Team
PCP
12%
0
1%
Last HbA1c value  9.5%
16%
0
0
Last LDL-C value‡
7%
–
1%
Last LDL-C value  3.6 mmol/L
(140mg/dL)‡
2%
1%
1%
Last LDL-C value < 3.6 mmol/L
(140 mg/dL) or on a statin (%)‡
2%
2%
5%
Intermediate Outcomes
Last HbA1c value
Process measures
Diabetes Care Indicator
Process Measures†
Hemoglobin A1c (HbA1c)
obtained
Low Density Lipoprotein
Cholesterol (LDL-C) obtained‡
Lipid profile obtained
Level of Care
Facility
Team
PCP
-
1%
8%
–
2%
8%
7%
2%
9%
Physician effect size
Physician effect size
Negligible
PCP Effect
200
Last LDL-C
Value (1%)
Panel size
150
100
Small
Moderate
Cost of home
glucose
monitoring for
patients not on
Insulin
Last LDL-C
value <3.6
mmol/L or
on a statin (5%)
Hemoglobin A1c
obtained (8%)
50
Median PCP Panel
size in study sample
0
.02
.04
.08
Variance attributable to level of care
.10
Implicit chart review – site level
.8
1
(to detect differences between sites)
.6
Site level
reliability
.4
HTN 0.17
DM 0.10
COPD 0.11
.2

How many reviews are needed?
0

Trained physician
reviewers
621 records
26 clinical sites
Reliability of site level quality score

0
10
20
30
cases
40
50
Conclusions



Measurement models are fundamentally
important to measuring and profiling
quality.
There is often little reason or capability
to profile at the physician level.
Profiles that ignore measurement error


Misrepresent the variability in quality
Are difficult (or impossible) to validate
Example – the imprecise thermometer

Budget cuts inspire innovation in the
clinic
85
90
95
100
105
Observed temperature
observed
85
90
95
100
105
Observed vs. true temperature
true
observed
85
Body temperature(F)
90
95
100
105
Strength in numbers
true
observed
average
0
.2
.4
.6
.8
1
Scale transformation
1
2
3
4
Rating of preventability
5
6
Reliability
“A person with one watch knows what
time it is”
“A person with two watches is never quite
sure”
Effect of gaming
Outlier Physicians (1991)
Deviations of MD profiles from mean Hgb A1c levels
0.4
0.2
0.0
These physicians eliminate from their 1992
panels the patients who in 1991 had HgbA1c levels
-0.2
in the top 5%.
-0.4
Non-outlier Physicians (1991)
0.4
0.2
These physicians have the same patient panels in 1992
as in 1991.
0.0
-0.2
-0.4
1991
1992