Physician Performance
Measurement:
Lessons from the California
Better Quality Information
Project
Cheryl L. Damberg, PhD, Ted vonGlahn, and David Hopkins, PhD
(PBGH)
Bob Houchens, PhD (Thomson Reuters)
Academy Health
June 29, 2009
Study Objectives and Context
 Growing interest by public and private purchasers in
assessing the performance of individual physicians
 Greater variation in care at the physician level
 Stimulate quality improvements through performance
feedback, transparency, and incentives
 CMS funded six pilot projects (2007-2008) to test the
feasibility of physician performance measurement:
 Pool data across multiple payers to enhance the ability to
measure the performance of individual physicians
 Examine methods issues, including attribution and reliability
(to minimize misclassification error).
 This study describes the methods work conducted
through the California Better Quality Information
(BQI) project
2
© Pacific Business Group on Health, 2009
California BQI Study Design
 Aggregated claims data from three large California
PPOs and Medicare FFS
 Claims for ~64,000 physicians in California
 ~4.7 million enrollees/beneficiaries
 Constructed performance scores for 18 clinical
measures at individual physician level
 Measurement year 2005, 2006, and 2007 data
 Tested 3 approaches to attributing patient events to
physicians based on E&M visits
1. All providers of any specialty: single touch of the patient
2. All measure relevant specialists: single touch rule
3. Single provider: Any relevant specialist for a given
measure,—based on plurality of E&M visits
3
© Pacific Business Group on Health, 2009
List of Clinical Measures
 Breast cancer screening
 Colorectal cancer screening
 Diabetes




LDL testing
HbA1c testing
Nephropathy testing
Eye exam screening
 Heart Failure
 LVF testing
 Warfarin for patients w/ CHF & atrial
fibrillation
 Spirometry testing for COPD
patients to confirm DX
 Rheumatoid arthritis patients
prescribed a DMARD drug
 Cardiovascular
 LDL testing
 CAD patients receiving lipidlowering therapy
 Persistence of beta blocker
therapy – Post MI
 Annual monitoring for patients
on persistent medications: Ace
inhibitors, Digoxin, Diuretics,
Statins
 Percent of patients >=65
that had glaucoma exam
 Women with osteoporosis
age >=67 who have had a
fracture bone density/
medication
Specifications: NCQA (HEDIS) or the Medicare Care Management Performance
(MCMP) Demonstration
Study Design

Assessed impact of 3 different rules on:






Number of denominator patients assigned
Number of physicians scored
Number of patients assigned per physician
Ability to Score Doctors Reliably
Validated attribution results with 50 physicians
(2,500 patient events)
Computed reliability scores for each performance
measure
 Defined the “n” to reach 0.70 for each measure
 Examined the fraction of physicians who met or
exceeded a threshold reliability of 0.70
5
© Pacific Business Group on Health, 2009
Attribution Rule Testing
Percent of Total Denominator Patients Included
by Attribution Method
Commercial & Medicare Data Combined
Method 1
Percent of Denominator Patients
100%
Methods 2 & 3
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
BCS
COL
CMC
Measure
LDL
HBA
Number of Denominator Patient Events
Assigned (Single Touch Rule)
Cycle 3—Relevant Specialists Only
Total Denominator
Patients
Patients Attributed
N
N
%
Measure
BCS
772,949
513,603
66.45%
COL
2,043,414
1,290,114
63.14%
CMC (LDL screen)
154,519
122,744
79.44%
LDL (diabetes)
396,046
261,284
65.97%
HBA
396,065
255,092
64.41%
BBH
7,739
6,861
88.65%
CAD
176,854
146,604
82.90%
HF2 (LV EF test)
20,044
18,790
93.74%
HF8 (warfarin)
30,503
25,853
84.76%
NPH
395,943
256,445
64.77%
PBH
843
709
84.10%
MPM
488,557
238,937
48.91%
CDC_Eye
301,818
209,545
69.43%
Reasons Patients go Unassigned
A
Total
Denom
Patients
BCS
CMC
COL
HBA
LDL
BBH
CAD
HF2
HF8
772,949
154,519
2,043,414
396,065
396,046
7,739
176,854
20,044
30,503
B
Total
Unique
Unassigned
Patients
256,287
21,982
746,792
105,326
99,914
740
22,303
878
3,364
C
%
Unassigned
Patients
(C=B/A)
33%
14%
37%
27%
25%
10%
13%
4%
11%
D
Patients
with E&M
Visits NonRelevant
Specialty
34%
50%
37%
48%
46%
32%
59%
48%
53%
E
Patients
with E&M
Visit/No
identifiable
MD
27%
16%
15%
15%
15%
32%
12%
28%
13%
F
G
No
E&M
Visit
39%
35%
49%
37%
39%
36%
29%
24%
35%
Total
Percent
(D+E+F)
100%
100%
100%
100%
100%
100%
100%
100%
100%
Number of Physicians Attributed,
by Attribution Method
All Physicians
Method 1
(single touch)
Measure Relevant
Specialties
Method 2
(single
Method 3
touch)
(plurality)
Total
Physicians
Assigned
NonAll
1 or More
Relevant
Relevant
Relevant
Measure
Events
Specialties Specialties Specialties
Single
Physician
(Relevant
Specialty)
BCS
42,255
18,654
23,601
23,601
20,797
COL
47,464
23,805
23,659
23,659
21,805
CMC
30,967
14,726
16,241
16,241
12,104
LDL
39,544
18,757
20,787
20,787
17,718
HBA
39,544
19,225
20,319
20,319
17,400
Average Number of Patients Assigned per
Physician, Method 2
10th
Percentile
50th
Percentile
90th
Percentile
42.90
2
26
102
117.17
3
62
303
Diabetes: HbA1c Screening
35.77
2
18
92
Diabetes: LDL Screening
35.97
2
18
93
Diabetes: Nephropathy Screening
35.87
2
18
93
Cardiovascular: LDL Testing
22.48
1
10
58
Cardiovascular: Beta Blocker Post
Discharge After a Heart Attack
4.44
1
2
11
Cardiovascular: Beta Blocker at 6
Months After a Heart Attack
1.60
1
1
3
Coronary Artery Disease: LDL
Drug Therapy
21.81
1
10
56
Heart Failure: Left Ventricular
Ejection Fraction Testing
8.56
1
4
23
Measure Name
Breast Cancer Screening
Colorectal Cancer Screening
Mean #
Physician’s Views of Responsibility
Relevant Specialties Only
Not Majority
Majority E&M
E&M Visits
Visits
(Method 2
Only)
(Method 3)
NonRelevant
Specialties
(Method 1
Only)
Relevant
Specialties
(Methods
1 & 2)
BCS
0%
37.3%
22.9%
68.9%
COL
0%
50.0%
31.7%
70.1%
CMC
0%
57.9%
42.9%
76.5%
LDL
0%
64.3%
42.2%
86.4%
HBA
0%
65.8%
43.7%
87.6%
Total
0%
52.0%
32.9%
75.1%
Measure
Reasons Why Physicians Claim NonResponsibility for Patient’s Care
Not
Primary
Care
Measure
Giver
74.7%
BCS
79.4%
COL
Treated
patient
Through
Single
Consult
Only
3.6%
Patient
Changed
Status; No
Longer
Able to
See
patient
0.3%
Never
Treated
This
patient
20.1%
Other
1.3%
8.2%
7.8%
9.2%
13.4%
CMC
84.3%
7.8%
0.0%
7.8%
0.0%
LDL
69.2%
9.2%
1.5%
17.7%
2.3%
HBA
63.8%
13.4%
2.4%
18.1%
2.4%
Ri =
2
σ MD
σ i2
2
σ MD +
ni
Reliability Calculation
 Reliability was calculated using the Spearman-Brown prophecy
formula:
 In this formula:




s2MD is the between-physician variance
s2i is the binomial variance associated with the rate for physician i,
ni is the denominator (sample size) for physician i.
Ri is the proportion of the total variance that is attributable to the
variance among physicians.
 Used a shrinkage estimator to compensate for the instability of
estimated rates for small-denominator physicians
 Ri, increases with
 the variation in rates among physicians, and
 the physician’s sample size (denominator)
Concentration of Patients Varies across
Measures
MDs
Measure
Den.
Reliable?
Threshold
ACE
25
ART
21
No
10,291
62.2
83.3
75,060
23.7
Yes
6,245
37.8
93.6
242,000
76.3
No
8,723
98.4
65.6
22,262
77.5
142
1.6
79.4
6,447
22.5
No
9,286
44.9
63.9
105,753
10.1
Yes
11,387
55.1
67.9
941,640
89.9
No
14,710
93.7
51.3
226,969
65.4
983
6.3
50.6
120,276
34.6
No
7,866
62.9
54.5
110,490
24.6
Yes
4,641
37.1
60.4
339,293
75.4
Yes
BCS
CAD
28
75
Yes
CDC_EYE
35 *
Mean
%
% MDs Score MD*Pats MD*Pats
Key Lessons Learned
 While a large fraction of denominator events can be
assigned, we need to identify ways to improve
assignment rates and increase denominator density
 Increasing data coding of rendering physician
 Alternative approaches to E&M based rule, when no
E&M visit in the measurement year
 Identifying “proxy PCP” when no visit with measure
relevant specialist in measurement year
 Payers and providers will need to pay greater attention
to data completeness issues to improve the validity of
the underlying data
 Broadening effort to include HMO data in future will
increase patient density at physician level
Conclusions
 Non-relevant-specialists claimed 0% responsibility
(even if they were the only physician the patient saw
during the measurement period)
 Measure relevant-specialty physicians claimed
responsibility between:
 68-85% of the time when they constituted the plurality of
visits
 22-43% of the time when they did not represent the plurality
of visits.
18
 The lack of ownership expressed by many physicians
highlights challenges related to accountability for
quality, particularly when multiple physicians touch a
patient only once during a measurement period or the
only physician seen is not-relevant to the
performance measure.
© Pacific Business Group on Health, 2009
Conclusions
 The “n” required to achieve a specified level of
reliability varies physician to physician and
reliabilities are measure dependent
 Achieving reliable denominator sizes at the
physician-level remains a challenge
 Possible methods to increase reliability
include:
 Use of composite measures
 Shrinkage estimates
 Pooling data over multiple years
 Incorporating data from more health plans
 Summarizing results at a practice site level
19
© Pacific Business Group on Health, 2009
More Information is Available
www.cchri.org – for more information
about the California Physician
Performance Initiative (CPPI)
Sponsored by the California Cooperative
Healthcare Reporting Initiative (CCHRI)
20
© Pacific Business Group on Health, 2009