Effects of Statistical Uncertainty on the
Classification of Physicians
Based on their Cost Profiles
John Adams
AcademyHealth Annual Research
Meeting
June 28th, 2009
Health Plans Pushing to
Reward Efficient Care
• Health care purchasers want to identify which
physicians deliver care in the least costly manner
(“efficiently”)
• Using episode based cost profiles
• Number of policy applications
– Tiered networks
– Pay-for-performance
– Public reporting
• All require method for determining who is a high or low
performer
AcademyHealth-2 06/09
Our Goal
• Compare common method being used by health
plans (cut-points) to another method (statistical
testing)
• Does it make a difference?
AcademyHealth-3 06/09
The Massachusetts Data
• 2004-2005 claims from 4 Massachusetts
health plans
• Combined database:
• ~80% state’s commercial health plan market
• 2.9 million enrollees, 44% of state residents
• Analyses based on continuously enrolled
adults between 18–65 years old
AcademyHealth-4 06/09
Creating Episodes of Care
• Each patient’s claims aggregated into
episodes of care
• Episode of care is all care provided over a
period of time for a specific condition
•
Pneumonia – first through last claim for
pneumonia-related care
•
Diabetes – all claims for 12 mo
• Used Symmetry’s ETG Grouper
AcademyHealth-5 06/09
Physician’s Cost Profile
• Each episode assigned to physician with greatest
fraction of costs
• Calculated for each episode Actual Costs &
Expected Costs
• Expected Costs = average costs among episodes
assigned to physicians of same specialty adjusted
for patient co-morbidities
Sum of Actual Costs
Cost Profile =
--------------------------------Sum of Expected Costs
AcademyHealth-6 06/09
Current Method - Empirical Cut Points
• Pick percentiles of the observed distribution and
put physicians into bins
– E.g. Bottom (lowest cost) 25% of MD “high
performing”
• Attractive because:
– It is easy
– It is “grading on the curve”
– You can directly set the size of your “high
performance” network
AcademyHealth-7 06/09
Noisy Cut Points Are A Problem
AcademyHealth-8 06/09
New Method – Statistical Testing
Testing Cost Profiles vs Mean
• First you need a standard error
var(Observed )
–
SE ( PROFILE ) = ∑
∑ Expected
– Plug in population quantities
– A “null hypothesis” style SE
• Then you test against the mean:
– t=
PROFILE − mean( PROFILES )
SE ( PROFILE )
AcademyHealth-9 06/09
0.2
0.4
0.6
0.8
1.0
Labeled as a low cost provider with
0.0
Probability labeled as a low cost provider using
Some comments about how tests work
-3
-2
-1
0
1
2
3
True Score
AcademyHealth-10 06/09
Comparing Two Methods
• Cut-points
– Top 25% = high cost
– Bottom 25% = low cost
• Statistical testing
– Significantly higher than average (p<0.05) =
high cost
– Significantly lower than average (p=<0.05) =
low cost
AcademyHealth-11 06/09
30% of MD Classified Differently Across
Two Methods
T-test (p=0.05)
Cut-Points
Low Cost
Average
Cost
High Cost
Low Cost
11%
14%
0
2%
47%
1%
0
13%
12%
Bottom 25%
Average
Cost
High Cost
Top 25%
AcademyHealth-12 06/09
Potential Concern Not Enough Outliers
• Some policy applications require a sufficient fraction of
physicians to be labeled as high performing
– Statistical testing – 12.9% are low cost
– Cut-points – 25% are low cost
• If you need to increase the number of MDs, use a
higher p-value
0.2
0.1
0.0
Probability
0.3
0.4
Null Distribution
-3
-2
-1
0
1
2
3
Null Distribution
AcademyHealth-13 06/09
Conclusions
• Empirical cut-points
– Pros
• Gives the big standard error (small sample size) providers a chance
to be flagged as good (mostly by mistake)
• Easy to explain
– Cons
• Can be very noisy
• Lots of misclassification for small sample size providers
• It may not help to be a low SE provider
• Statistical testing
– Pros
• Reduces the number of providers flagged as above/below average
by chance
• Conforms to typical medical evidence standards
– Cons
• May not fill a high performance network (without a relaxed statistical
standard)
• May be harder for some purchasers to implement
• Harder for most people to understand
AcademyHealth-14 06/09
Study Team
Beth McGlynn PI, Ateev Mehrotra, Bill Thomas, Julie Lai
Funders
Department of Labor
For More Information
John Adams
RAND
adams@rand.org