Methodological Issues in Measuring
Physician-Level Quality and Efficiency
Ateev Mehrotra MD MPH
RAND Health & University of Pittsburgh
AcademyHealth Annual Research Meeting
June 5th 2007
Applications of Physician Level
Profiles
• Public reporting – information to help people
•
•
make more value-based decisions
Pay-for-performance – financial rewards to
physicians with better performance
Tiering – differential co-payments tied to
physician performance
• $10 to see an “above average” efficiency &
quality physician
• $30 to see a “below average” efficiency &
quality physician
Overall Project Goals
• Identify key methodological choices that arise when
constructing physician quality and efficiency
profiles
• Evaluate whether decision on methodological
choice results in physicians being placed in
different categories
• Identify potential policy impact when applying these
metrics
Methodological Issues on Efficiency Profiles Being
Addressed
1. Constructing efficiency scores
2. Categorizing physicians into categories
3. Evaluating effects of reimbursement versus
utilization on efficiency scores
4. Examining the relationship between efficiency and
quality
5. Evaluating different units of analysis
6. Assessing alternate attribution rules
Overall Findings
• Number of choices necessary when creating
these profiles
• Approaches used are rarely transparent to
users or those being evaluated
• Empirical evidence that choice matters
Data Source
• 2003 & 2004 claims from 4 major health
plans in Massachusetts
• 2.9 million commercial enrollees
• Adults <65 who were continuously enrolled
for two years
• Aggregated database ~90% state’s
commercial health plan market
Quality Measurement Approach
RAND QA Tools
• Subset of the measures used in RAND’s national
study of health care quality
– Claims-based algorithms
– 129 measures of technical process quality
across 23 conditions
Efficiency Measurement Approach
Symmetry’s ETG
• Commonly used program among health plans
• Patient’s claims aggregated into episodes of care
• Episode of care is all care provided over a period of
time for a specific condition
•
e.g. Pneumonia – first through last claim for
pneumonia-related care
•
e.g. Diabetes - all diabetes care received in year
Physician’s Efficiency Profile
• Each episode assigned to the physician using
an attribution rule
• Calculated for each episode:
costs of given episode (observed)
----------------------------------------average costs for that type of episode
across all patients (expected)
• Overall score for a physician is the ratio of
observed to expected costs across all
assigned episodes
Analytic Sample
Characteristic
Massachusetts
Adults 18-65yo
1,863,883
Episodes of care
8,673,513
Quality events
2,968,867
Physicians with at least one quality
event or episode
21,946
Focus on Attribution
What is Attribution?
• How do we decide which MD is responsible
for care?
• Except when there is a contractual
relationship (gatekeeper), most approaches
are algorithmic
• Explore different algorithms and empirical
impact
Choice #1
Level of Analysis?
Patient-based
Episode-based
MD is responsible
for managing overall
care for patient
MD is responsible
for managing a
condition or
problem
Choice #2
What is Signal for Responsibility?
E&M Visits
Costs
Professional
services & Rx
Plurality
(>30%)
Majority
(>50%)
Evaluation
Plurality
(>30%)
Majority
(>50%)
Triggering Event
Visit that started the
episode (vs. who
played the most)
Choice #3
One or Multiple Physicians?
Single MD
Multiple MD
One MD is
responsible for
managing patient or
condition
Team approach to
managing a patient
or condition
Focus on One of these Choices:
Level of Analysis
Patient-based
Episode-based
Number of Doctors Seen in 2003-2004
for E&M Visits
EPISODE-BASED
PATIENT-BASED
4 MD
1%
3 MD
4%
1 MD
19%
5 or more MD
32%
5 or more MD
1%
2 MD
15%
2 MD
19%
4 MD
13%
3 MD
17%
1 MD
79%
How Often Do Different Attribution Rules Assign the
Same Episode to the Same Physician?
Patient-Based
Plurality Visits
Patient-Based
Episode-Based
Plurality
Visits
Plurality
Visits
Plurality
Costs
Episode-based
Plurality
Visits
Plurality
Costs
40%
39%
96%
Half of Massachusetts MDs Classified
Differently Under 2 Rules
Episode-Based
Patient-Based
Unknown
Efficiency
Less
Efficient
Average
More
Efficient
Unknown
Efficiency
0%
0%
0%
0%
Less
Efficient
8%
11%
4%
2%
Average
5%
8%
29%
8%
More
Efficient
8%
1%
6%
10%
Sample Size
Average # of Total Episodes Assigned to Different
Specialties
45
40
Patient Based
35
30
Episode Based
25
20
15
10
5
0
Less Common Surgical Specialties
urology, neurosurgery, plastic, vascular,
thoracic
What Percent of Episodes Relevant to Specialty
25
Patient
Based
15
Episode
Based
%
20
10
5
0
Among Endocrinologists, % All
Episodes for Diabetes
Policy Implications
• No “right” approach for attribution as it
depends on policy goal and desired behavior
change
• For tiering, patient-based might be best
– Patient usually chooses a primary provider
– Primary provider has a set referral network
• For P4P, episode-based might be best
– Locus of control
– Shared responsibility
Study Team
•
•
•
•
•
•
•
•
Elizabeth A. McGlynn, Ph.D.
Ateev Mehrotra, M.D.
Bill Thomas, Ph.D.
John Adams, Ph.D.
Scott Ashwood
Rodger Madison
Julie Lai
Fuan-Yue Kung
For More Information
mehrotra@rand.org
For more information
How Often Does only Single MD Care for a Given
Episode?
100
90
80
%
70
60
50
Preventive
Episodes
Acute
Episodes
Chronic
Disease
Episodes
Most Plans Contract with Physicians
Statewide
But Utilization Is Concentrated in
One Area of the State
So Aggregating These Plans May Not
Increase Sample Sizes
Or Any Two of These Plans
But Aggregating Data with this Purchaser
Increases Number of Observations
Average # of Episodes Assigned to Different
Specialties
100
Cost Basis
80
E&M Basis
60
40
20
0
PCP
Common
Specialties
Ob/Gyn,
Cardiology,
Neurology
Less Common
Surgical
Specialties
Urology,
Neurosurgery,
plastic, vascular,
thoracic
Relationship Between
Quality & Cost-Efficiency
High
100%
50%
25%
Low
Efficient
Inefficient
100
10
1
Cost-efficiency Score
0%
0.1
Quality Score
75%
For Many Physicians, We Do Not Have Enough
Information to Create Robust Profiles
3500
3000
2500
Number of
Providers
2000
1500
1000
1-9
500
10-29
30-59
0
1-9
Episodes
10-29
60+
30-59
60+
Quality Events
Scores Are Based on a Minority
of Patients and Claims
Enrollees
Claims
100%
100%
Limited to Adults
18-65
67%
88%
Limited further to those
continuously enrolled
39%
46%
Limited further to those
with at least one claim
32%
46%
All Enrollees