Methodological Issues in
Physician-Level Measurement
of Clinical Quality
Elizabeth A. McGlynn, Ph.D.
June 26, 2006
Information About Individual Physicians’
Performance is Increasingly Sought
• Health plans believe they can save money through
differential payments to physicians (pay for
performance)
• Employers believe they can save money through
increasing consumer cost-sharing (consumer
directed health plans)
• Medical groups believe they can negotiate higher
rates or market share by demonstrating better
performance (tiered networks, rate increases)
• Consumers are likely to demand information on
performance as the share they pay for health care
increases (public release)
McGlynnAcadHealth-2 06/26/06
What Is Being Measured?
McGlynnAcadHealth-3 06/26/06
Data Sources for Measuring Quality
• Available sources include:
–
–
–
–
–
Administrative (claims) data
Manual abstraction of medical records
Surveys of patients
Inspection of office practice
Extraction of data from electronic medical
records
– Board certification/Maintenance of certification
• Each of these sources has strengths and
weaknesses
• No single source is adequate to address all
questions
McGlynnAcadHealth-4 06/26/06
Most Existing Approaches to Measuring
Physician Performance Use Claims Data
• Data are readily available and impose less burden
on providers
• But they have some significant problems
– Generally available one payer at a time
– Information availability driven by the benefit
package and the ways coding systems are used
– Some confounding of physician practice
patterns with patient behavior
• Pressure to deliver answers driving widespread use
of these methods
McGlynnAcadHealth-5 06/26/06
Current Approaches to Quality Measurement
• “Leading indicators”
– One measure at a time
• Condition-specific aggregates/composites
– Multiple measures on the same population with
the same health problem
• Comprehensive cross-condition measures
– Patient as the unit of analysis
McGlynnAcadHealth-6 06/26/06
Examples of Where These Approaches
Are Currently Used
Approach
Use
Leading indicators
Pay for performance
Public reporting
Tiered networks
Disease composites
Recognition programs
Maintenance of certification
Comprehensive aggregates
Not in widespread use
McGlynnAcadHealth-7 06/26/06
What You Measure May Affect the
Conclusions You Draw
Internal
Medicine
Endocrinology
Family Practice
Cardiology
0
20
40
60
80
100
% of recommended care delivered
HbA1c
DM Overall
QA Tools Overall
McGlynnAcadHealth-8 06/26/06
Some Challenges in Measuring
Physician Performance
McGlynnAcadHealth-9 06/26/06
Physicians See Multiple [Different] Patients
PT1
PT3
PT2
MD1
PT3
PT4
MD2
PT5
So, representing the variety of practice matters:
Case Mix Adjustment
McGlynnAcadHealth-10 06/26/06
A Market Basket of Indicators May Be
Necessary to Reflect the Variety of Practice
Afib
CAD
80%
CHF
Diabetes
Headache
60%
Hypertension
Hyperlipidemia
40%
Pneumonia
Prenatal
Preventive
20%
UTI
Other
B
-G
O
ed
ic
In
te
rn
a
lM
YN
e
in
e
tic
P
ily
Fa
m
En
do
c
rin
ra
c
ol
o
ol
o
gy
gy
0%
C
ar
di
% of eligible events
100%
McGlynnAcadHealth-11 06/26/06
Patients See Multiple Providers
PT3
PT4
PT2
Hosp A
MD1
PT5
MD2
PT1
MD3
PT6
PT7
PT9
Hosp B
PT8
So, determining who is “responsible” matters
Attribution
McGlynnAcadHealth-12 06/26/06
Information Rarely Available to
Link Patients to Physicians a Priori
• As gatekeeper models decline, no clear assignment
of patients to a physician exists
• Algorithms are used to “assign” patients to
physicians
– Done most frequently in economic profiling
– Basis is majority of dollars or visits
• We are experimenting with other rules:
– First eligible provider seen in study period
– Provider “triggering” eligibility for indicator
• Critical to reality test assignments
McGlynnAcadHealth-13 06/26/06
Physicians Have Multiple Contracts
Humana
Wellpoint
PacifiCare
MD1
United
MD2
Medicare
Aetna
MD3
Anthem
Medicare
Medicare
So, putting the pieces together matters:
Aggregation
McGlynnAcadHealth-14 06/26/06
Few Physicians Can Be Evaluated Using
Single Indicators from One Payer
Internal Medicine
Endocrinology
Family Practice
Cardiology
0
10
20
30
40
50
% of MDs with >10 eligibilities
HbA1c
DM Overall
QA Tools Overall
McGlynnAcadHealth-15 06/26/06
Physicians Practice in Different Systems
So, understanding the organizational context matters:
Fair comparisons
McGlynnAcadHealth-16 06/26/06
Little Routine Information Available on
Physician Practice Setting
• Taking organizational context into account is
challenging because of data limitations
• Using location may be misleading
– Shared space vs. shared practice
• Rationale for constructing scores at group level:
– Increase sample size
– Demonstrate value of integrated medical groups
– Avoid scores at the physician level
• Relatively little known about within vs. between
group variation
McGlynnAcadHealth-17 06/26/06
Categorizing Physician-Level Results
• Many applications of physician-level scoring
require using results to categorize physicians
– In/out of network
– In/out of performance bonus
– Tiering
• We prefer statistical testing to straight cut-points
• Applied this to the three different approaches to
MD-level scoring
– Test performance compared to the mean
– Use 95% confidence interval around each
provider’s score
– Those with scores significantly below average
were assigned to the low performance category
McGlynnAcadHealth-18 06/26/06
Different Methods Will Result in Different
Category Assignments
QA Tools
Overall
Not rated
1 star
2 stars
3 stars
DM
Composite
HbA1c
0%
20%
40%
60%
80%
100%
% of internists in category
McGlynnAcadHealth-19 06/26/06
Different Results Under Different Systems
Likely To Produce Challenges from MDs
Agree
DM>QAT
DM<QAT
0%
20%
40%
60%
80%
100%
% of internists
McGlynnAcadHealth-20 06/26/06
Summary
• A number of methodological issues arise in
creating quality scores at the physician level
• We need to better understand the implications of
these methodological choices
• Because the data on which the scores are based
were not intended for this purpose, feedback loops
and data quality improvement are essential
• But, the world isn’t going to wait for us to get the
methods perfect…
McGlynnAcadHealth-21 06/26/06
This Train Is Headed Your Way!
McGlynnAcadHealth-22 06/26/06