Measuring Patients’ Experiences With
Individual Physicians:
Are We Ready for Primetime?
___________________________________________________________________________
Dana Gelb Safran, ScD
The Health Institute
Institute for Clinical Research and Health Policy Studies
Tufts-New England Medical Center
Presented at:
Academy Health Annual Research Meeting
San Diego, CA
7 June 2004
Commonwealth Fund and Robert Wood Johnson Foundation
Focusing on Physicians

Survey-based measurement of patients’ experiences with
individual physicians is not new.

What’s new: Efforts to standardize and potential for public
reporting.

IOM report Crossing the Quality Chasm gave “patient-centered
care” a front row seat.

Methods and metrics have been honed through 15 years of
research and through several recent large-scale demonstration
projects

But putting these measures to use raises many questions about
feasibility and value.
Ambulatory Care Experiences Survey Project

Statewide demonstration project in Massachusetts

Collaboration:




6 Payers
6 Physician Network Organizations
Massachusetts Medical Society
Massachusetts Health Quality Partners

Testing the feasibility and value of measuring patients’
experiences with individual primary care physicians and
practices

Primary impetus: plans seeking to standardize surveys

IOM “Chasm” report further propelled the work
Principal Questions of the Statewide Pilot

What sample size is needed for highly reliable estimate of
patients’ experiences with a physician?

What is the risk of misclassification under varying
reporting frameworks?

Is there enough performance variability to justify
measurement?

How much of the measurement variance is accounted for
by physicians as opposed to other elements of the system
(practice site, network organization, plan)?
Sampling Framework
Eastern, MA
Central, MA
Western, MA
Tufts, BCBSMA, HPHC,
Medicaid
BCBSMA, Fallon,
Medicaid
BCBSMA, HNE,
Medicaid
PNO1
PNO2
PNO3
PNO4
PNO5
PNO6
34 Sites
23 Sites
10 Sites
143 Physicians
35 Physicians
37 Physicians
Both commercially insured & Medicaid patients sampled
Only commercially insured patients sampled
Measures from the
Ambulatory Care Experiences Survey (ACES)
Trust
Organizational
Access
Continuity
Interpersonal
Treatment
Primary
Care
Communication
·longitudinal
·visit-based
Comprehensiveness
·whole-person
Integration orientation
•team
•specialists
•lab
·health promotion/
patient
empowerment
Sample Size Requirements for Varying
Physician-Level Reliability Thresholds
Number of Responses per Physician Needed to Achieve Desired
MD-Level Measurement Reliability
Reliability:
Reliability:
Reliability:
0.7
0.8
0.95
ORGANIZATIONAL/STRUCTURAL FEATURES OF CARE
Organizational
access
23
39
185
Visit-based continuity
13
22
103
Integration
39
66
315
Communication
43
73
347
Whole-person
orientation
21
37
174
Health promotion
45
77
366
Interpersonal
treatment
41
71
337
Patient trust
36
61
290
DOCTOR-PATIENT INTERACTIONS
What is the Risk of Misclassification?
 Not
simply 1- MD
 Depends
on:
Measurement
Proximity
Number
reliability (MD)
of score to the cutpoint
of cutpoints in the reporting framework
Risk of Misclassification at Varying Distances from the Benchmark
and Varying in Measurement Reliability (MD )
MD Mean Score
Distance from
Benchmark
(Points)
1
2
3
4
5
6
7
8
9
10
Probability of Misclassification at Varying Thresholds of
MD-Level Reliability
MD=.70
MD=.80
MD=.90
38.0
27.1
18.0
11.1
6.3
3.3
1.6
0.7
0.3
0.1
34.5
21.2
11.5
5.5
2.3
0.8
0.3
<0.001
<0.001
<0.001
27.4
11.5
3.6
0.8
0.1
<0.001
<0.001
<0.001
<0.001
<0.001
Certainty and Uncertainty in Classification
Comparison with a Single Benchmark
Significantly below
6.3
Significantly above
αMD=0.7
4.9
αMD=0.8
3.26
αMD=0.9
0
65
100
50th p’tile
= area of uncertainty
Certainty and Uncertainty in Classification
Cutpoints at 10th & 90th Percentile
Bottom Tier
Middle Tier
Top Tier
6.3
6.3
4.9
4.9
3.26
3.26
αMD=0.7
αMD=0.8
αMD=0.9
0
10th
53
p’tile
76
p’tile
100
90th
= area of uncertainty
Substantially Below Average
Average
Substantially Above Average
MEASURE RELIABILITY (MD)
0
0.9
50
0.01
0
50
0.01
0
0.8
50
0.6
0
50
0.5
0
0.7
50
2.4
0
50
2.4
0
64.6
76.3
50th ptile
90th ptile
52.9
10th ptile
88.0
100
Substantially
Below Average
Below Average
Average
Substantially
Above Average
Above Average
MEASURE RELIABILITY (MD)
0
0.9
50
19.7
0.8
50
0.7
3.3
50
2.2
0
50
17.6
3.2
50
0
0
28.5
11.1 50
8.8
0.4
50
27.0
11.2
50
0.4
0
50
33.0
17.3
50
14.7
2.0
50
32.0
17.4
50
2.3
0
0.6
50
36.4
22.5
50
19.9
4.7
50
35.4
22.8
50
5.4
0.1
0.5
50
38.7
27.7
50
25.2
8.7
50
27.3
50
9.7
0.4
52.9
58.5
64.6
70.8
10th ptile
25th ptile
50th ptile
75th ptile
38.3
76.3
90th ptile
100
Variability Among Physicians (Communication)
___________________________________________________________________________
100
97.5
95
92.5
90
85
80
77.5
75
0
10
20
30
Number of Doctors
40
50
60
Variability Across Practice Sites (Communication)
100
95
90
85
80
75
70
65
60
Eastern Region
25th-75th percentile
range of group scores
Central Region
Group Mean score
Western Region
Variability Among Physicians within Sites
(Communication)
100
95
90
85
80
75
70
65
60
55
50
Site A-1
25th-75th percentile
range of site scores
Site A-2
Site Mean score
Site A-3
25th-75th percentile
range of MD scores
Site A-4
MD Mean score
Allocation of Explainable Variance:
Doctor-Patient Interactions
100
80
62
60
Doctor
74
77
70
84
Site
Network
40
20
0
Plan
38
25
22
29
16
Allocation of Explainable Variance:
Organizational/Structural Features of Care
100
80
39
36
23
Doctor
60
Site
40
45
56
77
Network
Plan
20
16
0
Organizational
Access
8
Visit-based
Continuity
Integration
Summary and Implications
 With
sample sizes of 45 patients per physician, most surveybased measures achieved physician-level reliability of .7-.85.
 With
a 3-level reporting framework, risk of misclassification
is low – except at the boundaries, where risk is high
irrespective of measurement reliability.
 Individual physicians
and practice sites accounted for the
majority of system-related variance on all measures.
 Within
sites, variability among physicians was substantial.
Summary and Implications (cont’d)
 Feasibility of
obtaining highly reliable measures of patients’
experiences with individual physicians and practices has been
demonstrated.
 The
merits and value of moving quality measurement beyond
health plans and network organizations is clear.
 By
adding these aspects of care to our nation’s portfolio of
quality measures, we may reverse declines in interpersonal
quality of care.
Certainty and Uncertainty in Classification
Multiple Cutpoints
Bottom Tier
6.3
2nd Tier
6.3
6.3
4.9
4.9
4.9
3.26
3.26
3.26
3rd Tier
Top Tier
αMD=0.7
αMD=0.8
αMD=0.9
0
53
10th p’tile
65
50th p’tile
76
100
90th p’tile
= area of uncertainty