Quality Improvement Research
Methods: Issues in Detecting
Changes in Clinician
Performance
Haya R. Rubin, M.D., Ph.D.
Lynne Nemeth, R.N., Ph.D.
Hoangmai Pham, M.D., M.P.H.
Background
• Given multi-year projects to measure physician
performance, it is now possible to examine and
compare provider changes in performance on
quality indicators, providers want to track if
change is occurring
• Interventions are being applied that require
comparing changes in different groups, i.e.,
physician performance incentives
• This has made evident methodological issues in
determining whether, which, and how big a
change has occurred, and whether one change
exceeds another.
Overview of Issues
• Risk Adjustment
– Patient Characteristics*
– Clinician/Practice Characteristics*
– Insurance Type
• Indicator Specifications
– Denominators: relationship of indicator lag time to
frequency of indicator measurement
– Specific indicators: choice and volume
– Composites: Scope, weighting
– Provider exclusion from denominators
Risk Adjustment: What needs to be
adjusted for in design or analysis?
– Patient characteristics: practices differ from each
other, and over time, and interventions groups may
differ from each other in the patient populations they
serve. Should patient characteristics (i.e., age,
gender, race, ethnicity, diagnoses, chronic diseases,
comorbidity, health status, attitudes, behavior) be
adjusted for in comparing changes over time?
– Practice/provider characteristics: Practices differ in
size, scope and specialty, state of facilitators/barriers
to achieving quality care, and how well they are doing
on indicators at baseline. Do we need to adjust for
any of these practice characteristics when
determining if changes in quality have occurred and in
comparing changes to each other?
Risk Adjustment:
Insurance/Payment Type
• Payment incentives may be limited to
specific payers; can measures for specific
payers be generalized to others?
• Does it matter if measures are applied in
capitated payment systems vs. fee-forservice systems?
• What is the state of other payer
incentives/structures to facilitate quality
besides the intervention being evaluated?
Indicator Specification:
Does the nature of the measure
affect the determination of change?
• Indicator specification issues
– Specific indicators: Choice, volume issues
– Composite indicators: Scope and weighting
– Provider exclusion from denominators
– Lag time for indicator evaluation vs. frequency
of indicator measurement
Specific Indicators
• Choice: Does a specific indicator apply to
a practice’s work/role in improving
patients’ health? e.g., Should a BP control
indicator be used to compare urologists’
care for hypertensive patients?
• Volume: Does the provider see enough
patients with the condition to which the
indicator applies to allow an evaluation of
change in performance?
Composite Indicators
• Which indicators are included in a composite
indicator?
– Choice, scope: How well do they apply to each of the
practices being compared? How much of the valuable
work of each practice in improving health and survival
is included in the composite indicator?
– Weighting of individual component indicators: Does it
approximate that aspect’s impact on patient qualityadjusted survival? How do we determine how to
weight each one?
Provider Exclusion
• In some systems, providers can exclude
patients from denominators where there
are clinical exclusions specified
• Where possible, do some fail to use this
capability and look like they perform worse
than they do relative to other practices?
• Do some use this inappropriately and look
like they perform better than they do
relative to colleagues?
Lag Time vs. Frequency
• If indicator specifies a period for
measurement such as “last 6 months” and
is measured quarterly, how do we deal
with the overlap?
• What about replication of patients in each
period?
Panelists
• Lynne Nemeth, Ph.D.
– Medical University of South Carolina
– Will present on impact of baseline performance on
changes observed in A-TRIP study on 99 practices
• Hoangmai Pham, M.D., M.P.H.
– Center for Studying Health System Change
– Will present on impact of patient demographic
characteristics on changes in clinician preventive care
performance observed in Medicare