Practice-Based Evidence Research Model
Predictors of Clinician Performance
Risk Adjustment
Comparative Effectiveness Research
AAOMPT Conference
Anaheim, CA 2011
Dennis L. Hart, PT, PhD
Director, Consulting and Research
Focus On Therapeutic Outcomes, Inc.
Knoxville, TN
Faculty Affiliation & Disclosure
• Affiliation
– Director of Consulting and Research for FOTO.
– Develops outcomes and other measures used in FOTO’s
software to assist clinician’s ability to manage patients
designed to improve their outcomes.
– Publishes or assists in the publication of scientific
endeavors related to analyses of the FOTO database.
• Disclosure
– Employee and investor in FOTO, a proprietary international
medical/rehabilitation database management company.
2
The First Presentation
• Describe predictors of physical therapy clinician
performance identified using Practice-BasedEvidence (PBE) research designs.
• PBE methods/designs allow development of
comparative effectiveness research (CER) models.
• CER uses advanced risk-adjustment to control for
confounders and diverse patient case-mix for
meaningful interpretation of results to identify which
treatments are associated with better outcomes for
which patients.
3
Background
• FOTO is combining data from the U.S. and Israel
looking for best practices.
• Both countries are interested in public reporting,
value-based purchasing and finding best practices.
• Research teams are analyzing data from evidencebased practice from MDT and EIM trained clinicians.
• As these data become risk-adjusted, they become
the foundation for value-based purchasing and
public reporting and become the foundation for
comparative effectiveness research.
4
Predictors of Clinician Performance
• To predict clinician performance, one needs data
describing patients, clinicians and outcomes (etc!).
• We collect and analyze real life clinical practice data.
• Use of real life clinical data and their analyses are
“Practice-Based Evidence for Clinical Practice
Improvement Study Design” (PBE): Susan Horn, PhD.
• PBE analyzes content and timing of individual steps
of a health care process to determine how to
achieve superior outcomes for the least necessary
cost over the continuum of a patient’s care.
5
Predictors of Clinician Performance
•
•
•
•
•
•
•
•
•
Let’s look at some data from MDT trained clinicians.
N=743 patients with lumbar impairments.
Age 52 (17), 18 to 93: 57% female: 17% had surgery
52% chronic symptoms
40% had 4 or more comorbid conditions
53% PPO or HMO (11% military, 3% WC)
45% elevated somatization: 47% elevated depression
27% elevated FABQ-PA
41% CEN
6
Clinician Performance: Raw Data
Discharge FS By Clinician
90
85
Discharge FS
80
75
70
65
60
55
50
PT1
PT2
PT3
PT4
Therapists
PT5
PT8
PT 11
7
N=743, mean 106/therapist, range 31 - 288
Clinician Performance: Risk-Adjusted
Predictions for Discharge FS
RA Predicted Discharge FS
RA Predicted Discharge FS by Clinician
90
85
80
75
70
65
60
55
50
PT1
PT2
PT3
PT4
PT5
PT8
PT11
Therapists
N=448, R2 = 0.50, P<0.001
8
Raw & RA Predicted Discharge FS
Clinician Performance: Raw & RiskAdjusted Predicted Discharge FS
90
85
80
75
Raw FS
70
RA Pred FS
65
60
55
50
PT1
PT2
PT3
PT4
PT5
PT8
PT11
Therapists
N=448, R2 = 0.50, P<0.001
9
Clinician Performance: Risk-Adjusted
Residuals for Discharge FS
RA Residuals Discharge FS
RA Residuals Discharge FS by Clinician
3
2
1
0
PT1
PT2
PT3
PT4
PT5
PT8
PT11
-1
-2
-3
Therapists
N=448, R2 = 0.50, P<0.001
10
Clinician Performance: Ratings by Raw vs.
Risk-Adjusted Residuals for Discharge FS
Therapist
PT1
Rating Using Raw Rating Using RA
FS
Residual
5
6
PT2
3
2
PT3
2
7
PT4
6
1
PT5
7
4
PT8
4
3
PT11
1
5
11
Risk Adjustment
• Risk adjustment, also called ‘case-mix’ adjustment, is
a statistical method used to control effects of
confounding variables seen in patient populations.
• Risk adjustment considers factors other than the
health care intervention or processes of care that
help explain variation in patient outcomes.
• Without clinically appropriate and statistically sound
risk adjustment, you cannot compare outcomes
amongst clinicians in a meaningful manner.
12
Comparative Effectiveness Research
• CER: which treatments are associated with better
outcomes for which patients.
• Deutscher, Horn, Dickstein, Hart et al, 2009 is the
only paper I am aware of that describes a CER data
analysis plan in outpatient rehab.
• Observational study where all data are included.
• Bias is present to some extent in all observational
studies (systematic error from factors that influence
the decision to treat and the outcomes of interest,
like patient selection, confounding, misclassification,
timing of measurements, loss of follow-up data).
Dreyer NA et al Am J Man Care 2010
Dreyer NA et al Health Affairs 2010
13
Controlling for Bias
• Large, complete, diverse data sets.
• Statistical risk-adjustment to reduce effects of bias.
• Many variables (patient demographic, patient
classification, clinic/clinician details, process,
outcomes, timing of measurements) quantified.
• Exacting operational definitions for each variable.
• Variability in treatment.
• Groups being compared should have a reasonable
amount of overlap.
Dreyer NA et al Health Affairs 2010
14
Deutscher et al Results: Quick Review
• Data set large: 54 outpatient facilities in Israel, of
57,008 patients, 22,019 (39%) had both intake and
discharge data, diverse patients.
• Many variables: patient characteristics,
interventions, and outcomes.
• Variability in treatments.
• Operational definitions: working on it!
• Groups had reasonable overlap in needs for
treatment.
• Multivariable linear regression models.
Deutscher D et al Arch Phys Med Rehabil 2009
15
Deutscher et al Results: Quick Review
• Exploratory power of models: 30% to 39%.
• Patient variables explained most variation in
outcomes.
• Better outcomes were associated with: patient
compliance with self-exercise and therapy
attendance, use of therapeutic exercise and manual
therapy, completion of 3 or more outcomes surveys.
• Worse outcomes were associated with: women,
electrotherapy for pain, ultrasound for shoulder
impairments, use of depression medication, longer
waiting times before starting therapy.
Deutscher D et al Arch Phys Med Rehabil 2009
16
The MDT and PBE Studies
• To improve patient outcomes:
–
–
–
–
–
–
–
–
Collect all data – No missing data.
Data must be risk adjusted.
Add variables to the risk-adjustment models.
Define everything you do, like treatments
provided and methods of clinical examination.
Classify patients in meaningful ways.
Little difference between patients U.S. vs. Israel.
Follow the data, not the guru.
Integrate results into clinical practice.
17
EBP and Risk-Adjusted Outcomes
• Evidence-Based Practice provides structure to
patient examination and management.
• EBP is the use of the current best science to help
understand the patient for improved patient
management and outcomes.
• EBP has led to improved patient examination, which
has led to better ways of classifying/grouping
patients, and better ways of treating patients, for
improved statistical modeling of outcomes.
• Adding patient classification brings predictive power
of FS outcomes models to 50%.
Werneke MW et al JOSPT 2011
18
Extraneous Variables
• Observational studies have bias to some extent.
• The trick is to describe and quantify the bias, so you
can control the effect of the bias, which improves
interpretation of results.
• Outcome Confounder: Using MDT or TBC approaches
direct treatment to specific subgroups of patients,
which confounds outcomes.
• Treatment Mediator: Clinical finding like elevated
fear-avoidance that helps direct treatment but might
not be a good predictor of FS outcomes.
George SZ et al Spine 2003
Edmond SL et al JOSPT 2011
19
Extraneous Variables
• Observational studies have to some degree of bias.
• The trick is to describe and quantify the bias, so you
can control the effect of the bias, which improves
interpretation of results.
• Outcome Confounder: Using MDT or TBC approaches
direct treatment to specific subgroups of patients,
which confounds outcomes.
• Treatment Mediator: Clinical finding like elevated
fear-avoidance that helps direct treatment but might
not be a good predictor of FS outcomes.
George SZ et al Spine 2003
Edmond SL et al JOSPT 2011
20
Extraneous Variables
• Effect of treatment on outcomes.
• Be prepared to recognize that treatment might not
affect FS outcomes as much as we would like!
• As predictive models evolve, current risk-adjustment
variables control a large percent of FS outcomes data
variance, so new variables add little to predictions.
• Comorbidities: <2% additional variance control.
• Fear-avoidance: <2% additional variance control.
• Classification: ~50% additional variance control.
Resnik L et al JCE 2010
Werneke MW et al JOSPT 2010
Hart DL et al JOSPT 2011
21
Effect of Adding Variables:
Comorbid Conditions & Fear
0.14% gained
0.2531
0.254
0.2517
0.64% gained
R2 Values
0.252
0.25
0.248
With
FCI
With
FCI
&
Fear
0.2453
0.246
0.244
0.242
0.24
No FCI or Fear
No Fear
Full
Orthopedic/Lumbar: Functional Status Change
Resnik L et al 2010; Hart DL et al 2011
22
Effect of Adding Patient Classification:
Pain Pattern Classification
0.5
Doubled!
0.5
0.45
R2 Values
0.4
0.2531
0.35
0.3
0.25
0.2
0.15
MDT Project
0.1
0.05
0
Full
PPC
Orthopedic/Lumbar: Functional Status Change
Werneke MW et al 2010
23
Conclusions
• Have described predictors of physical therapy clinician
performance using Practice-Based-Evidence (PBE)
research designs.
• Introduced CER and risk-adjustment to control for
confounders for meaningful interpretation of results.
• Demonstrated how we have identified which
treatments are associated with better outcomes for
which patients.
• Be prepared to change your understanding of many
issues related to patients, treatments, outcomes and
payment.
24
Thank You
hart@fotoinc.com
25