Risk Adjusting Residential
Treatment Outcomes with
Clinical Factors
Neil Jordan, PhD
Interim Director, Mental Health Services & Policy
Program
Department of Psychiatry & Behavioral Sciences
Northwestern University Feinberg School of Medicine
Introduction
► Increasing
emphasis on accountability,
qualityy of care of mental health services
q
► Government funders are using outcomes
data to assess provider/program
performance & influence payment rates
► Not all providers are alike, nor do their
caseloads remain the same over time
 Differences in outcomes across providers
reflect more than just differences in quality
Risk Adjustment (RA)
► Method
M th d
th t accounts
that
t ffor group diff
differences when
h
comparing non
non--equivalent groups on outcomes
► Takes
T k into
i t accountt ffactors
t
nott iin th
the provider’s
id ’
direct control that are related to outcomes
► Risk
Ri k variables
i bl are those
th
that
th t influence
i fl
outcomes
t
but are not part of treatment
► Good
G d risk
i k adjustment
dj t
t models
d l allow
ll
for
f more
equitable and valid comparisons of performance
than using unadjusted values
Risk Adjustment in Mental Health
► Still
emerging
Adult psychiatric hospitalization (Banks 1999)
Short--term rehospitalization (Hendryx 2001)
Short
Seclusion & restraint incidents (Schacht 2003)
Consumer satisfaction (Greenberg 2004)
Youth community mental health services (Ogles
2008)
 Youth residential treatment (McMillen 2008)





Why is RA Uncommon in MH?
► Lack
of necessary data
 Outcomes across multiple providers/programs
 Preferably not generated and reported by
providers
 Predictor variables measured before the
intervention (to serve as risk adjustors)
 Existing administrative data (because primary
d t collection
data
ll ti for
f RA would
ld be
b very expensive)
expensive
i )
 Lack of comprehensive clinical data
Study Objective
► McMillen’s
M Mill ’
2008 study
d off RA in
i youth
h
residential treatment recommended
including
l d
measures off clinical
l
l functioning,
f
MH diagnoses, and/or MH need as risk
adjustors
► Purpose: to determine whether the addition
of these types of variables improves a RA
model for youth residential treatment (RT)
outcomes
Study Population
► Youth
in state custody receiving RT services
for behavioral health p
problems in Illinois
► 45 residential treatment and group home
providers participating in a pay for
performance initiative, excluding
 Those that serve youth with intellectual
disorders
 Shelter placements
Data Source
Source, Study Sample
► Administrative
data from Illinois Department
of Children & Familyy Services
► RT placement during 2007 or 2008 and
clinical data prior to RT spell (n=1486)
(n=1486)
► Sample characteristics:
 61% male
 Mean age = 14.5
14 5
 55% African
African--American
Outcomes
► Treatment
stability (TS): % of days at the facility
and not on run, in a psychiatric hospital, or
incarcerated
► 180
180--dayy p
post-discharge
postg placement
p
stabilityy (PDPS)
(
)
(dichotomous)
 “Favorable” discharge
g to less restrictive setting…
g
► Foster
care, independent living, transitional living
► Less restrictive residential or group home setting
► Placement in chronic mental illness setting
 …and sustained subsequent placement for 180 days
Primary Risk Adjustors – Prior Child
Systems Involvement
► Measures




Aggressive symptoms and antipsychotic use
Juvenile detention or corrections
Runaway
Psychiatric hospitalization
► Dichotomous
Primary Risk Adjustors - CANS
►
►
►
►
►
Child and
d Adolescent
Ad l
t Needs
N d &
Strengths (Lyons, 1999)
Clinical outcomes management
tooll used
d for:
f
 Assessing progress within
placement
 Determining
appropriateness for
placement change
p
g
Completed by trained rater
Continuous; item score ranges
from 0 (no need for action) to
3 (need for immediate action)
Used most recent CANS prior
to beginning of RT placement
►
59 items
it
across 6 needs
d
domains:
 Behavioral/emotional needs
(d
(depression)
i )
 Risk behaviors (danger to
others)
 Traumatic stress symptoms
(numbing)
 Trauma experiences
(emotional abuse)
 Life domain functioning
(social)
 Acculturation (cultural
stress))
stress
Analytic Methods_1
Methods 1
► Exploratory
factor
analysis of CANS items
► Factor analysis
 Principal
p factors
extraction
 Varimax rotation
► Yielded
77-factor
solution:
Externalizing
Trauma
Sexuality
School problems
Juvenile justice
interactions
 Acculturation
 Internalizing





Analytic Methods_2
Methods 2
► Regression
R
i
models
d l
 Ordinary least squares (TS)
 Logistic
L i ti regression
i (PDPS)
► Also
adjusted for:
 Demographic
D
hi characteristics
h
i i (age,
(
gender,
d child’s
hild’
geographic origin)
 Placement characteristics (spell length
length, severity level
and/or specialty population served by unit where child
is placed, program’s geographic location)
 Prior placement in residential treatment
Base Model (Prior Child System
Involvement Variables Only)
Post-Discharge
P
t Di h
Placement Stability
Treatment Stability
Risk Adjustor
Β
95% CI
Β
95% CI
Aggressive symptoms & antipsychotic
use
-.024*
(-.042, -.005)
-.35
(-.79, .09)
Detention or corrections
-.034***
(-.050, -.017)
-.53*
(-.95, -.12)
Runaway
-.040***
(-.059, -.020)
-.58*
(-1.03, -.13)
P
Psychiatric
hi t i h
hospitalization
it li ti
-.035***
035***
( 055 -.016)
(-.055,
016)
-.50*
50*
( 95 -.05)
(-.95,
05)
*p< 05 **p<.01,
*p<.05,
**p< 01 ***p<.001
***p< 001
Base Model + Clinical Factors
Post-Discharge
Placement Stability
Treatment Stability
Risk Adjustor
Β
95% CI
Β
95% CI
Aggress symptoms/antipsychotic use
-.022*
(-.041, -.003)
-.39
(-.85, .07)
Detention or corrections
-.031**
(-.050, -.013)
-.33
(-.79, .12)
Runaway
-.036***
(-.056, -.016)
-.48
(-.96, .002)
P
Psychiatric
hi t i h
hospitalization
it li ti
-.030**
030**
( 050 -.010)
(-.050,
010)
-.56*
56*
( 1 03 -.08)
(-1.03,
08)
Externalizing
-.004
(-.013, .005)
-.40**
(-.63, -.16)
Trauma
-.006
((-.015,, .004))
.15
((-.08,, .38))
Sexuality
-.002
(-.011, .008)
-.01
(-.25, .24)
School problems
-.015**
(-.025, -.005)
-.24
(-.49, .01)
Juvenile justice interactions
-.004
(-.016, .007)
-.16
(-.45, .12)
Acculturation
-.001
(-.009, .008)
.08
(-.12, .29)
Internalizing
- 016**
-.016
((-.027,
027 -.006)
006)
.02
02
((-.23,
23 .28)
28)
*p<.05, **p<.01, ***p<.001
Preliminary Conclusions
► Prior
child system involvement variables
explain unique variation in treatment
stability and postpost-discharge placement
stability
► Clinical needs variables explain additional
variation
a at o in tthese
ese outcomes
outco es
 Treatment stability: internalizing, school issues
 Post
Post--discharge placement stability: externalizing
► Prior
child system involvement variables do
not appear to be proxies for clinical needs
Limitations
► Risk
adjustment doesn’t create perfectly
equivalent
q
groups
g p or achieve the purism
p
of
random assignment
► Best models explain only 15%
15%--23% of
variation in outcomes
► CANS
rates child and adolescent needs, not
diagnoses
Next Steps
► Other
items
methods for incorporating clinical needs
 CANS item level: classification and regression trees
analysis (to identify significant interactions)
interactions)
 Rasch analysis (to derive a total clinical needs score)
► Explore
OLS alternatives for treatment stability
model
► Examine p
predictive power
p
of best models
Acknowledgements
►
Collaborators
 Jielai Ma, PhD, Northwestern
University
 Richard
Ri h d Epstein,
E t i PhD,
PhD
Vanderbilt University
 Scott Leon, PhD, Loyola
University Chicago
 Andy Zinn, PhD, Chapin Hall
at the University of Chicago
 Alan Morris,
Morris PsyD
PsyD,, University
of Illinois
Illinois--Chicago
 Deann Muehlbauer, MPH,
University of Illinois
Illinois--Chicago
 Kathleen Kearney, JD,
University of Illinois at
Urbana--Champaign
Urbana
p g
►
More collaborators
 Christopher Larrison, PhD,
University of Illinois at
Urbana--Champaign
Urbana
 Gary McClelland, PhD,
Northwestern University
 Brice Bloom
Bloom--Ellis,
Ellis MSW,
MSW
LCSW, Illinois Department of
Children & Family Services
►
Funding Support
 Children’s Bureau,
Administration for Children &
Families US Dept of Health
Families,
& Human Services
 Illinois Department of
Children & Familyy Services