Comparison of Alternative Employment Services for Persons with Serious Mental Illness:
Effects on Patient-Reported Mental Health Functional Status, Substance Use, and Arrests.
Salkever D, Steinwachs DM, Abrams M, Gibbons B, Baier K, Stuart EA, Skinner EA, Wu AW,
Salzberg C.
ABSTRACT
Objective: To evaluate the effect of initiation of evidence-based supported employment services
(Individual Placement and Support-IPS) for persons with serious mental illnesses (SMI) and
qualifying for state or federal disability on patient-reported mental health functional status,
substance use, and arrests.
Data Sources: Maryland Medicaid data was matched to data from the Division of Rehabilitative
Services and the Maryland Public Mental Health System to identify the study groups. Data from
Maryland Medicaid, Maryland Mental Hygiene Administration (MHA) and the Maryland Division of
Rehabilitation Services (DORS) were used to select study subjects, to identify initiation dates of
employment services that occurred between Sept 1, 2006 and Dec 31, 2007, and to assign each
study subject to a treatment group. Data from MHA's Outcomes Measurement System (OMS) were
used to measure baseline levels of the self-reported outcome measures and follow-up levels of
these same measures.
Study Design: An intent-to-treat design is used to compare three groups of Medicaid persons with
SMI: those initiating evidence-based IPS (placing people in competitive employment with support
services) meeting fidelity criteria (IPS-F), those initiating supported employment services without
evidence of fidelity to IPS (SE-NF), and those initiating traditional vocational services, emphasizing
training that is expected to lead to competitive employment (TVS). Propensity score weights are
used in comparing the three groups on a number of baseline variables from the Medicaid claims
data measured over the 12 months prior to the assigned initiation date of employment services for
each study subject. Outcome variables were measured from OMS data for follow-up periods of 12months, 24-months, and 36-months following each subject’s date of employment services initiation.
Data Extraction Methods: Data sets were provided by the Maryland Medicaid Program, the
Maryland DORS and the Maryland MHA's Public Mental Health System.
Principal Findings: Results based on a variety of regression methods with inverse-probability
propensity weights, showed inconsistent patterns of differences among the three treatment groups.
Failure to reject null hypotheses of no difference, however, must be viewed as tentative due mainly
to the relatively small number of study subjects with available OMS dependent variable data in
each of the three study groups.
Conclusions: The tentative conclusion is that the 3 study groups did not differ on these patientreported outcomes. Further assessment with additional data is, however, indicated because of
sample size and OMS data availability issues.
Keywords: Supported employment; individual Placement and Support (IPS); Patient-reported
mental health outcomes.
Acknowledgements: This research was supported by Contract No. HHSA290201000009I from
the Agency for Healthcare Research and Quality, US Department of Health and Human Services,
as part of the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) Program.
1
INTRODUCTION
Nationally, it is estimated that persons with serious mental illnesses (SMI) and serious
functional impairment comprise 4.1% of U.S. adults (SAMHSA, 2013). Recent data also suggest
that over 80% of persons with schizophrenia being treated in the community are not employed
(Salkever, et al., 2007). Moreover, employment is cited as a goal by almost 2/3 of all persons with
SMI in the U.S. public mental health system (Bond and Drake, 2014; Bedell et al. 1998;
Frounfelker et al. 2011; McQuilkenet al. 2003; Mueser, Salyers and Mueser, 2001; Ramsay et al.
2011; Rogers et al. 1991; Watkins et al. 2011; Woltmann 2009). In contrast studies show that only
about 15% report any current paid employment (Bond and Drake, 2014; Lindamer et al. 2003;
Pandiani and Leno 2012; Perkins and Rinaldi 2002; Rosenheck et al. 2006; Salkever et al. 2007).
Given this large gap between those wanting to work and those actually working, increasing
paid employment has been viewed as a principal recovery goal for rehabilitation programs for
those with SMI. Research shows that Individual Placement and Support (IPS), an evidence-based
supported employment intervention, can achieve competitive employment positions for half or
more of persons with SMI desiring work (Bond, Drake, and Becker, 2008). In randomized trials of
IPS vs. traditional rehabilitation programs that are not vocationally oriented, IPS has shown strong
positive effects in helping patients to gain paid (mainly competitive) employment (Marshall et al.,
2014). As Salkever (2010, 2013) has noted, comparisons with more traditional alternative
vocationally-oriented services, such as an enhanced vocational rehabilitation (EVR) (Drake et al.,
1999) or a diversified placement approach (DPA) (Bond et al. 2007) show significantly higher
competitive employment rates for IPS but similar overall paid employment rates (including agency
jobs and other non-competitive paid employment) for EVR and DPA.
In contrast to clear results about IPS effectiveness in achieving competitive employment,
IPS effectiveness in achieving nonvocational outcomes has been the subject of debate in the
recent literature. A recent article by two of the leading authorities in the field (Bond and Drake,
2014) forcefully argues that IPS does have positive nonvocational effects stemming from the
combination of (1) positive IPS effects on employment and (2) the positive effects of employment
on nonvocational outcomes. Their argument is as follows:
IPS is a highly effective approach to vocational rehabilitation…(Becker et al.
2011)….systematic reviews conclude that IPS enhances vocational outcomes (Bond 2004;
Bond et al. 2008; Burns et al. 2007; Crowther et al. 2001; Dixon et al. 2010; Twamley et al.
2003). About two-thirds of IPS participants succeed in competitive employment…(Bond et
al. 2012a) and sustaining employment for years (Becker et al. 2007b; Salyers et al. 2004)….
People who obtain competitive employment through IPS enhance their income, self-esteem,
quality of life, social inclusion, and control of symptoms (Bond et al. 2001; Burns et al. 2009;
Kukla et al.2012; Mueser et al. 1997;Turner et al. 2012). These enhancements to well-being
persist at 10-year follow-ups (Becker et al. 2007b; Salyers et al. 2004). People with SMI
often report that IPS is good treatment and central to their recovery (Bailey 1998;Becker et
al. 2007b; Strickler et al. 2009)….employment leads to decreased mental health costs
(Bond et al. 1995; Burns et al. 2009;Clark 1998; Henry et al. 2004; Latimer 2001; Perkins et
al. 2005; Rogers et al. 1995; Schneider et al. 2009). Long-term cost reductions appear to
be even greater (Bush et al. 2009).
An alternative view is expressed in a recent article by Kukla and Bond (2013), who report
on a two-year randomized trial of an IPS program vs. a DPA program. The nonvocational
2
outcomes studied pertained to symptoms, psychiatric hospitalizations, quality of life, and social
networks. They summarize their results and conclusion as follows:
Although the total sample showed improvement in several nonvocational domains over time,
there were largely no differences between groups in nonvocational outcomes at follow-up or
in their rates of improvement over time…Participation in supported employment alone is not
sufficient to positively impact most nonvocational outcomes in people with severe mental
illness.
To further reinforce Kukla and Bond’s (2013) alternative view that questions IPS effects on
these nonvocational outcomes, we have previously noted (Salkever 2010 and 2013) that evidence
of a positive contemporaneous association between employment status and these nonvocational
measures should not be interpreted as a causal influence of employment because of the strong
likelihood of unmeasured selection factors that are positively correlated with both employment and
the nonvocational measures. This same concern about inferring causation is in fact explicitly noted
in Kukla et al. (2012), Bond et al. (2001), Burns et al. (2009), and Mueser et al. (1997), all being
papers cited by Bond and Drake in support of their argument for the nonvocational effects of IPS.
Given the divergence of views (e.g., between Kukla and Bond (2013) and Bond and Drake
(2014)) about the effects of IPS on nonvocational outcomes, we undertook our own analyses that
expanded the range of the research in several ways. First, it presented comparisons between
certified fidelity-compliant IPS vs. the alternative “treatments” of (1) SE that was not certified as
fidelity compliant and (2) a range of other “traditional” vocational rehabilitation services provide by
a state vocational rehabilitation agency. Second, these analyses involved new outcome measures.
In the present analysis, we study patient-reported outcome measures from a new statewide
Outcomes Measurement System (OMS) used in the state of Maryland for all patients served under
Maryland’s Public Mental Health System-managed specialty mental health care program. In a
companion study, state Medicaid claims data are used to measure IPS vs. alternative treatment
effects on measures of continuity and coordination of treatment services, testing the hypothesis
that continuity and coordination of these services for persons with severe mental illness may be
influenced by IPS efforts to coordinate vocational services with other mental health treatment
services (Steinwachs, et al., 2014).
Traditional employment services in Maryland differ from the IPS supported employment
approach in that they emphasize training and non-competitive job experiences (e.g., enclave
employment) prior to or instead of placement in a competitive employment environment.
In 2002, the Maryland Mental Hygiene Administration implemented a statewide program to
provide evidence-based employment services for all persons with SMI wanting to work, largely
replacing traditional employment training and placement programs. For persons with SMI,
competitive employment has been recognized as key outcome of the recovery process
(President’s New Freedom Commission on Mental Health (2003); US Surgeon General (1999);
National Institute of Mental Health (1999)). We hypothesized that the IPS employment intervention
would have desirable effects on non-employment outcomes. Specifically, we expect that the IPS
employment intervention would have positive effects on individuals’ self-reported functional status,
and would reduce the frequency of negative outcomes such as substance use and arrests.
In this study, comparisons are made among consumers meeting SMI and continuous
Medicaid enrollment criteria who entered either an IPS program meeting established fidelity
standards (IPS-F), a supported employment program without fidelity (SE-NF) or a traditional
employment services program (TVS). We hypothesized that IPS-F would be associated with
3
improved coordination of services as compared to TVS and possibly similar to SE-NF employment
program outcomes. IPS-F is expected to:



Increase the positivity of each of 5 patient-reported ratings of their functional status
with respect to 1) participating in meaningful activities, 2) taking care of personal
needs, 3) coping with problems, 4) doing things the patient wants to do, 5) reducing
the perceived burden of mental illness symptoms.
Reduce the frequency of self-reported alcohol use, drug use, and substance abuse.
Reduce the frequency of self-reported arrests.
BACKGROUND
Individual Placement and Support (IPS) is a version of supported employment that uses a
“place and train” approach (Wehman and Moon, 1988) in contrast to traditional practices in which
extensive pre-vocational training is provided prior to job placement (“train and place” model). The
supported employment model was adapted to meet the needs of persons with SMI and multiple
randomized controlled trials have established its efficacy in improvement employment outcomes
(Bond, Drake and Becker, 2008). In nine RCTs conducted in eight states (including Maryland), IPS
employment rates ranged from 27% to >75%, compared to controls that achieved 7% - 40% IPS. A
recent Cochrane Review of 14 RCTs in the U.S. and elsewhere compared IPS with other programs
(largely emphasizing training prior to work placement) and focused on the employment outcomes
of time to obtain competitive paid job sand duration of employment. IPS programs consistently
outperformed traditional employment services (Kinoshita, 2013; Marshall et al., 2014)).
According to the Dartmouth Psychiatric Research Center (2011), IPS is based on 8
principles: (1) involvement in competitive employment from the outset instead of placement in a
sheltered or non-competitive work setting, (2) all consumers (the person with the SMI) desiring to
work are eligible for IPS, (3) rapid job search (placement frequently within a month), (4) integration
of mental health and employment services, (5) attention to consumer preference in the job search,
(6) time-unlimited individualized job supports, and (7) personalized benefits counseling, and (8)
systematic job development.
DATA AND METHODS
Study Population - The sample selection criteria identified persons with serious mental illness
(SMI) diagnoses and state or federal certified disability as identified in the Medicaid enrollment files
with at least 10 months per year of Medicaid enrollment over a 4-5 year period, 2 years prior and 3
years post initiation of an employment services intervention. Persons were eligible for inclusion if
their first employment services contact occurred between September 1, 2006, and December 31,
2007, which marked the beginning of statewide implementation of an Outcomes Measurement
System (OMS) that is used by Maryland’s Public Mental Health System (PMHS) to track the
progress of the patients that they serve. Persons were included in the study if they met all the
following selection criteria. The figures in parentheses show the numbers remaining after
completion of the step.
1. Had a serious mental illness diagnosis in calendar years 2006 or 2007 (n=131,820)
2. Was 20-63 years of age between 9/1/2006 and 12/31/2007) (n=79,759)
3. Eligible for disability benefits according to state or federal criteria during the period 7/1/2006
to 12/31/2007 (n=35,253)
4. Was not a Medicare enrollee at initiation of employment services (n=22,556), given that
Medicare data were not available for this investigation
4
5. Had at least 10 months of Medicaid enrollment in each 12 month period two years before
and two years after the employment services initiation period, 9/1/2006 and 12/31/2007
(n=16,321)
6. Had a matching DORS or PMHS record indicating initiation of employment services
between 9/1/2006 and 12/31/2007 (n=618)
7. Did not receive employment services in the year before the initiation period and lived in
Maryland during the same period (n=433)
Serious mental illness was defined as the presence of at least one of the following International
Classification of Diseases Version 9 (ICD-9) diagnostic codes, as recorded in the Medicaid data:
295.xx
Schizophrenic disorders
296.xx
Episodic mood disorders (mania, bipolar, major depression)
297.xx
Delusional Disorders
298.xx
Other Nonorganic Psychoses
299.8x, 299.9xOther pervasive developmental disorders
300.xx
Anxiety states
301.xx
Personality disorders
302.xx
Sexual and gender identity disorders (excluding 302.7x,
psychosexual dysfunction)
310.xx
Specific nonpsychotic mental disorders due to brain damage
311.xx
Depressive disorders, not elsewhere classified
Classification of subjects in three treatment groups - Individuals in our sample of 433 were
classified into one of three mutually exclusive groups based on the type of service they received at
the first employment service received between 9/1/2006 and 12/31/2007:
 Individual Placement and Support (IPS-F) programs certified as meeting the criteria for
fidelity to the IPS model at time of service initiation (N=136);
 Supported employment services that place clients in competitive employment (SE-NF);
these programs may or may not follow the IPS model and were not certified as meeting the
IPS fidelity criteria at time of service initiation (N=171); and
 Traditional vocational services (TVS) which offer a “train and place” approach (N=126).
Outcome Measurement System Data - The Outcome Measurement System (OMS) collects selfreported data from persons in Maryland’s PMHS on mental health status and related subject areas
such as substance use, employment status, and arrests. The reported interview data were
obtained by clinical provider staff who administered the interview to the PMHS patients. (Note that
dual-eligible patients were not included in the OMS interview process.)
The study start date is September 1, 2006, which is also the start date for OMS
implementation in the PMHS system. OMS data were obtained for the study initiation period,
September 2006 through December 2008, as well as for three years thereafter, 2009 through 2011.
Persons receiving outpatient mental health services covered by the PMHS who are not dually
eligible should have one or more OMS records as long as they are not receiving services from an
individual practitioner. Completed OMS questionnaires are submitted by Outpatient Mental Health
Centers (OMHC’s), Federally Qualified Health Centers (FQHC’s) and hospital based clinics (known
as HSCRC’s). The OMS questionnaire is reported approximately every 6 months for persons in
continuous treatment and helps track patient progress for providers as well as for the overall PMHS.
5
The number of persons in our study sample who have OMS records depends on whether
the person received mental health services covered by the PMHS in the study period and on
whether persons received services at an OMHC, FQHC, or HSCRC. Of our 433 study subjects, 52
persons did not have any OMS records. Across treatment groups, the numbers without any OMS
records are 10 (6.3%) for the IPS-F group, 13 (8.8%) for the SE-NF group, and 29 (23.0%) from
the TVS group. For those that have at least 1 OMS record, persistence in treatment helps
determine the number of OMS records each person has. Table 1 below displays descriptive
statistics on the number of records per person in the study period (excluding the 52 cases with 0
records). The TVS group has a lower average number of records (7.1) than both the IPS-F group
(10.1 records) and the SE-NF group (10.0).
Table 1: Total OMS records per person in Study Period
IPS-F
SE-NF
N
159
148
mean
9.6
9.3
sd
4.9
5.3
min
0
0
max
20
17
N with >0 OMS
Records
149
135
Mean # OMS
Records if >0
10.2
10.2
TVS
126
5.4
5.1
0
17
97
7.1
While many persons in the PMHS who are persistent in treatment have OMS records at
regular intervals of approximately 6 months, this is not the case for a substantial portion of this
population who are in and out of treatment. There is also an issue with matching the OMS records
with the initiation date. Table 2 below gives the mean and standard deviation for the number of
days between the initiation date and the closest OMS record, before or after the initiation date. The
average for the IPS-F and the SE-NF groups are 101 days and 136 days while the TVS group is
even higher at 308 days. These are highly skewed distributions with large outliers for persons who
had one or more records much later or in some cases much earlier than their initiation date.
Table 2: Descriptive Statistics for OMS-Initiation Date Time Intervals, by treatment group
IPS-F
n
initiation date
159
OMS Baseline record date
102
# of days between initiation
date and closest OMS record
149
1 yr. post-inititation date
(initiation date+365)
OMS 1 yr. Outcome record
date
2 yr. post-initiation date
(imitation date+730)
OMS 2 yr. Outcome record
date
159
106
159
111
Mean
Date
4-May
07
17May
07
101
3May08
21Apr-08
3May09
27Apr-09
SE-NF
(# of
days)
sd
TVS
n
Mean
Date
(# of
days)
sd
n
132
148
25-Apr-07
139
126
Mean
Date
18-Feb07
130
88
18-May
07
144
52
2-Mar07
116
163
135
136
234
97
308
488
132
148
24-Apr-08
139
126
144
103
21-Apr-08
158
56
132
148
24-Apr-09
139
126
146
91
15-Apr-09
152
50
18-Feb08
12-Feb08
17-Feb09
11-Feb09
(# of
days)
sd
110
110
122
110
117
6
3 yr. post-initiation date
(imitation date+1,095)
OMS 3 yr. Outcome record
date
# of days between baseline
OMS record date and OMS
1-yr. follow-up record date
# of days between baseline
OMS record date and OMS
2-yr. follow-up record date
# of days between baseline
OMS record date and OMS
3-yr. follow-up record date
159
102
3May10
3May10
132
148
24-Apr-10
139
126
17-Feb10
110
144
86
11-May10
147
40
23-Feb10
106
75
371
61
74
367
65
35
372
49
77
730
71
59
735
65
30
752
70
73
1093
68
58
1102
66
23
1104
61
Since the correspondence between the OMS dates and initiation dates was variable and
rarely exact, we adopted a set of rules for matching the various OMS records with the initiation
date and three annual follow-up dates. An OMS record for each person is determined to be a
baseline OMS record if the service date is 1) within 90 days before or after the initiation date and 2)
is the closest OMS record date to the initiation date for that person. The band of 90 days before or
after initiation date was chosen as a reasonable timeframe that balances the desire to have OMS
records be as close to the initiation date as possible with the need to have as many persons as
possible in the analysis. Descriptive statistics are presented in Table 2 by treatment groups for a)
the initiation date, b) the number of days between the initiation date and the closest OMS record
and c) the baseline OMS record chosen.
The OMS record assigned to the 1-year follow-up was chosen as the closest OMS record to
the 1-year anniversary of the initiation date, with the proviso that this OMS record must be within
90 days of that anniversary date. Similar rules were applied to assign the OMS records for the 2year and 3-year anniversaries of the initiation date for each person. The number of study subjects
for whom we found OMS records assigned to each of the anniversary dates are also reported in
Table 2.
Since the initiation dates could be as early as 9/1/2006, and the OMS surveys only began
on 9/1/2006, almost all of the persons with no baseline OMS record had an initiation date in the
latter part of 2006 or early 2007. The share of persons missing a baseline record ranges from
35.8% for the IPS-F group up to 58.7% for the TVS group. A total of 265 persons were found to
have had an OMS record within the time-window for the 1-year follow-up, including 66.7% of the
IPS-F group, 69.6% of the SE-NF group and only 44.4% of the TVS group. The number of persons
with an OMS record within the 2-year follow-up time window was 252; the corresponding figure for
those within the 3-year follow-up time window only dropped to 228.
The first 2 rows of the table compare the mean initiation dates and mean baseline OMS
dates across the 3 groups. The correspondence of the two dates is close for each of the groups.
Row 3 in the table shows that the difference between the initiation date and the closest OMS
record (which may not be in the time window to be designated as a baseline OMS record) is larger
for the SE-NF and TVS groups. The next 6 rows show that for each of the 3 groups the mean
follow-up dates based on the initiation date and the mean OMS follow-up record dates are similar.
The final 3 rows show that the mean time durations between the baseline OMS dates and the
follow-up OMS dates are also very similar across the groups. The latter result is of particular
interest since some of our analyses will examine OMS outcome variables controlling for baseline
values of these same variables.
7
Dependent Variables in the Analysis - The OMS dependent variables for this study are in three
subject areas: functional status, alcohol and substance use, and criminal justice issues. All
dependent variables come from the self-reported answers to the OMS questionnaire that is
facilitated by the mental health provider. A list of dependent variables names, definitions, sources,
and coded values are shown in Table 3. Descriptive data on these dependent variables are
reported in Table 4. Note that for each of these variables, in each of the three follow-up years, the
number of reported values is not quite as large as the numbers of persons with follow-up OMS
records shown in Table 2.
As part of the OMS interview, there are 24 questions that together make up a psychiatric
instrument called the BASIS-24 (Eisen et al., 2004). The BASIS-24 includes an overall score and
seven sub-scores. BASIS-24 scores are continuous variables that are constructed from categorical
questions whose responses have an ordinal nature. More detail is provided on each dependent
variable below. Other questions from the OMS, besides those in the BASIS-24, either have
ordered categorical or binary responses.
As shown in Table 3, the dependent variables include: 5 variables describing responses to
each of 5 questions about functioning (included in the OMS but not in the BASIS-24), a re-coded
BASIS-24 substance abuse sub-score (which covers both problematic alcohol and other substance
use in the past week), one binary variable pertaining to a question about alcohol use in the past
month, one binary variable pertaining to a question about drug use in the past month, and one
binary variable pertaining to a question about arrests in the past 12 months.
Table 3. OMS Dependent Variable definitions
VARIABLE NAME*
q1func_y1
DEFINITION
I do things that are meaningful to me
VALUE RANGE
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
q2func_y1
I am able to take care of my needs
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
q3func_y1
I am able to handle things when they go
wrong
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
q4func_y1
I am able to do things that I want to do
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
q5func_y1
My symptoms bother me
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
alc_pastmo_y1**
Alcohol use in the past month
1=any use; 0=No use
du_pastmo_y1**
Drug use in the past month
1=any use; 0=No use
b24sa_y1
BASIS-24 Substance Use
1=BASIS-24 Substance Abuse score>0 ;
0=BASIS-24 Substance Abuse score of
0
arrest12mo_y1
Arrested in the past 12 months
1=arrest in past 12 months; 0=no arrests
* “y1” refers to the 1-year follow-up. Analogous dependent variables are defined for the 2-year and 3-year
follow-ups, designated by the suffixes “y2” and “y3”.
**No 3-year follow-up is included in our study because the OMS dropped this question from its survey and
numbers of responses in the 3-year follow-up were very small in all treatment groups.
8
Descriptive statistics on these dependent variables are shown in Table 4.
Table 4: Descriptive Statistics (Unweighted) for Dependent Variables by
Treatment Group
EBP-SE
non-EBP SE
N
mean
Sd
N
mean
sd
N
TVS
mea
n
q1func_y1
q2func_y1
q3func_y1
q4func_y1
q5func_y1
alc_pastmo_y1
du_pastmo_y1
b24sa_y1
96
97
97
96
97
95
94
90
2.06
2.10
2.52
2.39
3.38
0.16
0.04
0.29
0.71
0.74
0.95
0.90
1.19
0.37
0.20
0.46
90
90
89
89
89
89
88
82
2.08
1.98
2.72
2.31
3.39
0.15
0.05
0.30
0.82
0.83
0.98
0.97
1.09
0.36
0.21
0.46
52
52
52
52
52
52
52
50
2.15
2.04
2.73
2.77
3.38
0.15
0.04
0.16
1.02
0.88
1.12
0.96
1.21
0.36
0.19
0.37
arrest12mo_y1
102
0.06
0.24
88
0.08
0.27
52
0.13
0.34
q1func_y2
q2func_y2
q3func_y2
q4func_y2
q5func_y2
alc_pastmo_y2
du_pastmo_y2
b24sa_y2
100
100
100
100
100
81
81
96
2.09
2.05
2.44
2.28
2.92
0.16
0.02
0.33
0.87
0.76
0.94
0.87
1.19
0.37
0.16
0.47
84
84
84
84
84
67
66
79
2.14
2.15
2.67
2.58
3.18
0.16
0.08
0.39
0.97
0.78
1.03
1.01
1.10
0.37
0.27
0.49
48
48
48
48
48
47
47
46
2.08
2.25
2.90
2.48
3.88
0.15
0.02
0.26
1.07
0.96
1.06
1.13
1.12
0.36
0.15
0.44
arrest12mo_y2
111
0.01
0.09
91
0.08
0.27
50
0.04
0.20
q1func_y3
q2func_y3
q3func_y3
q4func_y3
q5func_y3
b24sa_y3
91
91
91
91
89
88
1.99
2.04
2.48
2.40
2.80
0.34
0.75
0.76
0.81
0.89
1.17
0.48
73
73
73
73
73
72
2.00
2.15
2.68
2.51
2.58
0.46
0.76
0.79
0.93
0.96
1.12
0.50
39
39
39
39
39
36
2.23
2.13
2.54
2.51
2.13
0.33
0.90
0.83
1.05
0.88
1.15
0.48
arrest12mo_y3
102
0.01
0.10
86
0.02
0.15
40
0.03
0.16
Sd
Yr.1 Follow-Up
Yr.2 Follow-Up
Yr.3 Follow-Up
A number of salient features should be noted. First, the numbers of study subjects with
reported values are generally slightly below the numbers with OMS records for the corresponding
follow-up years (see Table 2). It thus appears that some OMS interviews did not have a complete
set of responses. Second, average rates of non-zero responses were very low for several of the
outcome variables, including the indicators for alcohol use and separately drug use in the prior
month, and the indicator for arrests in the prior 12 months. The small amount of variance in these
indicators, combined with the small sample size, implies that lack of power is a potential problem
9
for statistical tests across treatment groups for these outcomes. Third, there are no cases where
the differences between treatment groups exceeds the sizes of the standard errors. This suggests
that significant differences across these groups are unlikely to be observed in the absence of
controlling on differences between the groups in covariates either by propensity weighting or by
regression controls.
Covariates – The covariates used as regression controls in our analysis are defined in Table 5.
They include baseline OMS values for our dependent variables as well as diagnostic flags from
Medicaid claims data and baseline OMS data on employment status and on the person’s overall
BASIS-24 mental health status score.
Table 5: Sources and Definitions for Covariates
VARIABLE
NAME
q1func_bline
DATA
SOURCE
OMS
q2func_bline
OMS
q3func_bline
OMS
q4func_bline
OMS
q5func_bline
OMS
alc_pastmo_bline
OMS
du_pastmo_bline
OMS
b24sa_bline
OMS (B24)
arrest12mo_bline
OMS
emp_past6
OMS
b24ovrall_bline
schiz_diag
dep_bip_diag
other_diag
OMS (B24)
Medicaid
claims
Medicaid
claims
Medicaid
claims
DEFINITION
I do things that are
meaningful to me
I am able to take care of
my needs
I am able to handle
things when they go
wrong
I am able to do things
that I want to do
My symptoms bother me
Alcohol use in the past
month in binary form
Drug use in the past
month in binary form
BASIS-24 Substance
Use in binary form
Arrested in the past 12
months binary
currently employed or in
past 6 mos.
BASIS-24 overall score
VALUE RANGE
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
1=strongly agree, 2=agree, 3=neutral,
4=disagree, 5=strongly disagree
1=any use; 0=No use
1=any use; 0=No use
1=BASIS-24 Substance Use score>0 ;
0=BASIS-24 Substance Abuse score of
0
1=arrest in past 12 months; 0=no
arrests
1=employed in past 6 mos.; 0=not
continuous score from 1 (fewest
symptoms) to 5 (most symptoms)
Schizophrenia diagnosis
1=schizophrenia diagnosis; 0=other
Bipolar or major
depression diag.
Other SMI diagnosis
1=bipolar or depression diag.; 0=other
1=other SMI diagnosis; 0=not
Table 6 presents descriptive statistics on these variables across the 3 treatment groups. As
in the case of the dependent variables, we see relatively small differences across the groups in all
variables. The most substantial difference is in the primary diagnosis variables, where the fractions
of cases with a schizophrenia diagnosis is considerably smaller in the TVS group, while the
fractions of cases with bipolar or major depression primary diagnoses and with other primary
diagnoses is larger in the TVS group.
10
Table 6: Descriptive Statistics for Covariates
IPS-F
SE-NF
TVS
Variables
N
mean sd
N
Mean
sd
N
Mean
q1func_bline
98
2.06 0.82
82
2.11 0.98
52
2.15
q2func_bline
98
2.08 0.81
82
2.00 0.86
52
2.25
q3func_bline
98
2.66 0.91
82
2.45 1.01
52
2.62
q4func_bline
98
2.46 0.99
82
2.41 0.97
52
2.50
q5func_bline
98
3.45 1.15
81
3.25 1.21
52
3.79
alc_pastmo_bline
94
0.13 0.34
81
0.17 0.38
51
0.22
du_pastmo_bline
93
0.04 0.20
81
0.05 0.22
51
0.08
b24sa_bline
85
0.26 0.44
76
0.36 0.48
49
0.37
emp_past6
102
0.06 0.24
88
0.08 0.27
52
0.13
b24ovrall_bline
102
0.26 0.44
88
0.40 0.49
52
0.33
schiz_diag
84
1.21 0.60
76
1.09 0.67
49
1.31
dep_bip_diag
159
0.54 0.50 148
0.51 0.50 126
0.20
sd
1.07
1.12
1.12
1.32
1.07
0.42
0.27
0.49
0.34
0.47
0.68
0.40
other_diag
0.50
159
0.37
0.48
148
0.41
0.49
126
0.48
Demographic Comparisons – Table 7 provides comparative data, across the 3 treatment groups,
on age, race, gender, and location. The source for these data items is the Medicaid enrollment
records. Data are reported for those respondents with at least one OMS record.
Table 7: Descriptive Statistics for 0-1 Demographic Variables
IPS-F
N
mean
SE-NF
N
mean
TVS
N
mean
female
white_race
black_race
other_race
age
balt_suburbs
wash_suburbs
western_md
southern_md
easternshore_md
159
159
159
159
159
159
159
159
159
159
0.47
0.40
0.55
0.06
39.7
0.26
0.40
0.08
0.01
0.00
148
148
148
148
148
148
148
148
148
148
0.45
0.36
0.57
0.07
38.5
0.26
0.11
0.07
0.05
0.02
126
126
126
126
126
126
126
126
126
126
0.59
0.32
0.67
0.02
34.9
0.16
0.22
0.05
0.10
0.05
baltcity
159
0.21
148
0.38
126
0.41
The demographic differences between the groups are somewhat more pronounced. The IPS-F
group has lower percentages of females and blacks. It also has a much lower percentage of
persons residing in Baltimore City and a much higher percentage residing in the Washington
suburban areas. The TVS group has higher percentages of females, blacks, and persons residing
in Baltimore City and in Southern Maryland and the Eastern Shore.
Regression Analysis Methods - Separate regressions were estimated separately for the 1-year
follow up, the 2-year follow-up and the 3-year follow-up. In each case, 2 different model
specifications were used:
11

a simple model with only the treatment group dummies and the lagged (baseline)
value of the dependent variable as explanatory variables, and

a second model that included as additional covariates two baseline diagnosis
dummies, baseline employment status, and baseline overall BASIS-24 mental
health status score
The first model was estimated with un-weighted data while the second model used weighted data
with the weights derived from a propensity score analysis (described in the on-line supplement).
Thus, for each dependent variable we report 2 regression models for each of the 3 follow-up years.
The types of regression models that we applied varied with the type of dependent variables.
The 5 functional status variables are ordered qualitative responses so we applied the maximumlikelihood ordered probit regression estimation procedure. The four remaining dependent variables
were coded as binary outcomes and thus we applied the maximum-likelihood probit regression
model with binary outcomes. (In all regressions, TVS treatment is the reference category and the
reported results for the IPS-F and SE-NF groups indicated outcome differences relative to the TVS
group.
RESULTS
Self-Reported Functional Status Items from the OMS -We examined possible treatment effects
on responses to 5 different specific items in the OMS that were used to track functional status.
These items are to following statements:
1. I do things that are meaningful to me
2. I am able to take care of my needs
3. I am able to handle things when they go wrong
4. I am able to do things I want to do
5. My symptoms bother me
For each statement the respondent was asked to select one of five responses indicating their level
of agreement: strongly agree, agree, neutral, disagree, and strongly disagree. Since these
responses are ordered but do not represent interval data, we have estimated ordered probit
regression to obtain treatment effect estimates. It should also be noted that for the first 4 items,
higher responses imply poorer outcomes while for the last item, a higher response implies a better
outcome.
Item 1: I do things that are meaningful to me - Table 8 gives the estimated effects on the
responses to item 1 (I do things that are meaningful for me). Results indicate a consistent absence
of treatment effects regardless of the weights used or the set of covariates included in the
regression. The only consistently significant covariates (with positive coefficients) were the
baseline value of the outcome question and the baseline value of the BASIS-24 index (in 2 of 3
regressions).
Table 8. Ordered Probit Coefficients (and p-values) for Functional Status 1 (I do things that
are meaningful to me). (Negative sign indicates improving effect.)
1-year Follow-up
IPS-F
unweighted
0.0793
(0.738)
weighted
0.153
(0.510)
2-year Follow-up
unweighted
-0.031
(0.900)
weighted
0.279
(0.254)
3-year Follow-up
unweighted
-0.031
(0.912)
weighted
0.079
(0.770)
12
SE-NF
q1func_bline
-0.0864
(0.716)
0.283
(0.002)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
163
0.03
360.2
378.8
-0.046
(0.844)
0.223
(0.023)
0.001
(0.998)
0.502
(0.184)
0.056
(0.774)
0.284
(0.066)
150
0.071
335.5
365.6
-0.081
(0.748)
0.274
(0.007)
149
0.02
378
399.1
0.019
(0.939)
0.272
(0.020)
-0.074
(0.850)
0.030
(0.937)
0.367
(0.069)
0.435
(0.009)
135
0.082
344.9
376.9
-0.084
(0.770)
0.394
(0.001)
129
0.04
284.1
301.3
0.080
(0.779)
0.493
(0.000)
0.272
(0.482)
0.363
(0.348)
0.060
(0.789)
0.040
(0.830)
121
0.083
255.1
283.1
Item 2: I am able to take care of my needs - Table 9 gives the estimated coefficients on the
responses to item 2. Here again there are no significant treatment effects. The baseline value of
the outcome is again consistently significant (with positive coefficients), as is the baseline overall
BASIS-24 covariate (in 2 of 3 regressions). The only other significant covariate (in only 1
regression) is the baseline indicator that the respondent was employed in the 6 months prior to
baseline. The positive coefficient for this covariate implies that it is predictive of a poorer follow-up
outcome on this functional status indicator.
Table 9. Ordered Probit Coefficients (and p-values) for Functional Status 2 (I am able to take care of my
needs). (Negative sign indicates improving effect.)
1-year Follow-up
IPS-F
SE-NF
q2func_bline
unweighted
0.354
(0.135)
0.181
(0.452)
0.435
(0.000)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
164
0.057
365.6
387.3
weighted
0.292
(0.207)
0.141
(0.551)
0.495
(0.000)
0.008
(0.983)
0.105
(0.783)
0.470
(0.015)
-0.018
(0.902)
151
0.072
336
369.2
2-year Follow-up
unweighted
-0.099
(0.691)
0.149
(0.565)
0.353
(0.001)
149
0.04
314.2
332.2
weighted
-0.111
(0.663)
0.305
(0.236)
0.242
(0.059)
0.157
(0.711)
0.152
(0.713)
0.064
(0.760)
0.242
(0.129)
135
0.046
277.6
306.7
3-year Follow-up
Unweighted
0.146
(0.603)
0.253
(0.387)
0.216
(0.063)
129
0.014
282.7
299.9
weighted
-0.060
(0.832)
0.144
(0.618)
0.197
(0.158)
0.265
(0.506)
-0.137
(0.728)
-0.123
(0.588)
-0.109
(0.528)
121
0.032
240.4
268.4
13
Item 3: I am able to handle things when they go wrong - Table 10 presents ordered probit
results for the third of the five functioning questions. The only significant treatment effects are
positive (i.e., detrimental) effects of non-fidelity-certified SE vs. TVS, but only in year 3. Results for
the baseline outcome and baseline BASIS-24 covariates conform to our previous findings. There is
also some evidence, albeit inconsistent across years, of significant differences by diagnosis in this
particular outcome.
Table 10. Ordered Probit Coefficients (and p-values) for Functional Status 3 (I am able to handle things
when they go wrong). (Negative sign indicates improving effect.)
1-year Follow-up
IPS-F
SE-NF
q3func_bline
Unweighted
-0.0544
(0.810)
0.133
(0.564)
0.265
(0.003)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
163
0.022
435.5
457.1
weighted
0.136
(0.537)
0.378
(0.097)
0.226
(0.026)
-0.632
(0.076)
-0.457
(0.202)
0.177
(0.341)
0.460
(0.002)
150
0.08
387.1
420.2
2-year Follow-up
unweighted
-0.362
(0.129)
-0.076
(0.757)
0.449
(0.000)
149
0.061
389.7
410.8
weighted
-0.322
(0.181)
-0.025
(0.918)
0.340
(0.003)
0.199
(0.605)
0.656
(0.087)
0.024
(0.902)
0.253
(0.097)
135
0.098
342.7
374.6
3-year Follow-up
unweighted
0.201
(0.460)
0.511
(0.073)
0.548
(0.000)
129
0.087
319.2
339.2
weighted
0.340
(0.205)
0.576
(0.041)
0.702
(0.000)
-0.010
(0.979)
0.250
(0.497)
0.324
(0.136)
0.232
(0.196)
121
0.171
285.2
316
Item 4: I am able to do things I want to do - Results in Table 11 indicate only two significant
treatment effects, both for the SE-NF group, but the signs differ between year 1 (negative) and
year 2 (positive). Other results conform to our previous findings, with only the baseline dependent
variable and the baseline BASIS-24 value being significant.
Table 11. Ordered Probit Coefficients (and p-values) for Functional Status 4 (I am able to do things I want to
do). (Negative sign indicates improving effect.)
1-year Follow-up
IPS-F
SE-NF
unweighted
-0.302
(0.183)
-0.515
(0.025)
weighted
0.004
(0.985)
-0.238
(0.293)
2-year Follow-up
unweighted
0.023
(0.924)
0.350
(0.160)
Weighted
0.136
(0.573)
0.609
(0.013)
3-year Follow-up
unweighted
-0.074
(0.782)
0.111
(0.690)
Weighted
0.039
(0.881)
0.035
(0.899)
14
q4func_bline
0.269
(0.001)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
162
0.037
422.5
444.1
0.156
(0.107)
0.240
(0.511)
0.532
(0.155)
-0.137
(0.464)
0.511
(0.000)
149
0.079
384.9
417.9
0.366
(0.000)
149
0.05
392.3
413.3
0.284
(0.005)
-0.074
(0.850)
0.051
(0.896)
-0.154
(0.437)
0.362
(0.017)
135
0.08
350.9
382.8
0.242
(0.016)
129
0.021
327.6
347.7
0.199
(0.071)
0.249
(0.492)
0.021
(0.955)
-0.226
(0.290)
0.369
(0.028)
121
0.044
306.5
337.2
Item 5: My symptoms bother me – Results in Table 12 for this dependent variable show
significant treatment effects for both the IPS-F and SE-NF groups for years 1 and 2, but again the
signs are not consistent between the years. We also now observed inconsistent, and not always
significant results for the baseline dependent variable and BASIS-24 covariates. IN year 1, the
prior employment covariate is significant with a sign again suggesting a deleterious effect on the
outcome item, but results for years 2 and 3 are clearly not significant.
Table 12. Ordered Probit Coefficients (and p-values) for Functional Status 5 (My symptoms bother me).
(Positive sign indicates improving effect.)
1-year Follow-up
IPS-F
SE-NF
q5func_bline
unweighted
0.0304
(0.892)
0.214
(0.352)
0.314
(0.000)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
163
0.035
484.7
506.3
weighted
0.496
(0.025)
0.529*
(0.020)
0.121
(0.207)
0.257
(0.464)
0.615
(0.084)
-0.405
(0.027)
0.579***
(0.000)
150
0.089
440.9
474.1
2-year Follow-up
unweighted
-0.822
(0.001)
-0.619
(0.015)
0.040
(0.597)
148
0.027
453.7
474.6
Weighted
-0.632
(0.008)
-0.456
(0.062)
0.081
(0.411)
0.265
(0.488)
0.431
(0.251)
-0.003
(0.986)
0.155
(0.323)
134
0.036
426.2
458.1
3-year Follow-up
unweighted
0.347
(0.203)
0.215
(0.444)
-0.178*
(0.032)
126
0.021
378.7
398.5
weighted
0.205
(0.440)
0.243
(0.367)
-0.098
(0.324)
0.445
(0.216)
0.164
(0.649)
0.060
(0.773)
-0.545
(0.002)
118
0.071
345.9
376.4
15
Self-reported prior alcohol use and drug use – Regressions were estimated for 3 different
binary dependent variables pertaining to prior drug and alcohol use: 1) self-reported use of any
alcohol in the 30 days prior to the OMS interview, 2) 1) self-reported use of any illegal drugs in the
30 days prior to the OMS interview, and 3) a 0-1 recoding of the BASIS-24 substance abuse subscore.. This sub-score is derived from four of the questions from the BASIS-24 instrument and is in
the form of a cumulative score in which a lower score is better. Each of the four questions from
which the score is calculated ask whether a particular aspect of substance abuse occurred in the
past week. The four questions are the following: 1) Did you have an urge to drink alcohol or take
street drugs? 2) Did anyone talk to you about your drinking or drug use? 3) Did you try to hide your
drinking or drug use? 4) Did you have problems from your drinking or drug use? Answers to these
questions are in terms of the frequency they occurred in the past week: a) Never b) Rarely c)
Sometimes d) Often e) Always. Due to the sub-score having a high percentage of zeros indicating
no substance abuse problems, the decision was made to re-code the score into a binary variable
with 0 representing no substance abuse problems and 1 representing any substance abuse
problems.
Results for the first of these dependent variables are shown in Table 13. The only
significant treatment effect is a positive coefficient in year 1 for the IPS-F implying an increased
probability of alcohol use. The usual pattern of positive coefficients for the baseline dependent and
baseline BASIS-24 covariates and insignificant results for all other covariates was again observed.
Table 13. Marginal Effects (and p-values) for Probability of Alcohol use past 30 days.
1-year Follow-up
IPS-F
SE-NF
alc_pastmo_bline
unweighted
0.0841
(0.303)
0.0754
(0.359)
0.0214
(0.004)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
161
0.066
140.9
153.2
weighted
0.119
(0.014)
0.056
(0.267)
0.017
(0.001)
-0.100
(0.130)
-0.045
(0.507)
0.064
(0.091)
-0.068
(0.019)
150
0.095
283
307.1
2-year Follow-up
unweighted
0.017
(0.831)
0.051
(0.532)
0.021
(0.031)
124
0.051
102
113.3
weighted
-0.006
(0.905)
-0.020
(0.706)
0.021
(0.002)
-0.049
(0.549)
-0.032
(0.695)
-0.016
(0.705)
-0.066
(0.044)
116
0.064
197.6
219.6
The pattern of results for the binary indicator of self-reported drug use in the past 30 days, as
shown in Table 14, is similar. There is a finding of a positive treatment effect of IPS-F on probability
of drug use in year 2 when other (almost all insignificant) covariates are also included but no other
16
evidence of such an effect in other regressions. Positive effects are also observed for the baseline
value of the dependent variable.
.
Table 14. Marginal Effects (and p-values) for Probability
of Drug use past 30 days
1-year Follow-up
IPS-F
SE-NF
sa_pastmo_bline
unweighted
0.0109
(0.741)
-0.00459
(0.897)
0.0364
(0.045)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
160
0.33
42.27
54.57
Weighted
0.023
(0.262)
0.016
(0.470)
0.024
(0.001)
0.094
(0.997)
0.158
(0.996)
-0.144
(0.962)
-0.018
(0.157)
149
0.479
63.18
87.21
2-year Follow-up
unweighted
0.023
(0.701)
0.065
(0.270)
0.009
(0.020)
124
0.137
54.49
65.77
Weighted
0.081
(0.081)
0.090
(0.054)
0.006
(0.016)
0.352
(0.990)
0.391
(0.989)
0.001
(0.974)
-0.026
(0.270)
116
0.125
108.7
130.7
Regression results for the BASIS-24 substance abuse problem indicator are shown in Table
15. Treatment effect results for both year 1 and year 3 show significant increases in the probability
of any substance abuse problem for the IPS-F and SE-NF treatment groups; however in year 2
both groups show significantly negative coefficients (implying reduced probabilities of substance
use problems. Among the covariates, in year 1 we observe significantly negative coefficients for
prior employment and baseline overall BASIS-24, and we also observe a significantly positive
coefficient for the schizophrenia dummy in year 2. As in most other cases, the coefficient of the
baseline dependent variable is consistently and significantly positive.
Table 15. Marginal Effects (and p-values) for Probability of Any substance use problem in the past week
(BASIS-24).
1-year Follow-up
IPS-F
SE-NF
b24sa_bline
unweighted
0.217
(0.026)
0.197
(0.045)
0.291
(0.000)
weighted
0.235
(0.000)
0.186
(0.001)
0.315
(0.000)
2-year Follow-up
unweighted
0.049
(0.633)
0.040
(0.703)
0.299
(0.000)
Weighted
-0.115
(0.081)
-0.139
(0.043)
0.259
(0.000)
3-year Follow-up
unweighted
0.108
(0.379)
0.163
(0.189)
0.309
(0.000)
weighted
0.154
(0.035)
0.154
(0.047)
0.327
(0.000)
17
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
143
0.202
143.2
155.1
-0.089
(0.257)
-0.016
(0.844)
-0.124
(0.005)
-0.099
(0.004)
143
0.263
291.4
315.2
132
0.13
158.5
170
0.197
(0.092)
0.044
(0.711)
-0.076
(0.172)
-0.014
(0.727)
132
0.103
362
385
117
0.123
145.6
156.6
0.053
(0.596)
-0.061
(0.555)
-0.059
(0.323)
-0.006
(0.884)
117
0.128
328.2
350.3
Arrest in the Past 12 months – The last binary regression model is for the probability of an arrest
in the prior 12 months for each of the 3 follow-up years. In this case, no significant coefficients are
found for any treatment indicators or other covariates except for the baseline overall BASIS-24
score. This is not surprising since the observed rate of arrests in our data was very low. As shown
in Table 4, the fraction of respondents reporting any arrests in the prior 12 months did not exceed
o.1 for any of the three treatment groups for any of the three follow-up years with the sole
exception of the TVS group in the year 1 follow-up.
Table 16. Marginal Effects for Prob. of an Arrest in the past 12 months
1-year Follow-up*
IPS-F
SE-NF
unweighted
-0.0689
(0.123)
-0.0454
(0.302)
schiz_diag
dep_bip_diag
emp_past6
b24ovrall_bline
N
pseudo R-sq
AIC
BIC
242
0.018
141.6
152.1
weighted
-0.037
(0.212)
-0.024
(0.435)
-0.065
(0.149)
0.039
(0.342)
-0.037
(0.182)
0.109
(0.000)
209
0.208
260.8
284.2
2-year Follow-up*
unweighted
-0.049
(0.225)
0.026
(0.394)
156
0.074
64.42
73.57
weighted
-0.054
(0.115)
0.013
(0.649)
0.403
(0.988)
0.432
(0.987)
-0.045
(0.115)
0.043
(0.012)
143
0.136
127
147.7
3-year Follow-up*
unweighted
-0.016
(0.520)
-0.00131
(0.952)
weighted
0.00135
(0.930)
0.00889
(0.563)
144
0.057
33.51
42.42
228
0.007
85.05
95.34
* The baseline arrest variable perfectly predicts the outcome variable and is therefore dropped for all
regressions.
Conclusion
18
The analyses of patient-reported outcomes in the OMS provide little substantial evidence in
support of the alternative hypotheses that, compared to persons receiving traditional vocational
services, persons receiving IPS-F supported employment or SE-NF non-fidelity-certified supported
employment would have positive (improving) non-employment outcomes. There are, however, a
number of considerations which suggest that the power of our analysis to reject the corresponding
null hypotheses may have been low.
A principal concern is the relatively small sample sizes for the OMS regressions. Across
almost all regressions, the range of values observed for N was between 100 and 185; the only
exception was the regressions for year 1 and year 3 for arrest in the prior 12 months. Given the
variability inherent in individual self-report data, the multiple comparisons groups, and the need to
also include covariate controls, it is perhaps surprising that any regression coefficients turned out
to be significant.
Another aspect of problematic power in our analysis is that the rates of positive responses
for the binary outcome variables tended to be very low, indeed, well below our expectations.
For example, the Substance Abuse and Mental Health Services Administration (Epstein et al.,
2004) found that in 2002 28.7% of persons with a severe mental illness reported substance abuse
in the past year. Since the populations in our study are very different (e.g., all participants sought
and were receiving vocational services), it may not be surprising that percentages for reported
substance use were much lower, ranging from 2% to 6%, varying by treatment gr. These lower
self-reported percentages clearly imply a low level of power in the analysis that makes rejection of
the null hypothesis very difficult.
We also note the real possibility of selection on unobservable factors in our study samples
that bias the estimated effects of the treatment groups on the non-employment outcomes. The
observational nature of this study, with subjects self-selected into each of the three treatment
groups, introduces this possibility. While propensity-score weights help to balance the treatment
groups on observables, they do not necessarily correct for bias due to unobservable factors. There
is also some concern that the propensity-weights may not adequately balance the treatment
groups because the TVS group was small and may not have enough overlap.
In conclusion, we do not provide not evidence for the IPS-F treatment group having
superior non-employment outcomes relative to the TVS. There are a few particular results from the
different non-employment outcomes in certain follow-up years that are supportive. Overall though,
there are enough instances of contradictory results and a lack of consistency that undercut this
finding.
19
References
Bailey, J. 1998. “I’m Just An Ordinary Person.” Psychiatric Rehabilitation Journal, 22(1), 8–10.
Becker, D. R., Drake, R. E., and Bond, G. R. 2011. “Benchmark Outcomes In Supported
Employment.” American Journal of Psychiatric Rehabilitation, 14, 230–236.
Becker, D. R., Whitley, R., Bailey, E. L., and Drake, R. E. 2007. “Long-Term Employment
Outcomes Of Supported Employment For People With Severe Mental Illness.” Psychiatric Services,
58, 922–928.
Bedell, J. R., Draving, D., Parrish, A., Gervey, R., and Guastadisegni, P. 1998. “A Description And
Comparison Of Experiences Of People With Mental Disorders In Supported Employment And Paid
Prevocational Training”. Psychiatric Rehabilitation Journal, 21(3), 279–283.
Bond, G. R. 2004. “Supported Employment: Evidence For An Evidence-Based Practice.”
Psychiatric Rehabilitation Journal, 27, 345–359.
Bond, G. R., Campbell, K., and Drake, R. E. 2012. “Standardizing Measures In Four Domains Of
Employment Outcome For Individual Placement And Support.” Psychiatric Services, 63, 751–757.
Bond, G. R., Dietzen, L. L., Vogler, K. M., Katuin, C. H., McGrew, J. H., and Miller, L. D. 1995.
“Toward A Framework For Evaluating Costs And Benefits Of Psychiatric Rehabilitation: Three
Case Examples.” Journal of Vocational Rehabilitation, 5, 75–88.
Bond, G.R. and Drake, R.E. 2014. “Making The Case For IPS Supported Employment.”
Administration and Policy in Mental Health and Mental Health Services Research, 41:69–73.
Bond, G.R., Drake, R.E., and Becker, D.R. 2008. “An Update on Randomized Controlled Trials of
Evidence-Based Supported Employment”. Psychiatric Rehabilitation Journal 31(4):280-290.
Bond, G. R., Resnick, S. G., Drake, R. E., Xie, H., McHugo, G. J., and Bebout, R. R. 2001. “Does
Competitive Employment Improve Nonvocational Outcomes For People With Severe Mental
Illness?” Journal of Consulting and Clinical Psychology, 69, 489–501.
Bond, G.R., Salyers, M.P, Dincin, J., Drake, R., Becker, D.R., Fraser, V.V., and Haines, M. 2007.
“A Randomized Controlled Trial Comparing Two Vocational Models For Persons With Severe
Mental Illness.” Journal of Consulting and Clinical Psychology 75(6): 968-982.
Burns, T., Catty, J., Becker, T., Drake, R. E., Fioritti, A., Knapp, M., Lauber. C., Rössler, W., Tomov,
T., van Busschbach, J., White, S., Wiersma, D.; EQOLISE Group. 2007. “The Effectiveness Of
Supported Employment For People With Severe Mental Illness: A Randomised Controlled Trial.”
Lancet, 370, 1146–1152.
Burns, T., Catty, J., White, S., Becker, T., Koletsi, M., Fioritti, A., Rossler, W., Tomov T., van
Busschbach, J., Wiersma, D., Lauber, D., and for the EQOLISE Group. 2009. “The Impact Of
Supported Employment And Working On Clinical And Social Functioning: Results Of An
International Study Of Individual Placement And Support.” Schizophrenia Bulletin, 35, 949–958.
Bush, P. W., Drake, R. E., Xie, H., McHugo, G. J., & Haslett, W. R. 2009. “The Long-Term Impact
Of Employment On Mental Health Service Use And Costs.” Psychiatric Services, 60, 1024–1031.
20
Clark, R. E. 1998. “Supported Employment And Managed Care: Can They Coexist?” Psychiatric
Rehabilitation Journal, 22(1), 62–68.
Crowther, R. E., Marshall, M., Bond, G. R., and Huxley, P. 2001. “Helping People With Severe
Mental Illness To Obtain Work: Systematic Review.” British Medical Journal, 322, 204–208.
Dartmouth Psychiatric Research Center. “Practice Principles of IPS Supported Employment.”
10/11/2011. http://www.dartmouth.edu/~ips/page48/page79/files/ips-practice-principles002880029.pdf. Accessed August 20, 2014.
Dixon, L. B., Dickerson, F. B., Bellack, A. S., Bennett, M., Dickinson, D., Goldberg, R. W., Lehman,
A., Tenhula, W.N., Calmes, C., Pasillas, R.M., Peer, J., and Kreyenbuhl, J. 2010. “The 2009
Schizophrenia PORT Psychosocial Treatment Recommendations And Summary Statements.”
Schizophrenia Bulletin, 36, 48–70.
Drake RE, McHugo GJ, Bebout RR, Becker DR, Harris M, Bond GR and Quimby E. 1999. “A
Randomized Clinical Trial Of Supported Employment For Inner-City Patients With Severe Mental
Disorders.” Archives of General Psychiatry 56: 627-633
Eisen, S. V., Normand, S.-L., Belanger, A. J., Spiro, A., and Esch, D. 2004. “The Revised Behavior
and Symptom Identification Scale (BASIS-R): Reliability And Validity.” Medical Care, 42(12), 1230–
1241.
Epstein J., Barker, P., Vorburger, M., and Murtha, C. 2004) “Serious Mental Illness And Its
Co-Occurrence With Substance Use Disorders, 2002”. DHHS Publication No. SMA 04-3905,
Analytic Series A-24. Rockville, MD: Substance Abuse and Mental Health Services Administration,
Office of Applied Studies.
Frounfelker, R. L., Wilkniss, S. M., Bond, G. R., Devitt, T. S., and Drake, R. E. 2011. “Enrollment In
Supported Employment Services For Clients With A Co-Occurring Disorder.” Psychiatric Services,
62, 545–547.
Henry, A. D., Lucca, A. M., Banks, S., Simon, L., and Page, S. 2004. “Inpatient Hospitalizations
And Emergency Service Visits Among Participants In An Individual Placement And Support (IPS)
Model Program.” Mental Health Services Research, 6, 227–237.
Kinoshita, Y., Furukawa, T.A., Kinoshita, K., Honyashiki, M., Omori, I.M., Marshall, M., Bond, G.R.,
Huxley, P., Amano, N. and Kingdon, D. 2013. “Supported Employment for Adults with Severe
Mental Illness.” Cochrane Database of Systematic Reviews, Issue 9. Art. No,: CD008297. DOI:
10.1002/14651858.CD008297.pub2
Kukla, M., Bond, G. R., and Xie, H. 2012. “A Prospective Investigation Of Work And Nonvocational
Outcomes In Adults With Severe Mental Illness.” Journal of Nervous and Mental Disease, 200,
214–222.
Kukla M. and Bond GR. 2013. “A Randomized Controlled Trial Of Evidence-Based Supported
Employment: Nonvocational Outcomes.” Journal of Vocational Rehabilitation 38: 91–98.
Latimer, E. 2001. “Economic Impacts Of Supported Employment For The Severely Mentally Ill.”
Canadian Journal of Psychiatry, 46, 496–505.
Lindamer, L. A., Bailey, A., Hawthorne, W., Folsom, D. P., Gilmer, T. P., Garcia, P., Hough, R.L.,
21
Jeste, D.V. 2003). “Gender Differences In Characteristics And Service Use Of Public Mental Health
Patients With Schizophrenia.” Psychiatric Services, 54, 1407–1409.
Marshall, T., Goldberg, R. W., Braude, L., Dougherty, R. H., Daniels, A. S., Ghose, S. S., George,
P., Delphin-Rittmon, M.E. 2014. “Supported Employment: Assessing The Evidence.” Psychiatric
Services, 65, 16-23.
McQuilken, M., Zahniser, J. H., Novak, J., Starks, R. D., Olmos, A., and Bond, G. R. 2003. “The
Work Project Survey: Consumer Perspectives On Work.” Journal of Vocational Rehabilitation, 18,
59–68.
Mueser, K. T., Becker, D. R., Torrey, W. C., Xie, H., Bond, G. R., Drake, R. E., and Dain, B.J. 1997.
“Work And Nonvocational Domains Of Functioning In Persons With Severe Mental Illness: A
Longitudinal Analysis.” Journal of Nervous and Mental Disease, 185, 419–426.
Mueser, K. T., Salyers, M. P., and Mueser, P. R. 2001. “A Prospective Analysis Of Work In
Schizophrenia.” Schizophrenia Bulletin, 27, 281–296.
National Institute of Mental Health (1999). Bridging Science And Service: A report by the National
Advisory Mental Health Council’s Clinical Treatment and Services Research Workgroup. Rockville,
MD: National Institute of Mental Health.
Pandiani, J. and Leno, S. 2012. “Employment Of CRT Clients: Third Quarters (Jan–Mar) Of
FY2000–FY2012.” Waterbury: Vermont Agency of Human Services.
Perkins, R., and Rinaldi, M. 2002. “Unemployment Rates Among Patients With Long-Term Mental
Health Problems: A Decade Of Rising Unemployment.” Psychiatric Bulletin, 26, 295–298.
Perkins, D. V., Born, D. L., Raines, J. A., and Galka, S. W. 2005. “Program Evaluation From An
Ecological Perspective: Supported Employment Services For Persons With Serious Psychiatric
Disabilities.” Psychiatric Rehabilitation Journal, 28, 217–224.
President’s New Freedom Commission on Mental Health (2003). Report to the President from the
President’s New Freedom Commission on Mental Health. Washington DC: U.S. Government
Printing Office
Ramsay, C. E., Broussard, B., Goulding, S. M., Cristofaro, S., Hall, D., Kaslow, N. J., Killackey, E.,
Penn, D., and Compton, M.T. 2011. “Life And Treatment Goals Of Individuals Hospitalized For
First-Episode Nonaffective Psychosis.” Psychiatry Research, 189, 344–348.
Rogers, E. S., Walsh, D., Masotta, L., and Danley, K. 1991. “Massachusetts Survey Of Client
Preferences For Community Support Services: Final Report.” Boston: Center for Psychiatric
Rehabilitation.
Rogers, E. S., Sciarappa, K., MacDonald-Wilson, K., and Danley, K. 1995. “A Benefit-Cost
Analysis Of A Supported Employment Model For Persons With Psychiatric Disabilities.” Evaluation
and Program Planning, 18, 105–115.
Rosenheck, R. A., Leslie, D., Keefe, R., McEvoy, J., Swartz, M., Perkins, D., Stroup, D., Hsiao, J.K.,
Lieberman, J., and the CATIE Study Investigators Group. 2006. “Barriers To Employment For
People With Schizophrenia.” American Journal of Psychiatry, 163, 411–417.
22
Salkever, D., Karakas, M.C., Slade, E.P., Harding, C.M., Hough, R.L., Rosenheck, R.A., Swartz,
M.S., Barrio, C., Yamada, A.M. 2007. “Measures and Predictors of Community-Based
Employment and Earnings f Persons wth Schizophrenia in a Multisite Study.” Psychiatric Services
58(3):315-324.
Salkever, D. 2013. “Social Costs Of Expanding Access To Evidence-Based Supported
Employment: Concepts And Interpretive Review Of The Evidence.” Psychiatric Services 64:111–
119,
Salkever D. 2010. “Toward a Social Cost-Effectiveness Analysis of Programs to Expand Supported
Employment Services: An Interpretive Review of the Literature.” U.S. Dept. of Health and Human
Services, Office of the Assistant Secretary for Planning and Evaluation, Westat, Inc.December 31,
2010. http://aspe.hhs.gov/daltcp/reports/2010/supempLR.htm (accessed August 20, 2014).
Salyers, M. P., Becker, D. R., Drake, R. E., Torrey, W. C., and Wyzik, P. F. 2004. “Ten-Year
Follow-Up Of Clients In A Supported Employment Program.” Psychiatric Services, 55, 302–308.
Schneider, J., Boyce, M., Johnson, R., Secker, J., Slade, J., Grove, B., and Floyd, M. 2009.
“Impact Of Supported Employment On Service Costs And Income Of People With Mental Health
Needs.” Journal of Mental Health, 18, 533–542.
Strickler, D. C., Whitley, R., Becker, D. R., adn Drake, R. E. 2009. “First Person Accounts Of LongTerm Employment Activity Among People With Dual Diagnosis.” Psychiatric Rehabilitation Journal,
32, 261–268.
Substance Abuse and Mental Health Services Administration (SAMHSA), 2013. “Results from the
2012 National Survey on Drug Use and Health: Mental Health Findings.” NSDUH Series H-47,
HHS Publication No. (SMA) 13-4805. Rockville, MD: Substance Abuse and Mental Health Services
Administration. http://www.nimh.nih.gov/statistics/SMI_AASR.shtml
Turner, N., O’Mahony, P., Hill, M., Clarke, M., Larkin, C., Waddington, J., et al. 2012. “Employment
Outcome Over 12 Years In A First Episode Psychosis Cohort.” Paper presented at
the International Early Psychosis Association, San Francisco.
Twamley, E. W., Jeste, D. V., and Lehman, A. F. 2003. “Vocational Rehabilitation In Schizophrenia
And Other Psychotic Disorders: A Literature Review And Meta-Analysis Of Randomized Controlled
Trials.” Journal of Nervous and Mental Disease, 191, 515–523.
U.S. Public Health Service Office of the Surgeon General. 1999. Mental Health: A Report of the
Surgeon General. Rockville, MD: Department of Health and Human Services, U.S. Public Health
Service.
Watkins, K. E., Pincus, H. A., Paddock, S., Smith, B., Woodroffe, A., Farmer, C., Sorbero, M.E.,
Horvitz-Lennon, M,. Mannle, T. Jr., Hepner, K.A., Solomon, J., and Call, C. (2011). “Care For
Veterans With Mental And Substance Use Disorders: Good Performance, But Room To Improve
On Many Measures.” Health Affairs, 30, 2194–2203.
Wehman, P., and Moon, M.S. 1988. Vocational Rehabilitation And Supported Employment.
Baltimore: Brookes Publishing.
Woltmann, E. 2009. “Development And Evaluation Of A Consumer Directed Decision Support
System To Support Shared Decision Making In Community Mental Health.” Dartmouth:
23
Unpublished dissertation.
24