DRAFT – Do Not Cite Without Permission
Factors Supporting Success in Return-To-Work Programs for Persons with Severe
Disabilities: An Exploratory Analysis from Wisconsin’s SPI Project
Barry S. Delin
Stout Vocational Rehabilitation Institute
University of Wisconsin – Stout
Anne E. Reither
Utah State University
November 2006
Paper prepared for the Association of Public Policy Analysis and Management
Research Conference, Madison, WI, November 2-4, 2006. This paper builds on
research performed at the University of Wisconsin – Stout Vocational
Rehabilitation Institute and the Oregon Health and Sciences University Health
Policy Institute on behalf of the Pathways Projects, Office of Independence and
Employment, Wisconsin Department of Health and Family Services. The authors
thank the staff at the Pathways Projects for their cooperation and support. The
descriptions and interpretations in this paper are those of the authors and do not
reflect those of the Pathways Projects, the Wisconsin Department of Health and
Family Services, or of the institutions where the authors are employed.
DRAFT – Do Not Cite Without Permission
1
Introduction
From July 1999 through September 2004, the Wisconsin Department of Health
and Family Services (DHFS) operated a “return-to-work” demonstration project for
disabled Social Security beneficiaries and recipients as part of the State Partnership
Initiative. This demonstration was called Wisconsin Pathways to Independence (WPTI)
and was one of twelve State Partnership Initiative (SPI) projects funded by the Social
Security Administration (SSA) through co-operative agreements with the participating
states. Under these co-operative agreements states were given the opportunity to
develop and implement new or augmented programs intended to improve employment
outcomes for persons receiving Social Security Disability Insurance (SSDI) and/or
Supplemental Security Income (SSI). There was an expectation that the SPI
demonstration projects would also provide the states, SSA, and other federal entities
with information that could inform future program planning and policy development. 1
WPTI resulted in what can be characterized as modest, but significant, gains in
the probability of employment, in earnings, and in income for those who received the
intervention. At the time of study entry participants in the intervention had somewhat
lower employment rates and quarterly earnings than those in the comparison group. 2
Regression models of program effects showed that participants in the intervention group
exhibited a gain of 13% in their employment rate relative to the comparison group one
year after study entry. This difference remained at 12% two years following study entry.
Average gains in earnings were estimated at $216 per calendar quarter after one year,
$314 per quarter after two. Those receiving the intervention achieved an estimated
income growth of $314 per calendar quarter relative to that estimated for comparison
1
The Rehabilitation Services Administration (RSA), U.S. Department of Education also
sponsored a number of SPI projects. The RSA sponsored projects were not restricted to Social
Security beneficiaries and recipients and were not required to collect as extensive a range of data
as the SSA sponsored projects.
2
Delin, Barry S., Reither, Anne E., Drew, Julia A. and Hanes, Pamela P. 2004. Final Project
Report: Wisconsin Pathways to Independence. Menomonie, WI: University of Wisconsin – Stout
Vocational Rehabilitation Institute, pp. 93-97 and 99-101. These employment outcomes were
measured using Unemployment Insurance (UI) data from the Wisconsin Department of Workforce
Development.
WPTI did not use random assignment. The comparison group was recruited from Wisconsin
Division of Vocational Rehabilitation (DVR) consumers who appeared to meet WPTI eligibility
requirements. Additionally, the comparison group was limited to those DVR consumers who had
progressed far enough through the vocational rehabilitation process to provide behavioral
evidence of strong motivation to become employed or to increase earnings.
There were significant differences between the two groups in the distributions of some measured
baseline characteristics, including employment and earnings. We believe differences in how the
two groups were recruited and the typical timing of entry to the study explained much of the
observed differences. Comparison group members were typically well into and had sometimes
completed their latest span of DVR services. By contrast, intervention group members were
typically starting an initial or new span of DVR services and may in some cases have reduced
their work efforts in anticipation of seeking employment services. The original study utilized
analysis techniques to adjust for the baseline differences between groups.
DRAFT – Do Not Cite Without Permission
2
group members. The estimated difference per quarter was $456 two years after study
entry. 3
The WPTI evaluation emphasized average differences between the intervention
and comparison groups. This was in keeping with SSA’s first priority: learning what the
typical effects of the various SPI initiatives were over the populations defined by the
projects’ eligibility criteria and recruitment processes. 4 However, this paper does not
focus on differences between individuals in the intervention and comparison groups. The
point of departure is the fact that there was substantial variation in intervention group
outcomes. There was not a single calendar quarter in which a majority of those in the
WPTI intervention group (or, for that matter, the comparison group) had employment or
earnings reported in Wisconsin Unemployment Insurance records. This finding was not
surprising for a group of individuals who were all SSDI beneficiaries and/or SSI
recipients for reason of disability. After all, every one of these individuals had to
demonstrate that they met the Social Security definition of disability, a definition that
requires that an individual cannot work at any job in the economy that can generate
monthly earnings at or above the Substantial Gainful Activity (SGA) level.
It would seem useful to learn the sources of such variation in order to improve
return-to-work programs and policies and/or to identify those individuals more likely to
benefit from those program and policies. Indeed, under current law, any decision by a
beneficiary or recipient to seek employment is ultimately voluntary. This exploratory
study utilizes logistic regression to attempt to identify the reasons that some WPTI
participants (the term “participant” should from this point forward be understood as
referring only to those in the intervention group) had much better employment outcomes
than others.
The WPTI Intervention
Wisconsin Pathways to Independence was operated jointly by what is now the
Office of Employment and Independence (OIE) in the Department of Health and Family
Services and, until October 2003, the Division of Vocational Rehabilitation (DVR) in the
Department of Workforce Development. The project was intended to address a broad
range of issues related to service delivery and the negative work incentives embedded in
public policies that WPTI’s designers and proponents believed had negatively affected
the ability of those in the SSDI and SSI programs to become employed, remain
employed, increase their earnings or, if desired, engage in career development.
Consequently, WPTI involved features to integrate and improve service provision and to
3
Delin, Barry S., et. al. 2004. pp. xiv-xv. Also see chapter IV of that report, “Net Effects of WPTI,”
pp. 131-79. The actual observed differences between the intervention and comparison groups (as
opposed to estimated differences) were smaller for employment (8% for both years), but of
somewhat larger size for earnings and income. Income was approximated by the sum of earnings
and social security benefits, including a SSI state supplement. For example, the earnings
difference per quarter was $306 per quarter one year after study entry, $364 after two. All
monetary values were converted to constant dollars using 1996 GDP = 100.
4
SSA was initially interested in identifying ways of increasing employment and earnings for those
on SSDI and SSI in hope that some would be able to permanently exit the programs. Over time,
perhaps in part from what was learned through SPI, SSA placed increasing emphasis on having
beneficiaries and recipients reduce their dependency on SSA benefits without necessarily leaving
benefit status.
DRAFT – Do Not Cite Without Permission
3
implement, whether on a temporary or permanent basis, changes to public policy. The
intent was to create a process that would maximize the participant’s ability to overcome
whatever barriers impeded the achievement of the participant’s employment goals.
Existing deficiencies in service provision and policy were seen as having subjective as
well as objective effects; individuals might refrain from increased work effort because of
their fears, justified or not, about the negative consequences.
WPTI provided participants with a combination of “work incentive” benefits
counseling and person centered vocational planning services. Participants were enrolled
at and served by twenty, largely non-profit, community agencies. Enrollment at each
agency was restricted to consumers deemed to fit into a particular disability category.
Consequently, WPTI had two kinds of eligibility requirements. One type pertained
to overall project eligibility, the second type constrained entry to the project through
specific types of community agencies. Participants had to be in either the SSDI or SSI
program (for reason of disability), Wisconsin residents, between eighteen and sixty-four
years of age (inclusive) and eligible for DVR services. 5 WPTI designated four types of
community agencies for purposes of the project, each of which served persons with a
disabling condition that fit under one of four categories, physical disabilities, mental
health, developmental disabilities, and HIV/AIDS, that were defined based on categories
already used by DHFS. 6
If a potential participant met eligibility criteria, entry to the program was by selfselection, modified by potentially three considerations: the community agency had to
have the capacity to serve the individual, DVR had to be willing to authorize the
participant to receive WPTI services, and in some cases the participant had to meet
legal requirements for receiving services from a particular community agency. 7 The
probability that any individual would enroll in WPTI was greatly increased if that person
was a current or recent consumer at one of the community agencies and/or DVR.
Additionally, the proportion of participants at each of the four types of provider agencies
closely mirrored the availability of program slots. Thus, it isn’t surprising that the
distribution of disabilities and some socio-demographic characteristics of WPTI
participants diverged substantially from those exhibited by the comparison group.
Finally, there were two periods during the WPTI demonstration when, due to fiscal
5
Nine participants were allowed to enroll without DVR authorization. These individuals were
persons who had had AIDS or a disabling condition related to their HIV positive status and did not
wish to reveal their condition to DVR. Several sites were authorized to enroll persons sixteen and
seventeen years old who were engaged in transition activities. Only one such participant is
included in the 506 cases utilized in this paper.
6
Though a community agency was only allowed to enroll those consumers who met the WPTI
definition for these groups, many consumers met more than one of these definitions. In those
regions of the state where there were multiple WPTI associated community agencies, a
consumer potentially had some choice as to where to enroll and thus to the style of person
centered vocational planning the consumer would receive.
7
At certain agencies, entrants to WPTI had to become clients of the agency first. In some cases
this reflected agency philosophy, but generally this reflected a combination of state regulations
and limited public funding. This was particularly true in the case of the Community Support
Programs which serve persons with severe and persistent mental illness.
DRAFT – Do Not Cite Without Permission
4
shortfalls, DVR closed or limited access to its services. These Order of Selection
closures both slowed enrollment and affected the timing and level of service provision.
Benefits counseling was implemented using a single model. There were,
however, multiple models used to provide vocational planning services. Community
agencies were required to use a vocational planning approach the WPTI central office
had approved for use with consumers from the disability category each agency was
authorized to enroll. 8 The delivery of benefits counseling and vocational planning
services was supposed to be integrated through a consumer directed team process
involving the participant, community agency staff, the DVR counselor, and other persons
agreeable to the participant. Particular attention was given to conducting an in-depth
assessment of barriers that made achievement of employment goals difficult, followed by
development of a plan to identify and mobilize necessary resources to deal with
identified barriers. There was an acceptance that there could be substantial variation in
the delivery of the WPTI service approach based upon participant needs and
circumstances. In fact, this personalized approach was a key part of the intervention.
All participants, in principle, had access to the combination of benefits counseling
and vocational planning services. Additionally, it was expected that the consumer
directed team, especially the community agency staff and DVR counselor, would work to
help the participant gain or maintain access to services and supports that WPTI did not
itself provide. Although there was an expectation that most participants would receive
the largest amount of WPTI services during their first months of participation, it was also
expected that WPTI would continue to provide services and supports “as needed” until
the end of the demonstration. 9 WPTI was not intended as a means to achieve quick
placement into gainful employment. The designers believed that better long term
outcomes would be achieved through a vocational planning process that allowed
participants to identify goals and how to achieve them. So it was up to the participant
and her/his team to decide the best time to (re)enter the workforce. 10
8
The Physical Disability and HIV/AIDS agencies used the same vocational planning model, the
Vocational Futures Planning model that was originally intended to be the WPTI standard. The
Developmental Disabilities agencies were suppose to use a second model, but were in fact given
broad latitude to deliver whatever model they chose. The Mental Health sites used one of two
other models, depending on whether the agency was a Community Support Program or a
Clubhouse. Although all models were said to be team based and consumer centered, there was
strong evidence of variation in how agencies using the “same” model interpreted how that model
should be delivered. See Delin, Barry S., et al. 2004. pp. 17-23.
9
Only about 45% of benefits counseling and vocational services (excluding job coaching and
various forms of indirect support) were delivered in the first six to nine months of participation
(i.e., in the first two calendar quarters after the enrollment quarter and whatever portion of the
enrollment quarter followed the actual enrollment date). There were both programmatic (e.g. staff
attrition, DVR budget shortfalls) and participant (e.g., illness, reluctance to commit to the process)
reasons why an unexpectedly small proportion of service hours were delivered in the first months
following enrollment. See Delin, Barry S., et al. 2004. p. 110.
10
This proved to be an area of some tension between DVR staff and both DHFS and agency
staff. DVR as an agency and DVR area directors and counselors as individuals are evaluated on
the number/proportion of successful closures. In a context of insufficient funds and waiting lists,
DVR personnel had understandable concerns about the length and cost of the intervention.
DRAFT – Do Not Cite Without Permission
5
Additionally, WPTI sought and to some extent obtained changes to health care
and income support program provisions that could potentially support better employment
outcomes for WPTI participants. These policy changes were only applicable to a subset
of WPTI participants, though program designers initially expected these subsets to be
fairly large. WPTI was instrumental in getting Wisconsin to adopt a Medicaid Buy-in
program for working people with disabilities, abbreviated as MAPP. As this program was
created through statute, MAPP was available to any Wisconsin resident who met the
eligibility requirements. However, WPTI’s designers had sought MAPP in order to
provide access to public health care coverage for WPTI participants, especially SSDI
beneficiaries, who might be more willing to take their earnings “permanently” high
enough to leave SSA benefits if access to a public health care program might be
preserved. 11
SSA also approved a less severe earnings offset for SSI recipients in WPTI. This
“SSI waiver” allowed eligible participants to keep three dollars of their SSI cash benefit
for every four dollars of earnings and allowed participants to have savings not otherwise
allowable under SSI and Medicaid. 12 WPTI also asked SSA for permission to test an
offset for SSDI beneficiaries to address the severe disincentive to earnings over SGA
faced after the end of the Trial Work Period. SSA did not approve this request. 13 WPTI
also planned to seek waivers from provisions of the Foods Stamp and Section 8 Housing
programs, but quickly abandoned these efforts to concentrate on obtaining waivers from
SSA.
Ultimately, 956 individuals enrolled in WPTI. However, only 506 of these
participants were included in the outcomes evaluation completed December 2004,
largely because the excluded cases did not have the necessary two years of data
following their entry into the project that was required for inclusion in the analysis. These
506 participants constitute the sample that is used for the analyses presented in this
paper. 14
11
MAPP, as a Balanced Budget Act buy-in, is not restricted to persons under full Social Security
retirement age. Participants must meet the Social Security disability standard, though having
earnings at or above the SGA level does not preclude eligibility. While employment or
participation in a program leading to employment is required, there is no specific requirement that
participants have monetary earnings. Even though subsequent changes to federal law allowed
former SSDI beneficiaries to retain Medicare eligibility longer than the end of the Extended Period
of Eligibility, Medicaid (especially prior to Medicare Part D) provided a range of services and
support generally not available through Medicare.
12
The 1619 provision of the SSI includes a 1:2 offset and continued access to Medicaid after the
cash benefit is “zeroed out.” However, 1619 does not provide relief from Medicaid asset limits.
13
However, in 2005, Wisconsin became one of four states selected to pilot a SSDI earnings offset
in preparation for a national demonstration.
14
The 506 includes participants who enrolled through September 30, 2001. It would appear, in
theory, that this number could be augmented by those who enrolled through September 30 of the
following year. However, survey data is available only for those who enrolled by December 31,
2001; i.e. another three months. We decided that the work involved in augmenting an available
data set to gain about another twenty-five cases was not justified in the context of this exploratory
study.
DRAFT – Do Not Cite Without Permission
6
Conceptual Approach
Wisconsin Pathways to Independence was intended, above all else, to test an
approach to increasing the employment rates and earnings of working age persons
attached to a Social Security disability program. Unemployment Insurance (UI) data was
the principal source of employment and earnings data utilized to assess participant
outcomes for the WPTI evaluation. UI data are available on a calendar quarter basis.
The data set used for the WPTI evaluation was organized into a seventeen calendar
quarter time series with the calendar quarter of enrollment preceded and followed by
eight calendar quarters of data. 15
Though participants in Wisconsin Pathways to Independence achieved
significant gains in their employment rate and average earnings relative to members of
the comparison group, the stark fact remains that there was not a single calendar
quarter during the entire seventeen quarter period when their employment rate
calculated from UI records reached 50%. In the calendar quarter immediately prior to the
enrollment quarter only 35% of future WPTI participants were employed. The maximum
employment rate of 46% was reached in quarters four and five following the enrollment
quarter, followed by a fairly rapid decline to 40% at quarter eight. 16
These results were not completely unexpected. Though Social Security programs
include work incentives, SSDI beneficiaries and SSI recipients need to meet the Social
Security disability definition to retain attachment to the programs. Whatever the variation
may be in the actual application of the definition, the definition for adults remains strict:
the inability to engage in work that is compensated at the substantial gainful activity
(SGA) level due to a physical or mental impairment that has lasted or is expected to last
at least one year or to result in death. Second, as previously noted, SSDI beneficiaries
or SSI recipients face no legal compulsion to attempt paid employment as long as they
maintain their disability status. At the same time, those on SSDI and SSI always face the
possibility that work activity can serve as evidence of medical improvement. Moreover,
should they still decide to work, many persons on Social Security disability programs can
face significant declines in income or losses of important non-cash benefits, despite the
availability of work incentives. Given the time and effort it took many SSDI beneficiaries
and SSI recipients to establish their eligibility for Social Security and other public
benefits, it should not be surprising that many persons capable of and/or desiring to
perform some work do not. According to a frequently cited statistic from the (former)
15
Wisconsin UI data does not report all employment. Most notably self-employment and jobs at
out of state employers are excluded. However, in addition to this underreporting, there is a way in
which the UI data may overestimate the employment rate. The UI records do not include direct
information about how much time within the quarter an individual is employed, and therefore may
include even very brief periods of work activity.
16
The employment rate had its minimum value (32%) four quarters prior to the enrollment quarter.
Thereafter, approaching the enrollment quarter, the rate exhibited a slow rising trend, seemingly
inconsistent with the expectation that persons tend to leave work just before entering employment
programs. Though not appearing in the WPTI final report, this material has been presented at
several public events. For example, see Delin, Barry S., Reither, Anne E., and Drew, Julia A.
2003. “Employment and Earnings Trends in a Complex Program Delivery Environment.”
Presentation at the State Partnership Initiative Annual Meeting, Washington D.C., August 7-8,
2003.
DRAFT – Do Not Cite Without Permission
7
General Accounting Office, less than 1% of those participating in a Social Security
disability program ever leave the rolls to return to work. 17 Still, the proportion of those
attached to Social Security disability programs who have some reported earnings can be
much higher than implied by the GAO number. For example, the proportion of working
age Wisconsin SSI recipients performing some paid work during the period WPTI was
operating ranged between 17% and 21%. 18
We believe that one of the lessons learned, or at least reinforced, through WPTI
and the State Partnership Initiative is that there can be value to those with disabilities,
SSA and other government entities, and the wider community in increasing employment
and earnings and reducing dependency on benefits, even when that does not result in
individuals ending their attachment to a Social Security disability program. Thus, even
when a return-to-work program does not have a large impact on average employment
rates and earnings across the SSA disability program population, that program may
constitute good policy if one can both identify the attributes and conditions that make it
more likely for some to benefit and then identify and encourage participation in return-towork programs by such persons.
In our search for a method to identify the characteristics of those who had
benefited most from their WPTI participation, we found a study conducted by Chi-Fang
Wu, Maria Cancian, and Daniel R. Meyer that approached this issue in the context of
changes in earnings and employment of TANF participants. 19 While there are many
differences between the TANF and Social Security disability populations and the legal
requirements and political contexts in which the respective programs operate, there are
commonalities in that both populations can be viewed as being composed of individuals
who find it difficult to become employed, remain employed, and/or increase earnings.
Wu, et al. sought to identify TANF participants who achieved what they characterized as
better medium-term and long-term employment outcomes.
A key step in doing this was to categorize TANF participants into earnings and
employment patterns or trajectories associated with different levels of success over
17
Although this number has been used for both Social Security disability programs, its origin
appears to be as an estimate of the rate of departure from SSDI because of work. See General
Accounting Office. March 1997. Social Security Disability Programs Lag in Promoting Return to
Work. (GAO/HEHS-97-46) Washington, DC: General Accounting Office, p.1. Also, there have
been estimates of somewhat higher departure rates based upon the analyses of specific cohorts
of SSDI entrants. In the most optimistic case (a study of entrants under age 40), about 4% left the
program because of return to work. See Mashaw, Jerry L. and Reno, Virginia P., eds. 1996. The
Challenge of Disability Income Policy. Washington, DC: National Academy of Social Insurance.
pp. 109-11.
18
These data were obtained from relevant issues of the “SSI Disabled Recipients Who Work”
report and have been adjusted to reflect only recipients who were age 18-64. During the same
period the national rate was estimated as being in the 8% to 9% range.
19
Wu, Chi-Fang, Cancian, Maria, and Meyer, Daniel R. 2005. “Standing Still or Moving Up?
Evidence from Wisconsin on the Long-Term Employment and Earnings of TANF Participants”
Unpublished Manuscript presented at the Association of Public Policy Analysis and Management
Research Conference, Washington D.C., November 2005. Wu, et al. have graciously given
permission to cite this draft version of their work.
DRAFT – Do Not Cite Without Permission
8
some defined time period following TANF entry. Both earnings and employment
trajectories were defined in terms of the level of change, its directionality, and the
stability of trends; each case was assigned to a separate earnings and employment
trajectory. 20 Once participants were assigned to a trajectory, the authors were able to
compare the distribution of characteristics and experiences across these groupings. Wu,
et al. utilized both descriptive methods and multinomial logistic regression models to
analyze their data. Medium-term outcomes were defined as those occurring three years
after entering TANF; long-term outcomes were those occurring six years following entry.
As will become apparent, we borrowed heavily from the conceptual approach developed
by Wu, Cancian, and Meyer.
Like Wu, et al., we began by defining earnings and employment trajectories.
However we did so facing some limits imposed by the data available from WPTI. The
most important was the limited number of cases, 506 compared to the approximately
17,000 available for the TANF study. Consequently, we defined substantially fewer
trajectories for each outcome variable and then defined dichotomous categories of
relative success for each variable. We present evidence below that supports the
appropriateness of these choices.
Also, we had data for a period of time that Wu, et al. would characterize as
medium-term. The final calendar quarter of data available was for the eighth quarter
following the calendar quarter of WPTI enrollment. As we had access to Unemployment
Insurance earnings and employment data for a period prior to enrollment, we included
the quarter previous to the enrollment quarter so that the initial data period does not
include any delivery of WPTI services. 21 Thus, the data period for these outcome
variables is ten calendar quarters or two and one-half years. We also constructed an
alternative employment “success” indicator variable using participant reported data
collected on a monthly basis by staff at the community agencies. These data were
available only from the enrollment quarter forward, limiting the analysis period to nine
quarters. 22 These limited data spans could have an important consequence. Wu, et al.
observed that in their study population a desirable medium-term trajectory did not
consistently lead to good long-term outcomes. Given the cyclical nature of some
20
See Wu, et al. 2005. pp. 5-7.
21
We did not use UI data from before Q-1 as we are only interested in changes in outcomes
following WPTI enrollment. Additionally, we would not have encounter and survey data for that
earlier period.
22
Depending on the participant’s enrollment date, the period of WPTI participation can vary from
a minimum of twenty-four full months to a maximum of twenty-seven. Additionally, we had
originally hoped to be able to analyze outcomes over a much longer period for a significant
minority of our cases. Wisconsin is one of four states currently piloting an earnings offset for the
SSDI program. Program planners anticipated that approximately half of the 800 participants
would be former WPTI participants, as many of the same community agencies would be involved
in enrollment and the provision of benefits counseling to pilot participants. The former WPTI
participants are asked to sign an additional release allowing researchers to use data collected
during WPTI to the new data collected for the pilot project. Unfortunately, only 5% of pilot
participants (through August 2006) were former WPTI participants.
DRAFT – Do Not Cite Without Permission
9
disabilities, the progressive nature of others, and the time limited nature of some Social
Security work incentives, this is an important caution. 23
Outcome Measures: What Constitutes Success?
Three dichotomous outcome measures were constructed to summarize
participant earnings and employment outcomes for each of the 506 participants. Data
from Wisconsin unemployment Insurance (UI) records were used to construct indicators
of relative success in the areas of earnings and employment. Data from monthly
participant update forms were used to construct an alternative indicator of employment
success which would include employment not captured in the UI system. 24 The UI based
indicators utilize ten calendar quarters of data, the alternative form-based employment
indicator, for reasons identified above, utilizes nine quarters of data.
Each indicator was constructed from two types of underlying trajectories, one in
the area of growth, the other attempting to capture persistence in achieving what was
defined as a “good” outcome. Both growth and persistence were assessed by comparing
a base year to an outcome year. 25 While this approach has the undesirable property of
ignoring quarter to quarter variation in any individual’s outcomes, we think accepting this
limitation has the virtue of being consistent with the fact that many people with
disabilities will leave work for relatively short periods for reasons not under their direct
control such as variations in the symptomatic severity of their disabling condition or the
loss of important public program or natural supports.
We sought to define “success” in ways that would differentiate participants
without a large number near the boundaries between the categories. We also sought to
define success at levels that would insure enough cases in the “more successful” group
to allow analysis. Given the limited number of cases we chose to forgo analysis of what
might be characterized as the most successful participants of all (for example, those with
earnings more than twice the poverty level). In the case of the UI earnings indicator,
growth was defined as an increase of more than the mean increase between the base
and the outcome year ($949 in 1996 GDP dollars). 26 Persistence was defined as
23
Wu, et al. 2005. p. 12.
24
However, the employment data reported from the community agencies contains some
inaccuracies. That data did not always include jobs reported in the UI system. See Delin, et al.
2004. p.141.
25
For both UI based indicators the base year is Q-1 through Q2 (where Q0 designates the
enrollment quarter). For the Forms Employment Indicator the base year is Q0 through Q3. We
also constructed UI indicators using Q0 through Q3 as the base year. Their distributions proved
almost identical to those for the Q-1 through Q2 base years. We did some preliminary modeling
with these variables and found they performed similarly to the Q-1 through Q2 formulations. The
more serious problem is the relatively short duration between the base and reference years which
compromises our claim to be measuring medium-term, as opposed to short-term change.
26
The median change in UI earnings between the base and outcome years was $0 reflecting the
fact that a large number of participants had no UI earnings reported in either the base or outcome
year. The median value was nearly one quarter standard deviation below the mean value. There
is a significant tail representing cases where there was a decrease in earnings.
DRAFT – Do Not Cite Without Permission
10
maintaining earnings above $2500 per year in both the base and outcome year. This
seemingly arbitrary figure roughly captures the annualized trigger amount for a SSDI
Trial Work Period month when Pathways began in 1999. 27 There was a moderate
tendency for a participant who was identified as “more successful” in earnings
persistence to also be identified as being in the higher earnings growth category. 28
For both the UI and update form based indicators, growth was defined as moving
from having either zero, one, or two quarters in which employment was reported during
the base year, to having employment reported in either three or four quarters in the
outcome year. 29 Success in persistence meant having either three or four quarters in
which employment was reported in both the base and outcome years. 30
Table 1 shows that the success indicators meet the design criteria of capturing a
reasonably high number of the 506 cases in each of the two categories. Do they
adequately differentiate relatively successful cases from less successful ones?
Table 1: Earnings and Employment Success Indicators
(Numbers and Percentage per Category)
More Successful
Less Successful
UI Earnings Success
236 (46.6%)
270 (53.4%)
Indicator (UI$I)
UI Employment Success
174 (34.4%)
332 (65.6%)
Indicator (UIEI)
Form Employment Success
226 (44.7%)
280 (55.3%)
Indicator (FEI)
The employment and earnings trends displayed, respectively, in Figures 1 and 2
are consistent with an interpretation that the “more successful” and “less successful”
values of each dichotomous indicator variable represent groups with substantially
27
SSDI beneficiaries, including those with concurrent SSI eligibility are allowed to test their ability
to work during a nine month Trial Work Period (TWP). During the TWP, the beneficiary retains all
of their cash benefit. The months do not need to be consecutive, but months where earnings are
above a certain monthly amount count as one of the nine months. During the thirty-six month
Extended Period of Eligibility that follows completion of the TWP, beneficiaries who retain their
eligibility following a medical review and earn at or above SGA in any month will lose their cash
benefit for that month. Those who have earnings less than SGA will receive their full SSDI cash
benefit.
28
About 85% of those in the higher growth category were also in the higher persistence category.
The reciprocal association was much weaker. Only about 55% of those classified into the higher
persistence category were also classified into the higher growth category.
29
Because of the manner in which UI Employment is reported, it is impossible to know what time
period an individual was actually employed in a calendar quarter. It may have been the full three
months; it may have been a single day. We used the same convention for the participant reported
information, although these data could, in principle, be used to calculate the proportion of time in
a calendar quarter an individual actually was employed.
30
For the employment indictors, the persistence and growth criteria are, by definition, logically
exclusive.
DRAFT – Do Not Cite Without Permission
11
different and diverging outcomes following entry to WPTI. Starting with Figure 1, during
the enrollment quarter (Q0) those in the “more successful” earnings group already
average $1073 more in earnings per quarter than the “less successful” earnings group.
This difference increased to $1710 by Q8. Most of the increased difference resulted from
the upward movement of earnings for the “more successful” group by $630 (57%)
relative to Q0. By contrast, the “less successful” groups average earnings level at Q8
was actually slightly lower ($8) at Q8 than at Q0. 31
Figure 1: Mean UI Earnings Q-4 through Q8 by UI Earnings Success Indicator (UI$I)
UI Earnings per Quarter
$2,000
$1,800
$1,600
UI Earnings in 1996 $
$1,400
$1,200
Less Successful
More Successful
$1,000
$800
$600
$400
$200
$0
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Participation Quarter, 0 = Entry to WPTI
31
Though there was a small decline in average earnings for the “more successful” group over the
final quarters of the data series, there was quite a large one, in percentage terms (55%), for the
“less successful” group after they reached their maximum in Q3. The decline was small ($37) in
absolute terms as average earnings for members of this group peaked at $67 per quarter.
DRAFT – Do Not Cite Without Permission
12
Figure 2: Mean UI Employment Rate Q-4 through Q8
by UI Employment Success Indicator (UI$I)
UI Employment Rate
100.0%
90.0%
80.0%
Percent UI
Employment
70.0%
60.0%
Less Successful
More Successful
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Participation Quarter, 0 = Entry to WPTI
Figure 2 displays the UI employment rate trends for the “more successful” and
“less successful” groups for the UI Employment Success Indicator (UIEI). The general
trends seen in Figure 1 are again present. There is a substantial difference in the Q0
employment rate of about 50 percentage points. This difference grows to 75 percentage
points by Q8. Not unexpectedly, given the usually strong relationship between average
earnings and employment, the quarter to quarter post enrollment trends for both groups
are similar to those observed for earnings. The employment rate for the “less successful”
is never very high, but the maximum (23%) is reached in Q4, followed by some decline.
For the “more successful” group, the UI employment rate continues to rise through Q7,
with some decline in the last quarter. We have not included a graph for the Form
Employment Success Indicator (FEI) as those data exhibit the same basic pattern as
that presented for the UIEI indicator in Figure 2. The main difference is the slightly higher
employment rates observed in most quarters.
Additionally, the information in both Figures 1 and 2 indicate that substantial
divergence was present between the “more successful” and “less successful” groups for
both the UI Earnings Success (UI$I) and UI Employment Success (UIEI) indicators in the
quarters (-4 through -1) approaching that observed in the initial quarter of WPTI
enrollment. 32 The trend lines for this period also show that the pattern of increasing
divergence in earnings and employment outcomes was clearly present about a year
before project entry. This is an important finding as it suggests to us that the WPTI
32
The pre-enrollment trend lines for the ”less successful” participants, especially for UI earnings,
are consistent with the expectation of reduced work effort prior to entry to employment and
training programs (the “Ashenfelter” dip). In marked contrast, the participants classified into the
“more successful” groups are exhibiting strong trends toward increasing their work effort.
Unfortunately we do not have attitudinal data for this period. Additionally, as will be reported later
in this paper, we cannot distinguish the “more successful” from the “less successful” based on
attitudinal data collected at WPTI enrollment, at least as structured in the data set we used.
DRAFT – Do Not Cite Without Permission
13
intervention or any of its particular features is not likely to be the primary driver of the
increasing difference between the “more successful” and “less successful” groups.
Saying this, however, does not mean that WPTI did not help more successful
participants to exploit whatever advantages they possessed when entering the project.
Nonetheless, as there is no apparent increase in the rate of employment and
earnings growth for those in the “more successful” groups in the quarters following
enrollment, it is reasonable to ask why we think WPTI may have contributed to continued
growth in the mean values observed in Figures 1 and 2. We offer three arguments.
First, the employment rates and average earnings levels reach values far above
what is normally observed for those participating in Social Security disability programs.
While by itself this proves nothing, the finding suggests that something unusual has
happened. Second, our models include variables that measure the impact of various
features of the intervention. Should at least some of these variables have a strong and
positive impact on outcomes, there would be reason to think the intervention contributed
to the observed outcomes. This proved to be the case. Third, participants, through focus
groups and follow-up surveys, suggested that the Pathways experience helped them in
multiple ways to achieve better employment outcomes, including ongoing availability of
useful information from a trusted source, continuing personal/social support, and skill
development. A number of WPTI participants pointedly contrasted the usefulness of
having an available source of assistance and support over the long haul, in contrast to
their experience with other programs that had only offered a short-term fix (for example,
job placement services). 33
We close this section by looking at whether those who were classified as “most
successful” for one indicator would be so classified for the others. A quick look at Table
1 will confirm that the sets of the “most successful” are not perfectly equivalent.
However, cross-tabulations between the indicators confirm substantial overlap, though
least strongly between the UI earnings and forms employment indicators. 34 This is
probably a result of the additional jobs identified through the update form (predominately
self-employment) not being reported to the UI system. Spearman correlations also
indicate substantial, but far from complete, overlap between the “most successful”
groups. The correlation between the two UI based indicators was strongest at .691. The
weakest association among the three pairs of variables was .459 between the UI$I and
FEI indicators. Thus, with appropriate caution, we believe that it is reasonable to talk
about participants as having, in a general way, “more successful” and “less successful”
overall outcomes. Nonetheless, we will report findings only for the UI$I, UIEI, and FEI
variables.
33
34
See Delin, Barry S., et al. 2004. pp.123-25.
72% of the “more successful” using the FEI indicator were in the “more successful” group for
the UI$I indicator. 69% of those classified as “more successful” on the UI$I variable were also
classified as among the “more successful” on the FEI variable.
DRAFT – Do Not Cite Without Permission
14
Independent and Control Variables
We utilized the same data set as was used for the WPTI outcomes evaluation. 35
This data base includes a wide range of administrative, encounter, and survey data.
Although we had access to all of the data collected for the WPTI evaluation, given the
exploratory purpose of this paper we limited this study to those variables included in the
data set for the WPTI Final Project Report or, in several cases, to recoded or
transformed versions of those variables.
We planned to utilize independent and control variables in seven categories: (1)
variables capturing some aspect of the WPTI service model or its provision, (2) variables
capturing use of policy alterations created by or associated with WPTI, (3) sociodemographic variables, (4) benefit program participation variables, (5) work experience
variables, (6) variables describing disability type and severity, and (7) participant
attitudes and perceptions.
We did not use all available variables. We began by selecting variables that had
been found to have a significant impact on WPTI participant earnings and employment
outcomes relative to those for the comparison group. We also, with one exception
discussed in footnote #41, included variables that had a significant effect on outcomes
among WPTI subgroups that had been examined for the WPTI Final Project Report. We
then examined the bivariate relationships between these variables and the dependent
variables with the intention of excluding variables when the appropriate measure(s) of
association did not meet a significance level of .2. In some cases, we retained variables
that did not meet this standard, either because they measured some aspect of the
intervention we thought critical or, as in the case of certain socio-demographic variables,
we thought exclusion would have been questioned.
What follows is a listing, by area, of the seventeen independent and control
variables used in our regression models. In a few cases we include some discussion of
these choices or of those not made. Fuller descriptions of these variables and of
associated frequencies and/or basic statistics can be found in Appendices A through C.
WPTI service provision: 36
Hours of benefits counseling provided (variable name = BCtoQ5)
Hours of vocational services provided (VStoQ5)
35
Data elements included both those required by the SPI Project Office at Virginia
Commonwealth University and others specific to the WPTI evaluation and/or that of the closely
related 3-State Work Incentives Initiative. Pamela Hanes had the lead role in designing both the
WPTI and 3-State evaluations. Among the many individuals with some role in putting together the
WPTI data set, we wish to give special recognition to Janne Boone and Julia Drew. We also wish
to thank David Sage for his contributions to both the WPTI evaluation and this study.
36
The WPTI data set we used includes hours of service delivery in two formats: Q0 through Q2
and Q0 through Q5. We chose to use the longer duration because of WPTI’s emphasis on getting
follow-up services at need and the fact that many participants had to wait before receiving
services (especially benefits counseling) because of staff attrition or DVR budgetary problems.
When we looked at the strength and significance of measures of association between the service
provision and dependent variables, the bivariate relationships tended to be stronger with the Q0
through Q5 versions of the service provision variables.
DRAFT – Do Not Cite Without Permission
15
Hours of job coaching services provided (JCtoQ5)
Implementation Quality (IMPQUAL)
WPTI policy alterations:
SSI Waiver (SSIWAV)
Medicaid Buy-in (MAPP)
Socio-demographic:
Age (AGE)
Sex (SEX)
Race (RACE2)
Education (EDUCrev)
Benefit program participation: 37
SSDI/SSI status (SSACat) 38
37
With the exception of the SSDI/SSI status variable, we view the program participation variables
as control variables. The SSA Benefit Amount and Food Stamps participation, both at Q0, are
intended to control for the potential individual level tradeoffs each new WPTI participant faced as
a consequence of entering a project that had as its purpose increasing that individual’s work
effort. For example, consider the different situations (all else being equal) faced by a SSI only
recipient and a SSDI only beneficiary receiving a benefits check well over SGA. The SSI
recipient’s check will be significantly less than SGA and will only be subject to a $1 reduction for
each $2 earned, irrespective of whether or not earnings rise above SGA. By contrast for the SSDI
beneficiary any earnings over SGA, after completion of the Trial Work Period, result in the full
loss of the benefit check. To at least break even, the beneficiary must earn her/his former benefit
amount plus the SGA amount. In the real world, these calculations are further complicated by loss
of “leisure” time (no trivial consideration for many with a severe disability) and potential loss of
other public benefits.
More WPTI participants utilized Food Stamps than any other non-health care related public
program; participation at Q0 is used as a proxy for whether the participant faced potential loss of
non-Social Security benefits from increased work activity. The third of these variables, income
change between the base and outcome year, is intended to control for the actual trade off
between earnings and the Social Security benefit amount.
The SSA Benefit Amount at Q0 and income change between the base and outcome years are
monetary amounts. These variables were standardized because of their large scale relative to the
values of other variables used in the regression models. Additionally, the underlying measure of
income change is a proxy rather than an actual value. It includes UI earnings and the income
received from Social Security and the Wisconsin State SSI Supplement and thus excludes other
possible sources of income. Moreover, the proxy excludes income received by any other member
of a participant’s household.
38
An individual may participate in both the SSI and SSDI programs at the same time, most often
because that individual’s SSDI benefit is less than the SSI FBR amount. We chose to use a
variable that classified SSDI only and concurrent SSI/SSDI cases in the same category, all SSI
only cases in a second. This reflects our judgment that the overall impact of SSDI policies on
earnings is stronger than that of SSI policies. When a concurrent beneficiary works, the size of
their benefit check reflects the application of both SSDI and SSI rules. In most cases, the impact
of the SSDI rules is more powerful. In particular, concurrent beneficiaries face the same
constraints on earnings above SGA following completion of the Trial Work Period as a SSDI only
beneficiary. Any earnings over SGA result in a 100% loss of the SSDI portion of a concurrent
beneficiary’s Social Security check.
DRAFT – Do Not Cite Without Permission
16
Food Stamps (FSQ0)
SSA Benefit Amount at Q0 (Q0BS$_sdz)
Income change between the base and outcome years (Chg_sdz) 39
Work experience:
Post Disability Employment Ratio (EMPRAT) 40
Disability type and severity:
Primary Disability (PRIMDIS) 41
DVR Order of Selection Category (OOSrev) 42
To our disappointment, we did not include any measures of participant attitudes
or perceptions at program entry in our modeling. We had expected these variables to
explain at least some of the difference in employment outcomes, especially the
differences in earnings and employment rates that were exhibited even before the
quarter of project entry. To our surprise, we saw almost no evidence that participants
classified into a “more successful” group (for any of the three indicator variables) differed
from participants classified into a “less successful” group on any of the
39
In the WPTI final report “income” was treated as a dependent variable. In this study, we treat it
as a control variable, reasoning that the severity of the tradeoff between earnings and cash
benefits might affect participant decisions about work. Policy features of the WPTI intervention
such as the SSI Waiver and MAPP were intended to decrease the severity of the tradeoff.
Benefits counseling was intended to help participants to make informed choices, including
whether and to what degree to use these policy features and other available work incentives.
40
“Post-disability,” in this context, begins at the eligibility date for Social Security benefits, not the
actual date that the impairing condition started. The EMPRAT variable is a proxy for the actual
ratio of time employed between entering a Social Security benefit program and entry into WPTI.
41
Assignment was to one of the three disability groups was based on the participant’s RSA-911
code in their DVR record. Assignment was according to a crosswalk developed by Pamela Hanes
and her staff at Oregon Health and Sciences University and modified by the Wisconsin based
WPTI evaluation staff.
There was very considerable overlap between these disability categories and the type of WPTI
agency at which a participant enrolled. This was largely a product of the secondary WPTI
eligibility criteria that allowed each of the four agency types (Physical Disability, Mental Health,
Developmental Disabilities, and HIV/AIDS) to enroll participants who fit within one of these
categories as defined by their contracts with DHFS. As both variables more or less measure the
same thing, we included one. We chose to use the one that was the more direct indicator of a
disabling condition as we are interested in the impact of attributes that participants brought with
them to WPTI.
However, this choice has a cost. There were very large differences in the average employment
rates and earnings across the four agency types, differences that on the basis of interviews and
focus groups conducted during WPTI appear to be related to agencies’ missions and service
philosophies. See Delin, Barry S., et al. 2004. pp. 134-40 for differences in outcomes by WPTI
agency type, pp. 17-23 for differences in agency service philosophy by WPTI agency type.
42
The DVR OOS code is used as a proxy for severity, though one that reflects that agency’s
concern with assessing how difficult it will be to prepare an individual to reenter (or initially enter)
the labor force.
DRAFT – Do Not Cite Without Permission
17
attitudinal/perceptual measures, at least at the time of entry to WPTI. Our data set
included measures of work motivation, perceived barriers to work, self-esteem, mastery,
and subjective physical health (PCS12) and mental health (MCS12) scales from the
SF36 assessment instrument. Only one combination of the attitudinal and dependent
variables had an ordinal by ordinal measure of association lower than .3; most values
were .7 or higher. The single exception was the association measure between UI$IP and
the MCS12 score which just met the .2 significance level we specified for including
variables in our regression models. However, we did not include this variable in the
modeling because there were sixty-three missing cases. 43
Analysis Methods
This study uses binary logistic regression to identify the variables which seem to
most strongly account for the distribution of WPTI participants into the “more successful”
and “less successful” categories for each of the three dependent variables (UI$I, UIEI),
and FEI). The analysis seeks to account for outcomes over either a thirty month (for the
UI data based indicators) or a twenty-seven month period (for FEI), though the analysis
is not set up as a time series. Only a maximum of 483 of the 506 cases are available
due to missing data for two of the seventeen independent variables. 44
Given the exploratory purpose of this study, we decided to use forward selection
models utilizing the 483 cases. We also ran models after removal of outliers at or above
2.58 standardized residual level. When there are no major differences between the
original models and those with the outliers removed, we will present the original
models. 45 The standard .05 significance level is criterion used for rejecting null
hypotheses.
43
These 63 cases represent about 12% of the 506 cases. Indeed, the use of any of the attitudinal
variables would have reduced our N by a minimum of 38 cases. Nonetheless, we would have
used at least some of these variables in our models if there had been any indication that they
might have a meaningful effect.
44
Given the exploratory nature of this study, we decided to keep the modeling as simple as
possible. As noted earlier we were willing to ignore quarter to quarter variation in this initial
analysis, because it looked at a relatively short period for a population unusually subject,
especially because of variation in health and the availability of support services, to variability in
their work effort. We were also concerned about the technical difficulties of modeling when one of
the two analytical groups had majorities with no employment or earnings in every calendar
quarter.
Two independent variables had missing cases. There were 21 missing values for OOSrev and 5
for EDUCrev. As 3 cases had missing information for both these variables, the actual number of
excluded cases was 23, 4.5% of the original 506.
45
Models were run using SPSS for Windows V. 13 using the LR option to calculate the models.
We used the default values offered by SPSS, including those for inclusion (.05) and exclusion
(.10) of variables at each step.
We also ran models using backward selection to check for consistency between inclusion of
variables and their relative strength between the forward and backward selection models. While
there were two cases among five general models run where the backward selection models
included variables not chosen in forward selection, these variables were weak and either of
borderline significance or non-significant.
DRAFT – Do Not Cite Without Permission
18
In modeling the UI Earnings Success Indicator (UI$I) we used all seventeen
independent variables. For the two employment indictors, (UIEI and FEI) we did not
include the variable measuring hours of job coaching in the models because that
variable (JCtoQ5) measures service provision that, in principle, should have occurred
only after a participant became employed. We faced a more difficult choice about
whether to include the Medicaid Buy-in participation variable (MAPP) in the employment
models.
As a Balanced Budget Act authorized Buy-in, the Wisconsin program (MAPP)
requires employment as a condition of participation. The one exception to this
requirement is participation on a time limited basis in an approved employment
preparation program (HEC). At any time, only a handful of MAPP participants are using
HEC. As such, it would seem reasonable to exclude MAPP from the employment
models.
However, MAPP does not require any monetary earnings or minimum hours of
work as a condition of either initial or continuing eligibility. Employment that provides inkind compensation is acceptable. So, apparently, is any number of work hours above
zero, provided some work is performed every month. Though such individuals are
eligible for the Wisconsin Buy-in, they do not meet the definition of employment used in
WPTI or the other SPI projects. 46 Moreover, the fact that a significant proportion of
MAPP participants report either no earnings (12%) or no more than $67 per month
(50%) suggests that there is a difference between MAPP participation in itself and the
attainment of the relatively high employment outcomes that WPTI and, ultimately, the
Wisconsin legislature and U.S. Congress, hoped to motivate. 47 Given the ambiguity
concerning whether MAPP should be included in or excluded from the employment
models, we decided to run employment models both ways. To foreshadow our findings,
MAPP participation proves to be an extremely strong factor in determining whether
participants are in the “most successful” group in all models that include the MAPP
variable.
Finally, our models include three categorical variables, primary disability
(PRIMDIS), educational attainment (EDUCrev) and WPTI implementation quality
(IMPQUAL) that have more than two values. The reference category is “physical
disability” for PRIMDIS, “more than high school” for EDUCrev, and “highest” for
IMPQUAL.
46
Employment in all SPI projects, including WPTI, was defined as requiring monetary
compensation and being continuous, rather than episodic, in nature. The SPI definition did not
identify minima, but did include some examples of situations (e.g., receiving payment for
occasional odd jobs outside of contracting such work on a continuing basis) that did not meet the
definition.
47
APS Healthcare, Inc. 2005. Medicaid Purchase Plan Evaluation Annual Report (for 2004).
Madison WI: APS Healthcare, Inc. pp. 11-12. The values cited are based on information collected
by county economic support workers for October 2004.
DRAFT – Do Not Cite Without Permission
19
Logistic Model Results
Tables 2 through 6 provide results for the regression models we tested that are
the “best” representations of the relationships between our indicator variables. We have
placed “best” in quotation marks to signal that as a group these models, especially those
using the encounter data (FEI), are not particularly strong as explanations of what
distinguishes the “more successful” from the “less successful” for each of our
dichotomous outcomes. Nonetheless, we argue that the results can provide some useful
information to policymakers and stakeholders as to whom might be targeted for
participation in return to work programs and, at least in the area of policy modifications,
what might support the employment efforts of persons with disabilities who participate in
the Social Security income support programs.
Our discussion of Tables 2 through 6 focuses on the final two columns in each
table. The significance (Sig.) value indicates whether a variable has been correctly
included in the model and that the effect size reported for any independent variable is
credible. As previously noted, our criterion is a value no larger than .05; our discussion,
therefore, concentrates on variables that meet this standard. Exp(B) represents the
“odds ratio.” When the dependent variables are dichotomous, as they are in this study,
an Exp(B) of “X” can be understood as a change in the odds any case will have one
value of the dependent variable rather than the other. For example, in Table 2, the
Exp(B) value of 3.486 indicates, all else held constant, that a person signing up for the
SSI waiver had about 3.5 times the odds of being in the “more successful” earnings
group than in the “less successful” one. 48 We will use the Exp(B) to communicate the
direction and relative strength of a variable relative to others in a model.
Table 2 presents information about the UI Earnings Success Indicator (UI$I)
model. The UI$I model is statistically significant and, according to the Hosmer and
Lemeshow test, exhibits adequate “goodness of fit.” 49 The model appears to explain a
usefully large amount of variance (Nagelkerke R Square = .341).
The data in Table 2 indicate that WPTI’s policy features have strong and positive
effects, resulting in odds of about 3.5 to1 that participants using either the SSI waiver or
MAPP had earnings outcomes that placed them in the “more successful” category. By
contrast, most WPTI service variables are excluded from the model. The one exception,
the hours of job coaching provided, appears to have a very modest positive effect.
48
When the dependent variable has two values, an Exp(B) of 1 indicates no difference in the
odds of being in one group rather than the other. Exp(B) values greater than 1 indicate a positive
relationship between the independent and dependent variables, values less than 1 an inverse
relationship. The further the Exp(B) value is (as a ratio) in either direction from 1, the greater the
independent variable’s impact. Additionally, when a categorical variable has more than two
values, the Exp(B)s are interpreted in relation to a reference category.
49
However, the classification table for this and our other models do not indicate that the models
are especially good at correctly predicting the assignment of cases to their actual dependent
variable category. The correct classification rate for the UI$I model is typical with a value of about
77%.
DRAFT – Do Not Cite Without Permission
20
Table 2: Model Results for UI Earnings Success Indicator (UI$I). N=483.
Variable
B
S.E.
Wald
Sig.
Exp(B)
PRIMDIS
-.134
.304
.195
.659
.874
(cognitive/DD)
PRIMDIS
.729
.271
7.264
.007
2.073
(affective/MH)
OOSrev
.477
.216
4.852
.028
1.611
SSACat
-969
.282
11.812
.001
.380
SSIWAV
1.249
.257
23.615
.000
3.486
MAPP
1.260
.307
16.872
.000
3.524
EMPRAT
.670
.231
8.410
.004
1.954
JCtoQ5
.035
.015
5.421
.020
1.036
Q0BS$_sdz
-.302
.126
5.733
.017
.739
FSQ0
-.715
.271
6.953
.008
.489
Chg_sdz
.532
.127
17.524
.000
1.701
Constant
-6.03
.514
1.374
.241
.547
Variables not included by forward selection: AGE, SEX, RACE2, EDUCrev, IMPQUAL,
BCtoQ5, and VStoQ5.
Other variables that appear to have a strong positive effect on motivating
inclusion in the “more successful” earnings group include working a larger proportion of
time since becoming qualified for a Social Security program (EMPRAT, Exp(B) = 1.954),
having a less severe disability (OOSrev, , Exp(B) = 1.611), and having a primary
disability (PRIMDIS) classified as “affective/mental” (Exp(B) = 2.073) The first two
findings are consistent with expectations, the positive result for “affective/mental”
conditions compared to the reference category of physical disabilities is perhaps more
controversial. At least in the past, those with a mental illness (as a primary disability)
were somewhat more likely than those with a physical disability to participate in SSI
rather than SSDI. Those in SSI were thought to have less earnings capacity because of
their more limited work experience. Indeed, the Exp(B) value for SSACat (.380) indicates
a strong, but inverse relationship between being a SSI recipient and the odds of being
included in the “more successful group.”
All three of the benefits related control variables are included in the model and
have Exp(B) values consistent with our suppositions. Not unexpectedly, increases in the
income change variable (Chg_sdz, Exp(B) = 1.701) motivate inclusion into the “more
successful” group. This is good news as it suggests that earnings gains more than offset
losses in income from Social Security and the SSI State Supplement. The Exp(B) values
well below 1 for the Social Security/state supplement amount (Q0BS$_sdz, Exp(B) =
.739) and food stamps participation (FSQ0, Exp(B) = .489) are in concert with our
expectation that those with either higher levels of income support or relying on non-SSA
benefits programs would be less likely to achieve earnings outcomes that would place
them in the “more successful” category.
Finally, we were surprised that none of the socio-demographic variables were
selected in the UI$I model. Most notable is the absence of educational attainment
DRAFT – Do Not Cite Without Permission
21
(EDUCrev) given the strong and apparently strengthening relationship between
educational attainment and earnings. 50
Tables 3 and 4 present results for the UI Employment Success Indicator (UIEI)
models from which outliers were removed. For reasons discussed in the previous
section of this paper, we are presenting models both including and excluding Medicaid
Buy-in (MAPP) participation.
Table 3: Model Results for UI Employment Success Indicator (UIEI) with MAPP.
Outliers Removed, N=472.
Variable
B
S.E.
Wald
Sig.
Exp(B)
PRIMDIS
1.064
.309
11.842
.001
2.899
(cognitive/DD)
PRIMDIS
1.086
.284
14.588
.000
2.963
(affective/MH)
SSACat
-.694
.290
5.719
.017
.499
SSIWAV
1.228
.275
19.892
.000
3.416
MAPP
2.006
.327
37.582
.000
7.436
EMPRAT
.710
.261
7.410
.006
2.003
Chg_sdz
1.215
.176
47.792
.000
3.370
Constant
-1.606
.430
13.943
.000
.201
Variables not included by forward selection: AGE, SEX, RACE2, EDUCrev, OOSrev,
IMPQUAL, BCtoQ5, VStoQ5, Q0BS$_sdz, and FSQ0.
The results for the model including MAPP are found in Table 3. This model
exhibited adequate goodness of fit and had a Nagelkerke R Square of .391. Though
fewer variables were selected than for the UI employment model, many of the same
independent variables exhibit strong and positive relationships with the dependent
variable.
The policy components of the WPTI intervention have, if anything, an even
stronger effect than they have in the UI earnings model. The Exp(B) value for SSI
Waiver use (SSIWAV) is over 3.4, for MAPP over 7.4. Intervention service variables are
again conspicuous by their absence, as are the socio-demographic variables.
Once again, having a strong employment history after entry into SSDI or SSI
(EMPRAT) strongly increases the odds a participant is in the “more successful” category
(Exp(B) = 2.003). Participants who were SSI only again have reduced odds of being in
the “more successful” category. This negative impact (Exp(B) = .499) is not quite as
strong as for the UI earnings model (Exp(B) = .380). Two of the benefits related control
variables are absent from this model, but the remaining one, the income change variable
(Chg_sdz), has about twice the impact on the odds of being in the “more successful”
group than it did in the UI earnings model.
50
See Hanushek, Eric A. 1996. “Outcomes, Costs, and Incentives in Schools” in eds. Hanushek,
Eric A. and Jorgenson, Dale W. Improving America’s Schools: The Role of Incentives.
Washington DC: National Academy Press. pp. 31-33.
DRAFT – Do Not Cite Without Permission
22
Perhaps the most salient difference between the UI Earnings Success model and
this employment model (as well as the other employment models) is that inclusion in the
“cognitive/developmental” category of PRIMDIS strongly increases the odds of being in
the “more successful” category. The Exp(B) value is about 2.9. This finding was not
expected for much the same reasons discussed in regard to the surprising strength of
the “affective/mental” category in our discussion of Table 2 findings. Indeed, our
expectation is that those with cognitive/developmental disabilities would tend to have
even less work experience and education in comparison to those with physical
disabilities. 51 Finally, inclusion in the “affective/mental” category again strongly increases
the odds of inclusion among those exhibiting more successful outcomes.
Turning to Table 4, which shows results for the UIEI model run without MAPP,
we find a pattern of results generally similar to those observed for the UIEI model with
MAPP. This model had a slightly lower Nagelkerke R Square of .343.
Table 4: Model Results for UI Employment Success Indicator (UIEI) without MAPP.
Outliers Removed, N=472.
Variable
B
S.E.
Wald
Sig.
Exp(B)
PRIMDIS
.653
.304
4.611
.032
1.922
(cognitive/DD)
PRIMDIS
.975
.281
12.082
.001
2.652
(affective/MH)
RACE2
.972
.321
9.176
.002
2.643
SSACat
-.944
.307
9.491
.002
.389
SSIWAV
.817
.264
9.596
.002
2.264
EMPRAT
.803
.254
9.970
.002
2.232
BCtoQ5
.008
.004
4.802
.028
1.008
Q0BS$_sdz
-.333
.143
5.418
.020
.717
Chg_sdz
1.048
.157
44.635
.000
2.853
Constant
-2.759
.782
12.445
.000
.063
Variables not included by forward selection: AGE, SEX, EDUCrev, OOSrev, IMPQUAL,
BCtoQ5, VStoQ5, and FSQ0.
The most notable addition to this model is the dichotomous race variable
(RACE2), the first time that a socio-demographic variable appears in one of the models.
The Exp(B) value indicates that participants who identified themselves as “white” had
about 2.5 the odds of being in the “more successful” UIEI employment group than
someone who had self-identified as having a different racial background. 52 A WPTI
intervention service component (benefits counseling hours) also enters the model, but
has essentially no effect on outcomes.
The post-disability work history variable (EMPRAT), both primary disability
categories (PRIMDIS = affective/MH or cognitive/DD), SSA Waiver use (SSIWAV), and
the change of income variable (Chg_sdz) all remain strongly associated with having a
51
52
See Table 10 for information about educational attainment.
More than three-fifths of the “other” group identified themselves as “black.” The next largest
group was composed of those who saw themselves as having multiple racial identities.
DRAFT – Do Not Cite Without Permission
23
positive outcome. Though the effect of the “cognitive/developmental” category of
PRIMDIS is noticeably weaker than in the UI employment (UIEI) model that included
MAPP, the effect of being in the SSI waiver is stronger. SSI only participants again have
greater odds of poor outcomes than SSDI or concurrent beneficiaries; the strength of the
effect appears somewhat stronger than in the other UIEI model. Finally, the SSA
benefits level at enrollment variable (Q0BS$_sdz) is included in this model. As expected,
Q0BS$_sdz (Exp(B) = .717) is inversely related to inclusion in the “more successful”
group; the variable has about the same impact it did in the UI Earnings Success model.
Tables 5 and 6 present results for the Employment Success Indicator based on
information from the WPTI Enrollment and Monthly Update forms. These models, while
exhibiting adequate “goodness of fit,” explained much less of the variance than the UI$IP
and UIEI models presented in Tables 2 through 4. The Nagelkerke R Square for the
model including MAPP is .197, without MAPP .127. We first present findings from the
FEI model that included MAPP.
Table 5: Model Results for “Form” Employment Success Indicator (FEI) with MAPP.
N=483.
Variable
B
S.E.
Wald
Sig.
Exp(B)
PRIMDIS
.833
.266
9.776
.002
2.299
(cognitive/DD)
PRIMDIS
.550
.253
4.732
.030
1.733
(affective/MH)
SSIWAV
.628
.225
7.803
.005
1.875
MAPP
1.442
.293
24.244
.000
4.228
EMPRAT
.707
.219
10.465
.001
2.028
IMPQUAL
-.370
.305
1.475
.225
.690
(low)
IMPQUAL
-676
.221
9.343
.002
.509
(medium)
Chg_sdz
.242
.108
5.052
.025
1.274
Constant
-.978
.216
20.415
.000
.376
Variables not included by forward selection: AGE, SEX, RACE2, EDUCrev, OOSrev,
SSAcat, BCtoQ5, VStoQ5, Q0BS$_sdz, and FSQ0.
As with the UI Employment Success Indicator model that includes MAPP, both
Buy-in and SSI Waiver participation are strong (Exp(B) = 4.228 and 1.875 respectively)
and thus lead to substantially increased odds of inclusion in the FEI “more successful”
category, albeit that the Exp(B) values are considerably lower than the ones for the
comparable UIEI model. Much the same can be said for the two primary disability
(PRIMDIS) categories and the income change control, Chg_sdz. However EMPRAT,
the work history variable, is about equally strong and positively associated with desired
outcomes in both the employment models that include the MAPP variable.
The FEI model (including MAPP) does not include the Social Security program
variable (SSACat), but for the first (and only) time the implementation quality variable
(IMPQUAL) appears in a model. While the coefficients are in the anticipated direction
(that is, participants at agencies rated as having “low” or medium” implementation quality
DRAFT – Do Not Cite Without Permission
24
have lower odds of being in the “more successful” category than those at agencies rated
“high”), only one of the coefficients is statistically significant.
The FEI model that excluded the MAPP variable (see Table 6) presents the most
idiosyncratic results, probably reflecting its slight explanatory power. It is the only model
that excludes SSI Waiver use (SSIWAV). It also excludes the race variable that was
quite strongly related to good employment outcomes in the comparable UI Employment
Success Indicator model.
Table 6: Model Results for “Form” Employment Success Indicator (FEI) without MAPP.
N=483.
Variable
B
S.E.
Wald
Sig.
Exp(B)
PRIMDIS
.675
.256
6.928
.008
1.963
(cognitive/DD)
PRIMDIS
.608
.240
6.412
.011
1.837
(affective/MH)
EMPRAT
.877
.209
17.541
.000
2.403
BCtoQ5
.010
.003
8.014
.005
1.010
Chg_sdz
.311
.100
9.560
.002
1.364
Constant
-1.286
.206
39.157
.000
.276
Variables not included by forward selection: AGE, SEX, RACE2, EDUCrev, OOSrev,
SSAcat, IMPQUAL, VStoQ5, SSIWAV, Q0BS$_sdz, and FSQ0.
More typically, both of the primary disability categories, work history (EMPRAT)
and income change (Chg_sdz), are present and positively related to inclusion in the
“more successful group” at strengths similar to the other FEI model. Also, like the other
FEI model, the Social Security program variable (SSACat) is not selected into the model.
Finally, benefits counseling hours is included and significant, but the Exp(B) value is so
close to “1” that the variable appears to have but a trivial impact on results.
Discussion
In this section we review the impact of our independent and control variables on
inclusion into the more and less successful categories of our three dichotomous
dependent variables. Using descriptive data presented in Appendices B and C and some
material from the WPTI Final Project Report, we identify and discuss possible
explanations for unexpected results from our regression models. We also make some
suggestions for further analysis of the WPTI dataset.
First, however, we briefly examine whether classification into one of the “more
successful” categories of the earnings and employment indicators results in the
participant being “better off” because of their experience in WPTI. We define “better off”
as having a higher income, not just higher earnings or a higher probability of
employment. Not only is this a pertinent standard for WPTI’s value from a participant
standpoint, it also speaks to the issue of whether the gains in earnings and the
reductions in benefit costs desired by both Wisconsin and SSA could be expected to be
lasting should the “more successful” participants retain the capacity and opportunity to
maintain increased work effort. The data in Tables 7, 8, and 9 suggest that the desired
patterns of income growth, including earnings growth and benefit cost reduction are
DRAFT – Do Not Cite Without Permission
25
present for those in the “more successful” categories. As previously identified (see
footnote #37), our income measure is a proxy that is limited to the participant’s UI
earnings and Social Security and State SSI Supplement cash benefits.
Table 7: Change in Income (Earnings and Social Security/State Supplement Amount)
between the Base and Outcome Years (1996 GDP$)
Variable
Mean Change
Median Change
Standard Dev.
“More Successful”
UI Earnings
$1525
$1033
$5925
UI Employment
$2251
$1100
$5963
Form Employment
$1230
$384
$5031
“Less Successful”
UI Earnings
-$327
$50
$2041
UI Employment
-$361
$59
$2945
Form Employment
-$22
$70
$3743
Table 8: Change in UI Earnings between the Base and Outcome Years (1996 GDP$)
Variable
Mean Change
Median Change
Standard Dev.
“More Successful”
UI Earnings
$2134
$1423
$6702
UI Employment
$2899
$1343
$6474
Form Employment
$1823
$519
$5524
“Less Successful”
UI Earnings
-$86
$0
$457
UI Employment
-$72
$0
$2998
Form Employment
$244
$0
$3812
Table 9: Change in Social Security/State Supplement Amount between the Base and
Outcome Years (1996 GDP$)
Variable
Mean Change
Median Change
Standard Dev.
“More Successful”
UI Earnings
-$609
$51
$2936
UI Employment
-$647
$0
$2786
Form Employment
-$594
$39
$2468
“Less Successful”
UI Earnings
-$241
$70
$1959
UI Employment
-$289
$76
$2278
Form Employment
-$266
$78
$2462
Another prominent result is that for every mean presented in these tables the
standard deviation is much larger than the mean. Moreover, the standard deviations for
income and earnings change for each of the “more successful” groupings are very high
on an absolute basis. These results signify that the dichotomous variables obscure a
large amount of variation in UI earnings among both the “more successful” and “less
successful,” but especially among the former. Though we explained our reasons for
using dichotomous indicators in this exploratory effort and presented evidence showing
that the “more successful” and “less successful” groups were easily distinguishable, it
DRAFT – Do Not Cite Without Permission
26
would make sense to investigate the use of indicator variables with more than two levels
and the use of multinomial logistic methods. 53
Returning to the WPTI intervention, we begin by noting that differences between
the more and less successful on all three of our indicator variables were apparent well
before enrollment in WPTI. More importantly, the growing divergence between the more
and less successful groups on each indicator grew at about the same rates in the
quarters before and after enrollment, with increasing outcomes among the “more
successful” responsible for almost all of the growing difference (see Figures 1 and 2).
Therefore, we think it more appropriate to think about WPTI as a program that supported
continued growth in outcomes rather than as the principal cause of those outcomes. 54
None of the WPTI service provision variables, including the one measure of
(perceived) intervention quality, appear to have directly contributed to whether
participants ended up in the more or less successful categories at a level that we would
consider meaningful. In brief, we hypothesize that any effect of the delivered quality of
WPTI services (IMPQUAL) was small compared to differences rooted in WPTI agency
recruitment patterns and service delivery philosophies. We will discuss this in more
detail when we look at the primary disability (PRIMDIS) variable. There may also be
PRIMDIS specific patterns in the value of different amounts of provision of specific
services for promoting better outcomes (e.g. job coaching for those in the
“cognitive/developmental” category within PRIMDIS), but at the WPTI program level any
impact of these variables is lost in the relatively high variation in service hours provided
to both “more successful” and “less successful” participants. 55
By contrast, the impacts of WPTI’s policy features, the SSI waiver and MAPP,
were generally strong. In most models, use of one of these policy features made the
53
We were concerned that our control variable for income change, Chg_sdz, had such a large
earnings component that it weakened the explanatory power of the models. Despite feeling it was
conceptually less appropriate than “income change,” we tried a standardized change in benefits
(Social Security and state supplement) amount variable in our models. In no case did doing so
improve the explanatory power of the model or substantially change the inclusion/exclusion of
other variables.
54
It is apparent from interviews and focus group that some agency staff and early participants
viewed WPTI more as a test of policy options that would ameliorate the tradeoffs between
employment and access to public benefits than as an employment program. In addition to such
reports, this assertion gains support from the comparatively high employment rates of the earliest
WPTI enrollees compared to those who entered somewhat later. Participants who enrolled in the
first 9 months of the project had about a 15% higher employment rate at entry than those who
enrolled in the next nine months, a fact that cannot be accounted for by the modest decline in
labor market conditions that occurred over this time period (July 1999 through December 2000).
See Delin, Barry S., et al. 2004. pp. 193-94 and 29-30.
55
The WPTI evaluation was not set up to clearly distinguish the impact of key service
components, especially benefits counseling, from that of the intervention as a whole. However,
using data from Vermont’s SPI program, one study found that the delivery of benefits counseling
services was strongly and positively associated with subsequent employment and earnings
outcomes. See Tremblay, Tim, Xie, Haiyi, Smith, James, and Drake, Robert. 2004. “The Impact
of Specialized Benefits Counseling Services on Social Security Administration Disability
Beneficiaries in Vermont,” Journal of Rehabilitation, 70, No. 2, pp. 5-11.
DRAFT – Do Not Cite Without Permission
27
odds of being in a “more successful” group two to four times greater. This suggests that
well crafted changes to the structure of incentives and disincentives built into public
benefits programs (whether in a single program or, preferably in theory, across multiple
ones) may prove a powerful intervention approach. However, given what we know about
the modest employment outcomes associated with MAPP participation in general, we
speculate that some of the services and supports provided through WPTI (e.g. benefits
counseling) may have had an important role in generating the strong outcomes observed
for WPTI participants who were also MAPP participants. 56 This claim receives support
from participant survey and focus group data. We learned that participants vary greatly
in when they felt ready to make use of the SSI Waiver, MAPP, or indeed other available
work incentives. Readiness could vary for many reasons, including health and personal
or familial circumstances, not directly related to employment goals or work activity. 57 This
interpretation also receives support from our interviews with both community agency
staff and DHFS and DVR administrators in a position to observe WPTI operations.
As mentioned earlier, we were surprised at the lack of impact that sociodemographic variables had in our models, most particularly educational attainment
(EDUCRev). Indeed, the tables in Appendix B show only small differences in the
proportions in each category of EDUCrev between the “more successful” and “less
successful” groups for any of the dependent variables. We thought it possible that there
might be an unusually large number of very severely disabled participants with high
educational attainment that would obscure the expected relationship between education
and strong employment outcomes. The data in Table 10 does not support this possibility,
but it does suggest another.
Table 10: Educational Attainment (EDUCrev) by DVR Order of Selection (OOSRev) and
Primary Disability (PRIMDIS)
Variable
Less than High
High School or
More than High
School
GED
School
OOSrev
Most significant
54 (21%)
91 (36%)
108 (43%)
Other
37 (16%)
54 (24%)
139 (60%)
PRIMDIS
Cognitive/developmental
19 (20%)
43 (46%)
31 (33%)
Affective/mental
25(23%)
35 (32%)
49 (45%)
Physical
92 (18%)
151(30%)
258 (52%)
Persons classified as having a “physical” primary disability had a somewhat
higher percentage in the “more than high school” category of EDUCrev than those
classified as having an “affective/mental” condition as their primary disability. Table 10
also shows that WPTI participants with a physical condition as their primary disability
had, as a group, much higher levels of educational attainment than those with a
56
However, caution is advised in interpreting the strong beneficial results associated with MAPP
participation as only 16% of WPTI participants were enrolled in the Buy-in program. Even in the
“more successful” groups the participation rates varied between 24% and 30%.
57
We have not yet looked at the participant follow-up survey data based on the dichotomous
outcome variables we created for this study. Thus the characterization of results provided on
page 13 reflects the dominant response pattern among those participants who returned surveys.
The focus group data is much richer, but the number of cases is very small.
DRAFT – Do Not Cite Without Permission
28
classification of “cognitive/developmental.” These distributions are not unexpected. What
is unexpected is that, as a look at the PRIMDIS distributions in the Appendix B tables will
indicate, the proportion of those with “physical” primary disabilities in each of the “more
successful” distributions is always less than for the corresponding “less successful”
distributions. By contrast, those with “affective/mental” primary disabilities show a very
strong pattern in the opposite direction, as do, to a much lesser degree, those with
cognitive/developmental primary disabilities. As with the seemingly anomalous results
for the WPTI implementation quality variable, we will discuss this issue further in the
context of the PRIMDIS variable.
It is also possible that we have used a poor categorization for measuring
educational attainment in the context of the contemporary labor market. As a high school
diploma or GED increasingly becomes the effective minimum credential for work that
pays a “living wage” or even work which opens up that possibility, we might have been
better served by having a variable that distinguishes between different amounts of postsecondary education. Fortunately, the data to do this are available, though not in the
data set used for this paper.
We have little to add to our previous comments about our benefit program
participation and work experience variables. As a group, their impact was much as we
anticipated. There was an inverse relationship between receiving high levels of SSA
and/or other income support benefits and those on SSI only were less likely to achieve
earnings and employment outcomes that would result in assignment to the “more
successful” groups. Also, it was not surprising to learn that those employed in a greater
proportion of the years between entering Social Security benefit status and entering
WPTI (EMPRAT) were more likely to have strong employment outcomes. The Exp(B)
values for EMPRAT varied over a fairly small range (about 1.9 to 2.4) across all of the
five logistic models. However, we observed that the distribution of EMPRAT was strongly
bimodal, with more than 80% of the 506 cases either having reported earnings in none
or all of the years between entering a Social Security disability program and entering
WPTI. The bimodality was also observable in each of the “more successful” and “less
successful” groups. Consistent with expectations, the “always employed” peak
(EMPRAT = 1) was in all cases substantially higher than the “never employed” peak
(EMPRAT = 0) for the “more successful,” and visa versa for the “less successful.” We do
not know whether the distribution of EMPRAT across the Social Security (adult) disability
program population would be similar; but if it is, it suggests a potential targeting or
recruitment approach for return to work programs.
In all of the models, those with an “affective/mental” primary disability showed a
strong tendency to be included in the “more successful” groups compared to those in the
reference PRIMDIS category, “physical.” This was also true for those in the
“cognitive/developmental” category for all of the employment models, but not the
earnings model. Though assignment of cases to a PRIMDIS category was based on
information (RSA-911 codes) external to and usually predating WPTI enrollment, we
urge caution in interpreting the relatively high effect sizes (Exp(B) from about 1.7 to 3.0)
as an indicator of “physical” disabilities being a direct cause of less favorable
employment outcomes. WPTI participants with a “physical” primary disability had higher
level of educational attainment than either of the other disability categories. On other
variables of interest such as SSA status, level of severity (OOSrev), and employment
history since entering a SSA disability program, the distributions for the “physical”
category of PRIMDIS were similar to those for persons in the “affective/mental” category
DRAFT – Do Not Cite Without Permission
29
and much more promising than for persons in the “cognitive/developmental” category.
Finally, we know that persons enrolled at WPTI “physical disability” agencies (most of
whom were classified as having a “physical” primary disability) had, when employed,
much higher average earnings than employed participants enrolled at other types of
WPTI agencies. 58 There is nothing here to suggest why those with a PRIMDIS of
“physical” should be so much less likely to be included among the “more successful.”
Nonetheless, this result could have been anticipated from the results of the WPTI
evaluation. In the regression models that examined differences among the three
PRIMDIS disability groups within WPTI, those in the “affective/mental” and the
“cognitive/developmental” categories had significantly better employment and earnings
outcomes than those in the “physical” category.” The overall trends for the groups were
broadly similar and the degree of divergence in estimated outcomes tended to persist
over the full two year period following WPTI enrollment. 59
Previously we noted (see footnote #41) the strong relationship between inclusion
in a primary disability (PRIMDIS) category and enrollment at a particular type of WPTI
site. 60 This was not a chance association. It reflected both the secondary eligibility rules
for WPTI, the tendency of community agencies to do outreach to current or former
consumers, and in some cases state and/or organizational rules governing whom an
agency could serve. “Agency type,” rather than “primary disability,” was emphasized in
the modeling for the WPTI evaluation. Regression models restricted to WPTI participants
(i.e. excluding members of the comparison group) exhibited the same general pattern of
results as for the models incorporating PRIMDIS. 61 We believe that the evidence points
to the dominant effect of agency recruitment, mission, and service philosophy as one
principal cause of observed results. The cell sizes for combinations of PRIMDIS and
WPTI agency type that do not “coincide” are modest, though it would make sense to at
least examine their crosstabulation. 62
58
Delin, Barry S., Reither, Anne E., and Drew, Julia A. 2003. “Employment and Earnings Trends
in a Complex Program Delivery Environment.” Presentation at the State Partnership Annual
Meeting, Washington D.C., August 7-8, 2003. Average UI Earnings for employed participants at
physical disability agencies in the eighth quarter following the enrollment quarter was $2,486. The
amounts for those employed at mental health and developmental disability agencies were,
respectively, $1,850 and $1,641. These data are for persons enrolled by March 31, 2001, six
months prior to the final enrollment date used in the analyses contained in this paper.
59
See Delin, Barry S., et al. 2004. pp. 181-85 and p. 52. Unfortunately, no models examined the
impact of PRIMDIS between WPTI participants and comparison group members. This was a
result of using the PRIMDIS variable to compute the propensity scores needed to “equate” the
two groups.
60
87% of those classified as “physical/HIV” in PRIMDIS received WPTI services at a designated
“Physical Disability” or “HIV/AIDS” agency. The proportion of those with PRIMDIS = “affective”
enrolled through “Mental Health” agencies was 72%.The proportion of those with PRIMDIS =
“cognitive” enrolled through “Developmental Disability” agencies was 77%. See Delin, Barry S., et
al. 2004. p. 79.
61
62
Delin, Barry S., et al. 2004. pp. 134-37.
The data set used for this study does not include WPTI agency type, though that data can be
obtained. Actual cells sizes are unknown. However, cell sizes range from a low of 1 to a high of
44 for the enrollment group of 527 cases from which the 506 were drawn.
DRAFT – Do Not Cite Without Permission
30
Additionally, one of the two types of organizations, Community Support
Programs, selected as WPTI “mental health” agencies used an agency wide service
approach that encouraged employment mainly as a form of therapy and only secondarily
as a means to increase economic welfare. These agencies, despite achieving high
employment rates for their WPTI participants and, thereby, relatively high average
earnings, were rated poorly on the implementation quality variable. 63 While it is doubtful
that this alone accounts for the weak impact of the implementation quality (IMPQUAL)
variable, it appears to have contributed to that result.
Furthermore, we speculate that the combination of secondary eligibility criteria for
the four types of WPTI provider agencies (physical disabilities, mental health,
developmental disabilities, and HIV/AIDS), agency recruitment patterns, and agency
service philosophies may explain why educational attainment and other sociodemographic variables appear to have little or no impact on whether a participant will
have successful outcomes as we’ve defined them. This may also be the primary reason
why attitudinal factors seem to have little effect either. This combination may be so
powerful, in the specific context of how WPTI operated, as to swamp the expected
effects of characteristics generally associated with labor market outcomes, especially
those characteristics that might explain the different patterns in earnings and
employment rates observed in the quarters prior to WPTI entry. We are, however,
unwilling to speculate how frequently this type of selection bias might occur. While
apparently not “creaming” in intent, it raises similar concerns about the ability to
generalize results. 64
63
Based on our observations and interviews during WPTI, we think this observation would apply,
if less stringently, to at least one of the “developmental disability” agencies.
64
It is important to note that random assignment of participants to treatment and control groups
would not by itself have solved this problem. The problem is a consequence of the delivery
system chosen for WPTI. That choice was not made for research considerations, but to exploit
existing service delivery capacity and to work within an existing programmatic and legal
framework.
There is circumstantial evidence (mainly from interviews, surveys, and informal conversation) that
limited “creaming” occurred. As service provision was, with few exceptions, provided through
DVR authorizations (though the federal match was provided through DHFS funds), it is
understandable that DVR counselors and community agency staff wanting to maintain good
relationships with those counselors might give preference to individuals who appeared likely to
achieve successful DVR closures (“twenty-sixes”). Though DVR state-level managers do not
appear to have ever directly pressured counselors to withhold a DVR authorization for WPTI
services for individuals who appeared unlikely to succeed, these managers did communicate their
concern about the higher average cost per closure associated with WPTI compared to other
patterns of DVR funded services.
There is also some countervailing evidence that particular DVR counselors steered some
particularly difficult to serve consumers to WPTI. We have no means of comparing the relative
power of these two opposing effects on who entered the demonstration.
However, it must also be remembered that the WPTI enrollment process incorporated “ability to
benefit” judgments, making the identification of deliberate selection bias difficult. The first and
most important was that made by an individual in deciding to enroll. From the perspective of
DRAFT – Do Not Cite Without Permission
31
The issue is not, in our view, how the characteristics of WPTI participants differ
from those of the working age SSDI and SSI (for reason of disability) population. It is
how the characteristics of WPTI participants might differ from those within the population
of the putatively eligible who might choose to utilize a program similar to WPTI should it
become available through most, if not all, community agencies that provide services to
persons with severe disabilities. The observed differences between WPTI participants
and the comparison group of DVR consumers in their respective distributions for
disability characteristics and some socio-demographic variables suggest that the
characteristics of WPTI participants might not closely model those who might enter a
similar program that was more widely available. In this vein, it is important to consider
that a disproportionate amount of WPTI capacity was located at organizations that
predominately served persons with physical disabilities. Should it be determined that the
overall distribution of organizational capacity in Wisconsin is reasonably similar to that
mobilized for the WPTI demonstration, we would have even greater confidence in using
our results to inform policy design in at least that state. 65
Lastly, we noted that we had not included any of the baseline survey data about
participants’ attitudes and perceptions in our models, largely because of the absence of
association with our dependent variables at the .2 significance level and the large
number of missing responses for certain items. Yet, in the WPTI evaluation, several of
these variables proved to be significant predictors of employment status across time. 66 It
seems counterintuitive that none of the attitudinal variables are associated with inclusion
in the outcome groups. Thus, this is an area for further investigation, though with one
exception we have not identified an approach to guide future work.
The barriers measure included in the data set we used is an average score
across a range of conceptually distinct impediments to employment or increasing work
effort or earnings. Items include the participant’s perceptions of health status, needs to
maintain access to health care and other public benefits, the availability of transportation
and personal support, the respondents need for further education or training, and the
impact of disincentives to earnings in Social Security and other public programs. Over
the year between the baseline and follow-up surveys, WPTI participation appeared to
WPTI’s designers and most of its organized stakeholders, the principle of consumer choice was
paramount (among other things contributing to the decision not to use random assignment). Later
Moreover, both the community agency and DVR counselor applied ability to benefit
considerations. In the community agency’s case, this was supposed to be limited to having the
ability to serve the individual’s specific needs, not just those of the typical individual fitting into the
disability category the agency was authorized (for WPTI) to serve. The DVR counselor was to use
the same criteria used for all other consumers.
65
WPTI did not formally structure the community agency selection process to result in having
more of its enrollment capacity at the agencies serving those with “physical” disabilities. In the
site selection RFP, WPTI offered that it was seeking a equal number of “physical disability” and
“mental health” sites. In point of fact, the targeted enrollment capacity for each disability category
that emerged from the agency selection process did not closely match the estimates of
enrollment in each disability category that DHFS communicated to SSA, as part of Wisconsin’s
SPI proposal.
66
These models contrasted all those in the WPTI study, not just those who received the
intervention. See Delin, Barry S., et al. 2004. pp. 152 - 158.
DRAFT – Do Not Cite Without Permission
32
lessen the perceived severity of barriers to employment in two ways, both through the
intervention itself (e.g. benefits counseling and easy access to agency staff) and through
the “reality testing” of achieving better employment outcomes. 67 It may be useful to
explore changes in specific barrier items, treating the dependent indicator variables used
in this paper (UI$I, UIEI, and FEI) as independent variables.
To sum up, it appears certain that the differences between the more and less
successful WPTI participants were strongly rooted in attributes and circumstances in
place before entry into WPTI. The problem is that with the exception of maintaining
attachment to work after entry to a Social Security program we have little to say what
these were and, thus, have limited advice for those who might want to look for ways to
target recruitment for return-to-work programs. We have found evidence to support that
policy innovations like the Medicaid Buy-in and the SSI waiver can make a difference for
those participants “poised” to make stronger (by our criteria) return-to-work efforts, but
found no direct quantitative evidence of the role service provision aspects of WPTI, like
benefits counseling, may play in facilitating better outcomes. Certainly, we will explore
whether there are unexploited opportunities to look at this issue using available data.
Finally, we are left with the unresolved issue of how typical WPTI participants are of the
larger population that might choose to enter return-to-work programs and/or use existing
or future work incentives. The WPTI experience suggests that organizational (rather than
individual) selection can have a strong influence on individual participation and, it
appears, outcomes. To the extent that delivery of employment and, more generally,
social services continues to be highly decentralized in the United States, it is likely that
such organizational factors matter. We cannot say, however, that they will matter in the
same ways they did for WPTI.
67
Reither, Anne E., Delin, Barry S., and Drew, Julia A. 2005. WPTI Final Project Report –
Addendum: Barriers to Employment. Menomonie, WI: University of Wisconsin – Stout Vocational
Rehabilitation Institute. In particular, see pp. 10-12.
DRAFT – Do Not Cite Without Permission
33
Appendix A: Independent and Control Variables Used in Logistic Models
NAME
AGE
SEX
RACE2
EDUCrev
PRIMDIS
OOSrev
SSACat
EMPRAT
IMPQUAL
BCtoQ5
VStoQ5
DESCRIPTION
Age at Enrollment. Calculated in whole
numbers at birthday on or preceding
enrollment date.
Sex. 1=male 2=female
Race, recoded into a dichotomous variable.
1=other 2=white
Education, recoded into three categories.
1=less than high school 2=high school
diploma or GED 3=more than high school
Primary Disability, coded into three
categories. 1= cognitive/developmental
2=affective/mental 4=physical (including
HIV/AIDS)
Order of Selection assignment recoded. 1=
most significant 2= other
SSA status at enrollment. 1=SSDI or
concurrent SSDI/SSI 2=SSI only. As the SSA
administrative data available did not
unequivocally identify this status on the
enrollment date, any record of SSDI
participation in the year leading up to
enrollment resulted in that participant being
assigned to the SSDI or concurrent .
Ratio of years with reported earnings
between entry to a Social Security disability
program and WPTI enrollment (noninclusive). Values can range from “0” (no
years with earnings) to “1” (all years having
earnings).
Average assessment of WPTI implementation
quality as assessed by WPTI central program
staff housed at DHFS and DVR. This variable
measures perceived overall implementation
quality, not that of the quality of the
intervention received by any specific
participant. 0=low 1=medium 2=high
Hours of benefits counseling services
delivered from enrollment quarter through the
fifth quarter following enrollment.
Hours of vocational services, including
barriers assessment and vocational service
planning, delivered from enrollment quarter
through the fifth quarter following enrollment.
These hours do not include services provided
by vendors external to the WPTI agency.
DATA SOURCE
Encounter data
Encounter data
Encounter data
Participant survey,
supplemented with
encounter data
DVR administrative
records, participant
survey when RSA911 code
unavailable.
DVR administrative
records
SSA administrative
records
SSA administrative
records and
encounter data
Q sort procedure
Encounter data
Encounter data
DRAFT – Do Not Cite Without Permission
JCtoQ5
SSIWAV
MAPP
Q0BS$_sdz
FSQ0
Chg_sdz
Hours of job coaching services delivered from
enrollment quarter through the fifth quarter
following enrollment.
SSI Waiver participation (registering to use
the waiver, if eligible) 0=never on waiver 1=
on waiver at some time during WPTI
Medicaid Buy-in participation 0=never on
MAPP during WPTI 1= on MAPP at some
time during WPTI
Standardized amount of cash benefit (in 1996
GDP dollars) from SSA and the Wisconsin
SSI state supplement during the enrollment
quarter. Values were standardized by
subtracting the mean and then dividing by the
standard deviation.
Food Stamp participation during the
enrollment quarter 0=no 1=yes
Standardized change in income proxy (in
1996 GDP dollars) between base (Q-1
through Q2) year and outcome year (Q5
through Q8). Income includes UI earnings,
benefit payments from SSA and payments
from the Wisconsin SSI state supplement.
Values were standardized by subtracting the
mean and then dividing by the standard
deviation.
34
Encounter data
WPTI administrative
data
DHFS administrative
data
Computed from SSA
administrative data
DWD administrative
data
Computed from
SSA, DHFS, and
DWD/UI
administrative data
DRAFT – Do Not Cite Without Permission
35
Appendix B: Frequencies for Categorical Variables by Success Indicators (UI$I,
UIEI, & FEI)
Categorical Variable Frequencies and Percentages for UI Earnings Success
Indicator (UI$I) by “More Successful” and “Less Successful”
Variable
“More Successful”
“Less Successful”
SEX
Male
142 (60%)
165 (61%)
Female
94 (40%)
105 (39%)
RACE2
White
190 (81%)
201 (74%)
Other
46 (20%)
69 (26%)
EDUCrev
Less than H.S.
40 (17%)
52 (19%)
H.S.
77 (33%)
74 (28%)
More than H.S.
117 (50%)
141 (53%)
PRIMDIS
Cognitive/developmental
42 (18%)
53 (20%)
Affective/mental
71 (30%)
39 (14%)
Physical
123 (52%)
161 (66%)
OOSrev
Most significant
105 (46%)
149 (58%)
Other
121 (54%)
110 (42%)
SSACat
SSDI or concurrent
177 (75%)
187 (69%)
SSI only
59 (25%)
73 (31%)
IMPQUAL
Low
41 (17%)
39 (14%)
Medium
101 (43%)
124 (46%)
High
94 (40%)
107 (40%)
SSIWAV
Never
147 (62%)
214 (79%)
At some time
89 (38%)
56 (21%)
MAPP
Never
180 (76%)
246 (91%)
At some time
56 (24%)
24 (9%)
FSQ0
No
202 (86%)
204 (76%)
Yes
34 (14%)
66 (24%)
Note: Percentages calculated for non-missing cases. There are 21 missing cases for
OOSrev, 5 for EDUCrev.
DRAFT – Do Not Cite Without Permission
36
Categorical Variable Frequencies and Percentages for UI Employment Success
Indicator (UIEI) by “More Successful” and “Less Successful”
Variable
“More Successful”
“Less Successful”
SEX
Male
106 (61%)
201 (61%)
Female
68 (39%)
131 (40%)
RACE2
White
148 (85%)
243 (73%)
Other
26 (15%)
89 (27%)
EDUCrev
Less than H.S.
29 (17%)
63 (19%)
H.S.
55 (32%)
96 (29%)
More than H.S.
88 (51%)
170 (52%)
PRIMDIS
Cognitive/developmental
37 (21%)
58 (18%)
Affective/mental
53 (31%)
57 (17%)
Physical
84 (48%)
217 (65%)
OOSrev
Most significant
83 (50%)
171 (54%)
Other
83 (50%)
148 (46%)
SSACat
SSDI or concurrent
132 (76%)
232 (70%)
SSI only
42 (24%)
100 (30%)
IMPQUAL
Low
35 (20%)
45 (14%)
Medium
66 (38%)
159 (48%)
High
73 (42%)
128 (39)%
SSIWAV
Never
110 (63%)
251 (76%)
At some time
64 (37%)
81 (24%)
MAPP
Never
126 (72%)
300 (90%)
At some time
48 (28%)
32 (10%)
FSQ0
No
149 (86%)
257 (77%)
Yes
25 (14%)
75 (23%)
Note: Percentages calculated for non-missing cases. There are 21 missing cases for
OOSrev, 5 for EDUCrev.
DRAFT – Do Not Cite Without Permission
37
Categorical Variable Frequencies and Percentages for “Forms” Reported
Employment Success Indicator (FEI) by “More Successful” and “Less Successful”
Variable
“More Successful”
“Less Successful”
SEX
Male
136 (60%)
171 (61%)
Female
90 (40%)
109 (39%)
RACE2
White
185 (82%)
206 (74%)
Other
41 (18%)
74 (26%)
EDUCrev
Less than H.S.
36 (16%)
56 (20%)
H.S.
73 (33%)
78 (28%)
More than H.S.
114 (51%)
144 (52%)
PRIMDIS
Cognitive/developmental
53 (24%)
42 (15%)
Affective/mental
59 (26%)
51 (18%)
Physical
114 (50%)
187 (67%)
OOSrev
Most significant
105 (49%)
149 (55%)
Other
109 (51%)
122 (45%)
SSACat
SSDI or concurrent
166 (74%)
198 (71%)
SSI only
60 (27%)
82 (39%)
IMPQUAL
Low
36 (16%)
44 (16%)
Medium
87 (39%)
138 (49%)
High
103 (46%)
98 (35%)
SSIWAV
Never
148 (66%)
213 (76%)
At some time
78 (35%)
67 (24%)
MAPP
Never
170 (75%)
256 (91%)
At some time
56 (25%)
24 (9%)
FSQ0
No
191 (85%)
215 (77%)
Yes
35 (16%)
65 (23%)
Note: Percentages calculated for non-missing cases. There are 21 missing cases for
OOSrev, 5 for EDUCrev.
DRAFT – Do Not Cite Without Permission
38
Appendix C: Selected Statistics for Interval Variables by Success Indicators (UI$I,
UIEI, & FEI)
Mean, Median and Standard Deviation for UI Earnings Success Indicator (UI$I) by
“More Successful” and “Less Successful”
Variable
Mean
Median
Standard
Approx.
Deviation
Mean/StDev.
AGE
“More Successful”
37.4
38.0
10.4
3.6
“Less Successful”
38.8
39.0
10.1
3.8
EMPRAT
“More Successful”
0.66
1.00
.460
1.4
“Less Successful”
0.45
0.27
.457
1.0
BCtoQ5
“More Successful”
33.5
23.0
39.3
0.8
“Less Successful”
27.3
21.0
24.5
1.1
VStoQ5
“More Successful”
71.5
49.5
68.3
1.0
“Less Successful”
43.8
38.0
30.6
1.4
JCtoQ5
“More Successful”
4.6
0.0
17.0
0.3
“Less Successful”
1.1
0.0
5.4
0.2
SSA Q0 Benefit
Amount (1996
GDP$)*
“More Successful”
$1758
$1735
$851
1.9
“Less Successful”
$1983
$1928
$787
2.5
“Income” Change
Base to Outcome
Year (1996 GDP$)*
“More Successful”
$1525
$1031
$5925
0.3
“Less Successful”
-$327
$50
$2041
0.2
* Presented statistics based on actual data rather than for standardized versions of
these variables used in the regression models.
DRAFT – Do Not Cite Without Permission
39
Mean, Median and Standard Deviation for UI Employment Success Indicator (UIEI)
by “More Successful” and “Less Successful”
Variable
Mean
Median
Standard
Approx.
Deviation
Mean/StDev.
AGE
“More Successful”
36.9
37.0
10.0
3.7
“Less Successful”
38.8
39.0
10.3
3.8
EMPRAT
“More Successful”
0.65
1.00
.465
1.4
“Less Successful”
0.50
0.50
.464
1.1
BCtoQ5
“More Successful”
34.6
22.0
42.6
0.8
“Less Successful”
27.9
21.0
25.2
1.1
VStoQ5
“More Successful”
73.7
49.0
71.5
1.0
“Less Successful”
74.7
60.1
55.4
1.3
JCtoQ5
“More Successful”
5.2
0.0
18.9
0.3
“Less Successful”
1.5
0.0
6.5
0.2
SSA Q0 Benefit
Amount (1996
GDP$)*
“More Successful”
$1733
$1739
$865
2.0
“Less Successful”
$1953
$1893
$793
2.5
“Income” Change
Base to Outcome
Year (1996 GDP$)*
“More Successful”
$2251
$1100
$5963
0.4
“Less Successful”
-$361
$59
$2945
0.1
* Presented statistics based on actual data rather than for standardized versions of
these variables used in the regression models.
DRAFT – Do Not Cite Without Permission
40
Mean, Median and Standard Deviation for “Forms” Reported Employment Success
Indicator (FEI) by “More Successful” and “Less Successful”
Variable
Mean
Median
Standard
Approx.
Deviation
Mean/StDev.
AGE
“More Successful”
38.1
38.3
9.9
3.9
“Less Successful”
38.2
39.0
10.5
3.6
EMPRAT
“More Successful”
0.65
1.00
.457
1.4
“Less Successful”
0.47
0.33
.466
1.0
BCtoQ5
“More Successful”
34.7
22.5
39.3
0.9
“Less Successful”
26.6
21.0
25.1
1.1
VStoQ5
“More Successful”
77.9
54.5
71.6
1.1
“Less Successful”
71.4
59.5
51.5
1.4
JCtoQ5
“More Successful”
4.3
0.0
14.9
0.3
“Less Successful”
1.5
0.0
9.7
0.2
SSA Q0 Benefit
Amount (1996
GDP$)*
“More Successful”
$1781
$1739
$805
2.2
“Less Successful”
$1956
$1900
$832
2.4
“Income” Change
Base to Outcome
Year (1996 GDP$)*
“More Successful”
$1230
$384
$5031
0.2
“Less Successful”
-$21
$70
$3743
0.1
* Presented statistics based on actual data rather than for standardized versions of
these variables used in the regression models.