Does Work Incentive Benefits Counseling Improve Employment Outcomes for
Those with Serious Disabilities? Preliminary Evidence for the “Work Oriented”
from Two Demonstration Projects
Barry S. Delin, Ellie A. Hartman and Christopher W. Sell
Stout Vocational Rehabilitation Institute
University of Wisconsin – Stout
APPAM Research Conference
Boston, MA
November 5, 2010
The authors thank staff at Pathways to Independence, Office of Independence and
Employment, Wisconsin Department of Health Services for their cooperation and
support. This work was supported by the Centers for Medicare and Medicaid
Services, Medicaid Infrastructure Grant – CFDA No. 93.768 and Social Security
Administration Contract No. SS00-05-60008
The descriptions and interpretations in this paper are solely those of the authors.
1
I. Introduction
Work incentive benefits counseling has been characterized as an essential
service for helping persons with severe disabilities to either obtain employment or to
improve outcomes gained through employment. Though there is a range of definitions
for work incentive benefits counseling, we think that one offered by Golden et al. remains
the best starting point. 1 Work incentive benefits counseling refers to services and
supports aimed at (1) promoting and sustaining successful employment outcomes and
(2) doing so in ways that allow those receiving the services and supports to make
informed choices about their level of work activity. Key functional tasks include
assessing the consumer’s situation and goals, identifying available options, and, based
on the consumer’s preferences, tracking and managing benefits and/or work activity to
help consumers achieve their goals.
Nevertheless, claims about the effectiveness of work incentive benefits
counseling services rest heavily on anecdotal reports from those either receiving or
providing such services or on appeals to a largely implicit intervention theory. However,
there are several studies that provide quantitative evidence of the positive impact of
work incentive benefits counseling. This paper is intended as a contribution to this
developing literature.
The claim that work incentive benefits counseling is an essential service is really
an assertion about a particular context: participation in public benefit programs in the
United States where eligibility is contingent upon establishing that one is sufficiently
disabled. Thus, advocates for the “necessity” of work incentive benefits counseling
services focus on the two Social Security disability programs, that is, Social Security
Disability Insurance (SSDI) and Supplemental Security Income (SSI), and the public
health care programs (Medicaid and Medicare) where eligibility can arise from meeting
Social Security disability criteria. Perhaps the most important of the criteria is inability to
work at the Substantial Gainful Activity (SGA) level because of one’s disabling condition.
SGA is defined by having the capacity to work at any job that will result (in 2010) in
monthly earnings of $1000 or more. 2 For those participating in the SSDI and/or SSI
programs, such work activity has the potential to result in loss of eligibility for these
income support programs and for any others that rely on SSA disability criteria. Even
when eligibility is not threatened, program rules can result in very large reductions of
income when beneficiaries increase their earnings. Thus, work incentive benefits
counseling involves identifying ways to best ameliorate tradeoffs between having higher
earnings and maintaining program eligibility and/or benefit levels.
It is the focus on ameliorating the substantially negative interaction between
increases in earnings and disability program rules that most differentiates work incentive
benefits counseling from other types of benefits counseling, particularly the type that
1
Golden, T. O’Mara, S., Ferrell, C. and Sheldon, J. (2000) “A Theoretical Construct for Benefits
Planning and Assistance” Ithaca, NY: Cornell University, Program on Employment and Disability.
Retrieved August 9, 2010 from
http://www.ilr.cornell.edu/extension/files/download/theoretical%20construct.pdf
2
SGA is indexed to the Consumer Price Index for Urban Consumers (CPI-U). There is a
somewhat higher SGA level for those who qualify for disability because they are blind.
2
concentrates on helping individuals gain eligibility for and/or maximize benefits from
public programs. Henceforth, whenever the term “benefits counseling” appears, it refers
solely to work incentive benefits counseling.
Purposes of Study
Recently, we completed an evaluation of a pilot of a SSDI benefit offset
provision, a type of work incentive that reduces cash benefits more slowly than
increases in earnings. The project was called the Wisconsin SSDI Employment Pilot or,
for short, the SSDI-EP. 3 Those assigned to the treatment group had the opportunity to
make use of an offset provision after completion of their Trial Work Period. When
applied, the offset feature resulted in only one dollar of reduction in the SSDI benefit for
every two dollars of earnings over SGA. However, study participants, irrespective of
assignment, were promised access to benefits counseling services. This commitment
was substantially fulfilled. 4
The pilot was intended to provide the Social Security Administration with
information that would inform the design of a national demonstration of a SSDI benefit
offset. The pilot also provided a largely unanticipated opportunity to examine the role of
benefits counseling in promoting better employment outcomes. In brief, there were
statistically significant increases in earnings, the probability of employment, the
probability of having earnings at or above the SGA level, and imputed gross income
associated with larger dosages of benefits counseling services. Moreover, statistically
significant gains in employment outcomes appeared at surprisingly modest dosage
levels; that is, at as little as four hours of reported services over a two year period. 5 Data
for one of these outcomes (earnings) is presented in figure 1.
3
Delin, B., Hartman, E., Sell, C., and Reither, A. (2010) Testing a SSDI Benefit Offset: An
Evaluation of the Wisconsin SSDI Employment Pilot Menomonie, WI: University of Wisconsin –
Stout Vocational Rehabilitation Institute.
4
Through the end of the initial calendar quarter following the quarter of study entry, those in the
treatment and control group received essentially the same amounts of benefits counseling
services. Services were front loaded in the sense that the average participant would have
received about half of their total benefits counseling services in the first six months of the two
year period included in the analysis. Thereafter, those assigned to the treatment group tended to
receive greater service hours. See Delin, et al., 2010. pp. 104-06.
5
Results for between subject differences were significant for all employment outcomes at a 0.05
p-value. MANOVA techniques were utilized because the technique supports time series analyses
with limited sample sizes. Delin, et al. (2010) pp. 223-24. More detailed descriptions of model
results for benefits counseling dosage are interspersed through pp. 221-58.
3
Figure 1: Quarterly UI Earnings by Benefits Counseling Dosage Categories, All SSDI-EP
Participants
UI Earnings by Benefits Counseling Dosage Categories, All SSDI-EP Participants
2000.00
1800.00
1600.00
Quarterly UI Earnings
1400.00
1200.00
BC Hrs = 8.0 or >
BC Hrs. = 4.0 to 8.0
BC Hrs. = 0.1 to 3.9
BC Hrs. = 0
1000.00
800.00
600.00
400.00
200.00
0.00
q4pre
q3pre
q2pre
q1pre
q0
q1
q2
q3
q4
q5
q6
q7
q8
Quarter Relative to Entry Quarter
Source: Wisconsin Unemployment Insurance and SSDI-EP encounter data
Note: Constant Dollars (CPI-U, August 2005 adjusted to 100)
Consequently, we thought that further analysis of the data from the SSDI-EP was
warranted. Moreover, we had access to a deidentified data set from an earlier
demonstration that was similar to the SSDI-EP on numerous dimensions. This project,
Wisconsin Pathways to Independence (WI SPI) was one of the twelve projects the
Social Security Administration funded as part of the State Partnership Initiative.
Given the strong resemblances between the SSDI-EP and WI SPI and the
availability of comparable data about service delivery and employment outcomes, it
seemed reasonable to conduct analyses of the impact benefits counseling had in each
effort. Additionally, there was enough similarity between the projects to explore whether
the samples could be combined for some analyses.
In particular, given that for both studies data were collected about the amount
and timing of service provision, it would be possible to do go beyond previous studies
that had been limited to simply identifying an association between receiving any benefits
counseling and differences in employment outcomes. In this paper, we begin to explore
the relationship between dosage (the hours of service delivered) and employment
outcomes. We seek to answer two general questions, though we may not be able to
generalize our answers beyond our samples to the universe of “working age” persons in
the United States living with severe disabilities. These questions are:
4
•
•
What is the impact of work incentives benefits counseling on employment
outcomes (such as mean earnings and the probability of employment).
What is the influence of prior employment outcomes on the dosage of work
incentive benefits counseling services received?
We see these questions as closely related. We note that in most of the existing
quantitative studies about the effects of benefits counseling there is a strong increase in
measured employment outcomes in the time periods immediately following the start of
the intervention. By contrast, results in later time periods are quite mixed, with generally
either more modest increases or actual declines. This suggests the possibility that those
who received benefits counseling were already primed to seek employment or, if already
employed, to increase their earnings. Nonetheless, this observation does not pre-judge
whether these individuals would have attained the same level of employment and
earnings in the absence of benefits counseling.
Additionally, we examine how the relationship between the dosage of benefits
counseling and employment outcomes varies for subgroups based on employment
status prior to entering either the WI SPI or the SSDI-EP. As already noted, we are
interested in the effects of dosage for those:
•
•
Seeking to enter the labor market either after long absence or for the first time
Already employed at relatively high earnings, but seeking to increase their
earnings
Literature Review
There are relatively few studies that focus directly on the impact of work incentive
benefits counseling on employment outcomes. 6 In addition to limiting consideration of
dosage to an essentially dichotomous condition (received/not received), we are not
aware of any study that includes data about the quality of benefits counseling services
received at the individual level. 7
The two most widely cited papers about the impact of benefits counseling
services use data for vocational rehabilitation (VR) consumers in Vermont. Both studies
estimate earnings for individuals in Social Security disability programs who received
benefits counseling through Vermont’s State Partnership Initiative (SPI) project over a
period starting two years before the receipt of benefits counseling services and ending
two years thereafter. These individuals were volunteers. Their employment outcomes
6
We exclude analyses where the effects of benefits counseling are co-mingled with other
services as, in particular, was the case for many of the projects operated during the State
Partnership Initiative. Examples of such work include Shea, J. and Ekstrom, S. (2004)
California’s Individual Self-Sufficiency Planning (ISSP) Project: Final Evaluation Report of a State
Partnership Initiative (SPI) Demonstration Project. Napa, CA: Allen, Shea & Associates.
Retrieved August 20, 2010 from http://www.migrats.org/uploads/CASPIReport.doc and Delin, B.,
Reither, A., Drew, J., and Hanes, P. (2004) Final Project Report: Wisconsin Pathways to
Independence. Menomonie, WI: University of Wisconsin – Stout Vocational Rehabilitation
Institute.
7
Some studies do include information about the quality implementation of benefits counseling at
the project level. In fairness, this study suffers from the same limitation.
5
were compared to those for two matched groups of VR consumers who had not received
benefits counseling. One study focused on outcomes for consumers across the “full”
range of disabling conditions; the second analysis was restricted to subgroups of
individuals with psychiatric disabilities. 8
In both analyses, those receiving benefits counseling services exhibited much
stronger earnings performance over the eight post-intervention quarters than they had in
the pre-intervention quarters. More importantly, their earnings trends in the postintervention period were much better than those observed for either of the comparison
groups who had not received benefits counseling services. In the analysis including “all”
disability types, mean quarterly earnings for the intervention group increased to
approximately $1100 at the end of the post-intervention period after varying in a range
between $530 and $660 during the pre-intervention period. These gains were far larger
in both absolute and percentage terms than those for the two comparisons groups.
However, both papers show that gains for those in the intervention group were
disproportionately concentrated in the quarters immediately following the start of the
intervention. More than 50% of the increase in mean earnings occurred in the first full
quarter subsequent to study entry. In the case of the “all disability types” sample,
earnings gains continued, albeit at a slower pace. However, the psychiatric disabilities
subgroup attained their peak earnings in the third quarter and then began to experience
a decline. 9 The strong increase in outcomes in the quarter following initiation of services
raises an important question that is not examined in the Vermont papers. Are the
outcomes driven primarily by benefits counseling or does the service operate principally
as an effective means for individuals with severe disabilities to achieve goals they had
already embraced? The answer to this question is potentially important for deciding for
whom and especially when currently limited service capacity should be used and thus is
one area we explore in this paper.
A recent study utilizing Connecticut data examined the impact of benefits
counseling on earnings and employment for a sample of approximately 5700 consumers
divided into three groups: closed vocational rehabilitation cases who did not receive
benefits counseling services, closed VR cases who did and consumers who received
benefits counseling, but had not been served by the VR agency in the relevant period. 10
8
Tremblay, T., Smith, J., Xie, H., and Drake, R. (2004) “The Impact of Specialized Benefits
Counseling Services on Social Security Administration Disability Beneficiaries in Vermont,”
Journal of Rehabilitation, 70 (2) pp. 5-11; Tremblay, T., Smith, J., Xie, H., and Drake, R. (2006)
“Effects of Benefits Counseling Services on Employment Outcomes for People With Psychiatric
Disabilities,” Psychiatric Services, 57 (6) pp. 816-21.
9
This result is not explicitly identified in either paper, but can be discerned by looking at the
figures provided in the articles. For example, line sighting the intervention groups’ mean earnings
in figure 1 of Tremblay et al. (2004) one notices that there is an approximately $300 increase
between the final pre-intervention quarter and the first post-intervention quarter. This constitutes
about 60% of the roughly $500 growth between the final pre-intervention and post-intervention
quarters. The proportion of peak earnings growth for the intervention group examined in
Tremblay et al. (2006) appears to be greater (roughly 75%). See figure 1 in that paper.
10
The benefits counseling only group received services through the Social Security
Administration funded WIPA program or its predecessor, the BPAO.
6
Everyone in the sample received benefits through at least one Social Security disability
program. However, unlike in Vermont, those getting benefits counseling did not explicitly
volunteer for a Social Security sponsored demonstration project. Samples were
identified administratively. Thus, the Connecticut study has the virtue of examining a
disability program population more likely to be representative of SSDI beneficiaries and
SSI recipients than either the Vermont or, for that matter, the Wisconsin based studies. 11
The findings from Connecticut were broadly similar to those reported from
Vermont, but exhibited some important differences. In most of the comparisons of
average earnings or the proportions having non-zero earnings, the two groups getting
benefits counseling performed better than the group that did not. Among individuals
receiving benefits counseling services, those who had received VR services as well as
benefits counseling exhibited slightly better outcomes. As in the Vermont, gains were
concentrated in the period immediately following the start of the intervention. However,
the scale and persistence of gains was substantially less than those associated with the
Vermont SPI project. The authors suggested that the likely explanation for the more
modest outcomes were differences in sample characteristics related to project goals and
eligibility requirements.
In addition to reviewing the literature about the quantitative relationship between
benefits counseling and employment outcomes, we were interested in reviewing
literature that identified reasons why benefits counseling would be expected to support
better employment outcomes. We did not find a fully articulated intervention theory that
might guide both further research and efforts to improve service delivery. Still, many of
the building blocks that could be used for developing a formal intervention theory appear
to be in place.
Multiple observers claim that benefits counseling works because of the quality of
information provided. Good information supports informed decision making by
consumers; i.e., allowing them to pursue their goals however defined. 12 One observer
hypothesized that good information about available work incentives and when they might
be useful given a consumer’s specific goals and circumstances is particularly important.
That is, the positive impact of benefits counseling might result from increased use of
appropriate work incentives. 13 Indeed, discussions of practice domains in the literature
posit that benefits counseling involves the application of information to individual needs,
preferences, and circumstances. Consequently, benefits counseling may also be
effective to the extent it includes an interactive process between the consumer and the
professional that allows identification and clarification of consumer needs and goals. 14
11
Gruman, C., Shugrue, N., Kellett, K., Robison, J., and Porter, A. (2010) Medicaid Infrastructure
Grant: The Impact of Benefits Counseling and Vocational Rehabilitation on Earnings Farmington,
CT: University of Connecticut Health Center.
12
Golden, et al., (2000) p. 2. Also see Kregel, J., (2009) “Work incentive planning and
assistance: Assisting beneficiaries to obtain employment and reduce dependence on SSA
benefits,” Journal of Vocational Rehabilitation 31 (1), pp.1-9.
13
Personal communication from Susan O’Mara, Director of the WIPA National Training Center,
Virginia Commonwealth University, August 25, 2010.
7
Descriptions of effective benefits counseling also stress the need for trust
between the consumer and the professional. This is especially so when the consumer
faces potential loss in eligibility, the level of benefits received, or in time and peace of
mind if there are disputes with government entities over the benefits received or
rescinded due to work activity. 15 Trust is said to arise from consumer confidence in the
benefits professional’s competence and commitment to act in the consumer’s interest.
However, it is possible that trust has other dimensions, for instance in reducing fears that
work activity will result in harm. 16 It is also possible that exposure to benefits counseling
may increase consumers’ capacities to plan or to increase their confidence in their
abilities to make and implement decisions, thereby increasing their willingness to risk
employment or increasing earnings.
We think that a work group convened by the MIG-RATS (Medicaid Infrastructure
Grant Research Assistances to States) made significant progress toward specifying a
nuanced intervention theory. This work group created a visual representation of the likely
relationships between benefits counseling and both proximate and longer term
outcomes, identifying a chain of influences and in some cases the expected direction
(positive or negative) of outcomes. The work group also specified partial models that
researchers wanting to concentrate on a limited part of the overall model and/or facing
data limitations might use. 17 However, the work group never developed justifications for
why the model is ordered as it is. By not doing so, the group produced something less
than an intervention model. Nonetheless, we have found the visual diagrams useful for
helping to organize our thinking. Our analysis has certainly been influenced by what the
MIG-Rats work group termed the “Abbreviated Employment and Income Model.” Note
that this model omits intermediary conditions such as “understanding work incentives,”
“barriers to work,” and “informed decision making” that, in the full model, are located
between benefits counseling and the other elements in the abbreviated model shown in
figure 2. Given our emphasis on the direct effects of dosage on employment outcomes,
we think the use of the abbreviated model is appropriate.
14
Golden et al., (2000) pp. 2-5. and Lui, J., Chan, F., Lin, C., Anderson, C. and Peterson, M.
(2010) “Roles and functions of benefits counseling specialists: A multi-trait analysis,” Journal of
Vocational Rehabilitation 32 (3), pp. 163-73 (especially p. 171).
15
Kregel (2009) p. 2, Golden et al. (2000) p.2. and Lui, J. (2009) “An Emerging Profession Disability Benefits Specialists, “ Rehabilitation Counselors and Educators Journal 3 (2) pp. 48-54
(especially p. 52)
16
17
Personal communication from Susan O’Mara, August 25, 2010
This work group met in 2007-08. This discussion is based on a document distributed within the
MIG-RATS work group in September 2008, “Guidelines for Using Benefits Counseling Outcomes
Models.” The work group did not meet after this date. Though there was some external
distribution of this product, it must be considered a draft.
8
Figure 2: MIG-RATS Abbreviated Employment and Earnings Model 18
Abbreviated Employment
and Income Model
+
#1
Benefits
Counseling
#4a
Use of
Work Incentives
+
+
#5b
Cash Benefit
Program
Participation
+/?
?
-/?
#4b
Employment
#6b
Total
Disposable
Income
?
+
#5c
Earnings
from Work
Benefits Counseling and the WI SPI and SSDI-EP Projects
Both the WI SPI and the SSDI-EP were largely operated and designed by the
Pathways to Independence unit housed in what is now the Wisconsin Department of
Health Services. 19 Both projects were sponsored by the Social Security Administration
with the purpose of encouraging those on disability benefits to “return to work.” Both
projects were organized in a similar fashion with roughly twenty community based
(“provider”) agencies enrolling and serving participants and the Pathways office
providing project co-ordination, including providing or arranging for the provision of
training and technical assistance.
WI SPI operated between July 1999 and September 2004. The SSDI-EP started
in summer 2005 and officially continued through the end of 2008. 20 Participants in both
projects were volunteers. However there were differences in eligibility requirements, the
most important of which was, that while WI SPI participants could be attached to either
or both Social Security disability programs, eligibility in the SSDI-EP was restricted to
18
“Guidelines for Using Benefits Counseling Outcomes Models” (2008) pp. 5-6.
19
Wisconsin Division of Vocational Rehabilitation had a significant role in the design and
management of WI SPI. SSA mandated some of the critical features of the SSDI-EP (e.g., the
offset feature and the basic eligibility rules) but left decisions about project structure and service
delivery to Pathways.
20
Some members of the SSDI-EP treatment group retained their ability to use the offset provision
after this date, necessitating residual capacity to track their work activity and to provide benefits
counseling as needed.
9
SSDI beneficiaries who received benefits based on their own work histories and had no
attachment to the SSI program. Finally, SSDI-EP participants were randomly assigned to
a treatment and control group, whereas WI SPI participants were not.
Work incentive benefits counseling was a critical service element in both
projects. Participants were expected to have access to as much benefits counseling as
they needed to pursue their employment goals. 21 In WI SPI, benefits counseling was
supposed to be fully integrated into a consumer centered vocational planning process. In
the SSDI-EP, integration was encouraged, but not required. In both projects, training and
technical assistance for benefits counseling came from a common source and the
continuities in content and rigor were far stronger than the differences. 22 Counselors’
initial training required approximately fifteen days and all active counselors received
follow-up training on a quarterly basis. Benefits reviews produced by inexperienced
benefits counselors were subject to review by staff at the training entity.
Nonetheless, there were important differences between the projects in
participants’ receipt of benefits counseling services, a fact that complicates comparisons
between the projects. Those in the WI SPI sample received an average of thirty-nine
hours of benefits counseling in the nine quarters we look at, compared to slightly over
eight for the SSDI-EP sample. Though benefits counseling delivery was front loaded in
both projects, those in WI SPI received almost three times as much in the enrollment
quarter and that which immediately followed (Q0-Q1). In the WI SPI only about 4% of
participants had no reported hours of benefits counseling over the Q0-Q8 period
compared to 21% during the SSDI-EP. By contrast, it is likely that only a handful of WI
SPI participants received benefits counseling prior to entering the project, while about a
third of SSDI-EP participants reported having received some prior service. 23
21
However, the provider agencies were in the strongest position to make decisions about what
constituted the upper limits of service provision for individual cases.
22
During WI SPI, Employment Resources Incorporated (ERI) provided training and technical
assistance, generally in the tradition developed at Cornell University. As an outgrowth of the SPI
project, Pathways helped to organize a permanent training and TA capacity housed at a
consortium called the Wisconsin Disability Benefits Network (WDBN). WDBN was housed at ERI
and, with some modification, continued what ERI had developed.
23
In 1999, the SSA sponsored BPAO program had not yet begun. Within Wisconsin only four
organizations offered any benefits counseling services. However three of these became provider
agencies for the project. It is reasonable to suppose that some small number of WI SPI
participants had received services prior to project entry. By contrast, by the start of the SSDI-EP,
there were many more sources of benefits counseling services in Wisconsin. It is not surprising
that a third of participants had some exposure to the service. It is quite probable that our figure is
an underestimate as it only captures reports of benefits counseling at the provider agency where
a participant enrolled or imputes such receipt for those who had participated in WI SPI (about 4%
of SSDI-EP participants). However, as many participants enrolled at provider agencies where
they had already received services, we believe that the error is not large.
10
Table 1: Receipt of Benefits Counseling Services, WI SPI and SSDI-EP Participants
WI SPI (n=443)
SSDI-EP (n=468)
Benefits Counseling,
38.7
Mean Hours,
Q0 through Q8
Benefits Counseling, Mean
11.1
Hours, Q0-Q1 (“initial”)
% Receiving No Benefits
4.3%
Counseling Services
% with Reported Benefits
0%
Counseling Prior
to Study Entry
Source: WI SPI and SSDI-EP Encounter Data
8.2
4.1
21.2%
35.3%
We think it likely that these differences in service provision were rooted in
differences in how service provision was funded. During the first years of the WI SPI
project, provider agencies received a capitation meant to cover costs for each participant
that was negotiated between each agency and project management. Management
wanted the capitations to be generous enough to encourage provider agencies to remain
in the project and to fully implement the service approach. At the least, the capitation
rates were generous enough to provide for what, in comparison to the SSDI-EP, can be
viewed as generous service provision. 24 There was no specific funding source for
service provision in the SSDI-EP. Provider agencies were expected to find funding to
meet each participant’s service needs. In practice, Pathways was always available to
pay for benefits counseling services, though in the first years of the project not all
agencies took full advantage of this option. 25
Samples
Our analyses are limited to the WI SPI and SSDI-EP participants (433 and 468,
respectively) with nine quarters of both project service and Wisconsin Unemployment
Insurance (UI) earnings data starting with the calendar quarter in which enrollment
occurred. In addition, we have access to UI data for all these participants for the four
quarters prior to the enrollment quarter (Q-4 through Q-1) to look at employment
24
We have already noted that the mean amount of benefits counseling services for a WI SPI
participant was almost five times greater than for a SSDI-EP participant. Over the same nine
quarter period a WI SPI participant received fifty-nine hours of vocationally related services
compared to a little over nine hours, on average for a SSDI-EP participant (almost a 7:1 ratio).
This does not include VR funded services provided by other vendors (including other units within
the provider agency). Given the differences between the project samples in the likelihood of being
a VR consumer, we would expect differences to again favor WI SPI participants. Nonetheless,
other factors than how the projects were organized and funded probably had a role in motivating
observed differences. For example, when there is a larger proportion of participants classified as
having a cognitive/developmental disability (as in WI SPI) there is likely to be more utilization of
some high intensity services, especially those associate with supported employment.
25
Failure of participants to receive services were more often associated with other factors such
as attrition of benefits specialists, training delays, and, quite likely, participants’ failure or
reluctance to ask for services.
11
outcome trends in the year preceding entry to either WI SPI or SSDI-EP. Given that
participants in both projects entered over fairly long periods of calendar time, analyses
are always referenced to the calendar quarter of project entry. 26
Table 2 provides information about selected demographic, experiential and
program participation characteristics for the two samples. Many of the differences
between the WI SPI and the SSDI-EP samples reflect the inclusion of SSI recipients and
of SSDI beneficiaries who got benefits based on another’s earnings history in the WI SPI
project. Even so, a large proportion of the WI SPI sample (72%) had some form of SSDI
eligibility prior to study entry. Thus, these individuals were subject to losing all of the
cash benefit associated with that program should their earnings exceed SGA. Advocates
for work incentive benefits counseling would contend that this possibility would increase
the potential need for and value of benefits counseling services.
26
The WI SPI sample is limited to participants who entered in the first twenty-five months of the
project. There were some differences between these and later enrollees. In particular, early
enrollees were more likely to be employed at study entry. The SSDI-EP sample is drawn from the
full fifteen month enrollment period.
12
Table 2: Selected Demographic, Experiential, and Program Participation Characteristics
of WI SPI and SSDI-EP Participants
WI SPI (n=443)
SSDI-EP (n=468)
Age
Mean
38.2
45.2
Median
39.0
46.0
Gender
Female
37.4%
44.7%
Male
62.6%
55.3%
Race/Ethnicity
Non-White
22.3%
12.8%
White
77.7%
87.2%
Hispanic
3.6%
3.2%
(includes non-white &
white)
Education
Less than High School
16.5%
5.6%
Completion
High School Completion
30.2%
27.1%
(including GED)
More than High School
53.3%
67.3%
Completion
Disability Categories 27
Cognitive/Developmental
23.3%
7.4%
Disabilities
Affective/Mental Health
27.8%
36.3%
Physical (including Sensory
49.0%
53.5%
& HIV)
Other
--2.7%
28
SSA Disability Program
SSDI (including concurrent)
72.2%
100%
SSI (only)
27.8%
0%
Medicaid Buy-in
Participation, Q0-Q8
Yes
12.6%
51.1%
No
87.4%
48.9%
Source: WI SPI and SSDI-EP Encounter and Survey Data, Administrative Data from WI
Department of Health Services, WI Division of Vocational Rehabilitation, and the Social
Security Administration
Note: All percentages calculated based on available cases
27
The process of assigning cases to disability categories differed between studies. For WI SPI,
RSA 911 disability codes were recoded into the three categories by the evaluators and were
supplemented by survey data. For SSDI-EP, provider agency staff made the assignments based
on instructions from project management at Pathways.
28
The information in table 1 reflects program status prior to WI SPI enrollment. Any period of
inclusion in the SSDI program in the two years prior to enrollment resulted in a coding of SSDI.
13
We have already identified some differences between the WI SPI and SSDI-EP
in project eligibility criteria. These are not insignificant. A project restricted to SSDI
beneficiaries will probably have a higher proportion of participants with prior work
experience and higher levels of human capital development. Moreover, a project
restricted not only to SSDI beneficiaries, but also excluding those getting concurrent SSI
benefits or SSDI benefits based on a parent’s earnings record, will generally have a
smaller proportion of individuals with severe cognitive disabilities. The SSDI-EP sample
exhibits these tendencies, relative to the WI SPI sample.
Most importantly, despite some differences in eligibility requirements, both WI
SPI and SSDI-EP participants must be characterized as unusually “work-oriented”
relative to the typical working-age participants in Social Security disability programs. We
borrow this concept from Gina Livermore, who defines the category based on having
work goals or expectations. 29 Livermore found that such individuals were far more likely
than others in Social Security disability programs to report engaging in work related
activity such as job training, searching for work, or being gainfully employed. Though
Livermore’s definition leaves some ambiguity as to whether the concept is primarily
attitudinal or behavioral, we would lean to the latter formulation. That is statements or
measures of goals or intent only indicate “work-orientation” if their presence is strongly
indicative of behavior, either current or in the very near future.
Table 3 provides information about WI SPI and SSDI-EP participants’
employment histories, both since entering benefit status and, particularly, in the year
prior to entering one of the projects. Though volunteering for one of these projects could,
by itself, be viewed as an indicator of a strong orientation toward work, we think the data
documenting a relatively high level of workforce participation and earnings before
enrollment provides even stronger evidence.
According to data from the 2004 National Beneficiary Survey approximately 9%
of those in Social Security disability programs reported employment at the time of the
survey, with about 13% reporting employment during the previous year. 30 By contrast, in
the calendar quarter immediately prior to project entry, 35% of the WI SPI sample was
employed, as indicated by having positive earnings in Wisconsin UI records. The
comparable figure for those in the SSDI-EP sample was 44%. Another indicator of
unusually strong commitment to employment can be seen in the proportions of our
samples who reported working since going on either SSDI or SSI benefits: 67% for the
29
Livermore, G. (2008) “Disability Policy Research Brief Number 08-01: Earnings and Work
Expectations of Social Security Disability Beneficiaries,” Washington, DC: Center for Studying
Disability Policy, Mathematica Policy Research, Inc. p. 2 and Livermore, G. (2009) “Disability
Policy Research Brief Number 09-05: Work-Oriented Social Security Disability Beneficiaries:
Characteristics and Employment-Related Activities.” Washington, DC: Center for Studying
Disability Policy, Mathematica Policy Research, Inc. p. 2.
30
Livermore. (2008) p. 2. In point of fact, participants in the WI SPI and SSDI-EP appear to have
had, at project entry, higher employment rates than those in the work-oriented subgroup that
Livermore identified from the 2004 National Beneficiary Survey (21% at the time of interview, 29%
at some point during the prior year). See Livermore. (2009) p. 2.
14
WI SPI group and 81% for those entering the SSDI-EP. 31 However, as might be
expected for a SSDI only population, average earnings prior to enrollment and human
capital measures like educational attainment (see table 2) were somewhat higher in the
SSDI-EP participant sample than in the WI SPI sample.
Table 3: Selected Employment History Characteristics of WI SPI and SSDI-EP
Participants
WI SPI (n=443)
SSDI-EP (n=468)
67.0%
80.8%
Self-Reported
Employment between
SSA Disability Program
and Study Entry 32
34.8%
43.6%
UI Employment
in Quarter Prior
to Study Entry (Q-1)
49.2%
55.1%
UI Employment in Any
Quarter in the Year Prior
to Study Entry
$608
$894
Mean UI Earnings in
Quarter Prior to Study
Entry (Q-1)
22.6%
27.6%
UI Earnings =/>$1200 in at
least Two Quarters in the
Year Prior to Study Entry
Source: WI SPI and SSDI-EP Encounter and Survey Data, Administrative Data from WI
Department of Workforce Development
Note: All percentages calculated based on available cases
Note: Monetary amounts deflated using CPI-U (August 2005=100)
It also appears that the samples had, on average, somewhat higher earnings in
the period leading up to study entry than another group of employed persons with severe
disabilities in Wisconsin. On average, Wisconsin Medicaid Buy-in participants who would
have been eligible for the SSDI-EP earned about $400 per quarter, considerably lower
than the means for the quarter prior to enrollment (Q-1) for either the WI SPI sample
($608) or the SSDI-EP sample ($894). 33 Additionally, the proportion of WI SPI and SSDI31
The UI employment rate for the WI SPI comparison group in the quarter prior to their nominal
entry date (40%) was also unusually high for a group of persons receiving Social Security
disability benefits.
32
If there were positive earnings reported in WI Unemployment Insurance records for any quarter
in the year prior to study entry, a self-report of non-employment was changed to a report of
employment.
33
The Wisconsin Buy-in, with temporary exceptions, requires employment (though payment can
be through “in kind” services). The quarterly earnings data for two samples of Buy-in participants
was tracked, respectively, from 2005-07 and 2006-08. All amounts are inflation adjusted using the
CPI-U with August 2005 set as the 100 index value. See Delin, B. and Hartman, E. (2010).
“Characteristics of Joint Participants in the Wisconsin Medicaid Buy-in and the SSDI Benefit
Offset Pilot: A Comparison with the Population of SSDI Beneficiaries Enrolled in the Wisconsin
Medicaid Buy-in.” Pittsburgh PA: MIG/DMIE Employment Summit, April 2010. pp.16-19.
15
EP participants who earned well above the mean values was substantial. Roughly a
quarter of both samples had reported earnings of at least $1200 in at least two of the
four quarters preceding study entry.
The two samples are also unusual in that earnings growth begins well before
enrollment with no evidence of any reduction in employment effort on the expectation of
entering a program aimed at improving employment outcomes. The trend lines for mean
UI earnings can be seen in figure 3.
Figure 3: Mean Quarterly UI Earnings Q-4 through Q8, WI SPI and SSDI-EP Samples
Mean Quarterly UI Earnings for WI SPI and SSDI-EP Participants
2000
1800
1600
Quarterly UI Earnings
1400
1200
SSDI-EP
WI SPI
1000
800
600
400
200
0
q4pre
q3pre
q2pre
q1pre
q0
q1
q2
q3
q4
q5
q6
q7
q8
Quarter Relative to Entry Quarter
Source: Administrative Data from WI Department of Workforce Development
Note: Monetary amounts deflated using CPI-U (August 2005=100)
Employment rate trends (not shown) were generally similar, though
understandably less pronounced. These results are notable in that they show that a
large proportion of employment outcome gains, especially for the SSDI-EP sample, took
place before program entry and thus was unrelated to benefits counseling received
during the projects. This is another reason for our interest in seeking to identify factors
that increase use of benefits counseling services as well as seeking further
understanding of (apparent) dosage effects.
It is important to note that the observed differences in mean earnings largely reflect differences in
UI employment rates across groups. However, the average quarterly earnings for those employed
according to UI records were about 25% higher for SSDI-EP participants at project entry. That
difference grew considerably to approximately 65% higher by Q8.
16
Data Sources and Key Variables
Data used in this paper come from multiple sources: government agency
records, participant surveys, and encounter data collected by each project’s provider
agencies. Most of the data elements used were collected in identical or highly
comparable ways during the two studies, but we identify significant exceptions.
Information about employment and earnings are from Wisconsin Unemployment
Insurance records. Though considered accurate within their limitations, there are
important exclusions that result in underestimates of earnings and employment rates.
The most important exclusions, for our purposes, are self-employment and employment
by firms located outside Wisconsin.
Additionally, we use UI earnings data to construct a proxy for individual income.
This proxy is the quarterly sum of UI earnings and Social Security disability program
payments. Though the proxy measure has some important shortcomings, it allows us to
explore the relationship between benefits counseling and income. 34 Indeed, an important
reason that benefits counseling may be important is that marginal increases in earnings
can sometimes leave beneficiaries with far less income. Both UI earnings and the
income proxy are inflation adjusted using the CPI-U (August 2005 =100).
Information about the amounts of benefits counseling services and other project
services were obtained from monthly reports from the community agencies where WI
SPI and SSDI-EP participants were enrolled. Service provision was measured in hours
and reports were not limited to time spent with participants. Monthly reports were
aggregated on a quarterly basis. Definitions for benefits counseling and vocational
service categories were highly consistent across the two studies.
Nonetheless, the raw quarterly benefits counseling dosage variable raised a
difficult issue. Though the magnitude of change in employment outcomes was similar
across the two studies, the mean dosage provided to WI SPI participants over the Q0Q8 period was about five times larger than what was provided to SSDI-EP participants.
Given this, though we hypothesized that an hour of benefits counseling would be
associated with some gain in earnings and perhaps other employment outcomes, it was
clear that the estimates would have very different magnitudes. Therefore, we decided to
reconceptualize dosage differences in a relational framework. Given relatively easy
access to benefits counseling services, what was the effect of differences in dosage
relative to the mean for the project?
For our estimates of the impact of benefits counseling on employment outcomes,
we created a categorical variable based on the standard deviation of Q0-Q8 dosage in
each of the studies. Because the distributions include more cases below the mean than
above it, we used a half rather than full standard deviation score to create the following
dosage categories. The largest dosage category contains those cases that received an
amount of service more than one half standard deviation above the mean dosage for the
project. A middle group is defined by dosages within one half standard deviation of the
34
Most notably, it ignores income produced by other family or household members as well as
excludes some potentially important sources of individual income. Against this, it must be noted
that large proportions of study participants resided in households where they were the sole adult.
17
mean. The lowest dosage group includes those participants, who relative to others in
their project, received a dosage less than the amount that was one half a standard
deviation below the mean. This final group includes those participants who received no
benefits counseling services in their Q0-Q8 periods. Details about these dosage groups
are available in table 4. When estimating the impacts of employment related factors,
particularly those occurring prior to study entry on the amount of benefits counseling
received, we use the actual dosage amounts as the dependent variable.
Table 4: Benefits Counseling Dosage Categories
WI SPI
SSDI-EP
Mean Hours of Benefits
38.7
8.2
Counseling, Q0-Q8
½ Standard Deviation
21.1
6.2
% in High Dosage category
19.0%
20.5%
% in Medium Dosage
49.0%
38.2%
Category
% in Low Dosage Category
32.1%
41.2%
Source: WI SPI and SSDI-EP Encounter Data
Note: Dosage Categories Relative to One Half of a Standard Deviation Above or Below
Mean Dosage
In addition to our overall estimates for the WI SPI and SSDI-EP samples, we are
also interested in looking at two subgroups within the larger samples: (1) those seeking
to enter the labor market either after a long absence or for the first time and (2) those
already employed with relatively high earnings, but seeking to further increase their
earnings. We define the first group as those individuals who did not report any
employment between going on disability benefits and enrolling in the study. In the SSDIEP this data was collected as part of the enrollment process. For WI SPI this information
was collected by survey. Unfortunately, this resulted in missing data for about 9% of the
sample. We defined the relatively high earnings prior to study entry subgroup as those
with quarterly earnings of at least $1200 (inflation adjusted) for at least two calendar
quarters in the year prior to the quarter of project enrollment. Given our relatively small
sample sizes, we use these subgroups as control or independent variables in our
models, rather than running separate models for each subgroup. 35
The other control variables used in the models are split into two types: those
available, though not always in fully equivalent form, for both the WI SPI and SSDI-EP
and those applicable or available for only one of the studies. Variables of the first type
include age, gender, disability type, education, and vocational service hours over the
Q0-Q8 period. 36 Variables exclusive to the WI SPI study are SSA program type and use
of the SSI waiver. Variables exclusive to the SSDI-EP study include prior use of benefits
counseling, the Medicaid Buy-in, study group assignment, TWP completion over the Q035
As these two variables are necessarily strongly correlated, they are never used together in the
same model.
36
Differences in the source data used to construct the disability type variable have already been
discussed. Additionally, there are two variables in this group where there are a noteworthy
proportion of cases without information. Disability type information is missing for 5.3% of the
SSDI-EP sample. The education variable is missing for 1.4% of the WI SPI sample.
18
Q8 period, and actual use of the SSDI benefit offset provision. 37 These study specific
variables are used mainly in the outcomes models that examine the mediating effects of
work incentives.
Methods
Completing this paper required performing two rather different types of analytical
tasks. The first and more important task was to discern the impact of different dosages
of benefits counseling on employment outcomes over the nine quarter analysis period.
The aim was to identify trends over the Q0-Q8 time period as well as to estimate and
explain results at the end of that period. The second task was to account for the amount
of benefits counseling received by participants based on their characteristics, particularly
employment outcomes during the period leading to project entry. Different methods were
chosen to complete each of these tasks. Specific choices reflect data limitations, most
notably limits on the number of available cases and, secondarily, differences in the
magnitude of benefits counseling service delivery between the WI SPI and SSDI-EP
projects.
We estimate dosage impacts for both the combined samples and separately for
each project. We think both approaches have utility in what is in many respects an
exploratory study. The first approach maximizes sample sizes and is defensible given
the close similarity between the projects in both the character of the benefits counseling
provided and overall project structure and goals. Nonetheless, WI SPI and the SSDI-EP
had important differences in their eligibility requirements, use of random assignment,
work incentive features, and, above all, in the magnitude of service provision. Though
defining dosage categories relative to the mean is intended to support useful
comparisons across the two projects, one cannot wish away the possibility that dosage
hours have a direct impact on employment outcomes.
Modeling techniques were chosen based on sample size, though in all cases our
goal was to compare group trends across time. In all cases, we limited these analyses to
the Q0-Q8 period as the goal is to assess impacts for benefits counseling delivered
through the projects. For the analyses including participants from both projects we use a
mixed linear regression model to estimate the impact of three benefits counseling
dosage categories. For the smaller WI SPI and SSDI-EP samples, we use repeated
measures MANOVA (Multivariate Analysis of Variance) to estimate impacts. This
technique has the advantage of allowing us to run time series models with multiple
control variables despite relatively small sample sizes. Both types of techniques support
estimation of differences across time (within participant) and between groups.
While readers can be expected to be familiar with the interpretation of regression
statistics, it may be useful to provide some particulars about repeated measures
37
By definition, there are no SSI recipients in the SSDI-EP sample. Similarly, the study
assignment variable does not apply to WI SPI. Though it is possible that a handful of WI SPI
participants received benefits counseling before that project, it would not have been a large
enough proportion to include in our regression models. Similarly, as only 12% of the WI SPI used
the Medicaid Buy-in in their Q0-Q8 participation period, we chose not to use that variable in
estimating outcomes for that study. Finally, though it appears that some information about TWP
completion had been collected during WI SPI, for whatever reason it was not retained in the
deidentified data set from that project.
19
MANOVA. The significance of a variable for between subject comparisons is a
straightforward probability value. The significance of a variable for within subject
comparisons is the probability value for the wilks’ lambda statistic. In both cases, we use
the standard .05 level to denote statistical significance. The effect size of the variable
(i.e. the amount of variation explained) is estimated by the partial eta squared for both
within subject and between subject effects. This is estimated separately for each effect
type.
MANOVA involves some disadvantages relative to regression techniques. Most
notably, independent variables have to be categorical. 38 As a consequence, some of the
information available when a variable is in continuous form is lost. An additional
challenge is that MANOVA does not produce a direct equivalent to the beta coefficients
available from regression analyses. Though it is still possible to identify the rate of
change over a particular time period, this needs to be separately calculated using the
categorical (marginal) means.
In the process of conducting our analyses, we discovered circumstances when
an independent variable such as benefits counseling dosage was not significant though
the categories were seemingly associated with substantial differences in employment
outcomes over time. This suggested the possibility of there being important mediator
variables. We explore this possibility using standard OLS linear regression techniques,
though we restrict these analyses to the use of work incentives.
Finally, we also use linear regression techniques to explore the role that preenrollment conditions had on the total hours of benefits counseling services provided
through the projects over the Q0-Q8 period. These models also include a range of
demographic and disability control variables and, for the SSDI-EP sample, information
about the receipt of benefits counseling prior to enrollment. Regressions are performed
stepwise and final models are limited to variables that were statistically significant at the
.05 level.
Findings: Benefits Counseling Impacts
All analyses of the impact of the amount of benefits counseling service over the
Q0-Q8 period utilize a categorical measure of dosage relative to the mean dosage in the
project any given case participated in. This is done to adjust for the considerable
difference in the mean dosage values between WI SPI and the SSDI-EP. Dosage is
standardized into three categories, based on whether the amount of benefits counseling
provided was within one half standard deviation of the project mean.
Models for both the combined sample and those for each of the two projects use
a common set of control variables. Conceptually, the most important of these is a
dichotomous variable intended to distinguish participants based on having relatively
strong employment outcomes in the year prior to entering either WI SPI or the SSDI-EP.
Inclusion in the stronger outcome group is based on having two calendar quarters of
(inflation-adjusted) UI quarterly earnings of at least $1,200 during this period. Our
38
MANOVA allows multiple independent variables. While independent variables must be entered
into the model in categorical form, other covariates can be entered as continuous variables.
20
expectation is that strong employment outcomes prior to entering a “return to work”
program would be positively correlated with post-entry outcome levels.
Other control variables include gender, age, disability type, and education.
Excepting age, all of these variables are categorical. Disability type variables include 1)
physical/HIV/sensory disabilities, 2) cognitive/developmental disability, 3)
affective/mental disability, and 4) others. Education attainment is measured in three
categories: 1) having less than a high school diploma/GED, 2) high school (or GED)
completion, and 3) having any post-secondary education.
Three employment outcomes are examined in both the combined and separate
sample analyses: mean quarterly earnings, quarterly employment rates, and the
quarterly mean for the individual income proxy variable. We highlight the results for
earnings relative to the other outcomes. Our reasons are as follows: Employment, in
itself, has essentially no bearing on continued eligibility for Social Security disability
programs. While benefits counseling may encourage employment, the primary purpose
of the service is to enable consumers to earn more without loss of eligibility and/or
reduction in income. Though, given this, income could be seen as more important, from
a consumer’s perspective, than earnings, the inherent limitations of our income variable
suggest it would be imprudent to focus over much on those estimates.
Combined Sample Results
We use a mixed linear regression modeling approach to analyze the effects of
the predictor variables (benefits counseling, pre-enrollment earnings, and the four
demographic variables) on earnings, employment, and income. The following
specification is for the earnings model; except for the outcome variable, the employment
and income models have identical specifications.
Earnings = β 0 + β 1 (Quarter ) + β 2 (HalfSDBCCa tegory ) + β 3 (Pr eEarningsC ategpry )
+ β 4 (Quarter * BenefitsCo unseling ) + β 5 (Quarter * Pr eEarnings )
+ β 6 (Quarter * BenefitsCo unseling * Pr eEarnings ) + β 7 (Gender ) + β 8 ( Age)
+ β 9 ( Education ) + β 10 (Cognitive / DevelopmentalDisabil ity ) + β 11 ( Affective / MentalDisa bility )
+ β12 (OtherDisability ) 39
Beta coefficients should be interpreted as unit changes in amounts for both the earnings
and income models and as changes in percentage points for the employment model. As
these models are linear, the changes associated with any quarter are an average of
changes across the Q0-Q8 period and thus may oversimplify actual trends. Graphs of
the estimates for these models are available in Appendix A. We begin our discussion of
findings for these models by looking at earnings.
This earnings model looks at the influence of the predictor variables on earnings
from the enrollment quarter (Q0) up through eight quarters following enrollment.
Benefits counseling did not have a significant influence on earnings at the enrollment
quarter, but did influence changes in earnings over time. As expected, those who had at
39
The physical/sensory/HIV disability category serves as a reference category for the other three.
21
least two quarters of earnings of at least $1200 during the four quarter pre-enrollment
period did have significantly higher earnings during the enrollment quarter than
participants who did not earn as much during the four quarters prior to the enrollment
quarter. Gender, age, and disability type also had an overall influence on earnings, but,
unexpectedly, education did not. 40 Table five provides the p-values and dollar estimates
of the impacts of each of these predictor variables.
Table 5: Estimates of Effects on UI Mean Earnings for Combined Samples
Estimates of Effects in Mixed Linear Earnings Model (n = 911)
Parameter
Beta
Std. Error P-Value
Intercept
564.3
178.90
0.0016
Quarter
-14.1
22.69
0.5348
Benefits Counseling
96.7
57.58
0.0931
PreEarnings
1700.5
229.41 < 0.0001
Quarter * Benefits Counseling
34.3
11.42
0.0027
Quarter * PreEarnings
11.3
45.02
0.8016
Benefits Counseling * PreEarnings
48.8
110.90
0.6602
Quarter * Benefits Counseling * PreEarnings
-41.9
21.87
0.0557
Gender (reference = female)
97.9
49.64
0.0487
Age
-10.5
2.29 < 0.0001
Education
54.4
34.67
0.1165
Cognitive/Developmental Disability
-404.8
69.69 < 0.0001
Affective/Mental Disability
156.0
52.94
0.0032
Other Disability
-1197.4
211.72 < 0.0001
One way to interpret these results is to visualize the influence of each predictor
variable on earnings. To do this we need a starting place. For simplicity sake, let’s start
with individuals who are female, have a physical/HIV/sensory disability, are age
seventeen, and had less than two of four pre-enrollment quarterly earnings of at least
$1200. By adding the beta coefficient of each of these variables to the intercept value
one can calculate the Q0 mean earnings estimate. These individuals had predicted
enrollment quarterly earnings of $386.
Utilizing the coefficients in table 5, one can estimate changes from our initial
example reflecting differences in participant characteristics and time in the project. For
individuals with pre-enrollment earnings of at least $1200 in at least two quarters, their
enrollment quarter earnings are predicted to be $1701 more, adding up to $2086 in
baseline (Q0) earnings. If a participant was male, enrollment quarter earnings are
predicted to be $98 higher. For every one year increase in participant age at enrollment,
enrollment quarter earnings are predicted to be $11 less. Enrollment quarter earnings
are predicted to be $405 less for people with cognitive/developmental disabilities, $156
40
Though we do not explore this anomaly in this paper, we speculate that this result is a
reflection of the relatively low employment rates for participants classified as having a physical
disability. While participants classified as having physical disabilities tended to have greater
educational attainment and higher earnings when employed than other participants, mean
earnings in these projects appear to be strongly influenced by employment rates. Indeed, the
expected benefits of educational attainment on earnings appear in the income models, where
greater educational attainment and the higher earnings associated with it increased the
probability of SSDI eligibility and, when participating in that program, the size of the cash benefit.
22
more for people with affective/mental disabilities, and $1198 less for people with other
types of disabilities relative to that for persons in the physical/HIV/sensory disability
category.
Most pertinent to our analysis, the amount of benefits counseling positively
influences earnings growth over the post-enrollment period. The variable of interest is
the interaction between the dosage category and time (Quarter*Benefits Counseling)
available. For those individuals who received a benefits counseling dosage that was less
an amount one half a standard deviation below the mean for their project, estimated
mean earnings increased $34.30 each post-enrollment quarter. Individuals who
received an “average” amount of benefits counseling (within ½ a SD of their project
mean) increased their earnings by an estimated $68.60 per quarter. The largest
increases in earnings are found for those individuals who received dosages more than
one half standard deviation above their project mean over Q0-Q8. Their earnings
increase an estimated $102.90 per post-enrollment quarter. Consequently, those in the
medium dosage group gain an estimated $274 in quarter eight earnings relative to the
low dosage group. Those in the high dosage group gain about $549 relative to the low
dosage group. When differences in total earnings are calculated for the full period the
differences are even greater. The estimated relative difference in cumulative earnings
over the Q0-Q8 period between the low and medium dosage groups is nearly $1235.
The comparable difference between the low and high dosage group earnings is about
$2470.
In the employment model, the predictor variables exhibit a similar pattern of
influence on the outcome as they did for earnings. Nevertheless, there are key
differences. Education has a statistically significant but negative impact on Q0
employment rates. Employment rates did not vary between those with
physical/HIV/sensory, cognitive/developmental, and other disabilities, although those
with an affective/mental disability did have higher employment rates to match their
higher earnings found in the previous model. Finally, there is an overall predicted
decrease in employment during the post-enrollment quarters.
However, this downward trend in employment rates is actually offset for those
individuals in the middle and high benefits counseling dosage categories. Members of
both these groups display gains in their estimated employment rates. The following
table provides the p-values and beta estimates for the mixed linear employment model.
23
Table 6: Estimates of Effects on UI Employment Rates for Combined Samples
Estimates of Effects in Mixed Linear Employment Model (n = 911)
Parameter
Beta
Std. Error P-Value
Intercept
0.4831
0.0462 < 0.0001
Quarter
-0.0146
0.0059
0.0137
Benefits Counseling
0.0107
0.0152
0.4802
PreEarnings
0.4602
0.0605 < 0.0001
Quarter * Benefits Counseling
0.0130
0.0030 < 0.0001
Quarter * PreEarnings
0.0065
0.0117
0.5771
Benefits Counseling * PreEarnings
0.0299
0.0292
0.3066
Quarter * Benefits Counseling * PreEarnings
-0.0143
0.0057
0.0124
Gender (reference = female)
-0.0471
0.0126
0.0002
Age
-0.0035
0.0006 < 0.0001
Education
-0.0316
0.0088
0.0004
Cognitive/Developmental Disability
0.0217
0.0178
0.2223
Affective/Mental Disability
0.1757
0.0135 < 0.0001
Other Disability
0.0797
0.0538
0.1386
Like the mixed linear earnings model, the mixed linear employment model
estimates employment rates based on adding the coefficients for the relevant predictor
variables to the intercept value. For a starting point, let us assume a participant is
female, seventeen years old, has a less than high school education, a disability type
other than affective/mental, and did not have earnings of at least $1200 for at least two
of the four pre-enrollment quarters. The estimated employment rate at the enrollment
quarter for participants who matched this description is 38.3%.
Had we identified a group that was identical, except for having earnings of a least
$1200 in at least two of the four pre-enrollment quarters, there would have been a fortysix percentage point increase in the Q0 employment rate to 84.3%. By contrast, if the
only difference in participant characteristics had been being male rather than female, the
estimated employment rate for the enrollment quarter would have been nearly five points
lower. The estimated employment rate was also 0.4 percentage points less for every one
year older a participant was at enrollment, but a substantial 17.6 percentage points more
for participants with an affective/mental disability (relative to the reference physical
disability group). Somewhat surprising were lower estimated employment rates for
individuals with more educational attainment, about three percentage points lower at
project entry for individuals with a high school education and over six percentage points
lower for individuals with at least some post-secondary education.
Similar to the earnings model, the employment model predicted a larger growth
in employment rates during the post-enrollment quarters if the participant were included
in either the medium or high benefits counseling dosage groups for their project. Those
who received a dosage within one half standard deviation of their project mean exhibited
an estimated 1.1 percentage point gain in employment rates each quarter, whereas
those who received a dosage at least one half standard deviation above their project
mean had an estimated quarterly increase of almost two and a quarter percentage
points in their group’s employment rate. Those who received dosages of benefits
counseling less than one half standard deviation below their project mean faced an
estimated decline in employment rates of 0.2 percentage points per calendar quarter
24
over the Q0-Q8 period. Thus by Q8, the regression estimated employment rate for the
high dosage group had grown almost twenty percentage points relative to the estimated
rate for the low dosage group.
The final model utilizing the samples from both projects estimates the income
proxy variable. Overall results for this model are again very similar to those for the
earnings model. Still, there are differences of note. First, those with affective/mental
disabilities had income levels at enrollment that were not significantly different from
those for the physical/HIV/sensory disabilities group. Recall that in the earnings model
the beta coefficient for the affective/mental disability group was significantly higher than
that imputed for the “PD” reference group. Second, education did predict differences in
income, whereas it did not predict differences in earnings. Unlike in the other two
models, the higher an individual’s educational attainment category, the higher her
estimated income. Third, in the income model, older participants have higher estimated
incomes. By contrast, the earnings model predicts that older participants will have lower
earnings than younger ones. Finally, as one would expect, income estimates were
higher than earnings estimates. The predicted estimates and p-values for these
estimates are displayed in table 7.
Table 7: Estimates of Effects on UI Mean Income for Combined Samples
Estimates of Effects in Mixed Linear Income Model (n = 911)
Parameter
Beta
Std. Error P-Value
Intercept
1425.7
191.39 < 0.0001
Quarter
-26.4
22.72
0.2456
Benefits Counseling
52.2
60.52
0.3883
PreEarnings
1456.9
241.60 < 0.0001
Quarter * Benefits Counseling
36.7
11.54
0.0015
Quarter * PreEarnings
4.3
45.30
0.9252
Benefits Counseling * PreEarnings
111.6
116.70
0.3392
Quarter * Benefits Counseling * PreEarnings
-36.3
22.04
0.0995
Gender (reference = female)
364.5
53.30 < 0.0001
Age
15.1
2.46 < 0.0001
Education
314.8
37.17 < 0.0001
Cognitive/Developmental Disability
-680.1
74.63 < 0.0001
Affective/Mental Disability
-1.1
56.76
0.9844
Other Disability
-997.4
227.58 < 0.0001
As in the earnings model, these beta estimates can be interpreted as changes in
dollar amounts. For participants who are female, have either physical/HIV/sensory or
mental disability, were age seventeen at enrollment, and had earnings of less than
$1200 in three or more of their pre-enrollment quarters, the Q0 estimate for mean
income is just over $1997. For participants with at least two pre-enrollment quarters with
at least $1200 in earnings, estimated income for the enrollment quarter was $3454; that
is, $1457 above that for participants with our “reference” characteristics. Being male
increased the Q0 estimate by $365 compared to participants who were otherwise the
same except being female. Each additional year of age at enrollment was associated
with an additional $15 increase in estimated income. Education also increased
enrollment quarter income, with an estimated addition of $315 for those participants who
ended their education with a high school diploma (or equivalent) and $630 for those who
25
had any education beyond that. Individuals with cognitive disabilities had an estimated
$680 less in enrollment quarter income compared to the average for the
physical/HIV/sensory disability reference group.
Like quarterly mean earnings and employment rates, the rate at which income
estimates changed over the eight post-enrollment quarters depended on the amount of
benefits counseling the participant received, relative to others in their project, during this
period. For those individuals who received dosages of benefits counseling that did not
even reach within a half standard deviation of their project mean, income growth
appears to have averaged less than $37 per quarter. Quarterly growth was higher for
individuals who received an “average” amount of benefits counseling (within one half
standard deviation of their project mean); the rate of increase is estimated at $73 per
quarter. The highest rate of increase was exhibited by those individuals who received at
least one half standard deviation of service over their project mean. Their incomes grew
at an estimated $110 per quarter.
Single Sample Results
The mixed linear models for the combined samples provide support for the
hypothesis that higher dosages of benefits counseling services result in better
employment outcomes. This influence remains even when participant earnings in the
four quarters prior to the enrollment quarter are taken into account. Nonetheless, there
was a large difference in the typical amount of service provided to participants in the two
projects. We now examine the WI SPI and SSDI-EP samples independently to assess
whether there are comparable relationships between relative dosage and the
employment outcomes of interest. Additionally, we look at whether the “pre-earnings”
and demographic control variables exhibit similar patterns of influence as observed for
the combined samples regression models.
As discussed in the methods section, we perform these analyses using repeated
measures MANOVA because of the smaller sample sizes. Although we had to sacrifice
the beta estimates, we are still able to look at relationships both across time (within
participants) and across (between) participants. MANOVA statistics provide both
significance levels and measures of variance (partial eta-squared) explained by each
independent variable. Though pertinent results are discussed in the text below, tables
providing a fuller set of MANOVA statistics are available in Appendix B. We also provide
a series of figures displaying the estimated mean values of an employment outcome
associated with each of the three benefits counseling dosage categories when the
effects of the other predictor variables are accounted for. We supplement the figures
with a series of tables containing estimates of the change in an employment outcome
over the Q0-Q8 study period for each of the MANOVA models displayed below. These
tables are found in Appendix C.
In both the repeated measures MANOVA earnings models for the WI SPI and
SSDI-EP samples, there is a significant between subject relationship between benefits
counseling and earnings, though movement between dosage categories accounts for
only 2% of the variance in earnings. By contrast, pre-enrollment earnings accounts for
about 12% of the variance in earnings. It is of note that none of the demographic
variables are significant predictors of earnings for the WI SPI sample, but gender and
age are significant predictors of earnings for those participating in the SSDI-EP. In this
model, gender accounts for 4% of the within subject variance in earnings and age
26
accounts for 1% of the between subject variance. The following two graphs (figures 4
and 5) provide visual representations of the relationship between the benefits counseling
received and post-enrollment earnings for, respectively, the WI SPI and SSDI-EP
projects.
Figure 4: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) by Benefits
Counseling Dosage Category, Q0-Q8 for WI SPI Sample
Estimated Quarterly Earnings Means during the WI SPI Study
Half SD BC Category
Less than 1/2 SD of
BC Mean (n = 139)
Within 1/2 SD of
BC Mean (n = 214)
Greater than 1/2
SD of BC Mean (n
= 84)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
Quarter Compared to Enrollment
7
8
27
Figure 5: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) by Benefits
Counseling Dosage Category, Q0-Q8 for SSDI Sample
Estimated Quarterly Earnings Means during the WI SSDI EP Study
Half SD BC Category
Less than 1/2 SD of
BC Mean (n = 177)
Within 1/2 SD of
BC Mean (n = 172)
Greater than 1/2
SD of BC Mean (n
= 94)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
There is a little less consistency in the relationship between the benefits
counseling dosage categories and either the employment rate or income outcomes. In
the employment models, this relationship is not statistically significant for the WI SPI
sample, but is for the SSDI-EP sample. The opposite is true for the income models. This
time the relationship is significant for the WI SPI participants, but not for those in the
SSDI-EP. When the relationship is significant, the between subjects effects of benefits
counseling continues to explain about 2% of the variance in employment outcomes.
Meanwhile, the dichotomous pre-enrollment earnings variable explains between 20%
and 23% of the variance in employment rates and either 7% or 8% of the variance in the
income variable.
In both employment rate models, education accounts for about 4% of the
variance in within subject employment rates. Other demographic variables were
significant for only one of the samples. Disability type accounts for about 3% of the
between subject variance in the employment rate for those enrolled in WI SPI, whereas,
for SSDI-EP participants, age proves a significant factor accounting for about 2% of the
between subject variance in rates. The pattern of relationships between the
demographic control variables and estimated income shows even greater disparity than
that observed for the employment rate models. In the income model for the WI SPI
sample, age and disability category each accounts for approximately 1% of between
subject variance. For those in the SSDI-EP, gender accounts for a statistically significant
amount of both the within and between subject variance (4% and 1%, respectively).
28
Finally, education explained about 2% of the between subject variance in the income
models for both samples.
The following four figures provide visual representations of the relationships
between the benefits counseling dosage categories and both employment rates and
mean income. Separate graphs are provided for each combination of project and
employment outcome.
Figure 6: Estimated Quarterly UI Employment Rates by Benefits Counseling Dosage
Category, Q0-Q8 for WI SPI Sample
Estimated Quarterly Employment Rates during the WI SPI Study
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 139)
Within 1/2 SD of
BC Mean (n = 214)
Greater than 1/2
SD of BC Mean (n
= 84)
Estimated Employment Rates
1
0.8
0.6
0.4
0.2
0
0
1
2
3
4
5
6
Quarter Compared to Enrollment
7
8
29
Figure 7: Estimated Quarterly UI Employment Rates by Benefits Counseling Dosage
Category, Q0-Q8 for SSDI-EP Sample
Estimated Quarterly Employment Rates during the WI SSDI EP Study
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n =177)
Within 1/2 SD of
BC Mean (n =172)
Greater than 1/2
SD of BC Mean (n
= 94)
Estimated Employment Rates
1
0.8
0.6
0.4
0.2
0
0
1
2
3
4
5
6
Quarter Compared to Enrollment
7
8
30
Figure 8: Estimated Quarterly Mean Income (Inflation Adjusted) by Benefits Counseling
Dosage Category, Q0-Q8 for WI SPI sample
Estimated Quarterly Income Means during WI SPI Study
5000
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 139)
Within 1/2 SD of
BC Mean (n = 214)
Greater than 1/2
SD of BC Mean (n
= 84)
4500
Estimated Income Means
4000
3500
3000
2500
2000
1500
1000
500
0
0
1
2
3
4
5
6
Quarter Compared to Enrollment
7
8
31
Figure 9: Estimated Quarterly Mean Income (Inflation Adjusted) by Benefits Counseling
Dosage Category, Q0-Q8 for SSDI-EP
sample
Estimated Quarterly Income Means during the WI SSDI EP Study
5000
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 177)
Within 1/2 SD of
BC Mean (n = 172)
Greater than 1/2
SD of BC Mean (n
= 94)
4500
Estimated Income Means
4000
3500
3000
2500
2000
1500
1000
500
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
Single Sample Results for Participant Subgroups Reflecting Selected Pre-Enrollment
Employment Outcomes
The next analyses focus on two subgroups within the WI SPI and SSDI-EP
samples that were chosen to exemplify either relatively strong or weak employment
outcomes prior to entering one of the projects. These analyses focus exclusively on
earnings. Additionally, these analyses utilize the same demographic control variables
used in the MANOVA models already presented. In point of fact, these results are
nothing more than the estimated means for those participants in each sample who also
meet the subgroup criterion.
The first subgroup is composed of persons with relatively strong employment
outcomes. Inclusion in this group is defined by having at least two quarters in the four
quarters prior to the calendar quarter of project entry with UI earnings of at least
$1200. 41 The estimates for mean earnings trends for the WI SPI and SSDI-EP
participants with relatively high earnings in the “pre-enrollment” period are shown in
figures 10 and 11.
41
This is the higher value of the “pre-earnings” variable used in the full sample MANOVA
analyses. Thus, the MANOVA models are identical to those used to estimate quarterly mean
earnings for the full WI SPI and SSDI-EP samples.
32
In the models for both samples, participants identified as having relatively high
earnings before enrollment are also significantly more likely to have higher earnings
during the post-enrollment period than those who did not meet this criterion. This
difference in assignment to one of the pre-enrollment earnings categories accounted for
about 12% of the variance in post-enrollment earnings.
To determine whether the relationship between benefits counseling and earnings
is different for these higher earners, the p-value of the interaction between benefits
counseling and pre-enrollment earnings is examined. For both studies, this interaction is
not statistically significant (p > 0.05). Thus, the relationship between benefits counseling
dosage and earnings is not significantly different for high earners in the year before
enrollment compared to the overall relationship for all the participants in each study.
The following two graphs provide a visual representation of this relationship for the high
earners group. The trends displayed can be directly compared to the ones shown for
the full WI SPI and SSDI-EP samples in, respectively, figures 4 and 5.
Figure 10: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) for High Earners in
Pre-Enrollment Period by Benefits Counseling Dosage Category, Q0-Q8 for WI SPI
Sample
Estimated Quarterly Earnings Means during WI SPI Study
For Participants with At Least Two Pre-Enrollment Quarters with at Least
$1200
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 30)
Within 1/2 SD of
BC Mean (n = 40)
Greater than 1/2
SD of BC Mean (n
= 26)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
Quarter Compared to Enrollment
7
8
33
Figure 11: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) for High Earners in
Pre-Enrollment Period by Benefits Counseling Dosage Category, Q0-Q8 for SSDI-EP
Sample
Estimated Quarterly Earnings Means during the WI SSDI EP Study
for Participants with At Least Two Pre-Enrollment Quarters with at Least
$1200
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 41)
Within 1/2 SD of
BC Mean (n = 54)
Greater than 1/2
SD of BC Mean (n
=33)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
In fact, it is hard not to notice the visual overlap in earnings across the different
dosage categories of benefits counseling that participants in the higher pre-enrollment
earners group received, especially in the results for the SSDI-EP sample. For that
sample, the overlap is especially apparent for participants who received a dosage within
one half standard deviation of the project mean and those who received higher dosages.
When looking at the relationship between benefits counseling and earnings for all SSDIEP participants (figure 5), there is still some, but not nearly as much, overlap. This raises
the question of whether the relationship between benefits counseling dosage and
earnings may be stronger for those with lower earnings in the year prior to project entry.
The earnings trends exhibited in figure 12 are consistent with this interpretation.
There appears to be more differentiation between those in different dosage categories.
Receiving higher dosages is associated with higher earnings levels (though less clearly
with earnings growth). What one does not see for lower pre-enrollment earners group is
a decreasing trend in estimated earning for those who received “below average’
amounts of benefits counseling (i.e., less than ½ SD below the project mean). By
contrast, the decreasing trend was present, beginning in Q3, for high pre-enrollment
earners who received dosages that did not even reach one half standard deviation
beneath the project mean. Still, our overall assessment is that, even in the SSDI-EP, the
amount of benefits counseling received positively influenced post-enrollment earnings
for participants included in both the low and high pre-enrollment earnings groups.
34
Figure 12: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) for Low Earners in
Pre-Enrollment Period by Benefits Counseling Dosage Category, Q0-Q8 for SSDI-EP
Sample
Estimated Quarterly Earnings Means during the SSDI EP Study
for Participants with Less than Two Pre-Enrollment Quarters with at Least
$1200
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 136)
Within 1/2 SD of
BC Mean (n = 118)
Greater than 1/2
SD of BC Mean (n
= 61)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
The second subgroup of interest is composed of participants who had not
participated in the labor market between entering benefit status and enrolling in either WI
SPI or the SSDI-EP. Inclusion in this group is based largely on participant self-report,
though in a few cases persons were removed from this group based on having UI
earnings in one of the four pre-enrollment quarters. The MANOVA earnings model is
identical to the one used for the high pre-enrollment earners subgroup with one
exception. The “pre-earners” variable is replaced by a dichotomous indicator of reported
employment (EmpSinceDis) at any point in time between entering a Social Security
disability program and entry to either WI SPI or the SSDI-EP. 42
Estimated mean earnings for this subgroup are displayed in figures 13 and 14.
For WI SPI participants who did not report a span of employment after entering benefit
status, benefits counseling dosage is significantly related to earnings in the Q0-Q8
period. The dosage category accounts for about 2% of the between subject variance.
However, this relationship is not significant for SSDI-EP participants, with a between
subject p-value of roughly 0.2. However, having no reported employment after entering
Social Security benefit status is both significantly and negatively related to earnings.
Moreover, the variable accounts for the largest amount of between subject variance in
these earnings models, ranging from 4% in the SSDI-EP model to 8% in the WI SPI
model. No demographic variables are significantly related to earnings in the model for
the WI SPI sample, but gender does account for around 4% in the within subject
42
The pre-enrollment earnings variable is removed from this model because of a concern that
including both pre-enrollment employment related variables would cancel each other’s effects.
35
earnings variance and age accounts for almost 1% of the between subject variance in
the model results for the SSDI-EP sample.
Figure 13: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) for Those without
Employment after Entering Benefit Status by Benefits Counseling Dosage Category, Q0Q8 for WI SPI Sample
Estimated Quarterly Earnings Means during the WI SPI Study
for Participants who at Enrollment Had Not Been Employed since Receiving
Social Security Benefits
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 40)
Within 1/2 SD of
BC Mean (n = 70)
Greater than 1/2
SD of BC Mean (n
= 23)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
Quarter Compared to Enrollment
7
8
36
Figure 14: Estimated Quarterly Mean UI Earnings (Inflation Adjusted) for Those without
Employment after Entering Benefit Status by Benefits Counseling Dosage Category, Q0Q8 for SSDI-EP Sample
Estimated Quarterly Earnings Means during the WI SSDI EP Study
for Participants who at Enrollment Had Not Been Employed since Receiving
Social Security Benefits
HalfSDBCSplit
Less than 1/2 SD of
BC Mean (n = 43)
Within 1/2 SD of
BC Mean (n = 27)
Greater than 1/2
SD of BC Mean (n
= 12)
3000
Estimated Earnings Means
2750
2500
2250
2000
1750
1500
1250
1000
750
500
250
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
To determine whether the relationship between benefits counseling dosage is
different for those individuals who were not employed since receiving Social Security
benefits than for other WI SPI and SSDI-EP participants, the interaction between
benefits counseling and having employment since receiving social security benefits was
examined. No statistically significant interaction exists in the models for either sample.
In other words, there is no evidence that the relationship between benefits counseling
and earnings was any different for those who were not employed since receiving Social
Security disability benefits and enrolling in one of the studies.
Finally, we compared the estimated earnings trends for those who had reported
having been employed since entering benefit status to those for the subgroup who had
not reported any span of employment since going on benefits. Though we do not display
results for those participants, in general the earnings differences related to different
amounts of benefits counseling is more pronounced.
Findings: Mediating Effects of Work Incentives
As suggested by the MIG-RATS Abbreviated Employment and Earnings Model
(see figure 2) one explanation for the positive relationship between benefits counseling
and earnings is the increased use of work incentives. Specifically, receiving greater
amounts of benefits counseling may increase the likelihood that an individual in a Social
Security disability programs will use available work incentives. In turn, this increased
37
use of work incentives is hypothesized to increase earnings and (hopefully) income. In
other words, work incentives may mediate the relationship between benefits counseling
and earnings. The following diagram demonstrates this relationship.
Figure 15: Work Incentive Use as a Mediator between Benefits Counseling Use and
Earnings
Use of
Work Incentives
(Mediator)
b
a
Benefits
Counseling
(Independent
Variable)
c
Earnings
(Dependent
Variable)
In order to determine whether the use of work incentives actually mediates the
relationship between benefits counseling and earnings, four things must be true. First,
there must be a positive relationship between benefits counseling and the use of work
incentives (a in the diagram). Second, there must be a positive relationship between the
use of work incentives and earnings (b). Third, there must be a positive relationship
between benefits counseling and earnings (c). Finally, when both benefits counseling
the use of work incentives are included as predictors of earnings in a single model, the
relationship between benefits counseling and earnings should be reduced. 43
The presence of the first three relationships (a, b, and c) is first examined using
Pearson correlations. If three significant and positive relationships are identified, then a
two step linear regression model is used to determine whether the use of work
incentives reduces the relationship between benefits counseling and earnings. In this
two step model, the first model includes only benefits counseling and pre-enrollment
earnings as the predictors of post-enrollment earnings, whereas the second model
included benefits counseling, pre-enrollment earnings, and the use of work incentives as
the predictors of later earnings. Again, the benefits counseling dosage variable has
three ordinal categories based on whether dosage is within one half standard deviation
of the mean dosage for the project in which the participant was enrolled. The variable
used to indicate the level of pre-enrollment earnings also remains the same, dividing
participants based on having earnings of at least $1200 in at least two of the four preenrollment quarters. However, the earnings variable is different. Because the use of a
work incentive did not necessarily coincide with either project enrollment or the full Q0Q8 study period, we designate quarter eight UI earnings as the dependent variable.
These analyses were limited due to differences between the projects in eligibility
requirements, available work incentives, and data availability. To some extent we rely on
proxy variables to indicate the availability, though not necessarily the use, of a given
43
Howell, D.C. (2002) Statistical Methods for Psychology: Fifth Edition. Pacific Grove, CA:
Duxbury pp. 547-77.
38
type of work incentive. As a consequence, analyses are conducted separately for the WI
SPI and the SSDI-EP samples. For the SSDI-EP, the work incentives examined include
trial work period (TWP) completion during the Q0-Q8 post-enrollment period, Medicaid
Buy-in participation during the same period, whether the study participant was randomly
assigned to the treatment group (thus having the opportunity to use the benefit offset
being tested), and actual offset use. For the WI SPI study, the work incentive examined
includes the SSI waiver. 44 Although not a work incentive, we also examine whether
SSDI eligibility (including having concurrent SSI eligibility) mediates the relationship
between benefits counseling and earnings. Having access to SSDI benefits also gives
one access to specific work incentives such as the TWP and EPE.
Only TWP completion and the closely related use of the SSDI benefit offset
provision mediates the relationship between benefits counseling and earnings. Though
the TWP work incentive was available to WI SPI participants who were SSDI
beneficiaries, the relationships described below can only be ascribed to the SSDI-EP
participant sample. 45
There is a significant and positive correlation between the amount of benefits
counseling and TWP completion (r = 0.145; p = 0.001). There is also a significant
correlation between TWP completion and earnings (r = 0.357; p < 0.001). Additionally,
the third condition of having a significant correlation between the amount of benefits
counseling received and earnings (r = 0.167; p < 0.001) is also achieved. Thus the
conditions for using a linear regression model to confirm the mediating effect are met.
When both the amount of benefits counseling and TWP completion variables are
included in the same linear regression model as predictors of quarter eight earnings, the
relationship between benefits counseling and earnings is reduced, with the p-value
increasing from 0.002 to 0.034. The following table compares the results for the models
omitting and including the TWP completion variable.
Table 8: Linear Model Comparison to Check if TWP Completion Mediates the
Relationship between Benefits Counseling and Earnings
Model
Beta
Std. Error P-Value
1
(Constant)
230.8
245.61
0.3479
Half SD BC
388.2
125.98
0.0022
Pre-Enrollment Earnings
1197.0
213.71 < 0.0001
2
(Constant)
85.4
231.4
0.7123
Half SD BC
254.2
119.52
0.0340
Pre-Enrollment Earnings
1157.4
200.78 < 0.0001
TWP Completion Post-Enrollment
1687.2
212.34
< 0.001
Study WI SSDI EP Participant; n = 468; Dependent Variable = Quarter 8 Earnings
44
The SSI waiver did not go into effect until approximately a year and a half following the first
enrollments to the project. Though eligible participants had to explicitly sign-up for this waiver, the
standard 1619 provision would have applied as a matter of course. The SSI waiver involved a
decreased loss in the SSI benefit for each extra dollar of earnings compared to the existing 1619
provision. As application thresholds were identical, it is uncertain how much of the SSI waiver’s
effect on earnings can be directly attributed to the waiver.
45
Though TWP completion information was collected during WI SPI, it is not included in the
deidentified data set for that project.
39
Adding TWP completion to the SSDI-EP sample model also increases the
amount of variance in Q8 that is accounted for. The r 2 value for the model without the
TWP completion variable is 0.089. Thus 9% of the earnings variance is accounted for.
By contrast, the r 2 value for the model including TWP completion is 0.198 (accounting
for 20% of variance in earnings). This means that, for SSDI-EP participants, the amount
of benefits counseling an individual received during the study predicted TWP completion
during the Q0-Q8 post-enrollment period. In turn, TWP completion during this period
predicted Q8 earnings. In other words, TWP completion during the post-enrollment
period mediates the relationship between the categorical amount of benefits counseling
a participant received and Q8 earnings.
We also examined the mediating effect of benefit offset usage by SSDI-EP
participants (see table 9). While offset use required TWP completion and earnings above
SGA for the months it was applied, it is also true that the date of TWP completion could
have occurred after Q8 and the offset might not have been applied to the SSDI cash
benefit in any of the months contained in the Q0-Q8 period. Still, we observe a
significant relationship between the amount of benefits counseling received and offset
use (r = 0.179; p < 0.001). There is a significant relationship between offset use and
earnings (r = 0.370; p < 0.001). There is a significant relationship between the amount
of benefits counseling received and earnings (r = 0.167; p < 0.001). 46 Finally, including
offset use along with the amount of benefits counseling received into the linear
regression earnings model reduced the relationship between benefits counseling and
earnings, with the p-value increasing from 0.002 to 0.042. 47
Table 9: Linear Model Comparison to Check if Benefit Offset Use Mediates the
Relationship between Benefits Counseling and Earnings
Model
Beta
Std. Error P-Value
1
(Constant)
230.8
245.61
0.3479
Half SD BC
388.2
125.98
0.0022
Pre-Enrollment Earnings
1197.0
213.71 < 0.0001
2
(Constant)
309.3
232.99
0.1850
Half SD BC
246.2
120.94
0.0424
Pre-Enrollment Earnings
935.2
205.64 < 0.0001
SSDI Benefit Offset Use
2112.1
287.95
< 0.001
Study WI SSDI EP Participant; n = 468; Dependent Variable = Quarter 8 Earnings
Just as including TWP use increases the amount of variance in earnings that a
model accounts for, so too does inclusion of offset use. When offset use is not included
in the model, the r 2 value was 0.089, accounting for 9% of the variance in earnings.
When offset use was included in the model, the r 2 value increases to 0.184, now
accounting for 18% of the variance. Like TWP completion, benefit offset use mediates
the relationship between the amount of benefits counseling a participant received and
Q8 earnings.
46
47
As with TWP completion, we start the analysis of mediating effects with bivariate correlations.
Also note the reduction in the size of the beta coefficient for the dosage category from 388 to
246.
40
Findings: Determinants of Benefits Counseling Dosage
In our brief review of studies about the impact of benefits counseling we noted
that the growth in employment outcomes tends to be concentrated in the period shortly
after the service was initiated. While it is possible this reflects how the intervention
works, it is also possible that such rapid gains reflect conditions already in place when
an individual first receives the service. We use data from our samples to explore this
question, though we stress that caution should be exercised in applying our findings
beyond the immediate context of these demonstration projects.
In these analyses, the total hours of benefits counseling provided over the Q0-Q8
period (SumBCQ0Q8) is the dependent variable. We use OLS linear regression to
estimate total dosage as we are not interested in identifying dosage trends within the
Q0-Q8 period. 48 Our analysis concentrates on employment outcomes prior to entry to
either the WI SPI or the SSDI-EP as we hypothesized that persons already having
relatively good employment outcomes would be well positioned to use benefits
counseling to further improve their outcomes. To do this, we use the same variables we
use to identify either participants with unusually strong earnings prior to study entry or
those who reported having no employment after going on Social Security benefits.
These variables are labeled, respectively, as PreEarn1200Cat2 and EmpSinceDis.
Additionally, we ran models using our income proxy for the calendar quarter immediately
prior to study entry (Incqn1), as we thought it possible that participants with higher
incomes might feel a greater need for services such as benefits counseling as they
approached earnings or income thresholds that might threaten continued eligibility for
disability programs. 49 It is important to remember that relatively high income does not
necessarily indicate relatively high earnings. For SSDI beneficiaries, it often reflects a
higher than average benefit level.
All of the models estimating total hours of benefits counseling include the
following control variables: age, gender, disability type and education. Models using the
same set of control variables irrespective of project are referred to as “common.”
Additionally, we ran models with variables that were specific to one of the
studies. For the SSDI-EP sample we added variables that indicated whether a
participant had received benefits counseling before entering the project (PreStudyBC)
and which study group the participant had been assigned to (ssdieptx). For WI SPI, we
added the variable (SSDI) that distinguished participants who had some form of SSDI
eligibility from those who were SSI only. Table 10 summarizes the results for both the
common and study specific models after non-significant variables were removed.
48
Benefits counseling tended to be front loaded in both projects (see table 1). For the SSDI-EP,
quarterly means for the period after Q1 are too small to support analysis. Additionally, in the case
of the SSDI-EP, benefits counselors reported that a substantial proportion of later benefits
counseling was associated with problems in processing post TWP work reviews and offset
adjustments for those in the treatment group. Delin et al. (2010) pp. 116-18 and 120-24.
49
We examined the impacts of several other indicators of employment related outcomes prior to
project enrollment. Results proved highly consistent with those presented here.
41
Table10: Selected Regression Results for Predictors of Total Benefits Counseling Hours
Q0-Q8
WI SPI
SSDI-EP
Variables
Beta
Variables
Beta
Significant at .05
Significant at .05
Level
Level
18.242
“PreEarn1200Cat2”
3.246
Common Model “PreEarn1200Cat2”
for
“CognitiveDD”
14.503
“CognitiveDD”
-6.538
“PreEarn1200Cat2
“AffectiveMH”
-14.131
Common Model
for
“EmpSinceDis”
“CognitiveDD”
“AffectiveMH”
16.340
-10.344
“EmpSinceDis”
“CognitiveDD”
3.024
-6.225
Common Model
for “Incqn1”
“Incqn1”
0.004
(per $1
increase)
17.367
-11.110
“Incqn1”
“CognitiveDD”
“Age”
0.001
(per $1
increase)
-5.997
-0.139
18.242
14.503
-14.131
“PreEarn1200Cat2”
“CognitiveDD”
“PreStudyBC”
3.073
-6.602
3.982
18.772
-10.363
-9.730
“EmpSinceDis”
“CognitiveDD”
“PreStudyBC”
3.219
-5.787
4.066
“CognitiveDD”
“AffectiveMH”
Study Specific “PreEarn1200Cat2”
Model for
“CognitiveDD”
“PreEarn1200Cat2
“AffectiveMH”
Study Specific
Model for
“EmpSinceDis”
“CognitiveDD”
“AffectiveMH”
“SSDI”
0.004
“CognitiveDD”
-6.093
(per $1
increase)
“CognitiveDD”
17.367
“PreStudyBC”
3.970
“AffectiveMH”
-11.110
“Age”
-0.127
Source: WI SPI and SSDI-EP Encounter Data, Administrative Data from WI Department
of Workforce Development and the Social Security Administration
Study Specific
Model for
“Incqn1”
“Incqn1”
The common models suggest that having relatively high earnings for at least half
the quarters in the year prior to study entry substantially increases the amount of
benefits counseling one receives. The beta coefficient for WI SPI indicates that on
average these higher earning participants would receive about eighteen more hours of
benefits counseling than other participants. This increment of service is about 47% of the
mean service hours for the full WI SPI sample. Though the beta coefficient for SSDI-EP
“common” model is much smaller in absolute size, it implies a considerable proportion of
additional benefits counseling (39%) relative to the average dosage. The comparable
betas for the study specific models are either identical or marginally different.
By contrast, the EmpSinceDis variable is statistically significant for the SSDI-EP
sample, but not for the WI SPI sample. For SSDI-EP participants the results are very
similar to those for the high earners group. By contrast, the regression coefficients for
the WI SPI models never reach a p-value of 0.1 and are excluded from the model. We
42
do not understand why a variable that indicates whether there had been any
employment history is non-significant for the WI SPI sample, though the control variables
we discuss below may provide some insight into the matter.
The results for the income variable are mixed, but generally indicate that those
with higher income before study entry received more benefits counseling over the Q0-Q8
period. Results for both of the WI SPI model and for the common model for the SSDI-EP
are statistically significant with betas of .004 for the WI SPI models and .001 for the
SSDI-EP model. 50 Though these coefficients appear to be very small, it is because they
measure the estimated increase of benefits counseling dosage associated with a one
dollar increase in the income variable. For example, a beta of .001 implies that if a SSDIEP entrant had a Q-1 income $1,000 greater than another study entrant, the former
would be expected to be provided with one more hour of benefits counseling over the
Q0-Q8 period; i.e., about 12% of the sample average.
Among our control variables, one or more of the disability type variables prove
statistically significant in every model. Results for these variables are relative to those of
the reference category of physical disability. While in all models being categorized as
having a cognitive/developmental disability has a statistically significant effect on
benefits counseling dosage, the effects do not run in a consistent direction. In WI SPI,
this disability category is associated with receiving substantially more benefits
counseling than those in the reference category. By contrast, in the SSDI-EP, those in
the cognitive/developmental disability category typically received substantially less
benefits counseling than those in other disability groups. We have not followed up on
this finding, though we believe it may be related to differences in the eligibility rules and
the service populations associated with the specific groups of community agencies that
enrolled participants in each of the projects. 51 Such differences may also help explain
why those in the affective/mental health category received far less service in the WI SPI
project, but comparable levels to the reference group in the SSDI-EP.
With one exception, we do not discuss other control variables. That exception is
benefits counseling prior to study entry (PreStudyBC). That variable proves to have a
strong effect on dosage over the Q0-Q8 period in the SSDI-EP study specific models.
The coefficients are consistent across the three models and suggest that having prior
benefits counseling increases dosage approximately four hours (that is, nearly half of the
mean dosage). It is possible that consumers receiving benefits counseling services
develop what economists call a taste for the service. It is also possible that prior receipt
helped to motivate the better pre-enrollment employment outcomes we find associated
with dosage during the Q0-Q8 period. The data available to us does not support further
examination of this issue.
Nonetheless, though these models are always significant at a 0.00 p-value, none
of the models explain very much of the variance. In no case is the r 2 much greater than
0.1, suggesting that the models are either incompletely specified or that there is a great
50
For the SSDI-EP study specific model the Incqn1 variable was nearly significant (p-value =
0.069). The beta coefficient was again 0.001.
51
In WI SPI, each provider agency was restricted to enrolling consumers from a single disability
group. A substantial proportion of participants were enrolled at agencies restricted to enrolling
those defined as having a developmental disability.
43
deal of individual level variation. One important factor might be work motivation.
Unfortunately, the survey data still available for both studies provides only modest
resources to explore this possibility. We do not give much credence to responses to
questions about whether a severely disabled person would prefer to work if possible as
there is little correlation with actual labor force participation. However, there was one
survey item available for both studies that might be a reasonably credible indicator of
motivation, the extent that, on a five point scale, a respondent agreed that she had a
career plan.
We added the career plan variable to the three “common” models run for both
studies. The results are mixed. For the SPI sample, the variable is always statistically
significant with a positive but relatively small coefficient. Including the variable makes a
slight improvement in the model’s r 2 value. For the SSDI-EP, having a career plan was
never statistically significant and the variable’s inclusion resulted in a slightly smaller r 2 .
Conclusion
Researchers have established a reasonably strong case that better employment
outcomes for persons with severe disabilities, especially earnings, are positively
associated with the delivery of work incentive benefits counseling. The relevant service
population is defined not only by disability in a medical or functional sense, but also by
attachment to public programs that provide income support, health care, or other
services as a consequence of meeting disability related program eligibility requirements.
Our aim has been to better understand this association, especially whether the amount
of benefits counseling services provided, that is dosage, is positively related to better
employment outcomes.
The descriptive data from the SSDI-EP project (where about 21% of participants
did not get any benefits counseling in the Q0-Q8 period) are basically consistent with
those from other studies. Consumers who receive benefits counseling have better
employment outcomes. More to the point, when we look at the association between
dosage and employment outcomes we see clearly positive relationships. Figure 1 in this
report provides an example (from the SSDI-EP) for the positive association of benefits
counseling dosage with subsequent earnings levels and growth. Table 11 displays
information suggesting a strong association between dosage levels and income growth
over the Q0-Q8 period in both the WI SPI and SSDI-EP projects. Indeed, we would
argue that from an individual’s perspective, income growth should be expected to be
more important than earnings growth. 52
52
In saying this, we are not suggesting that many individuals might not place more value on a
dollar of income achieved through employment than one obtained from some other source.
44
Table 11: Repeated Measures MANOVA Estimated Mean Income by Received Amounts
of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 –
%
Counseling
Q0
Change
< ½ SD BC
$2672 $2866
$194
7.3%
WI SPI (n =437)
Within ½ SD BC
$2850 $3284
$434
15.2%
> ½ SD BC
$3008 $3698
$690
22.9%
< ½ SD BC
$3356 $3378
$22
0.6%
SSDI-EP (n =443)
Within ½ SD BC
$3512 $3879
$367
10.5%
> ½ SD BC
$3550 $4192
$642
18.0%
However, our purpose is also to explore whether the positive relationship
between the amount of benefits counseling and employment outcomes would remain
after controlling for other factors, such as indicators of previous employment outcomes
and personal characteristics such as age, gender, educational attainment, and broadly
defined disability categories. Ideally, we would have liked to have reported dosage
impacts in a standard form such as “one additional hour of benefits counseling leads, on
average, to x dollars difference in earnings.” As WI SPI participants typically received
many more service hours than those in the SSDI-EP, we decided to utilize dosage
categories relative to the mean dosage for each of the projects. Thus, results must
always be interpreted as dosage relative to that which was typical for project
participants. While we hope that other research efforts will be implemented that will more
directly measure the impact of a unit of benefits counseling services, as we explain
below, we are not sure that this is the best way to conceptualize how the service impacts
employment outcomes.
Though conclusions based on data from two relatively small projects located in a
single state cannot fully settle the issue, our results suggest that receiving higher
dosages of benefits counseling services leads to better employment outcomes. This is
unequivocally true for UI earnings. The results for the combined samples model suggest
that, after controlling for other factors, quarterly earnings at the end the study period for
those in the medium dosage category increased about $275 relative to those for
participants in the low dosage category. On average, Q8 earnings for those in the high
dosage category grew about $550 relative to those for participants in the low dosage
category.
Our results are less consistent for other employment outcomes, though higher
dosages are usually associated with higher probabilities of employment and higher
mean income. Though higher dosage levels are generally associated with better
outcomes, they are not as consistently associated with outcome change rates over the
study period. Much the same could be said for our subgroup analyses of the earnings
trends for relatively high earners in the year prior to study enrollment and those who had
reported no workforce participation in the period between going on Social Security
benefits and enrolling in one of the projects. The basic patterns are discernable, but
without clear differentiation of earnings trends across the dosage categories.
One of the claims made for the efficacy of benefits counseling is that it
encourages consumers to use available work incentives. The results from our analysis
are mixed, with the strongest evidence coming in the positive relationship of benefits
counseling dosage to TWP completion and benefit offset use in the SSDI-EP. As noted
45
earlier, our results for WI-SPI may have as much to do with the lack of good data about
work incentive use over the full Q0-Q8 period as with the impact of benefits counseling.
It is crucial to acknowledge that the explanatory power of our regression and
MANOVA models is very modest and the benefits counseling dosage categories
(including their interaction with time) never explains over a few percentage points of
observed variance. Generally, the pre-enrollment earnings variable is, by far, the
strongest predictor variable. Based on prior work, this possibility had been anticipated.
Thus, we performed an analysis of what factors best explain how many hours of benefits
counseling a participant received. Participants with stronger pre-enrollment employment
outcomes received more service than those who had weaker ones. Additionally (for the
SSDI-EP only) those who reported receiving benefits counseling services before
enrollment also received more service subsequent to project entry.
Nonetheless, the explanatory power of the models estimating how many hours of
benefits counseling participants received is low, with r 2 values never rising appreciably
above 0.1. Part of this may be the lack of potentially important attitudinal variables in our
models. However, we suspect that how one thinks about causality may also be an
important issue. We doubt that one should think about the impacts of benefits counseling
as one might think about the impacts of a drug. It might be better to view the service as a
tool that those already committed to achieving better employment outcomes take up to
pursue their goals. It might also be important to think about the context in which the
service is used. The results of a standardized unit of service might prove to be more
strongly affected by labor market conditions or program rules (such as the SSDI cash
cliff) than by many individual level characteristics.
Finally, our samples are limited to volunteers who appear to contain a far greater
proportion of “work oriented” individuals than the overall working age population of those
getting Social Security disability benefits. We cannot say with any certainty whether we
would have observed similar patterns between benefits counseling dosage and
employment outcomes if the typical WI SPI or SSDI-EP participant had entered the
projects with the same likelihood of employment or mean earnings as those of the
overall SSDI and SSI populations in Wisconsin or nationally. Much needs to be done to
determine the efficacy of benefits counseling services for the “non-work oriented,”
though there may be other good reasons to provide the service. In particular, we find
purchase in the argument that no person in clear need of retaining cash benefits, access
to health care, and other support services should be asked to risk their eligibility unless
they are fully aware of the risks they face and what might be done to mitigate them.
In any case, our paper adds to the body of evidence that supports the value of
benefits counseling services for facilitating “return to work.” To the extent that the public
and policymakers are interested in encouraging fuller labor market inclusion for those
with severe disabilities, whether the motivation be general economic development,
lowering outlays for government programs, or a concern for the economic welfare and
dignity of persons with disabilities, the continuation, indeed expansion, of work incentive
benefits counseling services appears to us to provide an effective policy tool for helping
to achieve that goal.
46
Appendix A: Graphs of Mixed Linear Model Predicted Outcomes
by Amount of Benefits Counseling Received for Combined Sample
Figure 16: Predicted Mean UI Earnings by Amount of Benefits Counseling Received,
Q0-Q8
Predicted Mean Quarterly Earnings by Amount of Benefits Counseling
Mean Predicted Earnings
2000
HalfSDBCSplit
1500
Greater than 1/2
SD of BC Mean
Within 1/2 SD of
BC Mean
Less than 1/2 SD
of BC Mean
1000
500
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
47
Figure 17: Predicted Mean UI Employment Rates by Amount of Benefits Counseling
Received Q0-Q8
Mean Predicted Employment Rates
study2: Study Participant
1.00
0.80
HalfSDBCSplit
0.60
Greater than 1/2
SD of BC Mean
Within 1/2 SD of
BC Mean
Less than 1/2 SD
of BC Mean
0.40
0.20
0.00
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
48
Figure 18: Predicted Mean Income by Amount of Benefits Counseling
Received Q0-Q8
study2: Study Participant
4500
Mean Predicted Income
4000
3500
3000
HalfSDBCSplit
2500
Greater than 1/2
SD of BC Mean
Within 1/2 SD of
BC Mean
2000
Less than 1/2 SD
of BC Mean
1500
1000
500
0
0
1
2
3
4
5
6
7
8
Quarter Compared to Enrollment
49
Appendix B: MANOVA Model Summary Statistics
Table 12: Repeated Measures Earnings MANOVA – WI SPI Participants (n = 437)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.776
0.011
< 0.001
0.032
Benefits Counseling
*Quarter
0.125
0.026
0.013
0.020
PreEarnings
*Quarter
0.045
0.037
< 0.001
0.118
Benefits Counseling *
*Quarter
0.042
0.031
0.281
0.006
PreEarnings
Gender
*Quarter
0.837
0.010
0.351
0.002
Age
*Quarter
0.916
0.008
0.202
0.004
Education
*Quarter
0.212
0.025
0.420
0.002
Disability
*Quarter
0.831
0.010
0.824
< 0.001
Table 13: Repeated Measures Earnings MANOVA – SSDI-EP Participants (n = 443)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.872
0.009
0.012
0.014
Benefits Counseling
*Quarter
0.059
0.029
0.021
0.018
PreEarnings
*Quarter
0.043
0.037
< 0.001
0.122
Benefits Counseling *
*Quarter
0.110
0.027
0.924
< 0.001
PreEarnings
Gender
*Quarter
0.016
0.043
0.721
< 0.001
Age
*Quarter
0.111
0.030
0.025
0.012
Education
*Quarter
0.822
0.010
0.107
0.006
Disability
*Quarter
0.296
0.022
0.200
0.004
Table 14: Repeated Measures Employment MANOVA – WI SPI Participants (n =
437)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.321
0.022
< 0.001
0.190
Benefits Counseling
*Quarter
0.839
0.012
0.376
0.005
PreEarnings
*Quarter < 0.001
0.068
< 0.001
0.198
Benefits Counseling *
*Quarter
0.423
0.019
0.392
0.004
PreEarnings
Gender
*Quarter
0.757
0.012
0.120
0.006
Age
*Quarter
0.807
0.011
0.092
0.007
Education
*Quarter
0.047
0.036
0.350
0.002
Disability
*Quarter
0.396
0.020
0.001
0.025
50
Table 15: Repeated Measures Employment MANOVA – SSDI-EP Participants (n =
443)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.755
0.012
< 0.001
0.102
Benefits Counseling
*Quarter
0.693
0.015
0.004
0.025
PreEarnings
*Quarter
0.102
0.031
< 0.001
0.236
Benefits Counseling *
*Quarter
0.555
0.017
0.240
0.007
PreEarnings
Gender
*Quarter
0.050
0.035
0.965
< 0.001
Age
*Quarter
0.436
0.018
0.007
0.017
Education
*Quarter
0.313
0.022
0.613
0.001
Disability
*Quarter
0.143
0.028
0.205
0.004
Table 16: Repeated Measures Income MANOVA – WI SPI Participants (n = 437)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.284
0.023
< 0.001
0.061
Benefits Counseling
*Quarter
0.216
0.023
0.020
0.018
PreEarnings
*Quarter
0.125
0.029
< 0.001
0.082
Benefits Counseling *
*Quarter
0.100
0.027
0.053
0.014
PreEarnings
Gender
*Quarter
0.899
0.008
0.119
0.006
Age
*Quarter
0656
0.014
0.044
0.009
Education
*Quarter
0.238
0.024
0.010
0.015
Disability
*Quarter
0.773
0.011
0.044
0.009
Table 17: Repeated Measures Income MANOVA – SSDI-EP Participants (n = 443)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.825
0.010
0.001
0.024
Benefits Counseling
*Quarter
0.110
0.027
0.118
0.010
PreEarnings
*Quarter
0.014
0.044
< 0.001
0.072
Benefits Counseling *
*Quarter
0.071
0.029
0.561
0.003
PreEarnings
Gender
*Quarter
0.019
0.042
0.028
0.011
Age
*Quarter
0.127
0.029
0.919
< 0.001
Education
*Quarter
0.838
0.010
0.004
0.019
Disability
*Quarter
0.327
0.021
0.169
0.004
51
Table 18: Employment Since SSA Benefits Repeated Measures Earnings MANOVA
– WI SPI Participants (n = 437)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.594
0.017
0.093
0.007
Benefits Counseling
*Quarter
0.802
0.014
0.048
0.001
Employment Since SSA
*Quarter
0.986
0.005
< 0.001
0.077
Benefits
Benefits Counseling *
*Quarter
0.944
0.007
0.380
0.005
Employment Since SSA
Benefits
Gender
*Quarter
0.747
0.013
0.570
0.001
Age
*Quarter
0.735
0.013
0.211
0.004
Education
*Quarter
0.250
0.026
0.220
0.004
Disability
*Quarter
0.784
0.012
0.458
0.001
Table 19: Employment Since SSA Benefits Repeated Measures Earnings MANOVA
– SSDI-EP Participants (n = 443)
With-In Subject (Wilks’ Lambda)
Between Subject
Sig
Partial Eta
Sig
Partial Eta
Squared
Squared
Intercept
Quarter
0.790
0.011
0.297
0.003
Benefits Counseling
*Quarter
0.606
0.016
0.200
0.007
Employment Since SSA
*Quarter
0.465
0.018
< 0.001
0.041
Benefits
Benefits Counseling *
*Quarter
0.905
0.011
0.428
0.004
Employment Since SSA
Benefits
Gender
*Quarter
0.022
0.041
0.815
< 0.001
Age
*Quarter
0.120
0.029
0.048
0.009
Education
*Quarter
0.797
0.011
0.054
0.009
Disability
*Quarter
0.211
0.025
0.245
0.003
52
Appendix C: Q8-Q0 Change in Employment Outcomes Predicted by MANOVA models
Table 20: Repeated Measures MANOVA Estimated Mean Earnings by Received
Amounts of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
$900 $1270
$370
WI SPI (n =437)
Within ½ SD BC
$1200 $1324
$124
> ½ SD BC
$1346 $1809
$463
< ½ SD BC
$1224 $1060
-$164
SSDI-EP (n =443)
Within ½ SD BC
$1490 $1751
$261
> ½ SD BC
$1683 $1887
$204
Table 21: Repeated Measures MANOVA Estimated Mean Employment by Received
Amounts of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
0.28
0.31
0.03
WI SPI (n =437)
Within ½ SD BC
0.28
0.39
0.11
> ½ SD BC
0.29
0.48
0.19
< ½ SD BC
0.32
0.28
-0.04
SSDI-EP (n =443)
Within ½ SD BC
0.36
0.47
0.11
> ½ SD BC
0.37
0.53
0.16
Table 22: Repeated Measures MANOVA Estimated Mean Income by Received
Amounts of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
$2672 $2866
$194
WI SPI (n =437)
Within ½ SD BC
$2850 $3284
$434
> ½ SD BC
$3008 $3698
$690
< ½ SD BC
$3356 $3378
$22
SSDI-EP (n =443)
Within ½ SD BC
$3512 $3879
$367
> ½ SD BC
$3550 $4192
$642
Table 23: Repeated Measures MANOVA Estimated Mean Earnings for PreEnrollment High Earners by Received Amounts of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
$1539 $1996
$457
WI SPI (n =437)
Within ½ SD BC
$2166 $1805
-$361
> ½ SD BC
$2178 $2516
$338
< ½ SD BC
$1937 $1571
-$366
SSDI-EP (n =443)
Within ½ SD BC
$2412 $2422
$10
> ½ SD BC
$2754 $2549
-$205
53
Table 24: Repeated Measures MANOVA Estimated Mean Earnings for PreEnrollment Low Earners by Received Amounts of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
$261
$544
$283
WI SPI (n =437)
Within ½ SD BC
$235
$843
$608
> ½ SD BC
$513 $1101
$588
< ½ SD BC
$511
$547
$36
SSDI-EP (n =443)
Within ½ SD BC
$567 $1079
$512
> ½ SD BC
$613 $1224
$611
Table 25: Repeated Measures MANOVA Estimated Mean Earnings for Participants
who at Enrollment were not Employed after Receiving Benefits by Received
Amounts of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
$0
$246
$246
WI SPI (n =437)
Within ½ SD BC
$0
$589
$589
> ½ SD BC
$22
$891
$869
< ½ SD BC
$83
$306
$223
SSDI-EP (n =443)
Within ½ SD BC
$67
$562
$495
> ½ SD BC
$38 $1160
$1122
Table 26: Repeated Measures MANOVA Estimated Mean Earnings for Participants
who at Enrollment were Employed after Receiving Benefits by Received Amounts
of Benefits Counseling
Study
Amount of Benefits
Q0
Q8
Q8 – Q0
Counseling
< ½ SD BC
$824 $1144
$320
WI SPI (n =437)
Within ½ SD BC
$968 $1263
$295
> ½ SD BC
$1352 $1938
$586
< ½ SD BC
$1085
$938
-$147
SSDI-EP (n =443)
Within ½ SD BC
$1348 $1676
$328
> ½ SD BC
$1559 $1768
$209