The Innovative
Strategies for
Increasing SelfSufficiency (ISIS)
Evaluation
APHSA Members Webinar
September 26, 2013
Introduction
 Purpose of webinar
 Introduction of speakers
 Review of agenda
– ACF Perspective
– Evaluation Summary
– Evaluation Partner: Workforce Development Council
of Seattle-King County
– Q&A
2
What Is ISIS?

ISIS – the Innovative Strategies for Increasing Self-Sufficiency
evaluation – is an evaluation of promising strategies for increasing
education, training, employment and self-sufficiency among lowincome families.
– Impact Study using Random Assignment
– Implementation Study
– Cost-Benefit Analysis




Evaluation by the Office of Planning, Research and Evaluation
(OPRE) in the Administration for Children and Families (ACF)
Led by Abt Associates, in partnership with MEF Associates,
APHSA, NCSL, NGA, consultants, Public Strategies, Inc. and the
University of Michigan.
Support for programs provided by the Open Society Foundations,
The Joyce Foundation, The Kresge Foundation, The Meadows
Foundation and ACF/OFA Health Professions Opportunity Grants.
10-year project, currently in the random assignment phase (year 6)
3
Why is ACF and OPRE
evaluating career pathways?
 OPRE is the principal research and evaluation office of ACF.
– OPRE is to serve as the “principal advisor” to the Asst Secretary on
research and evaluation matters.
– Our research covers the entirety of ACF’s program areas (though not
evenly).
 Our research and evaluation activities are guided by the principles of
rigor, relevance, transparency, independence and ethics.
 We have a long history of conducting experimental research,
particularly in welfare to work and employment and training
programs (see new book by Gueron and Rolston).
 Our approach to research and evaluation has also taken on greater
prominence and importance in the current Obama Administration
and OMB’s emphasis on “evidence-based policy-making.”
4
Poverty and Pathways:
The Motivation for ISIS
3 Key “Facts”
– Previous research has demonstrated effectiveness in moving families
off cash assistance and into employment. (NEWWS, GAIN, New Hope,
MFIP, SSP, state waiver experiments, etc.)
– Economic mobility is increasingly “skill-biased.”
– Individuals with lower skills/lower income have fewer financial and
human capital resources from which to draw on and make productivity
investments.
• Money is a major constraint, but time is also an important constraint.
 2 Important Limitations of Existing Research:
– Education and training has not been as promising or financially
beneficial for poor families as it has for the population as a whole.
– Families were not substantially better off even in the successful
interventions (large numbers remain poor.)
 1 Major Challenge
– How can we integrate education, workforce development and human
services to create pathways out of poverty and increase self-sufficiency
5
for a large and diverse set of American families?

Emerging Evidence: Career
Pathways and Sector-Based
Strategies
Year Up (Economic
Mobility Corp, 2010)
•
National org provides
6 months customized
training + 6 month
paid internship in IT
and finance sectors
w/local colleges,
completers earn 14+
credits
•
Results from small
initial experiment
(n=164)
Two key questions:
1) Do future earnings recoup those foregone while in training?
2) Do the benefits exceed the costs?
6
Emerging Evidence: Career
Pathways and Sector-Based
Strategies
Sectoral Training Strategies
(Maguire et al. 2010)
 3 experienced CBOs
provide customized shortterm training in varied
high-demand fields
 Disadvantaged adults with
HS+ education
 Careful screening
 Total sample n=1,014
7
The ISIS Evaluation
 ISIS aims to learn about promising strategies for increasing
employment and self-sufficiency among low-income individuals and
families
 Core principles
– Collaborate with stakeholders and partners to Identify most relevant
interventions and strategies
• Semi-structured discussions with over 250 individuals (key APHSA role)
 Test a few things well
– Experimental design with random assignment
– Test one type of intervention in nine sites, rather than nine disparate
interventions
 Aim for positive influences on social policy and practice
8
Key Career Pathways Ideas
 A series of connected education and training programs and
support services
– Addresses the wide range of skill and other needs of low-skilled
individuals
– Provide credentials valued in high demand occupations/sectors
 Enables individuals to secure employment within a specific
industry or occupational sector, and to advance to
successively higher levels of education and employment
within that sector
 Each step is designed to prepare the participant for the next
level of employment and education
 Build effective partnerships
– Education and training providers, social service providers,
workforce
9
Prospects for good-paying, stable employment
The Basic Career Pathways Model
V. BA+ Programs
Upper-Skilled Jobs
IV. 1-2-Year Certificate to AA Programs
Mid-Level Skilled Jobs
III. Short-Term Certificate Programs
Entry-Level Skilled Jobs
II. Sectoral Bridge Programs
Skilled Jobs
Semi-
I. Basic Bridge Programs
Occupational, academic, and life skills
Key Career Pathways Services
 Comprehensive assessment
– Academic and non-academic skills
 Basic and technical skills instruction
– Modularization, contextualization, acceleration, flexible delivery,
active learning
 Supports
– Proactive advising and guidance, supplemental instruction,
social supports, supportive services, financial assistance
 Employment connections
– During and after training
11
Take First/ Next
Step In Career
Pathway/Lattice
Participant Characteristics
• Demographic
• Educational
• Economic
Theory of Change for
Career Pathways
Increase Performance &
Persistence in Training
Foundational Academic Skills
• Certificate/Diploma
• 2- year, 4- year Degree
Occupational Skills
Comprehensive
Assessment
Psycho-social Factors
Improve Performance &
Advancement in Jobs
Core Curriculum
Career Orientation and Knowledge
Supports
•  Earnings
•  Benefits
•  Job security
Resource Constraints
Employment
Connections
Improve Other
Outcomes
Other Personal and Family Challenges
• Income & assets
• Child & adult well being
• Local economic growth
Contextual Factors: Institutional, Economic, Social
Program Inputs
Intermediate Outcomes
Primary Outcomes
TO NEXT STEP
Initial
Targeting &
Placement
Decisions
The ISIS Sites
Bellingham Technical College
I-BEST
Everett Community College
I-BEST
Year Up Puget
Sound
Whatcom Community College
I-BEST
Workforce Development
Council of Seattle-King
County
Health Careers for All
Instituto del
Progreso Latino
Carreras en Salud
Year Up Boston
Year Up Providence
Year Up
San Francisco Bay
Area
San Diego
Workforce Partnership
Bridge to Employment
Program
Madison Area
Technical College
Patient Care Pathways
Des Moines Area
Community College
Prepared Learner
Program
Year Up New York City
Year Up Chicago
Year Up National
Capital Region
Pima Community
College Pathways to
Healthcare Program
Year Up Atlanta
Valley Initiative for
Development and
Advancement (VIDA)
ISIS HPOG site
ISIS site
13
Health Professions
Opportunity Grants (HPOG)
 In 2010, the HPOG program was created as part of
the Affordable Care Act in order to provide TANF
recipients and other low-income individuals with
access to jobs that pay well and address workforce
needs in health professions.
 $67 million a year to 32 grantees across 23 states,
including 5 tribal grantees.
 Program is administered by the Office of Family
Assistance (OFA).
 Research and evaluation overseen by OPRE.
14
Health Professions
Opportunity Grants (HPOG)
 To evaluate HPOG ACF has implemented a multipronged research strategy that includes:
– Implementation, systems and outcomes evaluation for
HPOG TANF/low income grantees
– Evaluation of Tribal HPOG
– Performance tracking: HPOG PRS collects uniform
data across all grantees
– Impact studies in a subset of grantees
• HPOG Impact Studies
• Innovative Strategies for Increasing Self-Sufficiency (ISIS)
– University-based field initiated research
15
The Sample of ISIS Sites
 Career pathways is both a framework and a model
for particular programs – many variations
incorporating basic ideas possible
 9 ISIS programs exemplify central career pathways
ideas, but vary greatly in:
– Basic elements such as target populations (e.g.,
TANF), occupations and training steps, provider
partnerships
– Lead organization—CBOs (3), Community/Technical
Colleges (4) and WIBs (2)
– Key strategies included; their design, intensity, and
duration
16
Who is participating in ISIS?
Sex
100%
90%
80%
47.8%
51.7%
70%
72.1%
73.2%
60%
93.1%
50%
81.6% 85.2%
87.0%
78.1%
83.3% 85.7%
40%
30%
52.2%
48.3%
20%
28.0%
26.8%
10%
6.9%
0%
BTC
DMACC
Instituto
18.4% 14.8%
13.0%
Madison
Pima
Male
SDWP
Female
22.0%
16.7% 14.3%
VIDA
WDC
Whatcom
Year Up
NCR
Total
Sample
17
Who is participating in ISIS?
Age at Baseline
1.4%1.4%
100%
90%
13.8%
15.9%
10.0%
14.4%
80%
70%
36.4%
42.0%
31.0%
53.3%
45.8%
60%
85.7%
55.4%
42.5%
50%
40%
30%
38.4%
44.9%
98.6%
58.3%
44.6%
55.2%
20%
39.8%
29.5%
28.7%
10%
22.3%
27.1%
28.0%
19.6%
36.7%
19.0%
32.8%
14.6%
14.3%
0%
Under 25
25 to 39
40 and older
18
Who is participating in ISIS?
Total Family Income in Prior Year
80%
75.4%
75.0%
70%
68.1%
66.5%
60.4%
60%
50%
40%
54.5%
53.5%
53.2%
50.7%
44.8%
37.9%
28.6% 28.6%
30%
31.9%
27.4%
19.8%
20%
10%
9.4%
13.5%
15.0%
12.7%
7.1%
6.3%
0%
Below $15,000
$30,000 or Above
19
ISIS Program Diversity
Health
Office/
Business
DMACC
X
X
X
I-BEST
X
X
X
Instituto
X
MATC
X
PCC
X
SDWP
X
VIDA
X
WDC
X
Year Up
X
IT
X
X
Welding/
Electrical
Customer
Service
Building
Maint..
X
X
X
20
ISIS Research Questions
 What is the impact of the program on persistence in
education and achievement of credentials and
degrees?
 What is the impact of the program on career-track
employment and earnings?
 What is the impact on well-being?
21
Study Components
ISIS will include site-specific:
 Impact study
– Using baseline data, administrative data (UI, college records),
surveys
 Implementation study
– Interviews, staff surveys and site-level administrative data
 Cost-benefit study
Study parameters:
 Randomly assign a minimum of 1,000 eligible
individuals to treatment and control groups
– Control group can access other service in the community
22
Study Deliverables




Evaluation design
Program profiles (2013)
Cross-program implementation report
Program-specific 15 month reports
– Implementation
– Impact
– Cost
23
ISIS Program
Health Careers for All
Workforce Development Council of Seattle-King
County
24
What is HCA?
 Health Careers for All is funded by a 5-year HHS
HPOG grant
 Healthcare training via cohorts or individual training
support
 Career and education “navigation” support
 Efforts to better align systems that serve the target
population (workforce, education, social service)
Outcome Targets for HPOG
Over the 5 years of the grant period:
 Enroll 920 TANF recipients and other low-income
adults and youth
 Achieve a training completion rate of 70%
(benchmark = 55%)
 Achieve a training-related placement rate of 60%
(benchmark = 25%)
 Achieve a return-to-training rate of 25% (benchmark
= 11%)
Outcomes to Date
Over life of the HPOG grant
 Nearly 600 customers enrolled
 Approximately 300 customers completed healthcare
training
 More than 150 customers employed
 Approximately 60 customers continued/returned for
next level training
TANF Participation
 Project Goal to have TANF recipients comprise
approximately 1/3 of enrollments over the HPOG
project period
 TANF participation has averaged 40-45% over the
first 3 years of the project
 Project designed in close partnership with TANF
agency
TANF Partnership
 Partners in developing project model and proposal
 Members of project steering committee overseeing
implementation of project model
 Clearly articulated outreach and referral protocol
 Regular check-in meetings in addition to steering
committee
HCA in ISIS
 Implemented ISIS in summer 2012 (end of Year 2 of
HPOG grant)
 ISIS Goal: Randomly assign 700 individuals
 Outcomes to Date (end of Aug 2013):
–
–
–
–
267 individuals RA’d (51% TANF)
134 assigned to treatment group (51% TANF)
110 enrolled in HCA
94 entered training
30
TANF Partnership Post ISIS
 Re-doubled coordination efforts during
implementation of ISIS study
 Refined referral protocols
 Generated materials to support TANF staff
 Efforts to safeguard against over-enrollment in
control group during random assignment (ensure
50-50 treatment/control split among TANF referrals
in addition to overall RA pool)
Customer Story
Kim is a single mom of three boys who came to HCA in 2011 after nearly two years of
training and job hunting in medical administration produced no employment. She had
been trying to enter the healthcare field since 2008, was nearing the end of her lifetime
TANF benefits, and had nearly exhausted her allotted training time through DSHS. Kim
had 9 years of steady employment as a journeyman Industrial Maintenance Mechanic but
in 2008 was terminated from employment because of excessive time off related to
domestic violence issues. Due to the downturn in the economy, she was unable to find
alternate employment in her profession. After a period of homelessness, she found
transitional housing and subsequently entered subsidized housing. In addition to
employment and housing challenges, Kim was busy managing 3 young boys all with
behavioral issues and special needs.
After some counseling to figure out the best path for her to enter a health career as
quickly as possible, and to best utilize skills gained along the way, she enrolled in the
HCA Phlebotomy cohort, which she completed with the highest scores in the
class. Aside from excellent grades, Kim demonstrated patience, bilingual skills, a wry
sense of humor and excellent ability work calmly with diverse patients.
Following the phlebotomy training, Kim received job search assistance through WorkFirst
and the HCA job developer. In September 2012, she was offered a position at LabCorp
at $14.54 per hour, 25 hours per week to start. Three months later, Kim called to say
that she had received praise for the work she was doing and was offered a full-time
position. Kim is very thankful for the ongoing services received from the HCA program
and now wants to start nursing prerequisites so she can eventually become an RN.
Lessons Learned to Date
 Importance of understanding the TANF program and
how it is implemented in your area
 Importance of cultivating relationships at both the
local and regional/state level
 Benefits and challenges of grappling together with
barriers (e.g., participation requirements and 12month limit on training as a “countable” activity)
Questions?
34
Additional Information
www.projectisis.org
http://www.acf.hhs.gov/programs/opre
Karen Gardiner
Abt Associates
Karen_gardiner@abtassoc.com
Brendan Kelly
OPRE/ACF
Brendan.Kelly@acf.hhs.gov
Seanna Ruvkun
Workforce Development Council of
Seattle-King County
SRuvkun@seakingwdc.org
Cynthia Woodside
APHSA
cwoodside@aphsa.org
35