A Standardized
Net Impact
Evaluation Framework
For Minnesota
With preliminary results as of May 2014
Getting Started
Nick Maryns
Senior Policy Analyst
Governor’s Workforce Development Council
Nicholas.maryns@state.mn.us
Raymond Robertson
Professor of Economics
Macalester College
robertson@macalester.edu
Standardized
Net Impact
Evaluation Framework
Evaluation Design
Motivations, History,
Partners
Overview and Basic
Parameters
Pilot Project
Preliminary Results
Motivations, History, Partners
Motivations
History
Advisory Group
“Based on our rough calculations, less
than $1 out of every $100 of
government spending is backed by
even the most basic evidence that the
money is being spent wisely.”
- Peter Orszag and John Bridgeland,
The Atlantic Monthly, July 2013
Motivations, History, Partners
Motivations
History
Advisory Group
The National Conversation
Pew-MacArthur Results First Initiative
Results for America
Social Impact Bonds / Pay for Success
Motivations, History, Partners
Motivations
History
Advisory Group
Apples and Oranges Approaches
across the State
Motivations, History, Partners
Motivations
History
Advisory Group
The UPAM law required the
development of uniform ROI measure.
Motivations, History, Partners
Motivations
History
Advisory Group
The GWDC’s Role
(d) Functions. The State Board shall assist the
Governor in—
(6) development and continuous improvement of
comprehensive State performance measures, including
State adjusted levels of performance, to assess the
effectiveness of the workforce investment activities in
the State as required under section 2871 (b) of this title;
-
Section 111 of the Workforce Investment Act
Subd. 3. Purpose; duties.
(c) “Advise the governor on the development and
implementation of statewide and local performance
standards and measures relating to applicable federal
human resource programs and the coordination of
performance standards and measures among
programs”
-
Minnesota Statute 116L.665 Subd. 3c
Motivations, History, Partners
Motivations
History
Advisory Group
State Agencies
•Department of Employment and Economic Development
•Department of Corrections
•Department of Education
•Department of Human Services
•MN State Colleges and Universities
Local Workforce Boards
•City of Minneapolis Employment and Training Program
•Minnesota Workforce Council Association
•Workforce Development, Inc.
Community Organizations
•Greater Twin Cities United Way
•Lukeworks
•Twin Cities RISE!
Business / Employers
•Dolphin Group
•MN Chamber of Commerce
Researchers / Evaluators
•Anton Economics
•Invest in Outcomes
•Macalester College
•Minneapolis Federal Reserve Bank
•Wilder Research
Overview and Basic Parameters
Framework Design
Values
Objectives
Goal
Scope of Programs
Collaborations
Considerations and
Trade-Offs
A framework for measuring and
understanding the net impacts and
social ROI of all publicly-funded
workforce programs that is standardized
and credible, and that informs strategy
and continuous improvement
Overview and Basic Parameters
Framework Design
Values
Objectives
Scope of Programs
Net Impact
Analysis
Collaborations
Considerations and
Trade-Offs
Cost-Benefit
Analysis
Oversight / Management
Framework
Supportive Policies and Infrastructure
Overview and Basic Parameters
Framework Design
Values
Objectives
Scope of Programs
Collaborations
Considerations and
Trade-Offs
Manageable, feasible to administer
Useful, relevant, timely
Credible, transparent, trusted
Adaptable, sensitive to change
Overview and Basic Parameters
Framework Design
Values
Objectives
Scope of Programs
Collaborations
Considerations and
Trade-Offs
Improving Services, Driving Value
“What works, and for whom?”
“What disparities exist?”
Making Smarter Investments
“How do current investments align to what works,
and to disparities in our community?”
Communicating Value
“How do workforce services benefit
participants and taxpayers?”
Standardizing the Approach
Strengthening Transparency/Accountability
Overview and Basic Parameters
Framework Design
Values
Objectives
Scope of Programs
Collaborations
Considerations and
Trade-Offs
Publicly-administered and funded
workforce programs
DEED and other state agencies
Non-profit passthroughs
Public education (elements of K-12 and PS)
State and federal competitive
grants and special initiatives
Programs serving targeted populations
(e.g. people with disabilities, veterans)
Long-Term Vision
Independent nonprofits and
education providers
Other service areas
Overview and Basic Parameters
Framework Design
Values
Objectives
Scope of Programs
Collaborations
Considerations and
Trade-Offs
United Way
Wilder Research
Invest in Outcomes/
State Pay for Performance
National Governors Association
Overview and Basic Parameters
Framework Design
Values
Objectives
Scope of Programs
Collaborations
Considerations and
Trade-Offs
One methodology, many programs
Ensuring usefulness to program
managers and policy makers
(Unintended) incentives created by
measure
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
Net Impact
Analysis
What’s Not Included
Net Impact Analysis
Data, Not Assumptions
Net Impact
Cost-Benefit
Analysis
Management Framework
Other Features
Policy Framework
Evaluation Design
Cost-Benefit Analysis
Benefits
Break-Even Point
$
Costs
B/C Perspectives
Benefits
What’s Not Included
Net Impact Analysis
Time
Data, Not Assumptions
Net Impact
Other Features
Costs
ROI = (Benefits – Costs) / Costs
(Return)
(Investment)
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Employment
Income / Fringe Benefits
Taxes
Income / Payroll / Sales
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Public Assistance Savings
MFIP / SNAP / UI
Healthcare Savings
MinnesotaCare / Medical Assistance
Incarceration Avoidance
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Net Impact Analysis
Program Costs
Time-weighted / Service-weighted
(where possible)
Data, Not Assumptions
Net Impact
Other Features
Cost to Participant
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Benefits and Costs
to Participants
+
Benefits and Costs
to Taxpayers
=
Total Social
Benefits and Costs
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Some public benefits
Subsidized housing costs
Prescription Drug Program costs
Child Support payments
Other important but
difficult-to-quantify effects
Change in mental and physical health
Change in worker productivity
Reduction in criminal activity
Economic multipliers
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
Cloning
Randomized Trials
B/C Perspectives
What’s Not Included
Net Impact Analysis
Data, Not Assumptions
Causality /
True Attribution
Net Impact
Other Features
Kernel Density Propensity Score Matching
Difference-in-Difference Estimator
“As good as random”
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
Foundation:
administrative data at the
individual level
What’s Not Included
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Avoid broad assumptions
wherever possible
Not Used:
Self-reported program
performance indicators, e.g.
entered employment rate
six-month retention rate
earnings change
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Earnings
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Not the relevant comparison
Treatment Group
Net Impact
Comparison Group
Time
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Accounts for Many Factors
Personal Characteristics
Geography
Local Economic Conditions
Services Received
Also allows us to analyze
performance by these categories
Evaluation Design
Cost-Benefit Analysis
Benefits
Costs
B/C Perspectives
What’s Not Included
Contextualized Performance Goals
Adjusted for population served, local conditions
Net Impact Analysis
Data, Not Assumptions
Net Impact
Other Features
Leading Indicators
For near-term relevance; based on statistical
relationships between near-term indicators and
long-term outcomes
Pilot Project
Scope of Programs
Treatment
Comparison
Data Sharing
Timeframe
Purpose
Primarily for internal use, to test concept,
methodology, data process
The pilot evaluation comprises 950,000 individuals
and roughly 50 million data points
Pilot Project
Scope of Programs
Treatment
Comparison
Data Sharing
Timeframe
Initial Cohorts
(2007-08 and 2009-10)
WIA Adult Program
WIA Dislocated Worker Program
Twin Cities RISE!
New Cohorts
(2010-11)
FastTRAC I&B Grantees
MFIP / DWP Employment Services
Adult Basic Education
SNAP Employment and Training
Pilot Project
Scope of Programs
Treatment
Comparison
Data Sharing
Timeframe
Registrants at WorkForce Centers
and on MnWorks.net
Unemployment Insurance
Applicants
Pilot Project
Scope of Programs
Treatment
Comparison
Data Sharing
Timeframe
Pilot Project
Scope of Programs
Treatment
Comparison
Data Sharing
Timeframe
New round of Data Sharing
Agreements recently finalized
Data are currently coming in
Results this Fall
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
Preliminary results address earnings and
employment impacts across two programs:
• WIA Adult
• Dislocated Worker (both WIA and MN)
Treatment cohorts are defined as such:
Cohort
WIA Adult 0708
WIA Adult 0910
DW 0708
DW 0910
Exit Dates
July 2007 – June 2008
July 2009 – June 2010
July 2007 – June 2008
July 2009 – June 2010
Additionally, some initial findings on
FastTRAC data are also provided.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
See disclaimer to the left.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
The analysis that has produced the following
preliminary results has been guided by the
GWDC Net Impact Advisory Group and is still
under development.
The preliminary results that follow have been
reviewed by program directors and relevant
staff at DEED, who emphasized the value of
the findings and voiced their support for the
continuation of the effort.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
Treatment and control groups
have been matched along a
number of variables. Tables 1-4
show how similar the cohorts
are. The main difference is with
regard to race; in all cohorts,
treatment cohorts have a lower
percentage of white individuals.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
Treatment and control groups
have been matched along a
number of variables. Tables 1-4
show how similar the cohorts
are. The main difference is with
regard to race; in all cohorts,
treatment cohorts have a lower
percentage of white individuals.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
Treatment and control groups
have been matched along a
number of variables. Tables 1-4
show how similar the cohorts
are. The main difference is with
regard to race; in all cohorts,
treatment cohorts have a lower
percentage of white individuals.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
Treatment and control groups
have been matched along a
number of variables. Tables 1-4
show how similar the cohorts
are. The main difference is with
regard to race; in all cohorts,
treatment cohorts have a lower
percentage of white individuals.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
PreEntrance Log Wages
.6
WIA AD 2007-2008 Entrance
.4
4
8
Log of Quarterly Wage
12
kernel = epanechnikov, bandwidth = 0.0282
PreEntrance Log Wages
.6
WIA AD 2009-2010 Entrance
.4
.2
Figure 1b:
WIA AD 0910
Kernel Density
Distribution of
Pre Wages
UI
AD
0
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
0
Kernel Density Estimate
These charts show how similar
pre-enrollment earnings are
between treatment and control.
For WIA Adult, wages are
slightly lower than the controls;
for Dislocated Worker, the
match is closer.
0
INTERPRETATION:
UI
AD
.2
Figure 1a:
WIA AD 0708
Kernel Density
Distribution of
Pre Wages
Kernel Density Estimate
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
0
4
8
Log of Quarterly Wage
kernel = epanechnikov, bandwidth = 0.0282
12
Preliminary Net Impact Results
PreEntrance Log Wages
.8
WIA DW 2007-2008 Entrance
.6
.4
4
8
Log of Quarterly Wage
12
kernel = epanechnikov, bandwidth = 0.0282
PreEntrance Log Wages
.8
WIA DW 2009-2010 Entrance
.6
.4
.2
Figure 2b:
DW 0910
Kernel Density
Distribution of
Pre Wages
UI
DW
0
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
0
Kernel Density Estimate
These charts show how similar
pre-enrollment earnings are
between treatment and control.
For WIA Adult, wages are
slightly lower than the controls;
for Dislocated Worker, the
match is closer.
0
INTERPRETATION:
UI
DW
.2
Figure 2a:
DW 0708
Kernel Density
Distribution of
Pre Wages
Kernel Density Estimate
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
0
4
8
Log of Quarterly Wage
kernel = epanechnikov, bandwidth = 0.0282
12
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
This table tells us the average
time in program is between
three quarters and a year, with a
lot of variation.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
Figure 3: Unmatched Wage Distribution: WIA Adult 0708
INTERPRETATION:
In the earnings charts that
follow, 0 represents time of
enrollment. We worked to
match earnings in the preperiod. The net impact on
earnings is the average
difference in the post-period,
specifically quarters 5-8.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Matched Average
Net Impact
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For WIA AD 0708, the results at
right translate to a net 30%
increase in earnings for
program participants,
controlling for other observable
factors. The statistical
significance of the result is still
in progress.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 3: Unmatched Wage Distribution: WIA Adult 0708
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For WIA AD 0910, the results at
right translate to a net 31%
increase in earnings for
program participants,
controlling for other observable
factors. The statistical
significance of the result is still
in progress.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 4: Unmatched Wage Distribution: WIA Adult 0910
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For DW 0708, the results at
right translate to a net 52%
increase in earnings for
program participants,
controlling for other observable
factors. The result is statistically
significant.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 5: Unmatched Wage Distribution: Dislocated Worker 0708
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For DW 0910, the results at
right translate to a net 31%
increase in earnings for
program participants,
controlling for other observable
factors. The result is statistically
significant.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 6: Unmatched Wage Distribution: Dislocated Worker 0910
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For AD 0708, the results at right
translate to a net 30% increase
in the likelihood of
employment, controlling for
other observable factors. The
result is statistically significant.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 7: Unmatched Employment Distribution: WIA Adult 0708
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For AD 0910, the results at right
translate to a net 29% increase
in the likelihood of
employment, controlling for
other observable factors. The
result is statistically significant.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 8: Unmatched Employment Distribution: WIA Adult 0910
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For DW 0708, the results at
right translate to a net 6%
increase in the likelihood of
employment, controlling for
other observable factors. The
result is statistically significant.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 9: Unmatched Employment Distribution: Dislocated Worker 0708
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
INTERPRETATION:
For DW 0910, the results at
right translate to a net 5%
increase in the likelihood of
employment, controlling for
other observable factors. The
result is statistically significant.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Figure 10: Unmatched Employment Distribution: Dislocated Worker 0910
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
A statistical analysis of FastTRAC is forthcoming;
FastTRAC data have presented a number of
challenges, many of which illustrate common
data challenges we face.
Funded as a pilot project through the Joyce
Foundation, the MN FastTRAC model was not
initially designed to measure outcomes based on
placement, but instead focused on educational
attainment among a hard-to-serve population
(likely MFIP participants).
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
The model also allowed flexibility across local
service providers, which created greater
differences in self-reporting outcomes by each
provider.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Specifically, gathering data on program participants has
presented the following challenges:
1.
Participant data did not require entry into one database
but relied on local systems. Data entry is now entered
into WF1.
–
Some participants are excluded from data altogether
depending on program completion, placement, or
continuation of their academic program.
–
Entrance/exit dates may be defined inconsistently across
programs
–
Program activities/services may be used and/or defined
inconsistently (trying to adapt to other programs within
WF1)
2.
FastTRAC participants are characterized in part by the
multiple barriers they face; accordingly, it may be more
difficult to find strong control group matches for them.
3.
Small sample sizes and variance among FastTRAC
participant characteristics make statistically significant
results harder to obtain.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
As FastTRAC has evolved, the pilot recognized
challenges with collecting data to measure
program impacts.
Progress is being made to make data collection
practices more complete and consistent across
FastTRAC programs.
Data practices are improving, but it will take time
for those changes to be reflected in net impact
analyses since those analyses require at least one
year of post-enrollment data.
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Preliminary Net Impact Results
Summary Statistics
Earnings Impacts
Employment Impacts
FastTRAC: Initial Findings
What’s Next
DISCLAIMER: The results reported here are
preliminary and are subject to further testing
and refinement that could alter the direction
and magnitude of the results. A final report is
forthcoming later in 2014.
Further analysis is currently underway. Here’s
what to expect:
1.
Further refinement of matching and
estimation, to improve the statistical
significance of the results.
2.
Additional net outcomes measured over
longer time periods, including usage of public
benefits, reincarceration, and associated
monetary (ROI) impacts.
3.
Additional programs to be analyzed, including
FastTRAC, Adult Basic Education, MFIP
Employment Services, and SNAP Employment
and Training.
4.
Results disaggregated by participant
characteristics (e.g. race, gender) and other
factors.
Wrapping Up
Discussion and Questions
Raymond Robertson
Professor of Economics
Macalester College
robertson@macalester.edu
Nick Maryns
Senior Policy Analyst
Governor’s Workforce Development Council
Nicholas.maryns@state.mn.us