Accelerating Opportunity
Evaluation
Planning the Evaluation with the
Accelerating Opportunity States
February 10, 2012
11:30 a.m. – 1:00 p.m.
Webinar Agenda
Introduction of the Evaluation Team
Goals of the Evaluation
Evaluation Design
Plans for the Evaluation
Q&A with the Evaluation Team
AO Evaluation Team
Randall Wilson, JFF, Evaluation Director
Robert Lerman, Urban Institute, Principal
Investigator and Cost Analysis Lead
Lauren Eyster, Urban Institute, Project Director
and Implementation Analysis Co-Lead
Maureen Conway, Aspen Institute,
Implementation Analysis Co-Lead
Burt Barnow, George Washington University,
Impact Analysis Lead
Goals of the AO Evaluation
To generate evidence for state and federal
policymakers, college administrators, funders, and
other stakeholders about:
1. The process of implementing integrated college
and career pathway designs and taking these
designs to scale
2. Their impact for ABE and ESL students in
college and in the labor market
3. Their cost effectiveness and financial
sustainability
AO Evaluation Design
Impact on AO
Students
Implementation
and Scalability
Cost
Effectiveness
and Financial
Sustainability
Implementation Analysis
Questions
• How did states and community colleges
establish integrated pathway designs?
• What shifts in the culture (attitudes and
behavior) occurred within states and colleges
concerning ABE/ESL students?
• How did states and community colleges move
toward scale and the sustainability of pathways
through policy changes, innovative financing, the
use of data to support continuous improvement,
and other means?
Site Visits to States and Community
Colleges
• Twice during the grant period (Fall 2012 and
Spring 2014)
• Three-day visits with two evaluation team
members
• Two community colleges will be visited during
each site visit
• Selection of the colleges will be determined with
the state team
Community College Surveys
• Used to collect systematic data on implementation,
financing, and sustainability across ALL community
colleges
• Fielded once after the end of the first year of the
grant (December 2012) and once at the end of the
grant period (October 2014)
• Web-based survey but will be available in paper if
requested
• Will work with the states in fielding the survey –
reaching the right contacts at each college and
following up with non-responders
Quarterly Check-in Calls
• To keep updated on changes to AO program
design and implementation
• To provide states with an opportunity to ask
questions about the evaluation activities
• Will occur March, June, September, and
December each year during the grant period
Impact Analysis Questions
• What impacts do integrated career pathway
designs have on student progress and outcomes
in college and in the labor market?
• How do outcomes for participating students
change relative to comparison groups of
students similar to the population in AO, but not
participating in the program?
Requirements for the Impact Analysis
• Ensure that AO education and support services
differ significantly from services received by those
who do not participate
• Have a well-articulated selection process, which is
consistently applied by colleges
• Identify an appropriate comparison group whose
outcomes will be compared to those of AO
participants
• Have enough participants and comparison group
members to be able to detect statistically significant
differences in outcomes
Planned Quasi-Experimental Designs
Regression discontinuity design
• Compares the treatment group to nonparticipating applicants to the AO program
who are close to a test cutoff score used to
determine program eligibility
Propensity score matching
• Uses a large pool of similarly prepared and
motivated ABE/ESL students in the state to
compare to the treatment group on outcomes
of interest
Data for Impact Analysis
• Data need to be of high quality and collected for
both treatment and comparison group members
• Data will be submitted twice per year to the
Urban Institute through secure file transfer
protocols
• Test and retest scores on ABE/ESL tests are
necessary for developing comparison groups
• Data collected by states will include student
characteristics, services received, educational
progress, and labor market outcomes
Cost Analysis Questions
• What are the benefits and costs of implementing
and scaling up integrated pathway designs to
states, community colleges, and students?
• What methods were used to finance the
pathways, and how financially sustainable is the
initiative in the state?
• The key issue is how much more resources
must be devoted to the AO model than do
standard ABE/ESL programs and who bears any
extra costs
Data for Cost Analysis
• Template will be provided for documenting state
financing plans/budgets and college costs.
• College cost data will also be collected through
the implementation surveys.
• Student cost and foregone earnings data be
collected through semi-annual data reporting.
• Data on financial sustainability issues will be
collected in surveys and during site visits.
One-Day Planning Visits to States
(now scheduled for March)
• To learn more about states’ program design,
including selection procedures, test scores, and
career pathways
• To gain an understanding of data availability to help
the evaluators develop analysis plans for the
evaluation
• To work closely with the states in identifying
comparison groups
• To continue developing the states’ understanding of
the evaluation and its requirements
Who Should Participate in Planning Visits?
From states:
• State team leads
• State staff responsible for data submission
From one or two community colleges:
• College representatives who are knowledgeable
about pathway design, recruitment and selection
processes, and student data
Burt or Bob will each visit two states and will be
accompanied by JFF or other evaluation team staff
Main Topics for Visits
Continued discussion on the evaluation design and help the states and
community colleges meet evaluation requirements
AO program design, specifically:
• Recruitment and intake
• Eligibility requirements
• Educational milestones for students
• Contrast with services received by other adult education
students
Availability of data
• Student characteristics at intake, participation in AO pathways,
educational progress, and labor market outcomes
• Data confidentiality and security and FERPA
• Costs and benefits to students, community colleges, and states
Tools for Collecting Student Data
• Intake form (draft provided for review and
comment)
• Data template for semi-annual reporting (draft
provided for review and comment)
• Template for collecting state and college cost
data will be provided by spring 2012
Overall Evaluation Activity
Timeline
Evaluation Activity
Dates
Initial Evaluation Planning Visits
(One-Day Visits)
March 2012
First Full Site Visit (Three-Day Visits)
September-October 2012
First Community College Survey
December 2012
Second Full Site Visit (Three-Day Visits)
April-May 2014
Second Community College Survey
October 2014
Ongoing Activities
-Quarterly Check-in Calls
-Semi-Annual Data Submissions
March, June, September, and
December during the grant period
June 2012-2014 and January 20132015
Q&A
Accelerating Opportunity
Evaluation Plans