Early Lessons from
TAACCCT Evaluation
by Donna Tonini, Research Specialist, OCCRL
National Council for Workforce Education Conference
Milwaukee, WI 10/16/2013
Who is OCCRL?
Our Mission: A research office which studies policies, programs,
and practices designed to enhance outcomes for diverse youth and
adults who seek to transition to and through college to
employment.
Examples of our evaluation projects include:
•TAACCCT Grant Evaluations
•Shifting Gears
•Adult Education Bridge Program Evaluations
•College and Career Readiness
•Adult Career Pathways
What is TAACCCT?
•
Department of Labor (DOL) Grant: Trade Adjustment Assistance
Community College and Career Training, 10/2011 – 9/2014
•
Purpose: to provide funding to community colleges to scale up and
improve short-term (< 2 years) educational & career-training programs to
prepare students for employment in high-skilled, high-wage jobs.
•
Target Population: While the grant supports TAA (Trade Adjustment
Assistance) workers (who were retrenched /downsized due to overseas job outsourcing),
the DOL also expects the programs will benefit a wide range of individuals.
•
Goal: to enable the higher education community to develop, institute and
improve education and training programming which will assist the
target population in acquiring the skills, degrees, and credentials to meet
the high-skilled and high-wage employment needs mandated by
employers.
Understanding Priorities of the Sponsor and the
Goals of the Colleges
The DOL’s Round I Priorities for TAACCCT:
•Accelerate
progress for low-skilled workers
•Improve
retention and achievement rates and/or
reduce time to completion
•Build
programs that meet industry
•Strengthen
online and technology-enabled learning
Introducing the Colleges:
H2P: Health Professions Pathways Consortium
•
Cincinnati State – Lead (OH)
•
Anoka-Ramsey Community College (MN)
•
Ashland Community and Technical College (KY)
•
City Colleges of Chicago (IL)
•
El Centro College (TX)
•
Jefferson Community and Technical College (KY)
•
Owens Community College (OH)
•
Pine Technical College (MN)
•
Texarkana College (TX)
TAACCCT Strategies – H2P
•
Online assessment and enhanced career guidance
•
Contextualized developmental education
•
Competency-based core curriculum
•
Industry-recognized stackable credentials
•
Enhanced retention support
•
Training programs for incumbent health professions workers
•
Enhanced data and accountability systems
•
Galvanize a national movement
Develop Evaluation Plan Based on Priorities
& Goals - Three Linked Studies
Implementation Evaluation of the model, including
Programs of Study and core strategies, as articulated
through consortium theory of change.
Impact Evaluation of short- and longer-term
outcome measures for participants, including a
rigorous evaluation (quasi or true experimental
design).
Continuous Improvement through regular review of
the measures and outcomes, using consortium and
college data to support sustainability and scale up.
Implementation
Outcomes
Continuous
Improvement
Develop Questions that Address
Priorities & Goals
What is the baseline and progress on implementation related to
TAACCCT-impacted Programs of Study and strategies?
•Who
are your partners and what roles do they play in supporting the
implementation of TAACCCT-impacted Programs of Study and strategies? How
are student outcomes affected?
•What
strategies and activities are the consortium and co-grantee colleges
implementing, and what is the expected impact on the Programs of Study? How
are student outcomes affected?
•What
modifications and improvements are being made to funded Programs
of Study? How are student outcomes affected?
•What
barriers and supports exist, and how do they impact colleges’ efforts
in creating and enhancing Programs of Study and strategies? How are student
outcomes affected?
Lessons
•
Differing interpretations of DOL requirements
•
Clarity of expectations
•
Information sharing
•
Defining a common set of measurables
•
Creating value for the client
Open, Clear, and Constant Communication!
•
Identify key stakeholders:
 Program leadership
 Institutional Review (IR)
 External partners (Employers, Workforce Agencies)
•
Involve stakeholders in process:
 Engage colleges in shared vision
 Develop logic models
•
Leverage promising practices:




Communities of Practice (COP)
Monthly teleconferences
Basecamp
Blogs
Sample Logic Model: Implementation of H2P Grant Strategies
1. Online
Assessment &
Career
Guidance (No.
students, No.
credits)
Select and
Place
Par cipants
2. Contextualized
Dev Ed (No.
students
enrolled, No.
students increase
to college level,
No. students
earning academic
credit)
6. Incumbent
training (No.
enrolled, Min.
semester credit)
7. Data (No. students w/
educa on and
employment records)
3. Core Curriculum
(No. students
enrolled, No.
students complete)
5. Stackable
creden als (No.
cer ficates per 100
FTE, No. cer ficates)
4. Reten on (No. text
messages, No. enroll
fall to spring, No.
enroll fall to fall, %
completed of
a empted)
Outcomes:
• Employment
• Employment
reten on
• Average
earnings
• Credit
a ainment
• Cer ficate
a ainment
(< 1 and >1)
• Degree
a ainment
8. Na onal movement
(No. using labor market
info, No. core curriculum)
(Graphic developed by D. Bragg, 2013)
Develop Data Collection and Analysis Plans to Address
Priorities & Goals
List TAACCCT Partners (include
New
employers, community, workforce & other
Partnership?
relevant entities )
Partner 1
Partner 2
Partner 3
Partner 4
Partner 5
Role
Explanation Section
Tools for Collecting Data on Strategies, POS
TAACCCT Activities
No
No
Improved
Facilities
and/ or
Infrastructure
2012 No
No
Year
Month
6
Other
Yes/No
Yes
No
Year
Year
Month
Yes/No
Year
Month
Year
Yes/No
No
No
Upgrading
Classroom
Supplies/
Equipment
Month
Yes 6 2012 No
No
Yes 5 2014 Yes 5 2013 No
Month
Professional Internships or
Development ApprenticeFaculty/ Staff
ships
Yes/No
Year
Month
New
Courses
Yes/No
Year
Month
Yes/No
Year
Month
Yes/No
Year
Enhanced/
Upgraded
Curriculum
Yes/No
EXAMPLE: Medical Assistance Certificate
Month
New
Current Technology
Is this POS
and
New or
Planned
Enhanced? Activities
Current Yes 4 2012 No
Enhanced
Planned Yes 4 2013 No
Yes/No
Program of Study (POS)
Additional
course
Instructors
Highlighting Promising Practices
•
Curricular Innovations:
•
 Core curriculum
 Contextualized
developmental education
 Galvanizing a national
movement
•
Improved Student Services:
 Dedicated Retention
Specialist/Career Coach
 Online prior learning
assessments (PLA)
Strengthened Career
Pathways:
 Stackable credentials
 Incumbent worker
training programs
•
Enhanced Partnerships:
 Employers
 Workforce agencies
Locating the Stories to Tell
What is the baseline and progress on implementation related to
TAACCCT-impacted Programs of Study (POS) and strategies?
POS: The largest number of POS are Associate’s degree programs.
Top 3 (of 8) Strategies at Implementation Level:
* Enhanced Retention Support (8 of 9 colleges)
* Online Assessment and Enhanced Career Guidance (7 of 9 colleges)
* Enhanced Data and Accountability Systems (7 of 9 colleges)
Commonalities:
Barrier: Lack of TAA participants (1)
Strength: Value added by external partners (employers, etc.)
Area for Improvement: Communication, info-sharing (inter-college)
Finding Patterns in the Data
How are the consortia, students and employers impacted?
Outcomes:
•Enrollments
-
H2P: 3,835
Credentials - H2P: 430
Sample Outcomes from student demographic course-level data:
•Completers
•Upcoming
– Retention – Credit Hours -
Employment
Analysis: Multi-level analysis of student level data - degree
attainment, wage increases.
•Pathways to Results – Continuous Improvement
Annual Performance Report – Table 1
Obstacles and Opportunities For
Learning
•
Please share your experiences
from the field for discussion.
Office of Community College Research and Leadership University of
Illinois at Urbana-Champaign 51 Gerty Drive, 129 CRC | Champaign, IL
61820 | Phone: (217) 244-9390 | Email: occrl@illinois.edu