Emerging Issues of Data Collection
in TAACCCT Grant Implementation
National Council for Workforce Education
2014 Annual Conference
October 30, 2014
Presenters:
Pradeep Kotamraju
Bureau Chief, Career & Technical Education
Division of Community Colleges
Iowa Department of Education
Stephanie Oppel
Statewide Director
Iowa-Advanced Manufacturing Consortium
Des Moines Area Community College
TAACCCT in Iowa
• U.S. Department of Labor’s Employment and Training
Administration awarded over $2 billion to individual and
consortium grantees across the nation over four rounds of grant
awards.
• Approximately $35.5 million to Iowa across four rounds of grants
• Expand and improve education focusing on career pathways,
stacked and latticed credentials, enhanced industry engagement
supporting economic growth.
• TAACCCT focus on TAA eligible workers, veterans, unemployed
and under employed individuals.
TAACCCT in Iowa
• Bridges2Healthcare: Multi-state Consortium - $12,695,959
($3,284,296 total awarded to Iowa colleges - NICC and HCC)
• Engineering Technology: EICC - $2,500,000
• Iowa-Advanced Manufacturing: Statewide Consortium $12,951,165
• Information Technology Credentials to Careers: IWCC $2,500,000
• Iowa’s Information Technology, Healthcare, Utilities and
Manufacturing Network: Statewide Consortium - $15,000,000
In combination, these initiatives are projected to
impact more than 15,000 individuals, arming them
with the skills they need to engage in today’s workforce.
Priorities of the Iowa Advanced Manufacturing Consortium
• Priority 1: Build stacked and latticed curriculum and career
pathways in Signature Programs
• Priority 2: Build a steady pipeline of skilled workers for Iowa’s
advanced manufacturing in-demand occupations
• Priority 3: Improve the collaboration and alignment between
community college programs, the workforce system and targeted
industry employers to keep and create high quality jobs in Iowa.
Areas of Focus
• Alignment with relevant industry credentials
• Strengthen credit for prior learning
• Stacked and latticed career pathways
• Model for intensive advising
• Statewide outreach campaign
• Enhance technology enabled learning
• Strengthened relationships with industry
• Strengthened relationships with workforce
Elements of Data Collection
• Practical – Grantee collecting to help student achieve program
success, aid in implementation; carries into other reporting and
evaluation
• Compliance – Grantee collecting for reporting outcomes and
implementation progress to the U.S. DOL on a quarterly and
annual basis
• Implementation Evaluation – Third Party Evaluation team
preparing progress and final reports on the impact of the grant
• Outcomes Evaluation – Third Party Evaluation team preparing
outcomes analysis reports on the impact of the grant
Grantee and Third Party Evaluation team work collaboratively to
inform all aspects of data collection
The Grantee’s Role in Data Collection
• Practical Data
• Critical roles – Pathway Navigators, Faculty, Advisors
• Grant impact – Offerings, pathways, institutional practices all aimed
at increased student retention, completion and entrance into the
workforce
• Compliance Data
•
•
•
•
Establishment of consortium policies in line with DOL requirements
Continuous education and training for grant personnel
Creation of tools for collecting data
Establishment of MOUs to gain additional data sets
Grantee and Third Party Evaluation team work collaboratively to
inform all aspects of data collection
Navigating Data Collection on the Student Pathway
In any TAACCCT project, a student follows a pathway from
referral and college entrance, through the program study and
to the point of exit and employment. There are different
aspects of data collection as students navigate the pathway;
correspondingly, these aspects have training requirements for
staff and other stakeholders. Aspects of collection along the
pathway are:
•
•
•
•
•
Workforce Development Services Eligibility
Community College Entry
Enhanced Retention Support Services
Academic Program Progression
Graduation Exit and Employment Entry
Touchpoint: Workforce Development Service Eligibility
• For TAA and other dislocated workers, workforce development staff are
the first point of contact for finding training programs
• Many potential participants are referred by workforce development
agencies and career services to community college programs, as has
been the case in for many TAA-eligible workers.
• Community colleges work to educate workforce development staff
about available programs.
• Workforce development staff collect relevant data regarding prior
employment experience, education and training prior to the
employment dislocation, and any relevant information that can be of
potential use during community college application process.
• Students are referred to pathway navigators who continue to help
students determine which program they would be interested in and
suited for at the college
Touchpoint: Community College Entry
• Once the TAA-eligible worker or other individual chooses to enter
community college, there is specific information that needs gathering.
• This information could include things like prior learning experience,
more in-depth information regarding the applicant’s previous education
and training, any assessments or any pre-qualifications tests required
for program entry.
• Much of this information is entered on an intake form, either handwritten or electronically. Often the institution makes provision to enter
that data into its standardized student record databases.
• Connecting information on the intake form to standardized student
record databases often requires planning
• Goal here is to gather enough information about the applicant’s current
personal life and prior education and workforce training experience to
understand what student support services the applicant might require.
Touchpoint: Enhanced Retention Support Services
• A key element of TAACCCT programs is the provision of student support
services that target the retention of students through enhanced
supports.
• Given the variety of student support services offered and used, multiple
forms of data collection are necessary, including perceptual data
gathering techniques (e.g., focus groups, interviews) and service
delivery tracking (e.g., student interaction portfolios, checklists).
• The collected information should be linked to the extent possible to
intake forms and to information in the institutional and workforce
databases in order to understand the connections between and among
background circumstances, program elements, and outcomes.
• Need to emphasize the importance of gathering consistent data at the
most micro level possible (e.g., individual level whenever possible,
program level if not individual), and understanding how linkages must
be made between the different sources and methods of data collection.
Touchpoint: Academic Program Progression
• While other data collection sources and methods are used within
this stage, the primary source for data collection is the
institutional records that reside within the wider information
technology system.
• During this stage, much of the information required to gauge
academic progression is generally available in the institutional
record database.
• The student records data can be supplemented by other methods
of data collection, such as surveys. It is essential that the
methods by which data from these institutional systems are
extracted following standard protocols.
Touchpoint: Graduation Exit and Employment Entry
• Like the workforce development services eligibility stage, when it
comes to data collection, this stage requires explicit cooperation
between two different state agencies: workforce development
and the postsecondary education authorities.
• Requires the transfer of information between the two agencies to
gather information about the employment placement and
employment retention wages of graduates who have completed a
specific educational program.
• Protocols need to be established to facilitate the data collection
and transfer. These protocols could be formal such as
memoranda of understanding, or they could be less formal, such
as a letter of agreement.
• This stage focuses on the specific data needs of the two agencies,
the method of transfer, and ensuring data security.
I-AM: Workforce Development Service Eligibility
• Rapid Response to TAA and other closures and layoffs
• Workforce and Colleges connect with workers jointly
• Provide program information
• Workforce referrals and local relationships
• IWD/CC Partnership
• Colleges educate workforce staff on programming – local and state level
• Program recommendation and referral
• Self-assessment tool
• NCRC
• Program and support services eligibility
Practical data gathered through this process enables
workforce and college to deliver better service to students
for better outcomes
I-AM: Community College Entry
• Entrance Interviews, Testing & Advising
• Pathway Navigator
• Required entry exams & orientation
• Participant Intake Form
• Standard across colleges for all grant
participants
• Focus on DOL Required reporting
elements
• Developed with input from 3rd party
evaluation team
• Participant Database
• Web based, secure, InfoPath database
• Simple interface, user guide, webinars and
one on one training of users
• Embedded reports
Compliance data and practical data are gathered as the student enters college.
Pathway Navigators play a critical role in gathering both sets of data to ensure
accurate reporting and help ensure student success.
I-AM: Enhanced Retention Support Services
Practical Data & Navigators
• Advisor, coach,
cheerleader, tutor,
recruiter and MORE!
• Facilitates intake and
helps link student to
support services
• Intrusive advising model
• Tools for student tracking
During this aspect of data collection, I-AM staff focus on the individual student,
working with students to maximize opportunity and support to increase
retention, completion and career exploration.
I-AM: Academic Program Progression
• I-AM Database
• Guidelines and timelines for
updating
• Relies on student information
systems
• College and consortium access
• Follows student from intake to
exit
I-AM Consortium staff developed a step by step user guide for frontline data
entry staff to help ensure consistency of data. Webinars and one-on-one
trainings take place continuously. Data monitoring at local colleges provides IAM Consortium staff an opportunity to compare student records system data
to I-AM database data.
I-AM: Graduation, Exit and Employment Entry
• I-AM Database
• Exit and completion information
• Challenges with student
information systems
• Wage Data from IWD
•
•
•
•
MOU for data sharing
Secure data transfer
Reporting timeline issues
IDOE facilitation
Grantee Compliance Reporting
• Outcome data for DOL and to focus activity and resources
• College calendar v. grant calendar
• Ensuring accuracy at colleges
• Availability of data from wage records
• Financial reporting for DOL and to help manage spending of
colleges
• Required quarterly reporting
• Optional monthly reporting
• Programmatic reporting for DOL and to help manage grant
activities at each college
• Required quarterly reporting using form
• Quarterly and annual meetings with college teams
Compliance Reporting
• Gathering data
• Availability of data from wage records
• College calendar v. grant calendar
• Financial reporting for DOL and to help manage spending of
colleges
• Required quarterly reporting
• Optional monthly reporting
• Programmatic reporting for DOL and to help manage grant
activities at each college
• Required quarterly reporting using form
• Quarterly and annual meetings with college teams
Third Party Evaluators and Data Collection
• Implementation Evaluation
• Milestone Evaluation
• Data Source: Quarterly reports submitted by Project Leads
• Purpose: Track implementation progress & status
• Evaluation of Activities
• Data Source: Surveys of key stakeholders
• Purpose: Assess program benefits/strengths & challenges; formative &
summative information
• Outcome Evaluation
• Meeting DOL requirement for rigorous analysis of grant impact on
participants – Quasi-experimental design
• Establishing comparative cohort
• Accessing data records across sources
Implementation Evaluation: Milestones
Initial Reporting
• Quarterly Report
• General overview of
progress
• Covered only strategies
• True progress unknown
• Evaluation of progress was
cumbersome due to
reporting structure
Changes in Reporting as a Result
of the Evaluation
• Quarterly Report
• Very specific descriptions of
progress
• Covers each milestone
within each strategy
• True progress is known
• Evaluation of progress is
easier/faster
Implementation Evaluation: Activities
Surveys
Challenges
• Student (upon program
completion)
• Student Exit (6 months
after completion)
• Project Leads
• Welding Faculty
• Other Faculty
• Marketing
• Employer/Partner
• Committee
• College Leadership
• Large number of
stakeholder groups that
require unique surveys
• Difficulty in reaching survey
respondents due to
differences in how
community colleges
communicate with their
respective stakeholders
Outcome Evaluation: QUASI-EXPERIMENTAL DESIGN
Focus:
Students enrolled in credit welding programs at Iowa community colleges
between Fall, 2013 and Spring, 2014.
Why welding?
Largest signature program in I-AM.
13 (now to be 15) of the 15 participating Iowa community colleges have
credit welding programs funded by I-AM grant.
Outcome Evaluation: Quasi-Experimental Design
Focus: Students enrolled in credit welding programs at Iowa community
colleges between Fall 2013 and Spring, 2014.
Why welding?
Largest signature program in I-AM.
13 (now to be 15) of the 15 participating Iowa community colleges have
credit welding programs funded by I-AM
Data Sources:
Outcome Evaluation: MIS & IWD Data Timeline
Outcome Evaluation: Access to Data
Challenge of accessing data:
• Identification of stakeholders
• Grantee, Evaluator, State Agencies
• Identification of protocols for records release
• State agency policies and legal requirements
• Systems for distribution and retention of records
• Gaining CONSENSUS
• Aggregate data easily available
• Individual data difficult to obtain
• Development of a Memorandum of Understanding (MOU)
Grantee, Evaluation team, and state agencies all play a role in
Emerging Issues of Data Collection in TAACCCT Grant Implementation
Takeaways….
• Grantee and evaluation team must work collaboratively to
inform all aspects of data collection
• Cohesive data and evaluation plan
• Gaining access to data
• Availability of data from institutional records
• Identifying key data stakeholders – state agencies
• Begin MOU work early
• Training data collectors
• Face to face, webinars, troubleshooting
• Monitoring
• Communication with stakeholders