Youth Performance
Accountability
Data Accuracy and Reporting
for WIA and WIOA*
CWA Conference
Long Beach, CA
January 27, 2015
*As known today
1
Objective
Mutual Understanding of Data CollectionEntry-Reporting accountability from Local
Areas to State to DOL.
 Encourage discussion of data collection
and reporting requirements, procedures,
and guidance.
 Establish open forums for communication
for technical assistance.

2
Background

Oversight agencies like GAO and OIG
cite data quality issues with ETA’s data
(2002 and 2008)

Guidance issued annually containing
report submission deadlines and source
documentation requirements
 TEGL
6-14 for PY13/FY14 Reporting and
Data Validation
3
Policies/Procedures and
Training


Data management and the resultant quality of
reported data are derived from and influenced
by the policies, procedures and protocols
utilized at the state and/or local levels
Grantees should develop guidance for staff and
sub-grantees involved in the collection of data:




Definitions of data elements
Sources of information
Participant record and documentation requirements
Procedures for collecting, entering and reporting data and
associated “business rules” that cover timeliness and
completeness
 Procedures for entering data into an automated database
 Procedures for correcting data
4
Training and Monitoring

Data collection and data entry:
Routine training should be provided for
data management guidance
 All staff involved in the collection or entry of
data should be trained in the procedures
 The data entry process should include
steps for verifying entered data against
original sources on a sample basis or for
entire population of records

5
REPORTING
 If
Data is not
collected and
reported, it
didn’t happen!!
6
WIA (Workforce Investment
Act) effective through PY15






‘Services’ based
Participation and Exit
Data Validation required
Reporting Cohort
primarily 1st to 3rd Qtr
after Exit
Nine Common Measures
Reporting Participant
Information
Sequence of Services –
Core, Intensive, Training
WIOA (Workforce Investment and
Opportunity Act) effective PY16*






‘Services’ based
Participation and Exit
Data Validation codified
Reporting Cohort
extended 2nd to 4th Qtr
after Exit
Twelve Primary Indicators
of Performance
Expanded Reporting
Participant Information
‘Career Services’ and
Training
Performance Accountability
*unless State ‘early implementer’
7
WIOA Youth Program Design

Focus on Out of School Youth
 75%

Work Experience Expenditure
 20%

of formula youth funds minimum
of local area funds minimum
Eligible Out of School Youth Age
 16-24

year olds
New Program Elements
8
WIOA Youth Program Transition

Youth PY 2015 Funding
 April

1, 2015
WIOA Implementation
 July
1, 2015
While WIOA is effective July 1, 2015 and the Youth funds
allotments are provided April 1, 2015, and it will take some time to
transition, States and Local Areas must begin to incorporate
strategies for recruiting and serving more OSY with the receipt of
PY 2015 funds. Progress must be documented, including plans to
achieve 75% expenditure rate with future funds.
9
Participation
10
Participant
An individual determined eligible to
participate in the program who receives a
service funded by the program in either a
physical location (e.g., One-Stop Center)
or remotely through electronic
technologies.
 Three Components

 Determined
eligible to participate in the
program
 Receives a funded service
 In either a physical location or
through electronic technologies
11
Components of Participant

Individual determined eligible to participate
 Depends
on program/funding; doesn’t apply in the case of
W-P, which is based on universal access

Receives a service
 Not
all services trigger participation; it’s important to
understand the distinction between those that do and those
that don’t

In a physical location or remotely
 Many
substantial services are remotely accessed; this
needs to be captured
12
Multiple Program Participation
Counting Participants in Multiple Programs


Earliest date of service
Can participate in several programs
simultaneously
Counted as a participant in each of those
programs
The participant won’t exit from the program
unless there is a gap of no service for 90
days
13
WIOA Youth Program
Participant
When does participation begin?
 What services commence participation
and inclusion in performance
accountability?

14
A Service Is:
Any core, intensive or training activity
made available to eligible participants that
allows them to benefit from specific
programs in the workforce system.
15
When a Service is included in
Performance?

Core, intensive or training services made
available to eligible participants and
require significant staff involvement who
exit the program.
 These
aforementioned individuals are
included in the performance measures
 Those WIA Adult and DW program
participants who only receive self service or
informational activities are excluded from
performance
16
Services that Do Not Begin or
Extend Participation
Eligibility determination
 Case management administrative
activities to obtain information regarding
employment status, educational progress,
need for additional services, etc.
 Income maintenance or Support payments
 Visitors to One Stop Centers, etc., for
reasons other than its intended purposes
 Follow-Up Services

17
Exit
18
Exiter
A participant who hasn’t received a
program or partner-funded service for 90
consecutive days and no future services
are scheduled
 Three components

 Hasn’t
received a service
 For 90 consecutive days
 No future services scheduled
19
Components of Exiter

The participant hasn’t received a service
 Could

be program- or partner-funded
For 90 consecutive calendar days
A
gap in service can stop the 90-day clock if based on
specific/allowable circumstances

No future services scheduled
 Specific
services and activities as allowable
 Does not include any follow-up services or
circumstances where the participant voluntarily
withdraws or drops out of the program
20
Illustration: Participation and
Exit
Eligible and
Receives
Service
Participation
Last
Service
End of 90Day Period
No Services or Follow-Up Services
Participation
Date
Exit Date
21
Non-Compliance with EXIT
Requirements






Exit dates not reflective of dates of last
service
‘Case management’ used to extend exit date
Hard exits utilized
 Date of last contact = Exit date
 Date of employment = Exit date
Services provided within 90 days
Lack of common exit date (across core
workforce programs)
Exit dates not consistent with dates in MIS
22
How to Prevent Unintentional
Exits: Gaps in Service

Three allowable circumstances - the
condition must exist for at least 90
days



Delay before beginning of training
Health/medical condition of participant
or family member
Temporary move from the area that
prevents participation
23
Exclusions
Institutionalized
 Health/Medical or Family Care
 Deceased
 Reserved Forces called to Active Duty
 Relocated to Mandated Program (Youth)
 Invalid/Missing Social Security Number

24
WIOA Youth Program Exit
When is it appropriate to exit Youth
participants?
 Common or Program Exit?

25
Follow Up Services

Follow-up begins after the expected last
service

Youth are required to receive at least 12
months of follow-up services, which are
triggered at exit (the only exclusion is for
summer youth employment)

Not intended to take responsibility away
from case managers for WIA. Case
managers do not have to wait 90 days,
for instance, to begin providing follow-up
services.
26
Total Youth Common
Measures
27
PLACEMENT IN EMPLOYMENT
OR EDUCATION
Employed, in the military
or in post-secondary
education at participation?
YES
Excluded
YES
Numerator
NO
Has a qualifying outcome* in
the 1st quarter after exit?
NO
Qualifying Outcomes: employment, military,
enrolled in post-secondary education, advanced
training, or occupational skills training
Denominator
28
Placement in Employment or Education
Reporting Time periods
For the Quarter ending Sept 2014
2013
Jan-Mar
Apr-June
July-Sept
2014
Oct-Dec
Jan-Mar
Apr-June
July-Sept
Oct-Dec
Exit Cohort
Employment/Education Qtr
2 Qtr Wage Delay
Nov 15th
29
LITERACY/NUMERACY GAINS
Definition: Of those out-of-school youth
who are basic skills deficient, the
percentage who increase one or more
educational functioning levels within one
year of participation

30
Literacy/Numeracy Definitions
Basic
Skills Deficient
The
youth computes or solves problems, reads,
writes, or speaks English at or below the eighth
grade level or is unable to perform these tasks at a
level necessary to function on the job, in the
individual’s family, or in society.
States
and grantees may develop their own
definition, but it must include the language above.
31
Definitions (Continued)
Out-of-School
Youth
High
School Dropout (no diploma or
equivalent)
High
School Graduate (or equivalency) not in
postsecondary
High
School Graduate (or equivalency) in
postsecondary – but basic skills deficient
32
About the Assessments
All out-of-school youth must be assessed in
basic reading, writing and math

Pre-testing must occur within 60 days of the
first youth program service; can use pre-test
from up to six months prior to date of first
youth service

The same standardized assessment must
be used for pre- and post-testing

Youth should be post-tested by the end of
one year of participation and compared to
pre-test results obtained during initial
assessment

33
About the Assessments

If a youth continues to be basic skills
deficient after the first 12 months of
participation, they should continue to receive
training in literacy and/or numeracy skills

Youth should be post-tested and included in
the measure at the completion of the 2nd year
only if they complete two full years in the
program
34
Literacy/Numeracy Gains - 1st
Year of Participation
Of those out-of-school youth who are
basic skills deficient, the percentage who
increase one or more educational
functioning levels within one year of
participation

Number of youth participants who increase one or
more educational functioning levels
Number of youth who
completed a year of
participation (based on
date of 1st youth service)
+
Number of youth who
exit before completing
a year of participation
35
Literacy/Numeracy Gains
2nd and 3rd Years of Participation
Of those out-of-school youth who are
basic skills deficient, the percentage who
increase one or more educational
functioning levels within one year of
participation

Number of youth participants who increase one or
more educational functioning levels
Number of youth who completed a second or third
full year of participation (based on anniversary date of 1st
youth service)
36
ATTAINMENT OF DEGREE OR
CERTIFICATE
Enrolled in education at
participation or any time
during participation?
NO
Excluded
YES
Numerator
YES
Attained diploma, GED or
certificate by the end of the
3rd quarter after exit?
NO
Denominator
37
Credentials Defined
Awarded in recognition of an individual’s
attainment of measurable technical or
occupational skills necessary to obtain
employment or advance within an
occupation.

Work
Readiness Credentials and those
awarded by workforce investment boards are
NOT included in this definition.
38
Credentials
TEGL 15-10 established ‘credential’ as the
umbrella term encompassing postsecondary
degrees, diplomas, licenses, certificates,
and certifications.
 All states, regardless of the common
measure waiver, must collect information on
credential attainment in the WIASRD

39
Credentials: Approved Awarding
Institutions
A state educational agency, or a state agency responsible for
administering vocational and technical education within a state
 Institution of higher education (including community colleges)
 A professional, industry, or employer organization or a product
manufacturer or developer
 A registered apprenticeship program
 A public regulatory agency (e.g., FAA aviation mechanic
certification)
 A program approved by DVA (Veterans’ Affairs) to offer
education and training to veterans and other eligible persons
under the Montgomery GI Bill
 Office of Job Corps
 Institutions of higher education which are formally controlled,
or formally sanctioned or chartered by the governing body of an
Indian tribe(s)

40
WIOA Youth Performance Measures

Education/Employment


Education/Employment



Up to one year after exit
If HS Dip/GED, must include Employment/Postsecondary
In-Program Skills Gain


Median earnings 2nd quarter after exit
Credential Rate


4th quarter after exit
Earnings


2nd quarter after exit
Achieving measurable skills gains
Employer Effectiveness

before PY16
41
What’s Eliminated

Literacy/Numeracy indicator for youth
 Although
utilized in development of Skills Gain
measure

Customer Satisfaction as statutory measure

State Incentive Funds
 But
Governor’s reserve may be used for local
incentives
42
Additional Provisions

State Targets
 Must
use statistical adjustment model—use now codified
(Sec. 116(b)(3)(A)(viii))
 Targets for first two years included in State Plans

Additional Information required in Annual
Reports
 Example:
Amount of funds spent on each type of service
 Data Validation now codified (Sec. 116(d)(5))
43
Additional Provisions

Sanctions
 State
Level
If a state fails performance, Secretaries shall provide TA
(used to say will provide TA upon request)
 If a state fails for 2nd consecutive year or fails to submit
their Annual Report, it can lead to a reduction in statewide
funds (stronger language)

 Local

Level
If failure continues for a 3rd consecutive year, the
Governor must take corrective action which shall include
development of a reorganization plan (and new local
board)
44
Eligibility and Data
Validation Documentation
45
Data Validation and Eligibility

States have the burden of identifying
required Program Eligibility documents,
or have given that responsibility to the
Local Area.

While not required, many States use the
Data Validation Source Documentation
Guide as the required documents listing
for Program Eligibility.
46
Source Documentation
Whether scanned, paper, or system
cross-match, the purpose of source
documentation is to have an auditable
trail that documents the participant,
services delivered and outcomes
received.
47
Common Source Documentation
Issues
Using documents that are not on ETA’s
list of Acceptable Source Documents;
 Checkbox in MIS;
 Date of Birth;
 Youth who Needs Additional Assistance;
 Failure to Accurately record ‘Dates’; and
 Poor Case Notes.

48
Questions?
49