Measuring education outcomes in the
vocational education and training sector
Mark Cully
General Manager, Research
VET statistics are a patchwork quilt.
They serve the sector moderately well.
What we have:
o administrative unit record
collections of students, their
courses, and apprentices
and trainees
o sector specific surveys of
students and employers
AVETMISS provides the seaming
o complemented by ABS
household surveys on
participation and
attainment
Development of VET statistics by alphabet
soup – meets producer needs, not consumers
Development of VET statistics by alphabet
soup – meets producer needs, not consumers
Producers want to know:
Consumers want to know:
o how many students and
apprentices
o what courses are on offer
o what courses they are
doing
o what do people think
about the teaching they
get
o how many finish
o cost of delivery
o whether targets are on
track
o what will it cost
o what happens to people
who do these courses
VET sector subject to frequent reform and
revision to objectives.
Performance indicators have become more
outcome and goal oriented over time.
1994
Inputs
Actual vs target
student numbers
Outputs
Subject pass rates
Courses completed
Efficiency
Average cost per
student hour
Outcomes
Performance indicators have become more
outcome and goal oriented over time.
1994
2009
Inputs
Actual vs target
student numbers
Outputs
Subject pass rates
Courses completed
Literacy levels
Qualification
attainment
Efficiency
Average cost per
student hour
Skill to job match
Outcomes
Graduate
employment status
Job vacancies
2009 indicators on right track, but several
difficulties in collecting and interpreting.
Indicator
Collection issues
Interpreting issues
Working age popn at literacy
level 1-3
Sample survey
10 year cycle
Stock, not flow
Is level 2-3 literacy a real
concern?
2009 indicators on right track, but several
difficulties in collecting and interpreting.
Indicator
Collection issues
Interpreting issues
Working age popn at literacy
level 1-3
Sample survey
10 year cycle
Stock, not flow
Is level 2-3 literacy a real
concern?
20-64 year olds without postschool quals at Cert III +
Sample survey
Stock, not flow
Ignores Yr 12 & degrees
2009 indicators on right track, but several
difficulties in collecting and interpreting.
Indicator
Collection issues
Interpreting issues
Working age popn at literacy
level 1-3
Sample survey
10 year cycle
Stock, not flow
Is level 2-3 literacy a real
concern?
20-64 year olds without postschool quals at Cert III +
Sample survey
Stock, not flow
Ignores Yr 12 & degrees
Graduates employed after
training
Sample survey
Short-term
Ignores module
completors
2009 indicators on right track, but several
difficulties in collecting and interpreting.
Indicator
Collection issues
Interpreting issues
Working age popn at literacy
level 1-3
Sample survey
10 year cycle
Stock, not flow
Is level 2-3 literacy a real
concern?
20-64 year olds without postschool quals at Cert III +
Sample survey
Stock, not flow
Ignores Yr 12 & degrees
Graduates employed after
training
Sample survey
Short-term
Ignores module
completors
Graduates with improved
employment status
Sample survey
What is improvement?
Ignores module
completors
2009 indicators on right track, but several
difficulties in collecting and interpreting.
Indicator
Collection issues
Interpreting issues
Working age popn at literacy
level 1-3
Sample survey
10 year cycle
Stock, not flow
Is level 2-3 literacy a real
concern?
20-64 year olds without postschool quals at Cert III +
Sample survey
Stock, not flow
Ignores Yr 12 & degrees
Graduates employed after
training
Sample survey
Short-term
Ignores module
completors
Graduates with improved
employment status
Sample survey
What is improvement?
Ignores module
completors
Hard to fill vacancies
Sample survey
Collection on hold
Skill shortages a multifaceted issue
2009 indicators on right track, but several
difficulties in collecting and interpreting.
Indicator
Collection issues
Interpreting issues
Working age popn at literacy
level 1-3
Sample survey
10 year cycle
Stock, not flow
Is level 2-3 literacy a real
concern?
20-64 year olds without postschool quals at Cert III +
Sample survey
Stock, not flow
Ignores Yr 12 & degrees
Graduates employed after
training
Sample survey
Short-term
Ignores module
completors
Graduates with improved
employment status
Sample survey
What is improvement?
Ignores module
completors
Hard to fill vacancies
Sample survey
Collection on hold
Skill shortages a multifaceted issue
Skill-qualification mismatch
Sample survey
Boundary problems
What is right match?
Indicators require robust statistical collections
... and are not a substitute for analysis
What we need to know,
but currently don’t:
o finely detailed outcomes
for sub-populations of
interest (e.g. Indigenous)
o fee-for-service delivery of
private providers (supply)
o training expenditure by
employers and
individuals (demand)
o student pathways in and
out of VET
What we need to understand,
but indicators won’t tell us:
o determinants of student
course/qualification choice
o employers demand for skills
vs qualifications
o relative returns to
education and training
o determinants of course and
apprentice completion
rates
o what is quality training?
Conclusions
VET sector is moderately well served by statistics.
Conclusions
VET sector is moderately well served by statistics.
The sector could do a much better job in using
statistics to inform consumer choice.
Conclusions
VET sector is moderately well served by statistics.
The sector could do a much better job in using
statistics to inform consumer choice.
Objectives, and associated indicators, have not been
stable over time. Indicators have progressively
become more outcome oriented.
Conclusions
VET sector is moderately well served by statistics.
The sector could do a much better job in using
statistics to inform consumer choice.
Objectives, and associated indicators, have not been
stable over time. Indicators have progressively
become more outcome oriented.
New indicators focus too much on stocks, rather than
flows, and are not sufficiently robust to measure
outcomes for disadvantaged groups.
Conclusions
VET sector is moderately well served by statistics.
The sector could do a much better job in using
statistics to inform consumer choice.
Objectives, and associated indicators, have not been
stable over time. Indicators have progressively
become more outcome oriented.
New indicators focus too much on stocks, rather than
flows, and are not sufficiently robust to measure
outcomes for disadvantaged groups.
Indicators aid understanding. They need to be
complemented by careful statistical analysis.
Questions?