Brief No: 290
July 2001
ISBN 1 84185 552 9
FEASIBILITY STUDY FOR THE LONG-TERM EVALUATION OF
MODERN APPRENTICESHIPS
Joan Payne (Policy Studies Institute), Rebecca Riley (National Institute of Economic and
Social Research) and Nick Coleman (BMRB International)
Background
Although Modern Apprenticeships (MA) have been in operation for several years, no formal evaluation of their
long-term impact has yet been carried out. This report investigates how feasible it is:
(a)
to produce robust quantified estimates of the additional (net) impact of MA on individual trainees' longterm job chances and earnings;
(b)
to meet other research aims specified by the DfES, including investigating the benefits of MA for
employers or for the wider economy and variations in the effectiveness of MA between different
sectors of the economy.
Main Conclusions
Estimating net impacts on individuals

To measure the net impact of MA on trainees, we must estimate what would have happened to them if they
had not taken part in the programme (the counterfactual question). Given theoretical, ethical and practical
constraints, the best methodology to achieve this probably uses matched comparison groups.

The choice of comparison group depends on which policy questions are considered the most important.
Finding suitable sampling frames for potential comparison groups presents problems.

No existing data source meets the needs of the evaluation study very well.

The proposed new longitudinal survey of young people would provide excellent data for the evaluation study,
but would not yield results for several years.

A survey of a past cohort of entrants to MA could provide more timely results and improve existing
knowledge about the subsequent careers of MA trainees, but would only partially address the counterfactual
question.
Other research aims

The feasibility of conducting an evaluation of the impact of MA on individual employers is limited and is not
recommended.

The policy’s implications for the wider economy are mostly long-term. An evaluation will have to be
conducted in the form of an appraisal, relying on model based scenario analysis using simulations informed by
empirical investigation.
 An evaluation of the variations in the effectiveness of MA between sectors can be conducted for relatively
broad sectoral disaggregations and is easily facilitated in the evaluation of the impact of MA on individuals.
Estimating net impacts on individuals
Strategies
To measure the net impact of MA on trainees,
we must estimate what would have happened to
them if they had not taken part in the
programme. The main problem is that trainees
differ from non-participants in ways that affect
the chances of a successful outcome. Several
different strategies can be considered to deal
with this:

A random allocation design is ruled out on
ethical and practical grounds.

The instrumental variable approach relies
on finding variation in the probability of
taking part that is unrelated to the
probability of a successful outcome other
than through its effect on participation.
However it is hard to identify a good
instrument for participation in MA, and
there is likely to be political opposition to
artificially
creating
an
instrument
through randomly allocated financial
incentives to train.

A matched area design is not possible
because MA has been running nationally
for several years.

The 'difference in differences' strategy
is unlikely to yield clear results in this
case because MA does not have a welldefined target group and its launch did
not mark a significant change from
previous policies.

A matched comparison group design would
compare the progress of trainees with
that of non-trainees who are similar to
participants on relevant observable
factors. This approach appears to be
viable, given a suitable data set.
Specific issues
Different comparison groups for MA trainees
largely correspond to different parameters of
interest, which answer different policy
questions. Full-time vocational education, other
government-supported training for young people
(OT), jobs (especially in the newer MA sectors),
and young people not in education, employment
or training all provide potentially interesting
comparisons. Ideally we would have more than
one comparison group, and in principal these
could be constructed from an existing nationally
representative survey of young people. However
if a new survey of a comparison group is needed,
finding suitable sampling frames presents
problems.
The age at which outcomes are best measured
depends on the choice of comparison group. It
is also influenced by the fact that many
entrants to MA are aged over 18, that entrants
to Advanced Modern Apprenticeships (AMA) are
on average older than entrants to Foundation
Modern Apprenticeships (FMA), and that in the
traditional craft sectors entrants tend to be
younger than in the newer sectors.
A retrospective research design is not as strong
as a prospective design, but produces results
more quickly. It could be based on an age
cohort, an MA leaving cohort or an MA entry
cohort, the last of these being the most useful.
Early entry cohorts under-represent the newer,
non-traditional apprenticeship sectors, and
1998/99 is the first entry cohort with
reasonable numbers in FMA. Given the extent
of non-completion, the study should estimate
the impact of MA on young people who leave
early.
Data
No existing data source meets the needs of the
evaluation study very well.

Cohort 9 of the Youth Cohort Study
(YCS) has small sample numbers for MA
trainees
and
would
restrict
the
evaluation to 16/17 year old entrants,
with employment outcomes measured at
age 19/20 (there being no information on
earnings at that age).

Cohort 1 of the Educational Maintenance
Allowance Pilots Study has richer and
better
quality
data,
but
similar
limitations of age and sample numbers
and a sample that is not nationally
representative.
Some comparative
analysis would be possible with Cohort 10
of YCS.

The administrative database for MA, the
Trainee Database System (TDS), holds
limited background data and outcomes
are collected by a postal survey with a
very low response rate.
There is a
theoretical possibility of linking the TDS
with
administrative
data
on
unemployment and with data from the
New Earnings Survey.

The Labour Force Survey identifies
those who have completed MA, but
otherwise has minimal information on MA
training and limited background variables.

The DfES's proposed new Longitudinal
Survey of Young People potentially offers
the data that are best suited to the
matched comparison group methodology.
However it will not yield suitable data
until 2006 at the very earliest, and
including MA entrants aged more than 16
in the evaluation would entail an even
longer wait for results.
A new survey of MA entrants
A new survey of MA entrants would not fully
meet the evaluation aims, but could improve data
on outcomes from MA and give results quite
quickly. It would allow us to compare outcomes
for AMA, FMA and OT, and for trainees who
complete MA and early leavers. By modelling key
questions on the LFS, it would also permit
limited comparisons with the employment rates
and earnings of groups identified in the LFS who
had not completed apprenticeships.
The proposed new survey would be based on a
full-year entry cohort (either 1998/99 or
1999/2000) drawn from the TDS, covering all
ages and sectors and interviewed face-to-face
at least three years after entry. An achieved
sample size of 4,000-5,000 would be needed for
separate analyses of AMA, FMA and OT, but if
separate analyses were not wanted, sample size
(and costs) could be reduced substantially. It
would be vital to contain sample attrition
through pre-survey investigations to assess its
likely level, updating address information and
good fieldwork procedure.
Given this, a
reasonable response rate could be achieved
because of the relevance of the survey to
respondents.
Other research aims
Evaluating the net impact of MA from the
perspective of employers
In principle, an evaluation of the benefits of MA
to employers, in terms of improvements in
productivity and profitability, could be
conducted in much the same manner as an
evaluation of the benefits of MA to individuals,
in terms of improvements in earnings. Given the
difficulties involved in obtaining data on firm
characteristics, productivity and profitability,
the feasibility of conducting an evaluation of
this kind is limited and is not recommended.
Data difficulties are further compounded by the
evaluation problem itself. A comparison group
of employers is difficult to obtain, given that
for a number of occupations MA is the
predominant mode of entry and given the
relatively high sectoral concentration of MA.
Evaluating the net impact of MA on the wider
economy
An assessment of the policy’s wider economy
effects will provide information on the benefits
of the policy to employers. It is also useful in
informing the evaluation of the policy’s impact
on participants’ earnings and long-term job
chances, as the wider implications of the policy
in turn affect both participants and the
comparison group.
The introduction of MA is likely to affect
productivity and growth, earnings, employment
and competitiveness, if it is successful in raising
participants’ productivity and participation.
These effects are mostly long-term. Generally,
one can attempt to quantify the effects of
policy either by econometric investigation or by
scenario analysis using calibrated model
simulation. In terms of the aggregate economy,
the effects of MA particularly in the early
years will be too small to detect by econometric
investigation alone. Instead an evaluation will
have to be conducted ex ante, in the form of an
appraisal, relying on model simulation informed
by empirical investigation. An appraisal should
illustrate sensitivity to the specification of the
participation decision and the skill-specific wage
determination process, the extent to which
there may be spillovers from MA training and
the sectoral distribution of productivity
changes in terms of openness to trade.
An important input into an ex ante evaluation of
the impact of MA on the wider economy is an
estimate of the individual productivity gain
associated with MA participation. This is best
obtained from an evaluation of the impact of
participation on individuals’ earnings, but may
also be obtained from the literature on the
returns to training.
Variations in the effectiveness
between different sectors
of
MA
An
evaluation
of
differences
in
the
effectiveness of MA across relatively broad
sectors such as manufacturing and services
could be facilitated in the evaluation of the
impact of MA on individuals and would be useful
to feed into an evaluation of the policy’s wider
economic effects. It would not be feasible to
estimate sectoral differences at a more
disaggregated level, due to the occupational and
industrial concentration of MA trainees.
Copies of the full report (RR290) - priced £4.95
- are available by writing to DfES Publications,
PO Box 5050, Sherwood Park, Annesley,
Nottingham NG15 0DJ.
Cheques should be made payable to “DfES Priced
Publications”.
Copies of this Research Brief (RB290) are
available free of charge from the above
address. Research Briefs and Research Reports
can
also
be
accessed
at
http://www.dfee.gov.uk/research/
Further information about this research can be
obtained from Sophie Gerrard, Room W606,
DfES, Moorfoot, Sheffield S1 4PQ.
Email: sophie.gerrard@dfes.gsi.gov.uk