Further information on soft outcomes for House of Lords ESF Inquiry
During the presentation of evidence WEFO officials agreed to provide their
Lordships with some further information on how WEFO measures ‘soft
outcomes.’
WEFO’s methodology for monitoring and evaluating projects has been built up
as part of its experience of managing the ESF Programmes in Wales.
Evidence from 2000 – 2006 Programmes
Evaluations from the 2000 – 2006 Programmes found that ESF projects were
assisting highly disadvantaged individuals at some distance from the labour
market. For these individuals, the achievement of hard outcomes such as
qualifications and employment was a long-term process requiring multiple and
often intense interventions. As a result these individuals progressed through
a number of intermediate milestones as part of their journey towards
employment. Some of these milestones were achievements of basic life-skills
such as increased confidence or motivation. It was important that ESF
projects assisting these individuals could measure and demonstrate the
progress made by individuals as a result of the support provided.
In 2003, WEFO and the Department for Work and Pensions commissioned
jointly a consultancy organisation called GHK to produce the first practical
guide to measuring soft outcomes and capturing the distance travelled by
beneficiaries from inactivity towards employment. The research underpinning
the production of this guidance identified that there is no single, ‘off-the-shelf’
approach to suit all projects because of the wide variety of ESF supported
activities, the resources / skill-sets available to them, and the different client
groups with whom they work.
To determine the extent to which soft outcomes measurement systems were
used across the 2000 – 2006 Programmes and to investigate sponsors’
experiences of measuring soft outcomes, WEFO revisited this issue through
fieldwork with ESF project sponsors as part of the Mid Term Evaluation
Updates (MTEUs) for Objective 1 and Objective 3 undertaken in 2005. The
MTEUs found that overall, project sponsors attached great importance to the
achievement of soft outcomes and several collected soft outcomes data at
beneficiary level. However, many project sponsors were not using the
information in any systematic way: the information enabled them to
demonstrate progress of individual beneficiaries but did not allow project
sponsors to aggregate data to demonstrate the project’s achievement as a
whole. The MTEU also found that the systems developed by projects varied
considerably in their sophistication and the level of input required by the
project sponsor.
To explore project sponsors’ experience of measuring soft outcomes further
and to understand the barriers to systematic measurement, WEFO undertook
qualitative research with a sample of Objective 3 project sponsors in 2006.
The research found that sponsors used a variety of soft outcomes indicators
and tools but were sceptical of the possibility of aggregating soft outcomes
data.
The key messages from the research on the 2000 – 2006 Programmes were
that soft outcomes are important but that project sponsors all had their own
systems commensurate to the client groups with which they were helping.
And it was apparent they perceived it as very difficult to aggregate these
systems into a coherent aggregate individual indicator.
Approach in 2007 – 2013 Programmes
Based on the 2000 - 2006 research WEFO has designed its monitoring
system for the 2007 - 2013 Programmes using a two-pronged approach.
Firstly, the needs of participants at some distance from the labour market and
what this requires in terms of support and delivery by projects is better
understood. WEFO has developed a progression indicator, ‘Participants
gaining other positive outcomes’, to enable projects to report intermediary
outcomes, such as completing courses, entering voluntary work, or attending
a job interview. This indicator represents soft outcomes but with a hard edge:
it allows progression towards hard outcomes to be captured but, because the
progression is demonstrated by tangible actions set out in the definition1 of
this indicator, it can be measured consistently and therefore aggregated in a
way that concepts such as ‘increased confidence’ cannot because they mean
different things to different people.
Secondly, as many projects have their own soft outcomes monitoring
systems, WEFO encourages projects to provide details from these systems in
their project evaluations. WEFO has increased its emphasis on project
evaluation in the 2007 – 2013 Programmes and has set up a team to support
project sponsors to meet their evaluation requirements. This gives sponsors
an opportunity to report their soft outcome achievements as well as the harder
outcomes to WEFO.
This approach to soft outcomes complements the wider ESF monitoring
system, which involves the collection of information at participant level.
Collecting this level of information gives WEFO a better understanding of the
journey undertaken by participants as they progress from unemployment /
inactivity to employment. The enhanced ESF monitoring system and project
evaluation arrangements gives WEFO a richer set of information on which
judgements of the effectiveness of the Programmes can be made.
1
The number of participants gaining intermediary outcomes as a result of participation in an
ESF-funded project. Intermediary outcomes, such as completing courses, entering voluntary
work, or attending a job interview, are those which are achieved as part of the journey to
achieving final outcomes, such as entering paid employment or gaining qualifications.