Joint Report on the Evaluation of the
Socrates II, Leonardo da Vinci and
eLearning programmes
Executive Summary
Joint Report on the Evaluation of the
Socrates II, Leonardo da Vinci and
eLearning programmes
Executive Summary
January 2008
ECOTEC
u Priestley House
12-26 Albert Street
Birmingham
B4 7UD
United Kingdom
T
F
+44 (0)121 616 3600
+44 (0)121 616 3699
www.ecotec.com
ECOTEC
Contents
PAGE
1.0
Introduction............................................................................................... 1
1.1
1.2
1.3
Background ................................................................................................................1
Evaluation framework ................................................................................................2
Methodology ...............................................................................................................3
2.0
Impacts ...................................................................................................... 7
2.1
2.2
2.3
2.4
Impact in the schools sector .....................................................................................7
Impact in the VET sector..........................................................................................10
Impact in the higher education sector....................................................................13
Impact in the adult education sector ......................................................................16
3.0
Efficiency of the three programmes..................................................... 20
3.1
3.2
3.3
3.4
Efficiency of Socrates II ...........................................................................................20
Efficiency of Leonardo da Vinci II ...........................................................................20
Efficiency of eLearning ............................................................................................21
Efficiency of the Committee system .......................................................................21
4.0
The common impacts of the programmes and the contribution to
the Lisbon goals..................................................................................... 23
4.1
4.2
4.3
4.4
4.5
4.6
4.7
Introduction ..............................................................................................................23
Creation of a European education area..................................................................24
Improvements in teaching practice and approaches to learning and
management .............................................................................................................25
Impacts on policy and practice at EU and Member State levels ..........................26
Increased proficiency in EU languages..................................................................26
Socio-economic impacts .........................................................................................26
The contribution of the programmes to Lisbon-related objectives .....................27
5.0
Strategic Conclusions ........................................................................... 29
5.1
5.2
5.3
5.4
5.5
The relevance of the programmes ..........................................................................29
The effectiveness of the programmes: objectives, impacts and sustainability..29
Balancing grass-roots needs and top-down direction..........................................30
Modes of intervention ..............................................................................................30
Decentralisation and the role of Member States ...................................................31
ECOTEC
5.6
Overall contribution to the Lisbon agenda ............................................................31
6.0
Recommendations ................................................................................. 32
6.1
6.1.1
6.1.2
6.1.3
6.1.4
6.1.5
6.2
6.2.1
6.2.2
6.2.3
6.2.4
6.2.5
6.2.6
Strategic recommendations ....................................................................................32
Building on the achievements of 2000-06...................................................................32
Achieving the right balance between grass-roots needs and top-down direction .......32
Realising the potential for synergy between actions...................................................33
Improving synergies within countries..........................................................................33
Giving greater priority to social disadvantage .............................................................33
Operational recommendations................................................................................34
Developing indicators and targets ..............................................................................34
Improving monitoring ..................................................................................................34
Improving efficiency....................................................................................................34
Improving dissemination and sustainability ................................................................35
Structuring cooperation ..............................................................................................35
Improving how equality and accessibility issues are handled .....................................36
ECOTEC
1.0
Introduction
1.1
Background
In December 2006 ECOTEC Research & Consulting Ltd was commissioned to undertake
the Joint Final Evaluation of the Socrates II, eLearning and Leonardo da Vinci II (LdV)
programmes 1 over the period 2000-2006. Collectively, these programmes accounted for
in excess of 3,800 million euro of public spending2 and constituted a major way in which
the European Commission had the potential to make a contribution to the achievement of
the Lisbon goals in education and training3. This document is the Executive Summary for
the Final Joint Evaluation Report.
The Evaluation Terms of Reference set out the following questions to be answered for all
three programmes:
• Relevance: The extent to which an intervention’s objectives are pertinent to needs ,
problems and issues to be addressed
• Coherence and complementarity: The extent to which the intervention logic is noncontradictory and the intervention does not contradict other interventions with similar
objectives
• Effectiveness: The extent to which objectives set are achieved
• Efficiency: The extent to which the desired effects are achieved at a reasonable cost
• Sustainability: The extent to which positive effects are likely to last after an intervention
has terminated.
This was commissioned as a joint evaluation in order to be able to assess (a) individual
and collective impacts, and in addition to that, (b) combined effects and (c) those
combined effects in relation to Lisbon. This report is the main joint report of the evaluation.
Separate programme-level reports on Socrates II, Leonardo da Vinci II and eLearning are
available in annexed volumes with individual executive summaries. In order to avoid any
potential conflict of interest stemming from the involvement of ECOTEC in the
management of the Leonardo da Vinci II programme in the UK, the implementation of the
1
The legal basis for the evaluation was contained in the following Decisions of the European Parliament: Socrates – art.
14.2 Decision 253/200/EC; Leonardo – art. 13.2 Decision 1999/382/EC; eLearning – art. 12.2 Decision 2318/2003/EC.
2
Socrates: 1,850 million €,; LdV: 1,982 million €; eLearning: 44 million €.
3
See for example, Report from the Education Council to the European Council on The concrete future objectives of
education and training systems. http://ec.europa.eu/education/policies/2010/doc/rep_fut_obj_en.pdf
1
ECOTEC
Leonardo evaluation was sub-contracted to ECORYS NL. A series of measures and
quality checks were imposed to maintain this situation of independence.
1.2
Evaluation framework
The Terms of Reference for the evaluation required us to evaluate the combined effects of
the programmes and also their combined effects in relation to the Lisbon goals. This
required us to look both 'top-down' from policy and 'bottom-up' from the programmes.
The bottom-up component of this process was relatively conventional and comprised the
reconstruction of intervention logics for each programme/action. The top-down element
was more challenging as it required us to analyse how the Lisbon agenda had been
translated into the field of education and training, and how it linked to the programmes;
and then to construct an appropriate approach to evaluating the collective impact of the
programmes in relation to Lisbon. This process concluded that it would be inappropriate to
reconstruct an intervention logic in relation to Lisbon and that the evaluation should assess
the collective contribution of the programmes to the Lisbon goals through the relevant
body of education and training policy (notably the Copenhagen process in relation to VET
and the Bologna process in HE).
Further, it was determined that the evaluation should carefully extract coherent sets of
outputs, results and impacts that were common across the programmes, that did justice to
the wide range of activities carried out under them and that would also reflect the general
ways in which they might have contributed to Lisbon; the resulting categorisation would
then be used to structure the evaluation. Outputs were categorised into three broad types:
mobility activities; networks and partnerships; and methods/tools and frameworks.
Impacts were of two broad types: practice and policy and socio-economic. The
programmes were designed to deliver practice-related (and to a lesser degree policyrelated) impacts more than socio-economic ones. It was not the prime purpose of most of
the streams of funding to have a major direct socio-economic impact but rather to
stimulate change within education and training systems, which might ultimately feed
through into socio-economic benefits.
The evaluation also took into account the various structures and level of EU competence
and influence within the various educational sectors that the programmes targeted
(namely, schools, VET, higher education and adult education) when considering effects.
For example in the highly developed higher education sector there is a major role
identified regarding the Lisbon process not just in education but in providing the foundation
for European science, technology and innovation. This sector has reached the earliest
agreement of all the sectors at European level on common priorities (via the Bologna
2
ECOTEC
process), with comparatively strong progress towards the achievement of the infrastructure
needed to form a genuine European area of higher education with easy cross-border
movement for staff and students. In contrast, in the schools education sector where the
EU’s competence is more limited and the principle of subsidiarity is strongly maintained
with no equivalent of the Bologna and Copenhagen processes. European policy
interventions in schools thus target comparatively small sums of money at individual
institution and teacher levels. The adult learning sector too is relatively undeveloped,
even embryonic, across many member states with varying participation rates and a very
low base in terms of established cross-border co-operation and European dialogue in
2000. Similarly the VET sector, which is extremely varied across Member States in terms
of sector structure and has seen some agreement on priorities through the Copenhagen
process which is still in the early stages of implementation, with need for wider and more
rapid progress recognised.
1.3
Methodology
The evaluation methodology was designed to address the issues at the heart of the
evaluation framework given above, alongside the more ‘conventional’ eval uation questions
as set out by DG Budget1. The broad evaluation questions were translated into a series of
evaluation sub-questions and indicators (see Table 3.1, Chapter 3 of the Joint report).
The initial step was to reconstruct the logic for intervention for the programmes using
documentation and interviews with Commission staff. The intervention logics were then
translated into a series of meta-level impact indicators against which the programmes
were evaluated.
This was followed by a programme of primary and secondary research. This included
collation and analysis of programme documentation and financial data involving analysing
key documents about each programme and Action such as: policy and background
documents relating to establishing the programmes, calls for proposals, applications, the
role of committees, assessment, selection, monitoring, project details, previous
evaluations and dissemination. In particular, one purpose of the review was to review the
management and implementation of the three programmes and explore their strengths,
weaknesses, problems and solutions.
1
Evaluating EU Activities: A Practical Guide for the Commission Services; DG Budget, November 2003.
3
ECOTEC
We also undertook a series of web surveys targeted at successful applicant organisations
under each of the three programmes1. The surveys were publicised by email to lists of
successful applicant projects provided by the Commission. In addition National Agencies 2
provided support in promoting the survey in their countries.
The surveys received the following response rates. In Socrates II, invitation emails were
sent to 13,464 Socrates II participants from centralised actions. National Agencies were
asked to forward the invitation to participants from decentralised actions. A total of 4,499
responses were received to the surveys, this represents an overall response rate of 33%.
The survey focused on the main actions (Comenius, Erasmus, Grundtvig, Lingua,
Minerva) and also covered the Arion sub-action and mobility in Grundtvig (Grundtvig 3)
and Comenius (Comenius 2.2). In Leonardo over 46,500 email invitations were sent out,
and 7,149 responses were received reflecting a response rate of approximately 15%. In
eLearning emails were sent to all 62 funded projects and a total of 29 responses were
received, representing a response rate of 47%. A further survey of participants in Action 3
(eTwinning) also received 1,551 responses from schools across Europe. Of those, the
majority, 1,017, had completed a project through eTwinning.
We also conducted qualitative in-depth interviews with project co-ordinators, project
partners, Erasmus co-ordinators, National Agencies, Commission and Executive Agency
staff, Ministries, and other stakeholders3. In total we interviewed 128 individuals in
Socrates II, 186 individuals for LdV and 86 individuals for eLearning. We developed a
series of project case studies; these are presented in the individual programme reports.
We also looked at the impact of the programmes at national level in five countries to aid
understanding of how Member States contribute to delivering on Lisbon through the
development and implementation of lifelong learning strategies and the reform of
education and training systems. A sample of projects from the three programmes was also
assessed (45 Socrates, 58 Leonardo and 11 eLearning projects) through an independent
project scrutiny exercise using a template analysis framework primarily to enable us to
assess the level of bias in survey responses, although it also generated other useful
information which could be used in the evaluation. Finally, the National Reports prepared
on Socrates and LdV were synthesised and the results of this are presented under
separate cover.
The key findings presented below are based on the full range of data sources including
programme and financial data, the responses from the surveys, and in-depth interviews.
1
In addition a separate survey was undertaken among schools participating in Action 3 of the eLearning programme,
eTwinning.
2
And National Support Services in the case of the eLearning Action 3 survey for eTwinning.
3
Throughout this report 'stakeholders' are taken to refer to ‘people with a good knowledge of the education field and the three
programmes. They included Commission staff, Executive Agency staff, members of Working Groups, experts and evaluators.
4
ECOTEC
5
ECOTEC
Selected Headline Outputs
Comenius
Over 74,000 schools in partnerships
Over 5 million pupils engaged
eTwinning
Nearly 24,000 schools involved
Over 310,000 pupils took part
Leonardo da Vinci II
21,000 projects
367,000 mobility placements for VET students
Erasmus1
Over 783,000 university students
Over 100,000 university teachers
Over 2,500 institutions held an Erasmus University Charter
Grundtvig
Over 1,900 adult education institutions participating in partnerships by 2006
1
Figures to the end of 2005
6
ECOTEC
Selected Headline Outputs
455 multilateral projects, networks and thematic seminars
2.0
Impacts
2.1
Impact in the schools sector
The key programmes potentially impacting on the schools sector were the Comenius
action of Socrates II and eTwinning, Action 3 of the eLearning programme. A crucial factor
underpinning all activity in the sector is that the Commission is not responsible for school
education, this being a Member State responsibility. Schools policy during the period
2000-6 was an area where the EU had limited competence, primarily being there to
support Member States to develop and modernise their own systems. This therefore
limited the impact interventions were designed to achieve, particularly at the policy level.
Comenius aimed to involve a target of 10% of schools in activities supporting: enhancing
the quality and reinforcing the European dimension of school education; promoting the
learning of languages and promoting intercultural awareness through €599.5 million (27%
of the overall Socrates budget) over 2000-06. These objectives were operationalised
through three activities: school partnership projects (C1), training of school staff via
transnational co-operation projects (C2.1) and individual teacher mobility (C2.2) and
networks of teacher training institutions and other stakeholders (C3). In Comenius, over
74,000 schools participated in partnerships and over 5 million pupils were engaged1.
The intervention logic (IL) analysis for the Action highlighted that a key focus of
decentralised actions was the co-operation process itself – the carrying out of a project
with a number of partners from other European countries. In this case, tangible outputs
such as tools/methods were seen as additional, 'induced' outputs. Centrally managed
cooperation and networking actions in contrast were designed to develop new approaches
and share good practice.
1
European Commission (2007) Comenius success Stories. Europe Creates Opportunities. DG Education and Culture,
Belgium, p.4.
7
ECOTEC
In addition the IL analysis suggested Comenius partnerships probably faced barriers in
transferring lessons learned to other practitioners and to national policy makers, given the
relative inflexibility of national education policies and the EU’s limited competence in this
area.
eTwinning was launched in 2005 as the main Action (Action 3) of the eLearning
programme (2004-6), comprising 45% of the programme's €44 million budget. It focussed
solely on activity in schools: i.e. twinning schools through the internet. eTwinning
contributed to eLearning's global objective through encouraging the effective use of elearning and to its specific objectives through strengthening and developing European
level networking among schools, teachers and pupils, thereby enhancing the European
dimension in school education. The scheme differed from the European Commission's
other programmes/actions in that it did not give grants direct to participants. Rather, it
provided free access to an ICT infrastructure, the eTwinning Portal (at central and national
levels), which facilitated the establishment of partnerships between schools across
Europe; schools could then choose what type of project they wanted to undertake. It thus
constituted a wholly unique mode of intervention in the school sector. eTwinning involved
nearly 24,000 schools across Europe, with an estimated 310,600 pupils taking part 1. Our
analysis of the IL indicated that eTwinning’s impacts were most likely to be qualitative
rather than quantitative and when considering impacts, the relatively small size of the eLearning budget should be used to contextualise impacts.
Relevance (To what extent were the intervention's objectives pertinent to needs, problems
and issues to be addressed?)
Comenius and eTwinning were in line with the identified needs in terms of enhancing the
European dimension and improving skills and competences in schools, particularly around
ICT and languages. There was no formal overall rationale for intervention in the 'schools
sector' through Community action – meaning that the various activities in schools were not
conceived or designed to fit together – and their contribution to the Education 2010 work
programmes (and Lisbon) was never fully considered, despite schools having the potential
to contribute to both2. Programmes for schools were designed to impact from the bottomupwards through impact on individual teachers/pupils/schools; not to bring about systemic
changes. Regarding the Lisbon agenda, the programmes' objectives did not directly reflect
this; and reducing early school leaving was an aspect of the Lisbon priorities which was
not prioritised in the programme objectives and therefore did not translate into impact.
1
European Commission (2007) Comenius success Stories. Europe Creates Opportunities. DG Education and Culture,
Belgium, p.4.
2
OJC 142, 14.6.2002, Detailed work programme on the follow-up of the objectives of education and training systems in
Europe, p.1-22.
8
ECOTEC
Given the EU’s competence in schools, the scope for wider policy influence of the
programmes was also limited to sharing of good practices and facilitating networking.
The programmes focussed on clear activities and had a clear sense of purpose (they were
internally coherent), and activity between eTwinning and Comenius was clearly laid out to
avoid duplication (they were complementary). The evaluators concluded that activity that
would not otherwise have taken place without EU intervention had been funded
(additionality).
Effectiveness (To what extent were the programmes successful in attaining the objectives
set and achieving the intended results?)
Very broad objectives were set for programmes in the schools sector and as these were
not translated into verifiable outputs/results and impacts this made the task of evaluation
difficult. As a result, the evaluators imputed a set of impacts against which to evaluate the
programmes to identify impacts in key areas. One target that was identified in the Socrates
Decision was to attain a participation rate of around 10% of schools1. This was difficult to
measure since a formal count of the number of schools in Europe was not available 2, but
qualitative feedback from stakeholders confirmed that the target was not met. The
evaluation evidence demonstrated that eTwinning in particular appeared to have been
very effective in achieving its objective of supporting European co-operation, by creating a
network of interest and enthusiasm for co-operation among schools, which led to positive
impacts, through the scale of activity supported through its innovative approach. Both
programmes produced positive outputs such as tools, resources, DVDs and websites.
The key results and impacts in schools concerned: enhancing the schools’ European
dimension and outlook, and for the individual: improvements in skills and experience of
languages and ICT. An important area where the programmes impacted was in creating a
clearer sense of European identity and through schools benefiting from co-operating with
other schools. These areas clearly aligned with the needs and intended impacts the
programmes were designed to address. Another important area of impact was on teacher
training, for example teachers trained under Comenius’s decentralised mobility action
reported having improved skills and competences to utilise in their teaching when they
returned. Impact on curricula was more limited (not least because this was not a major
intended effect) and tended to be localised; whole school impacts were comparatively rare
(again, these were not intended). Policy impacts were also more limited; teachers could
make small curriculum improvements locally, but as predicted, wider national changes to
curricula were beyond the remit of the programmes and the EU’s competence. There were
1
2
Decision No. 253/2000/EC of the European Parliament and the Council of 24 January 2000
Not available in Eurostat Education Statistics, 2007.
9
ECOTEC
also some small scale impacts on language learning, with teachers reporting that pupils
had improved language skills and were more open to language learning in future. Little
impact was seen on less widely used/less taught languages as most projects were
undertaken in one of the main EU languages. Overall though the programmes funded
activity which led to impact in areas which met needs around skills levels and improving
European co-operation and thus, in evaluation terms, were of good utility. Both Comenius
and eTwinning provided good value for money with regard to the scale and extent of
activity being funded (i.e. the small scale of many Comenius funding grants provided for
teacher mobility and the fact that eTwinning schools did not receive direct grants at all). In
both programmes, the added value of significant additional person-hours spent on projects
by teachers has never been quantified. The innovative funding model of intervention used
in eTwinning deserves recognition as an excellent value for money method of supporting
transnational activity in the school sector. The evaluator’s opinion is that the value of both
programmes could be further improved through more effective support for dissemination,
sustainability and mainstreaming.
2.2
Impact in the VET sector
The evaluation looked at the impact of the programmes on the vocational education and
training (VET) sector primarily through the impact of the Leonardo da Vinci II programme.
Leonardo da Vinci II was a key part of the Commission's intervention in the field of VET.
The programme's objectives concerned improving the skills and competencies of people,
especially young people, in initial VET at all levels, to facilitate their integration and
reintegration into the labour market; improving the quality of, and access to, continuing
VET; and promoting the contribution of vocational training to the processes of innovation,
competitiveness and entrepreneurship. The key areas of activity (measures) were L1
mobility (including transnational placements and exchanges, and study visits); L2
transnational pilot projects including thematic actions; L3 projects concerning language
competencies; L4 transnational networks; and support for reference materials, joint actions
and accompanying measures. The total available budget for the programme increased
from 2000 to 2006 from €171 million to €251 million, an increase of 47%. Of that, funding
provided to mobility (L1) formed at least 39%; transnational pilot projects not less than
36% and funds for thematic action projects not more than 5 % (L2). In addition, language
competence projects were not less than 5% of the annual budget (L3). During the course
of the programme, the focus shifted from a balance between mobility and pilot projects to
an emphasis on the former. As with other programmes indicators of impact were not
specified at the outset so they were imputed from background documentation and
interviews with stakeholders. In terms of outputs, placements of young people undergoing
10
ECOTEC
initial vocational training were key (L1). In the period 2000-2006 approximately 21,000
projects were established, the vast majority being mobility projects (19,307) and the total
amount of mobility placements amounted to approximately 367,000. 70% of pilot projects
(L2) focussed on creating training courses, particularly in ICT. A further 30% created
vocational guidance products. The majority of language projects (L3) also produced
training courses (57%), whereas transnational networks focussed on exchange of
experience and dissemination activities.
Relevance (To what extent were the intervention's objectives pertinent to needs, problems
and issues to be addressed?)
In VET there was an identified need, and strong momentum, for example via the
Copenhagen Declaration, for strengthening the European dimension, improving
transparency, information and guidance systems, recognition of competences and
qualifications and promoting quality assurance. It is clear that looking at the intended and
actual outputs, results and impacts that these needs have been addressed in various
ways, something that would not have happened without the Programme (in that sense the
programme was promoting additional activity to what would have occurred anyway). There
was also a clear alignment between the design of the Leonardo programme and the
Lisbon guidelines through aspects such as improving employability, prioritising lifelong
learning and developing the knowledge economy. Similarly, the Leonardo programme
objectives did match the Integrated Guidelines1 during the period 2000-2006, with their
focus on: comprehensive and coherent strategies for Lifelong Learning, helping people
acquire and update the skills needed to cope with economic and social changes
throughout the entire life cycle; improving the quality and efficiency of education and
training systems; and equipping all individuals with the skills required for a modern
workforce in a knowledge-based society, to permit their career development and to reduce
skills mismatch and bottlenecks in the labour market. The budget increase awarded to
Leonardo over the course of the programme indicates the increased political priority
accorded to human capital development at EU-level.
The programme has proved to be coherent in terms of contributing towards the creation of
a European education area through the promotion of lifelong learning and continued
Community level cooperation between actors in the field of VET, and successive calls for
proposals have reinforced this. Furthermore, half of the survey respondents (46%) said
their project had helped to bring about convergence between Member States in policy and
practice in the field of VET.
1
Commission of the European Communities (2005) Integrated Guidelines for Jobs and Growth (2005-2008) Brussels
12.4.2005, COM (2005) 141 final, 2005/0057 (CNS)
11
ECOTEC
Effectiveness (To what extent were the programmes successful in attaining the objectives
set and achieving the intended results?)
Overall the evaluation found that the programme and its underlying measures contributed
to the development of a European area of co-operation in the field of education and
training and in this respect contributed to the promotion of a Europe of knowledge. The
specific objectives of the programme were felt to be broad and flexible enough to address
the needs in VET, taking into account local, regional and national diversity. The main
benefit for the participating organisations was the development of a greater European
outlook. In particular mobility and language projects contributed to this, whereas the other
measures mainly produced networks among institutions from different European countries.
The benefits for the participating VET students and the young workers consisted mainly of
improved knowledge, skills and competencies and an improved quality of VET, especially
through their participation in mobility projects.
The strongest area of impact for the programme was socio-economic, in relation to young
people in VET. In particular, the projects contributed to improvements in: the knowledge,
skills and competencies of young people in initial VET; capacities for lifelong acquisition of
skills and competencies; and VET quality. In particular, the acquisition of foreign language
skills was an important socio-economic benefit for young people participating in the
programme. Strong socio-economic impacts were reported in relation to the employability
and adaptability (to labour market developments) of participants in mobility, although most
of these impacts can only be demonstrated in the long run when the participants have
been active on the labour market.
The programme also impacted considerably on the curriculum in participating institutions,
especially via activities in mobility and pilot projects. Leonardo substantially increased the
quality of learning and teaching in the VET sector, for instance by improving and
introducing new teaching methods and curricula. Also in many respects it contributed to
the opening up of VET systems by organising transnational cooperation and placements,
resulting in greater transparency of curricula and qualifications.
Additional to the organisational/institutional impacts, Leonardo also impacted on VET
policy for example by developing standards, methods and tools that were integrated into
national or regional policy and practice. Policy impacts appeared strongest at local and
regional levels which was logical due to the limited geographical scope of most projects.
The impact of the Leonardo programme on increased transnational cooperation in the fi eld
of VET should not be underestimated, since without the programme little cooperation
would have taken place. Overall, the programme contributed to the development of a
European VET area which otherwise would not have developed at all or at best at a much
12
ECOTEC
slower pace. In addition, the most significant contribution of the programme towards the
Lisbon goals was around delivering a better skilled labour force and in improving the
labour market opportunities of young people.
In terms of sustainability, the programme was strong: the majority (73%) of survey
respondents1 stated that (all or some of) the project activities had continued after the end
of funding. In particular, the respondents of pilot projects indicated sustainability of project
activities (84%) while 91% of respondents indicated that their project outputs were still in
use in their own organisation and 64% in partner organisations. Furthermore, since the
majority of the Leonardo projects has achieved their objectives, and since only a minority
of the respondents stated that larger budgets would have contributed to higher quality
outputs/results, and since the evidence pointed to a considerable impact on a large range
of aspects of VET systems and its participants, the programme has probably produced
value for money.
2.3
Impact in the higher education sector
Of the various actions of the Socrates II programme, Erasmus was the only one that was
intended specifically for higher education at university and post-university level2. It
supported the mobility of EU students and teaching staff within the EU, as well as cooperation between European universities in the areas of teaching and research. However,
higher education institutions were also actively involved in other actions/programmes 3,
therefore when appropriate we draw on examples from those in the chapter.
Erasmus supported the mobility of EU students and teaching staff within the EU, as well
as co-operation between European universities in the areas of teaching and research . Its
objectives were to enhance the European dimension of higher education, encourage transnational co-operation, promote mobility for students and staff and improve transparency
and recognition of qualifications across Europe. These objectives were implemented via
three main sub-actions: European Inter-university co-operation through multi-lateral
projects (centralised), mobility opportunities for staff and students (which were managed in
a decentralised manner by NAs) and the centralised set of thematic networks which
involved HE and wider actors. The Erasmus budget for 2000-2006 amounted to 950
1
Of finished projects.
We took the view that the focus of Erasmus is on non-vocational education.
3
For instance, in eLearning, the virtual campus action aimed to add a virtual dimension to European cooperation in
higher education by encouraging the development of new organisational models for European universities (virtual
campuses) and for European exchange and sharing schemes (virtual mobility). This action line aimed at building on
existing co-operation frameworks created by Erasmus, giving them an e-learning component.
2
13
ECOTEC
million euros (of which approximately 750 million euros was for student grants)1. Student
mobility lay at the heart of the action, and during the implementation period there was
further impetus to increase the numbers of participants and to integrate such mobility into
a wider framework of co-operation activities that aimed at developing a "European
Dimension" within the entire range of a university's academic programme.
According to EC figures2, Erasmus mobilised 783,104 students and 109,399 teachers;
3,192 students took part in Intensive Language Courses and 2,520 institutions held an
Erasmus University Charter. The Action also implemented 479 curriculum projects,
developed 1,271 intensive programmes and supported 166 networks.
Relevance (To what extent were the intervention's objectives pertinent to needs, problems
and issues to be addressed?)
The programme fitted well the identified needs in higher education and it stimulated cooperation in the higher education sector across Europe. Socrates, through the Erasmus
action in particular, offered a wide range of measures designed to support the 'European
dimension' of higher education, by encouraging co-operation between higher education
institutions, promoting the mobility and exchange of their teaching staff and students, and
improving transparency and academic recognition of studies and qualifications.
Furthermore, by facilitating mobility, the programme potentially contributed to the
achievement of the aims of the Bologna process and the Europe Education and Training
2010 Strategy and, through them, to Lisbon. There is an underlying assumption that given
the number of people benefiting from Erasmus the programme will have had an impact on
the skills and knowledge of people arriving on the labour market and their mobility and,
consequently, a potential impact on the performance of the European economy in general
and the objectives of the Lisbon strategy. Evidence collated in previous evaluations that
focused on the professional value of Erasmus mobility indicated that a high percentage of
Erasmus students saw their first job search as having benefited from participation, that
there was far higher mobility in jobs and a stronger international dimension to the work
found by Erasmus students as result of their participation.
The programme demonstrated strong internal and external fit (coherence). It had clear
linkages between the global objectives (Socrates II) and the specific objectives ( Erasmus)
and the sub-actions supported. There was also a strong external fit with the Bologna
process (most of the action lines were directly relevant to the action) and with Education
and Training 2010 which emphasises the need to increase the quality and effectiveness of
1
Additional funds were provided in each country by public authorities, by the universities themselves and by other
organisations.
2
Data only available for the 2000-2005 period
14
ECOTEC
education in Europe and to open up the systems to the wider world. The focus of the
programme on the European dimension could be seen as complementary to much
national policy since internationalisation of the higher education sector is prioritised in
many Member States. Furthermore the evidence indicated that activity supported was
additional and most would not have taken place without EU support. Where activities
would have occurred without Socrates II funding, qualitative feedback from Erasmus
university co-ordinators and project co-ordinators gave an insight into the important
dimensions which additional EC funding had brought.
Overall it is important that higher education is not subject to a ‘common European policy’
because the authority over (and competence for) the content and the organisation of
studies remains at national level. However the autonomy and flexibility of higher education
institutions played an important role in facilitating the adoption of a European approach to
tackling the common problems faced by the sector. That 90% of European universities
participated in Erasmus1shows how instrumental it was in encouraging higher education
institutions to take part in exchange programmes and exchanging information and best
practice, showing the potential to influence the introduction of measures across the board.
Effectiveness (To what extent were the programmes successful in attaining the objectives
set and achieving the intended results?)
Overall it was concluded that Erasmus activities fitted well with the objectives of the Action,
which in turn fitted the perceived needs in the sector.
Increasing the capacity for mobility was the largest positive impact of Erasmus (89%
agreed with the statement that their activity had achieved this). The short-term impacts on
participants in terms of their personal and professional development were strong. Key
impacts were an increased capacity for mobility in the future (within and outside Europe), a
more open attitude and a clearer and better informed perspective for their subsequent
studies or professional life; greater understanding of Europe and ‘belonging to a European
family’, improved knowledge of EU language(s) and better contacts with European
colleagues. A further area of significant impact was in the improvement of professional
skills, and knowledge of other education systems and practices. As an ‘indirect’ effect of
the mobility period, participants became more ‘employable’ (due to international
experience and improved foreign languages skills). Erasmus's level of resourcing meant
that it was able to support a large number of individuals. However while it had strong
impacts on individuals, the 'real' effect on the entire student population was small at only
4%.
1
European Commission (2007), Erasmus Success Stories. DG Education and Culture, p. 3.
15
ECOTEC
Evidence of longer-term impacts was more difficult to identify. But there was some clear
evidence of impact on policy and practice, particularly in terms of the response to the
challenges of internationalisation in the sector and providing support to the intergovernmentally steered Bologna process. The Erasmus University Charter provided the
general framework for all European cooperation activities undertaken by higher education
institutions and helped to internationalise and to increase the European awareness and
cooperation of institutions (thus potentially impacting on all students and not just those
who took mobility periods). And overall Erasmus has acted as a driver for change in
European higher education by providing support to the objectives set out in the Bologna
process. The Bologna Process has led to a convergence in course structures, while
existing programmes including Erasmus have sought to provide the tools (e.g. the
European Credit Transfer System), stimulate the development of joint degrees and
develop collaborative teaching approaches. Sustainability was also strong in Erasmus with
50% of projects stating that outputs from the centralised sub-actions were still in use in
their own organisations. 72% of respondents also stated that all/most of the activities were
continuing and/or would continue beyond the funding period.
Erasmus demonstrated good utility and good value for money. The majority of Erasmus
funding went into the mobility aspect and data provided shows that the action has been
clearly cost effective in this respect, given the high number of exchanges and despite the
relatively low1 financial support. Outputs were appropriate and their generation has clearly
benefited participant organisations and individuals.
2.4
Impact in the adult education sector
The Grundtvig action within the Socrates programme was the key policy instrument
designed to target this sector. It aimed to raise demand for, and participation in lifelong
learning activities, improve basic education competences, enhance information and
support services and improve teacher training and recognition of competences. The
funding of €130 million was modest compared to other programmes/Actions, representing
some 7% of the entire Socrates II budget for 2000-2006. The objectives were
implemented through four measures: G1 European Co-operation Projects and Grundtvig
Training Courses (46% of the total budget), G2 Learning Partnerships: Exchange of
experience/practice/methods (40% total budget), G3 Individual Training Grants for Adult
Education Staff (6% total budget) and G4 Networks and Thematic Seminars (8% total
budget). G1 and G4 were centralised, the others de-centralised (managed by National
1
The Erasmus grant, on average, covered the additional expense of studying abroad. As a recent survey on the
economic background of Erasmus students confirmed, about half of the students who participated in the survey reported
the Erasmus grant as being insufficient.
16
ECOTEC
Agencies). The evaluation took account of the less advanced nature of the adult education
sector: more dispersed, less institutionalised, less well funded and with a shorter history of
support from EU programmes. The action was not designed primarily to impact directly on
systems for adult education, but rather emphasised the role that greater collaboration can
play in building capacity and subsequently promoting changes in structures and systems.
The main focus of the Grundtvig action was thus to promote the process of transnational
co-operation and collaboration among relevant bodies within the sector. Few, if any,
forums existed for the exploration of matters of common policy interest at European level.
It was therefore recognised from the beginning that the application of this objective should
focus on creating the capacity to explore such matters – particularly through the G4
networks.
Overall, in terms of participation, the number of adult education institutions participating in
G2 Learning Partnerships rose year-on-year from 478 in 2001 to 1,980 in 2006 1 and a total
of 455 multilateral projects, networks and thematic seminars (G1 and G4) were funded .2
The most common outputs were exchanges of experience and good practice (84%);
partnerships or networks (54%); websites (50%); seminars, workshops, conferences and
exhibitions (48%); transnational meetings (47%); and new teaching/training materials
(41%)3. Mobility was also a significant output of the Action, including mobilisation of
partners for meetings and workshops, many of whom experienced their first opportunity for
mobility.
Relevance (To what extent were the intervention's objectives pertinent to needs, problems
and issues to be addressed?)
Overall, this evaluation supported the conclusions of the Interim Evaluation of Grundtvig4,
which reported that it was seen as relevant to the needs of adult education in Europe,
since its objectives were broadly defined and thus provided flexibility to meet institutional
or national priorities, and for the Action to respond to shifts in needs over time. Nationally,
given the rather broad sweep of the global objectives, there was very little need to tailor
these at national level to achieve consistency with national priorities. While the Grundtvig
Action was not designed primarily to have a direct impact on whole adult education
systems as such, it did have the potential to contribute to major European policy agendas.
Its contribution to the Lisbon agenda was mainly in terms of internationalising
organisations in the sector, so that they are better positioned to work on common
challenges, including those particularly germane to the Lisbon goals – equipping people
1
European Commission (2007): Grundtvig Success Stories, Europe creates opportunities.
ibid.
3
Based on survey of Grundtvig participants, base size = 627.
4
ECOTEC (2004) The Interim Evaluation of the Grundtvig Action of SOCRATES programme.
2
17
ECOTEC
with the skills for a knowledge society and increasing access to lifelong learning
opportunities, especially for those without a basic education. The strategic objectives of
the Education and Training 2010 work programme also applied to the adult education
sector, given their emphasis on increasing the quality and effectiveness of education and
training systems, facilitating access for all and opening up education and training to the
wider world, notably their emphasis on learning at all stages of life.
Coherence within the Grundtvig Action appears to have been strong, with an appropriate
mix of measures selected to reflect the need for the sector to develop more European
platforms for joint working, allied with individual mobility and professional development. It
is also likely that much of the activity funded was additional, given the fragmented nature
of the sector, and since the majority of Grundtvig project co-ordinators reported that their
project would not have taken place without EU funding. Furthermore, lack of additionality
was certainly not an issue raised by National Agencies or other stakeholders.
Effectiveness (To what extent were the programmes successful in attaining the objectives
set and achieving the intended results?)
At programme level, Grundtvig's funded activities and results made a strong contribution to
meeting the operational objectives of the Action – in particular in terms of supporting
European co-operation projects, exchange of experience through transnational learning
partnerships, facilitating the training of adult education staff abroad and, to a lesser extent,
the development of European platforms in adult education and the exchange of
information to shape policy and research. However our analysis of the intervention logic
identified some weak linkages between operational objectives and the global objectives of
the action; and we thus concluded that if a more focused set of specific objectives, taking
into account intended impacts and recognising the specific contextual conditions in the
adult education sector had been developed, this may have resulted in a more robust set of
objectives, against which success could be measured. For example, the operationalisation
of the specific objective to boost 'demand for participation in lifelong learning activities' was
weak. This outcome might be expected to flow from the creation of a climate more
conducive to achieving increased participation rates, but the Grundtvig activities in
themselves were unlikely to have been able to contribute directly to the achievement of
this.
The evidence suggests that the strongest emerging impacts in adult learning were in terms
of increased and sustained co-operation between institutions/organisations, increased
capacity for mobility and increased European outlook for individuals and institutions. Selfsustaining networks/ communities of interest in lifelong learning at EU level were
18
ECOTEC
established and have been successful1 although the extent to which these were sustained
was more limited. There appears to have been a particularly significant impact on
European outlook for professionals and students, particularly through emphasis on
intercultural methods and issues. This was particularly important for first time overseas
travellers, e.g. adult learners mobilised by these projects, as well as for participants from
European Member States with relatively under-developed adult education sectors. While
activity has resulted in transnational co-operation and has benefited a number of
participating organisations and individuals, it cannot be said that this ha s been translated
into a stronger "European dimension" to the adult education sector. Such an impact will
require the sector and its main players to be able to exercise a much greater influence on
policy development, an outcome which appears to remain elusive. The evidence also
suggests a solid, if small-scale impact on the everyday lives and careers of professionals
in the adult learning sector through enhancing their skills and improvements to the quality
of curricula and teacher training. There were only limited impacts in terms of language
learning; this was consistent with the Commission's decision not to set aside a specific
funding pot for this objective within the Action. There was limited evidence of impact on
transparency/recognition of qualifications and limited impact on national policy influence.
In terms of utility, the action could not be said to have contributed greatly to the identified
needs in adult education, given the limits of its scale and scope. However it did lead to
some strong shorter term impacts at the individual level which were of high utility. In terms
of value for money, the budget allocated to Grundtvig was by no means extravagant.
Given the under-developed state of the sector, and if the desired effect was to sow the
seeds of a European dimension in adult learning, then the cost appears reasonable,
however limited the evidence of long term impacts may be. G2 learning partnerships were
particularly found to be cost-effective in terms of bringing new organisations into European
co-operation. That said, the amount of funding allocated cannot therefore be said to be
commensurate with the state of the sector, nor the scale of the challenges posed to the
sector by the Lisbon strategy.
1
Indeed, this impact may have been greater than our evidence suggests since our sample did not include
any action 4 projects 'Grundtvig networks'.
19
ECOTEC
3.0
Efficiency of the three programmes
Separate evaluations of the efficiency of the three programmes were carried out, as well
as a common analysis of the committee system. The key evaluation question regarding
efficiency was: To what extent have the desired effects been achieved at a reasonable
cost?
3.1
Efficiency of Socrates II
Overall, the management and monitoring of Socrates II facilitated effective and efficient
operation; in line with the identified needs; the programme’s objectives; and the expected
outputs and results. About two thirds of the Socrates II respondents considered the overall
application process to be 'good' or 'very good'. However there were some areas for
improvement, which should be taken into account in the delivery of future programmes;
e.g. co-ordinators expressed some dissatisfaction with the amount of administration
involved in application and selection, and monitoring and reporting requirements, which
they found to be complex and time-consuming. Difficulties were apparent in the availability
and quality of monitoring information, specifically basic information such as application
forms and interim and final reports. Evidence suggested there had been delays in
payments to the projects – and the evaluators inferred that improved ICT tools would
improve this aspect of the programme management.
3.2
Efficiency of Leonardo da Vinci II
The overall management and monitoring of the Leonardo programme generally facilitated
effective and efficient operation. Improvements made to the programme over the years
were generally welcomed; and the percentage of satisfied participants rose from 13% in
the period of the first call for proposal (2000-2002) to 29% in the last call period (20052006). Changes in procedures A and B were generally welcomed by respondents,
although some stated the division of labour between the National Agencies (NAs) and the
Commission within procedure B may have been causing some disparity in the quality of
accepted proposals. Most participants in Leonardo were satisfied with the application
procedures. Only the quality and usability of the application form was received less
positively. On a positive note, the delays in payment that were heavily criticised in the first
phase of the Leonardo Programme (1995-1999) were largely resolved in the second
phase (2000-2006) and generally budgets were felt to be adequate. At programme level
both the NAs and the Commission indicated that the total available budget was not
20
ECOTEC
sufficient compared with demand, especially for mobility – this indicates rising demand for
this type of activity. The main criticism of the programme management was the lack of a
comprehensive IT system/tool to manage programme data although some NAs had
developed management information systems, including one developed by the Norwegian
NA which was also used by others, and ten NAs had – following the encouragement of the
Commission – obtained ISO 9001. There was also some ex-ante and ex-post
measurement of impact (though Rap4leo). It was the evaluator's view that having a wellstructured and coordinated management information system would have improved the
Commission’s understanding of the programme at EC and national levels.
3.3
Efficiency of eLearning
The management and monitoring of the all actions of the eLearning programme facilitated
effective and efficient operation on the whole, and applicants in actions 1, 2 and 4 were
generally satisfied with the efficiency of the management processes: 96% of eLearning
project co-ordinators in our survey considered the overall application process to be
adequate or better, with more than half (55%) rating it 'good' or 'very good'. In particular
the qualitative evidence highlighted the proficiency of the Executive Agency (EACEA) in
responding to all queries promptly by email and circulating email newsletters to all funded
projects. Good practices like this could be extended to other programmes going forwards.
The innovative model of intervention used in eTwinning was welcomed by schools as
meeting a key need in the sector and feedback indicated schools found the tools very
useful and well-administered. Schools made some small suggestions for improvement of
the eTwinning services, including refinements to the partner search tool, which c ould be
taken into account in the delivery of future programmes.
3.4
Efficiency of the Committee system
Committees represented a forum to allow Member States to participate in the political and
strategic management of the three programmes and to comment on and approve
arrangements for programme management as well as to ensure some flexibility in delivery,
allowing them to adapt to changing needs and political contexts. Their key strength - as
identified in interviews with Committee secretaries - was their consideration of 'points for
information', since this allowed a more wide-ranging discussion (of issues not specified in
the programme Decisions), enabling Member State representatives to gain a deeper
understanding of programme implementation.
21
ECOTEC
Weaknesses respondents identified related mainly to the administration of the
Committees; there were challenges associated with the time pressures involved –
documentation was often distributed very close to the meeting date, as a result of tight
selection timetables. This reduced the amount of time and attention members could pay.
There was also no evidence of linkages between the three Committees, which the
evaluators concluded could have been beneficial.
22
ECOTEC
4.0
The common impacts of the programmes and the
contribution to the Lisbon goals
4.1
Introduction
To conclude the evaluation we returned to the evaluation framework prese nted in Chapter
2 to examine the programmes collectively and in relation to the Lisbon goals. Each of the
common high level impacts was assessed.
Key Impacts of the Programmes
•
Most significant impacts were on the creation of a European education area
•
Strong impacts on the development of ‘professional social capital’ with
significant effects on both individuals and organisations through mobility
activities, networks, and partnerships
•
Transnational cooperation firmly embedded in higher education where almost all
institutions participated and a platform established in VET for the first time
•
Sustainable cooperation mechanisms still patchy in the schools and adult
sectors
•
Improvements in teaching, learning and management formed the second most
common group of impacts
•
Widespread curriculum impacts: professionals brought back and applied their
learning quite extensively within the curriculum, alongside the significant
contributions made by 'strategic' interventions such as dissemination networks
•
Impacts on management were also registered such that there was quite a deep
and probably enduring impact within institutions in certain areas
•
Impacts on policy and practice were not intended goals of the programmes and
came mainly through the efforts of individual projects to disseminate their results
23
ECOTEC
4.2
•
Most policy impacts were at local level but Leonardo and in particular Erasmus
registered significant impacts at national and European levels
•
There were widespread impacts on language learning not just through the
development and dissemination of new methods but also through the act of
project participation itself, although this tended to involve the more common
languages, especially English
•
The main socio-economic impacts took place through mobility, this being an
especially prominent part of Erasmus and Leonardo. Erasmus was instrumental
in institutionalising and firmly embedding mobility – and its benefits – in
university life. In Leonardo mobility projects were the most successful and costeffective of all measures within the programme
•
Despite being a prominent horizontal theme, tackling social disadvantage was
not a prominent feature of the programmes either in terms of activity or impact,
though it was perhaps most prominent in Grundtvig and eLearning
Creation of a European education area
It was clear from multiple sources of evidence that in general the creation of a European
education area was the single most important common goal for the programmes evaluated
and that cooperation and increased European outlook were considered to be important
impacts on both organisations and individuals across all programmes (85% to 93%).
Furthermore, the added value of European cooperation was universally rated highly (79%
to 94%). All programmes contributed to the development of what we have termed
'professional social capital' through mobility activities, co-operation amongst professionals,
networks, partnerships.
Success in this area depends on developing a culture of European co-operation which is
sustainable and where co-operation becomes commonplace rather than unusual; and on
putting in place on-going mechanisms in the form of self-sustaining communities of
interests through which co-operation can take place. Such an infrastructure has arguably
been put in place through Erasmus, where almost all European universities participate,
and in VET a platform of transnational cooperation has been established for the first time,
but in other areas it is still patchy and this can be developed further.
24
ECOTEC
There is also an issue regarding the extent to which establishing a European area
depends on a large volume or 'critical mass' of mobile students or co-operation activity.
The most successful and best-resourced action – Erasmus – achieved a reach (or 'market
penetration') of only 4% of the student body. However, large numbers of organisations
were involved in the programmes and this therefore extended the programmes’ reach –
and their indirect impacts – much further. In the case of the programme that was most
successful in this respect – Erasmus – this meant a reach to almost all relevant
institutions. Given the nature of the programmes, it would be wrong to equate ‘critical
mass’ with volume. Rather, what becomes important is ensuring that individuals become
'change agents' who can take the benefits of European co-operation to others through
effective dissemination and exploitation.
In relation to the more specific issues of convergence and transparency, the programmes
have played the role of developing tools that can support and enhance ‘structural’
developments such as the development of the European Qualifications Framework (EQF),
European Credits in VET (ECVET) and Europass. However, outside of HE and VET there
are few structures to prevent divergence in future practice.
4.3
Improvements in teaching practice and approaches to learning and management
Improvements in teaching, learning and management formed the most common group of
impacts after those associated with the European education area. Like the latter, they
were seen as a positive impact by large proportions of participants and stakeholders
across all the programmes (62% to 78%). Evidence demonstrated that professionals
participating in the programmes brought back and applied their learning quite extensively
within the curriculum, alongside the significant contributions made by 'strategic'
interventions such as dissemination networks. Impacts on the curriculum, in the main,
tended to be in parts of the curriculum rather than all of it. Nonetheless, we regard it as
significant that, depending on the programme, some 8% to 21% of respondents indicated
that the activities in which they had been involved had impacted in all areas of the
curriculum, a significant achievement given the programmes’ design. Impacts on
management were also registered to an extent that had not really been anticipated in the
process of reconstructing the intervention logics for the programmes; this finding speaks of
quite a deep and probably enduring impact within institutions in certain areas.
25
ECOTEC
4.4
Impacts on policy and practice at EU and Member State levels
The extent to which activities were intended to have impacts on policy and practice beyond
the partners involved varied significantly depending on the programmes concerned, the
nature of the activity and the EU's competence in each sector. The fact that some types of
activity were decentralised whilst others were centralised is also highly pertinent here. Of
these two types it is decentralised activities that were more likely to have had an effect on
national policy and practice through dissemination. Impacts on policy and practice at
national levels within countries came through the efforts of individual projects to
disseminate their results; in the main national authorities did not play an extensive active
role in gathering together and disseminating the results from projects. Thus with the
exception of Erasmus, programmes/actions had the greatest impact on policy at local
level. There is also evidence that Leonardo registered good levels of impact at European
levels, which we regard as particularly impressive in light of the comparatively weak
baseline of international cooperation in the VET sector in 2000.
4.5
Increased proficiency in EU languages
All the programmes sought to cover language learning in some way and impact s were
widespread. Language proficiency developed not just through the development and
dissemination of new methods but also through the act of project participation itself,
although this naturally tended to involve the more common languages, especially English.
Indeed, the continuing trend towards globalisation and EU enlargement both provided
strong contextual factors that have almost certainly had more of an effect on language
proficiency than the programmes. Lingua in particular impacted in this area. While being a
small programme with a focus on devising new approaches to language learning, by no
means all projects saw their role to increase the volume of language learning and
teaching, but rather to improve the way in which language learning takes pl ace. Activity in
the schools sector through Comenius and eTwinning also demonstrates positive impacts
on early language learning.
4.6
Socio-economic impacts
The main way in which programmes achieved impact in relation to society and the
economy was through mobility. At the same time, an underlying theme within most of the
programmes and their component parts was the tackling of socio-economic disadvantage.
In relation to mobility the main contribution was through Erasmus and Leonardo, although
such activity was also part of Comenius and Grundtvig. Erasmus was instrumental in
26
ECOTEC
institutionalising and firmly embedding mobility – and its benefits – in university life. In
Leonardo mobility projects were the most successful and cost-effective of all measures
within the programme, and there was a strong contribution to the improvement of skills and
competencies of the people involved, which distinguishes it to a certain extent from
Erasmus with its stronger emphasis on social skills development. Staff trained abroad
under Comenius mobility reported good results in relation to the extent to which they could
use their skills when they returned home, particularly for language teachers.
Tackling social disadvantage was not a prominent feature of the programmes either in
terms of activity or impact. It was perhaps most prominent in Grundtvig, where some 42%
of projects said it was an important objective, compared to 9-19% amongst the other
Actions of Socrates II. eLearning also made a contribution here: 67-75% of respondents
agreed that their project had achieved a group of effects linked to socio -economic
disadvantage including: more individuals being able to use digital literacy skills in the
knowledge society context; improved digital literacy of specific target groups; improved
awareness of using e-learning to improve digital literacy. In Erasmus there was evidence
that the small grants available and the concomitant need for self-financing meant that
students from lower socio-economic backgrounds were less likely to participate than
people from higher socio-economic backgrounds and in Leonardo NAs had to strike a
balance between giving as many students as possible the opportunity to participate by
distributing more, but smaller, grants or giving enhanced financial support to a smaller
number of students.
4.7
The contribution of the programmes to Lisbon-related objectives
The programmes we have evaluated were not designed to achieve the Lisbon goals.
Indeed, our analysis of policy found that the links from the programmes, throug h the body
of education and training policy to Lisbon were complex, lacked clarity and were hard to
unravel. That said, the greatest contribution of the programmes appears to have been in
terms of 'opening up education and training systems to the wider world', following by
'increasing the quality and effectiveness of education and training systems'. Effects in
terms of 'facilitating the access of all' were found to be less widespread, although clearly
individual projects made important contributions in their own right. Unpacking this further,
we can say that the programmes' strongest, most consistent and widespread impact was
in terms of strengthening European co-operation and increasing mobility and exchanges,
to which all the programmes and actions made a contribution.
Looking in more detail at quality and effectiveness, all the programmes contributed in this
area, but as defined in Education and Training 2010 the objective of developing skills for
27
ECOTEC
the knowledge economy was not a strong feature of the programmes collectively, though
Leonardo registered important impacts in this regard, and improving recruitment to
scientific and technical occupations and making the best use of resources did not feature.
Ensuring access to ICTs was especially important in eLearning and Minerva.
The contribution of the programmes/actions to quality and effectiveness varied between
the programmes/Actions and the different modes of intervention. Whilst it was the task of
networks and cooperation projects to develop and disseminate new methods, tools and
approaches at a pan-European level, quality and effectiveness impacts were also
registered in the decentralised actions but in highly localised contexts, and in general not
across whole institutions. In terms of the extent to which the programme impacted on
entire education and training systems both Erasmus and Leonardo were influential in
respect of supporting developments which have and continue to influence the shape of
systems in their respective systems. In other sectors, systems impacts were not intended.
However, some countries used the opportunities presented by the programmes to bring
about change (as in the case of Grundtvig to enhance under-developed adult learning
systems) although these instances were rare.
28
ECOTEC
5.0
Strategic Conclusions
5.1
The relevance of the programmes
Overall it is clear that broadly speaking the programmes were relevant to the needs in the
sectors to which they were applied. Erasmus continued to build on the success it had
already established and has embedded a sustainable culture and platform for on-going
cooperation. A key question for the new Lifelong Learning Programme is how to maintain
the vitality of the Erasmus ‘product’ to ensure its future success: improvements in the
quality of the mobility period, support provided and the recognition of the study period
should assist in this regard. Leonardo has built capacity within the VET sector and helped
to place the sector on a sounder footing with regard to transnational cooperation and the
Copenhagen process and needs to continue to contribute to progress in the areas of
quality, attractiveness and recognition and transparency. In the schools and adult learning
sectors, the programmes were designed to stimulate cooperation and the development
and sharing of new approaches and good practice as well as to meet highly localised
needs. We consider this combination to have been appropriate to the task in this period
and to have developed both capacity and capability within the sector. For these sectors,
we believe the question is now how to develop activities so that they can make a stronger
contribution to Lisbon – if this is desired – rather than delivering more of the same activity.
To this end we recommend that firm priorities should be set for programmes in the schools
sector in relation to the reduction of early school leaving, and in the adult education sector
for raising participation in lifelong learning to assist in the achievement of the relevant EU
benchmarks, both of which are behind target.
5.2
The effectiveness of the programmes: objectives, impacts and sustainability
The programmes' global objectives concerned both social and economic impacts and
effects on practitioners, practice and to a lesser degree policy within the various
educational and training systems; in budgetary terms there was roughly a two thirds/one
third split in favour of the latter. This has been reflected in the pattern of their achieved
impacts. Practitioners benefited from a stronger European outlook, being brought together
in various co-operation activities and being provided with a means to learn new practices
which they have carried back into their teaching practice. Impacts on management were
also registered to a significant degree that had not been anticipated. Most impacts were
small scale, localised and within parts of institutions. The programmes thus had an
important impact on the development of professional social capital. Without the
29
ECOTEC
programmes much of the activity would not have taken place (additionality). Sustainability
remained an issue in the schools and adult education sectors which have less developed
capacity for cooperation: most projects will or have already fragmented and will not
maintain all their partners and activities. It is therefore likely that a high percentage of the
impacts will not just be localised but will also turn out to be short-term. The question that
arises however is whether more could have been done to ensure greater, more lasting
impact.
5.3
Balancing grass-roots needs and top-down direction
By design the programmes had very broad objectives, virtually no targets and a
voluntaristic modus operandi. This meant the programmes were highly flexible, that
learning ‘from the field’ could take place and that objectives could be fine-tuned to local
and sectoral needs. At the same time, two factors made it difficult for managing authorities
to balance this ‘bottom-up’ element with ‘top-down’ direction. First, although there was a
process for setting priorities this was found to suffer from a number of deficiencies: it was
not clear what status the priorities had alongside the objectives and the priorities could
vary from the specific to the highly generalised (looking in some case like general
objectives rather than priorities). Furthermore, the evidence points to the fact that in many
cases in practice they did not have a major effect on the direction of the programmes.
Secondly, there were deficiencies in management information systems such that it was
very difficult for managing authorities to access robust, accurate and timely information
about the outputs and results – in aggregate terms – that had been achieved at particular
points in the programmes. Naturally this inhibits the ability to make effective decisions
about the future direction of programmes.
5.4
Modes of intervention
There were two main modes of intervention: centrally managed transnational networks and
cooperation projects; and decentralised individual mobility programmes and interinstitution partnerships. We conclude that this division of types of activity between different
levels of management was appropriate.
Additionally, eTwinning adopted a unique mode of intervention whereby funding was
directed to a system of Central and National Support Services and an IT infrastructure (vi a
the eTwinning portal) instead of giving funding directly to applicants. In this mode, schools
(the target audience) benefited from using the tools and resources to find partners to twin
30
ECOTEC
with, and undertake school-level projects. This was a mode of intervention that provided
to be popular with the target audience and highly cost-effective.
5.5
Decentralisation and the role of Member States
Member States have a key role to play in modernising and reforming their systems in the
Lisbon context. However, the evaluation in the main found little systematic evidence of
Member State’s making use of the programmes strategically to adjust their policies and
systems, although some countries – particular the new EU members – did so in some
sectors. We consider this to have been a missed opportunity for most (though not all)
Member States.
5.6
Overall contribution to the Lisbon agenda
The programmes' effects were of two types: on practitioners, practice and policy where
their link to Lisbon was indirect and tenuous; and on individuals through the mobility
actions where the contribution was more direct in terms of delivering both social and
economic impacts. Any assessment must take into account the resources employed,
which though significant were small in comparison to education and training budgets. One
would not therefore anticipate major effects. The programmes served the function of
building underpinning capacity and capability within education and training to support
efforts to achieve the Lisbon goals. However, the evidence suggests that much of that
capacity will wither and dissipate now that funding has ceased. Many effects were
intangible and dispersed. In order to have a more significant effect on Lisbon such effects
would need to be brought together and focused. Furthermore, most Member States were
not actively engaged in the programmes to the extent of systematically seeking synergies
with their policies. Greater consideration also needs to be given to the role that the
programmes can play alongside and preferably in conjunction with other sources of
funding, reflecting the move towards greater integration of actions that is being
promulgated through the New Integrated Guidelines and associated National Reform
Programmes.
Lisbon accords Member States a central role and thus it is important in the new generation
of programmes that they seize the opportunities presented and also adopted a holistic
viewpoint of their potential.
31
ECOTEC
6.0
Recommendations
6.1
Strategic recommendations
6.1.1
Building on the achievements of 2000-06
The programmes/actions were designed and made different contributions to the varied
sectors at which they were directed which were appropriate to the diverse baseline
conditions they faced. After seven years of operations, it is important that the Lifelong
Learning Programme capitalises on the achievements made as much as possible.
In higher education, Erasmus needs to revitalise its mobility 'product': the Commission
and partners involved in the Bologna process should take steps to improve the quality
of the mobility period, the support provided, the range of partners involved from business
and civil society and the recognition of the study period.
In vocational education and training, Leonardo da Vinci II has developed capacity and
contributed to the establishment of transnational cooperation. The Copenhagen process
has acknowledged the need for greater progress in the areas of quality, attractiveness and
recognition and transparency and the Commission should ensure that Leonardo gives
priority to these areas.
In the school and adult learning sectors the priority should be on developing measures that
make a stronger contribution to Lisbon rather than delivering more of the same types of
cooperative capacity building and exchange activities, valuable though these clearly were.
The Commission should therefore give priority to measures that contribute to achieving
relevant benchmarks: the reduction of early school leaving in the case of schools and in
raising participation in lifelong learning.
6.1.2
Achieving the right balance between grass-roots needs and top-down direction
It is important that the right balance is struck between being responsive to grass-roots
needs and ensuring there is adequate top-down direction. Priorities that are set during the
course of the programme should be strengthened in order to have an effect on activities.
The Commission should clarify the relationship between priorities and objectives, setting
out the weight to be given to priorities, to ensure priority is given to proposals that reflect
them, and to ensure that priorities are SMART (i.e. Specific, Measurable, Achievable,
Realistic and Timebound).
32
ECOTEC
6.1.3
Realising the potential for synergy between actions
The evaluation found that few linkages or exchanges had been made between the actions
and programmes considered, which in the case of Socrates meant that little added value
appears to have flowed from bringing the diverse actions together in one programme. The
integration of the actions in the new Lifelong Learning Programme is expected to facilitate
better exchange across actions. However, the Commission and Member States should
implement a number of measures including the facilitation of network meetings and the
proactive exchange of information between EC staff/EACEA during the different stages of
the life of a project (selection, implementation).
6.1.4
Improving synergies within countries
The evaluation found little evidence of Member States having an holistic approach to the
programmes/actions or of making strategic use of them to adjust policies and systems.
Given the central role of national governments in education and training policy it is
important that Member States are more actively engaged with the Lifelong Learning
Programme. The Commission should stress to Member States the importance of taking
an holistic approach to decentralised actions and of forging links with national policy and
strengthen the collection of monitoring data to inform Member State action.
Member States should take a systematic look at the role that the LLP can play in taking
forward the EU agenda. They should also examine the potential for synergy between EU
programmes and national policy and take stock of the linkages between the LLP and other
EU programmes and funding streams such as ESF.
6.1.5
Giving greater priority to social disadvantage
Issues of social disadvantage had not been a prominent feature within most of the
programmes and actions examined, despite being emphasised as an important horizontal
issue. It would be beneficial therefore if greater emphasis were to be given to social
disadvantage in annual priority setting by the Commission. Additionally people from
lower socio-economic backgrounds tend to be under-represented within Erasmus mobility
periods, probably on account of the fact that they are predominantly self-financed. The
Commission should give consideration to ways of increasing grants for students from
disadvantaged backgrounds, as is already happening in some Member States.
33
ECOTEC
6.2
Operational recommendations
6.2.1
Developing indicators and targets
Almost all of the programmes had very broad objectives which did not perform well when
measured against the SMART yardstick (i.e. being Specific, Measurable, Achievable,
Realistic and Timebound). They were also deficient in terms of indicators and targets.
The Commission should ensure the development of coherent banks of indicators and
targets and that the any priorities set are SMART (i.e. Specific, Measurable, Achievable,
Realistic and Timebound).
6.2.2
Improving monitoring
It was very difficult for managing authorities to obtain robust, accurate and timely data on
outputs and results. The Commission should seek to develop current management
information systems so that they are able to collect systematically information on
programme outputs and results. Beneficial steps would include ensuring applicants
effectively identify at application stage their intended results and impacts as well as
outputs and to describe the evaluation methods they intend to use; to use typologies like
those used in the evaluation (e.g. mobility, methods, networks etc) to classify results and
impacts; and to identify a sample cohort of projects to supply data on results and impacts
achieved.
6.2.3
Improving efficiency
There are a number of recommendations directed towards the Commission and EACEA
that would help to improve the efficiency of programme management:
• more and better use should be made of ICT to strengthen and improve the efficiency of
all processes, for example through the use of common programme management and
monitoring databases of projects and the wider use of online application forms.
• practice can vary between National Agencies, not least in relation to quality assurance.
Good practice should be better shared, e.g. through workshops, audits, peer reviews,
evaluations, and consideration should be given to following the example of Leonardo
where NAs were encouraged to obtain ISO accreditation
• more comprehensive guidance and support from the Commission and EACEA would
improve the mainstreaming of impact and sustainability of the programme
achievements.
34
ECOTEC
6.2.4
Improving dissemination and sustainability
Much of the capacity and many of the capabilities that were developed are likely to
dissipate now that the programmes have ended, and many of the effects were – as
intended - not tangible, and were dispersed widely within the education and training
infrastructures of Europe.
To improve the collection of good practice and dissemination the Commission and
EACEA should require projects to allocate specific resources to dissemination and give
consideration to the possibility of including a 'dissemination phase' similar to Action 3
within the ESF Equal Programme. They should also facilitate the sharing at EU level of
good practice in dissemination and exploitation. National authorities should be required
to adopt effective strategies to disseminate and exploit results and impacts within their
countries which complement those of projects.
To improve sustainability, the Commission and EACEA should consider additional
support measures to try to ensure the continuance and mainstreaming of activity.
Consideration should be given to introducing measures such as the SALTOs in the
YOUTH programme 1. Support should also be provided to encourage participants to
understand and plan for sustainability. All applicants should be required to have a
sustainability plan built into their project design, which enables them to plan for their future
more effectively
Additionally Member States need to take steps to ensure that European co-operation
becomes embedded within curricula so that it becomes the norm rather than the
exception. Additional funding sources should be sought either in other EU programmes
where synergies might be made or in the form of public-private partnerships.
6.2.5
Structuring cooperation
Sharing and exchanging information amongst institutions across Europe was a central
aspect of the programmes. However it was observed that in multiple cases co-operation
was not developed in a clear framework with defined joint concrete objectives. To improve
this situation it is recommended that the Commission ensures the provision of guidance
and support, especially for schools and adult education institutions that do not have the
same level of transnational experience as in other sectors.
1
SALTOs are resource centres that provide support such as training courses and contact-making activities to support
organisations and National Agencies on various themes related to the programme. There are eight for YOUTH. Each
SALTO is hosted by a National Agency with funding provided by the Commission.
35
ECOTEC
6.2.6
Improving how equality and accessibility issues are handled
The evaluation concluded that equality and accessibility was one of the areas where the
programmes could be improved. A smaller than expected number of projects reported a
focus on this transversal issue and some projects have a very limited understanding of
inclusion issues. In the future participating institutions should demonstrate that they have
considered how equal opportunities and empowerment will be integrated into all aspects of
their activities.
Steps to be taken forward by the Commission and EACEA could include: setting a
comprehensive equality and empowerment policy that outlines the rights and
responsibilities of all participants; promoting and tracking progress in equality and
empowerment policy and practice; providing support, materials and processes that take
the needs of target groups into consideration.
36
ECOTEC