Fifth Conference on Evaluation of the Structural Funds
Budapest 26/27 June 2003
Challenges for Evaluation in an enlarged Europe
Paper presented for Conference Workshop 2
Comparative element to the ongoing mid-term evaluations
of the objective 2 Programmes in the
East of Scotland, Nordrhein-Westfalen in Germany and Western Scotland
Ulrike Kugler1, Carolyn Sawers1, Ruth Downes2, Regina Trenkler-Fraser2,
1Ulrike
Kugler - Objective 2 Secretariat (managed by agiplan ProjectManagement) in Nordrhein-Westfalen
(NRW) – Haroldstr. 4, 40190 Düsseldorf, ulrike.kugler@mwa.nrw.de or www.ziel2-nrw.de - and
Carolyn Sawers - Western Scotland Programme (managed by SEP - Scottish European Partnership Ltd);
www.wsep.co.uk/aboutsep.asp or for the East of Scotland www.esep.co.uk/
Both represent the regions engaging the mid term evaluations
Downes – European Policies Research Centre (EPRC) www.eprc.strath.ac.uk/iqnet/iq-net/index.html
Carrying out parts of the evaluation work in NRW and
Regina Trenkler-Fraser – EKOS regina.trenkler-fraser@ekos.co.uk,
Carrying out the evaluation of the horizontal themes and the benchmarking
2Ruth
17.02.16/b/106752927
1/24
-2-
Content
1 Introduction
2 Context - Background of regional co-operation between Scotland and NRW
2(a)
2(b)
2(c)
2(d)
The Partnership
The Regions
The Programmes
Integration of Horizontal Themes
3 Comparative Elements and Benchmarking study
3(a)
3(b)
3(c)
3(d)
Evaluation approaches - General Concept
Methodology
The results of the first workshop
Proposed structure of the Benchmarking Study and questions to be
answered
1 Introduction
Partnership connections and co-operation have been developed between NRW and Scotland for
a while. The co-operation and the benchmarking of certain aspects in the ongoing midterm
evaluation for objective 2 programmes of Nordrhein-Westfalen (NRW), West and East Scotland
are built on those experiences. This new co-operation process shall contribute to respective
understanding and knowledge-transfer in the context of EU structural funds and here in particular
in evaluation and the use of benchmarking as an approach in evaluation. The central objective of
this evaluation- and benchmarking- process is a discussion and reflexion on programme
experiences in different national and regional contexts. The engaged managing authorities ,
regional secretariats and evaluators believe, that the resulting experiences can contribute
significantly to a better understanding of complex problems and improve the ongoing
communication and support of the programme managing organisations.
The focus of the partnership is the following: There are three evaluations - one in each region carried out independent from each other. But the ongoing process and the main results of each
evaluation will be compared during and at the end of evaluation work. The benchmarking is
integrated into this process. It goes beyond a general comparison and considers the
implementation and relevance of horizontal themes in the Objective 2 programmes in depth.
The paper will address good practice in terms of programme evaluation at different stages of the
programming life cycle, benchmarking and cross-region co-operation projects. The evaluation’s
methodology has a number of interesting and innovative elements including:

Bringing an element of comparative evaluation to the mid-term evaluation process

Evaluation of the horizontal themes focussed on the practical process of mainstreaming

Co-operation and joint working: this project demonstrates co-operation between public
administrations in regions, between private sector evaluators and between the public and
private sectors generally.
3/..
/17.02.16/D:\106752927.doc
-3This Paper is a combined effort of the regional secretariats and the evaluators of the three
programmes. It describes the evaluation process from the beginning until June 2003. Therefore it
can be regarded as a working paper – not aiming to present final results. It gives opportunity for
the regions and evaluators to discuss the initiated process in a broader context. For this it will be
helpful to all being involved in the process to get new ideas where to do better, where to ask
other questions.
2 Context – Background of regional co-operation between Scotland and NRW
The following parts describe the origins of the partnership between the regions. They show the
basis that has been built up and that enables the regions with all their commonalities and
differences to try out a combined evaluation approach. Necessary for the understanding of the
co-operation and the evaluation approach is also a basic knowledge of the regional and
programme context. As it will be pointed out later, the integration of horizontal themes within the
programmes needs to be highlighted.
2(a)
The partnership
The startingpoint of Scottish-Nordrhein-Westfalian partnership was in the year 2001. The project
has been chaired and co-ordinated by Scottish Enterprise under the auspices of the Scottish
Executive and the Ministerium für Wirtschaft und Arbeit des Landes Nordrhein-Westfalen (MWA).
Also involved were experts from different organizations in both regions.
The partners agreed that an examination into some of the economic development issues
surrounding common problems of land reclamation and environmental renewal would be mutuall
beneficial.
This collaborative work resulted in a document "Partners in Development", supported also by
officials from the respective Ministries.
The initial concept developed into a project proposal for collaborative work examining four
specific themes, with additional consideration of the part played in the economic development
process by the EU-Structural Funds. The report, to some extent, tests some of the assumptions
we tend to make about the way others do their business; are systems in Nordrhein-Westfalen
more geared to long term thinking than in Scotland, and if so, can Scotland learn from that? Are
partnership structures in Scotland more fully developed than in Nordrhein-Westfalen, and if so,
what can, or should, be transferred?
The result was a project which demonstrated partnership in action, and underlined the benefits of
the regions of Europe working together. The diversity of regions within the Member States, even
where they face similar problems, leads to different approaches and different solutions. Work
such as this allows the regions to learn from each other and grow together.
The project was a comparative, but concise, discussion of four areas of economic development
activity:





Land Reclamation and Environmental Renewal
Clusters / ‘Competence Fields’
Sustainable Development and Environmental Technology Industries
Entrepreneurialism and Sustainable New Business Start-Ups.
Use and management of EU-Structural Funds
So the project was much more about seeking positive outcomes for the participating regions and
organisations. In seeking to understand each others’ systems, the authors of the relevant
chapters have noted, and described, differences where that is helpful in terms of contextual
4/..
/17.02.16/D:\106752927.doc
-4understanding. They have quickly moved on, however, to identify very specific areas of policy
and practice which can be influenced by joint learning, or where further work is called for. The
project team has identified a number of outcomes of value:




Joint learning, both in exploring systems in the partner region, and in re-examining
systems in the home region
Opportunities for further collaborative investigation
Opportunities for influencing policy development at home
The opportunity to leave a legacy which may be of value to candidate countries
developing policies in similar fields.
Due to the opinion of all project partners an abundance of knowledge was gathered by the first
partnership approach in 2001. Both regions signalised lively interest to continue co-operation and
with that going beyond description of situation. A new approach is ment to lead to concrete
projects and to regard more dynamic aspects.
In addition to the study Partners in Development the contact network IQ-Net1 has to be
mentioned as the initial point of collaborative work. Since 1996, IQ-Net has offered to partner
regions of Objective 1 and 2 areas research and exchange on structural fund topics- including
evaluation.
In context of the IQ-Net meeting in December 2001 in Grobbendonk/Belgium the three regions
agreed to intensify the cooperation between Scotland and NRW. Result of this discussion was ,
that the midterm evaluations of objective 2 Programmes offered a good chance to start-up a new
co-operation project.
Why choosing the mid term evaluations?:
- All three regions have the same regulations in timetable, procedure and methodologies
to be fulfilled in evaluation. In comparison to other political aspects the conditions for
setting up a regional benchmark are advantageous.
- Evaluations will give chance to get deeper into strategies and implementation of
structural policy of the regions, to assess the effectiveness of funding and to re-think the
own practice compared to each other region.
- The implementation of the Scottish and the NRW structural funds programmes are
regarded as good practice in terms of the EU-Commission. The quality of the three
programmes is said to be high.
- The size of eligible areas, the ERDF funding and -with restrictions- the economic
problems/challenges are comparable.
Further on, in February 2003 a co-operation agreement between Scotland and NordrheinWestfalen was signed by ministers of both regions in Düsseldorf. The agreement on a high
political level covers several areas EU-Policies, Research and Development and industrial
reconversion. It is a political commitment,
2(b)
The Regions compared2
The following part is giving a brief overview on the regional context – showing the eligible area,
population, contribution of the structural funds (objective 2), the main strengths and weaknesses,
the main similarities and differences are additionally pointed out.
1
IQ-Net is an exchange of experience network involving Objective 1 and 2 regions run by the European Policies Research Centre at
the University of Strathclyde in Glasgow. More details can be found at www.eprc.strath.ac.uk/iqnet
Informations are based completely/taken of the "Hintergrundbericht"/Backgroundreport: Downes,R; Trenkler-Fraser,R
(2002):Benchmarking Study Nordrhein-Westfalen and East and West Scotland; Background Report , First Workshop in Gelsenkirchen
28-29 October 2002; Glasgow – EPRC and EKOS Ltd.
2
5/..
/17.02.16/D:\106752927.doc
-5Objective 2 area North Rhine-Westfalia
Total population of Objective 2 and phasing out areas
4 million
Total Structural Fund contribution for 2000-06 programme
970 mill. euro
ERDF contribution
823 mill. euro
ESF contribution
147 mill. euro
ERDF per Capita
206 TEuro
6/..
/17.02.16/D:\106752927.doc
-6Objective 2 area Scotland
West of Scotland
East of Scotland
Total population of Objective 2 and
phasing out areas
2.3 million
550,000
Total Structural Fund contribution for
2000-06 programme
483 mill. euro
250 mill. euro
ERDF contribution
419 mill. euro
250 mill. euro
ESF contribution
64 mill. euro
-
ERDF per Capita
210 TEuro
455 TEuro
7/..
/17.02.16/D:\106752927.doc
-7-
Key regional strengths and weaknesses – NRW
Strengths
Weaknesses
largest market area in Germany with
certain areas of untapped market potential
for regional firms
central European location with good
access to key transport access and with
high international accessibility
comprehensive network of technology and
incubator units
good supply of business sites, many in
inner-city locations
strong network of universities and technical
colleges, research institutes and higher
education bodies
high number of graduates
good pool of qualified labour
financially strong large firms and high
concentration of HQs in or near the
assisted area
significant improvement in environmental
situation and wide-ranging skills in
environmental protection and renewable
energies
experience with innovative approaches to
regional development
Growth rate and investment quota significantly
below federal and Land average leading to
increasing disparities in terms of GDP and per
capita income
Below average business productivity and
competitiveness
considerable lack of job and above average
unemployment
low share of SMEs and craft firms
no growing network or cooperation structures
(except in mining sector)
below average rate of new firm formation,
including in technology sectors
poor rate of R&D activities and lack of regional
innovation strategies
below average growth of service sector
qualification deficits in ‘new’ sectors
particularly poor labour market prospects for
women given traditional focus on ‘maledominated’ sectors
poor image of Ruhr hinders service and tourist
sector and the attraction of new technologies
congested road system
environmental damage from industrial past
KEY COMMONALITIES
 Low or underdeveloped business and growth rate
 Skills shortages, particularly in new or growth sections (particularly NRW and West of
Scotland)

Strong research/university presence
8/..
/17.02.16/D:\106752927.doc
-8Key regional strengths and weaknesses – Scotland
West of Scotland
East of Scotland
Strengths
Strengths
Significant presence of growth sectors
and strategic sites
Excellent research base
Wide range of visitor attractions and
conference facilities
The region's natural environment
Well-established infrastructure
Strong partnership approach
Available match funding via Social
Inclusion Partnerships
Experience in intermediate labour
market initiatives
Weaknesses
Low rate of new firm formation
(particularly among women) and low
business performance
Gaps in ICT infrastructure and lack of
relevant strategy
High number of vacant/derelict sites
High levels of multiple deprivation in
concentrated areas
Low level of educational attainment
Strong vertical gender segregation
Poor transport infrastructure
Private sector attitudes towards
employing previously unemployed
Weak interface between research and
business sectors
Large/diverse manufacturing base
Attracting overseas investment
Well established tourism product
Research facilities/R&D presence
International business outlook
High quality environment
Well-qualified workforce
Weaknesses
Underdeveloped business base
Number of disadvantaged communities
Over-represented in sectors in
transition
Fewer businesses in growth sectors
Low levels of business investment in
R&D
Large areas of vacant/derelict land
Skills shortages in key areas
Towns struggling to adjust
Shortage of suitable sites and premises
KEY CONTRASTS
 NRW Objective 2 area considerably larger in population terms and with stronger heavy
industrial tradition
 Environmental conditions in Scottish regions relatively better than in NRW

Higher profile of disadvantaged/deprived communities and areas in Scottish programmes
9/..
/17.02.16/D:\106752927.doc
-9-
2(c)
The programmes – main objectives and structure
The following part presents the main strategic priorities of each programme, the structure of
measures and implementation.
Strategic priorities - NRW
Core strategic aims
“Creation of new and securing of existing jobs, particularly in SMEs, through improving the
competitiveness of the region”
Strategic objectives
Increasing investment activity including the creation of new firms
Developing and strengthening regional competencies/clusters
Improving the infrastructure framework conditions
Horizontal aims
Long-term and environmentally friendly development
Creation of equal opportunities for men and women in the labour market
Integration of disadvantage groups in the economic renewal process
Innovation orientation and optimal preparations for the information society.
Strategic priorities – Scotland
West of Scotland
Core strategic aim:
“strengthen the capacity to create and
maintain sustainable economic
growth that leads to greater economic
and social cohesion, that protects and
enhances the environment, and
results in increased prosperity, jobs
and an improved quality of life for
residents in Western Scotland”
Strategic objectives:
to enhance the competitiveness and
innovative capacity of the Western
Scotland economy in order to improve
growth
to develop greater economic and
social cohesion and reduce the
disparities within the region
to promote equal opportunities
to protect and enhance the
environment
East of Scotland
Core strategic aim:
“To promote sustainable economic development
in the East of Scotland which is founded on the
key principles of enterprise, learning and social
justice”
Strategic objectives:
build a strong regional and community capability
in order to leave a durable legacy beyond the
end of the Plan period
assist and encourage the development of a
more competitive, dynamic and innovative
business base
embed an inclusive and collaborative approach
to economic and community development
throughout the region
work towards an equality of opportunity for all
people in the region
improve the environmental performance of the
enterprises, communities and organisations in
the region
meet the challenges of economic and social
change through fostering entrepreneurship and
lifelong learning
10/..
/17.02.16/D:\106752927.doc
- 10 Strategic vision for the region:
“a dynamic, innovative and competitive regional
economy where people and businesses are well
equipped to face the challenges of new and
emerging European and international markets”
Strategic development drivers and themes:
sustainable development (equal opportunities,
economic and social inclusion and
environmental enhancement)
enterprise learning (business creation and
growth, social entrepreneurship, community
enterprise)
learning and innovation (life-long learning,
programme and project innovation)
building regional capacity (hard infrastructure –
spatial priorities & strategic sites – and soft
infrastructure – innovation in strategic
development partnerships and financial
engineering)
11/..
/17.02.16/D:\106752927.doc
- 11 -
Priorities and measures – NRW Objective 2 programme
Priority 1: Business and start-up finance
1.1
1.2
1.3
1.4
1.5
Investment grants for commercial
investment
Loan capital
Funds for business start-ups by
university graduates
Start-up premium for SMEs
Wage subsidies for unemployment
(ESF)
Priority 2: Innovation and development of
competencies/clusters
2.1
2.2
2.3
Technology and innovation
Start-up initiative
Business development initiative for
SMEs
2.4 Integrated environmental technologies/
environmental protection
2.5 Information and communication
technologies
2.6 Tourism, culture and leisure industries
2.7 Consumer and business related services
2.8 Renewable/future energies
2.9 Regional development strategies and
interregional cooperation
2.10 Training, consulting and coaching of
unemployed (ESF)
Priority 3: Innovation related
infrastructure development
3.1 Premises for manufacturing and
service industries
3.2 Greening of industrial sites and
emission-reducing infrastructures
3.3 Technology and training infrastructure
3.4 Logistical services and infrastructure
3.5 Integrated employment and
infrastructure support (ESF)
Priority 4: Support for selected target
groups
4.1 Vocational training for young people
4.2 Integrated development of urban
problem areas (ERDF + ESF)
4.3 Integrated development of rural areas
4.4 Improvement of labour market
opportunities for women
KEY COMMONALITIES
 Emphasis on technology and innovation
 Overall focus through strategic aims, priorities and measures on competitiveness
 Focus on SME development
 Inclusion of new forms of finance (eg. risk and loan capita)

Development of strategic sites, locations and infrastructure
12/..
/17.02.16/D:\106752927.doc
- 12 Priorities and measures – Scottish Objective 2 programmes
West of Scotland
Priority 1: Develop competitiveness and
innovative capacity of region’s SMEs
1.1
1.2
1.3
Enhance access to finance for SMEs
Enhance SME services to develop
competitive and innovative business
base
Develop a competitive workforce
East of Scotland
Priority 1: Strategic economic
development
1.1
1.2
1.3
Priority 2: Strategic locations and sectors
2.1
Priority 2: To develop the region as a
competitive location
2.1 Develop the region’s competitive
locations
2.2 Develop SME facilities to support
competitive sectors and clusters outside
competitive locations
2.3 Marketing and promoting the region as
a competitive location
Priority 3: Increase the economic and
social cohesion of the region
3.1
3.2
SME creation and development
Access to risk capital
Technology and knowledge transfer
2.2
Strategic locations and sectors
(revenue)
Strategic locations and sectors
(capital)
Priority 3: Community economic
development
3.1
3.2
3.3
Community engagement and capacity
building
CED implementation – spatial
targeting
CED implementation – thematic
activity
Community area regeneration
Routes to opportunities
KEY CONTRASTS
 Fewer priorities and measures in Scottish programmes (8 measures in Scottish programmes
vs. 24 in NRW)
 More explicit focus on community economic development and social cohesion in Scottish
priorities and measures
 More explicit focus on environmental issues in NRW priorities and measures

More overt cluster development orientation in NRW programme
13/..
/17.02.16/D:\106752927.doc
- 13 Implementation structures – NRW
Overall approach – ‘Subsumed’ system
In a ‘subsumed’ system3 like NRW, Structural Fund resources are linked closely to pre-existing
policies, schemes and administrative structures. Project appraisal and selection is carried out
mainly by domestic sectoral government departments and funding organisations using the same
basic criteria as the national funding programmes for which they are responsible.
Project applications
Ministry of
Economic
Affairs
[Managing
Authority]
Objective
2
secretariat
Information
on approved
Structural
Fund
projects
Other
sectoral
Land
Ministries
Regional
administrations
(Bezirkregierungen)
Documentation / award decisions for
eligibility check with programme objectives
Confirmation
of available
Structural
Fund
resources
Investment
Bank NRW
[Paying
authority]
Other funding
organisations
(including
Investment
Bank NRW)
Project
appraisal
and selection
Written confirmation
of Structural Fund
resources
Award (national
+ Structural
Fund) to
successful
projects
Funded projects
There are some specific/new features in NRW system in the period 2000-2006 to be mentioned:
The creation of an Objective 2 Secretariat. In a German context, this body is unique to NRW and
acts as a central coordination unit for the programme. Critically, the Secretariat ensures that
Objective 2 specific criteria are also applied to project applications (as well as those of the
specific sectoral Ministry/funding organisation), as well as having an important technical support
function for the monitoring system.
‘Differentiated’ features. The implementation of over half of the programme is not subsumed
through national schemes but comprises projects or initiatives specific to the NRW Objective 2
programme.
3
See Taylor S, Bachtler J, and Rooney M L (2000) Implementing the New Generation of Programmes: Project
Development, Appraisal and Selection, IQ-Net Thematic Paper 7(2), EPRC, Glasgow,
http://www.eprc.strath.ac.uk/iqnet/iq-net/reports.html (German and English summary).
14/..
/17.02.16/D:\106752927.doc
- 14 Implementation structures – Scotland
Overall approach – ‘Differentiated’ system
In a ‘differentiated’ system, Structural Fund resources are implemented through dedicated
organisations which are separate from the national structures. Project appraisal and selection,
together with all other aspects of implementation and monitoring, are carried out by these
organisations, based on Structural Fund specific criteria.
In the Scottish context, the overall Managing Authority responsibility is held by the Scottish
Executive which have representatives on the Programme Monitoring and Management
Committees.
Project Applications
Regional Objective 2
Programme
Management
Executive
Technical and
eligibility checks
Project appraisal / selection,
based on selection criteria
Programme Monitoring
Committee
Thematic and Representational
Advisory Groups
Recommendation
Programme
Management
Committee
Strategic
oversight/
programme
implementation
Award decision making
Decision of approval/rejection
Regional Objective 2
Programme
Management
Executive
Structural Fund award / letter of
approval or rejection to applicants
Projects
15/..
/17.02.16/D:\106752927.doc
- 15 2(d) Integration of Horizontal priorities – NRW
The integration of Horinzontal Themes is the central aspect of the benchmarking approach.
Therefore it is necessary to point them out in detailed manner. The following lists the horizontal
priorities and quantified indicators reported of the SPDs.
Long-term and
environmentally friendly
development

Focus on achieving environmentally sustainable development through
all measures and ensuring that funded activities make the widest
possible contribution to the maintenance and improvement of the
quality of the natural environment.
Overall ambitions of the programme:
Creation of equal
opportunities for women
and men in the labour
market
Integration of
disadvantaged groups in
the economic renewal
process

At least 50 percent of projects (based on public resources) should
achieve an improvement in the environmental situation, or at least one
component of it, or have a positive effect through the improvement of
environmental knowledge/awareness or the integration environmental
aspects into the development process, concept or strategy (43 percent
achieved in last period)

Decontamination of 40 ha of industrial land (compared with 37 ha in last
period)

Conversion of 520 ha derelict sites (Brachflächen) into green sites
(compared with 469 ha in last period)

Focus on creating equal opportunities for men and women through the
consideration of this priority in all measures and the parallel explicit
promotion of projects to increase female participation.
Overall ambitions of the programme:

To increase the share of women in newly created jobs to 40 percent
(compared to 20-38 percent in various measures in last period)

To aim for a share of at least 40 percent of women in newly created
training positions (compared with 30-55 percent in various measures in
last period)

To increase the share of women in supported new firm formations to 35
percent (compared to ca. 20 percent in individual Land programmes) –
only one-person firms will be included in this calculation

Focus on integrating disadvantaged groups (low qualified, long-term
and youth unemployed, migrants, older workers) in the economic
renewal process and avoiding their potential social and economic
marginalisation
Overall ambitions of the programme

Reduction of the share of social security recipients in particularly
disadvantaged urban areas by two percent compared with the 1990-97
period (compared with a share of between 5-19 percent in urban areas
of the programme with particular renewal needs)

Reduction of unemployment in particularly disadvantage urban areas
by two percent compared with the 1990-97 period (compared with
shares of between 15-30 percent in urban areas of the programme with
particular renewal needs)
16/..
/17.02.16/D:\106752927.doc
- 16 Innovation orientation and
optimal preparation for
the information society

Focus on the promotion of innovation and the information society to
ensure the adequate spread of ICTs through all sectors of the economy
Overall ambitions of the programme:

At least 30 percent of supported projects should use or develop
innovative solutions or methods for which there is no previous model

At least 30 percent of all supported projects should develop or use new
ICTs
Horizontal priorities – Scotland
West of Scotland
Equal
Opportunities
Sustainable
Development
Information
Society

The Programme focuses on supporting a range of actions to assist in the elimination
of inequalities. Supported actions include promoting equal opportunities and reducing
disparities relating to gender, ethnicity, disability or other forms of discrimination.

A dual strategy approach encompasses mainstreaming and positive action.

Detailed guidance is provided to project sponsors and adjustments made to the
project appraisal and scoring system to ensure the application of the approach to
equal opportunities throughout the Programme.

The commitment to sustainable development focuses on the economic, social and
environmental requirements of the region.

Supported actions in each Measure ensure that improvements within the economy
are achieved in a manner that does not have a detrimental affect on the environment
and increases the capacity for the use of renewable and sustainable resources.

Detailed guidance is provided to project sponsors and adjustments made to the
project appraisal and scoring system to ensure the application of the approach to
sustainable development throughout the Programme.

The programme focuses on supporting the demand side of the Information Society,
promoting the adoption of ICT in the region, through e-commerce, e-learning and ecommunities initiatives. Further supported actions encompass the dissemination and
application of new/existing knowledge, processes and technologies to SMEs.

The Programme's focus on the Information Society is reflected in guidance to
partners on project development and implementation and in selection/scoring criteria.

Various measures require that the environmental aspects and implications of a
project are an integral part of the project business plan.

Specific actions such as environmental knowledge and technology transfer,
environmental networks, marketing of green tourism, environmental land
improvements, recycling, waste minimisation, energy efficiency or habitat and
landscape restoration will directly contribute to the environmental objectives.

Targets for sustainable development are expressed as a percentage of selected
indicators and vary from measure to measure.

The support provided within each measure of the Programme especially seeks to
create and safeguard jobs in disadvantaged rural and urban communities.

Most measures anticipate 30% of overall targets will relate to social inclusion criteria
on the basis that there is a maximum of 30% coverage of Programme Area
population in social inclusion target areas. In specific social inclusion target areas,
this rises to 70%.
East of Scotland
Sustainable
Development
Social Inclusion
17/..
/17.02.16/D:\106752927.doc
- 17 Equal
Opportunities
3

There is a requirement for all applicants and benefiting SMEs in the region to
demonstrate an effective commitment to equal opportunity goals and processes in
their recruitment, training, promotion and reward practices.

Aspirational target of 40% female participation set for most indicators with reference
to the proportion of female employment in eligible sectors. 20% target set for
assistance to SMEs where female participation in senior management is generally
lower. 100% of overall target for number of organisations introducing Active
EO/Family Friendly policies relevant to females, ethnic minorities, disabled.

Aspirational target of 1% ethnic minority participation set for most indicators.

Aspirational target of 5% disabled participation set for most indicators by reference to
the proportion of disabled in population. 2% for assistance to SME where disabled
participation in senior management generally lower.
Comparative Elements and Benchmarking Study
3(a)
Evaluation approach – General Concept
The evaluation takes the form of a comparative element to the ongoing mid-term evaluations of
the Objective 2 Programmes in the East of Scotland, Nordrhein-Westfalen in Germany and
Western Scotland and adds value to this process by using the mid-terms as the basis for a
comparative policy evaluation. The evaluation work is delivered alongside the mid-term
evaluation process and therefore includes contextual analysis, stakeholder research, data
gathering, output and result analysis and survey work
The cooperation of the regional programmes and the common benchmarking of selected
horizontal themes, is an innovative component of the NRW mid-term evaluation. At its heart, lies
the aim of stimulating interactive debate and reflection based on programme experience in
different national and regional contexts, with particular reference to the horizontal themes.
The outcome of this process will be practical lessons of relevance to programming authorities and
partnerships in the implementation of the programmes and increased confidence and capacity to
address innovative issues.
There are two elements buildung the frame for the comparative element between the evaluations:
First, two common workshops will be held in the context of the mid term evaluation:

One at the start of the evaluation, in order to agree the work programme and the methodology
(Autumn 02)

One at the end of the evaluation, for presentation and discussion of the results (Autumn 03)
The commissioned consultants and members of the relevant Steering Groups attend the
workshops.
Second, the comparative element will focus on the region‘s approaches to and experiences of
mainstreaming the horizontal themes, allowing for benchmarking of performance and exchange
of good practice. In particular, the benchmarking will look at the horizontal themes of gender
mainstreaming and sustainable development. Building on the results in the regions, the
consultants should use NRW/Scotland as a reference point for benchmarking. For this, it will be
necessary for the consultants to agree on methods and work planning and will work closely with
each other. The co-operative element of the evaluation is novel and innovative. It is likely that, as
the co-operative work progresses, results from the mainstream evaluation will be shared by the
18/..
/17.02.16/D:\106752927.doc
- 18 NRW and Scottish consultants. Where this occurs, and under the guidance of the respective
Steering Groups, a synthesis chapter should be prepared, drawing together and comparing the
most important results of each of the mid term evaluations, in particular those outwith the scope
of the work on the horizontal themes, but drawing together the relationship between the
horizontal themes and the general results.
The integration and implementation of these themes present significant challenges to programme
managers and therefore mutual learning has the potential to be particularly.
The specific agreed outcomes of the benchmarking and cooperation component of the mid-term
evaluation are deliverables in the form of reports and active facilitation of two workshop events.
However, at a more fundamental level, a key outcome will be to design and focus these
deliverables to encourage an active process of dialogue and learning. The ultimate aim is to
promote genuine impacts in terms of the ability of programme actors to successfully meet the
significant and complex demands associated with the Structural Fund programmes and with the
horizontal themes in particular.
By now, a strong and active partnership between the German and Scottish evaluation teams has
been established. Regular contact is maintained between the evaluators to ensure an on-going
exchange on the progress of the respective mid-term evaluations. This widens the potential for
areas of useful comparison to be identified and relevant information to be gathered as part of the
evaluations. . It is proposed that, on this basis, the evaluators will be able to generate integrated
outputs which will present common issues, problems and potential solutions as well as
conclusions of specific relevance to individual programmes. This is also in line with the
maintenance of some degree of flexibility in the comparative study. As it is delivered through the
mid term evaluations in each region, the work is ongoing. By June 2003 it is expected that
preliminary results in all areas can be reported and that process lessons in terms of
benchmarking and co-operation can be highlighted.
3 (b)
Methodology
As already stated, the benchmarking and comparative component of the mid-term evaluations will
focus on the integration and mainstreaming of the horizontal themes of sustainable development
and gender mainstreaming. In addition, a synthesis of the main results of the overall mid-term
evaluations will be undertaken as appropriate. The raw material for the comparative work will be
drawn from the wider evaluation work for each programme and the benchmarking process itself
involves:



reaching a final agreement on the scope of this exercise;
ensuring the wider evaluations generate the necessary information and undertaking
targeted comparative analysis on the results; and
encouraging the dissemination and practical use of the outcomes. Each of these
stages involves the evaluators in an active dialogue with programme actors who
ultimately are at the heart of this exercise.
In practice, the benchmarking exercise will involve a series of stages, some of which have
already taken place:

Inception meetings (NRW on 12 September 2002). This provided an opportunity to
present and confirm the broad principles of the benchmarking exercise and the proposed
content of the Launch Workshop.

Meeting of evaluators (27 September 2002). EPRC (responsible for NRW
benchmarking) met with EKOS (responsible for Scottish benchmarking) to discuss overall
approaches to the benchmarking exercise and the linkages into the wider evaluations. In
19/..
/17.02.16/D:\106752927.doc
- 19 -








addition, issues such as practical coordination and potential input into the Launch
Workshop were raised.
Background report for launch workshop. EPRC/EKOS drafted a background report
stating the comparative context for the study and proposing possible options for the scope
and conduct of the benchmarking exercise. A key aim of this report is to facilitate
decision-making at the Launch Workshop. The background report drew on existing
knowledge about the programmes as well as initial consultation with programme
managers.
On the basis of initial discussions between the evaluators, it is envisaged that the
benchmarking exercise could focus on a number of areas: (i) how the horizontal themes
are communicated to all relevant actors/participants in the Objective 2 programmes; (ii)
the integration of the horizontal themes throughout the processes of programme
implementation from application through appraisal to monitoring and follow-up; (iii) the use
and practical challenges of indicators to identify and track the horizontal themes; (iv) the
scope and content of activities which have been achieved so far in the area of the
horizontal themes; and (v) issues of capacity building in supporting the implementation of
the horizontal themes throughout the programme process.
Launch workshop (in Gelsenkirchen, NRW, 28-29 October). This event brought
together relevant German and Scottish partners and their evaluators for the first time, and
had three aims: (i) to establish the contacts which will be the basis for exchange during
the study; (ii) to agree the scope and methods of the study; and (iii) to initiate the process
of learning and exchange, including through case study visits and initial discussions. A
central outcome of the Launch Workshop will be clarity and agreement on the themes to
be addressed in the benchmarking work.
Revised background report. EPRC/EKOS revised the initial background report to
reflect the agreements reached at the Launch workshop about the specific scope and
aims of the benchmarking. The report is a key document guiding all subsequent activities.
Evaluation. On the basis of the revised background report, the evaluators coordinate
with the wider evaluations teams in NRW and Scotland to ensure that the information
needs of the benchmarking work are being taken into account in the evaluation
methodology. Any necessary adjustments to the scope of the wider evaluation will be
made as soon as possible after the finalisation of the revised background report.
Concluding workshop (Scotland, Autumn 2003) The Concluding Workshop will be
designed to discuss and disseminate the results of the benchmarking/cooperation
element of the mid-term evaluations. Effective dissemination will be important to ensuring
a real value-added from this innovative part of the mid-term evaluations. Close
coordination with respective programme managers will take place to ensure that the right
target groups are included in the Workshop and that the format presents as many
opportunities for active learning as possible.
Benchmarking report. Based on the information drawn from the wider evaluations, a
benchmarking report on the agreed horizontal themes will be written in coordination with
the Scottish evaluator. The format and approach of the report will draw on the proposals
in the revised background report, as well as on-going consultation with programme
managers and partnerships. The report will aim to set individual programmes in a wider
context and also provide experience and lessons which are practically relevant to
programme managers and implementers.
Synthesis report. A synthesis report, drawing together a number of the key wider
conclusions of the NRW and Scottish evaluations, will also be drafted. The exact format
and focus of the synthesis report will be agreed in conjunction with the programme
managers and evaluation teams to ensure that the end result provides a genuine basis for
useful learning.
20/..
/17.02.16/D:\106752927.doc
- 20 The following diagram shows the proposed methodology for the benchmarking study:
Questions
Identification and
agreement on key
themes and
questions
Comparative Analysis
on agreed themes and
questions
Results
Mid-Term Evaluations of NRW, West of
Scotland and East of Scotland Objective
2 programmes
First Workshop,
Gelsenkirchen
October
2002
July 2003
Second Workshop
Feedback and
discussion of findings;
consideration of
dissemination options
September/
October
2003
Synthesis work and
finalisation of
comparative analysis
Final Report
- including agreed
dissemination formats
January/Fe
bruary
2004
21/..
/17.02.16/D:\106752927.doc
- 21 -
3 (c)
The results of the First Workshop
The first Workshop on the cooperation and benchmarking, held in Gelsenkirchen on 28-29
October 2002, kicked off this process. It provided an opportunity for key personnel involved in
the management and implementation of the three Objective 2 programmes to (re)establish
personal contact. The mutual understanding of the respective regional and programme contexts,
critical for establishing the basis for on-going comparison and learning, was be deepened through
presentations and study visits. The Workshop also provided the forum for discussion and
decision-making on the themes and approach of the comparative work, ensuring that a common
agenda is identified and that the relevant actors are engaged in the process from the start.
The Background Report prepared for the Workshop had two primary objectives:
First, to provide basic comparative information on the three programmes. Previous reports have
looked at various comparative themes relating to NRW and Scotland (including the Structural
Funds) and contribute to an understanding of the specific regional settings4. The information in
this report is focused specifically on key elements of the Objective 2 programmes and should act
as a summary guide and a background document for the more detailed regional presentations
given in the workshop.
Second, to put forward initial proposals for the approach to the benchmarking work and report.
These proposals draw on discussions between the evaluators as well as initial consultation with
programme managers in the three regions. They should act as a basis for debate and decisionmaking at the Workshop in the definition of an agreed common agenda and methodology for this
part of the mid-term evaluations.
Outcome of the Workshop Discussion
The proposed approach to the benchmarking study was presented in the Background Report and
discussed at the. The proposal was based on discussions between the two evaluators (Regina
Trenkler-Fraser (EKOS) for the Scottish programmes and Ruth Downes (EPRC) for the NRW
programme) and initial consultation with programme management authorities and horizontal
theme representatives. The discussion at the Workshop provided broad support for the proposal
and included the following main points:
A process-oriented approach was considered the most beneficial to the benchmarking and
cooperation exercise and exhibited the greatest potential for outputs of practical use within
different regional and programme environments.
Linked to the emphasis on process, the main focus of the comparative study was considered to
be issues which could make a difference within the practical implementation of the three
programmes involved. This may also include learning at project level. The aim would be to
generate usable results rather than simply a better knowledge of a different region.
The clarification of definitional issues with regard to the treatment of the horizontal themes in
each programme was considered important. However, this should be included primarily as a
‘scene-setting’ exercise rather than as a central focus of the benchmarking work.
Given the innovative nature of the comparative/benchmarking exercise, it would be useful to
maintain some flexibility in the coverage of the work to allow for amendments or new elements
emerging from the mid-term evaluation results.
4
See, for example, the recent report “Partners for Development/Partner in der Strukturpolitik” (2001)
22/..
/17.02.16/D:\106752927.doc
- 22 With regard to the methodology, the following points were emphasised in the Workshop
discussion:
Regular contact will be maintained between the two evaluators to ensure an on-going exchange
on the progress of the respective mid-term evaluations. This should widen the potential for areas
of useful comparison to be identified and relevant information to be gathered as part of the
evaluations. This is also in line with the maintenance of some degree of flexibility in the
comparative study.
A realistic approach to the benchmarking/comparative work was also underlined. Given the
scope of the comparative component, it will be inevitable that, overall, the mid-term evaluation
results will drive the comparison more than the requirements of the comparison will determine the
approach to the wider evaluations. However, the point was also raised that, should the need and
interest arise, an extension of this collaborative work beyond the bounds of this particular study
could be explored.
In terms of the timing of the Second Workshop, it was considered important that substantive
results were available for discussion. In the light of this, a Workshop in September 2003, to be
held in Scotland, was viewed as a likely option.
3 (d)
Proposed structure of the Benchmarking Study and questions to be
answered
Scope of Benchmarking Study
A benchmarking study in the more traditional sense is potentially too rigid an approach to
accommodate the range of areas of interest expressed by the three regions. Further, given the
mid-term status of programme implementation, it is anticipated that data on the outcomes and
impacts of project activity, required as the base material for a benchmarking analysis, may not be
available in the form and extent required. Feedback from the initial consultation, however,
indicated that some form of direct benchmarking would nevertheless be considered important as
part of the comparative study. One option, therefore, would be to select and focus on a particular
priority or measures where datasets from the mid-term evaluations show the potential for more
detailed comparative and benchmarking work.
Proposed structure of the comparative study
Building on discussions between the evaluators and the initial consultation with the programmes,
the following structure of the comparative study has been agreed. Essentially, the core questions
being addressed in each region and then brought together for comparison are very simple: What
has been done? Why has it been done? How has it been done? And to what effect? The
structure below defines the main components of these questions and the more detailed issues
which could be addressed in answering them.
Context: important contextual issues include (i) definitional clarification and explanation relating to
the understanding of the term mainstreaming and the focus of the horizontal themes themselves
and (ii) establishing the baseline situation for the horizontal themes in each region, including
approaches used to set targets.
Programme Processes: the question of how the horizontal priorities are represented throughout
the process of programming, from the generation, appraisal and selection of projects to their
monitoring and follow-up will be examined;
Communication Processes and Capacity Building: the extent and means through which each
Objective 2 Programme communicates the horizontal themes to their audiences and potential
23/..
/17.02.16/D:\106752927.doc
- 23 applicants will be assessed. An element of this process includes capacity building initiatives
undertaken at various levels of programme implementation designed to strengthen know-how
and effective communication.
Indicators and Monitoring: the range of indicators used in each Programme to measure
achievements against set targets, and their integration and follow-up through the monitoring
system, will be explored. A more detailed benchmarking exercise on specific priorities and/or
measures may be possible depending on the availability of relevant and comparable datasets.
Detailed questions
Within this proposed framework for the comparative study, the following specific questions and
issues could be addressed.
Context
What definitions are applied by each Programme regarding the meaning and scope of the
horizontal themes and the concept of mainstreaming?
What is the key socio-economic context for the horizontal themes in each region?
How was the baseline for horizontal themes established in each programme region?
Which indicators were chosen, and which targets were set?
Programme Processes
How far has the message of mainstreaming gone down the various levels of project
implementation?
Which are the key actors targeted and influenced by the Programmes' mainstreaming objectives?
What is done in practical terms to integrate horizontal themes into decision-making?
How are horizontal themes mainstreamed in standard programme areas such as business
development and infrastructure?
How can a mainstreaming approach to environmental sustainability and equal opportunities
actually encourage better implementation of sectoral policies (i.e. is there a way of encouraging
sectoral ministries to take these issues more into account)?
Whilst the regions have developed relative sophisticated appraisal and project selection
processes, how far have these processes really facilitated the implementation and improvement
of horizontal themes at beneficiary level?
How far has the often encountered attitude ‘horizontal themes have nothing to do with my project’
been overcome, and what sort of techniques have been employed?
To what extent have Structural Funds influenced Government bodies and their commitment
towards mainstreaming of horizontal themes?
And how does this manifest itself (degree of linkage between national Government agendas and
key Structural Funds policies)?
What are the key influences on the approach to implementing the horizontal themes – previous
experience, the framework conditions, sectoral policy structures?
Who is involved in appraising applications and how are these individuals trained/prepared?
What sort of systems in terms of human resources are in place to assess an application’s/
project/s potential impact on HT?
How can relevant selection criteria be selected and put into practice?
How are horizontal themes weighted/assessed in relation to other core indicators during the
appraisal and selection process?
How are issues of multiple disadvantage identified, if a Programme’s equal opportunities theme is
exclusively based on gender?
24/..
/17.02.16/D:\106752927.doc
- 24 Communication Processes and Capacity Building
What sort of consistency has been applied in communicating (awareness raising, training)
horizontal themes at all levels, from top policy level via intermediaries to beneficiaries?
What sort of relationships have been established in the process of implementing and
mainstreaming horizontal themes (training, capacity building)?
What techniques are used to influence change in beneficiaries?
How can the simple presentation of information for purposes of securing funding be avoided?
How is good practice defined and identified?
What sort of pro formas are applied in the pursuit of good practice in
implementing/mainstreaming horizontal themes?
How is good practice communicated in a Programme and across Programmes?
Are there any specific techniques used to facilitate the successful implementation of horizontal
themes (project sponsor and beneficiary/ levels), particularly with regard to encouraging SMEs to
become more aware and active?
What are the most effective communication techniques to raise applicants’ awareness levels?
Indicators and Monitoring
How is the implementation of horizontal themes monitored? Are horizontal themes integrated in
the overall monitoring framework, including monitoring visits?
How is the impact of horizontal themes on beneficiaries measured? And is reality captured, or
not?
Are there any tools regarding monitoring and tracking of HT, particularly with regard to equal
opportunities indicators and targets? Which monitoring tools are currently employed?
What sort of data-handling problems regarding HT exist? And what has been done to overcome
them?
When is mainstreaming regarded as success? When is the objective of mainstreaming perceived
as achieved in each Programme?
Benchmarking: depending on availability of relevant datasets and the comparability of established
baseline situations, the study could identify one programme measure and benchmark its
achievement levels against a selected set of horizontal indicators in the respective programmes.
This could include relevant project level comparisons.
Additional Areas of Potential Interest
Innovation: How important is its role in creating lasting impact, durability and sustainability whilst
facilitating positive change in the overall development of the regions? How much is risk
avoidance in Structural Fund Programmes hindering innovation to happen?
Social Economy/Social Enterprise: How sustainable is it and what relationship exist between
social economy projects and HT?
What is the long-term sustainability of HT? How is it measured and assessed?
Next Stages
The evaluators will ensure that the issues and questions identified as important for the
benchmarking study are reflected, as far as possible, in the methodologies of the mid-term
evaluations in each of the three regions. The differences between both the programmes and the
methodological approaches to the evaluations will inevitably result in some disparity in the way in
which the issues are addressed. On-going, regular contact between the evaluators, facilitated by
their location in Glasgow, will try to ensure that key issues of comparison are covered as
comprehensively as possible throughout the evaluation process.
/17.02.16/D:\106752927.doc