Slovak Society for Evaluation
Monitoring and evaluation
of Cohesion Policy
Austrian experience and
perspectives
Bratislava,
November 30th 2010
R. Hummelbrunner, Graz
Austria
Regional and Cohesion
Policy in Austria

Key features of regional policy in Austria
 Regional policy is competence of “Länder” (regions - NUTS II)
 National level: coordination, orientation (Federal Chancellery)
 Mechanism for aligning regional policies at national level (ÖROK)

Cohesion Policy – Structural Funds in Austria
 8 OPs for Objective 2 at NUTS II level, 1 OP for Objective 1 – Phasing
Out (Burgenland), 7 OPs for Objective 3 (ETC) - CBC strand
 Programmes at national level for Employment and Rural Development
 No other thematic / sectoral OPs at national level
 Central Monitoring System for all ERDF Programmes
 Austrian Conference for Spatial Planning (ÖROK) functions as network
of MAs for ERDF Programmes
Financial Resources 2007-13
Objective
Convergence / Phasing Out
of which ERDF
of which ESF
Regional Competitiveness and Employment
of which ERDF
of which ESF
Territorial Co-operation
SF in Mio €
Current prices
177,1
125,0
52,1
1.027,3
555,0
472,3
256,6
of which cross-border Co-operation
223,9
of which trans-national Co-operation
32,7
Total:
1.461,1
Source: ÖROK
Co-ordination / exchange of
evaluations at national level
Platform Evaluation (ÖROK)
Cross-programme exchange, horizontal themes
Evaluation
Team
Evaluation
Team
Evaluation
Team
Evaluation
Team
Contracts
Operational Programmes
Evaluation
Team
Key innovation during 2000 – 2006:
On-going evaluation
 Voluntary activity of various ERDF Programmes
 Objective 1 Programme Burgenland, five Objective 2 Programmes, four
INTERREG IIIA Programmes, URBAN II Graz
 Period between MTE and MTE Up-date (2003 – 2005)
 Implementation of on-going evaluations
 Integral element of evaluation contracts by MAs
 Specification and steering mostly by Working Group at programme level
 Focus: Result oriented (impact assessment) and/or process oriented
(understanding of implementation and framework conditions)
 Documentation: Internal papers, minutes of workshops/meetings,
contributions for Annual Reports
 Exchange / reflection on process and results of on-going evaluations at
national level (ÖROK)
Evaluation components in Austria
Programme period 2000-2006
Mid-Term Evaluation & Up-date embedded in on-going evaluation process
2000 - 2001 - 2002 - 2003 - 2004 - 2005 - 2006
MTE
On-going
Up-Date.
MTE
On-going
Purpose
External needs
Internal needs
Requirements
Reg.1260; EC-WP
Managing Authority
Timing
Set deadlines
continuous
Focus
primarily result-oriented
Result & Processoriented
Relevance of evaluation for
steering (actors, functions)

Mid-Term Evaluation and Up-date
 External requirement, timing and topics pre-defined
Relevant for:
 MAs (e.g. use of resources, n+2, achievement of targets)
 Monitoring Committee (e.g. Programme changes, reallocations)

On-going Evaluation (between MTE and Up-date)
 Utilization-focused: Information needs/questions of programme actors
 Quick results for programme modifications, quality improvement
Relevant for:
 Implementing Bodies at Measure level (e.g. funding conditions, project
selection and development, publicity and visibility of measures)
 MAs (e.g. implementing mechanisms, results and impacts)
Experience with On-going
Evaluations 2000 – 2006
 Value-added of On-going Evaluation
 External view on programme implementation, „External mirror“ for MAs
 New role of evaluators: „Coaching“ of programme actors in view of
professionalising the entire programme system
 In-depth treatment of selected topics / areas
 Accumulation of insights into programme realities
 Processing of information in small portions, easier to handle
 Quick and flexible reaction on questions / problems of MAs
 Rapid implementation of findings / recommendations
 Timely reaction and adjustments at programme or measure level
Innovations 2007-13:
Strategic governance process

Governance process for NSRF (strat.at) and nine Objective 2 OPs, to
 Promote learning and exchange of experience on OPs
 Provide ideas and practical knowledge for various stakeholders
 Generate contributions for future regional policy (EU, national )

Based on established networks and structures
 Network of Managing Authorities (within ÖROK)
 Steering Group (on-going definition of thematic priorities, work plan)

Main elements of governance process
 Work commissioned by ÖROK (evidence, expertise)
 Workshops to discuss/validate expert inputs
 Events for wider public (dissemination, information)
Outline of strategic governance
process in Austria
Community
Strategic Guidelines
Objectives and Strategy
(Sub)objectives per Priority
Strategy fields
NSRF
STRAT.AT
GOVERNANCE PROCESS NSRF
(= M&E, interactive exchange, Reflection >
institutional learning)
Operational
Programmes
- Priorities
- Actions
- Mechanisms
Analysis
Contributions
to objectives
PROGRAMME GOVERNANCE
(= Achievement of objectives, regional impacts)
Inputs
(Resources, Rules)
Outputs
(Projekts)
Use of outputs
Results
External influence
Regional
Impacts
Territorial framework conditions (physical, socio-economical, institutional)
Innovations 2007-13:
Cross-programme activities

Analysis of common themes / challenges
 Example: Assessment of Structural Funds implementation system in
Austria („Governance Check“)

Joint reflection and learning
 Example: 15 Years of INTERREG / ETC programmes in Austria ‘Looking back - looking ahead’

Comparative implementation analysis
 Example: Cross-programme evaluation of national funding scheme
for environment / climate change

Participation in larger scale cross-programme evaluations
 Example: Cross-programme evaluation of ETC programmes in
Central and Southeast Europe, Operational and Thematic aspects
Innovations 2007-13:
Process Monitoring of Impacts

What is Process Monitoring of Impacts?
 Tool for impact-led monitoring of an intervention
 Focus on processes, which should lead to impacts
 Use of Logic Models (impact diagrammes)

What is it for?
 Illustrates causal relations between programme outputs and effects
in a plausible manner
 Enables a concise, systemic overview of impact patterns of
complex interventions, e.g. programmes
 Provides early information for programme management (MAs, IBs)
required for effective steering
Monitoring of change processes
 Impact chains as basic scheme
Outputs are used
(by someone, in a
specific manner)
to reach results
Inputs are used to
produce outputs
(= projects)
Inputs
Financial Table
Commitments
Disbursements
Outputs
Results will lead to impacts
(intended / unexpected) in
a plausible manner
Results
Typology of projects
(output) to be
supported
Objectives at the level
of Areas of intevention /
Activities
OutputIndicators
Resultindicators
Impacts
Objectives at the
level of Priority or
Program
Impact diagramme AoI A 4.1 –
Softmeasures to strengthen
innovations in entreprises
Advise for co-operation
AF6
Enterprises (especially SME)
collaborate in networks (also with
large enterprises)
Enterprises (especially SME
carry out reorganization
processes
Enterprises (especially SME)
carry out product finding
processes
External expertise,
process consulting
AF6
Enterprises (especially SME)
introduce new technologies
Enterprises (especially SME)
gain new markets
Services to sensitize for
innovations
AF6
Enterprises (especially SME)
conceive innovation-/ investment
projects
Increased linkages between
enterprises at regional level /
scale
Sustainable stabilization of
enterprises
Adaptation to international
competition
New / improved products,
services and production
processes
Increase in employment /
new jobs
New contacts with clients/ new
orders
Pilot applications with programmes
Reg. Competitiveness 2007 - 2013
 Programme Styria
 Based on impact diagrams established during programme preparation
 Application with 5 Areas of Intervention (of totally 11)
 Project sample (used for up-dating of impact diagrams): 70
 Projects approved (used for weighting of impact chains): 840
 Duration of pilot application: July - November 2009
 Programme Carinthia
 Internal assessement of projects approved so far (ca. 40) based on
impact diagrams (dating from ex-ante evaluation)
 Assessment accompanied externally
 Finalization (Report) October 2009
Application with programme
Styria 2007 - 2013

Procedure / steps
1. Up-dating impact diagrams from the programming process (based on
current version of internal „Program Complement“ )
2. Attributing all (840) projects in line with their primary impact chain
(basis: short descriptions contained in ERDF Monitoring system,
information provided by IBs)
3. Weighting of impact paths according to number of projects and
financial volume, calculation and graphic representation
4. Integrating indicators (from ERDF Monitoring database) to provide
information on final achievements of processes
5. Validating and complementing impact hypothesis through interviews
with (selected) project owners
6. Discussing findings / conclusions with IBs (Funding Authorities)
Impact diagramme AoI 4.1 –
Softmeasures to strengthen
innovations in entreprises
2
Advise for co-operation
AF6
Enterprises (especially SME)
collaborate in networks (also with
large enterprises)
Increased linkages between
enterprises at regional level /
scale
40
Enterprises (especially SME
carry out reorganization
65
processes
25
Adaptation to international
competition
Enterprises (especially SME)
carry out product finding
processes
143
40
External expertise,
process consulting
AF6
2
Services to sensitize for
innovations
AF6
Enterprises (especially SME)
introduce new technologies
Sustainable stabilization of
enterprises
New / improved products,
services and production
processes
Increase in employment /
new jobs
9
Enterprises (especially SME)
gain new markets
Enterprises (especially SME)
conceive innovation-/ investment
projects
New contacts with clients/ new
orders
Impact diagramme AoI A
4.1 –
Softmeasures to strengthen
innovations in entreprises
Weighting according to financial volume (public funds)
127.900,00
Advise for co-operation
AF6
Enterprises (especially SME)
collaborate in networks (also with
large enterprises)
Increased linkages between
enterprises at regional level /
scale
352.997,53
Enterprises (especially SME
carry out reorganization
processes
789.269,49
Enterprises (especially SME)
carry out product finding
processes
3.620.240,85
1.652.266,92
External expertise,
process consulting
AF6
436.771,66
Sustainable stabilization of
enterprises
Adaptation to international
competition
New / improved products,
services and production
processes
Increase in employment / new
jobs
Enterprises (especially SME)
introduce new technologies
1.084.671,36
439.728,68
Services to sensitize for
innovations
AF6
Enterprises (especially SME)
gain new markets
Enterprises (especially SME)
conceive innovation-/ investment
projects
New contacts with clients/ new
orders
Pilot applications with programmes
Territorial Co-operation
 INTERREG IIIA programmes (2000-2006)
 On-going evaluation Austria – Slovenia
 Ex-post evaluation of Swiss participation in INTERREG III
 ETC Cross-border Co-operation Programmes (ex-ante evaluation)
 Austria – Slovenia
 Austria – Bavaria
 Lake Constance (AT / DE / CH / LIE)
 ETC Trans-national Co-operation Programmes (ex-ante evaluation)
 Central European Space
 South East European Space
 Studies / Work for INTERACT downloadable at www.interact-eu.net/mt
Results and experience
gained so far


Main results of method:
 Sound and quantifiable connection between supported projects and
effects at programme level (expected results, impacts)
 Systematic capture (respectively clarification) of project types and
categorisation of their use dimensions
 Capture and representation of linkages (within and between Areas of
Intervention, with other programmes / interventions)
 Focus on supported project types and their use, thus early indication of
likely achievement of impacts
Limits of the method:
 Provides assumptions on impact paths, but does not capture the actual
achievement of results / impacts
 However, they can be integrated (once achieved), e.g. via data input from
ERDF monitoring system, surveys, project reports
Evaluation of OPs
2007 - 2013
 National Evaluation Plan 2007 – 2013 (gradually specified)
 Preparation by ÖROK, imbedded in Governance process of NSRF
 Combination of considerations of MAs and national-level actors
 Considerations by MAs for on-going evaluations
 Utility for MAs in implementing their tasks (e.g. Annual Reports,
justifications for programme changes)
 Strategic assessments (e.g. achievement of objectives/results, mid-term
reviews)
 Operative assessments based on data from Central ERDF Monitoring
System (often done internally)
 Work commissioned ad-hoc, programme specific approach
Thank you for your attention
Richard Hummelbrunner
ÖAR - Regionalberatung
Alberstrasse 10,
8010 Graz, AUSTRIA
T: +43/316/31 88 48
hummelbrunner@oear.at
www.oear.at