Evaluation
within the context of the intervention
of the Structural Funds
in Portugal
IFDR
1 October 2008
General contents of the presentation
 A.
Fundamental issues for
evaluation of public interventions
the
 B. Evolution of evaluation processes
over the CSF
 C. Evaluation in the NSRF
Evaluation
Fundamental issues for the evaluation of
public interventions
Fundamental issues for the evaluation of
public interventions
Aspects to take into consideration in the evaluation:

Relevance – Evaluation of the closeness of the strategy to reality

Efficiency – Evaluation of the way in which resources were
transformed into output and results

Efficacy – Evaluation of the way in which the resources contributed
towards realising the objectives

Utility – Evaluation of the benefits for recipients

Sustainability – Evaluation of the extent and durability of the
alterations made
Fundamental issues for the evaluation of
public interventions
Relevance
Evaluation of the closeness of the strategy to reality
Fundamental issues for the evaluation of
public interventions
Efficiency
Evaluation of resources and its outputs
Fundamental issues for the evaluation of
public interventions
Efficacy
Evaluation of the way in which resources were
applied
Fundamental issues for the evaluation of
public interventions
Utility
Evaluation of the benefits for recipients
Fundamental issues for the evaluation of
public interventions
Sustainability
Evaluation of the extent and durability of the
alterations made
Principles of Evaluation
– Proportionality
– Partnership
– Independence
– Transparency
Principles of Evaluation
Principle of Proportionality
Financial and administrative resources of evaluations
Proportional to the total amount of the expenses allocated to the OP
Principle of Partnership
Cooperation of the major agents throughout the planning process
and implementation of evaluations
Principles of Evaluation
Principle of independence
Evaluation exercises by entities (internal or external to the
Public Administration) functionally independent of the
Management Authorities and of the entities with responsibilities
in the NSRF and of the OP
Principle of transparency
Disclosure of the main results of the evaluations, as well as
their use as a resource for the qualification of public debate
ACCOUNTABILITY
General contents of the presentation
A. Fundamental issues for the evaluation of the
public interventions
B. Development of Evaluation processes over
the CSF
C. Evaluation in the NSRF
Evaluation throughout the CSF
1.
Evaluation in the CSF I (1989-1993)
2.
Evaluation in the CSF II (1994-1999)
3.
Evaluation in the CSF III (2000-2006)
CSF I Evaluation
1989-1993
CONCEPTUAL MODEL
Institucional and Functional independence between Management & Evaluation
Different Structures
Political
Decision
Makers
Observatory of CSF I
methodologies
 Ex-ante, ongoing e ex-post Evaluations
 In collaboration: Commission - MemberState
 Monitoring and evaluation functionally &
institucionally independent from Management
 Independent Experts
 Ex-post Evaluation
 “IFDR” Coordination
Terms of
Reference
CSF II Evaluation
1994-1999
Main changes:
 Increasing importance of the ex-ante
appreciation and ex-post evaluation
 Evaluations were carried out by each
Operacional Intervention.
 The Commission encouraged the expost evaluation of the CSF II.
CSF II Evaluation
Ex-post evaluation 1994-1999
Main conclusions:
 High performance
 Around 7.7% of the GFCF in the period 1994-2000 was directly
induced by the CSF II
 Around 77.000 jobs were created and maintained up to 1999
directly induced by the investments made in the CSF II
CSF II Evaluation
Ex-post evaluation 1994-1999
Main Recommendations
 Changes into the Information System
creation of a single Information System
Incorporating all of the Funds.
 Improve the efficacy of the programmes
implementation of a system of
management by objectives
 Improve monitoring functions
 Separation between the activities of project control and monitoring
Control
≠
Monitoring
CSF III Evaluation
Evaluation Phases (2000-2006)
Ex-ante Evaluation
 from the Member State responsibility
 carried out by independent evaluators
Diagnosis
of
the
departure situation and
analysis of the expected
results.
Mid Term Evaluation – December 2003
 carried out by independent evaluators - responsibility of the MA
Analysis of
 allocation of the Performance Reserve
Mid Term Evaluation Update – December 2005
first
results
of
the
interventions,
taking
into account the ExAnte Evaluation.
Ex-post Evaluation – 3 Years after programming period
 carried out by independent evaluators - responsibility of the EC
the
CSF III Evaluation
3 different moments of evaluation
Ex-ante
Mid Term Review
Ex-ante evaluation
Important
input
Ex-Post
= basis for the
preparation of the interventions
* being the responsibility of the Member
State
Checks
strategy and objectives
coherence
execution and monitoring rules planned
results of the evaluations
CSF III Evaluation
Mid Term Review
Enables:
 the knowledge and analysis of the first results
 the pertinence and the implementation of the objectives
 the follow-up of financial allocations
 the functioning of the monitoring and the execution of activities.
Each OP
CSF
* being the responsibility of the Management Authorities in collaboration
with the Commission
CSF III Evaluation
Ex-post Evaluation
 Independent evaluators
 3 years after end programming period
 Commission in collaboration with Member State
Intended to report on achievements and
effects, the use of resources, the efficacy
and efficiency of the OP, The analysis of the
causes of inefficacy, the impacts
CSF III Evaluation
Mid Term Review
Included
 results of the evaluation of macro-economic impacts
through the analysis of different reports
 cross-referencing them with the results of the
evaluation studies
 Performed by independent evaluators – until Dec 2003
Recommended
Changes in OP
Mid Term Review
redefinition
priorities
of
strategies
and
redistribution of financial allocations
Funds
OP
Priorities
CSF III Evaluation
Mid Term Review
Performed by independent evaluations
until Dec 2003
Changes suggested
Intermediate reprogramming of 2004
Update of evaluation in 2005
Preparation of the subsequent interventions
indicating priorities for the following
programming period
CSF III Evaluation
Mid Term Review
quality of the evaluation
exercise
attribution of the efficiency
reserve
results of the Mid Term Review
4% of the total
allocations planned
for each Member
State
To reward the more
efficient OP
Reprogramming of 2004
CSF III Evaluation
Attribution of the Efficiency Reserve
Stage 1
 Selection of the OP which meet the criteria
Stage 2
 Determination of the amount of the efficiency
reserve to be awarded
CSF III Evaluation
Efficiency Reserve
Criteria for all OP - Considering the fulfilment by the OP of good
 Management System Quality
Quality of the system of control
Quality of the selection criteria
Quality of the evaluation system
 Global financial execution
 Financial execution in LVT
 Leverage effect
Efficacy
criteria
Financial
criteria
Management
criteria
management practices
 Measures the effects of the Funds through output
of result indicators
CSF III Evaluation
Attribution of the Efficiency Reserve
Common
Management
Criteria
Financial Criteria
Efficacy Criteria
 Very efficient: 5 of 6 criteria
 Efficient: 4 of 6 criteria
 Not Efficient: 3 criteria
 Very efficient: reaches 90% of targets
 Efficient: 75 – 90% of targets
 Not Efficient: < 75%
Very efficient: at least 80% of output & result
Indicators
Efficient: at least 60% of output & result
Indicators
Not efficient: < 60%
CSF III Evaluation
Results of The Mid-Term Review
Main Conclusions – 2003
 Relevance of the European Strategy For Employment
Reduction of unqualified employment
Reduction of early school leavers
Integration of education investment in companies
Modernization of Public Administration
Focus on Telecommunications and Energy (with direct impact of the
Portuguese economy)
CSF III Evaluation
Results of The Mid-Term Review
Main Conclusions - 2005
 Slow but progressive increase in the qualification of
labour supply
Reduced expression of entrepreneurship in Portugal
High level of performance of OP in terms of approval,
without perfect correspondence in terms of financial
execution
CSF III Evaluation
Mid-Term Review
Main Conclusions for 2007 – 2013
Integration of policies
of innovation and
productivity
Development of
territorial
competitiveness
Active employment
policies bringing them
closer to the regional
and local dimensions of
structural
unemployment
Develop articulation of
institutions and
organizations as to
avoid losses in the
efficiency
Improve global
coordination
Analysis of the evaluation exercises
In the whole period 2000 – 2006
133 evaluation studies
39 = evaluations of an obligatory nature
One significant investment – but only 3% of the global
financial allocations of technical assistance of the CSF III
44% of the financial resources were intended for
evaluations of an obligatory nature
Analysis of the evaluation exercises
Evaluation studies of an obligatory nature
Mid-Term Review
Follow up of Mid Term-Review
19
20
4 086
2 293
N.º of
studies
Total Cost l
(1000 €)
o 15
Institutions
o 1
Consultant Companies
Public Administration
o 3 Research Universit. Centers
20
o 15 Consultant Companies
Consultant Companies
o 1 Public Administration
o 3 Research Universit. Centers
Analysis of the evaluation exercises
Evaluation studies of an obligatory nature
Strong points
 Overall quality: credibility and utility
 Significant progress on analysis of implementation processes; global
rationality of the interventions; quantification of outputs;
 Institutional involvement and use of the evaluation as a support tool
for decision making
Weak points



Rigid calendars
Exaggerated scope
Weaknesses in quantification or estimate of impacts
Analysis of the evaluation exercises
Evaluation studies of an obligatory nature:
Recommendations
Good practices of
 Generalization of exercises of this nature in management
practice
 Flexibility and subsidiarity
 Disclosure
General contents of the presentation
A. Fundamental issues for the evaluation of
the public interventions
B. The evolution of the evaluation processes
throughout the CSF
–
–
Evaluation in the CSF I (1989-1993) and CSF II(19941999)
Evaluation in the CSF III (2000-2006)
C. Evaluation in the NSRF
Evaluation in the NSRF
2007-2013
Short presentation of the NSRF
Strategic
purpose:
EDUCATION and QUALIFICATION
Through:
 knowledge
 science,
 technology
 innovation
 promotion of high and sustained
levels of economic and sociocultural development and
 of territorial qualification
Structural Funds and
Cohesion Fund
Equal opportunities.
Increased efficiency and quality of
public institutions.
Promoted by
Operational Programmes of three large Thematic Agendas
Short presentation of the NSRF
EU Strategic
Guidelines
Growth and
Employment
Renewed Lisbon
Agenda
National
Public
Policies
O
U
T
P
U
T
S
National
strategic goals
and priorities
NSRF
OP
Short presentation of the NSRF
Thematic Operational Agendas
Human
Potential
NSRF
Operational
Programmes
Factors of
Competitivity
Enhancement of
the Territory
Short presentation of the NSRF
Articulation of instruments of national public intervention
ENDS
PNACE
PNAC
Territorial
Development Plans
Technological Plan
National Strategy
For Energy
National
Employment Plan
NSRF
National Nature and
Biodiversity Conservation
Strategy
National Strategy
for the Oceans
National Programme
for Inclusion
National Coastal
Zone Integrated Strategy
National Plan
for Equality
Cultural Development Plan
PNPOT
Autonomous
Regions Plans
National Water Plan:
Hydrographic Basins and
Efficient Use of Water
PEAASAR
National Strategic
Plan for Tourism
Nat. Strat.
Rural Develop.
Plan / FEADER
National Waste Plan
Nat. Forestry Plan
Cities Policy
Characteristics
OBJECTIVE
Evaluation
To improve the quality, efficacy, efficiency and coherence of the
intervention of the funds and of the strategy and execution of the
OP.
TIMING
Before the start of the
programming period
Ex ante
During the programming
period
On going
After the programming
period
Ex post
Characteristics
Evaluation
NATURE
Strategic
Operational
RESPONSI
BILITY
Environmental
Strategic
Analysis of the evolution of an
OP or group of OP in relation
to the community and national
priorities
To support the monitoring
of an OP
Evaluation of the effects of
the Interventions on the
Environment
EC
Ex-post
MS
Ex-ante
On-going
Institutional Framework
Evaluation – Responsibility
Institutional Framework
On-going Evaluation Responsibility
Strategic Monitoring
Financial and
Operational Monitoring
Management, Evaluation,
Internal Control
Evaluation Plan
 To ensure the efficient use of the Evaluations
 To
ensure that the evaluations are integrated and
considered as a management tool during the
implementation of the OP
 Nature of the evaluations
 Description of the coordination and articulation mechanisms
 An indicative list of Evaluation exercises
 Foreseeable calendar
Covers the whole programming period
Evaluation Plan
Includes
 Strategic evaluations
 Operational evaluations
Level :
 NSRF
 OP/Fund
Parts:
I - Coordination
II – Evaluation activities and reports
From the Mid Term Review
to the
on-going Evaluation
Regulation no. 1083 allows for a NEW CONCEPT of
Evaluation:
 More flexible
 In line with the needs of the policy decision-making
process and with the more efficient management of
resources
From the Mid Term Review
to the
on-going Evaluation
More
Flexible
By Topic
By OP
By Priority
By Actions
Major projects
For all OP
For one OP
At NSRF level
From the Mid Term Review
to the
on-going Evaluation
Two closely related concepts, but with different
objectives and functions
Monitoring
analysis, monitoring and verification of
results
Evaluation
analysis and interpretation of the information
obtained through the monitoring, and other sources
in order to find and explain the effects of the
interventions.
From the Mid Term Review
to the
on-going Evaluation
Monitoring
Evaluation
Provides quality information and
respective analyses
Helps the decision-making
process
From the Mid Term Review
to the
on-going Evaluation
Regular monitoring
Alerts for the need for
Evaluation
Regular follow up of
the Evaluation
Ensures implementation
of reccommmendations
Evaluation in the NSRF
Practical applications
Reflections of the Evaluation of the CSF III on
the NSRF
CSF III
NSRF 2007-2013
Insufficient
concentration
of
financing options on the key areas
corresponding
to
the
major
development problems of the
country
Reduction
in
the
no.
of
operational
interventions
More refined criteria in the
prioritization of projects
On-going
strategic monitoring
mechanisms
Insufficient strategic alignment of
Consolidation
of
rationality
operational instruments and of
centers
projects supported
Articulation between strategic
objectives and financing models
Reflections of the Evaluation of the CSF III on
the NSRF
CSF III
NSRF 2007-2013
Dissemination of good practices,
Difficulty
in
making
the monitoring systems, Evaluation and
innovative potential of agents benchmarking,
inter-institutional
emerge
coordination
and
innovative
approaches
Notion of Strategic State:
Insufficient attention to bolstering
Strategic planning
the institutional capacity of the
Inter-sectorial coordination
Public Administration
Monitoring and Evaluation
Quality
of
the
expense:
Insufficient focus on the quality
Relevance of the investments and
of the effects, efficiency and
economic and financial sustainability
sustainability
of
co-financed
Results to be achieved and the effects
operations
this will bring
SWOT Analysis of the Ex-ante Evaluation of the
NSRF
What is it?
It defines a strategy through the relationships existing
between the strong and weak points with the most
important trends noted in the global external context,
whether due to the economy, legal impositions, etc.
SWOT Analysis of the Ex-ante Evaluation of the
NSRF
What is it?
The term SWOT comes from the initials of the words:
 Strengths
 Weaknesses
 Opportunities
 Threats
SWOT Analysis of the Ex-ante Evaluation of
the NSRF
Strategy
SWOT Analysis of the Ex-ante Evaluation of the
NSRF
What is it?
Opportunities
Threats
Limiting exogenous
factors
(external analysis)
Strong Points
Weak Points
Existing applications
On-going dynamics
(internal analysis)
For a good strategist there are no threats, only opportunities
SWOT Analysis of the Ex-ante Evaluation of
the NSRF
How?
Build a table with the four elements: with the strong and
weak points on one side and the opportunities and threats
on the other
Strong
Points
Weak
Points
Opportunities
Threats
Identify the key elements which help to establish priorities and
take strategic decisions
SWOT Analysis of the Ex-ante Evaluation of
the NSRF
Analysis of the opportunities and threats which confront the
development process, as well as of the strengths and weaknesses of the
Portuguese situation.
Opportunities
Threats
Broadening of the process of integration of Iberian economies
Financial
difficulties in realising a
Completion of the High Speed Lisbon – fundamental project for the internaMadrid project
tional connectivity of the Portuguese
economy – high speed trains
Extending the market ado to companies
which until now have been more
focussed on the domestic market
(potentially interesting for Portuguese
SME in industry and services), within the
space of proximity which is the Spanish
economy
Growing affirmation of Spain as an
European force, affecting the balanced
development of Luso-Spanish trade and
the maintenance of national decisionmaking centres in strategic sectors
SWOT Analysis of the Ex-ante Evaluation of the
NSRF
Strengths
Weaknesses
Environment and natural heritage
Generalised
coverage
of
the Insufficient levels of service in basic
population in terms of water supply
areas with emphasis on the drainage
and treatment of waste waters
Great diversity of natural heritage with
high conservational value; 22% of the
national territory is calssified with the
status of nature protection and
conservation
Insufficient
protection
and
enhancement of natural heritage,
frequently associated to the lack of
knowledge for the management of
protected species and habitats
SMART Criteria in the preparation of the
Indicator File
What is it?
The definition of the objectives may follow SMART criteria
Specific
Measurable
Attainable
Realistic
Timely
SMART Criteria in the preparation of the
Indicator File
What is it?
Specific
They should clearly define what is intended

What?

Why?

How?
SMART Criteria in the preparation of the
Indicator File
What is it?
Measurable
If it cannot be measured, it cannot be managed!!
The objectives should be quantifiable
SMART Criteria in the preparation of the
Indicator File
What is it?
Attainable
The objectives should be achievable
SMART Criteria in the preparation of the
Indicator File
What is it?
Realistic
Objectives should be defined taking in to account the effort
required to achieve them
SMART Criteria in the preparation of the
Indicator File
What is it?
Timely
Deadline - clarifies when it is intended for
SMART Criteria in the preparation of the
Indicator
File
Indicator File
S
Identification
M
A
R
T
Planning and
Execution
Characterization
Appreciation of the Quality of the Evaluation
Mid Term Evaluation CSF III
Qualitative Appreciation
Criteria
Excellent
1. Information needs are satisfied
2.Pertinence of the scope of the
Evaluation exercise
3. Appropriate character of the
Methodology
4.Reliability of the Data
5.Validity of the Analysis
6.Credibility of the Results
7.Impartiality of the Conclusions
8.Utility of the Recommendations
9.Clarity of the Report
Good
Sufficient
Unacceptable
Thank you for your
attention!