3nd Steering Committee
Meeting
Malta, 28/6/2006
FUTURREG Evaluation
Objectives and
methodology
1
 Evaluation concepts
 Objectives of the Futurreg Ongoing Evaluation
 Theory Background
 Evaluation Methodology
 Evaluation activities
2
The evaluation is a systematic and
objective process which values the
relevance,
efficiency
and
effectiveness
of
the
policy,
programmes and projects in
relation with the established
objectives (Fahernkrog et al 2002).
The evaluation has to carry out five levels of assessment:
Relevance and Coherence
Cost and Benefit, Efficiency
Effectiveness
Utility
Sustainability.
3
Relation of the evaluation process
Utility /
Sustainability
Effectiveness
Efficiency
Relevance
Actual/Future
Needs
Objectives
General/Spe
cific
Input/Cos
t
Activit
y
Outputs
Results
/Benefit
Impact
Internal Coherence
4
Why to Evaluate?
 Control and Accountability – what have
been done with all the money?
 Knowledge about results – what are the
effects of the activities and measures? Did we
really achieve what we wanted to achieve?
 Learning approach – what went well, which
things went wrong, why did we (not) achieve
the objectives? How can we improve working
in the future?
 Building link between involved actors
5
ISSUES FOR FURTHER DEBATE
• Institutional Learning in project and programme
management is necessary to exploit fully the potential
of evaluations.
• Training and awareness-raising for both, project and
programme managers and evaluators.
• Integrate Evaluation into the project-cycle,
internalise the monitoring and evaluation function.
• Free evaluation from the stigma of being an
instrument for control and punishment.
6
building links
Regional foresight capabilities
Suggestion for
Implementation
improvement
and impact
Learning
approach
Taking desitions
& policy makers
Goal control and
follow-up
Triangle of Futurreg evaluation
functions
7

to provide information and recommendations in
order to improve the Futurreg project and to
contribute to the development of a regional
foresight culture.

Evaluation in conformity with the provision under
Interreg IIIC normative

To evaluate the relevance and coherence of
Futurreg project.

To evaluate the utility
Futurreg as a project.

To derive practical lessons and realistic
recommendations which are important for the
dinamization of foresight capabilities at regional
level.
and
sustainability
of
8
The Foresight can be defined as
a
systematic,
participatory,
future intelligence gathering
and
medium-to-long-term
vision-building process aimed
at present day decisions and
mobilising joint actions.
(FOREN, EU, 2002)
9
 Complex Subject and Project, wide
variety of stakeholders.
 Since the project is still running, it is often
not possible to identify or examine final
results or impacts.
 Limited visibility and sometimes indirect
effects of Futurreg actions on final
beneficiaries (SMEs, enterprises), leds to
difficulties integrating their perspectives in
the analysis to the same extent as the
points of view of intermediaries.
10
Evolution of Innovation policy
Main Objectives
Basic sciences
Key Technologies
Innovation
Social
Futures&
Foresight
Industrial
Military defense
1945
1970
2000
2020
Caracostas y Muldur (1998)
11
Evolution and foresight: interrelations
- Impact assesment
Policy makers’
- technology assesment
environment
Valovirta ja Hjelt (2005)
12
Evolution and foresight: interrelations
Policy makers’
environment
Valovirta ja Hjelt (2005)
13
• Development of Evaluation Methodology
• Evaluation criteria for
applications/implementation projects
• Development of indicators to evaluate
each component of the project
• Development of a questionnaire for the
partners
• On-going evaluation
• Questionnaire of satisfaction for the actual
users of the application
• Interviews with the regional actors benefit
of the project
• Ex-post evaluation report
14
• Methodological tools for evaluation:
– Desk research (background literature, work
programmes, project implementation reports,
former foresight projects evaluations, studies,
statistics, etc.)
– Work research (Interviews with partnership
and select stakeholders at regional level, Online Questionnaire).
– Permanent relationship with partner (meeting,
e-mail, phone). Contrast dynamics and
consent.
15
• Evaluation Framework of the project:
Evaluation
Topics
Evaluation Questions
Main
Additional
Indicators
Data
Sources
Data
gathering
(Methods)
Relevance
Effectiveness
Efficiency
Effects and
Impacts
16
CONCLUSION
• Evaluation will give new insights on
how to improve.
• Evaluation will give you information on
which measures work and which don’t.
• Evaluation gives the public institutions
a proof of what you have done and
achieved, and legitimise what you do
at regional level.
17
The contribution of Futurreg Evaluation:
 Ensures the RELEVANCE, FEASIBILITY and
SUSTAINABILITY of a project
 Facilitates a dialogue / OWNERSHIP
 Ensures that fundamental questions are asked
and weaknesses are analysed
 Defines the key elements & the settings of a
project
 Identifies measurements/ indicators of the projects
achievements
 Systematic common sense (helping to adapt
18
Structure of the Report
•
•
•
•
Executive Summary
Introduction
Evaluation Methodology
Analysis of:
– Futurreg deliveries
– Futures toolkit
– Regional foresight capabilities impact
• Conclusions and Recommendations on the
Overall Futurreg
19