Role of evaluation in
the stage of measures
and project planning
prof.dr.Branko Kontić,
dr.Davor Kontić
CIVITAS Elan Study Tour
Ljubljana, 23 October 2013
Aim of this discussion

Key message to the EC – over
VANGUARD (why not POINTER?;
expecting synthesis report) – for
improving the CIVITAS strategic
evaluation system (top responsibility for
quality projects and efficiency of
budgeting)

Potential for contributing to the benefits of
CIVITAS projects at operational level
(e.g., city mobility level, measures level,
common measures, transport policy
development)
CIVITAS Famework
How does EC know that the
project will meet the goals?
Impact evaluation
Integrated package
evaluation
CIVITAS
Plus
CIVITAS II
CIVITAS I
DISSEMINATION
Process evaluation
POINTER
GUARD
METEOR
City level evaluation
Cost-Benefit Analysis
Evaluation Framework
(by POINTER, 2009)
CIVITAS Evaluation Objectives
CIVITAS
Measures
Impact Evaluation
 Guidelines for defining
evaluation scenarios
 Common core indicators
 Guidelines for measurement
 Guidelines for up-scaling
 Guidelines for reporting
Process Evaluation
Evaluation at
Project and
City Levels
Reporting
Evaluation Results
Cross-site Evaluation
 Comparing results across sites
 Assessing transferability
 Conclusions and recommendations
 Collecting information on
all stages of the measure
 Feedback and support
 Factors of success and
unexpected barriers
 Input for policy
recommendations
Impact Evaluation Framework
(by POINTER, 2009)
Detailed Measure Descriptions
 Effects/Impacts
 Indicators
P
O
I
N
T
E
R
Evaluation Plans
‘Baseline’
‘Business-as-Usual’
Measure or Grouped Measure
level Evaluations
Project & City
understandings
Cross-site Evaluation
and Transferability
Support for
Recommendations to other
EUROPEAN CITIES
‘After’
BAU and CBA Evaluation
(by POINTER, 2009)
Process Evaluation Framework
(by POINTER, 2009)
POINTER
Support
Process
Topics & Issues
City
level:
26 cities
Process Topics &
Issues Form
Selection
Measure
level:
Ca. 120
measures
(Ca.
30%)
Focussed Measure Process Evaluation
Focused Measure
Process
Evaluation Form
Selection
Measure Process Evaluation
Measure
level:
336
measures
(100%)
Preparation
Implementation
Measure Process
Evaluation Form
Operation
(by POINTER,
2009)
Overview - LCA of measures and evaluation
Phases/activities of a measure
Evaluation of measure’s
and project goals
planning
implementation
operation
Process evaluation
Impact evaluation
CBA
Transfer&upgrade
Missing components

Strategic view in the overall concept (not
transparent and explicit; expectations to
be provided by POINTER and Project
Manager, however not clearly
presented/formulated)

Loop, feedback, monitoring (possibility for
consistent adaptation in terms of project
goals – need for additional evaluation
approach/understanding/tools in the
planning stage of the project)
Missing elements in the planning
stage 1/2

Role of the Evaluation Plan
•
•
Purpose (holistic view); it is not only a deliverable
•
Changes in project implementation (measures’ design,
implementation, operation) and links to the evaluation plan
Project management context (consultation); better cooperation
needed

Agreements between MLs and Evaluation Team
(objectives, data collection, etc.); see next slide

Performance assessment (PA) of a measure in the
context of evaluation; see next slide; PA follows all
(internal and external) barriers and drivers, process
evaluation follows only outside barriers and drivers
Missing elements in the planning
stage 2/2

Links between evaluation and dissemination: evaluation
of dissemination activities, dissemination of evaluation
results

Links and preparations to strategic and management role
of the evaluation, specifically PA and CBA: Is/was money
being reasonably spent? Was it worth doing? What could
be done better? Which mistakes we should not repeat?
Post “lessons learned” approach is less effective!
A sample of an agreement
between ML and SEM
A sample of the Performance
Assessment table
Possible improvements 1/3

Within the hierarchy the areas of responsibility are not
defined between measure, city and project level. Especially
the role of the measure leader with regard to the evaluation
needs to be defined clearly.

Not everyone knows how evaluation works and what is
needed. The measure leaders are not experts in
evaluation.

The planned evaluation procedure does not always work
according to plan, it needs to be flexible.

Ljubljana SEM signed agreements with measure leaders
on the collection of data, this is a good tool to clarify the
role of the measure leaders, asking them to provide data
on a regular basis.

There is a pressure from the PM (Project Management)
regarding ticking boxes only without checking the content
of the evaluation and its wider/strategic purpose.

Measure leaders need to understand what evaluation is
about!! Evaluation should be continuous activity in line with
measure planning and implementation.
Possible improvements 2/3

Process evaluation is important for all measures not only
focus ones; it clarifies whether implementation is
progressing well or not.

Ljubljana introduced performance assessments from the
early beginning. It was defined as an addition to the
process evaluation – covers also internal barriers and
drivers, and monitors some key components of a ML's work
and performance of the measure itself (e.g. management,
implementation, evaluation). The results are aggregated in
a score (colour score system with comprehensive
explanation of the rules for assigning a specific
score/colour).

Internal Progress Report (IPR) serves primarily for
management purposes; evaluation results loose their
strength in the context of this reporting. Transposing
performance assessment scores into "traffic light system”
of the IPRs should be consistent.
Possible improvements 3/3
Role of the evaluation should be somehow
integrated in the project approval procedure by
the EC!
Justification: In the case of ELAN more than a year
after official start of the project the evaluators were
working with MLs and PEM on "what is to be done in
the framework of a certain measure and how will the
work/measures be evaluated". The consolidation of
DoW in terms of evaluation activities is necessary.
This could be formalized by a requirement that a
DoW quality check is done by the evaluators too, not
only by the project manager and project coordinator.
Improved organization of
CIVITAS Elan
European Commission
Financial & Administrative
Reporting
Political Steering Group
Project Coordinator
Project Consortium
Meeting
Partners
Project Manager
Project Management Group
DisseminationEvaluation
Dissemination
Evaluation
Manager
Manager
Site
Coordinators
Coordinators
Local
Evaluation
Evaluation
Manager
Manager
Measure
Measure
Local
Dissemination
DisseminationMeasure
Manager
Measure
Measure
Measure
Scientific
Coordinator
Improved organization of
CIVITAS Elan
EC
Pointer
Project Coordinator
Improved Evaluation
Project
Manager