Jela Tvrdonova, 2014
1








Strategic approach to rural development
Common approach to evaluation: legal
background and CMEF
Monitoring and evaluation – what we are
talking about?
What is evaluated?
Intervention logic in evaluation
Indicators
Evaluation questions
CMEF guidance
2
Focus on limited number of objectives
Competitiveness – Environment – Quality of
life
Axes connected with strategic objectives
Strong and dynamic agri-food sector
Agriculture and forestry with high added
value
Employment and growth in rural areas
Improvement of the governance in rural areas
and mobilisation of the endogenous
potential



Exact definition of
objectives in Strategic
Guidelines for Rural
Development, Council
Regulation 1698/2005
(art. 77-87), EC
Regulation 1974/2006
(art. 60-62), Health
check,
National strategic plans
and EU strategic
monitoring
Single framework for all
program interventions





CMEF: Common monitoring
and evaluation framework –
Handbook, Annexes,
Guidance notes
Baseline indicators at the
program start
Suitable combination of
input, output, result and
impact indicators,
Common evaluation
questions for all RD
programs
Guidance for ex ante, midterm, ex post and ongoing
evaluation


Monitoring – measurement of immediate
outputs and results at measure and axis level
(different from strategic monitoring)
Evaluation - measurement of long-term
effects/impacts of intervention, within the
program development context (rural areas)
Accountability, comparability
and lessons learned for next
interventions
5



On-going process which monitors the
gradual implementation of the program at the
level of financial inputs, physical outputs and
results
Subject of annual reporting
Instruments – common and programme
specific indicators linked to outputs and
results,



Since 2010 and every other two years
Focus of the EC on the progress of national
strategic plans and its objectives
implementation
Assesment of the contribution of national
plans to the EU Rural Development Strategy
On-going evaluation is based on CMEF and
country´s own methodology to carry on the
evaluation during the program implementation
Process based on annual reporting on result/impact
indicators including periodical exercises:
◦ Ex – ante: evaluation of relevance of the planned
intervention and optimizing of the RDP budget,
WILL PROGRAMME
WORK AS IT SHOULD?
◦ Mid-term: for the assessing the progress related to the
specific and overall/program objectives,
IS PROGRAMME
WORKING AS IT SHOULD?
◦ Ex post : for the assessment of impacts and acievements
towards objectives.
DID PROGRAMME
WORK AS EXPECTED?



Good programme is the basis for the high
quality of monitoring and evaluation
Normally the monitoring and evaluation is
linked to programme intervention logic
But also specific topics, such as:
◦ Technical assistance
◦ Administrative arrangements
◦ Delivery mechanism
9
Intervention which logically responds to the most
important needs of the targeted area
Hierarchy of objectives:
• overall,
• specific,
• operational
Hierarchy
of expected effects:
• impacts on the territory,
• results on the
supported beneficiaries
• immediate outputs
at project level
Challenges of the CMEF &
Ongoing Evaluation
4/5 June 2009
10
Intervention logic of
EU rural development programmes
Challenges of the CMEF &
Ongoing Evaluation
4/5 June 2009
11
EU policy objectives
Context , its description
SWOT and needs assessment
Complementarity
Relevance
Overall objectives
EU/MS
Programme level
Impacts
Specific objectives
EU/MS
Axis level
Results
Operational
objectives EU/MS
Measure level
Outputs
Inputs
Source: EENRD 2014
Measures, projects
and their
management and
implementation
Efficiency
Effectiveness
Coherence
RDP Intervention logic
tools to measure the programme effects and
achievements towards expected objectives by
measures or the whole programme
should be Specific, Measurable, Achievable,
Relevant for the programme, and Timely (SMART)
Indicators should be filled with quantitative
statistical data; however in some cases, indicators
might also be linked to qualitative assessments or
logical assumptions
1
3



Baseline indicators: they relate to general socioeconomic context of the programme area (contextrelated baseline indicators) and to the state of the
economic, social or environmental situation in
direct relation with the wider objectives of the
programme (objectives-related baseline indicators)
Financial execution (input) indicators: they refer to
the budget or other resources allocated to the
programmes
Output indicators: measure activities directly
realized within programmes
1
4


Result indicators: measure the direct and
immediate effects of the intervention within
the group of programme beneficiaries and
provide information on changes that have
taken place
Impact indicators: refer to the benefits of
the programme both at the level of the
intervention but also more generally in the
programme area. They are linked to the
wider objectives of the programme
1
5
Impact
Result
Ouputs
Activities
Inputs

A common set of baseline, output, result, and
impact indicators for the RDPs (Art. 62 Reg.
1974/2006) “shall form the Common Monitoring
and Evaluation Framework (CMEF)’’ (Annex VIII
lists the common indicators)
1
7


Since common indicators may not fully capture all effects of
programme activities, it is necessary to define additional
programme specific indicators (see Guidance notes A & K),
which relate to programme specific issues.
They are developed if:
◦ Programme contains programme specific objectives and common
indicators are not sufficient to capture achievements
◦ There is the need to evaluate specific issues such as delivery
mechanism etc.
1
8


Define the focus of evaluations
Demonstrate the progress, impact, achievements,
effectiveness, efficiency and relevance of rural development
policy
Common Monitoring and Evaluation System distinguishes EQs:
a)
Common Evaluation Questions for Rural Development (CEQ-RD)
b)
Programme-Specific Evaluation Questions (PSEQs)
Source: EENRD 2014
19
COMMON
EQS FOR RURAL
DEVELOPMENT
Evaluation of matters
relevant at the EU policy
level
PROGRAMME
SPECIFIC EQS
Evaluation of matters
relevant to the programmespecific policy
Encourage the assessment of programme results and
impacts
Enhance comparability
across RDPs
Evaluation of specific RDP
related topics
Demonstrate the contribution of programme interventions
Source: EENRD 2014
20
Evaluation
questions
and
judgment
criteria
Policy
objectives
Triangular consistency
between objectives,
questions and indicators
which allows for:
• Less EQ and more
targeted towards policy
• Clearer formulation of EQ
and harmonization of all
terms used
Indicators
• More evidence to answer
EQ
• Better planning the
evaluation
Source: EENRD 2014
21
1.
Common Evaluation Questions

Linked to RD policy objectives

Causal-effect
extent..?)
questions
(To
what
2.
Common judgment criteria

Set the foundations to assess the
success of intervention

Formulate explicit judgments on the
basis common terms

Facilitate the identification of data,
information and analysis needed
3.
Common RD indicators

Additional
necessary
Source: EENRD 2014
information
3. Identification of
relevant common
rural development
indicators
1. Development of
CEQs linked to the
RD policy
objectives
2. Development
of common
judgment criteria
when
22
Handbook
Annex 1:
 Guidance notes - A: Choice and use of indicators,
B: On-going evaluation , C: Mid-term evaluation
Annex 2:
 Guidance notes – D: Hierarchy of objectives, E –
Measure Fiches
Annex 3:
 Guidance notes – F – K, Common indicators Fiches
Annex 4
 Guidance J – O Other guidances
2
3

Original set of 150 common evaluation questions
of CMEF has been simplified and reduced to the
essential demand for knowledge from the European
perspective.
3 groups of CEQs
Programme-related:
Community strategic priorities
Health Check objectives
7 impact indicators
TA and NRN
Efficiency of RDP resources
Source: EENRD 2014
Measure-related:
Contribution of
the measure
to its axis objectives
Other measure effects
and contribution
to other axes objectives
Leader approach-related:
Community strategic priorities:
Employment, diversification
and governance
Leader approach
LAG contribution
to LDS and RDP
2
4
RDP objectives
Evaluation Question
Indicators
Baselines and their
quantification,
Target levels
Overall strategic
objectives
Evaluation questions –
cross-cutting
Impact indicators
Horizontal baselines,
objective related,
contextual
Targets for
impact
indicators
Axis specific
objectives
Evaluation questions
Axis/sector specific
Result indicators
Axis based baselines,
objective related,
contextual
Targets for
results
indicatiors
Measure/operatio
nal objectives
Common and program
spec. Eval. questions
Output indicators
Baseline values of
output indicators
Targets for
output
indicators
2
5
CMEF provides guidance for MAs in setting up indicators at
each level of intervention and baseline indicators
Annex 3 of the CMEF Handbook provides detailed description
of all common indicators (Guidance notes F – K):
◦
◦
◦
◦
◦
F: COMMON INDICATOR LIST (overview of all common indicators)
G. BASELINE INDICATOR FICHES (detailed description of indicators)
H. OUTPUT INDICATOR FICHES
I. RESULT INDICATOR FICHES
J. IMPACT INDICATOR FICHES
2
6
Each indicator fiche contains the following elements:











Type of indicator
Related measures
Measure Codes
Definition of the indicator
Subdivision
Unit of measurement
Level of collection
Responsible actor for collection
Collection method/good practice
Sources
Registration frequency
2
7
Guidance on practical use of indicators in
monitoring and evaluation of RDPs
Guidance note E provides the following on
use of indicators within each particular
measure:

◦
◦
◦
◦
◦
◦
◦
Measure Code
Rationale of the measure
Target group
Target area
Common indicators
Link rationale of the measure and indicators
Evaluation questions
2
8
Introduction
Part I
For everyone
Introduces
ex post evaluation
and its role in policy
cycle
the scope and focus
of ex post
evaluation, legal
requirements and
common evaluation
elements
Source: EENRD 2014
Mainly
Evaluators
Toolbox
Introduces
Discusses and
explains
Provides
Explains also
specificities with
respect to NRN
29
Part III
Mainly
Managing
Authorities
the process of the
ex post evaluation,
steps to be
conducted and role
of evaluation
stakeholders
Explains
Part II
intervention logic,
evaluation
questions,
indicators, methods
and data
additional practical
tools for ex post
evaluation preparing,
implementation and
reporting.
EU policy objectives
SWOT and
needs
assessment
Basis of
evaluation
Focus of
evaluation
Operational
objectives
EU/MS
Collection of
evidence
Impacts
Methods
Data
Results
Outputs
Measures and
their
implementation
Efficiency
Source: EENRD 2014
Indicators
Specific
objectives
EU/MS
RDP Intervention logic
Evaluation
questions
Overall
objectives
EU/MS
Attribution of
impacts
Measurement
tools
Thank you for your attention
jela@ruralevaluation.eu
jelatvrdonova@gmail.com