Training Workshop on
Evaluation of
Agricultural and Rural Development Programmes
Independent Office of Evaluation (IOE), IFAD
Beijing, China
1 November 2013
Training Workshop Agenda
Module 1: Introduction to Development Evaluation
Module 2. Evaluation of Agricultural and Rural Poverty
Alleviation Interventions (ARD)
Module 3: How to Manage and Conduct ARD Evaluations
Module 4: A Practical Evaluation Exercise
Module 5: Wrap-up and Practical Suggestions
2
Module1. Introduction to
Development Evaluation

Development Evaluation: Development + Evaluation

Evaluation of development interventions

Different types of development interventions

Different types of evaluation
3
Why are Evaluations Important?
Evaluations for accountability: to show for which

purposes resources were used. Were the programme’s
or project’s resources used to achieve its objectives?
Evaluations for learning: to draw lessons from experience

and apply them to improve operations. What lessons can
be drawn from the programme’s experience?
4
Relations between
Monitoring and Evaluation

MONITORING: observing and collecting data on the
implementation of an intervention, including inputs,
outputs and outcomes, & on contextual factors

EVALUATION: on the basis of monitoring data and a set
of criteria, assessing the effectiveness of interventions

Evaluation uses monitoring data and evaluation
requirements determines which data should be
monitored ME & EM
5
Evaluation Criteria

Relevance: were the objectives consistent with
beneficiaries’ requirements, country needs, institutional
priorities and partner and donor policies?

Effectiveness: to which extent the development
intervention’s objectives were achieved, or are expected
to be achieved?
6
Evaluation Criteria

Efficiency: how economically resources/inputs
(funds, expertise, time, etc.) were converted into
results? how reasonable were the costs?

Sustainability: will the results achieved be durable?
Is it likely that those results will be sustainable?
Innovativeness: were innovative pro-poor
approaches introduced by the intervention?
Replicability and Scaling-up: to what extent the
interventions introduced have been (or are likely to
be) replicated and scaled up?


7
Evaluation Criteria
The first three criteria (relevance, effectiveness and

efficiency) are called “core performance criteria”
IFAD also uses other evaluation criteria such as

“rural poverty impact”, “performance of partners”
Equity: how equitable is the distribution of benefits

(by gender, social group or other attributes) is an
additional criterion
8
Impact Domains

Household income and assets

Human and social capital and empowerment

Food security and agricultural productivity

Natural resources and the environment

Institutions and Policies
See key questions for each of the impact domains in Table 7, ps. 37
& 38 of the IFAD Evaluation Manual.
9
Different Types of
Development Interventions

Projects: achieving development objectives
(e.g. reducing rural poverty) through a set of instruments
or components (e.g. microfinance)

Programmes: set of projects

Policies: sets of objectives that are implemented through
a set of programmes or projects

POLICIES  PROGRAMMES  PROJECTS
10
Development Evaluation and
Different Types of Evaluation Cont.’

Ex-ante evaluation (appraisal)

Process evaluation

Mid-term evaluation

Completion evaluation

Ex-post evaluation – Impact evaluations

----------------------------------------------- > time

Formative (to improve) Summative (to judge)
11
Reference e-materials

Conducting Quality Impact Evaluations Under Budget Time and Data
Constraints (2006)
http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/4585672-1251461875432/
conduct_qual_impact.pdf

Book Impact Evaluation in Practice (2011) Eng/Fr/Sp.
http://siteresources.worldbank.org/EXTHDOFFICE/Resources/5485726-1295455628620/
Impact_Evaluation_in_Practice.pdf

IFAD Evaluation Manual
www.ifad.org/evaluation/process_methodology

Glossary of Key Terms in Evaluation and Results Based Management
www.oecd.org/dataoecd/29/21/2754804.pdf
http://www.oecd.org/dac/evaluation/evaluationandaideffectivenessseries.htm
(Chinese version under doc. No. 6)
12
Module 2: Evaluation of Agricultural and
Rural Poverty Alleviation Interventions

Interventions whose objective is to alleviate rural poverty,
through agriculture and rural development

Programmes and projects with different types of
components: microfinance, agricultural extension, support
for rural non agricultural activities

Area based interventions and country wide programmes.
13
Examples of Evaluations of ARD

Evaluation of a rural development project in the
Northeast of Brazil

Country Portfolio Evaluation (CPE) of IFAD in Brazil
and CPE for China

Evaluation of a rural microfinance project in
Colombia

Thematic evaluation of agricultural extension services

IFAD and AfDB joint evaluation of ARD in Africa

IFAD Completion evaluation of the West Guangxi
Poverty-Alleviation Project
http://www.ifad.org/evaluation/public_html/eksyst/doc/prj/region/pi/china/guangxi
14
IFAD’s Methodology for
the Evaluation of ARD

IFAD’s Evaluation Manual, Methodology and Processes
www.ifad.org/evaluation/process_methodology


The manual builds on internationally recognized
evaluation practices and criteria
Guidelines for project completion report validation (PCRVs) and
project performance assessment (PPA), complement the Evaluation
Manual
http://www.ifad.org/evaluation/policy/new_policy.htm
where you can also find IFAD’s “Guide to Project M&E”
15
IFAD’s Methodology for the Evaluation of ARD:
Evaluation Criteria
1.
Project performance
2.
Household income and assets
b. Human and social capital and
empowerment
c. Food security and agricultural
productivity
d. Natural resources and the
environment
e. Institutions and policies
a.
Relevance
b. Effectiveness
c. Efficiency
a.
3.
Other Performance
Criteria
a)
b)
Sustainability
Promotion of pro-poor
orientation, replication and
scaling-up
Drawing upon 1,2 and 3, overall project
achievement of the project
16
Rural Poverty impact
4.
Performance of Partners
IFAD
b. Government
c. Cooperating Institution
d. NGO/CBO
a.
IFAD’s Methodolgy for the Evaluation
of ARD: Rating System

A six-point rating system for the evaluation criteria as a
way to quantify qualitative judgements of evaluators

6: highly satisfactory
5: Satisfactory

4: moderately satisfactory
-----------------------------------------------
3: moderately unsatisfactory

2: unsatisfatory
17
1: highly unsatisfactory
Other Approaches to
the Evaluation of ARD

Goal-free evaluation
Its strength in avoiding a limited focus on objectives can
be compensated by an explicit focus on “unintended
consequences”
Weak in accountability.
Impact evaluation with control groups

Explicit counterfactual to model the without
programme situation
18
M&E of innovation, replication and
scaling up
19
Evaluation Challenges

The contribution of the IFAD portfolio to innovation and scaling up is
important to justify IFAD’s lending to China, - the financial volume of which
is insignificant from a national financial perspective. However, extremely
challenging to evaluate this contribution

Project designs seldom have specific and concrete innovation agenda –
what is an innovation in the local context?

Often innovations emerge during implementation in response to
problems/opportunities – by farmers, NGOs, researchers, implementers,
IFAD supervision missions etc.

Often, implementation and project M&E systems do not clearly identify
innovations and lack systems to confirm value added/viability of innovations
and their replication and scaling up - M&E methodology for innovations is
the same as for standard interventions

A large part of the scaling up may take place after project completion and
in areas not covered by the project
20
Identifying innovation and scaling up
and the processes involved
Which interventions are innovative and
what is the innovation process?
Where and how are innovations scaled up –
processes and mechanisms?
Typical Project
Scaling up
inside
project
area
15
Standard Interventions
(road etc)
Innovative
Interventions
85
21
Innovation
Scaling up
outside
project
area
Innovation and scaling up
different processes and evaluation approaches
Innovation




Define the hypothesis of the
innovation – ”a scientific
experiment”
In addition to before-after,
with/without analysis may be
necessary
Engage independent third party to
assess viability, relevance and
applicability
If viable and relevant, formulate
scaling up strategy
22
Replication and Scaling up



Spontaneous market-driven
replication with some facilitation
or managed scaling up, requiring
substantial support, - policy
dialogue, KM, partnerships etc.
Who are the partners and the
financial sources required in the
scaling up process?
How can the scaling up process be
monitored – proxy indicators, data
sources etc.
M&E of support for enterprise
development and employment
generation
&
The Standard of the OECD Donor
Committee on Enterprise Development
(DCED)
23
Rural poor will increasingly depend on
enterprise development and employment





Current strong trends of land consolidation, agricultural
commercialisation, outmigration and ageing of farm population are
changing rural socio-economic landscape
Small commercial farms and rural enterprises, including agroindustries, are becoming important players
For their livelihoods, rural poor will increasingly depend on the
welfare system, leasing out their user rights, remittances and
employment on- and off-farm (often seasonal)
For development partners, obvious niche to support development of
MSMEs and cooperatives, creating employment for the poor – but
how to monitor and evaluate
DCED Standard provides a common tool to design support and
measure results and impact of support for enterprise development
24
Universal Impact Indicators

Scale: Number of target enterprises who realize a financial benefit as a result of
the programme’s activities per year and cumulatively. The programme must define
its “target enterprises.” (may be farms, MSMEs, cooperatives etc but also poor
workers)

Net income: Net additional income (additional sales minus additional costs)
accrued to target enterprises as a result of the programme per year and
cumulatively. In addition, the program must explain why this income is likely to be
sustainable.

Net additional jobs created: Net additional, full time equivalent jobs created in
target enterprises as a result of the programme, per year and cumulatively.
“Additional” means jobs created minus jobs lost (displacement and crowding in).
“Per year” comprises 240 working days. The program must explain why these jobs
are likely to be sustainable. Jobs saved or sustained may be reported separately.
These enterprise-level impact indicators can be translated into household-level impact but it is a demanding
process, particularly with respect to attribution, because households may have multiple income streams.
25
Eight Control Points
1.
Articulating the theory of change/results chain
2.
Defining the indicators of change
3.
Good measurement practices
4.
Handling attribution
5.
Capturing wider changes in the systems or markets
6.
Relating to programme costs
7.
Reporting results (internal/external)
8.
Managing the system for results measurement
26
Articulating the Results Chain
Obligatory
1.
An appropriate, sufficiently detailed and logical results chain(s) is
articulated explicitly for each of the interventions.
2.
Each results chain is supported by adequate research and analysis.
3.
Mid and senior level programme staff are familiar with the results
chain(s) and use them to guide their activities; key partners can explain
the logic of interventions.
4.
The results chain(s) are regularly reviewed to reflect changes in the
programme strategy, external players and the programme circumstances.
Recommended
 The results chain(s) include the results of broader systemic change at key
levels.
 The research and analysis underlying the results chain(s) take into account
the risk of displacement.
27
Example of Results Chain
www.Enterprise-Development.org
LG increases turnover
and profits
LG has more and better
quality exports
Silkworm farmers
increase income
Silkworm farmers
increase sales to LG of
high quality cocoons
Silkworm farmers increase
yields, quality and
production of cocoons
LG improves processing
LG company sells highyielding mulberrys to
silkworm farmers
28
LG sells improved
silkworm eggs to
silkworm farmers
LG trains silkworm
farmers in modern
cultivation methods
Module 3: How to manage and conduct
ARD evaluations

The first strategic decision is choosing WHAT to
evaluate. Decide on the basis of expected value added
for accountability and/or learning

The terms of reference should clarify expectations
concerning the evaluation that will be carried out.
Answers to WHY, HOW, and WHEN TO EVALUATE?
See “Writing Terms of Reference for an Evaluation: A How-To Guide”
https://ieg.worldbankgroup.org/evaluations/evaluationcapacity-development-ecd
29
How to manage and conduct
ARD evaluations Cont.’

Selecting the type(s) of evaluation to be used
Methods adequate to the circumstances (purpose,
availability of time and resources).
See IFAD Manual and “Monitoring and Evaluation: Some Tools, Methods and
Approaches”
https://ieg.worldbankgroup.org/evaluations/evaluation-capacitydevelopment-ecd
30
How to manage and conduct
ARD evaluations Cont.’
 Consulting
with stakeholders
Why? To get their insights, and
to nurture their interest in the evaluation (which will
facilitate the use of the evaluation).
 Identifying
key stakeholders:
Partners, intended beneficiaries, government officers, civil
society representatives
31
How to manage and conduct
ARD evaluations Cont.’
 Reviewing
relevant documentation
In addition to PROJECT DOCUMENTS, reports on the
country, internet searches for thesis. other evaluations
and academic papers on the project/programme area and
on the instruments used (e.g., rural microfinance or
agricultural extension in the area)
During the review prepare notes and a bibliography, as inputs for the report.
32
How to manage and conduct
ARD evaluations Cont.’
 Applying

evaluation criteria
Relevance, Effectiveness, Efficiency, Sustainability,
Innovativeness, Replicability and Scaling-up, Rural Poverty
Impact , Performance of partners

Ratings for the criteria

Rating overall project achievement

Providing a justification for the ratings
33
How to manage and conduct
ARD evaluations Cont.’
 Drafting

See section 3B of IFAD Evaluation Manual,p.30
 Using


34
and reviewing evaluation reports
evaluation reports
Evaluation reports are important INPUTS for evaluations,
as indicated before under “reviewing documentation”.
Evaluation reports are also key OUTPUTS of the
evaluation process. Show in a clear way the links between
findings and recommendations to facilitate the use of the
evaluation report.
Module 4: Exercise: Evaluation of
the Northern Someland ARD Project

With support from the XFund, the 10,000,000 euros Northern
Someland ARD project aimed to provide low prized fertilizers to
5,000 farmers in the northern region of Someland, a land-locked
country, where small farmers are fruit producers.

The objective of the project was to improve the welfare of the
farmers and their families. A project implementation unit was set
up to organize the delivery of fertilizers and to establish links
between the farmers and local institutions involved in the
provision of complementary services.

How would you evaluate this project? What would you do
to evaluate the project? Please discuss with your
colleagues and prepare a 5 minutes presentation
35
Module 5: Wrap-up and Practical
Suggestions

Review key points discussed

Your suggestions for ARD evaluations
36
Feedback survey on the training
A.
What have you learned?
B.
Do you have any suggestion for future training on
evaluation in China?
C.
To what extent to you think that the objectives of
the workshop were achieved?
37
6. Highly Satisfactory
3. Moderately Unsatisfactory
5. Satisfactory
2 . Unsatisfactory
4. Moderately Satisfactory
1. Highly Unsatisfactory
Thanks for your participation!
38