Seminar on selected evaluation
methodology issues
评估方法论研讨会
Independent Office of Evaluation of IFAD
Beijing, 16 July 2014
国际农业发展基金独立评估办公室
2014年7月16日，北京
Objectives
• Discuss selected methodology issues for
evaluation of agriculture development
interventions
• Collect feedback for the development of the
new IFAD/IOE evaluation manual
2
Context
• IOE and Ministry of Finance joint Statement of
Intend for Evaluation Capacity Development
• Development of new evaluation manual, to be
issued in 2015
• Joint PPA of the Environment Conservation and
Poverty-Reduction Programme in Ningxia and
Shanxi
3
Key IOE products
• Corporate-level Evaluations
• Country Programme Evaluations
• Project Evaluations
• Evaluation Synthesis Reports
• Impact Evaluations
4
Evaluation criteria used
Relevance
Gender
Effectiveness
Innovation and Scaling-up
Efficiency
Borrower performance
Impact
Lender performance
Sustainability
Use of six point rating scale to assess performance
5
Methodology Issues (1)
Data Quality
Challenges: No baseline survey; weak M&E system; data mostly
on outputs; deficiencies in project log frame; too many
indicators
Options: Evaluability assessments; reconstruct log frame and
underlying theories of change at evaluation; primary data
collection (RRAs, surveys, focus groups, case studies);
triangulation
6
Methodology Issues (2)
Assessing efficiency
(A measure of how economically resources/inputs (funds,
expertise, time, etc.) are converted into results)
Challenges: Lack of data on outcomes; difficulty in quantifying
benefits; weak data on costs and inputs; inadequate capacity to
measure EIRR; not all projects have ex-ante EIRR
Options: use of proxy indicators (delays from loan approval to
effectiveness; time-over-run, project management costs, cost per
beneficiary, final disbursements; etc.); benchmark unit costs (e.g., for
infrastructure; CBAs
7
Methodology Issues (3)
Evaluation and hindsight
(How to evaluate a project in an evolving context? Is it fair to
evaluate a project using today’s standard?)
Challenges: New priorities emerge during implementation (e.g.,
climate change, scaling-up, etc.); evolution of policy and institutional
context
Options: Learning accountability; retrofitting design during
implementation; If project objectives and/or outcome targets were
changed during implementation, both the original and revised
objectives are taken into account
8
Methodology Issue (4)
Impact Evaluation
Definition of impact: The changes that have occurred
– as perceived at the time of evaluation – in the lives of
the rural people (whether positive or negative, direct
or indirect, intended or unintended) as a result of IFAD
interventions”
IOE Evaluation Manual 2009
9
Methodology Issue (4) – cont.
IOE and impact evaluation
• IOE has introduced impact evaluations as new
independent evaluation products in 2013
• 1 impact evaluation finalized (Sri Lanka); 1 ongoing
(India); 1 planned in the IOE 2015 Work Programme
• Selectivity framework guiding the identification of
the (project) impact evaluation
10
Methodology Issue (4) – cont.
IOE Objectives for Impact Evaluations
• Assess the results and impact of the programme in a
more quantitative manner;
• Generate recommendations; and
• Contribute to innovative mixed evaluation
methodology development
11
Methodology Issues (4) – cont.
How do impact evaluations provide evidence of the
changes caused by an operation?
• Impact evaluations compare the situation of the beneficiaries who
participated in the program (treatment group) against those who did not
participate (comparison group), as well as “before and after” the program
• Mix-methods approach applying quantitative quasi-experimental and
qualitative participatory methods:





Pre-Post
Simple Difference
Difference-in-difference
Matching
Multivariate (Multiple) Regression
12
Methodology Issues (4) – cont.
Impact evaluations: key challenges
• Complexity of IFAD’s programmes and scarce availability
of baseline data
• Incubation time for impact measurement
• Establishment of a counterfactual that will effectively
allow the attribution of impact to the program
13
Methodology Issues (4) – cont.
Impact evaluations: opportunities
• To improve accountability
 through assessing not only “if”, but also “how” and “why” the
program has, or has not, had an impact and its magnitude
• To improve systematic and dynamic learning
through:
 enhancing IFAD’s ongoing programs and ultimately better allocate
funds across programs
 feeding results into relevant CPEs and CLEs undertaken by IOE
 strengthening the internal debate on impact evaluations
14