Impact Evaluation
Sri Lanka Dry Zone Livelihood Support and
Partnership Programme
Impact Evaluation Workshop
Colombo, Cinnamon Lakeside Hotel – 3 December 2013
1
FF
Background
• IFAD-wide: 30 project impact evaluations by 2015
• This has been IFAD’s first impact evaluation with
primary data collection on a large scale
• Testing internationally recognised methodologies
in an IFAD-project context. Identify opportunities,
constraints and limitations
2
FF
Criteria and data
• Whole set of IFAD project evaluation criteria:
Relevance, effectiveness, efficiency, impact, sustainability,
innovation, up-scaling, gender equality
• Primary data collection: mixed-method approach:
Survey of 2560 households (with / without project)
Qualitative survey: focus groups: 41 with beneficiaries
and project and 30 with line agencies’ staff
Synthesis mission
• Collaboration with: GreenTech Consultants Pvt. (SL);
Social Impact, Washington State University (USA)
3
FF
Methodology
1. Baseline survey conducted in 2006 but dataset was lost
2. Project targeted poorer communities and households
→ sampling bias
3. Farmer household’s characteristics can affect participation
in project: → sampling bias
4. Economic growth and poverty reduction in Sri Lanka during
past 10 years: attribution issues
5. Multi-component: different impact pathways, difficult to
capture through standardized questionnaire
4
FF
Approach followed
1. Data recall (reconstructing selected baseline data)
2. Econometric techniques: (i) Propensity-score matching;
(ii) Heckman treatment effect
 1st step: determine community and hh
characteristics associated with project participation
 2nd step: compare hh assisted by project vs. hh
without project
This is expected to reduce sampling bias
3. Cross validation between mixed methods (qualitative &
quantitative)
5
ES
General project information
• Total cost US$ 27.2m, IFAD loan US$ 22.0m + 0.3m grant
• Approved Sep 2004, effective Dec 2005, compl. Mar 2013
• Components:
(i) rain-fed upland agricultural & livestock developm. (10%);
(ii) small-scale irrigation rehabilitation (15%);
(iii) marketing and enterprise development (13%);
(iv) microfinance and income-generation (12%);
(v) community infrastructure development (27%);
(vi) programme management (22%)
• Slow implementation 2005-2008. Catch-up 2009-2012
6
ES
The
programme
area
7
ES
Findings – Project Performance
• Relevant project design and component mix. Exceptions: (i)
initial marketing component design and (ii) subsidised interest
rates for credit
• Adaptation to evolving country context (emphasis on highervalue crops, dairy farming, linkages with existing value chains)
• Concomitant focus on seed production schemes increased
output quality and added value for the seed producers,
especially in the case of seed potatoes
• Immediate objectives achieved in majority of components.
High coverage of agricultural extension packages, although
quality not always at par with quantity
8
ES
Findings – Project Performance (2)
• Efficiency – a sluggish first half of project implementation, with
23% disbursements after 43 months, with a vigorous turnaround after the MTR in 2009
• Common pattern in IFAD-funded projects, depressing efficiency
• The catch-up effort has emphasized quantitative targets and less
consolidation of results
• The PCR infers high internal return rates while the impact
evaluation is more cautious on methodological grounds
• Subsidized interest loan rates are inefficient → credit rationing.
Alternative: matching grants coupled with market interest rates
9
FF
Targeting – statistical analysis
Pro-poor geographical targeting (communities)
• Communities with project are more distant from District
Secretary Division’s capital
• Less equipped with basic infrastructure (e.g. school,
police, health post)
Household with project had:
• Poorer asset base before project start-up (recall data)
• More diversified cropping patterns
• Slightly better educational status of household head
10
FF
Findings: impact
1) Project M&E data show generalised and consolidated
impacts. Evaluation findings are more nuanced. Why ?
 Evaluation survey sample more representative (random
sampling) and takes into account attribution issues
(included households without project)
2) Impact survey finds that farmers are better aware of
improved techniques, higher value crops and products
3) Knowledge and awareness ≠ adoption. Initial investment
costs can be a barrier, and farmers require extended
technical support
11
FF
Findings – impact (cont.)
4) Project contributed to small farmers’ investments in dairy
farming in dry zone area
5) Households invested in dairy farming (and on higher value
crops) out of their own savings (limited credit access)
 crowding out investment in other hh assets
 Mixed effects on hh expenditures (as an income proxy)
6) Project activities peaked 2009-2012, impacts still emerging
12
ES
Other findings
• Moderately sustainable: creation of farmers’ federations,
linkages with public extension programmes and private
businesses. But farmers’ federations still weak
• Multiple innovations (e.g., potato and onion multiplication
technology, quality seeds for cowpeas, groundnuts, cooling
machines for milk, FFS approach).
Partnerships with public extension and private sector facilitate
up-scaling. Progress will be slow without follow-up support
• Gender equality: men and women both benefitting from
technical packages. Women well represented in farmers’ society
governing bodies
13
ES
Recommendations for the future
• Need for a follow-up phase to consolidate results
• Promote: (i) linkages with existing value chains through
public-private sector partnerships; (ii) support farmers’
societies and federations; (iii) avoid subsidised credit
• Advocacy on policy issues needs to continue. Not only
land tenure, also formal registration of farmers’ societies
• Commitments to provide a financial contribution to
revolving funds to be honoured
• Better accuracy and quality control in M&E is required.
Retain good practice of conducting thematic studies.
Conduct and preserve baseline survey
14
ES
Take home lessons
• The DZLISPP evaluation shows that supporting smallholder
agriculture, modernizing agriculture and fostering market
integration can be compatible objectives
• The presence of large retailing and processing companies
in rural areas provides important opportunities for market
linkages
• DZLISPP provided a seminal contribution but without
additional support, progress may be slow in the future
• Mixed findings on impact reflect the burden of initial
investments on smallholder farmers’ households.
15