Impact Evaluation:
Presentation to DAC Evaluation
Working Group, Paris, June 2, 2005
Howard White
Operations Evaluation Department
World Bank
What is impact evaluation?

Has had different meanings





Sector or country-wide evaluation
Long-run effects
Establishing the counterfactual
Focus on final welfare outcomes
OED adopt a combination of the last two: ‘a
counterfactual-based analysis of how the
intervention has affected welfare outcomes’
Impact evaluation in the development
context


Been associated with poverty, gender and
environment effects, using mixed methods
with bias toward qualitative
But growing tide of quantitative impact
evaluation. Driven by:


New techniques and technology
Results-based agenda (including MDGs)
Impact evaluation at the World Bank


As with all evaluation, much work takes place
outside of OED
For impact there is a new initiative (DIME)
promoting greater use of impact evaluation.
These studies are:



All prospective
Attempt to promote randomization
OED’s own program is adopting a range of nonexperimental approaches, firmly grounded by
context
The existing OED program



Impact evaluation not new to OED
Over 80 studies classified as impact, and others
not so classified adopt different approaches to
measuring impact
Current program under OED-DFID partnership
supporting three studies:



Ghana basic education
Bangladesh maternal and child health
India rural poverty
Ghana: Method and approach





Main data collection was survey to follow up
GLSS2 from 1988 education module
Combined with time spent in the field in three
visits
Background analysis: budget and political
economy (context)
Multivariate analysis of determinants of
educational outcomes. Link those determinants to
donor-financed activities
Work in collaboration with MOEYS and GSS
Ghana: findings





Enrolments are rising
Learning outcomes are improving
Better school infrastructure is part of explanation
Hence Bank-financed school investments lay
behind a substantial part of these improvements
in education outcomes
But growing dichotomization of public sector
(partly rectified in new Bank-supported program)
Bangladesh: method and approach







Initial meta-analysis
Using existing data sets (mainly DHS)
Own analysis and commissioned studies
Multivariate analysis of determinants of mortality
and nutrition (Oaxaca decomposition)
For BINP using propensity score matching
combining two datasets (problem of poor quality
and contaminated control)
Holistic approach
Work with IMED, and local research company
Bangladesh findings


Role of publicly-supported interventions in
successful reduction of fertility and mortality
Immunization most cost-effective, but so are




Training TBAs
Female secondary stipends
Strong cross-sectoral effects (need not imply
multi-sectoral programs)
BINP: theory-based approach identifies weak
links that help explain poor outcomes
India rural poverty

Looking at two interventions




AP Irrigation II and III
AP Rural Livelihoods Project
Three rounds of surveys (2005-2007)
Using innovative data collection
instruments for ‘immeasurables’
Do you want to do your own impact
studies?

Pros



Cons



Demonstrate results
Perceived as high quality products, appear to find
audience in country teams
Expensive
Technically demanding
So when to use


Periodic validation
Pilots
What do you need to do your own
impact study?





Some opportunism in selecting cases
Sufficient scale intervention to justify cost
Lengthy lead time, especially if collecting
own data (18 months from start to finish is
best you can hope for, 24-30 months is
more realistic)
Appropriate skills mix
Promote prospective evaluation
Data collection for impact evaluation
Approach
Description
Project-specific survey
Own data collection in project and
control areas
Synchronized survey
Carry out project survey at same time
as related national-level survey
Piggy-backing
Utilize an existing national survey,
possibly paying for (1) over-sampling
project areas, and (2) adding a
project-related module
How OED is planning to take its
program forward




Partnership has helped consolidate
commitment to continuing impact
evaluation
‘Study a year’ is built into work program
One prospective study being agreed
(possibly Karnataka health)
Open to discussion of expanded program