The Future of
Monitoring and
Evaluation:
Lessons from a Decade
of Impact Evaluations
Eric Foster-Moore
The World Bank | Africa Water Resources
efostermoore@worldbank.org
Development
works
Our project will directly benefit
400,000 people…
Our project will benefit the people of
Mozambique…
Number of women benefitted
Growth rate in country X increased by
2 percentage points
Jeffrey Sachs
Big Push, Millennium Villages Project,
The End of Poverty
“$195 billion in aid
per year for 20 years would
eliminate poverty.”
William Easterly
“No it won’t.”
The White Man’s Burden
Dead Aid by Dambisa Moyo
Can the West Save Africa?
The white band’s burden…
Art credit: @ChrisPlanicka
Three questions
1.
2.
3.
Is aid effective?
How do we know?
How do we apply this to our own work?
Does aid increase growth? A macro perspective
Boone (1996): Nope.
Burnside and Dollar (2002). It depends.
Effective when combined with “good policy.”
Easterly, Levine, and Roodman (2003). Not really.
Original Burnside-Dollar results
Easterly, Levine, and Roodman
Does aid increase growth? A macro perspective
Boone (1996): Nope.
Burnside and Dollar (2002). It depends.
Effective when combined with “good policy.”
Easterly, Levine, and Roodman (2003). Not really.
Clemens, Radelet, Bhavnani, and Bazzi (2012).
It still depends.
(Chris Blattman)
What about at the micro
level?




Often just make them up: “This project will directly
benefit 300,000 people.”
Move beyond correlations. Establish causality. Do
an experiment. Some are natural, some are
contrived
Natural experiment: half of a community displaced
by a reservoir
Contrived experiment: the Fiala (2013) paper
Fiala (2013):
Stimulating Microenterprise
Growth: Results from a Loans,
Grants, and Training Experiment in
Uganda
Grants, loans, training, or a combo?
Treatment groups:





Loans
Loans plus training
Cash
Cash plus training
Control
Follow up surveys after 6 and 9
months.
Results

Loan only: effect is gone after nine months

Loans and training: 54 percent increase in profits for men


Grant only: no effect whatsoever. For men, Fiala speculates, “Knowing that
the loan had to be repaid appears to have led them to use the money more
effectively in the business.”

Grants and training: no effect
 Women: no positive effects. “Family pressure on women appears to have
significant negative effects on business investment decisions: married women
with family nearby perform worse than the control group in a number of the
interventions.”
Acemoglu, Johnson, and Robinson
(2000): The Colonial Origins of
Comparative Development
What is the effect of institutions on economic performance?
Instrumental
variable
Methods
Strong
institutions
Other stuff
Strong economic
performance
Hypothesis
Results




“Differences in institutions account for roughly
three quarters of the differences in income per
capita.”
Not driven by outliers such as the United States,
Canada, Australia, etc.
Not driven by simply being on the African
continent.
Not driven by distance from the equator (i.e.
geography).
What RCTs do really well:




Compare program alternatives
Quantify effects of program X in place Z at time T
Force you to identify and test assumptions
Microfinance, cash transfers, education, health, job
training, etc.
Monitoring and evaluation: some examples
What you see:






Policy and legal frameworks
strengthened
Cash collection ratio; debts repaid
Number of people in project area with
access to “Improved Water Sources”
Percent of biological samples failing
Number of direct beneficiaries,
percent of which are female
Number of staff trained
What you don’t see:







Household income
Business profitability
Governance
Strength of political parties
Comparison to other
interventions
Opportunity cost
Articulation of assumptions
Two problems with RCTs
1.
2.
There’s an entire sector of projects that don’t get
tested and where assumptions that are probably
decades old dominate. Let’s bring the lessons of
impact evaluation to these projects.
Need to be strategic about their application in
order to learn something about human behavior.
Three questions
1.
2.
3.
Is aid effective?
How do we know?
How do we apply this to our own work?
Backup
Solow model of growth
From E-L-R
(2003)
From AJR