w w w. n s p - i e . o r g
Evaluation Team
Andrew Beath (Harvard University)
Fotini Christia (M.I.T.)
Ruben Enikolopov (New Economic School, Moscow)
Shahim Kabuli (World Bank)
Sakhi Frozish (World Bank)
Partners
Vulnerability Analysis Unit (MRRD)
AfghanAid, CHA, IC, IRC, NPO/RRAA, Oxfam, PiN
National Solidarity Program (NSP)
Funding
National Solidarity Program
Food and Agriculture Organisation(FAO)
World Bank TFs
National Solidarity Programme
Executed by Ministry of Rural Rehabilitation
and Development (MRRD)
Implemented by 28 Facilitating Partners (local and int’l NGOs)
Funded by Multilateral, Bilateral Donors
Phase-I: 2003-7; Phase-II: 2007-11; Phase-III: 2010+
2003 – 2010: Covered 22,500 communities at a
cost of $929 million
National Solidarity Programme
Two Principal Village-Level Interventions:
Create Gender-Balanced
Community
Development
Community
CouncilsDevelopment
(CDCs) through
Secret Council
Ballot, (CDC)
Universal
Suffrage Election
Fund Projects Selected by
Water Supply
24% and
CDCs
and Villagers
Roads
& Bridges
25%
Managed
by CDCs
Projects
Irrigation
18%
(Average
Grant: $33,000;
Electricity
13%
Max.: $60,000)
6 months
18-24 months
What areas does NSP potentially impact?
Social Cohesion & Conflict
Political Attitudes
Gender
Community
Development
Council (CDC)
NSP
Projects
Access to Services
Local Governance
Economic Activity
The
The
evaluation
evaluation
measures
estimates
impact
theseof
impacts
NSP-II
byon
.....
Social Cohesion & Conflict
Political Attitudes
Gender
Access to Services
Local Governance
Economic Activity
The evaluation
The evaluation
estimates
uses these
this data
impacts
to . . by
. ...
collecting data over 3½ years in 500 villages:
250 NSP (treatment) & 250 non-NSP (control)
Structure of Evaluation and Data Collection
Aug. – Sep. 2007
Treatment
Villages
(NSP)
Control
Villages
Create
CDCs
Baseline
Survey
May – Oct. 2009
Select
Sub-Projects
Implement
Sub-Projects
1st
FollowUp
Survey
Spring 2011
SubProjects
Finished
2nd
FollowUp
Survey
(Non-NSP)
Similarity of Treatment
and Control Villages
Interim Estimates
(18% of Projects
Complete at Survey)
Final Estimates
Differences
The evaluation
estimation
uses
is accurate
this data because
to . . . . . .
compare changes in treatment villages (NSP) with
changes in control villages (non-NSP)
Estimation of Impacts of NSP
Treatment
Villages
(NSP)
Control
Villages
(Non-NSP)
- -
Baseline
Survey
Baseline
Survey
Baseline
Survey
=
=
1stChange
Follow-in
Treatment
Up Survey
1st Villages
=
FollowUp
Survey
1stChange
Follow-in
Control
Villages
Up Survey
-
=
Impact of NSP
If treatment and control villages are identical at
Difference-in-Difference
Difference Estimates
Estimates
baseline . . .
Differences estimation is accurate because . . .
250 treatment villages selected randomly from 500
study
is
a
randomized
control
trial
surveyed villages - other villages to control group
Partitioned
Randomization: FPs denoted 15 villages
Village A
B
to be excluded from randomization and evaluation
Treatment Group
(NSP)
Randomization embedded in FPControl
contracts
Group
(Non-NSP)
Partitioned
Randomization
Contractual
Embedding
Due
to funding
constraints&and
lack of village-level
Improved
Chance was
of Successful
Randomization
data,
randomization
fairest way
to decide which
villages received NSP
Interim Estimates of
Program Impact
• Local Governance
• Access to Services
• Economic Activity
• Social Cohesion & Conflict
• Gender
• Political Attitudes
Local Governance
Creates Functional Councils which Assume Authority
over Different Aspects of Local Governance
Creates Avenues for Women’s Participation;
Increases Responsiveness to Women’s Needs;
Increases Satisfaction of Women with Village Leaders
Increases Participation by Men in Local Governance;
No Evidence of Impact on Men’s Satisfaction with
Village Leadership
Access to Services
Increases Access to Safe Drinking Water
Increases Access to Electricity
Weak Impact on Transportation (at this stage)
Mixed Evidence of Impact on Medical Care
No Evidence of Impact on Boys’ Schooling (but
Possible Increase in Girls’ Enrolment Rates)
Economic Activity
Improves Perceptions of Economic Situation and
Optimism
No Evidence (at this stage) of Impact on Household
Income, Consumption, Assets or Borrowing
Limited Impacts on Production
Social Cohesion & Conflict
Decreases Out-Migration of Households
No Evidence of Impact on Attacks, but Improves
Perceptions of Security
No Evidence of Impact on Disputes & Feuds
No Evidence of Impact on Trust among Villagers
Gender
Increases Involvement in Economic Activities, but
Doesn’t Affect Involvement in Decision-Making
Increases Inter-Village Connections and Availability
of Support Groups
Increases Respect for Women and Acceptance of
Female Participation in Local Governance
Decreases Incidence of Extreme Unhappiness among
Women
Political Attitudes
Connects Villages with Central Government
Improves Perceptions of Government & NGOs
No Evidence of Impact on Acceptance of Government
Jurisdiction, but Increases Support for Electing
Local Leaders
Access
to Services
Local Governance
Social
Cohesion
Economic
Activity
Political
Attitudes
Gender
Perceptions of Gov’t, Civil Society, and Military
Male Villager Believes Official Works for the Benefit of All Villagers
+ 4%
+ 6%
+ 5%
+ 7%
+ 5%
80%
76%
71%
65%
65%
59%
+ 4%
+ 5%
71%73%
69%
64%
56%
50%
68%
64%
+ 3%
51%
46%
29%
26%
President
Provincial
District
Members of Central Gov't Government
Governor Administrator Parliament
Officials
Judges
Police
NGO Workers
ISAF / US
Army Soldiers
Test
of Two DifferentInterventions
Types of CDC (STIs)
Elections
Sub-Treatment
NSP-II
impact
incorporates
two STIs which
STIs
testevaluation
different
implementation
test changes in two program components:
strategies or changes in program design
Community
Development
Council (CDC)
Election Type
Method of
Sub-Project
Selection
STIs provide real-time evidence-based
feedback on how to improve program
effectiveness
Test
Effect
of Two
of Different
DifferentTypes
TypesofofCDC
CDCElections
Elections
Treatment Village A
B
125
Villages
Neighborhood
Election
Villages
Election
250 treatment villages randomly assigned to
Neighborhood
250 Election: One male & one female from each
elect
CDC
either
by
neighborhood
or
Treatment
→
Guarantees
representation
neighborhood elected to CDC
Villages
village election
125
Village
Village Election: Highest male & female vote-getters in entire
village elected to CDC → Preferred candidates are elected
Test
Effect
of Two
of Different
DifferentTypes
Typesof
of CDC
Project
Elections
Selection
Neighborhood
Election
85%
Village
Election
60%
97%
95%
. . . on Electoral Competitiveness
82%
67%
48%
72%
75%
81%
64%
51%
Village Elections (compared to Neighborhood
Elections) increase electoral competitiveness
Test
Impact
of Two
of Different
DifferentTypes
TypesofofProject
ProjectSelection
Selection
125
Villages
Consultation
Meeting
Villages
Referendum
250 treatment villages randomly assigned to
Consultation
Meeting:
Villagers
hold
meeting and decide
250
select project
either
by
consultation
meeting
Treatment
collectively
which project is best → consensus-based
Villages
or secret-ballot125referendum
Secret Ballot
Secret Ballot Referendum: Villagers select projects through
selecting preferred project → directly democratic
Test
Impact
of Election
of Different
/ Project
TypesSelection
of Project
Interactions
Selection
Stage of Sub-Project Selection
Sub-Project
Baseline
Survey
asked
male
villagers,
female villagers,
Demographic
Selection Method
Proposal
Selection
Prioritization
and village leaders which village
projects
they preferred
~
+
+
Male Villagers
Obtained
information
FPs~do
on not
which
projects
were+
Preferences
ofReferendum
femalefrom
villages
influence
proposals,
+
proposed, selected,
and prioritized
for implementation
selection,
or prioritization
Meeting
+
+ Selection
+
Village
. Leaders
. . on Elite Capture of Project
Referendum
~ influence
~
Preferences
of male
villagers
and
Compared projects
preferred
by
different~selection
groups with
Meeting
prioritization
in both
meetings
and~referenda
~ prioritized
~
Preferences
of village
leaders
influence
proposals,
Female
projects
proposed,
selected,
and
to find out
Villages
selection,
andReferendum
prioritization
in~process
meetings, but
~ not referenda
~
who
is influential
in selection
Meeting
Preferences
of village
leaders groups
more able
influence
Compare
influence
of different
undertotwo
selection
in consultation
(but influence
of
selection
processes
to find outmeetings
how selection
type affects
elite capture
villagers in affecting selection is not affected)
Test of Election / Project Selection Interactions
Randomization
of Election
Type
250
Treatment
Villages
125
62
Villages
Neighborhood
Election &
Neighborhood
Consultation
Election
Meeting
125
Villages
63
Villages
Village
Neighborhood
Election&
Election
Referendum
Villages
125
Villages
Consultation
Meeting
125
62
Villages
Village
Secret Ballot
Election &
Referendum
Referendum
250
Randomization
Separately
Treatment of Election Type and Project Selection Type Done
Village
Villages
Election &
63
Randomization
of Project
Selection Type
250
Treatment
Villages
Four Randomly-Assigned Combinations of Election and Selection Type
Use to Test Interactions between Election and Selection Type
Impact of Combinations on Selection Outcomes
Instrument
Combination
Proposal
Selection
Prioritization
~
+
+
Neighborhood
Election / Referendum
~
+ prioritization
~
Villager
leaders
influence
proposals, selection,
and
Male
Villager
Village Election
/ Meetingare combined
when village
elections
~ with consultation
+
~
Village Election / Referendum
meetings ~
~
+
Difference between Types
~
~
~
. . . onNeighborhood
EliteElection
Capture
of Project
Selection
/ Meeting
~
~
~
Neighborhood
Election
/ Referendum
~ and prioritization
~
~
villagers
influence
selection
Village Male
Leaders
Village Election
/ Meeting and referendum
+
+
Combinations
of election
type+do not affect
Village Election
/ Referendum of male ~
influence
villagers ~
~
Difference between Types
+
+
+
Neighborhood Election / Meeting
Probability of Elite Capture Maximized by Combination
of Village Elections with Consultation Meetings
Problems with Solutions
CDD is interesting b/c it can change behaviors, attitudes,
institutions, social cohesion etc. Very difficult to measure well
→ invest in extensive pilot-testing and consult widely
‘Parrot Bias’ in Surveys: Does program really change attitudes and
behaviors or does it just responses to survey questions?
→ measure actual behaviors rather than simply asking questions
Randomization: theoretically easy, practically difficult
→ tweak procedure to be resistant to pressures & communicate
→ successful quasi-experimental eval. better than failed RCT
Prospective Evaluations take ages . . .
→ STIs can provide (quicker) real-time feedback to program
→ Manage expectations and communicate . . .
Issues That Remain
Independence of Researchers vs. Accountability of Evaluations
to Programs, Implementers, and Donors
Importance of Non-Results vs. Lack of Interest and Lack of
Incentives for Researchers or Program to Disseminate
Small proportion of programs are subjected to prospective IEs.
How do we ensure the evaluated ones aren’t punished for non- or
bad results?
Who will synthesize IE results for policy-makers and make sure the
results are used in policy decisions and program design?
Who will coordinate researchers to make sure evaluations address
questions useful for policy decisions and program design?
w w w. n s p - i e . o r g