Presentation prepared for DAC, 15th November 2006


The general evaluative questions proposed by the IDB’s ex post policy, approved in 2003, are (i) “…the extent to which the development objectives of IDB-financed
projects have been attained.” and (ii) “… the efficiency
with which those objectives have been attained” (para1.1 )

Policy left for practice: sampling, methodology, organizational framework, and the forum for the presentation of results.

Note the task is to evaluate already approved and/or closed projects ( average project time is about six years).


Project Sampling Strategy: Option: random or meta-evaluation.
Decision :meta-evaluation.

Method and Project types: (i) process cum naïve or treatment (impact) evaluations. Decision Treatment effect evaluations; and (ii) projects with partial or national coverage. Decision partial coverage models

Organizational. Decisions: (I) separate activity within the office; (ii) evaluations to be carried out both in-house and outsourced.Therefore:
(I) hired staff with appropriate expertise; and (ii) created EVALNET, a register of evaluators;

Forum for presenting results. Decision: overall report (sent to the
Board) with background-working papers (discussed in ad hoc seminars).

 what were the problems that the program was designed to tackle?
 what was the policy response, i.e. the design features of the program? (theory based evaluation)
 was the program of a sufficient size given the size of the problem(s)?
 were the program’s deliverables provided in a cost efficient (and cost effective manner)?

What was the incidence and was the program well targeted?
 what was the impact on welfare outcomes of the program?;and
 what were the benefits relative to the cost of the program ?


(I)
(II)
(III)
To answer the question OVE normally use three approaches in the same evaluation:
Naïve evaluation
Regression based (cross-section and panel)
Treatment effects

4
3
2
1
0
(1)
(2)
(3)
(4)

Profile:
– Social Investment Fund. Panama
– Basic Infrastructure to poor communities
Social Investment Fund - Panama

Data:
– Distribution of benefits by municipalities from administrative data
– Baseline and results of outcome indicators from households surveys
1994-2001
Naïve Impact


Technique:
– Treatment and comparison group using
PSM in double difference. The sample included 75 municipalities.
– Potential to work with a sample of more than 250 smaller geographic units but household survey was not representative at that level
Results:
– Naïve evaluation: the program failed.
Impact evaluation: the program succeeded

14
12
10
8
6
4
2
0
(2)
(4)
Employment rate
Labour Training - Dominican Republic
Monthly income
Hours worked per week
Hourly wage Health insurance in primary job

Profile:
– Labor Training program – Dominican
Republic

Data:
– Simple randomization including a followup survey done at 10-14 months after graduation from training
– 786 treated and 563 controls
– Baseline has universe, follow up was a stratified random sample (size determined by standard formulas)

Technique:
– Estimated average Intention-to-treat on treated by simple diff of means, verified with weighted diff and regression analysis (no DD b/c faulty baseline)

Results:
– Employability, income and health insurance access increased. Program succeeded

Poverty incidence
Indigence incidence
Ocupation ratio
Education: School attendance
Health: Child undernourishment
Household Completeness
Quality of the dwelling
Overcrowding
Electricity access
Sewerage connection
Access to potable water
(5) 0
Progressive Housing Phase I - Chile
5 10 15
Absolute change (%)
20 25 30




Profile:
– Progressive Housing Phase I – Chile
– Provision of low cost basic dwellings to poor families
Data:
– Household Surveys identified beneficiaries and applicants to the specific housing program
Technique:
– Treatment from beneficiaries and comparison from applicants using PSM.
Single difference from a sample of 508
Beneficiaries and 476 applicants
Results:
– Quality of dwellings improved
– Little or not change in other welfare outcome indicators.
– Difference between naïve versus impact

1,200
1,000
800
600
400
200
0
Quality of dwellings by household income and Progressive Housing benefits - Chile
0.0
Improving dwelling quality
All non-beneficiaries
PHP-I Beneficiaries
0.2
0.3
0.5
0.7
Quality of the dwellings (Composite Index)
0.8
1.0

Profile:
– Progressive Housing Phase I – Chile
– Provision of low cost basic dwellings to poor families

Data:
– Household Surveys identified beneficiaries and applicants to the specific housing program

Technique:
– The benefits of the program are the additional (necessary) household income required to obtain equivalent dwelling

Results:
– IRR: greater than 18%
– Benefits: Net present value per solution
~1150 US$

80
60
40
20
0
(20)
140
120
100
Short-term
Motorized Roads
Rural Roads - Peru
Medium-term
Per capita income per year
Per capita consumption per year
Short-term Medium-term
Non-motorized Roads

Profile:
– Rural Road – Peru
– Construction and upgrade of roads in rural areas

Data:
– Specific survey of beneficiaries.
Baseline collected after program started. Follow-up survey 3 years after program closed

Technique:
– single difference and double difference

Results:
– Positive impact on income and assets’ values of rural households.
– Decreasing impact for motorized roads not for non-motorized roads.

2
1
0
(1)
4
3
(2)
(3)
(4)
(5)
-8
-6
-4
-2
0
-14
-12
-10
-20
-18
-16
2
Accumulated impact by level of per capita investment - Chile 
Profile:
– National Fund for Regional Development
– Decentralized investment to finance infrastructure and productive projects
3 4 6 9 11
Per capita investment ratio respect control group
Impact by treatment type - FNDR Chile
Education intensive investment
Composite welfare indicator (Index)
Poverty Incidence (%)
17
Diversified investment
25

Data:
– Administrative data for distribution of benefits by municipalities
– Baseline and results of outcome indicators from households surveys 1994-2001. The sample included 343 municipalities.

Technique:
– Impact evaluation using PSM in double difference.
The municipalities grouped by per capita investment using cluster analysis.

Results:
– Positive and increasing impact on poverty incidence (reduction) on per capita investment
– Not impact on poverty if investment is intensive in education
– Greater impact on welfare composite index in municipalities with diversified investment

0 500 ranking
Rechazados
Predicion_rechaza do s
1000
Aprobados
Prediccion_aprobados
1500




Profile:
– Science and Technology – Chile
– Financing for R&D projects
Data:
– All projects that between 1988 and 2004 received the financial support of the program and a stratified sample of projects submitted to the program, which were not financed because they ranked below the threshold defined for being admitted to the financing.
– 2,936 different research projects (932 financed by the
FONDECYT and 1704 not financed) 4,959 publications recoded in the ISI – SCI (1873 by financed researchers and 3806 by not financed researchers).
Technique:
– Discontinuity regression design. The selection process drawn by a “threshold” quality value that separates beneficiaries from non-beneficiaries
Results:
– Unsuccessful. FONDECYT has no significant positive impact on the scientific production of the financed projects.

Employment
Sales
Exports
Productivity (TF)
Patents
Crowding out
Chile Argentina Brazil
+, *
+,
+,
+,
+, *
+, *
+, *
-, *
+,
0,*
-,
-, na
0, * na
-,
+,*
+, *


Profile:
Public grants-credits to firms for innovation
Data:

Administrative data on firms and firm level surveys (OSLO design)
Technique:
Double difference with propensity score matching

Results
Generally positive and significant effects on employment, and sales, but little evidence of effects on patents and total productivity .

0.1
Impact on infant mortality
0
-0.1
-0.2
-0.3
-0.4
All Sample At least Primary
-0.5
Bottom 25% 25%-50%
Expenditure level
50%-75% Top 25%

Positive effect on health outcome
(treatment less than naïve effect)
 heterogeneity of results important. a regressive relationship between treatment effect and income, where more educated (and wealthier) households did better than less educated (and poorer) households

Ramification for project design: projects should include or be coordinated with, as a hypothesis to be tested, a health education component together with potable water expansion.



But
Since 2004 have produced about 23 evaluations
Cost per evaluation was about $60,000



Problem of obtaining effective counterparts (in Bank and country) to accompany the evaluation from beginning to end. Started outreach program to obtain formal counterparts in the country, and form ad hoc interested specialist for each thematic study.
Mainstream impact evaluations into other evaluations of the Office
Problem of communicating the findings. Started producing different reports for different audiences for the same evaluations.

Before After


Panel data

Lny it
 a
0 , i
  i
Lny i , t

1
 
D it
 j m 
 a j , i
LnV i , t j
 j m 

 b j , i
LnV i , t j
  ij

Cross section data
Lny it
 a
0 , i
 
D it
 m  a j , i
LnV i , t j    ij j

Where y is the outcome of interest, D is the dummy for participation in the program, V control variables