From impact evaluations to
cost-benefit analysis?
DEC Course Poverty and Inequality - Module 6:
Evaluating the impacts of assigned programs
March 28, 2008
Jed Friedman
Development Research Group
Outline
Conceptual framework
 Challenge of accounting for ‘full’ benefits
and costs
 Evidence C/B education programs
 Broader comparison across programs:
C/B nutrition interventions

Cost benefit analysis

Traditional cost-benefit analysis focuses on
‘efficiency’ arguments: determine the effect of a
program φ on total output or income
B( )   Yi ( )
i
And compare it to its costs C(φ)
Cost benefit analysis

Note that:
1. Distribution of program impact are irrelevant
(costless lump-sum transfers/taxes)
2. Ignore problem of aggregation of individual
valuation of the outcomes
 Y ( )  W ( )  W [U (Y ( )),..(U (Y
i
i
1
N
( ))]
Cost benefit analysis
3.

Generally ignore distributional aspects of policies
(imp. for programs with redistributive objectives
or with altruistic motivations)
Initial position in the distribution does not matter
Social justice/positive political economy might be
interested in:



Gains to specific groups of population at base
state (ex. poor)
Proportion of people who gain relative to their
base state
Proportion of people who reach an absolute level
of outcome (ex. social values of merit goods,
health and education)
Cost benefit analysis
4.
Full social costs of the program should include
the welfare cost of raising funds, generally
ignored.
General framework




Consider a vector of policy variables φ, with social
cost c(φ)
N1(φ) and N0(φ) is the number of people in each
state {1, 0}
Assume constant relative prices
Total output:
N1 ( ) E (Y1 | D  1,  )  N 0 ( ) E (Y0 | D  0,  )  c( )

Now consider a marginal change in φ. What is the
corresponding change in output?
Conceptual framework
(1) persons induced to change state/sector:
“entry effects”
N1 ( )
( ) 
[ E (Y1 | D  1,  )  E (Y0 | D  0,  )]

E (Y0 | D  0,  )
E (Y1 | D  1,  )

N1 ( ) 
N 0 ( )


c( )
(2) change outcomes

within state/sector
 (3) marginal social
(direct + indirect)
cost
What does the conventional
evaluation parameter measure?
1. Entry effects generally ignored
2. Indirect effects are also generally ignored
(spillover/substitution, GE on prices)
3. Does not incorporate costs
The conventional parameter ATT
[ E (Y1  Y0 | D  1,  )]
does not correspond to a marginal change and
includes the gains accruing to participants…
What does the conventional
evaluation parameter measure?
1. Entry effects generally ignored (Moffitt 1992,
2003, Heckman 1992):

effect of the program on the rate at which
individuals apply or gain admittance to the
program.
 Direct: policy changes the way applicants
are handled
 Indirect: policy reform changes the
attractiveness of the program
What does the conventional
evaluation parameter measure?
2. Indirect effects are also generally ignored
(spillover/substitution, GE on prices)

Program evaluations poorly designed to
estimate the effect of system-wide structural
reform which alters the entire environment.
What does the conventional
evaluation parameter measure?

However, evaluation parameter still informative
under special conditions ..:




No GE effects, with partial participation
When benchmark/counterfactual state is ‘no
program’
.. about the “gross” gains from the existence
of a program at a given level φ1 relative to
shutting it down (all or nothing)
Not appropriate for ‘fine tuning’/marginal changes,
nor for economy-wide reforms → “Policy Relevant
Treatment Parameters”
Challenges in getting good
estimates of benefits





Impact may accrue over the life-cycle
Place value on all relevant impacts
Pricing of impact might be distorted by market
failures
Distinguishing between private and social costs
(central for the efficiency motive for policies)
Selecting the appropriate discount rates to
obtain the net present value of future stream of
benefits
Challenges in getting good
estimates of costs



Account for true private and public cost,
including distortions costs (and not nec. budget
costs)
Assessing the cost when a project is scaled-up
(marginal vs average in the presence of
economies of scale). Reasonable to assume
they are similar in the long run
Distinguish private vs public costs
1. C/B analysis of a voucher
education program in Colombia

Vouchers for private secondary schooling in
Colombia (PACES) – Angrist et al 2002






Offered to low income families residing in poor
neighborhoods and attending public schools
Value voucher: US$190 per year, ages 13-15,
equal annual installments
average monthly fee for private schools US$340
Average per pupil cost public secondary school
slot US$350
Assignment by lottery: natural policy experiment
Enrolment primary almost universal, enrollment in
secondary 73% overall, 55% in poorest quintile
1. voucher education program:
impact estimates


Randomized assignment → compare lottery
winners/losers (OLS, OLS with controls) recover
an intention-to-treat (ITT) parameter
Take-up: model scholarship use.
 Lottery losers were awarded other
scholarships: alternative financing (24%)
 90% lottery winners used the scholarship
 Model selective compliance with 2SLS and
recover the average treatment-on-the-treated
(ATT) parameters
1. voucher education program:
impact estimates

Lottery winners:






No effect on enrollment
15% more likely to attend a private school
0.12-0.16 additional grades completed (mainly
due to reduced repetition (-5/6%))
No effects on drop-out rates
0.2SD higher test scores
10% more likely to have completed 8th grade
1. C/B voucher program
Estimated cost on hh’ld exp.

Total increase in hh’ld expenditures:


Increase in gross fees for winners $52 (greater
likelihood of attending private school + spend
voucher on more expensive private schools)
Increased opportunity cost: 1.2 fewer hours
working, average h/wage 71c: 1.2x.71x48
weeks=$41 lost earnings
Lottery winners receive on average $74 more
aid than losers
extra net expenditure per lottery: 52+41-74=$19

1. C/B voucher program
Estimated cost on gov. budget


Lottery winners receive on average $74 more aid than
losers.
Reduced public schools costs:
 lottery winners no difference in likelihood of
attending school, but 0.14 less likely to attend
public school $350*.14~$50
extra public expenditure per lottery: 74-50=$24
1. C/B voucher program
Estimated total cost



Estimated public + private costs $19+$24=$43 per
year
Administrative + distortionary costs assumed 30%
increase in gov.expenditure
Costs incurred over three years
1. C/B voucher program
Estimated gains

Gains to lottery winners:
1.
2.
3.

0.12-0.16 additional grades completed
0.2SD higher test scores
10% more likely to have completed 8th grade
Valuing gains (1):



Estimated return to 1 year schooling ~ 10% 
program increases winner wages by 1.2% per
year
Annual earnings parents $2,400 per worker (with
lower education levels) ~ $3,000
Annual benefits occur at ages 16-60
1. C/B voucher program

Net present value (Berhman, Knowles 2004):



Benefit to cost ratios:
at 3% discount: 3.8
 at 5% discount: 2.7
 at 10% discount: 1.4
Note:
 Conservative: Adding gains from completing 8th grade
+ increased tests scores brings B/C ratio from 2.7 to
3.9 (at 5%)
 Heavy discounting (benefits spread over large period)


discount rate is 3%, 5% or 10% per annum.
Benefits (of gains in grade completed) and costs over
life-cycle discounted to age 13
2. C/B analysis ECD
intervention in Bolivia





Daycare, nutritional and educational services 6-72
months old –poor urban areas Bolivia (Berhman,
Cheng, Todd 2004)
Child care centers at women’s houses (training
+loans/grants to upgrade homes)
Up to 15 children
Longitudinal information on participants +
representative sample children same age group in
program and non-program areas
Cross-sectional, matching, marginal PSM
2. C/B analysis ECD
intervention in Bolivia: costs



Cost: estimated $43/month child enrolled
(516/year)
Costs presented with/without 25% upward
adjustment for distortionary costs of government
financing
Costs calculating assuming that children take
part to the program for 3 years (age 2-5)
2. C/B analysis ECD
intervention in Bolivia: benefits

Benefit: use impact estimate of:




height gains (2%)
Shorten length time to complete education (1year)
Increase cognitive skills by 5%
to calculate long-life earnings via:



Increasing adult physical stature (1% increase
height →2.4% increase wages (urban Brazil,
Strauss Thomas 1997)
Increased number grades completed
Increased cognitive skills (conditional on grades
completed) (1% increase cognitive skills →0.233%
increase earnings, Alderman et al 1996)
2. C/B analysis ECD
intervention in Bolivia


Implied benefit to cost ratio range from 1.7 to 3.7
Largest contribution gains is increasing the number of
grades completed
2. C/B analysis across programs:
hunger and malnutrition in LDCs



Copenhagen Consensus 2004
Objective MDG1: halving prevalence underweight
rates by 2015
Income alone insufficient: assuming a stable annual
economic growth rate of 5%, growth would result in 0.5%
yearly reduction in malnutrition

Not a single intervention: broad overview of
different groups of interventions aiming at:




Reducing prevalence low birth weight
Infant and child nutrition and exclusive breastfeeding
Reducing the prevalence of micronutrient
deficiencies (iron, vitamin A, iodine, zinc)
Investing in agriculture technology
3. C/B analysis
hunger and malnutrition in LDCs


Attach a value to the benefit of averting 1 child from
being LBW from not LBW
Ex. Adverse outcomes for low birth weight (LBW:
birth weight<2.5 kg)





Excess mortality (40% more likely to die in the
neonatal period)
Higher likelihood of being stunted
Cognitive and neurological impairment → lower
earnings and productivity as adults
Higher vulnerability to diseases (infant and lifetime)
Intergenerational transmission (LBW more likely to
give birth to LBW children)
3. hunger and malnutrition:
Accounting for ‘full’ benefits


Ex. 1: Estimating benefits from reducing
incidence of LBW on infant mortality
Do not attach value life saved



Study from Guatemala, India: risk neonatal
mortality for LBW 4 times higher than 2-2.9kg, 10
times higher than 3-3.5kg
Value: $800 cost save life through measles
immunization in early 1990s adjusted for inflation
and distortionary taxation=$1,250
Value to be adjusted for risk death for LBW gives
estimated gain of about $93
3. hunger and malnutrition:
Accounting for ‘full’ benefits

Ex. 2: Estimating benefits increased lifetime
productivity through decreased stunting




UK long-term follow-up study estimates 1,000 gr
less than normal birth →loss 0.5SD height (1.6 cm
or 1% studies identical twins in the US)
1% increase height →2-2.4% increase wages
(urban Brazil)
Assumed annual earnings $500 a year in constant
prices over a work life 15-60 years old
PDV at 5% discount rate of about $99
3. hunger and malnutrition:
overview PDV different benefits
3. hunger and malnutrition:
overview PDV different benefits
3. hunger and malnutrition:
review various types of
intervention




Overall benefit per LBW averted: 580-986$
60% benefits accounted for increased productivity
Review costs of different types of interventions
addressing LBW (ex. treatment women with bacterial
infections, treatment women with STD, drugs pregnant
women with poor obstetrics history)
Calculate implied benefit to cost ratios ranges
3. hunger and malnutrition:
Conclusions



Essential to account for full benefits and costs
Awareness of the assumptions underlying costbenefit analysis and impact estimates
Parameter generally estimated in the evaluation
literature do not lend themselves to marginal
changes in policies


Recent literature on distributional treatment
effects
Ex-ante simulations. And structural modeling