The evaluation of the Familias en
acción programme in Colombia:
do conditional subsidies improve
education, health and nutrition
outcomes?
Alice Mesnard
EdePo , IFS
Outline
1. the Familias en acción Program
2. how do we evaluate its effects ?
Methodological issues and data
3. Results
Education, health and nutrition in poor households
4.What can we conclude ?
significant effects on development of children in
Colombia
Is it enough to assess the effectiveness of the program?
Familias en acción
• Familias en Accion is a Conditional Cash Transfer program. Its
final aim is to foster the accumulation of human capital.
Why is it important ?
• CCT provides monetary incentives for families to invest in
human capital (education and health)
• Conditional cash transfers are now being implemented in:
•
Mexico, Honduras, Nicaragua, Panama, Brazil, Argentina, Colombia,
Turkey, Bangladesh
• In Colombia, the program is targeted towards the 20% poorest
households living in towns with less than 100k people, with
enough education and health infrastructure, and with a bank
•
•
Within each villages 20 % poorest hhds how?
indicator summarises living conditions (has a toilet, running water,
quality of walls, income..)
Create Sisben levels 1 to 6 : take only Sisben 1 hhds = poorest of the
poor
Need to be registered in the village in dec 1999
Infrastructure :
villages with banks otherwise transferring cash is dangerous ,
complicated
Schools : aim is to evaluate enrolment in schools. Assumes that
schools are supplied
Familias en acción
• The program is made of two components: Education and Health
• Education:
• Eligible families with children aged 6-17 receive a subsidy
per child conditional on school attendance.
• The subsidy is about 10US$ per month for primary school
children and 20US$ for secondary school children
• Nutrition and health:
• Eligible families with children aged 0-5 receive about 30US$
per month conditional on registering the children on growth &
development checkups
• Mothers are also ‘encouraged’ to attend some ‘talks’ with
health professionals
• All subsidies are given to the mothers
Political context
In 2002 the programme started, financed by World Bank, Inter
American Bank for Development and the Colombian govnt
Pgm is viewed as a replacement of Hogares Communitares, which
has been providing free child care and subsidised food to
children of all households since 20 years
The loan is running until next year : if gvt wants to continue it has to
refinance it alone: how ?
Maybe by cutting pre-existing program
He needs to evaluate the FA programme
Main features of the beneficiary families
• Average family size: 7
• Average monthly consumption 150US$
• Including consumption in kind
• Food consumption accounts for about 60% of
consumption
• 85% of households report consumption in
kind
• Under civil war
Age structure
Treatment
municipalities
800
700
600
500
400
300
200
100
0
0
4
8
12
16
20
24
28
32
36
40
44
48
52
56
60
64
68
72
76
80
84
88
92
96
100
Focus on children development
Children 0 to 6 years old
• Chronically malnourished: 23.7%
• Diarrhoea in 15 days: 17.1%
• Symptom of respiratory disease: 44%
• Compliance Growth and Dev checkups: 30%
School enrolment
School enrolment by age and type of town
100,00%
80,00%
60,00%
40,00%
20,00%
0,00%
7 a 9 años
10 a 11
12 a 15
16 a 17
control con pago Tratamiento con pago Control sin pago Tratamiento sin pago
2. The evaluation of Familias en Accion
• The evaluation will be based on the comparison of outcomes of
interest between individuals living in treatment towns and those
living in control towns
these outcomes are : enrolment in schools, food consumption,
anthropometrics, self assessed symptoms, compliance with G &
D check ups
• Idea of an experimental set-up
Methods: Potential outcomes model
Yi  denotes outcome if treated
1
Yi  denotes outcome if non  treated
0
T  1 denotes living treatment municipali ty
t  1 denotes post treatment data
Effect  E (Y 1i | T  1, t  1)  E (Y 0 i | T  1, t  1)
Why is it difficult to measure ?
 We do not have the counterfactual E (Y i | T  1, t  1)
Treatment municipalities have received the
program
 Can we consider instead the outcome in control
municipalities ?
It is important to consider the existence of preexistence differences between treatment and
control municipalities.
Differences in outcomes reflect pre-existing
differences or effect of different variables
0
Methodology in a nutshell
To take into account pre-existing differences and differences in other
variables we use a combination of two standard evaluation techniques:
Method 1 : Selection on observables (parametric and non parametric)
Method 2 : Difference in difference
effect of the program= differences in school enrolment between treated
and non treated children minus pre-program differences
We need a baseline database and a follow up one
The data
 Sample: 11,500 households in 67 treatment and
62 control municipalities
 Baseline collected between June and November 2002
 First follow up collected between July and December 2003
 Complete household survey. Info about all household
members. Household survey is quite long: about three hours
and a half
 Questionnaires to household heads, mothers etc....
 Survey to major, health care centres, schools, nurseries
 Good quality of data : attrition =6%
Method 1 Selection on observables
• Suppose:
• Y=b X + c D + u
Where X are some observable variables, u is an unobserved random
variable and D=1 indicates that the program is on (so that is 1 in
treatment towns and 0 in non treatment towns).
• Assume that u is orthogonal to D conditional on X.
• This implies that after controlling for observable household and town
characteristics the program is assigned randomly
• What is the effect of the program ?
• In this case, we can estimate the effect of the program (c) by pooling
treatment and control households and estimating the equation above.
Selection on observables (continued)
• We can even generalize the technique above to take
into account in a flexible way the effect of X and D
• The general idea is to compare treated households to
control households that are similar in the X’s .
• Generalisation if there are several variables :
summarised by a score of been treated
Let Prob{D=1 | X} = p(X)
p(X) is called the
propensity score
We compare the outcomes between households with
close propensity scores.
This is called the Propensity score matching method
Variables included in the Propensity Score (to match)
Head and spouse, household composition (age, education, height,
number of brother and sisters, age composition)
Household infrastructure( water by pipe, gas by pipe, connection to
sewage, rubbish collection, telephone, house walls materials,
house ownership, household has suffered shocks related to
illness, death or violence in the last three years)
Municipality level variables (region of the country, altitude, index of
quality of life of the municipality, population size, number of
schools, number of students per teacher, number of square
meters in the school per student, presence of public hospital in
the municipality, number of health care centres, number of small
health clinics, number of pharmacies, percentage of households
with water and sewage by pipe, occurrence of strikes/taskforce
desertion in the health care providers of the municipality)
Method 2 Difference in difference
• The 2 methods above assume that conditional on the X’s
treatment and control villages are statistically identical.
• Of course it is possible that there is some unobserved
characteristic of the town that is related to the program and
will confuse the effect.
i.e. Y=b X + c D + v+u
with v correlated to D ?
• If such characteristics v are constant over time and we have
data pre and post program, we can use diff in diff methods
The effect of the program = the post program average
differences in school enrolment between the treatment and
control municipalities minus the pre program differences
Difference in difference effect
Y
treatment
effect
Control
t=0
t=1
Example : diff and diff effect
• Before the program, school enrolment rates
In treatment towns : 60%
In control towns : 50%
• After the program,
in treatment towns : 80%
in control towns : 60%
What is the effect of the program ?
Answer
10 %
Diff and diff effect = (80% - 60%)-(60%- 50%)
3 Evaluation Results
 In most cases, we look separately for urban and
rural parts of municipalities
 More important results
 School attendance
 Household consumption
 Nutrition
 Health
 Other results : child labour , migration
See the report on the IFS website
www.ifs.org.uk/edepo/
Table 1. Impact of FA on percentage of children who attend school
With FA
Impact
Without FA
Rural
Aged 8–11
93.0%
93.1%
0.1
Aged 12–17
46.2%
56.3%
10.1*
Aged 8–11
95.2%
96.6%
1.4
Aged 12–17
68.5%
73.7%
5.2*
Urban
* Statistically significantly different from zero at the 95% confidence level.
Impact on school attendance
 Higher impacts in rural parts
 Higher impacts on older children in rural parts
 There are also positive impacts on older children living in
urban areas
 Unsignificant effects on young children.
if the aim is to increase enrolment in primary school :
wasted money . redesigning pgm ?
 Other issue : nothing on quality of school ?
We know repeated school
 Child labour : effects in terms of number of hours
Probably school and work are not perfect substitutes
Table 2. Impact of FA on total consumption and on food
consumption (in Colombian pesos)
With FA
Impact
Without
FA
Total
consumption
Rural
450,343
538,057
87,714*
Urban
477,460
521,846
44,386*
Rural
279,042
349,213
70,171*
Urban
254,767
295,041
40,274*
Food
consumption
* Statistically significantly different from zero at the 95% confidence level
Impact on consumption
 Household consumption increases in about 15%
most of it due to food (+20%) : not very interesting : poor people
increase consumption if their income increase
 More interesting results on food components:
The ones that increase more are protein rich foods (meat,
chicken, milk and eggs).

Apart from food, there are increases in:
Clothes and shoes for children (12,000 pesos)
Education (in the urban area) (8,000 pesos)
No effect on alcohol, tobacco, or adult clothes.
Why is it important ?
Pb of answers ?
 Very interesting results : FA improves nutritional
outcomes
Table 3. Impact of FA on selected components of household
consumption (in Colombian pesos)
Rural
Urban
Meat, chicken and milk
21,831.4*
21,717.2*
Potatoes, yucca and
other tubers
2,938.9*
4,133.1
Cereal
5,008.8*
9,094.6*
Fruit and vegetables
1,399.3
4,249.4
Pulses
313.6
2,008.4
Fat and oil
1,887.8*
3,139.4*
Sugar and pastries
1,234.6
647.2
Clothes and footwear for
men
–3,952.4
–2,090.4
Clothes and footwear for
women
–1,410.0
58.7
Clothes and footwear for
children
12,088.1*
11,634.2*
Healthcare
1,898.7
3,641.9
Education
8,005.5*
–610.7
Alcohol and tobacco
2,175.1
–1,184.2
Impact on number of days per week that children had…
Milk
Eggs
Chicken
Urban 24-48
months
0.23
(0.26)
0.56*
(0.34)
0.28**
(0.14)
Urban 49-60
months
0.81**
(0.21)
0.30
(0.21)
0.25**
(0.10)
Rural 24-48
months
1.07**
(0.26)
0.45*
(0.24)
0.37**
(0.08)
Rural 24-48
months
1.09**
(0.28)
0.84**
(0.21)
0.38**
(0.12)
*, 10%
**, 5%
Table 4. Impact of FA on percentage of children with up-to-date
schedule of preventive healthcare visits
With FA
Age
Without
FA
Impact
<24 months
17.2%
40.0%
22.8*
24–48
months
33.6%
66.8%
33.2*
>48 months
38.9%
40.4%
1.5
* Statistically significantly different from zero at the 95% confidence level.
Health Impacts
Significant effects
Probability that children adequately complied the
growth and monitoring visits according to the health
ministry guideline increased by 23 % for below 2
years old, and 32% for 2 to 4 years old.
Probability that children suffered diarrhea in the rural
part decreased by 10% in the rural part
Table 5. Impact of FA on percentage of children who
suffered from diarrhoea in the 15 days prior to the interview
Age
Without
FA
With FA
Impact
Rural
<24 months
32.6%
22.0%
–10.6*
24–48
months
21.3%
10.4%
–10.9*
>48 months
8.5%
7.0%
–1.5
<24 months
38.6%
23.6%
–15.0
24–48
months
16.8%
13.5%
–3.3
>48 months
12.3%
8.1%
–4.2
Urban
Table 6. Impact of FA on boys’ heights (in centimetres)
Age
Without
FA
With FA
Impact
12 months
72.70
73.14
0.44*
36 months
87.54
87.58
0.04
60 months
104.22
104.27
0.05
Effect on height
• 1/2 cm effect on young children only
• What about older children ? Effects take more
than a year to be significant
• Hope to evaluate longer term effects with next
follow ups (3 years effect)
4 Conclusions:
Is the program effective ?
• FA improves school enrolment among older children
• FA improves positively the nutritional status of
younger children, and the occurrence of diarrhoea in
the rural part of municipalities
• FA has a very large effect on the attendance to
preventive health care visits
other issues ?
•
•
Cost benefit analysis
Comparison with other programs
supply of schools, training of teacher etc…
Which ones are more effective ?
How can we compare their results ?
If we have similar data on children attending both programs ?
ex : Hogares comunitarios is a nutrition/child care program introduced all over
Colombia in the early 1980s
FA widely perceived as a substitute for Hogares
• Targeting issues : waste of money
• Political economy : whom do you want to help ?
example : FA & Hogares comunitarios: which
one?
• Hogares comunitarios is a nutrition/child care
program introduced all over Colombia in 1986.
• The program is targeted to poor people (sisben 1,2,3)
• Subsidized child care:
• Children 0-6 can attend for a small fee
• Receive food and child care from a ‘community
mother’
• FA widely perceived as a substitute for Hogares
Hhds have to choose between two pgms
Percentage of children attending
Hogares Comunitarios
Age
Girls
Boys
0
3
3
1
19
16
2
38
44
3
46
48
4
36
34
5
20
24
6
11
11
Hogares Comunitarios & FA
• Evaluation performed in the ‘control towns’
• Different methodology as the program is universal:
• Compare participants to non participants
• But take into account that participation might be
related to the outcome of interest
• Use distance as an ‘instrument’
• Startling results:
• HC has improves height of children 0-5 by about 2
cms.
• It has also long run effects on school achievement
• It increase considerably employment of mothers
• Are Familias and Hogares really substitute?
Interactions between HC and FA
• We know that HC benefit mostly those
children older than 24 months
• and FA seems to benefit most those children
younger than 12 months.
• HC and FA seem to be complements rather
than substitutes