July 2015
EDePo
Evaluating development policy at IFS
Issue
What’s inside
• Special feature: bottlenecks to
productive investment
p2
• Using structural models to
evaluate development policies
p6
• From the field: how big should
sample size be?
• Case study: taxation and public
spending in Brazil
p5
p7
03
Welcome to the
conference edition of
the EDePo Digest. In
this issue, we take
an extended look at
some of the work
being presented at our
summer conference:
‘Improving productivity
in developing countries:
identifying bottlenecks
and obstacles to
productive investments
and technology
adoption’.
Orazio Attanasio
and Sonya Krutikova,
Co-directors, EDePo
This conference is being held at
Goodenough College in London, on 8
and 9 July 2015, to mark the end of our
associated three-year Department for
International Development (DfID) and
Economic and Social Research Council
(ESRC) grant.
The conference is organised around
four core constraints to productivity:
informational resources available to
individuals; market failures; operational
resources; and social norms. Plenary
sessions will be given by Mark
Rosenzweig (Yale), Eliana La Ferrara
(Bocconi), Pascaline Dupas (Berkley)
and Imran Rasul (University College
London and Institute for Fiscal Studies).
The closing session of the conference,
led by the DfID’s Chief Economist, Stefan
Dercon, will focus on what it means to
carry out policy-relevant research.
Following the conference, the papers
presented, along with footage of the
plenary presentations, will be available
on the EDePo web site.
So what’s coming up inside? We
start with three articles looking at
how information, social networks
and credit constraints affect some
key determinants of welfare and
productivity. EDePo Senior Research
Economist Britta Augsburg looks at
the role of marketing, information and
credit constraints in the purchase of
toilets – key to reducing the impact of
water-borne diseases – in India and
Nigeria. Bansi Malde discusses her work
on social norms, networks, risk-sharing
and child health in Malawi. And Bet
Caeyers summarises her work on social
networks and knowledge about about
public programmes in Tanzania.
We also have an article on public finance
issues. Research fellow Lucie Gadenne
finds that locally raised tax revenues
appear to be better spent than central
government grants in Brazil.
EDePo co-Director Orazio Attanasio
explains the importance of structural
economic modelling to EDePo’s work
in our “Methodology Matters” column.
Finally, Marcos Vera-Hernández
discusses a key issue in collecting data
“From the Field”: determining the size
and structure of the sample needed.
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
01
Main Feature: Part I
Bottlenecks to productive
investment
Individuals in developing countries face many barriers and constraints that can
prevent them undertaking productive investments in their human, physical or
social capital. Understanding these constraints, and how they can be overcome, is
the focus of a series of projects being undertaken by EDePo staff, funded in-part by
a major ESRC-DfID grant.
Some of the biggest constraints include access to credit, and access to information.
In a series of 3 articles, Britta Augsburg, Bansi Malde and Bet Caeyers explain
some of our research that looks at these constraints, and the role family and social
networks play in overcoming them.
Is a toilet a worthwhile investment? Views
from India and Nigeria
Britta Augsburg
In addition to its importance
for public and private health,
safe sanitation benefits
individual households in a
number of ways, including
time savings, comfort, greater
safety and increased status.
In light of these personal
gains as well as the enormous
investment necessary to
meet the United Nations’
target of eliminating open
defecation by 2025, many
sanitation programmes aim
for households themselves to
invest in the construction of a
toilet.
In surveys conducted as part of three
sanitation impact evaluation studies
in India and Nigeria, researchers at the
IFS have asked over 7,000 households
why they do not own a toilet. The
overwhelming majority (>80%)
state that toilets are too expensive.
In a recent policy note, additional
detailed data on sanitation beliefs and
2
practices have been analysed with the
aim of understanding whether this
statement is driven by misperceptions
of costs and returns or by households
being liquidity constrained. Such an
understanding is important for the
design of sanitation interventions. If
the cost of toilets is seen as appropriate
given the expected returns (i.e.
households would be willing to invest if
they were able to), then policy makers
and development agencies should
focus on interventions that alleviate
liquidity constraints. However, if
households underestimate returns and/
or overestimate costs, any intervention
that aims at inducing sanitation uptake
should provide households with
information on the actual costs and
benefits.
The data provide compelling evidence
that liquidity constraints for private
latrine investments are substantial and
binding. Any sanitation intervention
that aims at increasing household
sanitation coverage therefore needs
to address such financial barriers.
Although the data also suggest that
households have a fairly accurate
understanding of the investment
costs and benefits, on average, many
households continue to place a fairly
low priority on sanitation investment.
This implies that information provision
should accompany interventions aimed
at alleviating financial constraints.
The IFS is currently conducting two
impact evaluations of sanitation
interventions that tackle both
informational and financial barriers
but use different approaches. In India,
the evaluation is designed to shed light
on the differential uptake of private
household sanitation when sanitation
micro-credit is provided and when this
credit is combined with a sequence
of ‘soft activities’ aimed at informing
and influencing potential customers.
In Nigeria, the implementing partner
combines the popular soft approach
of Community-Led Total Sanitation
(CLTS) , which focuses on triggering a
change in sanitation behaviour through
community participation events (and
associated social pressure), with
sanitation marketing. This is a more
commercial approach, which more
directly addresses both the demand
and supply of sanitation. The first
results of both of these studies are
expected to be available in around one
year’s time.
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
02
Main Feature: Part II
Social norms and the role of the extended
family in rural Malawi
Bansi Malde
The extended family is an
institution that plays an
important role in influencing
individual and household
choices in developing
countries, and through
this, their well-being. These
relatives are a key source of
information and resources.
For example, grandmothers,
aunts and sisters provide
information on how to care
for children, while siblings,
parents and uncles may
be a source of resources
(monetary and in-kind) for
investments, such as education
or businesses.
Moreover, social norms interact with
the presence of extended family
to affect the overall well-being of
individuals and households. Importantly,
such interactions may also affect the
effectiveness of policies targeting
specific behaviours and outcomes. Thus,
it is essential to understand how the
extended family influences various
choices and outcomes.
IFS researchers have been examining
how extended family networks
influence informal risk-sharing and
child health in Mchinji District, Malawi.
On risk-sharing, historical norms imply
that a woman’s brothers are likely to
play a key role in ensuring the wellbeing of her household. In contrast,
for child health, the extended family
is a conduit for traditionally accepted
practices and beliefs associated with
child-care practices, and may also be a
barrier to the uptake of interventions
aimed at changing practices and beliefs.
3
Extended family and risksharing
We consider how the size of the
extended family – defined as the
brothers and sisters of the head and
spouse – affects the degree of risksharing that households achieve in
response to idiosyncratic crop losses.
Economic theory indicates that two
forces are at play in influencing the
relationship between risk-sharing and
the size of the risk-sharing group. On
the one hand, larger groups are better
able to diversify risk and thus should
provide better insurance. On the
other hand, they are more vulnerable
to subgroups reneging and choosing
to form new smaller groups, which
could potentially reduce the amount
of insurance that a larger group can
provide (or even prevent such a group
functioning at all).
In Malawi, we find that the latter effect
outweighs the former: larger groups
provide less insurance and risk-sharing.
In particular, households where the
woman has many brothers – a group
considered to be crucial for informal
insurance given prevailing social
norms – achieve less risk-sharing. Such
a pattern is not found for sisters. This
suggests that the relationship found
for brothers is not simply the result of
larger families being poorer, but instead
reflects the strength of the family
network’s risk-sharing ability.
Extended family and infant
health promotion
In the second project, we exploit a
cluster randomised control trial to
analyse whether the success of a
home-visit intervention, to provide
pregnant women and new mothers
with information on how best to feed
their infant (including the importance
of exclusive breast feeding during the
first few months of life), is affected
by whether certain members of the
extended family are alive – relatives
who may provide information based on
tradition, rather than modern medical
practice.
Exposure to the intervention
significantly increased child height for
age z-scores, which is a good predictor
of long-term health. However, the
effect size was reduced if the paternal
grandmother was still alive. This could
be the result of a disagreement between
the beliefs of grandmothers and the
messages delivered by the intervention.
Maternal grandmothers do not have
any effect on intervention impact
(but appear to be associated with
some reduction in child height more
generally). Overall, the results point to a
highly influential role for grandmothers
on child health in this area of Malawi,
and suggest that the intervention could
be made more effective if it also sought
to inform and influence senior women,
rather than focusing exclusively on
mothers.
Fitzsimons, E., Malde, B. and VeraHernandez, M. (2015), ‘Group Size,
Coalitional Constraints and Informal Risk
Sharing’, IFS mimeo.
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
03
Main Feature: Part III
Social networks and development
programme awareness of vulnerable
groups in Tanzania
Of course, the characteristics of
neighbours – including whether they
are informed about TASAF – may be
endogenous: people may cluster in
neighbourhoods with similar people.
The paper avoids this problem by
using the exogenous characteristics of
neighbours’ neighbours as instruments
for the proportion of neighbours who
were well-informed. (This approach
was originally developed by Bramoulle,
Djebbari and Fortin (2009) and De
Giorgi, Pellizzari and Redaelli (2010).)
Bet Caeyers
There are a number of
challenges involved in
operating and evaluating
community-based
development (CBD)
programmes, which place
greater control of resources
and decision-making in local
hands.
One challenge, in particular, concerns
how information spreads through
communities: those who might
stand to benefit most from these
programmes might be least likely to
hear about them. CBD initiatives often
rely on village meetings and media
campaigns to sensitise communities
and to encourage participation.
However, vulnerable groups, such as
the elderly and disabled, may not have
(easy) access to such direct sources
of information, because of physical or
socio-economic constraints.
One such programme is the Tanzania
Social Action Fund II (TASAF II), which
supports vulnerable groups in rural
4
Tanzania to collectively apply for
funding to start up income-generating
activities. Despite the Tanzanian
government’s information campaigns
during the first years of the programme
(2005–2007), in 2008 less than half
of TASAF’s eligible non-beneficiary
households reported to have ever heard
about the programme. Baird, McIntosh
and Őzler (2013) identify households’
direct access to information (e.g.
village meeting attendance rate)
and their immediate access to local
decision makers (e.g. through blood
relationships) as key determinants of
being informed about TASAF.
In a recent working paper, Bet Caeyers
(2014) investigates whether social
interactions with well-informed
neighbours played a role in TASAF
information transfer, and if so, whether
different households benefitted
differently from such interactions. To
do this, Caeyers exploits a unique
household dataset collected by the
World Bank in 2008, capturing
details – including GPS coordinates
and information about knowledge
about TASAF – of the households of
all 19,916 elderly and disabled people
in 100 villages in rural Tanzania.
The results are striking. On average,
each additional informed neighbour
increases the likelihood of a vulnerable
survey respondent knowing about
TASAF by 8 percentage points. Having
two informed neighbours had, on
average, a similar impact on the
probability of being informed as
attending a village meeting (15% and
14%, respectively). Further analysis
explores whether certain groups face
particular difficulties in accessing
information via social networks, and
finds that such groups include femaleheaded households living in a maledominated neighbourhood, less wellconnected households, the uneducated
unemployed and minority religious
groups.
The results are an important reminder
that some isolated households are not
only restricted in their access to direct
sources of information (e.g. village
meetings, radios, television, etc.) but
also lack indirect access to information
through social networks. It may be
desirable to target these demographic
groups separately with complementary
– specifically designed – information
strategies that take these constraints
into account.
Caeyers, B., ‘Peer effects in development
programme awareness of vulnerable
groups in rural Tanzania’ (2014), Centre
for the Studies of African Economies
series, WPS/2014-11
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
04
Article
From the field
How big should the sample be?
Marcos Vera-Hernández
You have your research idea
in mind, and you spot a call
for a proposal that fits nicely
with it. What are the next
steps? Polishing the research
design, assembling the
team … and determining the
sample size needed for the
analysis. This is a tiresome
but important question that
many of us dread.
of the control group. However, at
the very busy moment of writing
the research proposal, we do not
have the time to be digging into
statistics journals, which tend to
use jargon and notation we are not
used to! The final result is that we
stick to the basic calculations.
A new working paper by Brendon
McConnell and Marcos VeraHernández has been written
to make more complex sample
size calculations for randomized
α is the chance of wrongly rejecting H0; β is the chance of wrongly rejecting H1
1-β measures the statistical power of testing hypothesis H1 against H0
We want to have a large enough
sample to be able to detect the
effects of our trial, but not so large
that we will have too little money
left to fund our research time. The
truth is that we know that there
are more complex formulae that
will fine-tune our sample size
calculations. We also know that
in many instances it is possible to
save on costs and to achieve the
same power by increasing the size
5
controlled trials (RCTs) accessible
to busy researchers. Entitled
“Going beyond simple sample size
calculations: a practioner’s guide”,
the paper compiles formulae for
continuous and binary outcomes
with and without covariates, for
both clustered and non-clustered
RCTs. Formulae for unbalanced
designs (where there are different
numbers of treatment and control
observations) are provided, as well
as panel data methods, including
formulae to take advantage of
baseline information when making
sample size calculations. And if
your specific design is not covered,
a useful contribution of the paper
is to show you how to perform
simulations to compute the power
of whatever complex design you
have in mind.
There are also two further
extensions. First, there are
formulae that help you to design
sampling regimes that save costs,
which is particularly useful when
your research budget is also
paying for the interventions.
Second, you are shown how to
adjust the sample size calculations
if you have several outcome
variables and if you are planning
to correct the p-values of your
impact estimates when you
perform the analysis. This is not
us trying to be overly clever or
complex: funders are increasingly
concerned about ensuring that
statistical inference is robust so
that they can be confident about
policy conclusions drawn from
the work they fund. (EDePo
researchers have already been
asked to make such an adjustment
in the sample size calculations.)
Knowing just how big your
sample has to be is therefore an
increasingly important part of
collecting data from the field.
Access ‘Going beyond simple sample size
calculations: a practioner’s guide’, and
associated tools from
www.ifs.org.uk/centres/EDePo/
resources.
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
05
Methodology Matters
Using structural
models to evaluate
development policies
Orazio Attanasio
In recent years, randomised
controlled trials (RCTs) have
received much attention
in applied economics and,
in particular, development
economics. Economists take
the identification of causal
links seriously and worry
about the possibility that
observed differences between
individuals who receive a
programme and individuals
who do not could be driven
by systematic differences
in those individuals rather
than being induced by the
programme itself. The use
of a randomisation device
to assign a programme
eliminates this issue, in
most cases, thus allowing
researchers to make causal
inferences.
However, RCTs do not provide
all the information needed for
policy analysis. The design of
effective policies, the simulation
of counterfactual policies and
the extrapolation of findings to
different contexts require the
identification of the mechanisms
that determine the observed
outcome. This means that
modelling individual, household or
firm behaviour is often essential.
Therefore, even when RCTs
on a specific programme are
available, the use of structural
models of behaviour is a key
part of a more rounded policy
analysis. Indeed, the availability
of RCTs makes the estimation
of more realistic and credible
structural models possible: it
is possible to use the variation
induced by the RCT, which is
exogenous by construction, when
estimating the structural model.
RCTs and structural models are
thus complements rather than
substitutes. At EDePo and the IFS
more generally, we have always
pursued this approach vigorously.
Take as an example the evaluation
of a conditional cash transfer
(CCT) programme, such as the
celebrated PROGRESA in Mexico.
Such programmes change the
relative price of schooling and
working: this price is determined
partly by the wage available to
youths in the local labour market
in the absence of the CCT, whilst
it is determined by the wage
minus the grant, in the presence
of the CCT. One could therefore
think of using variation in youth
wages across different local labour
markets, and relate it to school
attendance in order to infer what
the impact of the introduction of
a CCT may be. Such an approach,
however, assumes two things that
may not hold: that variation in local
youth wages is not correlated with
other determinants of schooling
decisions (or that one can control
for such correlation using observed
variables); and that the effect of
a reduction in wages is exactly
equivalent to a similar increase
in the grant. The availability of
a RCT where a CCT operates in
some randomly chosen labour
markets allows the estimation
of more flexible models that are
more robustly identified. These
models can then also be used to
extrapolate the effects of different
levels and structures of grants, as
in an influential study by EDePo
researchers (Attanasio et al.,
2012).
Another example is our recent
study (Attanasio et al., 2015) of
an early childhood development
programme in Colombia. In
addition to using a clustered RCT to
assess the impact of the policy, we
also estimate a production function
for children’s human capital and
a model of the determinants of
parent’s investments in their
children’s human capital. This
allows us to test whether the
6
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
06
Paper in Focus
.
Taking good care of
their own money
Lucie Gadenne
observed impacts derived from an
increase in the skills of parents,
or from the material and time
investments parents made in their
children. The results suggest that
the programme’s impact can be
fully explained by increases in
parental investments – a valuable
additional insight from the
structural modelling, which can
potentially inform improvements to
the programme’s design.
Attanasio, O., Meghir, C. and Santiago,
A., ‘Education Choices in Mexico: Using
a Structural Model and a Randomized
Experiment to Evaluate PROGRESA’,
Review of Economic Studies (2012) Vol.
79, No. 1, pp. 37-66
Attanasio, O., Cattan, S., Fitzsimons, E.,
Meghir, C., Rubio Codina, M., ‘Estimating
the production function for human
capital: results from a randomized
controlled trial in Colombia’, (2015) IFS
working paper W15/06
7
Given the growing focus on
revenue mobilisation in
the development agenda,
the findings from the
literature concerning the
impact of increases in
government resources on
the quality and quantity
of public services in
developing countries are
somewhat disappointing
and surprising: studies
typically find that
additional resources have
little impact. However,
such studies often consider
variations in non-taxfunded resources, possibly
because variations in tax
revenues that are unrelated
to other determinants of
public spending are hard to
come by.
A new EDePo paper uses
Brazilian data to examine
whether governments spend
resources that come from
taxes that they levy themselves
better than they spend non-tax
resources, such as grants from
higher levels of governments or
from official development aid.
This is done as part of an
evaluation of a programme that
helps Brazilian municipalities
increase their tax revenues by
subsidising investments in local
tax administrations. Making use
of the staggered programme
uptake to identify impacts, the
paper finds that the programme
was successful in raising local tax
revenues: a $1 investment in tax
administration is estimated to
lead to an annual increase in tax
revenues of roughly $1 after five
years.
There is also evidence that
the revenues generated by the
programme were invested, at
least in part, in local educational
infrastructure. In particular,
the increases in tax revenues
generated by the programme
are estimated to increase the
quantity of municipal education
infrastructure by 5% to 6%, and
are associated with a significant
improvement in an index of the
quality of the infrastructure.
In contrast, estimates suggest
that an additional grant from the
federal government – variation
in which comes from rules
determining grant allocations
– does not have an impact on
local education infrastructure.
Why is this? Other papers in the
literature suggest that grants
from central government are
associated with an increase
in local corruption. Further
analysis in our Brazilian study
finds evidence that additional
tax revenues are not lost to
corruption in the same way.
Gadenne, L., ‘Tax me, but spend
wisely? Sources of public finance and
government accountability’ (2015),
Institute for Fiscal Studies, mimeo
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council
EDePo
Evaluating development policy at IFS
July 2015 Issue 3
07
08
February 2015
IFS Working Paper, W15/06
8-9 July 2015
EDePO ESRC-DfID Conference: Improving
productivity in developing countries:
identifying bottlenecks and obstacles to
productive investments and technology
adoption
Recent Publications
Estimating the production function for human capital: results
from a randomized controlled trial in Colombia
Orazio Attanasio, Sarah Cattan, Emla Fitzsimons, Costas Meghir and
Marta Rubia Codina
March 2015
American Economic Journal: Applied Economics, Vol. 7, No. 2, pp
35– 52
Should cash transfers be conditional? Conditionality,
preventative care, and health outcomes
Orazio Attansio, Veruska Oppedisano and Marcos Vera-Hernandez
March 2015
Oxfam Effectiveness Review
Livelihoods in Armenia: Evaluation of new economic
opportunities for small-scale farmers in Tavush and Vayots
Dzor regions
Bet Caeyers and Natalie Naïri Quin
March 2015
Oxfam Effectiveness Review
Women’s Empowerment in Indonesia: Evaluation of Papua
women’s empowerment
Bet Caeyers and Simone Lombardini
March 2015
Oxfam Effectiveness Review
Women’s Empowerment in Rwanda: Evaluation of women’s
economic leadership through horticulture planting material
business
Bet Caeyers and Robert Fuller
June 2015
IFS working Paper, W15/15
Sanitation dynamics: toilet acquisition and its economic and
social implications
Britta Augsburg and Paul Rodriguez-Lesmes
Upcoming Events
9 July 2015
Laura Abramovsky presenting ‘Social
protection, taxation and inequality’ at the
Overseas Development Institute
10 July 2015
Fiscal redistribution in developing
countries, workshop at the Institute for
Fiscal Studies
August 2015
Bansi Malde will be presenting the paper
‘Network structure and risk sharing
in extended family networks’ at the
Econometric Society World Congress (7 21 August) and the EEA Conference (24-27
August)
09
New Projects
Adolescent girl intervention (Safe
Spaces), funded by Children’s Fund
Foundation (CIFF), in collaboration
with Oxford University.
Economic research, analysis and
debate (Jordan) with Adam Smith
International, funded by the Foreign
and Commonwealth Office.
Contact
Safija Jusupovic
Tel: 020 7291 4800
email: edepo@ifs.org.uk
8
Institute for Fiscal Studies
7 Ridgmount Street
London WC1E 7AE
EDePo is hosted at the Institute for Fiscal Studies (IFS) and funded by the Economic and Social Research Council