State of the Field:
The Need to Understand and Incorporate Variation in Impact in Seeking
to Influence Outcomes for Women and Children
Kate Schwartz & Yeshim Iqbal
Outline
Impact Evaluations (scale/cost)
What we Know (and what we don’t)
Women’s Economic Empowerment
Child Development
How to Know More
Overview of Additional Insights
Year 2 Goals
Impact Evaluations
 2,640 (1,721 RCTs) published impact evaluations conducted over the
past 3 decades in 120+ low- and middle-income (LAMI) countries
 863 (648) in Sub-Saharan Africa
 655 (352) in Latin America and the Caribbean
 534 (340) in South Asia & 444 (276) in East Asia and the Pacific
 112 (91) in the Middle East and North Africa
Impact Evaluations
EXPONENTIAL growth in impact evaluations in recent years….
 52% (N=1372) of published impact evaluations (and RCTs, N=895)
conducted in LAMI countries have been conducted since 2010
 Especially true in Middle East and North Africa (64% since 2010)
 410 (271 RCTs) in 2012 alone compared with 45 (28 RCTs) in 2002
Impact Evaluations
EXPONENTIAL growth in impact evaluations in recent years….
 The World Bank averaged 16 per year from 1999-2004 and 62 per
year from 2005-2010.
Impact Evaluations
EXPONENTIAL growth in impact evaluations in recent years….
Impact Evaluations
Cost of the average evaluation is around $500,000 USD, exclusive of all
intervention costs
Yet, this is only 1.4% of the total cost of the evaluated intervention.
(IEG, 2012, Thomas, 2014)
So what do we know from all of these?
“What I want to talk shit on is the paradigm of the Big
Idea—that once we identify the correct one, we can simply
unfurl it on the entire developing world like a picnic
blanket...There are villages where deworming will be the
most meaningful education project possible. There are
others where free textbooks will. In other places, it will be
new school buildings, more teachers, lower fees, better
transport, tutors, uniforms… The point is, we don’t know
what works, where, or why…”
Michael Hobbes, Human Rights Consultant in Berlin
So what do we know from all of these?
 79% of completed World Bank IEs & 35% of completed
IFC IEs have estimated distribution of program impacts
 27% evaluated the specific contribution of specific
elements of treatment
 21% and 46% looked at cost-benefit, rate of return, or
cost effectiveness across programs
 11% and 20% looked at medium/long-term impacts
So what do we know from all of these?
So what do we know from all of these?
So what do we know from all of these?
DE-WORMING INTERVENTION
So what do we know from all of these?
DE-WORMING INTERVENTION
Kenya, RCT, ~30,000 students
(Baird et al., 2014; Miguel & Kremer, 2004)
 Absence rates decline 25%
 Health increases
 Educational attainment and salaries increase (long-term outcomes)
Cited 1,266 times
Featured in/by the Atlantic, Huffington Post, NY
Times, World Economic Forum Newsletter, the
Clinton Global Initiative, and more
So what do we know from all of these?
Kremer founded Deworm the World in 2007
Large scale replication (largely un-evaluated)
YET, Taylor-Robinson et al. (2000, 2009, 2012) find very low to low
evidence that deworming affects weight, height, cognitive tests, or
school attendance in multiple systematic reviews.
 2000: 30 RCTs in 17 countries
 2009: 34 trials (10 new, 6 prior ones excluded due to updated
stricter inclusion criteria)
 2012: 41 trials in 23 countries
So what do we know from all of these?
TEXTBOOKS
So what do we know from all of these?
MICROFINANCE




Hyderabad, India: No effect on women’s economic empowerment
Andhra Pradesh, India: Positive effect
Bangladesh: Positive and negative effects
Qualitative reviews: Overwhelmingly positive
“It remains unclear under what circumstances, and for whom, microfinance has been
and could be of real, rather than imagined, benefit to poor people. Unsurprisingly we
focus our policy recommendations on the need for more and better research… on
whom, where, and when (e.g. under what circumstances), and the mechanisms which
account for these effects…While there is currently enthusiasm for RCTs as the gold
standard for assessing interventions, there are many who doubt the universal
appropriateness of these designs. Indeed there may be something to be said for the idea
that this current enthusiasm is built on similar foundations of sand to those on which
we suggest the microfinance phenomenon has been based.”
How to Know More
For Whom, Under What Conditions, How?
Mixed Methods Approaches
Including Community more in Process
Standardized Measures across Evaluations
Quasi-Experimental Designs
Strategic, Coordinated Impact Evaluation Designs
Translating as opposed to transferring findings
Additional
Insights: 21st
Century
Strategies to
Promote
Economic
Empowerment
and Child
Development
Additional
Systematic review of available data Insights: 21st
sets and available data
Century
YEAR TWO
Strategies to
Promote
Economic
Empowerment
and Child
Development
All of you!
Additional
Insights: 21st
Century
Strategies to
Promote
Economic
Empowerment
and Child
Development
Thank you!
Impact Evaluations
 300 (146 RCTs) related to women’s economic empowerment*




76 (51) in Sub-Saharan Africa
100 (53) in Latin America and the Caribbean
85 (24) in South Asia & 29 (12) in East Asia the Pacific
5 (3) in the Middle East and North Africa
*Tagged sub-group Gender AND at least one of the following sectors: Agricultural Reform, Agricultural Credit, Rural
Livelihoods, Economic Policy, Girls’ Education, Vocational/Technical Education & Trade. Scholarships, Finance,
Conditional Cash Transfers, Community Action Program, Community Driven Development, Social Funds, Private
Sector Development, Social Protection
Impact Evaluations
 794 (146 RCTs) related to child development*




188 (144) in Sub-Saharan Africa
290 (174) in Latin America and the Caribbean
134 (86) in South Asia & 140 (95) in East Asia the Pacific
27 (24) in the Middle East and North Africa
*Tagged as at least one of the following sectors: Educational Inputs; Girls’ Education; Pre-Primary and Primary
Education; Public/Private Sector Education; Secondary Education; Scholarships; Child Nutrition; Conditional Cash
Transfers; Early Child Development Programs