Annette N. Brown, Deputy Director for AIES
TWO RECENT SYSTEMATIC
REVIEWS FOR DEVELOPMENT
International Initiative for Impact Evaluation
www.3ieimpact.org
Outline
•
•
•
•
•
•
Review questions
Inclusion criteria
Theory of change
Search pipeline
Results
Fun methods finding
International Initiative for Impact Evaluation
www.3ieimpact.org
“Interventions in Developing Nations
for Improving Primary and Secondary
School Enrollment of Children: A
Systematic Review”
By Anthony Petrosino, Claire Morgan, Trevor A.
Fronius, Emily E. Tanner-Smith, Robert F.
Boruch, November 2012
International Initiative for Impact Evaluation
www.3ieimpact.org
Review questions
• What are the effects of interventions
implemented in developing countries on
measures of students’ enrollment,
attendance, graduation, and progression?
• Within those studies that report the effects
of an intervention on [the above
measures], what are the ancillary effects
on learning outcomes?
International Initiative for Impact Evaluation
www.3ieimpact.org
Inclusion criteria
Population/participants: Primary and
secondary school aged children in LMICs
Interventions: Intended to affect one of the
four primary outcomes
Comparison: No specific program comparison
Outcomes: Enrollment, attendance,
graduation, and progression
Studies: RCTs and QEDs
International Initiative for Impact Evaluation
www.3ieimpact.org
Typology of education interventions
Demand
Reducing
costs
Providing
information
CCTs, scholarships
and non-fee subsidies
Supply
Increasing
preparedness
Buildings
Early child
development
Vouchers
Abolishing school fees
and capitation grants
International Initiative for Impact Evaluation
Health/
nutrition
Teachers
Materials
Management
School
feeding
www.3ieimpact.org
International Initiative for Impact Evaluation
www.3ieimpact.org
International Initiative for Impact Evaluation
www.3ieimpact.org
International Initiative for Impact Evaluation
www.3ieimpact.org
International Initiative for Impact Evaluation
www.3ieimpact.org
International Initiative for Impact Evaluation
www.3ieimpact.org
“Farmer field schools for improving
farming practices and farmer
outcomes in low- and middle-income
countries: a systematic review”
By Hugh Waddington, Birte Snilstveit, Jorge
Hombrados, Martina Vojtkova, Daniel Phillips,
and Howard White, December 2012
International Initiative for Impact Evaluation
www.3ieimpact.org
Review questions
• What is the impact of farmer field schools
on their objectives in terms of ‘endpoint’
outcomes such as increased yields, net
revenues and farmer empowerment, and
intermediate outcomes such as capacity
building and adoption of improved
practices?
• Under which circumstances and why: what
are the facilitators and barriers to FFS
effectiveness and sustainability?
International Initiative for Impact Evaluation
www.3ieimpact.org
Inclusion criteria
• Population/participants: Farm households in low and
middle income countries
• Intervention: Programs explicitly referred to as ‘farmer
field school’
• Comparison: No specific program comparison
• Outcomes: effectiveness across the causal chain
– Knowledge → adoption →
– Impact on yields, revenues, environment, health, empowerment
• Studies:
– Effects: experimental, quasi-experimental with controlled
comparison
– Barriers/facilitators: qualitative (Critical Appraisal Skills
Programme checklist 2006)
International Initiative for Impact Evaluation
www.3ieimpact.org
Input 1 Training
of trainers
Input 2 Field
school
T of Change
Capacity
building (FFS
participants)
Capacity
building (FFS
neighbours)
Adoption
(FFS
participants)
Adoption
(FFS
neighbours)
Measured impacts:
International Initiative for
Yield, input-output ratio,
income, empowerment,
environmental
Impact Evaluation
outcomes, health
www.3ieimpact.org
Input 1 Training
of trainers
- Facilitators
adequately trained
- Farmers and
facilitators attend
sufficient meetings
- FFS synchronised
with planting
season
- Curriculum
relevant to
problems facing
farmers
-Farmer attitudes
changed
(convinced
message
appropriate)
- Relative
advantage over old
techniques Initiative
International
Input 2 Field
school
Capacity
building (FFS
participants)
Capacity
building
(neighbours)
Adoption
(FFS
participants)
Adoption
(neighbours)
for
- Field
days/follow-up
- High degree of
social cohesion
- Geographical
proximity to other
farmers
(observation) or
market
(communication)
- New technology
appropriate
- Market access
Measured impacts:
- Favorable prices
Yield, input-output ratio,
- Environmental factors
income, empowerment,
including weather, soil
environmental
fertility
Impact Evaluation
www.3ieimpact.org
outcomes. health
Effectiveness
27,866 titles screened
1453 abstracts
screened
Causal Chain
Analysis
1,112 abstracts
screened
751 excluded
126 no access
369 full text obtained
186 excluded:
128 on relevance
58 on design (no
comparison)
183 Extension impact
papers:
134 FFS
49 non-FFS
134 FFS impact papers
80 individual FFS
studies
International Initiative for Impact Evaluation
312 full text sought
49 no access
257 excluded
Qualitative
Synthesis
25 qualitative
papers
30 IE and sister
papers
20 individual
FFS studies
11 individual
FFS studies
BB+ Synthesis
www.3ieimpact.org
Study
ID
ES (95% CI)
FFS participants
Huan et al., 1999 (Vietnam)
Endalew, 2009 (Ethiopia)
Price et al., 2001 (Philippines)
Rao et al., 2012 (India)
Reddy & Suryamani, 2005 (India)
Kelemework, 2005 (Ethiopia)
Mutandwa & Mpangwa, 2004 (Zimbabwe)
Dinpanah et al., 2010 (Iran)
Khan et al., 2007 (Pakistan)
Bunyatta et al., 2006 (Kenya)
Erbaugh, 2010 (Uganda)
Rebaudo & Dangles, 2011 (Ecuador)
Subtotal (I-squared = 93.3%, p = 0.000)
.
FFS neighbours
Khan et al., 2007 (Pakistan)
Reddy & Suryamani, 2005 (India)
Ricker-Gilbert et al, 2008 (Bangladesh)
Rebaudo & Dangles, 2011 (Ecuador)
Subtotal (I-squared = 0.0%, p = 0.610)
.
NOTE: Weights are from random effects analysis
0.02 (-0.06, 0.10)
0.27 (-0.06, 0.60)
0.42 (-0.17, 1.01)
0.43 (-0.02, 0.87)
0.45 (-0.04, 0.94)
0.54 (-0.22, 1.29)
0.59 (0.25, 0.92)
0.67 (0.41, 0.92)
0.79 (0.29, 1.29)
1.03 (0.65, 1.41)
1.14 (0.93, 1.34)
1.79 (1.17, 2.41)
0.66 (0.33, 1.00)
Positive
impacts on
knowledge
among
participants
-0.13 (-0.68, 0.42)
0.05 (-0.45, 0.56)
0.17 (-0.25, 0.59)
0.38 (-0.15, 0.91)
0.13 (-0.12, 0.37)
-.5
0
.5 1
Favours intervention
International Initiative for Impact Evaluation
3
www.3ieimpact.org
Study
ID
ES (95% CI)
FFS neighbours
Pananurak, 2010 (India)
Khan et al., 2007 (Pakistan)
Feder et al, 2004 (Indonesia)
Labarta, 2005 (Nicaragua)
Pananurak, 2010 (China)
Wu Lifeng, 2010 (China)
Pananurak, 2010 (Pakistan)
Yamazaki & Resosudarmo, 2007 (Indonesia)
Subtotal (I-squared = 49.5%, p = 0.054)
.
FFS participants
Pananurak, 2010 (India)
Van Rijn, 2010 (Peru)
Naik et al., 2008 (India)
Huan et al., 1999 (Vietnam)
Labarta, 2005 (Nicaragua)
Rejesus et al, 2010 (Vietnam)
Feder et al, 2004 (Indonesia)
Wu Lifeng, 2010 (China)
Ali & Sharif, 2011 (Pakistan)
Pananurak, 2010 (China)
Gockowski et al., 2010 (Ghana)
Yang et al., 2005 (China)
Hiller et al., 2009 (Kenya)
Khan et al., 2007 (Pakistan)
Cavatassi et al., 2011 (Ecuador)
Davis et al, 2012 (Tanzania)
Birthal et al., 2000 (India)
Pananurak, 2010 (Pakistan)
Dinpanah et al., 2010 (Iran)
Wandji et al., 2007 (Cameroon)
Mutandwa & Mpangwa, 2004 (Zimbabwe)
Palis, 1998 (Philippines)
Zuger 2004 (Peru)
Carlberg et al., 2012 (Ghana)
Yamazaki & Resosudarmo, 2007 (Indonesia)
Van den Berg et al., 2002 (Sri Lanka)
Davis et al, 2012 (Kenya)
Pande et al., 2009 (Nepal)
Dinpanah et al., 2010 (Iran)
Orozco Cirilo et al., 2008 b) (Mexico)
Todo & Takahashi, 2011 (Ethiopia)
Subtotal (I-squared = 93.0%, p = 0.000)
.
NOTE: Weights are from random effects analysis
0.79 (0.63, 1.00)
0.97 (0.74, 1.26)
0.99 (0.97, 1.01)
1.00 (0.99, 1.01)
1.02 (0.98, 1.07)
1.03 (0.99, 1.08)
1.03 (0.86, 1.25)
1.43 (1.05, 1.96)
1.01 (0.98, 1.03)
0.80 (0.61, 1.05)
0.86 (0.63, 1.18)
0.89 (0.83, 0.96)
0.95 (0.92, 0.98)
0.97 (0.92, 1.02)
0.97 (0.72, 1.31)
0.98 (0.96, 1.01)
1.08 (1.03, 1.14)
1.09 (1.03, 1.15)
1.09 (1.04, 1.14)
1.14 (1.03, 1.25)
1.15 (0.94, 1.41)
1.17 (0.53, 2.56)
1.17 (0.97, 1.42)
1.22 (0.97, 1.53)
1.23 (1.00, 1.51)
1.24 (1.13, 1.36)
1.24 (1.01, 1.54)
1.32 (1.22, 1.42)
1.32 (1.07, 1.63)
1.36 (1.06, 1.73)
1.36 (0.97, 1.92)
1.44 (1.09, 1.92)
1.58 (1.19, 2.10)
1.67 (1.23, 2.26)
1.68 (1.30, 2.18)
1.81 (1.15, 2.84)
2.11 (1.25, 3.56)
2.52 (2.05, 3.11)
2.62 (2.23, 3.08)
2.71 (1.11, 6.60)
1.23 (1.16, 1.32)
.5
1
2
Increased
yields among
FFSbeneficiaries
not
neighbours
3
Favours intervention
International Initiative for Impact Evaluation
www.3ieimpact.org
Reduced environmental risk factors
Study
ID
ES (95% CI)
FFS participants
Pananurak, 2010 (India)
0.52 (0.32, 0.85)
Praneetvatakul & Waibel, 2006 (Thailand)
0.54 (0.39, 0.76)
Pananurak, 2010 (Pakistan)
0.55 (0.41, 0.75)
Cavatassi et al., 2011 (Ecuador)
0.82 (0.55, 1.23)
Subtotal (I-squared = 8.0%, p = 0.353)
0.59 (0.49, 0.71)
.
FFS neighbours
Pananurak, 2010 (India)
0.58 (0.24, 1.41)
Pananurak, 2010 (Pakistan)
0.64 (0.37, 1.10)
Cavatassi et al., 2011 (Ecuador)
0.69 (0.43, 1.11)
Praneetvatakul & Waibel, 2006 (Thailand)
1.04 (0.32, 3.40)
Subtotal (I-squared = 0.0%, p = 0.878)
0.68 (0.49, 0.93)
.
NOTE: Weights are from random effects analysis
.1
.2
.5
1
2
Favours intervention
International Initiative for Impact Evaluation
www.3ieimpact.org
Summary of quantitative findings
• FFS increase knowledge and improve
adoption of the FFS practices
• On average increasing yields and/or
incomes
• Suggestions of farmers feeling
empowered
• Limited, if any, spillovers
• Neighbours do not adopt the practices
consistently
International Initiative for Impact Evaluation
www.3ieimpact.org
Study
ID
ES (95% CI)
High risk of bias
Naik et al., 2008 (India)
Huan et al., 1999 (Vietnam)
Labarta, 2005 (Nicaragua)
Gockowski et al., 2010 (Ghana)
Yang et al., 2005 (China)
Hiller et al., 2009 (Kenya)
Khan et al., 2007 (Pakistan)
Birthal et al., 2000 (India)
Dinpanah et al., 2010 (Iran)
Wandji et al., 2007 (Cameroon)
Mutandwa & Mpangwa, 2004 (Zimbabwe)
Palis, 1998 (Philippines)
Zuger 2004 (Peru)
Carlberg et al., 2012 (Ghana)
Van den Berg et al., 2002 (Sri Lanka)
Pande et al., 2009 (Nepal)
Dinpanah et al., 2010 (Iran)
Orozco Cirilo et al., 2008 b) (Mexico)
Subtotal (I-squared = 95.4%, p = 0.000)
.
Medium risk of bias
Pananurak, 2010 (India)
Van Rijn, 2010 (Peru)
Rejesus et al, 2010 (Vietnam)
Feder et al, 2004 (Indonesia)
Wu Lifeng, 2010 (China)
Ali & Sharif, 2011 (Pakistan)
Pananurak, 2010 (China)
Cavatassi et al., 2011 (Ecuador)
Davis et al, 2012 (Tanzania)
Pananurak, 2010 (Pakistan)
Yamazaki & Resosudarmo, 2007 (Indonesia)
Davis et al, 2012 (Kenya)
Todo & Takahashi, 2011 (Ethiopia)
Subtotal (I-squared = 81.0%, p = 0.000)
.
Overall (I-squared = 93.0%, p = 0.000)
NOTE: Weights are from random effects analysis
0.89 (0.83, 0.96)
0.95 (0.92, 0.98)
0.97 (0.92, 1.02)
1.14 (1.03, 1.25)
1.15 (0.94, 1.41)
1.17 (0.53, 2.56)
1.17 (0.97, 1.42)
1.24 (1.13, 1.36)
1.32 (1.22, 1.42)
1.32 (1.07, 1.63)
1.36 (1.06, 1.73)
1.36 (0.97, 1.92)
1.44 (1.09, 1.92)
1.58 (1.19, 2.10)
1.68 (1.30, 2.18)
2.11 (1.25, 3.56)
2.52 (2.05, 3.11)
2.62 (2.23, 3.08)
1.35 (1.19, 1.52)
0.80 (0.61, 1.05)
0.86 (0.63, 1.18)
0.97 (0.72, 1.31)
0.98 (0.96, 1.01)
1.08 (1.03, 1.14)
1.09 (1.03, 1.15)
1.09 (1.04, 1.14)
1.22 (0.97, 1.53)
1.23 (1.00, 1.51)
1.24 (1.01, 1.54)
1.67 (1.23, 2.26)
1.81 (1.15, 2.84)
2.71 (1.11, 6.60)
1.10 (1.03, 1.17)
Sensitivity
analysis:
Yields by
risk of bias
status
High risk of
bias studies
overestimate
impacts
1.23 (1.16, 1.32)
.5
1
2
3
Favours intervention
International Initiative for Impact Evaluation
www.3ieimpact.org
www.3ieimpact.org
THANK YOU!
International Initiative for Impact Evaluation
www.3ieimpact.org