The role of civil society to strengthen
accountability and to build evaluation
systems
The experience of Fundación IDEA
November 2013
Content
•
•
•
About Fundación IDEA
The Indigenous Education Gap Index
PEC FIDE (School based management program): evaluation method and results
•
•
Civil society’s role in M&E
M&E and policy-making
2
Fundación IDEA is a non-partisan think tank that provides rigorous
analysis and sound solutions.






Our vision: informed policy decisions
Rigorous analysis and sound solutions
8 years of experience
Projects in Mexico, Guatemala, Colombia, USA, Italy, Peru
Impartial and non-partisan
Work on a variety of topics: education, economic development…
3
Fundación IDEA works on a variety of topics.





“Evaluación Transversal: políticas y programas para el desarrollo de la juventud”
“Estrategia nacional para la vivienda sustentable”
“¿Qué hacer con la diabetes?”
“¿Quiénes son los emprendedores innovadores mexicanos?”
“Propuesta de programa de inclusión laboral para jóvenes”
Fundación IDEA has developed expertise on education.
 “Estudio sobre calidad docente en zonas rurales e indígenas en México”
 “Índice compuesto de eficacia de los sistemas escolares”
 “Índice de equidad educativa indígena”
 “Evaluación de impacto PEC-FIDE (Fortalecimiento en inversión directa a escuelas)”
4
Fundación IDEA found that indigenous people in Mexico have achieved
66% of the educational achievement compared to non-indigenous.
There is a gap of 34%.
Implications of the study:
 Showing the gap to push
for targeted policy.
6
Impact evaluation of the federal School-Based Management Program
PEC FIDE (PROGRAMA ESCUELAS DE CALIDAD-FORTALECIMIENTO EN INVERSIÓN DIRECTA A
ESCUELAS)






Pilot variation of program
School zone approach
Focus on very low-income neighborhoods
Focus on at risk students
Focus on training (shift from insfrastructure)
More money per school than average PEC
Relevant SBM indicators
Graduation
rate
Learning
outcomes
Dropout
rates
Short-run
Long-run
Fundación IDEA was the external evaluator.
PEC FIDE
 Impartial, rigorous approach, publish results;
 Usefulness of recommendations;
 Implications of evaluation: design programs based on evidence, keep programs that
work, improve programs that do not work.
 Evaluation context:
• 50% of schools were beneficiaries (selection bias).
• Info available: annual school census; ENLACE test; population census.
 Methodology (mixed methods):
• quantitative: diff-in-diff w/ propensity score matching.
• qualitative: interviews, participant observation and document analysis in 10
schools.
8
The evaluation showed positive impact for PEC FIDE (2008-09) in
graduation rates.
Resultados de Skoufias y Shapiro (2006a)
Resultados de Abreu y Santibañez (2010)
Tasa de
aprobación
Tasa de
aprobación
Tasa de
aprobación*
1.6 pp
0.24 pp
0.11 pp
Efectos de un ciclo escolar
Efectos después de tres ciclos es escolares
Efectos de un ciclo escolar
PEC FIDE
Resultados de Murnane, Willett y Cárdenas (2006)
vs
PEC
Other results:
 Approx 20 more points in ENLACE test (reading comprehension); approx 1/3
of standard deviation.
 Approx -4 percentage points of students at insufficient level in ENLACE test.
The evaluation showed impact on reading comprehension at the
elementary school level (ENLACE test).
Avg ENLACE scores
19.5 puntos de
ENLACE/año
PRIM - Tendencia
general
PRIM - PEC-FIDE
12.8 puntos de ENLACE/año
PRIM - PEC-FIDE sin
PEC
Inicio de ciclo
2008-2009
Fin de ciclo
2008-2009
Fundación IDEA presented results to administrators of the program.
 Qualitative analysis showed increased motivation.
 Investment focused on infraestructure, hard shift to training.
 Limited participation: how to promote more and better participation?
 The importance of targeting neighborhoods?
 Importance on school zone program design?
 Targeting students at risk?
Content
•
•
•
About Fundación IDEA
The Indigenous Education Gap Index
PEC FIDE (School based management program): evaluation method and results
•
•
Civil society’s role in M&E
M&E and policy-making
12
Civil society needs to promote an evidence based policy culture.
Public policy based on
evidence
Usefulness of
recommendations
Credibility of
conclusions
Rigorous
analysis
Programs may be implemented because of:
-
Evidence
Experience
Ideology
Politics
Budget
Technical capacity
13
Evaluation should be used as a tool for decision making and efficient
spending. We are experiencing a paradigm shift in M&E.
 Evaluation is a tool for accountability and transparency
- programs use public resources and so society needs to know if objectives are
accomplished and if programs are effective
 Evaluation as a learning process
- to improve design and operation of programs
- to generate knowledge about complex problems and about interventions that
work
- to know if a program works or not
- “bang for the buck”
M&E, if appropriately used, can improve how programs are managed and
the outcomes they can have.
M&E is used for decison
making
The use of M&E by organizations
Good consumer and capacity
to perform M&E activities
Some M&E activities
No M&E
t
15
There are some misconceptions on M&E practice.
Types of evaluation:
 Needs assessment
 Program theory (theory of change)
 Design evaluation
 Implementation or process evaluation
 Results (change)
 Impact (causality)
 Efficiency (CBA, cost effectiveness)
Impact eval
Implementation eval
Sound indicators
Needs assessment
16
Impact evaluation can change policy if implemented based on a good
understanding of M&E practice.
Key lessons for capacity building:
 Valid Counterfactual
- understand when to evaluate impact and when not
- many organizations decide to evaluate before-after or compare beneficiaries
with non-beneficiaries: not always good use of resources
 Theory of Change
- important for understaning what the organization is doing and how they are
measuring it
- sometimes there is no clarity/consensus of the purpose of a program
among stakeholders
Objectives
Inputs
Products
Intermediate
results
Impact
Thank you
18