Fabretto's Monitoring and Evaluation System:
Recommendations and Possibilities for the Future
EXECUTIVE SUMMARY
June 2nd, 2015
Authored by:
Matthew J. Tarditi, M.S.Ed.
University of Pennsylvania, Graduate School of Education
Will Slotznick, Amy Summer,
Amarsingh Gawande, Edgar Agudelo, Conner Evans
Global Impact Collaborative (PennGIC)
Contributing Editor:
Cesar Dubois
Fabretto Children's Foundation
TABLE OF CONTENTS
Introduction
2
1. Overall Framework of M&E Evaluation
3
2. Evaluation Criteria
4
3. Recommendations
5
Appendices
A. PENN Review Plan 2014-2015: Fabretto's M&E System
6
B. Indicator Evaluation Criteria
7
C. List of Recommended Indicators by Strategic Objective
8
D. Detailed Descriptions of Recommended Indicators
12
Indicator 1: Coefficient of Efficiency
Indicator 2: Danielson Framework
Indicator 3: Devereux Student Strengths Assessment (DESSA)
Indicator 4: Education Level of Students vs. Parents
Indicator 5: Frontline SMS Follow-up Survey (FSFS)
Indicator 6: Gender Parity Index
Indicator 7: Progress out of Poverty Index (PPI)
Indicator 8: Tripod Student Surveys
Authors' Biographies
32
References
33
1
INTRODUCTION
Fabretto’s Monitoring and Evaluation (M&E) program attempts to streamline and
advance the organization’s efforts to improve the lives of individuals and communities through
education. To strengthen this crucial component of Fabretto’s work, the University of
Pennsylvania’s Graduate School of Education has partnered with the Penn Society for
International Development’s Global Impact Collaborative to analyze Fabretto's existing M&E
framework and build its relevance and efficacy. Our first step involved building an
understanding of Fabretto’s mission and goals as an organization, and then developing a logical,
collaborative, and multi-step plan to carry out these actions. In the second step of this process we
developed a framework of criteria to critically examine existing indicators. These were assessed
on the degree to which they related to Fabretto’s objectives, fit within the organization’s
resources, and advanced their mission through new partnerships and increased international
relevancy. In all, the team undertook a five-phase process to evaluate Fabretto’s M&E program.
This involved personal research, responding to feedback from Fabretto and Timshel, and a
collective review process as an approach through which we were better informed to recommend
specific indicators and to demonstrate their relation to Fabretto’s objectives and expected
outcomes. We invite Fabretto to identify opportunities for improvement in our work and to
consider modifications to the indicators, processes, and products that we have suggested.
Fabretto’s M&E system is fundamental to their success in what is an increasingly
important goal in the field of international development—access to quality education. "Of about
112 million children born annually in developing countries, 23% or about 26 million will not
complete primary school. Of these 11.6 million will never start school, and getting them to
complete the primary cycle will be costly" (Orazem, Glewwe & Patrinos, 2009, p. 4). While
these statistics reveal the challenging future that NGOs and various development organizations
face, a wide variety of literature supports the conclusion that an investment in education can
bring substantial returns to the individual, community, and nation (Orazem, et al., 2009, p. 13;
UNESCO, 2010; Psacharopoulos & Patrinos, 2002). Achieving “universal primary education” is
even listed as the second United Nations Millennium Development Goal, aiming to ensure that
all boys and girls can complete a full course of primary schooling by 2015. Understanding that
the completion of schooling alone does not guarantee "quality" education, it is vital that Fabretto
offers programming and utilizes measures that strengthen education and monitor and evaluate
progress and results. Multiple organizations and studies support the importance of nutrition in
both combating poverty and improving education (i.e. school enrollment and student
performance), and therefore Fabretto emphasizes school nutrition programs in order to
supplement their education interventions.
As a result of Fabretto’s long-term involvement in Nicaraguan communities and
established status within the country, its impact and monitoring and evaluation system can
provide important insight for the field of international development. As Fabretto continues to
grow and participate in the global movement to tackle poverty through education, health, and
nutrition initiatives, it is crucial that it remains an engaged participant in relevant international
discourse. While the team has developed a replicable model of evaluation for instruments and
indicators, this ongoing involvement in international dialogue will facilitate perpetual monitoring
2
and assessment of the M&E system and allow for informed adjustments and updates that will
ensure its continued functionality and resonance across Fabretto's communities.
OVERALL FRAMEWORK OF M&E EVALUATION
The framework for the evaluation of Fabretto's Monitoring and Evaluation System is
comprised of six phases: (1) Phase One: Preliminary Research and Development of Evaluation
Plan; (2) Phase Two: Develop Evaluation Criteria and Conduct Initial Evaluation of Potential
Indicators; (3) Phase Three: Evaluation of Existing Indicators and Continued Evaluation of
Potential Indicators; (4) Phase Four: Evaluation of Processes and Organizational Structure of
M&E System; (5) Phase Five: Final Analysis and Recommendations; and (6) Phase Six: In-field
Collaborative Review and Operationalization of M&E Recommendations with Fabretto staff.
The specific activities and timeline for these six stages of the evaluation are outlined in depth in
the Penn Review of Fabretto's M&E System chart (Appendix A).
In Phase One we engaged with Fabretto staff and foundation documents to better
understand both the mission and objectives of Fabretto as well as staff perspectives and
understandings of the current M&E system and visions for the future. Additionally, the
exploratory and preparatory phase sought to critically understand the international landscape of
Monitoring and Evaluation, including the prevailing indicators and instruments used by
foundations, non-governmental organizations, and institutions across the globe. It culminated
with the creation of an overarching structure and logic to our evaluation. Next, we entered into
Phase Two, during which we developed an Indicator Evaluation Criteria (discussed in the
following section) to facilitate a systematic, rigorous examination of potential and existing
indicators. The completion of the first report on potential indicators for Fabretto entitled
Preliminary Review of Potential Indicators: An Evaluation, Rating and Analysis was followed by
extensive review and critical feedback from Fabretto's directors and staff, a collaborative
approach that enabled us to refine the evaluation criteria and to further incorporate priorities and
perspectives from staff into our ongoing work. This signaled the start of the subsequent phase.
Phase Three included a thorough review and refinement of the evaluation criteria
followed by the identification of additional potential indicators (based on Fabretto feedback and
an increased emphasis on alignment with the mission and objectives of Fabretto). To enable a
holistic approach, we evaluated these additional indicators along with Fabretto's existing
indicators and began to map the indicators onto the Strategic Plan 2015-2017 created by
Timshel. In tandem to Phase Three, we examined the processes and organizational structure of
Fabretto's M&E (Phase Four), including the mapping of two representative indicators (EGRA
and MINED Statistics) in order to understand the steps, processes, involved parties, timelines
and overall life of data within the system.
Lastly, in Phase Five, we assembled the different pieces of the puzzle from the previous
phases and conduct concentrated analysis across our work in order to produce a concise,
comprehensive report on Fabretto's M&E system that focuses on recommendations and
opportunities for improvement. The Executive Summary represents a condensed version of the
final report, with an emphasis on recommended indicators and metrics. In the final report, we
will include the procedural and organizational elements of the overall M&E System.
3
In the following section, we discuss the specific evaluation criteria we employed to
examine potential and existing indicators for Fabretto's M&E system.
EVALUATION CRITERIA
Central to the evaluation of Fabretto's M&E System was the creation of a set of criteria to
systematize and clarify the examination of indicators. Informed by application and feedback
from Fabretto staff, the criteria, including weights and scoring, have been iterated over time and
have evolved into their current state. Four major developments have occurred since the original
criteria were established. First, the set of criteria itself has changed. Second, the weights
assigned to each criterion have been adjusted. Weighting represents the numerical rankings of
each individual criterion based on a review of literature and critical conversations with Fabretto
staff. Third, the scoring for each criterion has been reduced from a 0-10 scale to 0-4. And lastly,
detailed descriptions for each criterion and score have been added. The results of these changes
are reflected in the overall evaluation criteria (Appendix B).
All indicators, potential and existing, were evaluated using our criteria in order to enable
comparability and to strengthen a more objective review and analysis. Furthermore, the criteria
are a useful tool for Fabretto as the organization continues to periodically and systematically
(re)consider metrics moving forward. Based on a wide spectrum of variables and issues related
to synergy with Fabretto, implementation, resources, and audience, we created the following six
criteria: (1) Alignment with Fabretto's Mission and Objectives; (2) Financial Sustainability; (3)
Partnership Opportunities; (4) National and International Relevance; (5) Necessary Resources
to Implement Indicator; and (6) Audience.
The complete breakdown of criteria is included at the conclusion of this section;
however, we will highlight two examples to demonstrate the reasoning behind our decisions.
The most heavily weighted criterion, Alignment with Fabretto's Mission and Objectives,
measures to what extent the metric aligns with Fabretto's Strategic Plan (expected results,
objectives and targets), supports the mission and vision of Fabretto, and is consistent with
Fabretto's values. The most important determinant of an indicator's inclusion in Fabretto's
Monitoring and Evaluation system is whether or not the metric directly aligns with the goals and
objectives of the organization. The second most weighted criterion, Financial Sustainability,
refers to the direct and associated costs of an indicator and how those may be addressed through
existing or potential financial partners. If Fabretto is unable to cover the expenses to utilize an
indicator, regardless of its alignment to goals and objectives, it is significantly less feasible to
include the metric into the M&E system. In sum, we developed the evaluation criteria as a
systematic approach and a set of prioritized characteristics to provide Fabretto with a useful,
practical tool in the face of a rapidly changing world of international M&E.
The following section highlights our recommendations for indicators to be included in
Fabretto's Monitoring and Evaluation System.
4
RECOMMENDATIONS
In this section we present a final selection of 41 indicators to support Fabretto’s system of
M&E. It includes sets of existing, adjusted, and new indicators, differentiated by color in the
charts below. These recommended indicators are the final product of a multi-step investigative
and evaluative process. Each stage in this process had been informed by feedback from our midyear Pilot Study. In line with the interests of Fabretto staff, particular attention has been given to
finding and reviewing indicators for students, teachers, and graduates.
Stage One of the evaluation involved individual research to identify a new set of potential
indicators. Following this investigation (Stage Two), our team compiled a draft list of indicators
through a collective review process. In Stage Three, team members applied the evaluation
criteria to better understand the strength and relevance of each proposed indicator. Aided by the
Timshel Strategic Plan, we considered the alignment of indicators to Fabretto’s mission,
outcomes and objectives, and the feasibility of implementation. In Stage Four, the team engaged
in a second round of collective review followed by an individual voting process (Stage Five) to
narrow our draft list of indicators. The team held a third round of collective review (Stage Six) in
which we shared votes and decided upon a final list of relevant indicators.
Throughout this process, team members engaged with the providers and creators of select
instruments and indicators (e.g. Tripod, DESSA, and Danielson). This engagement offered our
team concrete ideas on the feasibility of new indicators, both in terms of implementation and
analysis. Further, this outreach allowed us to explore potential partnerships between certain
providers and Fabretto.
In the appendices you will find the Final List of Indicators (Appendix C), which includes
how each indicator corresponds to the outcomes and objectives from Timshel’s strategic plan,
and details on the New Indicators (Appendix D) our team recommends.
5
Appendix A
PENN Review Plan 2014-2015: Fabretto's M&E System
Responsible
Deadline
Penn-Fabretto
Penn-Fabretto
Penn-Fabretto
Penn-Fabretto
10/13/14
10/13/14
10/14/14
10/15/14
Phase One: Preliminary Research and Development of Evaluation Plan.
1.1. Preliminary Research of M&E System (via meetings, conversations and document review).
1.2. Create Evaluation Plan to Review Fabretto's M&E System.
1.3. Review Evaluation Plan with Fabretto Staff for Feedback.
1.4. Adjust Plan based on Fabretto feedback.
Phase Two: Develop Evaluation Criteria and Conduct Initial Evaluation of Potential Indicators.
2.1. Create Indicator Evaluation Criteria.
2.2. Identify Potential Indicators to be reviewed.
2.3. Conduct Evaluation of Potential Indicators using Evaluation Criteria.
2.4. Produce Report on Potential Indicators for Fabretto.
2.5. Share Report with Fabretto Staff and Receive Feedback.
2.6. Review Feedback and Adjust Overall Plan for Review of Fabretto M&E System.
Penn-Fabretto
Penn-Fabretto
Penn
Penn
Penn-Fabretto
Penn-Fabretto
11/1/14
11/1/14
11/15/14
11/24/14
1/10/15
1/13/15
Phase Three: Evaluation of Existing Indicators and Continued Evaluation of Potential Indicators.
3.1. Adjust Indicator Evaluation Criteria.
Penn-Fabretto
1/13/15
3.2. Review and Revise Evaluation Criteria Descriptions.
3.3. Identify Additional Potential Indicators based on Fabretto feedback.
Penn-Fabretto
Penn-Fabretto
1/13/15
2/19/15
3.4. Evaluate Fabretto's Existing Indicators.
Penn-Fabretto
3/1/15
Penn
Penn
3/18/15
4/10/15
4.1. Map the life of representative selections of data (EGRA & MINED data): Preparation, data
collection and management, analysis, dissemination and decision-making.
Fabretto-Penn
4/1/15
4.2. Revise Map of Data with Fabretto M&E team and Finalize Representative Sample.
4.3. Analyze Current Organizational Structure of M&E System.
4.4. Finish Review of Existing Fabretto Documents on Processes and Organizational Structure.
Penn
Penn
Penn-Fabretto
4/21/15
4/22/15
5/22/15
5.1. Create Structure and Outline for Final Report.
Penn-Fabretto
3/18/15
5.2. Share Proposed Structure and Outline with Fabretto for feedback.
Penn-Fabretto
3/18/15
5.3. Cross-Reference Indicators-Instruments with Timshel's Strategic Plan Outcomes-Objectives.
5.4. Write Executive Summary for Board of Directors Meeting (May 7th).
Penn-Fabretto
Penn
3/18/15
4/29/15
5.5. Review Strategic Plan Matrix (Outcomes, Objectives, Indicators) with Timshel.
Penn-Fabretto
5/1/15
5.6. Present Executive Summary to Board of Directors to elicit feedback.
5.7. Final Draft of Report with Analysis and Recommendations for Fabretto's M&E.
Penn
5/7/15
6/1/15
3.5. Evaluate Additional Potential Indicators.
3.6. Determine Recommended Indicators for Future of Fabretto M&E.
Phase Four: Evaluation of Procedural and Organizational Structures of M&E System.
Phase Five: Cumulative Analysis and Draft of Final Recommendations Report.
Penn
Phase Six: Collaborative Review and Operationalization of M&E Recommendations with Fabretto staff.
6.1. Review Final Report with Fabretto Staff (i.e., Programs, M&E, Directors, Centers).
6.2. Select Indicators & Instruments for Pilot (work alongside Program Directors & M&E Staff).
Penn
Penn-Fabretto
6/12/15
6/19/15
6.3. Critically Consider Procedural and Organizational Recommendations for M&E.
Penn-Fabretto
6/26/15
6.4. Design Indicator/Instrument Pilot Plan (2015).
Penn-Fabretto
7/3/15
6.5. Devise Structure, Content and Overall Plan for M&E Workshops.
Penn-Fabretto
7/24/15
6.6. Determine Penn Involvement with Fabretto M&E Post-August 2015
6.7. Draft Action Plan for M&E Post-August 2015.
Penn-Fabretto
Penn-Fabretto
8/1/15
8/15/15
6
Appendix B
Indicator Evaluation Criteria
Indicator Evaluation Criteria (Weight and Score Breakdown)
Criteria
Alignment with
Fabretto
Weight
10
The indicator helps to
measure desired outcomes
and impacts that align with
the Strategic Plan
(expected results,
objectives and targets) and
support the mission and
Criteria Description
vision of Fabretto. The
indicator is consistent with
Fabretto's values.
Financial
Sustainability
9
Sustainbility arises from
manageable costs and
financial partners to assist
in covering these costs.
Full implementation of an
indicator (from preparation
and data collection to
analysis and decisionmaking) requires financial
resources that can be
provided internally and/or
externally (i.e., funders,
donors, partners and
Fabretto).
Partnership
Opportunities
National and
International
Relevance
8
7
Necessary
Resources to
Implement
Indicator
6
Using the indicator could
The position and
The amount and type of
provide increased or new relationship to national and resources (funds, personnel
relationships with various international discussions
hiring, duration of the
NGOs, donor agencies,
with respect to social,
project, needed expertise
individuals and/or public
educational and health
knowledge and skills)
and private institutions.
programs. The obtained required for the collection,
results of Fabretto's
processing, analysis and
intervention are
publication of data for
comparable with statistics project implementation,
from national and
reporting and decisioninternational agencies.
making.
Audience
5
The people concerned with
and interested in the
information provided by
the indicator (e.g.
beneficiaries, donors,
communities, Mined and
government, Fabretto staff
and Board of Directors,
individuals and
organizations).
Score
0
1
2
There is no funding
available to implement the There are no apparent
indicator AND/OR the
opportunities to partner
indicator is prohibitively with other agencies or
expensive to justify
organizations.
application.
There are limited economic
resources AND/OR
The indicator aligns with at
temporary funding to
Affords little potential for
least one objective of
enable the full
future partnerships.
Fabretto's strategic plan.
implementation of this
relatively costly indicator.
Not applicable because
there is only funding for
The indicator aligns with the start of the
Provides at least one
one or more of the
implementation of this
opportunity to partner with
objectives, goals and
indicator but not the end
an organization, agency,
values of Fabretto.
AND/OR the indicator is institution, NGO or donor.
moderately costly to
implement.
The values and objectives
of Fabretto do not align
with the data provided by
this indicator.
3
4
There are short-term and
long-term funding
The indicator aligns with available to fully
Fabretto's mission, vision, implement the indicator
objectives and values.
AND/OR the indicator
costs little to nothing to
implement.
The resources necessary to
implement the indicator are
There is no audience for
nearly impossible for
this indicator.
Fabretto to develop, access
or deploy.
The indicator has little to
no relevance to prominent
discussions and trends
(nationally or
internationally).
The resources and
Few people are interested
expertise necessary to
in the information provided
implement far exceed
by the indicator
Fabretto's current capacity.
There is moderate
relevance to national OR
international metrics, but
not both.
Fabretto is able to partially
Some people are interested
meet the resources required
in the information provided
to fully implement the
by the indicator
indicator.
The indicator provides
relevant metrics to national
and international
organizations, institutions
and agencies focused on
social, educational and
health programs.
The indicators affords
Provides great and real
highly relevant metrics on
opportunities for
the national and
partnerships with different international scale. Data
organizations, private
provided is popularly
institutions, the
measured by respected
government, etc.
institutions, organizations
and agencies.
The indicator is relatively
inexpensive to implement Affords potential for
AND/OR funding is
partnerships with several
available to cover the
organizations, institutions
majority implementation and/or individuals.
costs.
It aligns with Fabretto's
mission and most of its
goals and objectives.
The indicator has no
relevance to the national
and international context
The indicator requires
minimal resources
Many people are interested
AND/OR Fabretto can
in the information
meet the majority of the
provided by the indicator.
necessary resources to fully
implement.
Most (almost all) people
are interested in the
The necessary resources to
information provided by
fully implement the
the indicator. Also, the
indicator can be easily met
indicator provides relevant
by Fabretto.
and/or useful information
for multiple users.
7
Appendix C
Final List of Indicators
NEW INDICATOR
Indicator
ADJUSTED INDICATOR
Instrument
EXISTING INDICATOR
Objective
Poll graduating
seniors (traditional &
SAT) on parents'
education and track
SAT graduates yearly.
1
Education of Graduating Students vs
Parents
FSFS
1A
2
Gender Parity Index
MINED Statistics
1A
3
Number of students matriculated in
preschool education
MINED Statistics
1A
3E
4
Number of students matriculated in
primary education
MINED Statistics
1A
3E
MINED Statistics
1A
3E
MINED Statistics
1A
3E
MINED Statistics
1A
3E
MINED Statistics
1A
3E
PPI Survey
1A
1D
6
Training session
attendance lists
1D
3D
6D
List of parents
who attend
school/center/cha
pel events
1D
3D
4C
5
6
7
8
9
10
11
Percentage of attendance in preschool
students
Percentage of promotion of primary
students (aka Promotion Efficiency)
Percentage of retention for preschool
students
Percentage of retention for primary
students
Progress out of Poverty Index (PPI)
Number of mothers and fathers who
have participated in 75% of the
training, sensitization, and promotion
workshops during the year
Number of parents who collaborate in
school and/or program activities
(primary & SAT): Examples are
gardens, road repair, use of land, food
preparation.
Notes/Additions
1D
6B
6C
6D
To add: Case study of
involved parents
8
Indicator
12
Post-Graduate Employment and
Education Status
13
Teacher Quality
14
15
16
17
18
19
DESSA
Danielson Framework (Planning and
Preparation; Classroom Environment;
Instruction; Professional
Responsibility)
Number of preschool teachers who
have received pedagogical assistance
visits (quarterly)
Number of instructors and promoters
trained (attended 75% of trainings)
Number of parents trained in Early
Stimulation (attended 75% of sessions).
Number of correct responses (EGMA)
Instrument
FSFS - Frontline
SMS Surveys
(Graduation
tracking)
TRIPOD
Student-sourced
teacher
evaluations
DESSA Survey
Objectives
1D
5A
5B
5C
2A
3A
3C
4B
3A
Accompaniment
Visit Form
3A
Training session
attendance lists
Training session
attendance lists
EGMA Exam &
Protocol
4D
3D
3D
3
20
Number of girls and boys benefited by
the Early Stimulation program
List of boys and
girls, including
names, ages, and
community
3B
21
Percentage of preschool teachers
trained
Training session
attendance lists
3E
22
Percentage of students that read with
fluidity and comprehension (EGRA)
EGRA Exam &
Protocol
3
23
Case study of projects and
entrepreneurial initiatives started by
students
Case study
Guide*
3B
5D
Pilot in select schools
with Internet.
*If online/internet
2C* 3A
Danielson
Form/Guide
Notes/Additions
4A
3C
# of boys and girls
matriculated in the
Early Stimulation
program; % of student
retention in Early
Stimulation program
Number of preschool
teachers trained
(attended 75% of
trainings)
3E
5B
5C
5D
Number of students
involved in projects.
*To be created with
Fabretto staff
9
Indicator
Instrument
AFPF Statistics
& Initial
Matriculation
AFPF Statistics
& Youth Lists
24
Number of students who complete
Educational Enrichment Program
25
Number of students who received
scholarships.
26
Percentage of student promotion in VT
(Vocational Training) program
AFPF Statistics
Percentage of student retention in VT
courses according to gender, age and
region
Total dollar amount of scholarships
awarded to students
AFPF Statistics,
Initial
Matriculation
Scholarship
Information
29
Percentage of SAT staff trained (attend
75% of training sessions)
Training session
attendance lists
30
Number of on-site pedagogical
advisory visits to SAT tutors
31
Number of pedagogically equipped
classrooms (in preschool, primary,
secondary and SAT)
32
Number of students matriculated in
SAT program
33
Percentage of retention for SAT
students
27
28
Visit Form for
Technical
Advisory
Inventory of
Central Storage,
Inventory of
Delivery from
AFPF Centers.
AFPF Statistics
& Initial
Matriculation
AFPF Statistics,
Final
Matriculation
Objectives
3
3B 3D
Notes/Additions
To add: More detailed
division of activities
4A
5A
5C
5D
5A
5C
5D
# and % of student
attendance in FV
program
# and % of student
attendance in FV
program
4A
# of SAT staff trained
(attend 75% of
training sessions)
4B
4D
4D
5B
To add: Definition of
pedagogically
equipped classroom
4E
4E
Grade, gender, and
region
# of students who
pass the Public
University Admission
Exams in Spanish
and/or Math
34
Percentage of passing students (Public
University Admission Exam) in
Spanish and/or Math
Public University
Admission Exam
5A
35
Number of daily school lunches
supplied
Fabretto Data
Base
6A
36
Percentage of boys and girls from age
1-5 years with normal height for their
age
Fabretto Data
Base
6A
# of boys and girls
age 1-5 with a normal
height for their age
10
Indicator
Instrument
37
Percentage of boys and girls from age
1-5 years with normal weight for their
age
Fabretto Data
Base
6A
38
Percentage of schools that have
adequate waste disposal systems
Observations by
AFPF technicians
6B
39
Number of home visits to families with
children showing risk factors in biopsycho-social development
Training session
attendance lists
6D
6E
40
Number of households (mother, father,
uncle, guardian) who attend four talks
on early stimulation
Training session
attendance lists
6D
6E
41
Percentage of teachers and directors
trained in topics of health and nutrition
Training session
attendance lists
Objectives
6D
Notes/Additions
Number of boys and
girls from age 1-5
years with normal
weight for their age
Number of teachers
and directors trained
in topics of health and
nutrition (attend 75%
of sessions)
11
Appendix D
Detailed Descriptions of Recommended Indicators1
Indicator 1: Coefficient of Efficiency
Definition: How many students should ideally graduate to the next level of school compared to
how many students actually graduate (UNESCO Institute for Statistics, 2009).
Purpose: Shows the efficiency of the education system, showing the quantity of dropouts and
students who have to repeat a year in school.
Calculation method: Divide the actual number of graduates by the ideal number of graduates, and
then multiply by 100.
Distilled Formula:
Coefficient = Actual Amount of Graduates
Ideal Number of Graduates
X 100
Process (incl. data required/data sources): Need the number of graduates with and without
repetition. Also need the amount of dropouts. Data will come from school census and registry.
Interpretations: A high percentage indicates a high level of efficiency in students graduating
without needing to repeat a year in school. Lower percentages indicate the amount of students
dropping out and repeating years in school.
Limitations: This indicator only shows the efficiency in graduating students, but does not show the
exact level of achievement of the students.
Justification: This indicator not only shows the retention rate of students, but also shows how
many students are repeating years in school. It is promoted by UNESCO’s Institute for Statistics.
------------------------------------------------------------------------------------------------------------------------------------------
Indicator 2: Danielson Teaching Framework2
Definition: A framework for teaching that identifies the aspects of a teacher's responsibilities that
promote improved student learning (supported by empirical studies). Organizes teaching among
various areas of competence in which teachers need to develop expertise. There are twenty-two
components clustered into four domains (listed below).
Purpose: The framework provides a universal/shared vocabulary as a means to discuss quality and
performance. It also offers a clear pathway and set of objectives within four teaching-related
1
More information and content are available upon request.
Following a conversation about Danielson, core materials of the framework were shared with Danea Mairena for
her review and feedback.
2
12
domains. Lastly, having four clear domains serves to focus the work of professional development and
accompaniment activities.
According to the Danielson Group website (http://danielsongroup.org/framework/), "the Framework for
Teaching is a research-based set of components of instruction, aligned to the INTASC standards, and
grounded in a constructivist view of learning and teaching." The InTASC provides a set of standards that
"outline the common principles and foundations of teaching practice that cut across all subject areas
and grade levels and that are necessary to improve student achievement" (CCSSO, 2011)
The Framework may be used for many purposes, but its full value is realized as the foundation for
professional conversations among practitioners as they seek to enhance their skill in the complex task of
teaching. The Framework may be used as the foundation of a school or district’s mentoring, coaching,
professional development, and teacher evaluation processes, thus linking all those activities together
and helping teachers become more thoughtful practitioners. Developing a common understanding is
critical to accuracy, teaching advancement, and the Framework’s impact on students’ core learning.
Primary Questions to Consider for Next Steps
1. Which domains and components of the Danielson Framework are most relevant to Fabretto's
primary school and secondary programming and the foundation's expected outcomes and
objectives?
2. Will the Danielson Framework be utilized for supporting teacher development, evaluating
teacher performance or a combination of the two approaches?
3. What school or center will be the pilot group to test the viability and impact of utilizing the
Danielson framework?
4. Does Fabretto want to formally partner with Danielson Group or simply utilize the framework?
If a formal partnership is the path, to what extent does Fabretto want to collaborate?
a. Need to schedule an informational discussion with Danielson Framework to further
discuss potential options and possible steps moving foward.
5. To what extent is Fabretto willing and able to cover the potential costs related to the Danielson
Framework?
a. (1) Translation of materials; (2) Travel, accommodations and per diem for Danielson
experts to facilitate initial trainings; and (3) Additional training workshops for Fabretto
staff and local stakeholders among others.
Issues related to Preparation
1. Coordinate with the Danielson Group to determine next steps for potential collaboration with
Fabretto
2. Define terms and specifics of communication and collaboration (informal vs. formal)
3. Identify potential costs: translation, Danielson-lead workshops (i.e., travel, accommodation,
logistics)
4. Arrange workshops and facilitation sessions about the overall framework and its approach to
evaluating and developing teachers
5. Translate Framework's central materials
6. Select specific domains and components to utilize for evaluative and/or development purposes
13
7. Determine the purpose of the Framework - Which domains does Fabretto use for teacher
evaluation and which do they use for teacher professional development?
8. Organize workshops on the Frameworks' domains and components
Four Domains of Teacher Evaluation
Four Levels of Performance: (1) Unsatisfactory, (2) Basic, (3) Proficient, (4) Distinguished
Critical Attributes of Proficient and Distinguished Levels and Possible Examples
 Domain 1: Planning and Preparation
 Domain 2: The Classroom Environment
 Domain 3: Instruction
 Domain 4: Professional Responsibility
Domain 1: Planning and Preparation
 Effective teachers plan and prepare for lessons using their extensive knowledge of the content
area, the relationships among different strands within the content and between the subject and
other disciplines, and their students’ prior understanding of the subject. Instructional outcomes
are clear, represent important learning in the subject, and are aligned to the curriculum. The
instructional design includes learning activities that are well sequenced and require all students
to think, problem solve, inquire, and defend conjectures and opinions. Effective teachers design
formative assessments to monitor learning, and they provide the information needed to
differentiate instruction. Measures of student learning align with the curriculum, enabling
students to demonstrate their understanding in more than one way.
Domain 2: Classroom Environment
 Effective teachers organize their classrooms so that all students can learn. They maximize
instructional time and foster respectful interactions with and among students, ensuring that
students find the classroom a safe place to take intellectual risks. Students themselves make a
substantive contribution to the effective functioning of the class by assisting with classroom
procedures, ensuring effective use of physical space, and supporting the learning of classmates.
Students and teachers work in ways that demonstrate their belief that hard work will result in
higher levels of learning. Student behavior is consistently appropriate, and the teacher’s
handling of infractions is subtle, preventive, and respectful of students’ dignity.
Domain 3: Instruction
 In the classrooms of accomplished teachers, all students are highly engaged in learning. They
make significant contributions to the success of the class through participation in high-level
discussions and active involvement in their learning and the learning of others. Teacher
explanations are clear and invite student intellectual engagement. The teacher’s feedback is
specific to learning goals and rubrics and offers concrete suggestions for improvement. As a
result, students understand their progress in learning the content and can explain the learning
goals and what they need to do in order to improve. Effective teachers recognize their
responsibility for student learning and make adjustments, as needed, to ensure student success.
Domain 4: Professional Responsibilities
 Accomplished teachers have high ethical standards and a deep sense of professionalism,
14
focused on improving their own teaching and supporting the ongoing learning of colleagues.
Their record-keeping systems are efficient and effective, and they communicate with families
clearly, frequently, and with cultural sensitivity. Accomplished teachers assume leadership roles
in both school and LEA projects, and they engage in a wide range of professional development
activities to strengthen their practice. Reflection on their own teaching results in ideas for
improvement that are shared across professional learning communities and contribute to
improving the practice of all.
Calculation method: For each domain there are specific components comprised of two to five elements
to describe the specific characteristics of the component. Using the teaching performance rubric (i.e.,
unsatisfactory, basic, proficient, distinguished), teachers and evaluators can assess each component and
provide a roadmap for improvement in teaching. The Danielson Framework is a supportive, interactive
tool that is best used in a collaborative, conversational manner between teachers and evaluators.
Formula: Although the tool is not best used to provide an overall score for reporting purposes, one
could calculate the numerical score for each element within each component in all four domains to
reach an overall total. For example, the maximum score for the 22 components would be 88 (22
components multiplied by 4 for Distinguished).
Process (incl. data required/data sources): The first step is to present and discuss the Danielson
framework to teachers in order to clarify the goals, domains, components and specific elements of the
teaching profession. Once the initial workshop has been delivered, teachers will self-assess their status
based on the domains and evaluation rubric. Concomitantly, evaluators can assess teachers using the
same framework and then discuss the results in order to plan strategies for enhancement and/or
improvement.
Interpretations: Standardized teaching indicators and assessment have widespread critics and
detractors. They lack a nuanced, flexible framework that can truly account for local, individualized
characteristics that may not fit smoothly into the predetermined, universal rubric.
Limitations: Great emphasis, power and interpretative license are given to the observer. He or she must
be an expert observer and pedagogue in order to offer critical feedback and constructive suggestions to
the teachers being observed. Additionally, observations have long been criticized for their limited view
of the classroom, teacher and student, based on the minimal time in the class, thus providing an
extremely incomplete account of the teacher's performance. Besides the subjectivity of and extreme
reliance on the observer, the traditional, observational approach lacks a complete, nuanced picture of
the teacher's performance due to the narrow window into the classroom dynamic and pedagogical
practices of teachers.
Justification: Although constrained by the expertise of the observer and the focus on a universal
framework of assessment, the Danielson framework can clarify the vocabulary around effective teaching
and provide scaffolding for teachers and schools in the identification, enhancement and development of
propitious practices and performance for the improvement of student learning.
Example from NYC Department of Education's Adaptation of the Danielson Framework
15
Adopted the framework for evaluation and development approaches (NYC Department of Education,
2014).
1. Rubric components: School leaders will rate teachers for evaluative purposes on eight components
of the Framework:
 1a Demonstrating Knowledge of Content and Pedagogy
 1e Designing Coherent Instruction  2a Creating an Environment of Respect and Rapport  2d Managing Student Behavior  3b Using Questioning and Discussion Techniques  3c Engaging Students in Learning  3d Using Assessment in Instruction  4e Growing and Developing Professionally *The entire Danielson Framework for Teaching (2013 Edition) will be used for formative purposes. 2. Forms and evidence: A new evaluator form will be created and will apply to all observation options.
The new form will require evaluators to provide lesson-specific evidence gathered during an observation
for all components rated, and it will enable supervisors to include Domain 1 or 4 evidence observed
within 15 school days prior to the classroom observation as part of an assessment of a teacher’s
preparation and professionalism.
3. Timely feedback: To help ensure that teachers are provided with timely feedback on their practice,
evaluators will be required to provide lesson-specific feedback to teachers (verbally or in writing) within
15 school days of an observation. Evaluator forms must be completed within 45 school days of an
observation; in order to ensure that teachers get constructive and timely feedback, evaluators can only
conduct one additional evaluative observation between the time of an observation and the completion
of the form for that observation. ------------------------------------------------------------------------------------------------------------------------------Indicator 3: Devereux Student Strengths Assessment (DESSA)
Definition: The Devereux Student Strengths Assessment is a method of assessing social-emotional
competencies of students in kindergarten through the 8th grade. The assessments can be completed by
teachers/school administrators or parents/guardians. It focuses on the students’ awareness of their
emotions and how they manage them, depending on the context. The DESSA tool can also provide
classroom profiles that may help teachers manage and improve social-emotional intelligence. Finally, it
has been used in the past at many different levels, including individual, classroom, school and district,
and they have also been used abroad by NGOs. Measured elements include self-awareness, socialawareness, self-management, goal-directed behavior, relationship skills, personal responsibility,
decision making, and optimistic thinking. According to the report, the profiles are also very easy to read
and understand.
16
Purpose: Analyze whether or not Fabretto programs are helping students to develop the socialemotional skills that are highly valued by Nicaraguan employers.
Calculation method: DESSA comes with a manual that describes the scoring process, and also provides
instructions on how to interpret and understand results.
Formula: Unknown. Will be provided in the DESSA handbook.
Process (incl. data required/data sources): Data required: The surveys should be filled out by teachers,
and concerns the emotional and social behaviors of students. The full DESSA has 72 items that are rated
on a scale from “never” to “very frequently.” Another option in the DESSA-Mini, which is an 8-item
assessment with 4 different but equivalent versions.
Interpretations: The DESSA manual will be necessary to interpret the data. These initial interpretations
may provide information on how Fabretto students rank on a global average. It may also be helpful to
assess students over time to see how their social and emotional skills improve.
Limitations: The feasibility of using DESSA will depend on the generosity of the Devereux Center for
Resilient Children. Originally, DESSA costs $115.95 for the full kit, and additional packages of 25 forms
are $39.95. But, they may be willing to negotiate a more reasonable price since Fabretto is a non-profit
organization. Further, DESSA (although less for DESSA-mini) is time consuming for teachers, and make
take away from valuable instruction time.
Justification: A FUNIDES report found that Nicaraguan employers say that the most needed
competencies for employees are honesty, following basic norms of conduct, enthusiasm, respecting and
listening to superiors, and the ability to work in a group. The study showed that while social-emotional
skills were clearly most important, they were difficult to find, especially in individuals with lower
education levels. If Fabretto aims to improve individuals’ quality of life, particularly through improving
their livelihood, it is essential that students graduate from programs with the necessary skills to succeed
in the work force. Moreover, if Fabretto programs can gain the reputation of producing students who
demonstrate such socio-emotional skills, employers may see these schools as valuable recruiting sites.
Primary Questions to Consider for Next Steps:
Question 1: Would a paper and pencil version or a digital version of the assessment be most useful to
Fabretto?
Necessary knowledge to make decision:
● The paper and pencil version would be more costly.
● The paper and pencil version is much more time consuming in terms of scoring and
analysis.
● An alternative option is to have teachers fill out the DESSA using paper and pencil, and
the assessment can later be entered online. This would drastically reduce the necessary
time for teachers, but would require another individual to enter the information.
17
Question 2: There are three different versions of the DESSA that could be employed that are of short,
medium, and extensive lengths. Which would be most useful to Fabretto?
Necessary knowledge to make decision:
● The DESSA Mini has 8 questions, and would take about two minutes per student (if done
with paper and pencil)
○ This version provides just one score
○ This version would provide insight into change in children’s social and emotional
skills over time, but would not provide a strategy to improve them
● The DESSA Second Step Addition has 36 questions and 4 different scales, and would take
about 8 minutes per student (if done with paper and pencil)
○ This version would allow profiling of students and facilitate the creation of a
strategy
● The full DESSA has 72 questions, and would take about 20 minutes per student (if done
with paper and pencil)
○ This version provides multiple scores in different categories
○ This version provides profiles of students and information on how students’
social and emotional skills can be improved
*Important note: Because the digital version of the DESSA is much faster at scoring and analysis, this
type of implementation may make the longer DESSA versions more feasible.
(1) Preparation: Preparation would require the necessary decisions to be made (as outlined above), and
then the proper version of the DESSA to be secured. Negotiation will be necessary in determining costs.
Paper and pencil versions would have to be printed and then mailed to Nicaragua, while the digital
version would have to be set up on computers (if this option is selected, more information will be
provided on setup).
(2) Implementation: The DESSA will be filled out by teachers or school administrators with the help of
the DESSA manual.
Paper and pencil version: Paper assessments will need to be distributed to schools and classrooms
implementing DESSA. Teachers are given the DESSA manuel to walk them through the answering,
scoring, an analysis process. They must either score and analyze data by hand or send the paper
assessments to be entered digitally by another member of the Fabretto team.
Digital version: An annual fee must be paid to gain access to the web-based assessment. Educators and
administrators (who are familiar with students) fill out the DESSA on a computer with internet
connection.
(3) Analysis: Paper and pencil version: Completed by hand using a manual or if entered online,
completed with a scoring assistant. Digital version: Completed with a scoring assistant
(4) Dissemination of Results: The teacher or school administrator will have immediate access to the
results. They will report these back to Fabretto. The individual schools and teachers can assess DESSA
18
recommendations and strategies to improve social and emotional skills. Focus groups may facilitate this
process.
(5) Decision Making: Schools, in collaboration with Fabretto, can use DESSA recommendations in order
to strengthen students social and emotional skills. This will require conversation between the schools
and organization in order to devise a strategy that fits with Fabretto’s objectives while allowing schools
to tailor programs to their specific needs.
------------------------------------------------------------------------------------------------------------------------------------------
Indicator 4: Education Level of Students vs. Parents
Definition: Average number of years of schooling received by parents of current and past Fabretto
program participants vs. average number of years of schooling received by current and past Fabretto
program participants age 19 and over.
Purpose: Analyze whether or not Fabretto programs are encouraging students to stay in school longer
and whether or not they helping to develop the educational system offered to these communities over
the generations.
Calculation method: Parents’ education data should be collected upon the matriculation of their
children. The sum of the parents’ total years of schooling will be taken and then divided by the number
of parents who participated in the study. This will be directly compared with the sum of Fabretto
program participants age 19 and over, divided by the number of students who participated in the study
to potential developments in the education system brought about by Fabretto programs.
Formula:
Total years of parent schooling / n
Total years of schooling of current and past Fabretto program participants age 19 and over / n
Process (incl. data required/data sources):
Data required: years schooling of Fabretto participants’ parents, years of schooling of current and past
Fabretto participants over 19
Sources: Household surveys, school archives
Interpretations: If the average years of schooling attained by Fabretto participants are greater than the
average of their parents, it can be interpreted that Fabretto is achieving its goals to help Nicaraguans
“reach their full potential, improve their livelihoods, and take advantage of economic opportunity
through education.” It has been shown by various studies that every year of additional schooling raises
personal income for the individual, but also gross domestic product of the state itself.
Limitations: Similar to the calculation for School-Life Expectancy by UNESCO, this indicator runs the risk
of double-counting repeated years (p. 7, 2009). It also fails to take into consideration what type of
education was received during these years, how long the school years were, and attendance rates
during the year. Fabretto, at this point, does not have strong communication channels between the
19
organization itself and former program participants, which would have to improve in order to properly
measure this indicator.
Justification: UNESCO, the World Bank, and other prominent institutions agree that for each additional
year of schooling, the individual and the nation benefit economically (UNESCO, 2010 and
Psacharopoulos & Patrinos, 2002). Taking its inspiration from UNESCO’s School-Life Expectancy (p. 7,
2009), this indicator will show over the course of a generation whether or not Fabretto’s programs are
succeeding in empowering students, improving livelihoods, and increasing economic opportunity. It’s
crucial that Fabretto’s day-to-day activities and goals are streamlined into these broader, more longterm objectives through such measurements.
------------------------------------------------------------------------------------------------------------------------------------------
Indicator 5: Frontline SMS Follow-up Survey (FSFS)
Brief Description of Instrument: The Frontline SMS Follow-up Survey is a method of assessing the
physical, familial, and financial status of graduates of the SAT secondary school program. The survey is
sent out one multiple-choice question at a time to the mobile phones of the graduates. The survey
focuses on questions relating to health, income, and personal evaluation of the SAT program. Follow-up
surveys are used often to evaluate experimental education programs. Although these surveys are
usually administered through online software, the FSFS allows the surveys to be administered through
mobile phones. The results of the survey are organized automatically based on the answer choice
selected to simplify the analysis process.
Primary Questions to Consider for Next Steps:
Question 1: How many of the current SAT students own mobile phones?
 Affects the number of mobile phones and cellular service that Fabretto must purchase
to implement these surveys.
 If not enough current students own cellular phones, the surveys could be sent to
randomly selected graduates, rather than every single graduate.
Question 2: What are the specific survey questions that should be asked?
 Suggested questions:
o What is the graduate’s monthly income?
o What is the graduate’s employment status?
o What is the graduate’s job sector?
o What type of geographical area is the graduate living in? (Urban vs Rural)
o How many people live in the graduate’s household?
o How many people in the graduate’s household are employed?
o What is the graduate’s marital status?
o How many children does the graduate have?
o How many meals a day does the graduate eat?
o Does the graduate have any chronic/recurring health problems?
20
o
o
Does the graduate find the skills he/she learned at Fabretto useful in his/her
daily life?
How satisfied is the graduate in his/her Fabretto education?
(1) Preparation: Preparation would require the framing of the survey questions along with the answer
choices. Additionally, mobile phones with cellular service (device and minutes) must be accessible to
participants.
Follow the instructions below to set up the survey instrument using the freeware FrontlineSMS.
a) Download Frontline SMS from http://www.frontlinesms.com/
b) Open Frontline SMS (the program will open in web browser window, but doesn’t require
internet)
c) First, you must connect the FrontlineSMS program with a mobile network. I connected an
account with my personal mobile phone’s network by first installing the android app
FrontlineSync Beta onto my android smartphone. There are other methods as well, detailed on
the connection page.
d) Next, you should add all of the students’ phone numbers into the contacts and put them all into
a group. Click on the “People” tab at the top of the page. Then, click on the + sign next to the
groups tab on the left of the page. Follow the instructions to create a group.
e) Now, you should add the student’s phone numbers to the group. Click “Add Contact”. Follow
the instructions to add each student into the contacts. Make sure you add the student to the
group created earlier.
f) Now you can set up the survey question. Click on the Activities tab at the top of the screen.
Then, click on the “Create and Activity” option at the top right corner of the screen. Click the
“Poll” option from the options that appear.
g) Follow the instructions to create the response options.
i) Make sure you name the polls differently.
ii) Keywords are the words that, when sent back as a response to the survey, will register as
one of the set responses. You can have multiple keywords per response. For example, if you
are sending out a gender survey, you can have the response words m, male, man, and boy
all go into the category of “Male” when analyzed.
iii) The tags are the category names. For example, if sending out a gender poll, your categories
could be “male”, “female”, or “other”.
iv) If you wish to create a free-response question, you can leave the response options blank.
The responses will be filed into the “uncategorized” category.
h) Once finished, click save. You can turn on your question by clicking the “enabled” switch.
However, remember to only turn on one question at a time so as to not confuse the responses.
A graduate could be replying to one poll question, but if multiple questions are enabled, the
reply will be registered under all of the questions that had the reply as a keyword.
i) Create all of the questions by repeating steps (f) through (h).
(2) Implementation: In order to implement the project, the mobile phones with cellular service must be
disseminated among the soon-to-be graduates of Fabretto. Depending on the number of mobile phones
21
that the students already own and the number of mobile phones that Fabretto is willing to purchase, the
phones could be distributed either at random or to every graduate.
Every one or two weeks (depending on how many times a year you would like to record the graduates’
responses) Fabretto would send out a survey question.
1. Click on the “Inbox” tab at the top of the page.
2. Click on the “Send SMS” button on the top right corner of the screen. For recipients, click the
button “Show all Contacts.”
3. Select the group you created earlier with all of the students.
4. Type out the question and answer choices (make sure the answer choices include the keywords
you used when setting up the poll) into the text box.
5. Click “send.” The replies will be filed according to the keywords replied back.
6. Whenever you send out a new question, be sure to disable the previous poll in order to avoid
conflicting responses.
7. You can also send reminder messages to those who have not yet responded through the “Send
SMS” button in the Inbox tab.
(3) Analysis: The results are already organized. Click on the Activities tab in the top left side of the
screen. Scroll down to see each poll question that was set up. Click on whichever question you wish to
analyze. On the left side of the screen in a chart, you will see the number of responses in each answer
category.
The responses to each question should give the organization a rough idea as to the living, employment,
and health conditions of its graduates. The results will also let Fabretto know if their graduates find the
vocational-based training useful.
(4) Dissemination of Results: We suggest transcribing the results to an excel spreadsheet so that the
results can be disseminated with ease throughout the organization. Unfortunately, I do not believe
there is a quick way to do this through the FrontlineSMS software itself.
(5) Decision Making: The results of the FSFS are especially useful in the long term. The results of the
FSFS will give the organization an idea of the success of their SAT program and vocational training. The
graduates’ personal opinion about the vocational training programs (probably best framed as a freeresponse question) will be useful in tweaking such programs in the future.
(6) Possible Questions: There should be thirteen questions, sent out either every two weeks or week,
depending on whether Fabretto wants the surveys to be quarterly or semiannually. The questions
should ask about: (i) income/employment (ii) household (iii) health.
1) Are you employed?
a) Yes
b) No
2) What is your personal monthly income?
a) The answer choices should be ranges of incomes. I am not sure what the ranges should be,
but that can be determined by Fabretto.
22
3) In what job sector are you employed?
a) Agriculture
b) Services
c) Production (not agricultural)
d) Tech
e) Financial
f) Other
4) In what geographical area are you living?
a) Urban
b) Rural
5) How many people in your household are employed (including yourself)?
a) 0
b) 1
c) 2
d) 3
e) 4
f) 5
g) More than 5
6) How many children (if any) do you have?
a) 0
b) 1
c) 2
d) 3
e) 4
f) 5
g) More than 5
7) When was the last time you visited the doctor?
a) In the last week
b) In the last month
c) In the last 2-6 months
d) In the last 6-12 months
e) In the last 1-2 years
f) Longer
8) Health-related questions problems (muscle pains, breathing problems, etc.) and health
problems in the family.
------------------------------------------------------------------------------------------------------------------------------------------
23
Indicator 6: Gender Parity Index (GPI)
Definition: Ratio of female to male values of a given indicator. This can be attached to indicators
involving student and teacher populations, such as student performance, attendance, retention, access
to ICTs, and more (UNESCO Institute for Statistics, 2009).
Purpose: The GPI measures progress towards gender parity in education participation and/or learning
opportunities available for women in relation to those available to men. It also reflects the level of
women’s empowerment achieved through the program.
Calculation method: Divide the female value of a given indicator by that of the male.
Formula:
GPI = F/M
F = Female value of a given indicator; Example: 50 female students enrolled in primary school A
M = Male value of a given indicator; Example: 45 male students enrolled in primary school A
Process (incl. data required/data sources): Female and male values of a given indicator are required.
Interpretations: From UNESCO Technical Guidelines on Education Indicators: “A GPI equal to 1 indicates
parity between females and males. In general, a value less than 1 indicates disparity in favor of
boys/men and a value greater than 1 indicates disparity in favor of girls/women. However, the
interpretation should be the other way round for indicators that should ideally approach 0% (e.g.
repetition, dropout, illiteracy rates, etc.). In these cases, a GPI of less than 1 indicates a disparity in favor
of girls/women and a value greater than 1 indicates a disparity in favor of boys/men” (UNESCO Institute
for Statistics, 2009).
Limitations: The index does not demonstrate whether disparities are due to a random improvement in
one gender group, or of a more fundamental issue in the program.
Justification: Fabretto’s mission is to improve access to quality education for all children; and it
promotes collaboration and equality in all facets of its program. The GPI is a widely-used socioeconomic
index used by UNICEF, UNESCO, and education-related NGOS, often to measure the relative access to
education of males and females in a nation or community. Many international donors are eager to see
indicators that demonstrate female access to education, given the historic gender gap in global
education.
------------------------------------------------------------------------------------------------------------------------------------------
Indicator 7: Progress out of Poverty Index (PPI)
Brief Description of Instrument: The Progress Out of Poverty Index (PPI) is a tool used by businesses and
international organizations to measure poverty, defined as living bellow or marginally above the poverty
line. The PPI is a scorecard system that gives a poverty rating to a series of countries. The PPI is
determined from a series of 10 questions that are surveyed in households around the country. These 10
24
questions (i.e. are all household members aged 7 to 18 enrolled this school year in the formal education
system?) are chosen from a database of 200-1000 questions and are selected based on their correlation
with poverty, how inexpensive and easy they are to measure, and how liable they are to change over
time as the nation’s level of development changes. After the responses are collected and averaged, the
nation is given a score based on a scale of 0-100 (0 being most likely to be poor, 100 being least likely to
be poor).
Primary Questions to Consider for Next Steps.
1. What questions are most important to Fabretto to ask households?
a. Can PPI provide more information than just poverty ‘likelihood’?
b. What data would governmental agencies and communities be interested in monitoring?
2. What type of communication and coordination with PPI experts does Fabretto envision?
a. Can Fabretto prepare, implement and analyze PPI? Hire a third party?
3. What are the benefits and costs to Fabretto (and communities, Nicaraguan government
agencies, and other NGOs) associated with implementing PPI?
a. What is the likelihood that the government would be interested in PPI?
b. Would local government officials/agencies be willing to cooperate and coordinate with
Fabretto?
c. What are the possible ways for local communities to participate?
i. Enriching content of surveys; Administering surveys; Sharing results; Creating
action plans based on results
4. How often should the PPI be collected?
(1) Preparation: The PPI is relatively simple to use, but does require some training. People who will be
going into the field to collect data should be trained in the definitions of certain terms and concepts on
the scorecard. They should review the directions for collecting scores and how to interpret those scores.
This would require minimum resources from Fabretto. “A Simple Scorecard for Nicaragua” describes
these steps in great detail and provides the list of 10 questions, scorecard, household information chart,
and poverty conversion table. This document is also available in Spanish.
(2) Implementation: The PPI is easy to use and takes about ten minutes per household. Field workers
will need to go to the households in which they want to survey (i.e. Fabretto students). At the house,
they collect information about household members, their education, and occupation. Then, the field
worker should ask the head of household the ten indicator questions. The answers to these questions
should be recorded on paper and then entered into a database at Fabretto’s offices.
(3) Analysis: The PPI guide provides a chart to convert scores based on multiple international
organizations. These scores are converted into percentages that show the likelihood of poverty. A score
of 0 is ‘most likely to be poor’ and a score of 100 is ‘least likely to be poor’. The PPI can be used to score
individual households or a group of households.
(4) Dissemination of Results: Stakeholders to which the PPI results would be shared are: Fabretto; local
communities; Ministry of Education and Government agencies; and external individuals and
organziations.
25
(5) Decision-Making and Considerations:
1. Financial Resources Necessary to prepare, implement and analyze data;
2. Human Resources to administer surveys in communities;
3. Political Context: (A) Need to cooperate with local government; (B) Consider Somoto or
Cusmapa as pilot zone.
i. Explain the survey, the objectives, and the potential benefits to the community,
government and Fabretto.
ii. Share the guide with locals and adjust given their participation.
SAMPLE: Índice comunitario - Fabretto
-----------------------------------------------------------------------------------------------------------------------------------------Entidad
Nombre
Identificación
Fecha
(DD/MM/AA)
Participante: ______________________
__________________
Inscripción:
_______________
AFPF:
______________________
__________________
Hoy:
_______________
_____________________________________________________________________________________
Indicador
Respuesta
Puntos
Valor
-----------------------------------------------------------------------------------------------------------------------------------------1. ¿Cuántas personas viven en el hogar?
A. Ocho o más
0
B. Siete
5
C. Seis
9
D. Cinco
13
E. Cuatro
19
F. Tres
27
G. Uno o dos
41
-----------------------------------------------------------------------------------------------------------------------------------------2. En el presente año escolar, ¿se matricularon en el
A. No hay miembros 7 a 18
0
sistema de educación formal todos miembros
B. No
2
del hogar de las edades 7 a 18?
C. Si
6
-----------------------------------------------------------------------------------------------------------------------------------------3. En su ocupación principal en los últimos siete días, ¿cuántos
A. Ninguno
0
miembros del hogar trabajaron como empleados/obreros?
B. Uno
3
C. Dos o más
7
-----------------------------------------------------------------------------------------------------------------------------------------4. ¿De cuántos cuartos dispone el hogar (no incluya cocina, baños,
A. Uno
0
pasillos ni garaje)?
B. Dos
2
C. Tres o más
5
-----------------------------------------------------------------------------------------------------------------------------------------5. ¿Qué material predomina en
A. Tierra, u otro
0
el piso de la vivienda?
B. Madera (tambo), ladrillo de barro, o
5
embaldosado o concreto
26
C. Ladrillo de cemento, mosaico, terrazo o cerámica
10
-----------------------------------------------------------------------------------------------------------------------------------------6. ¿Qué combustible utilizan
A. Leña no comprada
0
usualmente para
B. Leña comprada, carbón, o no cocinan
5
cocinar?
C. Gas butano o propano, gas kerosén, electricidad
u otro
10
-----------------------------------------------------------------------------------------------------------------------------------------7. ¿Tiene este hogar una plancha?
A. No
0
B. Si
2
-----------------------------------------------------------------------------------------------------------------------------------------8. ¿Tiene este hogar una licuadora?
A. No
0
B. Si
4
-----------------------------------------------------------------------------------------------------------------------------------------9. ¿Con cuántos teléfonos celulares cuenta el hogar?
A. Ninguno
0
B. Uno
3
C. Dos o más
8
-----------------------------------------------------------------------------------------------------------------------------------------10. ¿Tiene este hogar una bicicleta, bote, caballo, burro, mulo,
A. No
0
motocicleta, o vehículo?
B. Si
6
-----------------------------------------------------------------------------------------------------------------------------------------Puntaje:
Índice comunitario - Fabretto
-----------------------------------------------------------------------------------------------------------------------------------------11. ¿Cuántos miembros del hogar son adultos?
A. Ocho o más
B. Siete
C. Seis
D. Cinco
E. Cuatro
F. Tres
G. Uno o dos
-----------------------------------------------------------------------------------------------------------------------------------------12. ¿Cuántos adultos del hogar trabajan?
A. Ocho o más
B. Siete
C. Seis
D. Cinco
E. Cuatro
F. Tres
G. Uno o dos
-----------------------------------------------------------------------------------------------------------------------------------------13. ¿Cuáles son los ingresos quincenales de los miembros quienes trabajan?
27
Nombre
(1)
(2)
(3)
(4)
(5)
(6)
(7)
(8)
______________________________
______________________________
______________________________
______________________________
______________________________
______________________________
______________________________
______________________________
Edad
Ganancias Quincenales
________
________
________
________
________
________
________
________
________________________
________________________
________________________
________________________
________________________
________________________
________________________
________________________
-----------------------------------------------------------------------------------------------------------------------------------------14. ¿Cuál es el estado propietario del hogar?
A. Rentada
B. Familiar
C. Propia
-----------------------------------------------------------------------------------------------------------------------------------------15. Si la casa es rentada, ¿cuánto cuesta mensual?
-----------------------------------------------------------------------------------------------------------------------------------------16. ¿Cuál es la fuente de agua en el hogar?
A. Tubería
B. Pozo propio
C. Pozo comunitario
Preparation and Implementation Guide: Pautas para la efectiva realización de la entrevista Según las
página 10–13 del Manual, “En la realización de la entrevista, se debe tener en cuenta algunos aspectos
que le pueden ayudar a conducir con éxito la misma. Presentación “Se requiere de usted como
encuestador(a) mucha atención y cuidado al momento de la entrevista. Dado que usted y el o la
informante no se conocen, se debe demostrar simpatía, sensibilidad e interés por el (la) informante.
“En el contacto inicial entre el encuestador(a) y el (la) informante, es conveniente que el (la)
encuestador(a) se presente amablemente, indicando la institución que representa, el objetivo de la
encuesta [‘aprender más en cuanto a como viven los participantes de la organización’], y solicitando la
disposición del o la informante para brindar la información requerida. Una introducción efectiva puede ser lo siguiente. ‘Buenos días. Soy <nombre>, encuestador/a de [su
organización] que está llevando a cabo una encuesta de [todos/una muestra] de sus participantes con la
finalidad de recabar información que permita conocer sus condiciones de vida. . . . Entonces, quisiera
hacerle algunas preguntas y espero que usted tenga la bondad de cooperar conmigo.’
“No debe presentarse realizando preguntas como ‘¿Está usted muy ocupado?’, ‘¿Puede concederme
unos minutos?’, o ‘No sé si será posible que conteste algunas preguntas?”, ya que este tipo de
preguntas sugieren respuestas negativas y es conveniente que inicie de forma positiva.
Confidencialidad de las respuestas: “Antes de realizar la primer pregunta, el o la encuestadora, debe dar
a conocer al (la) informante el carácter confidencial de la información, explicando que en ningún
28
momento se publicarán nombres de personas y que la información será utilizada solamente para fines
estadísticos.
Ambiente de la entrevista: “Sería preferible que el entrevistado escoja el lugar de la entrevista para que
esto facilite la comunicación. Este lugar debe permitir que la encuesta cumpla con el carácter de
confidencialidad.
Carácter privado de la entrevista: “Es muy importante que la entrevista se realice en privado. Evite en lo
posible realizar la entrevista en presencia de personas que no pertenecen al hogar, ya que el o la
informante podría alterar la respuesta u omitirla. De no ser posible, explique la necesidad de la
privacidad de la entrevista y pida cortésmente al o (la) acompañante, que los deje solos.
Neutralidad: “Acepte las respuestas del o la informante, y evite la crítica, opiniones personales, juicios,
aprobación o desacuerdo con sus palabras o gestos. Recuerde que no debe imponer al entrevistado(a)
sus opiniones, ni sugerir respuestas.
Control de la entrevista: “El o la encuestadora es quien dirige la entrevista y por lo tanto es quien debe
controlar la situación. Si el o (la) entrevistada da respuesta de temas ajenos o habla de asuntos que no
tienen que ver con la entrevista, no es necesario que se le interrumpa; pero en la primera
oportunidad—y con mucho tacto—realice de nuevo la pregunta y regrese al tema de la entrevista.
No asumir respuestas: “Las características socio-económicas y sociológicas de los entrevistados, como
área de residencia o condiciones de la vivienda, no deben llevar al o (la) encuestador(a) a asumir
respuestas o formarse expectativas anticipadas.
“Así mismo, no debe sugerir respuesta en razón del ‘aparente’ nivel cultural del entrevistado. Si el
formulario lo señala, deben realizarse preguntas de sondeo.
No apresurar la entrevista: “Las preguntas deben ser realizadas de forma tal que el o la informante
comprenda lo que se le está preguntando. Una vez hecha la pregunta debe dársele el tiempo necesario
para pensar, de lo contrario puede que le responda con evasivas.
Tratamiento con personas indecisas: “En muchas ocasiones el (la) entrevistado(a) responderá ‘No sé’, o
dará una respuesta con evasivas o simplemente rehusará a contestar más preguntas. En estos casos, el o
(la) encuestador(a) tratará de darle más confianza y hacerle sentir más cómodo antes de continuar con
la siguiente pregunta. Le explicará la importancia de sus respuestas.
Tratamiento con personas renuentes: “En ocasiones el (la) entrevistado(a) se negará a brindar la
información desde un inicio, o simplemente se rehusará a seguir contestando más preguntas. En estos
casos, el o (la) encuestador(a) intentará de forma cortés tratar de convencer al informante sobre la
importancia del estudio y la necesidad de que brinde la información.
Fin de la entrevista: “Una vez finalizada la entrevista y antes de despedirse del o la entrevistada, revise el
formulario por si ha omitido alguna pregunta o si quedó incompleta alguna respuesta. Agradezca la
colaboración prestada, y despídase.”
29
Indicator 8: Tripod Student Surveys
Brief Description of Instrument: Tripod Surveys offer student-sourced metrics for assessing teacher
quality. These instruments acknowledge that students have large stakes in teaching effectiveness and
are perhaps the best positioned to comment on their teachers’ performance. Tripod offers these
surveys free-of- charge and in Spanish, but only in digital format. The use of student-sourced metrics is
becoming increasingly popular in education, and has earned endorsement by the Bill and Melinda Gates
Foundation. This instrument measures results directly aligned with two of six outcomes outlined in
Fabretto’s strategic document.
Primary Questions to Consider for Next Steps:
1. Does Fabretto have the technical capabilities/infrastructure to implement digital surveys in
classrooms?
2. Will students and teachers have the necessary experience to complete and analyze digital
surveys?
3. If Fabretto does not have the technical capabilities to implement digital surveys, is it willing
to invest significant time and resources to translate the surveys into written format?
4. Once translated to written format, are the surveys easily analyzed? Will Fabretto invest time
and resources to manually analyzing survey results?
5. Is Fabretto interested in utilizing student-sourced metrics?
(1) Preparation
- PennGSE researchers and/or Fabretto may request Tripod Student Surveys in digital format3
- If necessary, PennGSE or Fabretto staff may translate digital surveys into written format.
- Identify schools, teachers, and classes of students to participate in pilot study
- Prepare necessary materials for completion of surveys
(2) Implementation
- Pilot implementation of student surveys in select schools and classes
(3) Analysis
- Aggregate results within each class and assess general feedback on teacher quality
- Draft recommendations to improve teaching practice
(4) Dissemination of Results
- Draft report on results and analysis for dissemination
- Inform teachers and school heads of survey results and recommendations
- Coordinate meeting of teachers and school heads to review survey results and consider future
steps
(5) Decision Making
- Based on survey results, emphasize, add to, or implement existing teacher training modules.
Alternatively, develop new modules to fill particular gaps in teaching practice
3
Official text form "Request to conduct research using tripod instruments or data" can be provided upon request.
30
-
Invite teachers to participate in new or re-designed PD modules
Evaluate effectiveness of new PD modules
Propose permanent changes to teacher training program in consultation with program
coordinators
Figure 1.2: Tripod Survey Sample Questions4
CARE
My teacher in this class makes me feel that s/he really cares about me.
→ Agree Strongly
→ Agree
→ Somewhat Agree
→ Neutral
→ Somewhat Disagree
→ Disagree
→ Strongly Disagree
My teacher really tries to understand how students feel about things.
CONTROL
Students in this class treat the teacher with respect.
Our class stays busy and doesn’t waste time.
CLARIFY
My teacher has several good ways to explain each topic that we cover in this class.
My teacher explains difficult things clearly.
CHALLENGE
In this class, we learn a lot almost every day.
In this class, we learn to correct our mistakes.
CAPTIVATE
My teacher makes lessons interesting.
I like the ways we learn in this class.
CONFER
Students speak up and share their ideas about class work.
My teacher respects my ideas and suggestions.
CONSOLIDATE
My teacher checks to make sure we understand what s/he is teaching us.
The comments that I get on my work in this class help me understand how to improve.
4
Source: Bill & Melinda Gates Foundation
31
Authors' Biographies
Matthew J. Tarditi, M.S.Ed., is a doctoral candidate in the Teaching, Learning, and Teacher
Education program at the University of Pennsylvania’s Graduate School of Education (PennGSE).
Matthew’s research and professional work resides at the intersection of international
educational development, school-community partnership, and the integration of film and
multimedia to foster participatory and collaborative approaches to research, knowledge
construction and overall practice. Tarditi is trained as a qualitative researcher steeped in
ethnography and visual anthropology with several years experience as a teacher, coach and
teacher-trainer. Currently, his applied research involves two non-profit organizations in
Nicaragua: the Seeds for Progress Foundation and the Fabretto Children's Foundation. Tarditi is
the co-founder and former director of camra, a student-lead, faculty-supported multimedia
pedagogy lab at the University of Pennsylvania. Matthew received his master’s degree from the
Education, Culture, and Society program at PennGSE. His master’s work included ethnographic
and qualitative methods, linguistics, technology in education, teacher professional
development in technology, and international teacher education. Contact him at
mtarditi@gse.upenn.edu or @mjtarditi
Will Slotznick is a sophomore at the University of Pennsylvania, studying International Relations
and Development. Will is passionate about international education, and has spent time in
Ghana and Nicaragua working on several rural education initiatives. At Penn, Will conducts
research relating to educational M&E, program assessment, and teacher training for both
international and local groups. He is actively involved in the Penn Graduate School of Education,
Wharton Social Impact Initiative, and Penn Society for International Development. Will serves
as the founding director of Global Impact Collaborative (PennGIC) and advises on projects
relating to M&E and program evaluation. Contact him at wslotz@sas.upenn.edu
Any Summer
Amarsingh Gawande is a freshman at the University of Pennsylvania studying Physics and
Economics. Amar is interested in international development and global education. In the past
six years, he has worked with NGOs and educational centers in India and rural Nepal. At Penn,
Amar has been involved with GIC for the past year, primarily conducting research relating to
educational M&E.
Edgar Agudelo
Conner Evans
32
Global Impact Collaborative (PennGIC) is a not-for-profit, student-run action-research team at
the University of Pennsylvania. PennGIC's mission is to leverage academic resources to help
international organizations achieve tangible social impact. As a sponsored branch of the Penn
Society for International Development (PennSID), GIC provides students with an opportunity to
work closely with non-profit leaders and perform team-based and independent research
related to international development.
References
Orazem, P., Glewwe, P., & Patrinos, H. (2009). The Benefits and Costs of Lowering Parental
Schooling Costs to Improve Educational Outcomes. Frederiksberg, Denmark:
Copenhagen Consensus Center.
Psacharopoulos, G. & Patrinos, H. (2002). Returns to Investment in Education: A Further
Update. World Bank elibrary. Retrieved from
http://elibrary.worldbank.org/doi/pdf/10.1596/1813-9450-2881.
33