Fabretto's Monitoring and Evaluation System: Recommendations and Possibilities for the Future EXECUTIVE SUMMARY June 2nd, 2015 Authored by: Matthew J. Tarditi, M.S.Ed. University of Pennsylvania, Graduate School of Education Will Slotznick, Amy Summer, Amarsingh Gawande, Edgar Agudelo, Conner Evans Global Impact Collaborative (PennGIC) Contributing Editor: Cesar Dubois Fabretto Children's Foundation TABLE OF CONTENTS Introduction 2 1. Overall Framework of M&E Evaluation 3 2. Evaluation Criteria 4 3. Recommendations 5 Appendices A. PENN Review Plan 2014-2015: Fabretto's M&E System 6 B. Indicator Evaluation Criteria 7 C. List of Recommended Indicators by Strategic Objective 8 D. Detailed Descriptions of Recommended Indicators 12 Indicator 1: Coefficient of Efficiency Indicator 2: Danielson Framework Indicator 3: Devereux Student Strengths Assessment (DESSA) Indicator 4: Education Level of Students vs. Parents Indicator 5: Frontline SMS Follow-up Survey (FSFS) Indicator 6: Gender Parity Index Indicator 7: Progress out of Poverty Index (PPI) Indicator 8: Tripod Student Surveys Authors' Biographies 32 References 33 1 INTRODUCTION Fabretto’s Monitoring and Evaluation (M&E) program attempts to streamline and advance the organization’s efforts to improve the lives of individuals and communities through education. To strengthen this crucial component of Fabretto’s work, the University of Pennsylvania’s Graduate School of Education has partnered with the Penn Society for International Development’s Global Impact Collaborative to analyze Fabretto's existing M&E framework and build its relevance and efficacy. Our first step involved building an understanding of Fabretto’s mission and goals as an organization, and then developing a logical, collaborative, and multi-step plan to carry out these actions. In the second step of this process we developed a framework of criteria to critically examine existing indicators. These were assessed on the degree to which they related to Fabretto’s objectives, fit within the organization’s resources, and advanced their mission through new partnerships and increased international relevancy. In all, the team undertook a five-phase process to evaluate Fabretto’s M&E program. This involved personal research, responding to feedback from Fabretto and Timshel, and a collective review process as an approach through which we were better informed to recommend specific indicators and to demonstrate their relation to Fabretto’s objectives and expected outcomes. We invite Fabretto to identify opportunities for improvement in our work and to consider modifications to the indicators, processes, and products that we have suggested. Fabretto’s M&E system is fundamental to their success in what is an increasingly important goal in the field of international development—access to quality education. "Of about 112 million children born annually in developing countries, 23% or about 26 million will not complete primary school. Of these 11.6 million will never start school, and getting them to complete the primary cycle will be costly" (Orazem, Glewwe & Patrinos, 2009, p. 4). While these statistics reveal the challenging future that NGOs and various development organizations face, a wide variety of literature supports the conclusion that an investment in education can bring substantial returns to the individual, community, and nation (Orazem, et al., 2009, p. 13; UNESCO, 2010; Psacharopoulos & Patrinos, 2002). Achieving “universal primary education” is even listed as the second United Nations Millennium Development Goal, aiming to ensure that all boys and girls can complete a full course of primary schooling by 2015. Understanding that the completion of schooling alone does not guarantee "quality" education, it is vital that Fabretto offers programming and utilizes measures that strengthen education and monitor and evaluate progress and results. Multiple organizations and studies support the importance of nutrition in both combating poverty and improving education (i.e. school enrollment and student performance), and therefore Fabretto emphasizes school nutrition programs in order to supplement their education interventions. As a result of Fabretto’s long-term involvement in Nicaraguan communities and established status within the country, its impact and monitoring and evaluation system can provide important insight for the field of international development. As Fabretto continues to grow and participate in the global movement to tackle poverty through education, health, and nutrition initiatives, it is crucial that it remains an engaged participant in relevant international discourse. While the team has developed a replicable model of evaluation for instruments and indicators, this ongoing involvement in international dialogue will facilitate perpetual monitoring 2 and assessment of the M&E system and allow for informed adjustments and updates that will ensure its continued functionality and resonance across Fabretto's communities. OVERALL FRAMEWORK OF M&E EVALUATION The framework for the evaluation of Fabretto's Monitoring and Evaluation System is comprised of six phases: (1) Phase One: Preliminary Research and Development of Evaluation Plan; (2) Phase Two: Develop Evaluation Criteria and Conduct Initial Evaluation of Potential Indicators; (3) Phase Three: Evaluation of Existing Indicators and Continued Evaluation of Potential Indicators; (4) Phase Four: Evaluation of Processes and Organizational Structure of M&E System; (5) Phase Five: Final Analysis and Recommendations; and (6) Phase Six: In-field Collaborative Review and Operationalization of M&E Recommendations with Fabretto staff. The specific activities and timeline for these six stages of the evaluation are outlined in depth in the Penn Review of Fabretto's M&E System chart (Appendix A). In Phase One we engaged with Fabretto staff and foundation documents to better understand both the mission and objectives of Fabretto as well as staff perspectives and understandings of the current M&E system and visions for the future. Additionally, the exploratory and preparatory phase sought to critically understand the international landscape of Monitoring and Evaluation, including the prevailing indicators and instruments used by foundations, non-governmental organizations, and institutions across the globe. It culminated with the creation of an overarching structure and logic to our evaluation. Next, we entered into Phase Two, during which we developed an Indicator Evaluation Criteria (discussed in the following section) to facilitate a systematic, rigorous examination of potential and existing indicators. The completion of the first report on potential indicators for Fabretto entitled Preliminary Review of Potential Indicators: An Evaluation, Rating and Analysis was followed by extensive review and critical feedback from Fabretto's directors and staff, a collaborative approach that enabled us to refine the evaluation criteria and to further incorporate priorities and perspectives from staff into our ongoing work. This signaled the start of the subsequent phase. Phase Three included a thorough review and refinement of the evaluation criteria followed by the identification of additional potential indicators (based on Fabretto feedback and an increased emphasis on alignment with the mission and objectives of Fabretto). To enable a holistic approach, we evaluated these additional indicators along with Fabretto's existing indicators and began to map the indicators onto the Strategic Plan 2015-2017 created by Timshel. In tandem to Phase Three, we examined the processes and organizational structure of Fabretto's M&E (Phase Four), including the mapping of two representative indicators (EGRA and MINED Statistics) in order to understand the steps, processes, involved parties, timelines and overall life of data within the system. Lastly, in Phase Five, we assembled the different pieces of the puzzle from the previous phases and conduct concentrated analysis across our work in order to produce a concise, comprehensive report on Fabretto's M&E system that focuses on recommendations and opportunities for improvement. The Executive Summary represents a condensed version of the final report, with an emphasis on recommended indicators and metrics. In the final report, we will include the procedural and organizational elements of the overall M&E System. 3 In the following section, we discuss the specific evaluation criteria we employed to examine potential and existing indicators for Fabretto's M&E system. EVALUATION CRITERIA Central to the evaluation of Fabretto's M&E System was the creation of a set of criteria to systematize and clarify the examination of indicators. Informed by application and feedback from Fabretto staff, the criteria, including weights and scoring, have been iterated over time and have evolved into their current state. Four major developments have occurred since the original criteria were established. First, the set of criteria itself has changed. Second, the weights assigned to each criterion have been adjusted. Weighting represents the numerical rankings of each individual criterion based on a review of literature and critical conversations with Fabretto staff. Third, the scoring for each criterion has been reduced from a 0-10 scale to 0-4. And lastly, detailed descriptions for each criterion and score have been added. The results of these changes are reflected in the overall evaluation criteria (Appendix B). All indicators, potential and existing, were evaluated using our criteria in order to enable comparability and to strengthen a more objective review and analysis. Furthermore, the criteria are a useful tool for Fabretto as the organization continues to periodically and systematically (re)consider metrics moving forward. Based on a wide spectrum of variables and issues related to synergy with Fabretto, implementation, resources, and audience, we created the following six criteria: (1) Alignment with Fabretto's Mission and Objectives; (2) Financial Sustainability; (3) Partnership Opportunities; (4) National and International Relevance; (5) Necessary Resources to Implement Indicator; and (6) Audience. The complete breakdown of criteria is included at the conclusion of this section; however, we will highlight two examples to demonstrate the reasoning behind our decisions. The most heavily weighted criterion, Alignment with Fabretto's Mission and Objectives, measures to what extent the metric aligns with Fabretto's Strategic Plan (expected results, objectives and targets), supports the mission and vision of Fabretto, and is consistent with Fabretto's values. The most important determinant of an indicator's inclusion in Fabretto's Monitoring and Evaluation system is whether or not the metric directly aligns with the goals and objectives of the organization. The second most weighted criterion, Financial Sustainability, refers to the direct and associated costs of an indicator and how those may be addressed through existing or potential financial partners. If Fabretto is unable to cover the expenses to utilize an indicator, regardless of its alignment to goals and objectives, it is significantly less feasible to include the metric into the M&E system. In sum, we developed the evaluation criteria as a systematic approach and a set of prioritized characteristics to provide Fabretto with a useful, practical tool in the face of a rapidly changing world of international M&E. The following section highlights our recommendations for indicators to be included in Fabretto's Monitoring and Evaluation System. 4 RECOMMENDATIONS In this section we present a final selection of 41 indicators to support Fabretto’s system of M&E. It includes sets of existing, adjusted, and new indicators, differentiated by color in the charts below. These recommended indicators are the final product of a multi-step investigative and evaluative process. Each stage in this process had been informed by feedback from our midyear Pilot Study. In line with the interests of Fabretto staff, particular attention has been given to finding and reviewing indicators for students, teachers, and graduates. Stage One of the evaluation involved individual research to identify a new set of potential indicators. Following this investigation (Stage Two), our team compiled a draft list of indicators through a collective review process. In Stage Three, team members applied the evaluation criteria to better understand the strength and relevance of each proposed indicator. Aided by the Timshel Strategic Plan, we considered the alignment of indicators to Fabretto’s mission, outcomes and objectives, and the feasibility of implementation. In Stage Four, the team engaged in a second round of collective review followed by an individual voting process (Stage Five) to narrow our draft list of indicators. The team held a third round of collective review (Stage Six) in which we shared votes and decided upon a final list of relevant indicators. Throughout this process, team members engaged with the providers and creators of select instruments and indicators (e.g. Tripod, DESSA, and Danielson). This engagement offered our team concrete ideas on the feasibility of new indicators, both in terms of implementation and analysis. Further, this outreach allowed us to explore potential partnerships between certain providers and Fabretto. In the appendices you will find the Final List of Indicators (Appendix C), which includes how each indicator corresponds to the outcomes and objectives from Timshel’s strategic plan, and details on the New Indicators (Appendix D) our team recommends. 5 Appendix A PENN Review Plan 2014-2015: Fabretto's M&E System Responsible Deadline Penn-Fabretto Penn-Fabretto Penn-Fabretto Penn-Fabretto 10/13/14 10/13/14 10/14/14 10/15/14 Phase One: Preliminary Research and Development of Evaluation Plan. 1.1. Preliminary Research of M&E System (via meetings, conversations and document review). 1.2. Create Evaluation Plan to Review Fabretto's M&E System. 1.3. Review Evaluation Plan with Fabretto Staff for Feedback. 1.4. Adjust Plan based on Fabretto feedback. Phase Two: Develop Evaluation Criteria and Conduct Initial Evaluation of Potential Indicators. 2.1. Create Indicator Evaluation Criteria. 2.2. Identify Potential Indicators to be reviewed. 2.3. Conduct Evaluation of Potential Indicators using Evaluation Criteria. 2.4. Produce Report on Potential Indicators for Fabretto. 2.5. Share Report with Fabretto Staff and Receive Feedback. 2.6. Review Feedback and Adjust Overall Plan for Review of Fabretto M&E System. Penn-Fabretto Penn-Fabretto Penn Penn Penn-Fabretto Penn-Fabretto 11/1/14 11/1/14 11/15/14 11/24/14 1/10/15 1/13/15 Phase Three: Evaluation of Existing Indicators and Continued Evaluation of Potential Indicators. 3.1. Adjust Indicator Evaluation Criteria. Penn-Fabretto 1/13/15 3.2. Review and Revise Evaluation Criteria Descriptions. 3.3. Identify Additional Potential Indicators based on Fabretto feedback. Penn-Fabretto Penn-Fabretto 1/13/15 2/19/15 3.4. Evaluate Fabretto's Existing Indicators. Penn-Fabretto 3/1/15 Penn Penn 3/18/15 4/10/15 4.1. Map the life of representative selections of data (EGRA & MINED data): Preparation, data collection and management, analysis, dissemination and decision-making. Fabretto-Penn 4/1/15 4.2. Revise Map of Data with Fabretto M&E team and Finalize Representative Sample. 4.3. Analyze Current Organizational Structure of M&E System. 4.4. Finish Review of Existing Fabretto Documents on Processes and Organizational Structure. Penn Penn Penn-Fabretto 4/21/15 4/22/15 5/22/15 5.1. Create Structure and Outline for Final Report. Penn-Fabretto 3/18/15 5.2. Share Proposed Structure and Outline with Fabretto for feedback. Penn-Fabretto 3/18/15 5.3. Cross-Reference Indicators-Instruments with Timshel's Strategic Plan Outcomes-Objectives. 5.4. Write Executive Summary for Board of Directors Meeting (May 7th). Penn-Fabretto Penn 3/18/15 4/29/15 5.5. Review Strategic Plan Matrix (Outcomes, Objectives, Indicators) with Timshel. Penn-Fabretto 5/1/15 5.6. Present Executive Summary to Board of Directors to elicit feedback. 5.7. Final Draft of Report with Analysis and Recommendations for Fabretto's M&E. Penn 5/7/15 6/1/15 3.5. Evaluate Additional Potential Indicators. 3.6. Determine Recommended Indicators for Future of Fabretto M&E. Phase Four: Evaluation of Procedural and Organizational Structures of M&E System. Phase Five: Cumulative Analysis and Draft of Final Recommendations Report. Penn Phase Six: Collaborative Review and Operationalization of M&E Recommendations with Fabretto staff. 6.1. Review Final Report with Fabretto Staff (i.e., Programs, M&E, Directors, Centers). 6.2. Select Indicators & Instruments for Pilot (work alongside Program Directors & M&E Staff). Penn Penn-Fabretto 6/12/15 6/19/15 6.3. Critically Consider Procedural and Organizational Recommendations for M&E. Penn-Fabretto 6/26/15 6.4. Design Indicator/Instrument Pilot Plan (2015). Penn-Fabretto 7/3/15 6.5. Devise Structure, Content and Overall Plan for M&E Workshops. Penn-Fabretto 7/24/15 6.6. Determine Penn Involvement with Fabretto M&E Post-August 2015 6.7. Draft Action Plan for M&E Post-August 2015. Penn-Fabretto Penn-Fabretto 8/1/15 8/15/15 6 Appendix B Indicator Evaluation Criteria Indicator Evaluation Criteria (Weight and Score Breakdown) Criteria Alignment with Fabretto Weight 10 The indicator helps to measure desired outcomes and impacts that align with the Strategic Plan (expected results, objectives and targets) and support the mission and Criteria Description vision of Fabretto. The indicator is consistent with Fabretto's values. Financial Sustainability 9 Sustainbility arises from manageable costs and financial partners to assist in covering these costs. Full implementation of an indicator (from preparation and data collection to analysis and decisionmaking) requires financial resources that can be provided internally and/or externally (i.e., funders, donors, partners and Fabretto). Partnership Opportunities National and International Relevance 8 7 Necessary Resources to Implement Indicator 6 Using the indicator could The position and The amount and type of provide increased or new relationship to national and resources (funds, personnel relationships with various international discussions hiring, duration of the NGOs, donor agencies, with respect to social, project, needed expertise individuals and/or public educational and health knowledge and skills) and private institutions. programs. The obtained required for the collection, results of Fabretto's processing, analysis and intervention are publication of data for comparable with statistics project implementation, from national and reporting and decisioninternational agencies. making. Audience 5 The people concerned with and interested in the information provided by the indicator (e.g. beneficiaries, donors, communities, Mined and government, Fabretto staff and Board of Directors, individuals and organizations). Score 0 1 2 There is no funding available to implement the There are no apparent indicator AND/OR the opportunities to partner indicator is prohibitively with other agencies or expensive to justify organizations. application. There are limited economic resources AND/OR The indicator aligns with at temporary funding to Affords little potential for least one objective of enable the full future partnerships. Fabretto's strategic plan. implementation of this relatively costly indicator. Not applicable because there is only funding for The indicator aligns with the start of the Provides at least one one or more of the implementation of this opportunity to partner with objectives, goals and indicator but not the end an organization, agency, values of Fabretto. AND/OR the indicator is institution, NGO or donor. moderately costly to implement. The values and objectives of Fabretto do not align with the data provided by this indicator. 3 4 There are short-term and long-term funding The indicator aligns with available to fully Fabretto's mission, vision, implement the indicator objectives and values. AND/OR the indicator costs little to nothing to implement. The resources necessary to implement the indicator are There is no audience for nearly impossible for this indicator. Fabretto to develop, access or deploy. The indicator has little to no relevance to prominent discussions and trends (nationally or internationally). The resources and Few people are interested expertise necessary to in the information provided implement far exceed by the indicator Fabretto's current capacity. There is moderate relevance to national OR international metrics, but not both. Fabretto is able to partially Some people are interested meet the resources required in the information provided to fully implement the by the indicator indicator. The indicator provides relevant metrics to national and international organizations, institutions and agencies focused on social, educational and health programs. The indicators affords Provides great and real highly relevant metrics on opportunities for the national and partnerships with different international scale. Data organizations, private provided is popularly institutions, the measured by respected government, etc. institutions, organizations and agencies. The indicator is relatively inexpensive to implement Affords potential for AND/OR funding is partnerships with several available to cover the organizations, institutions majority implementation and/or individuals. costs. It aligns with Fabretto's mission and most of its goals and objectives. The indicator has no relevance to the national and international context The indicator requires minimal resources Many people are interested AND/OR Fabretto can in the information meet the majority of the provided by the indicator. necessary resources to fully implement. Most (almost all) people are interested in the The necessary resources to information provided by fully implement the the indicator. Also, the indicator can be easily met indicator provides relevant by Fabretto. and/or useful information for multiple users. 7 Appendix C Final List of Indicators NEW INDICATOR Indicator ADJUSTED INDICATOR Instrument EXISTING INDICATOR Objective Poll graduating seniors (traditional & SAT) on parents' education and track SAT graduates yearly. 1 Education of Graduating Students vs Parents FSFS 1A 2 Gender Parity Index MINED Statistics 1A 3 Number of students matriculated in preschool education MINED Statistics 1A 3E 4 Number of students matriculated in primary education MINED Statistics 1A 3E MINED Statistics 1A 3E MINED Statistics 1A 3E MINED Statistics 1A 3E MINED Statistics 1A 3E PPI Survey 1A 1D 6 Training session attendance lists 1D 3D 6D List of parents who attend school/center/cha pel events 1D 3D 4C 5 6 7 8 9 10 11 Percentage of attendance in preschool students Percentage of promotion of primary students (aka Promotion Efficiency) Percentage of retention for preschool students Percentage of retention for primary students Progress out of Poverty Index (PPI) Number of mothers and fathers who have participated in 75% of the training, sensitization, and promotion workshops during the year Number of parents who collaborate in school and/or program activities (primary & SAT): Examples are gardens, road repair, use of land, food preparation. Notes/Additions 1D 6B 6C 6D To add: Case study of involved parents 8 Indicator 12 Post-Graduate Employment and Education Status 13 Teacher Quality 14 15 16 17 18 19 DESSA Danielson Framework (Planning and Preparation; Classroom Environment; Instruction; Professional Responsibility) Number of preschool teachers who have received pedagogical assistance visits (quarterly) Number of instructors and promoters trained (attended 75% of trainings) Number of parents trained in Early Stimulation (attended 75% of sessions). Number of correct responses (EGMA) Instrument FSFS - Frontline SMS Surveys (Graduation tracking) TRIPOD Student-sourced teacher evaluations DESSA Survey Objectives 1D 5A 5B 5C 2A 3A 3C 4B 3A Accompaniment Visit Form 3A Training session attendance lists Training session attendance lists EGMA Exam & Protocol 4D 3D 3D 3 20 Number of girls and boys benefited by the Early Stimulation program List of boys and girls, including names, ages, and community 3B 21 Percentage of preschool teachers trained Training session attendance lists 3E 22 Percentage of students that read with fluidity and comprehension (EGRA) EGRA Exam & Protocol 3 23 Case study of projects and entrepreneurial initiatives started by students Case study Guide* 3B 5D Pilot in select schools with Internet. *If online/internet 2C* 3A Danielson Form/Guide Notes/Additions 4A 3C # of boys and girls matriculated in the Early Stimulation program; % of student retention in Early Stimulation program Number of preschool teachers trained (attended 75% of trainings) 3E 5B 5C 5D Number of students involved in projects. *To be created with Fabretto staff 9 Indicator Instrument AFPF Statistics & Initial Matriculation AFPF Statistics & Youth Lists 24 Number of students who complete Educational Enrichment Program 25 Number of students who received scholarships. 26 Percentage of student promotion in VT (Vocational Training) program AFPF Statistics Percentage of student retention in VT courses according to gender, age and region Total dollar amount of scholarships awarded to students AFPF Statistics, Initial Matriculation Scholarship Information 29 Percentage of SAT staff trained (attend 75% of training sessions) Training session attendance lists 30 Number of on-site pedagogical advisory visits to SAT tutors 31 Number of pedagogically equipped classrooms (in preschool, primary, secondary and SAT) 32 Number of students matriculated in SAT program 33 Percentage of retention for SAT students 27 28 Visit Form for Technical Advisory Inventory of Central Storage, Inventory of Delivery from AFPF Centers. AFPF Statistics & Initial Matriculation AFPF Statistics, Final Matriculation Objectives 3 3B 3D Notes/Additions To add: More detailed division of activities 4A 5A 5C 5D 5A 5C 5D # and % of student attendance in FV program # and % of student attendance in FV program 4A # of SAT staff trained (attend 75% of training sessions) 4B 4D 4D 5B To add: Definition of pedagogically equipped classroom 4E 4E Grade, gender, and region # of students who pass the Public University Admission Exams in Spanish and/or Math 34 Percentage of passing students (Public University Admission Exam) in Spanish and/or Math Public University Admission Exam 5A 35 Number of daily school lunches supplied Fabretto Data Base 6A 36 Percentage of boys and girls from age 1-5 years with normal height for their age Fabretto Data Base 6A # of boys and girls age 1-5 with a normal height for their age 10 Indicator Instrument 37 Percentage of boys and girls from age 1-5 years with normal weight for their age Fabretto Data Base 6A 38 Percentage of schools that have adequate waste disposal systems Observations by AFPF technicians 6B 39 Number of home visits to families with children showing risk factors in biopsycho-social development Training session attendance lists 6D 6E 40 Number of households (mother, father, uncle, guardian) who attend four talks on early stimulation Training session attendance lists 6D 6E 41 Percentage of teachers and directors trained in topics of health and nutrition Training session attendance lists Objectives 6D Notes/Additions Number of boys and girls from age 1-5 years with normal weight for their age Number of teachers and directors trained in topics of health and nutrition (attend 75% of sessions) 11 Appendix D Detailed Descriptions of Recommended Indicators1 Indicator 1: Coefficient of Efficiency Definition: How many students should ideally graduate to the next level of school compared to how many students actually graduate (UNESCO Institute for Statistics, 2009). Purpose: Shows the efficiency of the education system, showing the quantity of dropouts and students who have to repeat a year in school. Calculation method: Divide the actual number of graduates by the ideal number of graduates, and then multiply by 100. Distilled Formula: Coefficient = Actual Amount of Graduates Ideal Number of Graduates X 100 Process (incl. data required/data sources): Need the number of graduates with and without repetition. Also need the amount of dropouts. Data will come from school census and registry. Interpretations: A high percentage indicates a high level of efficiency in students graduating without needing to repeat a year in school. Lower percentages indicate the amount of students dropping out and repeating years in school. Limitations: This indicator only shows the efficiency in graduating students, but does not show the exact level of achievement of the students. Justification: This indicator not only shows the retention rate of students, but also shows how many students are repeating years in school. It is promoted by UNESCO’s Institute for Statistics. ------------------------------------------------------------------------------------------------------------------------------------------ Indicator 2: Danielson Teaching Framework2 Definition: A framework for teaching that identifies the aspects of a teacher's responsibilities that promote improved student learning (supported by empirical studies). Organizes teaching among various areas of competence in which teachers need to develop expertise. There are twenty-two components clustered into four domains (listed below). Purpose: The framework provides a universal/shared vocabulary as a means to discuss quality and performance. It also offers a clear pathway and set of objectives within four teaching-related 1 More information and content are available upon request. Following a conversation about Danielson, core materials of the framework were shared with Danea Mairena for her review and feedback. 2 12 domains. Lastly, having four clear domains serves to focus the work of professional development and accompaniment activities. According to the Danielson Group website (http://danielsongroup.org/framework/), "the Framework for Teaching is a research-based set of components of instruction, aligned to the INTASC standards, and grounded in a constructivist view of learning and teaching." The InTASC provides a set of standards that "outline the common principles and foundations of teaching practice that cut across all subject areas and grade levels and that are necessary to improve student achievement" (CCSSO, 2011) The Framework may be used for many purposes, but its full value is realized as the foundation for professional conversations among practitioners as they seek to enhance their skill in the complex task of teaching. The Framework may be used as the foundation of a school or district’s mentoring, coaching, professional development, and teacher evaluation processes, thus linking all those activities together and helping teachers become more thoughtful practitioners. Developing a common understanding is critical to accuracy, teaching advancement, and the Framework’s impact on students’ core learning. Primary Questions to Consider for Next Steps 1. Which domains and components of the Danielson Framework are most relevant to Fabretto's primary school and secondary programming and the foundation's expected outcomes and objectives? 2. Will the Danielson Framework be utilized for supporting teacher development, evaluating teacher performance or a combination of the two approaches? 3. What school or center will be the pilot group to test the viability and impact of utilizing the Danielson framework? 4. Does Fabretto want to formally partner with Danielson Group or simply utilize the framework? If a formal partnership is the path, to what extent does Fabretto want to collaborate? a. Need to schedule an informational discussion with Danielson Framework to further discuss potential options and possible steps moving foward. 5. To what extent is Fabretto willing and able to cover the potential costs related to the Danielson Framework? a. (1) Translation of materials; (2) Travel, accommodations and per diem for Danielson experts to facilitate initial trainings; and (3) Additional training workshops for Fabretto staff and local stakeholders among others. Issues related to Preparation 1. Coordinate with the Danielson Group to determine next steps for potential collaboration with Fabretto 2. Define terms and specifics of communication and collaboration (informal vs. formal) 3. Identify potential costs: translation, Danielson-lead workshops (i.e., travel, accommodation, logistics) 4. Arrange workshops and facilitation sessions about the overall framework and its approach to evaluating and developing teachers 5. Translate Framework's central materials 6. Select specific domains and components to utilize for evaluative and/or development purposes 13 7. Determine the purpose of the Framework - Which domains does Fabretto use for teacher evaluation and which do they use for teacher professional development? 8. Organize workshops on the Frameworks' domains and components Four Domains of Teacher Evaluation Four Levels of Performance: (1) Unsatisfactory, (2) Basic, (3) Proficient, (4) Distinguished Critical Attributes of Proficient and Distinguished Levels and Possible Examples Domain 1: Planning and Preparation Domain 2: The Classroom Environment Domain 3: Instruction Domain 4: Professional Responsibility Domain 1: Planning and Preparation Effective teachers plan and prepare for lessons using their extensive knowledge of the content area, the relationships among different strands within the content and between the subject and other disciplines, and their students’ prior understanding of the subject. Instructional outcomes are clear, represent important learning in the subject, and are aligned to the curriculum. The instructional design includes learning activities that are well sequenced and require all students to think, problem solve, inquire, and defend conjectures and opinions. Effective teachers design formative assessments to monitor learning, and they provide the information needed to differentiate instruction. Measures of student learning align with the curriculum, enabling students to demonstrate their understanding in more than one way. Domain 2: Classroom Environment Effective teachers organize their classrooms so that all students can learn. They maximize instructional time and foster respectful interactions with and among students, ensuring that students find the classroom a safe place to take intellectual risks. Students themselves make a substantive contribution to the effective functioning of the class by assisting with classroom procedures, ensuring effective use of physical space, and supporting the learning of classmates. Students and teachers work in ways that demonstrate their belief that hard work will result in higher levels of learning. Student behavior is consistently appropriate, and the teacher’s handling of infractions is subtle, preventive, and respectful of students’ dignity. Domain 3: Instruction In the classrooms of accomplished teachers, all students are highly engaged in learning. They make significant contributions to the success of the class through participation in high-level discussions and active involvement in their learning and the learning of others. Teacher explanations are clear and invite student intellectual engagement. The teacher’s feedback is specific to learning goals and rubrics and offers concrete suggestions for improvement. As a result, students understand their progress in learning the content and can explain the learning goals and what they need to do in order to improve. Effective teachers recognize their responsibility for student learning and make adjustments, as needed, to ensure student success. Domain 4: Professional Responsibilities Accomplished teachers have high ethical standards and a deep sense of professionalism, 14 focused on improving their own teaching and supporting the ongoing learning of colleagues. Their record-keeping systems are efficient and effective, and they communicate with families clearly, frequently, and with cultural sensitivity. Accomplished teachers assume leadership roles in both school and LEA projects, and they engage in a wide range of professional development activities to strengthen their practice. Reflection on their own teaching results in ideas for improvement that are shared across professional learning communities and contribute to improving the practice of all. Calculation method: For each domain there are specific components comprised of two to five elements to describe the specific characteristics of the component. Using the teaching performance rubric (i.e., unsatisfactory, basic, proficient, distinguished), teachers and evaluators can assess each component and provide a roadmap for improvement in teaching. The Danielson Framework is a supportive, interactive tool that is best used in a collaborative, conversational manner between teachers and evaluators. Formula: Although the tool is not best used to provide an overall score for reporting purposes, one could calculate the numerical score for each element within each component in all four domains to reach an overall total. For example, the maximum score for the 22 components would be 88 (22 components multiplied by 4 for Distinguished). Process (incl. data required/data sources): The first step is to present and discuss the Danielson framework to teachers in order to clarify the goals, domains, components and specific elements of the teaching profession. Once the initial workshop has been delivered, teachers will self-assess their status based on the domains and evaluation rubric. Concomitantly, evaluators can assess teachers using the same framework and then discuss the results in order to plan strategies for enhancement and/or improvement. Interpretations: Standardized teaching indicators and assessment have widespread critics and detractors. They lack a nuanced, flexible framework that can truly account for local, individualized characteristics that may not fit smoothly into the predetermined, universal rubric. Limitations: Great emphasis, power and interpretative license are given to the observer. He or she must be an expert observer and pedagogue in order to offer critical feedback and constructive suggestions to the teachers being observed. Additionally, observations have long been criticized for their limited view of the classroom, teacher and student, based on the minimal time in the class, thus providing an extremely incomplete account of the teacher's performance. Besides the subjectivity of and extreme reliance on the observer, the traditional, observational approach lacks a complete, nuanced picture of the teacher's performance due to the narrow window into the classroom dynamic and pedagogical practices of teachers. Justification: Although constrained by the expertise of the observer and the focus on a universal framework of assessment, the Danielson framework can clarify the vocabulary around effective teaching and provide scaffolding for teachers and schools in the identification, enhancement and development of propitious practices and performance for the improvement of student learning. Example from NYC Department of Education's Adaptation of the Danielson Framework 15 Adopted the framework for evaluation and development approaches (NYC Department of Education, 2014). 1. Rubric components: School leaders will rate teachers for evaluative purposes on eight components of the Framework: 1a Demonstrating Knowledge of Content and Pedagogy 1e Designing Coherent Instruction 2a Creating an Environment of Respect and Rapport 2d Managing Student Behavior 3b Using Questioning and Discussion Techniques 3c Engaging Students in Learning 3d Using Assessment in Instruction 4e Growing and Developing Professionally *The entire Danielson Framework for Teaching (2013 Edition) will be used for formative purposes. 2. Forms and evidence: A new evaluator form will be created and will apply to all observation options. The new form will require evaluators to provide lesson-specific evidence gathered during an observation for all components rated, and it will enable supervisors to include Domain 1 or 4 evidence observed within 15 school days prior to the classroom observation as part of an assessment of a teacher’s preparation and professionalism. 3. Timely feedback: To help ensure that teachers are provided with timely feedback on their practice, evaluators will be required to provide lesson-specific feedback to teachers (verbally or in writing) within 15 school days of an observation. Evaluator forms must be completed within 45 school days of an observation; in order to ensure that teachers get constructive and timely feedback, evaluators can only conduct one additional evaluative observation between the time of an observation and the completion of the form for that observation. ------------------------------------------------------------------------------------------------------------------------------Indicator 3: Devereux Student Strengths Assessment (DESSA) Definition: The Devereux Student Strengths Assessment is a method of assessing social-emotional competencies of students in kindergarten through the 8th grade. The assessments can be completed by teachers/school administrators or parents/guardians. It focuses on the students’ awareness of their emotions and how they manage them, depending on the context. The DESSA tool can also provide classroom profiles that may help teachers manage and improve social-emotional intelligence. Finally, it has been used in the past at many different levels, including individual, classroom, school and district, and they have also been used abroad by NGOs. Measured elements include self-awareness, socialawareness, self-management, goal-directed behavior, relationship skills, personal responsibility, decision making, and optimistic thinking. According to the report, the profiles are also very easy to read and understand. 16 Purpose: Analyze whether or not Fabretto programs are helping students to develop the socialemotional skills that are highly valued by Nicaraguan employers. Calculation method: DESSA comes with a manual that describes the scoring process, and also provides instructions on how to interpret and understand results. Formula: Unknown. Will be provided in the DESSA handbook. Process (incl. data required/data sources): Data required: The surveys should be filled out by teachers, and concerns the emotional and social behaviors of students. The full DESSA has 72 items that are rated on a scale from “never” to “very frequently.” Another option in the DESSA-Mini, which is an 8-item assessment with 4 different but equivalent versions. Interpretations: The DESSA manual will be necessary to interpret the data. These initial interpretations may provide information on how Fabretto students rank on a global average. It may also be helpful to assess students over time to see how their social and emotional skills improve. Limitations: The feasibility of using DESSA will depend on the generosity of the Devereux Center for Resilient Children. Originally, DESSA costs $115.95 for the full kit, and additional packages of 25 forms are $39.95. But, they may be willing to negotiate a more reasonable price since Fabretto is a non-profit organization. Further, DESSA (although less for DESSA-mini) is time consuming for teachers, and make take away from valuable instruction time. Justification: A FUNIDES report found that Nicaraguan employers say that the most needed competencies for employees are honesty, following basic norms of conduct, enthusiasm, respecting and listening to superiors, and the ability to work in a group. The study showed that while social-emotional skills were clearly most important, they were difficult to find, especially in individuals with lower education levels. If Fabretto aims to improve individuals’ quality of life, particularly through improving their livelihood, it is essential that students graduate from programs with the necessary skills to succeed in the work force. Moreover, if Fabretto programs can gain the reputation of producing students who demonstrate such socio-emotional skills, employers may see these schools as valuable recruiting sites. Primary Questions to Consider for Next Steps: Question 1: Would a paper and pencil version or a digital version of the assessment be most useful to Fabretto? Necessary knowledge to make decision: ● The paper and pencil version would be more costly. ● The paper and pencil version is much more time consuming in terms of scoring and analysis. ● An alternative option is to have teachers fill out the DESSA using paper and pencil, and the assessment can later be entered online. This would drastically reduce the necessary time for teachers, but would require another individual to enter the information. 17 Question 2: There are three different versions of the DESSA that could be employed that are of short, medium, and extensive lengths. Which would be most useful to Fabretto? Necessary knowledge to make decision: ● The DESSA Mini has 8 questions, and would take about two minutes per student (if done with paper and pencil) ○ This version provides just one score ○ This version would provide insight into change in children’s social and emotional skills over time, but would not provide a strategy to improve them ● The DESSA Second Step Addition has 36 questions and 4 different scales, and would take about 8 minutes per student (if done with paper and pencil) ○ This version would allow profiling of students and facilitate the creation of a strategy ● The full DESSA has 72 questions, and would take about 20 minutes per student (if done with paper and pencil) ○ This version provides multiple scores in different categories ○ This version provides profiles of students and information on how students’ social and emotional skills can be improved *Important note: Because the digital version of the DESSA is much faster at scoring and analysis, this type of implementation may make the longer DESSA versions more feasible. (1) Preparation: Preparation would require the necessary decisions to be made (as outlined above), and then the proper version of the DESSA to be secured. Negotiation will be necessary in determining costs. Paper and pencil versions would have to be printed and then mailed to Nicaragua, while the digital version would have to be set up on computers (if this option is selected, more information will be provided on setup). (2) Implementation: The DESSA will be filled out by teachers or school administrators with the help of the DESSA manual. Paper and pencil version: Paper assessments will need to be distributed to schools and classrooms implementing DESSA. Teachers are given the DESSA manuel to walk them through the answering, scoring, an analysis process. They must either score and analyze data by hand or send the paper assessments to be entered digitally by another member of the Fabretto team. Digital version: An annual fee must be paid to gain access to the web-based assessment. Educators and administrators (who are familiar with students) fill out the DESSA on a computer with internet connection. (3) Analysis: Paper and pencil version: Completed by hand using a manual or if entered online, completed with a scoring assistant. Digital version: Completed with a scoring assistant (4) Dissemination of Results: The teacher or school administrator will have immediate access to the results. They will report these back to Fabretto. The individual schools and teachers can assess DESSA 18 recommendations and strategies to improve social and emotional skills. Focus groups may facilitate this process. (5) Decision Making: Schools, in collaboration with Fabretto, can use DESSA recommendations in order to strengthen students social and emotional skills. This will require conversation between the schools and organization in order to devise a strategy that fits with Fabretto’s objectives while allowing schools to tailor programs to their specific needs. ------------------------------------------------------------------------------------------------------------------------------------------ Indicator 4: Education Level of Students vs. Parents Definition: Average number of years of schooling received by parents of current and past Fabretto program participants vs. average number of years of schooling received by current and past Fabretto program participants age 19 and over. Purpose: Analyze whether or not Fabretto programs are encouraging students to stay in school longer and whether or not they helping to develop the educational system offered to these communities over the generations. Calculation method: Parents’ education data should be collected upon the matriculation of their children. The sum of the parents’ total years of schooling will be taken and then divided by the number of parents who participated in the study. This will be directly compared with the sum of Fabretto program participants age 19 and over, divided by the number of students who participated in the study to potential developments in the education system brought about by Fabretto programs. Formula: Total years of parent schooling / n Total years of schooling of current and past Fabretto program participants age 19 and over / n Process (incl. data required/data sources): Data required: years schooling of Fabretto participants’ parents, years of schooling of current and past Fabretto participants over 19 Sources: Household surveys, school archives Interpretations: If the average years of schooling attained by Fabretto participants are greater than the average of their parents, it can be interpreted that Fabretto is achieving its goals to help Nicaraguans “reach their full potential, improve their livelihoods, and take advantage of economic opportunity through education.” It has been shown by various studies that every year of additional schooling raises personal income for the individual, but also gross domestic product of the state itself. Limitations: Similar to the calculation for School-Life Expectancy by UNESCO, this indicator runs the risk of double-counting repeated years (p. 7, 2009). It also fails to take into consideration what type of education was received during these years, how long the school years were, and attendance rates during the year. Fabretto, at this point, does not have strong communication channels between the 19 organization itself and former program participants, which would have to improve in order to properly measure this indicator. Justification: UNESCO, the World Bank, and other prominent institutions agree that for each additional year of schooling, the individual and the nation benefit economically (UNESCO, 2010 and Psacharopoulos & Patrinos, 2002). Taking its inspiration from UNESCO’s School-Life Expectancy (p. 7, 2009), this indicator will show over the course of a generation whether or not Fabretto’s programs are succeeding in empowering students, improving livelihoods, and increasing economic opportunity. It’s crucial that Fabretto’s day-to-day activities and goals are streamlined into these broader, more longterm objectives through such measurements. ------------------------------------------------------------------------------------------------------------------------------------------ Indicator 5: Frontline SMS Follow-up Survey (FSFS) Brief Description of Instrument: The Frontline SMS Follow-up Survey is a method of assessing the physical, familial, and financial status of graduates of the SAT secondary school program. The survey is sent out one multiple-choice question at a time to the mobile phones of the graduates. The survey focuses on questions relating to health, income, and personal evaluation of the SAT program. Follow-up surveys are used often to evaluate experimental education programs. Although these surveys are usually administered through online software, the FSFS allows the surveys to be administered through mobile phones. The results of the survey are organized automatically based on the answer choice selected to simplify the analysis process. Primary Questions to Consider for Next Steps: Question 1: How many of the current SAT students own mobile phones? Affects the number of mobile phones and cellular service that Fabretto must purchase to implement these surveys. If not enough current students own cellular phones, the surveys could be sent to randomly selected graduates, rather than every single graduate. Question 2: What are the specific survey questions that should be asked? Suggested questions: o What is the graduate’s monthly income? o What is the graduate’s employment status? o What is the graduate’s job sector? o What type of geographical area is the graduate living in? (Urban vs Rural) o How many people live in the graduate’s household? o How many people in the graduate’s household are employed? o What is the graduate’s marital status? o How many children does the graduate have? o How many meals a day does the graduate eat? o Does the graduate have any chronic/recurring health problems? 20 o o Does the graduate find the skills he/she learned at Fabretto useful in his/her daily life? How satisfied is the graduate in his/her Fabretto education? (1) Preparation: Preparation would require the framing of the survey questions along with the answer choices. Additionally, mobile phones with cellular service (device and minutes) must be accessible to participants. Follow the instructions below to set up the survey instrument using the freeware FrontlineSMS. a) Download Frontline SMS from http://www.frontlinesms.com/ b) Open Frontline SMS (the program will open in web browser window, but doesn’t require internet) c) First, you must connect the FrontlineSMS program with a mobile network. I connected an account with my personal mobile phone’s network by first installing the android app FrontlineSync Beta onto my android smartphone. There are other methods as well, detailed on the connection page. d) Next, you should add all of the students’ phone numbers into the contacts and put them all into a group. Click on the “People” tab at the top of the page. Then, click on the + sign next to the groups tab on the left of the page. Follow the instructions to create a group. e) Now, you should add the student’s phone numbers to the group. Click “Add Contact”. Follow the instructions to add each student into the contacts. Make sure you add the student to the group created earlier. f) Now you can set up the survey question. Click on the Activities tab at the top of the screen. Then, click on the “Create and Activity” option at the top right corner of the screen. Click the “Poll” option from the options that appear. g) Follow the instructions to create the response options. i) Make sure you name the polls differently. ii) Keywords are the words that, when sent back as a response to the survey, will register as one of the set responses. You can have multiple keywords per response. For example, if you are sending out a gender survey, you can have the response words m, male, man, and boy all go into the category of “Male” when analyzed. iii) The tags are the category names. For example, if sending out a gender poll, your categories could be “male”, “female”, or “other”. iv) If you wish to create a free-response question, you can leave the response options blank. The responses will be filed into the “uncategorized” category. h) Once finished, click save. You can turn on your question by clicking the “enabled” switch. However, remember to only turn on one question at a time so as to not confuse the responses. A graduate could be replying to one poll question, but if multiple questions are enabled, the reply will be registered under all of the questions that had the reply as a keyword. i) Create all of the questions by repeating steps (f) through (h). (2) Implementation: In order to implement the project, the mobile phones with cellular service must be disseminated among the soon-to-be graduates of Fabretto. Depending on the number of mobile phones 21 that the students already own and the number of mobile phones that Fabretto is willing to purchase, the phones could be distributed either at random or to every graduate. Every one or two weeks (depending on how many times a year you would like to record the graduates’ responses) Fabretto would send out a survey question. 1. Click on the “Inbox” tab at the top of the page. 2. Click on the “Send SMS” button on the top right corner of the screen. For recipients, click the button “Show all Contacts.” 3. Select the group you created earlier with all of the students. 4. Type out the question and answer choices (make sure the answer choices include the keywords you used when setting up the poll) into the text box. 5. Click “send.” The replies will be filed according to the keywords replied back. 6. Whenever you send out a new question, be sure to disable the previous poll in order to avoid conflicting responses. 7. You can also send reminder messages to those who have not yet responded through the “Send SMS” button in the Inbox tab. (3) Analysis: The results are already organized. Click on the Activities tab in the top left side of the screen. Scroll down to see each poll question that was set up. Click on whichever question you wish to analyze. On the left side of the screen in a chart, you will see the number of responses in each answer category. The responses to each question should give the organization a rough idea as to the living, employment, and health conditions of its graduates. The results will also let Fabretto know if their graduates find the vocational-based training useful. (4) Dissemination of Results: We suggest transcribing the results to an excel spreadsheet so that the results can be disseminated with ease throughout the organization. Unfortunately, I do not believe there is a quick way to do this through the FrontlineSMS software itself. (5) Decision Making: The results of the FSFS are especially useful in the long term. The results of the FSFS will give the organization an idea of the success of their SAT program and vocational training. The graduates’ personal opinion about the vocational training programs (probably best framed as a freeresponse question) will be useful in tweaking such programs in the future. (6) Possible Questions: There should be thirteen questions, sent out either every two weeks or week, depending on whether Fabretto wants the surveys to be quarterly or semiannually. The questions should ask about: (i) income/employment (ii) household (iii) health. 1) Are you employed? a) Yes b) No 2) What is your personal monthly income? a) The answer choices should be ranges of incomes. I am not sure what the ranges should be, but that can be determined by Fabretto. 22 3) In what job sector are you employed? a) Agriculture b) Services c) Production (not agricultural) d) Tech e) Financial f) Other 4) In what geographical area are you living? a) Urban b) Rural 5) How many people in your household are employed (including yourself)? a) 0 b) 1 c) 2 d) 3 e) 4 f) 5 g) More than 5 6) How many children (if any) do you have? a) 0 b) 1 c) 2 d) 3 e) 4 f) 5 g) More than 5 7) When was the last time you visited the doctor? a) In the last week b) In the last month c) In the last 2-6 months d) In the last 6-12 months e) In the last 1-2 years f) Longer 8) Health-related questions problems (muscle pains, breathing problems, etc.) and health problems in the family. ------------------------------------------------------------------------------------------------------------------------------------------ 23 Indicator 6: Gender Parity Index (GPI) Definition: Ratio of female to male values of a given indicator. This can be attached to indicators involving student and teacher populations, such as student performance, attendance, retention, access to ICTs, and more (UNESCO Institute for Statistics, 2009). Purpose: The GPI measures progress towards gender parity in education participation and/or learning opportunities available for women in relation to those available to men. It also reflects the level of women’s empowerment achieved through the program. Calculation method: Divide the female value of a given indicator by that of the male. Formula: GPI = F/M F = Female value of a given indicator; Example: 50 female students enrolled in primary school A M = Male value of a given indicator; Example: 45 male students enrolled in primary school A Process (incl. data required/data sources): Female and male values of a given indicator are required. Interpretations: From UNESCO Technical Guidelines on Education Indicators: “A GPI equal to 1 indicates parity between females and males. In general, a value less than 1 indicates disparity in favor of boys/men and a value greater than 1 indicates disparity in favor of girls/women. However, the interpretation should be the other way round for indicators that should ideally approach 0% (e.g. repetition, dropout, illiteracy rates, etc.). In these cases, a GPI of less than 1 indicates a disparity in favor of girls/women and a value greater than 1 indicates a disparity in favor of boys/men” (UNESCO Institute for Statistics, 2009). Limitations: The index does not demonstrate whether disparities are due to a random improvement in one gender group, or of a more fundamental issue in the program. Justification: Fabretto’s mission is to improve access to quality education for all children; and it promotes collaboration and equality in all facets of its program. The GPI is a widely-used socioeconomic index used by UNICEF, UNESCO, and education-related NGOS, often to measure the relative access to education of males and females in a nation or community. Many international donors are eager to see indicators that demonstrate female access to education, given the historic gender gap in global education. ------------------------------------------------------------------------------------------------------------------------------------------ Indicator 7: Progress out of Poverty Index (PPI) Brief Description of Instrument: The Progress Out of Poverty Index (PPI) is a tool used by businesses and international organizations to measure poverty, defined as living bellow or marginally above the poverty line. The PPI is a scorecard system that gives a poverty rating to a series of countries. The PPI is determined from a series of 10 questions that are surveyed in households around the country. These 10 24 questions (i.e. are all household members aged 7 to 18 enrolled this school year in the formal education system?) are chosen from a database of 200-1000 questions and are selected based on their correlation with poverty, how inexpensive and easy they are to measure, and how liable they are to change over time as the nation’s level of development changes. After the responses are collected and averaged, the nation is given a score based on a scale of 0-100 (0 being most likely to be poor, 100 being least likely to be poor). Primary Questions to Consider for Next Steps. 1. What questions are most important to Fabretto to ask households? a. Can PPI provide more information than just poverty ‘likelihood’? b. What data would governmental agencies and communities be interested in monitoring? 2. What type of communication and coordination with PPI experts does Fabretto envision? a. Can Fabretto prepare, implement and analyze PPI? Hire a third party? 3. What are the benefits and costs to Fabretto (and communities, Nicaraguan government agencies, and other NGOs) associated with implementing PPI? a. What is the likelihood that the government would be interested in PPI? b. Would local government officials/agencies be willing to cooperate and coordinate with Fabretto? c. What are the possible ways for local communities to participate? i. Enriching content of surveys; Administering surveys; Sharing results; Creating action plans based on results 4. How often should the PPI be collected? (1) Preparation: The PPI is relatively simple to use, but does require some training. People who will be going into the field to collect data should be trained in the definitions of certain terms and concepts on the scorecard. They should review the directions for collecting scores and how to interpret those scores. This would require minimum resources from Fabretto. “A Simple Scorecard for Nicaragua” describes these steps in great detail and provides the list of 10 questions, scorecard, household information chart, and poverty conversion table. This document is also available in Spanish. (2) Implementation: The PPI is easy to use and takes about ten minutes per household. Field workers will need to go to the households in which they want to survey (i.e. Fabretto students). At the house, they collect information about household members, their education, and occupation. Then, the field worker should ask the head of household the ten indicator questions. The answers to these questions should be recorded on paper and then entered into a database at Fabretto’s offices. (3) Analysis: The PPI guide provides a chart to convert scores based on multiple international organizations. These scores are converted into percentages that show the likelihood of poverty. A score of 0 is ‘most likely to be poor’ and a score of 100 is ‘least likely to be poor’. The PPI can be used to score individual households or a group of households. (4) Dissemination of Results: Stakeholders to which the PPI results would be shared are: Fabretto; local communities; Ministry of Education and Government agencies; and external individuals and organziations. 25 (5) Decision-Making and Considerations: 1. Financial Resources Necessary to prepare, implement and analyze data; 2. Human Resources to administer surveys in communities; 3. Political Context: (A) Need to cooperate with local government; (B) Consider Somoto or Cusmapa as pilot zone. i. Explain the survey, the objectives, and the potential benefits to the community, government and Fabretto. ii. Share the guide with locals and adjust given their participation. SAMPLE: Índice comunitario - Fabretto -----------------------------------------------------------------------------------------------------------------------------------------Entidad Nombre Identificación Fecha (DD/MM/AA) Participante: ______________________ __________________ Inscripción: _______________ AFPF: ______________________ __________________ Hoy: _______________ _____________________________________________________________________________________ Indicador Respuesta Puntos Valor -----------------------------------------------------------------------------------------------------------------------------------------1. ¿Cuántas personas viven en el hogar? A. Ocho o más 0 B. Siete 5 C. Seis 9 D. Cinco 13 E. Cuatro 19 F. Tres 27 G. Uno o dos 41 -----------------------------------------------------------------------------------------------------------------------------------------2. En el presente año escolar, ¿se matricularon en el A. No hay miembros 7 a 18 0 sistema de educación formal todos miembros B. No 2 del hogar de las edades 7 a 18? C. Si 6 -----------------------------------------------------------------------------------------------------------------------------------------3. En su ocupación principal en los últimos siete días, ¿cuántos A. Ninguno 0 miembros del hogar trabajaron como empleados/obreros? B. Uno 3 C. Dos o más 7 -----------------------------------------------------------------------------------------------------------------------------------------4. ¿De cuántos cuartos dispone el hogar (no incluya cocina, baños, A. Uno 0 pasillos ni garaje)? B. Dos 2 C. Tres o más 5 -----------------------------------------------------------------------------------------------------------------------------------------5. ¿Qué material predomina en A. Tierra, u otro 0 el piso de la vivienda? B. Madera (tambo), ladrillo de barro, o 5 embaldosado o concreto 26 C. Ladrillo de cemento, mosaico, terrazo o cerámica 10 -----------------------------------------------------------------------------------------------------------------------------------------6. ¿Qué combustible utilizan A. Leña no comprada 0 usualmente para B. Leña comprada, carbón, o no cocinan 5 cocinar? C. Gas butano o propano, gas kerosén, electricidad u otro 10 -----------------------------------------------------------------------------------------------------------------------------------------7. ¿Tiene este hogar una plancha? A. No 0 B. Si 2 -----------------------------------------------------------------------------------------------------------------------------------------8. ¿Tiene este hogar una licuadora? A. No 0 B. Si 4 -----------------------------------------------------------------------------------------------------------------------------------------9. ¿Con cuántos teléfonos celulares cuenta el hogar? A. Ninguno 0 B. Uno 3 C. Dos o más 8 -----------------------------------------------------------------------------------------------------------------------------------------10. ¿Tiene este hogar una bicicleta, bote, caballo, burro, mulo, A. No 0 motocicleta, o vehículo? B. Si 6 -----------------------------------------------------------------------------------------------------------------------------------------Puntaje: Índice comunitario - Fabretto -----------------------------------------------------------------------------------------------------------------------------------------11. ¿Cuántos miembros del hogar son adultos? A. Ocho o más B. Siete C. Seis D. Cinco E. Cuatro F. Tres G. Uno o dos -----------------------------------------------------------------------------------------------------------------------------------------12. ¿Cuántos adultos del hogar trabajan? A. Ocho o más B. Siete C. Seis D. Cinco E. Cuatro F. Tres G. Uno o dos -----------------------------------------------------------------------------------------------------------------------------------------13. ¿Cuáles son los ingresos quincenales de los miembros quienes trabajan? 27 Nombre (1) (2) (3) (4) (5) (6) (7) (8) ______________________________ ______________________________ ______________________________ ______________________________ ______________________________ ______________________________ ______________________________ ______________________________ Edad Ganancias Quincenales ________ ________ ________ ________ ________ ________ ________ ________ ________________________ ________________________ ________________________ ________________________ ________________________ ________________________ ________________________ ________________________ -----------------------------------------------------------------------------------------------------------------------------------------14. ¿Cuál es el estado propietario del hogar? A. Rentada B. Familiar C. Propia -----------------------------------------------------------------------------------------------------------------------------------------15. Si la casa es rentada, ¿cuánto cuesta mensual? -----------------------------------------------------------------------------------------------------------------------------------------16. ¿Cuál es la fuente de agua en el hogar? A. Tubería B. Pozo propio C. Pozo comunitario Preparation and Implementation Guide: Pautas para la efectiva realización de la entrevista Según las página 10–13 del Manual, “En la realización de la entrevista, se debe tener en cuenta algunos aspectos que le pueden ayudar a conducir con éxito la misma. Presentación “Se requiere de usted como encuestador(a) mucha atención y cuidado al momento de la entrevista. Dado que usted y el o la informante no se conocen, se debe demostrar simpatía, sensibilidad e interés por el (la) informante. “En el contacto inicial entre el encuestador(a) y el (la) informante, es conveniente que el (la) encuestador(a) se presente amablemente, indicando la institución que representa, el objetivo de la encuesta [‘aprender más en cuanto a como viven los participantes de la organización’], y solicitando la disposición del o la informante para brindar la información requerida. Una introducción efectiva puede ser lo siguiente. ‘Buenos días. Soy <nombre>, encuestador/a de [su organización] que está llevando a cabo una encuesta de [todos/una muestra] de sus participantes con la finalidad de recabar información que permita conocer sus condiciones de vida. . . . Entonces, quisiera hacerle algunas preguntas y espero que usted tenga la bondad de cooperar conmigo.’ “No debe presentarse realizando preguntas como ‘¿Está usted muy ocupado?’, ‘¿Puede concederme unos minutos?’, o ‘No sé si será posible que conteste algunas preguntas?”, ya que este tipo de preguntas sugieren respuestas negativas y es conveniente que inicie de forma positiva. Confidencialidad de las respuestas: “Antes de realizar la primer pregunta, el o la encuestadora, debe dar a conocer al (la) informante el carácter confidencial de la información, explicando que en ningún 28 momento se publicarán nombres de personas y que la información será utilizada solamente para fines estadísticos. Ambiente de la entrevista: “Sería preferible que el entrevistado escoja el lugar de la entrevista para que esto facilite la comunicación. Este lugar debe permitir que la encuesta cumpla con el carácter de confidencialidad. Carácter privado de la entrevista: “Es muy importante que la entrevista se realice en privado. Evite en lo posible realizar la entrevista en presencia de personas que no pertenecen al hogar, ya que el o la informante podría alterar la respuesta u omitirla. De no ser posible, explique la necesidad de la privacidad de la entrevista y pida cortésmente al o (la) acompañante, que los deje solos. Neutralidad: “Acepte las respuestas del o la informante, y evite la crítica, opiniones personales, juicios, aprobación o desacuerdo con sus palabras o gestos. Recuerde que no debe imponer al entrevistado(a) sus opiniones, ni sugerir respuestas. Control de la entrevista: “El o la encuestadora es quien dirige la entrevista y por lo tanto es quien debe controlar la situación. Si el o (la) entrevistada da respuesta de temas ajenos o habla de asuntos que no tienen que ver con la entrevista, no es necesario que se le interrumpa; pero en la primera oportunidad—y con mucho tacto—realice de nuevo la pregunta y regrese al tema de la entrevista. No asumir respuestas: “Las características socio-económicas y sociológicas de los entrevistados, como área de residencia o condiciones de la vivienda, no deben llevar al o (la) encuestador(a) a asumir respuestas o formarse expectativas anticipadas. “Así mismo, no debe sugerir respuesta en razón del ‘aparente’ nivel cultural del entrevistado. Si el formulario lo señala, deben realizarse preguntas de sondeo. No apresurar la entrevista: “Las preguntas deben ser realizadas de forma tal que el o la informante comprenda lo que se le está preguntando. Una vez hecha la pregunta debe dársele el tiempo necesario para pensar, de lo contrario puede que le responda con evasivas. Tratamiento con personas indecisas: “En muchas ocasiones el (la) entrevistado(a) responderá ‘No sé’, o dará una respuesta con evasivas o simplemente rehusará a contestar más preguntas. En estos casos, el o (la) encuestador(a) tratará de darle más confianza y hacerle sentir más cómodo antes de continuar con la siguiente pregunta. Le explicará la importancia de sus respuestas. Tratamiento con personas renuentes: “En ocasiones el (la) entrevistado(a) se negará a brindar la información desde un inicio, o simplemente se rehusará a seguir contestando más preguntas. En estos casos, el o (la) encuestador(a) intentará de forma cortés tratar de convencer al informante sobre la importancia del estudio y la necesidad de que brinde la información. Fin de la entrevista: “Una vez finalizada la entrevista y antes de despedirse del o la entrevistada, revise el formulario por si ha omitido alguna pregunta o si quedó incompleta alguna respuesta. Agradezca la colaboración prestada, y despídase.” 29 Indicator 8: Tripod Student Surveys Brief Description of Instrument: Tripod Surveys offer student-sourced metrics for assessing teacher quality. These instruments acknowledge that students have large stakes in teaching effectiveness and are perhaps the best positioned to comment on their teachers’ performance. Tripod offers these surveys free-of- charge and in Spanish, but only in digital format. The use of student-sourced metrics is becoming increasingly popular in education, and has earned endorsement by the Bill and Melinda Gates Foundation. This instrument measures results directly aligned with two of six outcomes outlined in Fabretto’s strategic document. Primary Questions to Consider for Next Steps: 1. Does Fabretto have the technical capabilities/infrastructure to implement digital surveys in classrooms? 2. Will students and teachers have the necessary experience to complete and analyze digital surveys? 3. If Fabretto does not have the technical capabilities to implement digital surveys, is it willing to invest significant time and resources to translate the surveys into written format? 4. Once translated to written format, are the surveys easily analyzed? Will Fabretto invest time and resources to manually analyzing survey results? 5. Is Fabretto interested in utilizing student-sourced metrics? (1) Preparation - PennGSE researchers and/or Fabretto may request Tripod Student Surveys in digital format3 - If necessary, PennGSE or Fabretto staff may translate digital surveys into written format. - Identify schools, teachers, and classes of students to participate in pilot study - Prepare necessary materials for completion of surveys (2) Implementation - Pilot implementation of student surveys in select schools and classes (3) Analysis - Aggregate results within each class and assess general feedback on teacher quality - Draft recommendations to improve teaching practice (4) Dissemination of Results - Draft report on results and analysis for dissemination - Inform teachers and school heads of survey results and recommendations - Coordinate meeting of teachers and school heads to review survey results and consider future steps (5) Decision Making - Based on survey results, emphasize, add to, or implement existing teacher training modules. Alternatively, develop new modules to fill particular gaps in teaching practice 3 Official text form "Request to conduct research using tripod instruments or data" can be provided upon request. 30 - Invite teachers to participate in new or re-designed PD modules Evaluate effectiveness of new PD modules Propose permanent changes to teacher training program in consultation with program coordinators Figure 1.2: Tripod Survey Sample Questions4 CARE My teacher in this class makes me feel that s/he really cares about me. → Agree Strongly → Agree → Somewhat Agree → Neutral → Somewhat Disagree → Disagree → Strongly Disagree My teacher really tries to understand how students feel about things. CONTROL Students in this class treat the teacher with respect. Our class stays busy and doesn’t waste time. CLARIFY My teacher has several good ways to explain each topic that we cover in this class. My teacher explains difficult things clearly. CHALLENGE In this class, we learn a lot almost every day. In this class, we learn to correct our mistakes. CAPTIVATE My teacher makes lessons interesting. I like the ways we learn in this class. CONFER Students speak up and share their ideas about class work. My teacher respects my ideas and suggestions. CONSOLIDATE My teacher checks to make sure we understand what s/he is teaching us. The comments that I get on my work in this class help me understand how to improve. 4 Source: Bill & Melinda Gates Foundation 31 Authors' Biographies Matthew J. Tarditi, M.S.Ed., is a doctoral candidate in the Teaching, Learning, and Teacher Education program at the University of Pennsylvania’s Graduate School of Education (PennGSE). Matthew’s research and professional work resides at the intersection of international educational development, school-community partnership, and the integration of film and multimedia to foster participatory and collaborative approaches to research, knowledge construction and overall practice. Tarditi is trained as a qualitative researcher steeped in ethnography and visual anthropology with several years experience as a teacher, coach and teacher-trainer. Currently, his applied research involves two non-profit organizations in Nicaragua: the Seeds for Progress Foundation and the Fabretto Children's Foundation. Tarditi is the co-founder and former director of camra, a student-lead, faculty-supported multimedia pedagogy lab at the University of Pennsylvania. Matthew received his master’s degree from the Education, Culture, and Society program at PennGSE. His master’s work included ethnographic and qualitative methods, linguistics, technology in education, teacher professional development in technology, and international teacher education. Contact him at mtarditi@gse.upenn.edu or @mjtarditi Will Slotznick is a sophomore at the University of Pennsylvania, studying International Relations and Development. Will is passionate about international education, and has spent time in Ghana and Nicaragua working on several rural education initiatives. At Penn, Will conducts research relating to educational M&E, program assessment, and teacher training for both international and local groups. He is actively involved in the Penn Graduate School of Education, Wharton Social Impact Initiative, and Penn Society for International Development. Will serves as the founding director of Global Impact Collaborative (PennGIC) and advises on projects relating to M&E and program evaluation. Contact him at wslotz@sas.upenn.edu Any Summer Amarsingh Gawande is a freshman at the University of Pennsylvania studying Physics and Economics. Amar is interested in international development and global education. In the past six years, he has worked with NGOs and educational centers in India and rural Nepal. At Penn, Amar has been involved with GIC for the past year, primarily conducting research relating to educational M&E. Edgar Agudelo Conner Evans 32 Global Impact Collaborative (PennGIC) is a not-for-profit, student-run action-research team at the University of Pennsylvania. PennGIC's mission is to leverage academic resources to help international organizations achieve tangible social impact. As a sponsored branch of the Penn Society for International Development (PennSID), GIC provides students with an opportunity to work closely with non-profit leaders and perform team-based and independent research related to international development. References Orazem, P., Glewwe, P., & Patrinos, H. (2009). The Benefits and Costs of Lowering Parental Schooling Costs to Improve Educational Outcomes. Frederiksberg, Denmark: Copenhagen Consensus Center. Psacharopoulos, G. & Patrinos, H. (2002). Returns to Investment in Education: A Further Update. World Bank elibrary. Retrieved from http://elibrary.worldbank.org/doi/pdf/10.1596/1813-9450-2881. 33