M&E Plan Overview M&E Capacity Strengthening Workshop Addis Ababa 4 to 8 June 2012 Arif Rashid, TOPS What is an M&E Plan? An M&E plan is: • A flexible guide to the steps needed to: – Inform and guide monitoring and evaluation activities; – Document, report, and communicate project activities; – and illustrate progress toward achievement of project goals and objectives. Slide # 1 What is an M&E Plan? Slide # 2 What are the components of the M&E Plan? • M&E goals and objectives • M&E methodologies & implementation plan – Data collection procedures – Data management and analysis procedures – Reporting and communications plan • Description of M&E activities – Output monitoring – Process monitoring – Outcome monitoring – Monitoring of environmental indicators – Commodity end-use monitoring Slide # 3 What are the components of the M&E Plan? – Calendar of M&E activities – Data quality assurance plan – Data collection instruments – Organizational structure of M&E unit – Communication strategy and use of data in decision making – Budget for M&E activities Slide # 4 M&E Plan components Focus of this workshop M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT PMP - Analysis - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 5 Unpacking the M&E Plan: Key elements Slide # 6 M&E Plan components M&E Methods/tools PIRS /M&E Matrix A M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT PMP - Analysis - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 7 A. Results framework (RF) “A results framework …[is a]…type of logic model which represents the food aid program’s theory of change by laying out the activities and outputs that will lead to short, medium and long term outcomes and objectives” Source: http://www.usaid.gov/our_work/humanitarian_assistance/ffp/ffpib_09.06.pdf Slide # 8 RF example GOAL SO 1 SO2 SO3 IR1.1 IR1.2 IR2.1 IR2.2 IR3.1 IR3.2 Activities Activities Activities Activities Activities Activities Slide # 9 M&E Plan components M&E Methods/tools B PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT PMP - Analysis - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 10 B. IPTT “The indicator performance tracking table (IPTT) …includes performance indicators (at the impact, outcome and output levels) linked to the food aid program proposal’s objectives, and baseline and target values for each indicator (estimated)” Source: http://www.usaid.gov/our_work/humanitarian_assistance/ffp/ffpib_09.06.pdf Slide # 11 FY2012 IPTT Template Slide # 12 M&E Plan components M&E Methods/tools C PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT PMP - Analysis - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 13 C. M&E Matrix / PIRS/PMP A performance management plan (PMP) includes: “definitions for each indicator listed in the IPTT, the data collection methodology and frequency” Source: http://www.usaid.gov/our_work/humanitarian_assistance/ffp/ffpib_09.06.pdf Slide # 14 C. M&E Matrix / PIRS/PMP An M&E matrix (also known as an Indicator Matrix/ or Performance Indicator Reference Sheet): • summarizes of detailed information for each project indicator. • serves as a helpful tool for planning and managing data collection, analysis and use. • expands on the IPTT/PMP to identify key information required for each indicator and summarize the key M&E tasks for the project. Adapted from: Chaplowe, Scott G. (2008) Slide # 15 M&E Plan components M&E Methods/tools C PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 16 D. Key M&E activities 1. Monitoring • Output monitoring • Process monitoring (quality monitoring) • Outcome monitoring • Environmental indicator monitoring • Commodity end-use monitoring 2. Evaluation 3. Reporting and communications 4. Review of M&E process Slide # 17 Output monitoring • Reporting and tracking project outputs to compare with targets and resource used • Information Source • Project reporting • M&E plan needs to develop: • Reporting forms • Data base(s) Slide # 18 Process monitoring • Monitoring the “quality” of services delivered to beneficiaries • Information Source • Surveys of “final service providers” • M&E plan needs to develop: • Sample design • Instruments • Analysis plan Slide # 19 Outcome monitoring • Monitoring outcome-level changes of project beneficiaries and institutions • To what extent have project interventions altered • Knowledge • Perceptions, and • Behaviors of beneficiaries Slide # 20 Outcome monitoring • To what extent have project interventions altered • Institutional competency • Policy change or • Quality of service changes. • Information source • Surveys of beneficiaries • M&E plan needs to develop for outcome monitoring: • Sample design • Questionnaire • Analysis plan Slide # 21 Environmental indicator monitoring • Monitoring the indicators identified in the IEE as negative determination with conditions • Information Source • Based on the proposed mitigating measures in IEE • M&E plan needs to develop: • List of indicators and mitigating measures • Instruments • Analysis plan Slide # 22 Commodity end-use monitoring • Monitoring end-use of the commodities distributed by the project • Information Source • Sample survey of beneficiaries that received commodities from the project • M&E plan needs to develop: • Sample design • Instruments • Analysis plan Slide # 23 Evaluation activities Evaluating project impacts should be taken into consideration from the inception of the project. Project evaluation is composed of three separate events that should be considered parts of a whole. [Note: Food for Peace does not require a quantitative survey during the mid term review] Baseline Mid-term Final Slide # 24 USAID Evaluation policy “In most cases, the evaluations will be external (i.e., a third-party contractor or grantee, managed directly by USAID, will evaluate the projects), and the contract or grant for the evaluation will be managed by an operating unit’s Program Office.” Please note that: The Title II Project evaluations are currently managed by the PVOs not by FFP/USAID which will continue until FFP decides otherwise. Slide # 25 Evaluation activities M&E units are responsible for organizing evaluation activities: Steps: 1. Identify study objectives/research questions, target population(s), coverage and reach of study, timeline and budget 2. Identify specific indicators to be measured 3. Compile sample frame (from appropriate population) 4. Using information above, develop SOW for external consultants 5. Organize logistics for evaluation studies to support external consultants Slide # 26 Review of M&E plan M&E plan should be regularly reviewed to assess whether the current plan is meeting objectives and needs of project: • Reporting needs to donor • Project management needs Review should assess: • Indicators • Data collection and analysis procedures • Communication strategy Slide # 27 M&E Plan components M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys E M&E Timeline Communication strategy - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 28 E. M&E Tools and methods M&E tools and methods include: • Project monitoring reporting forms • Database formats and management procedures • Survey sampling strategy and analysis plans • Reporting processes and procedures Slide # 29 M&E Plan components M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline F Communication strategy - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 30 F. M&E timeline An M&E timeline is “a schedule of M&E events, describing procedures and timing for analysis and use of performance data in order to modify activities based on lessons learned” Source: http://www.usaid.gov/our_work/humanitarian_assistance/ffp/ffpib_09.06.pdf Slide # 31 M&E Timeline example Year 1 Year 2 Year 3 Year 4 Year 5 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 General activities M&E Staff hiring Equipment procurement Baseline survey Baseline results reported and disseminated Mid-term evaluation MTE results reported and disseminated Final evaluation Final evaluation results reported and disseminated SO 1 activities Project monitoring/reporting SO 2 activities Project monitoring/reporting SO 3 activities Project monitoring/reporting Slide # 32 M&E Plan components M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy G - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 33 G. Communication Strategy An important (and often overlooked) component of M&E plans is how to disseminate and use the information obtained by the M&E system Although the M&E Officer or Unit does not have much control over the managers or the technicians to ensure that M&E data inform the programmatic decisions, M&E unit can facilitate the process. M&E Slide # 34 G. Communication strategy Careful thought needs to be given to disseminating and using M&E information including: •What information should be distributed? •Who needs what information? •Why do they want this information? •How much information they want to know? •How does the information get distributed? Source: Family Health International (2004) Slide # 35 M&E Plan components M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing H - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality DIP Slide # 36 H. M&E Staffing plan Based on information needs outlined in earlier steps, the following capacities may be required for an M&E unit (Pls refer to core competencies for detailed skills): •M&E plan design •Data collection methods (quantitative & qualitative) •Data management skills •Data analysis skills •Data reporting •Project monitoring Slide # 37 Sample M&E organizational chart M&E Manager (Extensive knowledge of: M&E plan design, data collection methods, analysis and reporting) M&E Officer M&E Officer (Project monitoring & reporting, data collection, supervising enumerators, etc.) (Project monitoring & reporting, data collection, supervising enumerators, etc.) M&E Assistant/ Data entry Clerk Slide # 38 M&E Plan components M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan I Data Quality DIP Slide # 39 I. M&E Budget • One challenge in developing M&E plans for Development Food Aid Projects is that the M&E budget is normally established before the detailed M&E plan is finalized • As a result, the M&E budget will need to be revisited after the M&E plan is developed • Budget must be aligned with activities in M&E plan Slide # 40 M&E Plan components M&E Methods/tools PIRS /M&E Matrix M&E Activities - Monitoring - Surveys M&E Timeline Communication strategy - Secondary data IPTT - Analysis PMP - Reporting M&E Staffing - Dissemination Results Framework - Review M&E Plan M&E Budget General implementation plan Data Quality J DIP Slide # 41 J. Data Quality: Assurance and Assessment About the Network It willQuality also serve as a place -for • Data Assurance A formalized organizations and institutions to process for defining the appropriate post relevant meeting, training or dimensions and criteria of data quality, and other event notices along with associated links to similar sites procedures to ensure that data quality criteria and resources. are met over time • Data Quality Assessment –Review of project M&E system to ensure that quality of data captured by the M&E system is acceptable. Slide # 42 What’s a Data Quality Assurance Plan? About the Network • A data quality assurance plan includes It will also serve as a place for activities to and improve data toquality: organizations institutions relevant meeting, training or tabulation plan and A.post Clear definition of indicators including other event notices along with measurement protocols toofsimilar sites aggregation and analysis B.associated Appropriatelinks design data collection, and resources. procedures C. Designing data base with appropriate checks and bounds to reduce data entry error D. Well designed data management procedures E. Follow-up verification of indicator values F. Allocate adequate Resources and Skilled Staff G. Periodically Assess Data Quality using established protocols Slide # 43 What’s a Data Quality Assessment? About the Network • data quality ItAwill also serveassessment as a place foris a periodic review organizations and institutions to that: post relevant meeting, training or –other Helps thenotices project to determine event along with and document “How good are links the data?” associated to similar sites and resources. – Provides an opportunity for capacity-building of implementing partners. Slide # 44 We have now identified all the components of an M&E plan M&E Methods/tools PIRS/ M&E Matrix M&E Activities - Monitoring - Surveys Results Framework IPTT M&E Timeline Communication strategy - Secondary data A PMP - Analysis - Reporting M&E Staffing - Dissemination - Review M&E Plan M&E Budget Data Quality All these components should be in the M&E plan document Slide # 45 Outline for M&E plan 1. Objectives of M&E system 2. Description of – – – – Results framework IPTT PMP/M&E Matrix/PIRS Project staffing and organizational structure Slide # 46 Outline for M&E plan 3. Description of Monitoring Activities – Description of processes, tools and frequency 4. Evaluation – Description of processes and methodology 5. Reporting and communications strategy 6. Data Quality Assurance Plan 7. M&E review process 8. M&E staffing and organization 9. M&E budget 10.Use of information in Decision Making Slide # 47 Outline for M&E plan • Annexes – – – – – – – – Results framework IPTT M&E matrix/PIRS/PMP Reporting formats for output monitoring Questionnaire forms for process/outcome monitoring Timeline for M&E activities M&E unit organizational chart M&E budget Slide # 48 The remainder of the workshop will provide guidance on how to develop these components of the M&E plan Slide # 49 This presentation was made possible by the generous support of the American people through the Office of Food for Peace in United States Agency for International Development (USAID). The contents are the responsibility of Save the Children and do not necessarily reflect the views of USAID or the United States Government.