Somalia Return Consortium Monitoring & Evaluation System Manual Version 7 January 2013 TABLE OF CONTENTS Figures ................................................................................................................................................................... 3 Acronyms .............................................................................................................................................................. 3 1. Objective of the Manual ......................................................................................................................... 5 2. Background ............................................................................................................................................. 5 2.1 Rationale............................................................................................................................................ 5 2.2 Target Audience................................................................................................................................ 5 2.3 Document Structure ......................................................................................................................... 5 3. Data Collection........................................................................................................................................ 6 3.1 Timeline of Data Collection .............................................................................................................. 7 3.2 Sources of Data ................................................................................................................................ 7 3.3 Data Collection Methodologies ....................................................................................................... 8 3.3.1 Intention Survey................................................................................................................. 8 3.3.2 Beneficiary Registration Form ........................................................................................... 9 3.3.3 Distribution List and Cash Transfer Records ..................................................................... 9 3.3.4 Post Return Monitoring ...................................................................................................... 9 3.3.5 Post Distribution Monitoring (by third party)..................................................................... 11 3.3.6 Post Return Assessment (by third party) ......................................................................... 13 3.3.7 Feedback and Complaint Mechanism ............................................................................. 16 3.4 Training............................................................................................................................................ 16 3.5 Data Verification ............................................................................................................................. 17 4. Data Entry .............................................................................................................................................. 18 4.1 Interagency Database..................................................................................................................... 19 4.2 Steps for Data Entry ....................................................................................................................... 19 4.3 Data Verification ............................................................................................................................. 22 5. Data Analysis ........................................................................................................................................ 22 6. Data Sharing & Protection ................................................................................................................... 25 7. FAQs ...................................................................................................................................................... 25 8. Annexes ................................................................................................................................................. 26 2 Tables Table 1: Specific Objectives & Corresponding Indicators ....................................................................................... 6 Table 2: Tools and Sources of Data........................................................................................................................ 7 Table 3: Post Return Monitoring Frequency Matrix ............................................................................................... 10 Table 4: Sample Size Examples ........................................................................................................................... 12 Table 5: Sample Size Examples ........................................................................................................................... 14 Table 6: Reliability/Credibility Matrix ..................................................................................................................... 17 Figures Figure 1: Data Collection Timeline .......................................................................................................................... 7 Figure 2: Database Dashboard ............................................................................................................................. 20 Figure 3: Login Interface ....................................................................................................................................... 20 Figure 4: Add New Record .................................................................................................................................... 21 Figure 5: View/Edit Record ................................................................................................................................... 21 Figure 6: Upload Multiple Records ........................................................................................................................ 22 Figure 7: Static Maps ............................................................................................................................................ 23 Figure 8: Webmap................................................................................................................................................. 24 3 Acronyms DRR GSV FGD HHs IDP MoU M&E PDM PRA RC SOPs VoO Disaster Risk Reduction Go and See Visit Focus Group Discussion Households Internally Displaced Person Memorandum of Understanding Monitoring and Evaluation Post Distribution Monitoring Post Returns Assessment Return Consortium Standard Operating Procedures Village of Origin 4 OBJECTIVE OF THE MANUAL The Somalia Return Consortium Monitoring and Evaluation System Manual is produced in consultation with all members of the Return Consortium (RC) for the purpose of standardizing indicators as well as data collection, sharing and analysis procedures. The manual is a key document in the larger body of RC documents intended to ensure a common voluntary return methodology to leverage synergies and individual agency expertise for more effective programming and fundraising as well as collective transparency and accountability. This manual, in a very practical way, expresses the RC’s commitment to delivering quality services and building accountability at both the individual and institutional level. It builds upon existing work by RC members and is intended for use as a reference document. As the RC becomes more established and its methodology is further refined, elements within this manual may need to be changed accordingly. BACKGROUND 2.1 Rationale The RC has established a Memorandum of Understanding (MoU) stating that, “The Consortium Members have come together for the purpose of standard-setting, creating operational synergies, fundraising and advocating for, and implementing, voluntary return programming with the aim of ensuring voluntariness of return, safety, dignity and sustainable reintegration for internally displaced person (IDP) households in South and Central Somalia in accordance with agreed SOPs.” In addition, the RC MoU states in article 4.2 that, “Specific guidelines for Monitoring and Evaluation (M&E) of the Programme will be developed and endorsed by the Steering group within three months from the signature date of the MoU, in order to standardize M&E objectives, procedures, reporting format and dissemination of data.” This manual is a tool for the consortium to develop a more systematic approach to the collection, management, and reporting of key data in order to inform and improve operational and strategic decision-making and allow for the evaluation of the outputs and outcomes of its programming. The M&E system framework is based on the IASC Framework on Durable Solutions for Internally Displaced Persons, establishing the context for which indicators will measure the: (1) extent returnee households are enabled to make an informed choice concerning return; (2) extent returnee households are able to meet basic human needs and access social services; and, (3) resilience and capacity of returnee households and receiving communities to mitigate shocks. The system is intended to not only measure impact of RC activities across all indicators, but also can be used as a project management tool to track progress and gaps. The IASC Framework represents best practices on IDP returns and is thus used a backbone to the design of the monitoring and evaluation (M&E) system. 2.2 Target Audience This manual is targeted toward both remote managers and field staff involved in the standardized returns methodology employed by the RC. While the manual seeks to present material in a common language that all readers can follow, it is recommended that the manual be used by staff well-versed in both setting-up and implementing M&E systems. The manual can also be used to train staff to collect data and contribute to the use of the M&E system. 2.3 Document Structure The manual is structured as a practical reference tool to support practitioners in the implementation of the M&E framework approved by the members of the consortium. With this in mind, each element of the framework is outlined and operationalized throughout the four stages of the system: (1) data collection (2) data entry; (3) data 5 sharing and protection; and, (4) data analysis. Each section will include a review of the methodologies and tools used in line with the standard operating procedures (SOPs) established by the RC. DATA COLLECTION Each member of the RC will use the M&E Framework found in Annex I as the basis for measuring standard indicators at the output and outcome levels. These indicators must be measured by each sending organization in order to provide the most comprehensive data necessary for full RC analysis. Each outcome indicator directly relates to the three specific objectives of the RC Standard Operating Procedures (SOPs). Table 1 outlines the standard objectives and indicators for the RC. Each sending organization will be responsible for using the tools defined in Section 3.2 according to the timeline in Section 3.1 to collect the data for the indicators below. Sending organizations will then enter this data into the interagency online database, as specified in Section 4. Table 1: Specific Objectives & Corresponding Indicators Output Indicator (at result level) % of HHs surveyed with Intention Survey # of HHs’ intent to return confirmed via Intention Survey # of IDP delegates involved in GSV (disaggregated by age, gender) % of target communities supported with GSV % of returnee households informed of what items they can take, available transport and arrangements for those with special needs1 PROXY INDICATORS: # of returnee HHs assisted with safe, successful transport2 to VoO % of returnee HHs reporting critical incidents3 during the transport # of returnee and receiving HHs assisted with RC Standard Minimum Package # of litres (average) of potable water available per person per day per returnee HHs % returnee-assisted households that have access to emergency shelter as per Shelter/NFI Cluster Somalia guidelines4 % of returnee HHs with an average food consumption of 2,100 Kcal5 per person per day % of returnee children enrolled in primary school % of returnee HHs benefitting from free health care service in the VoO or within a certain distance # of returnee HHs earning incentives / wages PROTECTION RELATED INDICATORS: # of reported acts of violence or intimidation targeting returnees on the basis of their refugee/IDP or minority status in the last 3 months % of assisted HHs involved in land disputes having access to formal justice system/ community reconciliation mechanisms # of assisted HHs involved in land/property disputes % of Vulnerable6 returnee HHs monitored by sending organizations on regular basis # of family reunification mechanisms in place and accessible in the area of return # assisted communities where disaster risk reduction (DRR) systems are in place # of emergency interventions carried out to respond to shocks # of feedbacks and complaints acquired by sending organizations through the F&C mechanism % of complaints addressed by sending organizations with response provided to the complainer Outcome indicators (at sp. Objective level) Specific Objectives of RC Extent returnee households are enabled to make an informed choice concerning return To facilitate Informed and Voluntary Decision of 20,000 HHs from displaced communities about voluntary return in South Central Somalia. 1 Extent returnee households are able to meet basic human needs and access social services To support a Sustainable and Durable Reintegration of 20,000 returnee HHs in their villages of origin. 2 Resilience and capacity of returnee households and receiving communities to mitigate shocks To adapt the Return Programme through M&E inputs in order to mitigate shocks affecting the returnee and receiving communities. 3 Minimum information that should be conveyed, based on IASC guidelines. “Safe, successful transport” defined as returnees reaching intended destination without injury or death. 3 “Critical incidents” defined as: (1) acts based on IDP status; (2) SGBV acts; (3) inter-communal conflict; (4) armed combatant act. Based on IASC guidelines. 4 This may refer to emergency shelter kit distribution, shelter voucher distribution or transfer of equivalent amount of cash, based on IASC guidelines. 5 2,100 Kcal per person per day to be linked with Post-Distribution Form 6 “Vulnerable” as defined by registration form and Intention Survey 1 2 6 PROXY INDICATORS: # of returnee HHs forcibly displaced from the VoO to other location % of returnee HHs assisted residing in the VoO for more than 3/6/12 months 3.1 Timeline of Data Collection RC members have agreed to the procedures established in the RC SOPs, setting up a pre-determined data collection process that will take place in-line with key steps within the SOPs. Figure 1 outlines when specific tools will be used according to the seven steps laid out in the SOPs. The tools or sources along the bottom of the timeline correspond to each step before it. The monitoring processes are split into activities that will be undertaken by RC members and those that will be conducted by external parties. Each of these sources is explained in more detail in Section 3.2. Figure 1: Data Collection Timeline Legend RC Monitoring External/Third Party Monitoring 3.2 Sources of Data The RC M&E System uses pre-defined tools and methodologies for primary and secondary data collection. Each source defined in Table 2 below directly links with the indicators in the RC M&E framework, providing the necessary data to measure their progress. All relevant tools are located in the annexes designated in Table 2 and the methodologies for their use can be found in Section 3.3. Table 2: Tools and Sources of Data Tool/Source Type Intention Survey (Annex 2) Objective Type Target Used By Internal Data Sources (directly from sending organizations) Clarify how return to survey household sending the place of origin, level orgs local integration and settlement elsewhere Frequency each round of returns Data Entry Mechanism online 7 Beneficiary Registration Form (Annex 3) Distribution Lists (for in-kind provision and voucher) Cash Transfer Records Post Return Monitoring (FGD, HH interview with vulnerable families) (Annex 4) Post Distribution Monitoring by third party (for in-kind provision, voucher and cash) (Annex 5) Post Return Assessment by third party (Annex 6) Satellite imagery Feedback and complaint mechanism (according to RC SOP) in the country are both durable solutions available for the IDPs Summarize key characteristics of registered households in returning process Record beneficiary reception of NFIs and other goods To verify reception of cash by beneficiaries Gather information on sustainability after refugees have returned Verify quality of distributions, impact and diversion rate. Improve future programming. Assess the process for returnees’ transport and integration into Villages of Origin Verify data already collected and provide additional authentication source Ensure high quality program implementation and program adaptation form household level sending orgs each round of returns online list household level sending orgs at site list household level household level; village level sending orgs sending orgs each distribution/ intervention each cash transfer once per month for 6 months third parties after each distribution offline then uploaded online forms External Data Sources survey household level upon transaction offline then uploaded online survey household level third parties month 2, 4 and 6 following return online photograph, GPS coordinates village and region levels sending orgs, third parties ongoing online SMS, phone calls, emails, individual level sending orgs ongoing online1 1 Initially, this will be downloaded from the F&C online system and uploaded to the interagency database through Excel spreadsheets. Eventually, the two systems will be linked to automatically transfer data from the F&C system to the interagency database. 3.3 Data Collection Methodologies The following section outlines the specific methodologies that are to be used for each tool highlighted in Table 2. The methodologies for the tools located in the Internal Data Sources section can also be found in the RC SOPs. 3.3.1 Intention Survey The stated purpose of the intention survey is as follows: To verify the genuine intention of the targeted community. To know the real intention of the community about voluntary return. To know and advocate for durable solutions alternative to return, if requested by part of the targeted community. The intention survey is designed to be administered for all potential IDPs wishing to return to their VoO. All sending organizations are required to implement the intention survey using a sample size of all potential IDP returnees. The standard intention survey form (Annex 2) is used and interviews are held at the household level with one individual from the household. 8 This is linked to Step 2 of the RC SOPs 3.3.2 Beneficiary Registration Form The stated purpose of the registration form is as follows: To have picture, age and gender data about the returning households. To acquire written consent by the head of household and the other family members (above 18 only). To make sure about the spontaneous and genuine character of the return, double-checking the list from the intention survey with the actual registration figures. To identify persons (adults and children) with special needs in order to plan a customized assistance package. To identify vulnerable families in the villages of origin. To provide additional information about the rights IDPs to durable solutions and related implications. The beneficiary registration form is designed to officially register all households intending to return to their VoO following their own consultation with leaders and reports from the GSV. Each family that intends to return must register with a sending organization using the consolidated family registration form (Annex 3). Thus, the sample size for this tool is all households intending to return. Pre-registration arrangements and a logistical setting that allows for reduced registration exercise time are recommended. Registration is a sensitive step of the return process where pressure from the community, security risks, risk of fraud and/or request to enlarge the target are only some of the challenges. Reduce time and improve efficiency using experienced and skilled staff (including staff from the areas of return who may support in the vetting process). This is linked to Step 5 of the RC SOPs 3.3.3 Distribution List and Cash Transfer Records There are not any standardized formats for distribution lists or cash transfer records; these forms are designed using each sending organization’s own templates. The purpose of these records is to compare what is reported by each sending organization as being distributed and what each IDP household reports they received during the third party Post Distribution Monitoring. These forms should, at a minimum, include: (1) the data of distribution; (2) the distributing agency; (3) the family code and name of the recipient, age and the size of the household; (4) the number/amount of items distributed; and, (5) the mode of distribution (i.e. in kind, cash or voucher). These forms are linked to Step 6 of the RC SOPs 3.3.4 Post Return Monitoring The stated purpose of post return monitoring is to provide information about to the re-integration of returnees, the sustainability of the return process and protection risks returnees are facing, with a particular focus on vulnerable households. This will result in referral, support with family reunification of cases of concern by the monitors and should lead to the identification of priority issues for advocacy purposes. The results of the monitoring will also inform planning and implementation of future returnee programs. The methodology currently will be carried out once per month for six months for vulnerable households in the VoOs where IDPs have been returned with Focus Group Discussions (FGDs) and meetings with local authorities three times during the six month period. Monitoring activities for vulnerable households are based on monthly visits of at least 2 monitoring staff (at least 1 male and 1 female) of the sending organization to the village of 9 origin for 6 months and the permanent availability of monitors for referrals and assistance, if required. The frequency of the activities during visits will take place according to Table 3 below. It is recommended that the sending organization organizes a 1-day training session on the SOPs and forms for all their monitoring staff. The monitoring methodology includes: Monitoring of distributions Home visits to vulnerable households FGDs Meetings with local authorities Regular phone calls with community focal points and availability of monitors at all times through a “hotline” Table 3: Post Return Monitoring Frequency Matrix Month after return Frequency and timing of activities FGD 1 1 x FGD – at location and time of distribution 2 3 Meeting with local authorities 1x 1 x at distribution date 1x 1 x FGD – at location and time of distribution 4 5 6 Home visits to all identified vulnerable households 1x 1 x at distribution date 1x 1x 1 x FGD – at location and time of distribution 1x 1 x at distribution date Distribution Monitoring: The monitors will be present at each distribution of the items of the return package as well as during each any other distribution (food, voucher, seeds, tools/equipment, livestock, etc.). The first distribution after arrival provides an opportunity for the monitors to: Share a list of locations and contact details of all medical services and schools in the proximity as well as judicial referral mechanisms and family reunification services. To provide awareness, including distribution of leaflet and other IC materials, to the community about RC the Feedback and complaint mechanism. Share at least 2 “hot-line” mobile phone numbers of monitoring staff from the sending organization with all returnees. In case any urgent issues occur related to access to health, education, land and property or other protection concerns the monitors are available for referrals. If required services are not in place or protection issues arise, monitors flag the matter to their main office. Hold the first FGD as per the SOPs below. Subsequent FGDs take place during third and the sixth distribution according to the matrix above. The monitors ensure the distributions are done in an orderly manner and make sure any assistance required to vulnerable individuals is provided. Any issues arising during distribution are flagged to their main office. 10 Vulnerable Household Visits: When returnees arrive in their village of origin, the sending organization provides the monitors with a list of the names of heads of vulnerable households and their exact locations (GPS coordinates or using landmarks/local address systems – whichever is most feasible and appropriate in the specific location), based on their identification during registration prior to the return. Data related to vulnerable households is subject to the highest protection measures; circulation is restricted to those conducting the household visits. The Post-Return Vulnerable Household Visit Form (Annex 4) will be used for the interview with the head of each of the vulnerable households and to note the monitor’s observations. In consecutive visits and for some categories of vulnerable households a few of the questions might become self-evident. In those instances the monitor is required to be flexible in completing the form. Vulnerable households must be sensitized before their return that monitors of the sending organization will visit them regularly after their return and the objective of the monitoring must be clearly explained to them. Based on the home visits, the monitors refer vulnerable households to medical services, schools, judicial referral mechanisms and family reunification services, if required. If possible, the monitors accompany the vulnerable household members to the relevant service. Focus Group Discussions: FGDs take place at the time and location of the distributions according to Table 3 above. The Post-Return FGD Reporting Form (Annex 4) will be used to record the responses of the FGD. Each session of FGDs is split into 3 separate groups: 1. 1 FGD with 10-20 youth (16 – 25 years) present at distribution 2. 1 FGD with 10-20 women present at distribution 3. 1 FGD with 10-20 men (> 25 years) present at distribution Only female staff will facilitate and attend the women’s FGD and only male staff will facilitate and attend the men’s FGD. The youth FGD can be facilitated by either. During the first round of FGDs the 3 groups of participants elect a community focal point: 1 from the youth, 1 man and 1 woman. The focal points are appointed to keep regular communication with monitors and will contact them by cell phone on a weekly basis to report on any updates on the situation in the area. It is very important to not expose the focal point to any risk and to keep their engagement low profile and informal. It recommended that the identified focal points are not community leaders. This is linked to Steps 6 and 7 in the RC SOPs 3.3.5 Post Distribution Monitoring (by third party) The stated purpose of the post distribution monitoring (PDM) is: To assess the quality of the distributions To evaluate the impact of the goods received on individual households To estimate diversion rates of the distributed goods outside the household The post distribution monitoring process consists of one method of data collection: household surveys with a random sample of returned IDP households that have received at least one distribution through the RC returns methodology. The questionnaire used during household interviews for post distribution monitoring is located in Annex 5. The PDM should be conducted 0-45 days following the distribution. 11 Sampling: Given the large number of potential distribution recipients, it is not possible to survey every beneficiary because it would be too costly or time consuming. Because of this, third parties contracted by sending organizations will use the following sampling methodology to draw valid conclusions about the entire beneficiary population. Table 4 shows examples of sample sizes for different population sizes. The population size can be defined as the total number of households from each returnee cohort7 that received a distribution. Sending organizations can also use the following website to calculate sample sizes when they do not fall within the parameters in Table 4: www.surveysystem.com/sscal.htm. Table 4: Sample Size Examples Population Size (HH) 100 200 400 500 1000 5000 10,000 Confidence Interval 5 5 5 5 5 5 5 Confidence Level 95% 95% 95% 95% 95% 95% 95% Sample Size (HH) 80 132 196 217 278 357 370 Sampling will be done randomly to ensure that each distribution household from each returnee cohort has an equal chance of being chosen for the sample and thus eliciting representative results. The steps below outline how to sample from a total population of distribution recipients using simple random sampling: 1. Obtain/create a list of households who received items during the most recent distribution in Excel format 2. Use the RAND() function in Excel to randomize the beneficiary list a. Insert a column to the left of the distribution list b. Insert RAND() in the first cell of the new column and copy for all cells in the column to generate a list of random numbers c. Highlight the entire table and sort the data from lowest to highest using the new column as the reference point for the sort function These steps have now provided the sending organization with a fully randomized list from which to sample the required sample size. Once selecting the first X number of households according to the required sample size, generate a back-up list of possible households to interview in case of non-response from the first list. This list should be the same size as the first and randomized in the same way. When any household from the first list is not available to respond, the sending organization then pulls from the second list as a replacement. A NOTE ON SAMPLING Sampling will only be done within returnee cohorts. Stratification by other population groups within cohorts will only be done on an ad hoc basis at the request of the sending organization for more detailed information about a specific population group. This is done to limit the sample size of each monitoring exercise and is based on the logic that each returnee cohort is likely fairly homogenous in livelihood strategies, income level, etc. 7 Returnee cohort: Returnee households that returned to the same VoO at the same time. 12 Keep in Mind The following points should be considered during the post distribution monitoring Each sending organization uses different modes (i.e. in-kind, cash, voucher) for distributing required goods to households. The household questionnaire form (Annex 5) includes options for beneficiaries to respond about all of these different modes. Sampling, however, will only be done within returnee cohorts and not within modes of distribution. It is important that the post distribution monitoring is conducted about one week after the distribution took place in order to capture information about household use of the items, but also ensure that the recall period has not been too long or that other distributions are not happening concurrently. Once the households that will be sampled are identified, it is then time to conduct the household survey. Household Questionnaire: The household questionnaire can be found in Annex 5 of this manual. Below outlines some guidance points for third parties in customizing the questionnaire tool: 1. Section D: Direct Distribution a. Sub-section 3 (Food Distributions): Customization is required for this section given the different items distributed by each organization to add up to the 1650 kcal minimum requirement. Third parties are required to obtain from the sending organization the list of items distributed to be input under sub-section 3 in the D.2 Item column. b. Sub-section 4: (Livelihoods Distributions): Customization is required for this section given the different items distributed by each organization, as appropriate for each community’s livelihoods practices. Third parties are required to obtain from the sending organization the list of items distributed to be input under sub-section 4 in the D.2 Item column. 2. Section E: Cash a. Column E.3 (Amount USD Received): These amounts may vary depending on when the distribution took place, how often it is to take place, etc. It will be important for all third party organizations to understand how this works for the sending organization being assessed. b. Sub-section 3 (Food Distributions): Customization is required for this section given the different items distributed by each organization to add up to the 1650 kcal minimum requirement. Third parties are required to obtain from the sending organization the list of items distributed to be input under sub-section 3 in the E.2 Item column. c. Sub-section 4: (Livelihoods Distributions): Customization is required for this section given the different items distributed by each organization, as appropriate for each community’s livelihoods practices. Third parties are required to obtain from the sending organization the list of items distributed to be input under sub-section 4 in the E.2 Item column. This is linked to Steps 6 and 7 in the RC SOPs 3.3.6 Post Return Assessment (by third party) The stated purpose of the post returns assessment (PRA) is to assess returnee integration into VoO relative to agreed-upon RC indicators in the M&E Framework. This is necessarily conducted by a third party to ensure full transparency and dispel any possibility of a conflict of interest. The PRA is conducted using three discreet methods: (1) household survey; (2) focus group discussions; and (3) satellite imagery. Sampling for the household survey is also done according to a simple and widely used methodology, as described below. 13 Sampling (for household survey): Given the large number of returnees, it is not possible to survey every beneficiary because it would be too costly or time consuming. Because of this, third parties contracted by sending organizations will use the following sampling methodology to draw valid conclusions about the entire beneficiary population. Table 5 shows examples of sample sizes for different population sizes. The population size can be defined as the total number of households from each returnee cohort. Sending organizations can also use the following website to calculate sample sizes when they do not fall within the parameters in Table 4: www.surveysystem.com/sscal.htm. Table 5: Sample Size Examples Population Size (HH) 100 200 400 500 1000 5000 10,000 Confidence Interval 5 5 5 5 5 5 5 Confidence Level 95% 95% 95% 95% 95% 95% 95% Sample Size (HH) 80 132 196 217 278 357 370 Sampling will be done randomly to ensure that each distribution household from each returnee cohort has an equal chance of being chosen for the sample and thus eliciting representative results. The steps below outline how to sample from a total population of distribution recipients using simple random sampling: 1. Obtain/create a list of households who received items during the most recent distribution in Excel format 2. Use the RAND() function in Excel to randomize the beneficiary list a. Insert a column to the left of the distribution list b. Insert RAND() in the first cell of the new column and copy for all cells in the column to generate a list of random numbers c. Highlight the entire table and sort the data from lowest to highest using the new column as the reference point for the sort function These steps have now provided the sending organization with a fully randomized list from which to sample the required sample size. Once selecting the first X number of households according to the required sample size, generate a back-up list of possible households to interview in case of non-response from the first list. This list should be the same size as the first and randomized in the same way. When any household from the first list is not available to respond, the sending organization then pulls from the second list as a replacement. A NOTE ON SAMPLING Sampling will only be done within returnee cohorts. Stratification by other population groups within cohorts will only be done on an ad hoc basis at the request of the sending organization for more detailed information about a specific population group. This is done to limit the sample size of each monitoring exercise and is based on the logic that each returnee cohort is likely fairly homogenous in livelihood strategies, income level, etc. Once the households that will be sampled are identified, it is then time to conduct the household survey. 14 Household Survey: Based on the sampling strategy explained above, each identified returnee household in the sample will be interviewed by a third party team. The interview should be conducted with the head of the household or other adult in the household to ensure accuracy of the information. The third party will use the Post Return Assessment Questionnaire Form located in Annex 6 to conduct the interview. Each interview team will consist of one man and one woman (2 total) in order to facilitate comfortable interviews. During and interview with a female head of the household, the female team member will conduct the interview while the male team member will record the answers (and vice versa for interviews with male heads of households). At the end of each data collection day, the interview teams will return to the base location with all questionnaires in hand. These will then be entered by the third party data entry team into the interagency database, as outlined in Section 4 below. Focus Group Discussions: For the PRA, FGDs will be conducted in order to measure progress on implementation of community-level disaster risk reduction (DRR) projects and emergency management measures. The PRA FGD Reporting Form (Annex 6) will be used to record the responses of the FGD. Each session of FGDs is split into 2 separate groups: 1. 1 FGD with 10-20 women of the community 2. 1 FGD with 10-20 men of the community Only female staff will facilitate and attend the women’s FGD and only male staff will facilitate and attend the men’s FGD. Satellite Imagery: Satellite imagery is an effective method of verifying data that has been collected at the household level or to assess areas that are currently inaccessible. For the purposes of the RC, it will be highly useful in analyzing how established the returnees are in their VoO and whether they have been able to establish livelihood practices that would keep them in their VoO. Linking this with the data that is collected through the household surveys and FGDs will provide a fully triangulated and highly accurate picture of the situation of returnees post return. The following structures or features will be necessary to capture satellite images of immediately after return as well as 2, 4 and 6 months after return along with the assessment methodology above. The third party will be responsible for procuring the satellite imagery as well as providing the following analyses per VoO: 1. 2. 3. 4. 5. 6. Number of shelters Types of shelters Hectares of cultivated land Visible DRR activities/projects Infrastructure Progress on the agreed early-recovery/development plan (if any) This is linked to Step 7 in the RC SOPs 15 3.3.7 Feedback and Complaint Mechanism The feedback and complaint system set-up by Danish Refugee Council (DRC) on behalf of the Return Consortium allows beneficiaries to send Short Message Service (SMS) messages and make calls to share their feedback, both positive and negative, related to programme implemented by RC members. SMSs and a summary of phone calls are stored on-line in a RC website. The website has two sections, one private, which is password-protected and one public. The private section stores the original SMS (or transcript of the phone call) with details of the sender. The public section shows both the original version in Somali and its translation into English, but without the contact details to protect the privacy of the senders. The feedback is grouped around pre-determined categories as follows: 1. Type of Feedback: FAQ/Inquiry/Suggestion/Appreciation/Not related to RC interventions 2. Complaint: General Complaint/Selection of Beneficiaries/Management of funds/Procurement process/Not related to RC interventions 3. Status: Processed/Pending 4. Return Stages: Pre-departure, travel, security/safety on route, security/safety at the place of origin, access to basic needs, livelihood, service availability, monitoring, 5. Regions 6. Projects A NOTE ON DRR Disaster Risk Reduction (DRR) is an important aspect of the RC methodology. The projects and awareness raising as part of the DRR interventions, however, are not effectively monitored at the beneficiary level. With this in mind, project records and consultations with cluster members will be key in ensuring that DRR projects are indeed being carried out in and around the VoOs. Satellite imagery will triangulate this information. The website has been made user-friendly with a map of Somalia and a search function. The website can be accessed at: http://somcdrd.org/hif/ All the information is processed at the RC level and forwarded to a focal person identified by the concerned RC member. Feedback that is categorized as “complaint” is then directed to the relevant sending organizations of the RC. Once the investigation is concluded, the sending organization provides feedback to the RC which will inform the sender. A summary of this feedback is made public in website of the Beneficiary Feedback System. A NOTE ON THE FEEDBACK & COMPLAINT SYSTEM Currently, this data must be downloaded from the separate feedback system and uploaded into the RC interagency database. Eventually the two systems may be linked to provide for seamless and automatic data input. See instruction manual for feedback and complaint system 3.4 Training Training is a critical element of any M&E system, as it allows for all team members to have a full understanding of the purpose of the research at hand, how it will be conducted, the tools that will be used, and what each of 16 their roles will be throughout the timeline. Trainings should be practical and cover every detail of the M&E system element being implemented. Below is a checklist to use for training field staff on the RC M&E System. CHECKLIST Provide each team member with a training covering the following topics Tools to be used Plan of action, including methodology to be used and time frame Working relationships: responsibility of each team member, reporting lines, etc. Logistical arrangements for the data collection (transport, accommodation, etc.) Security: existing situation and procedures during data collection 3.5 Data Verification It is important to verify all data that is collected. Each phase of data collection, outlined in Figure 1 should be accompanied by data verification processes. Below are some guidelines for the data collection phase: Data collection phase: Data collection team leaders in the field should check all data that is collected each day. The most effective method to do this is for the team leader to collect all completed tools at the end of the day and check for errors or unanswered questions. S/he would then go back to the appropriate individual and explain how it needs to be changed in subsequent data collection and, if necessary, would have the individual return to the location in question and collect the correct data. This verification should take place daily as well as on a randomized basis by someone other than the field data collection leader. It is also important that all sources used to monitor activities for the RC are verified as reliable and credible. In order to do this, the following matrix will be used for each source used (e.g. distribution records, transport lists, etc.). Any documents recorded at a D-F and/or 4-6 will require replacement or secondary documents to ensure reliability and credibility of the information contained within. Table 6: Reliability/Credibility Matrix Reliability of Source A. Completely reliable B. Usually reliable C. Fairly reliable D. Not usually reliable E. Unreliable F. Reliability cannot be judged Credibility of Data 1. Confirmed by other sources 2. Probably true 3. Possibly true 4. Doubtful 5. Improbable 6. Truth cannot be judged Reliability of source: A Completely reliable refers to a tried and tested source which may be depended upon with confidence. These are extremely rare and should be kept for special occasions. B Usually reliable refers to a source which has been successful in the past but for which there is still some element of doubt in a particular case. This should be used for sources of known integrity such as EU and UN agencies, military entities, some major NGOs, etc. C Fairly reliable refers to a source which has occasionally been used in the past and upon which some degree of confidence may be based. Some press sources and NGOs could fit in here. D Not usually reliable refers to a source which has been used in the past but has proved more often than not to be unreliable. Some press sources and NGOs could fit in here. 17 E F Unreliable refers to a source which has been used in the past and has been proven unworthy of any confidence. Reliability cannot be judged refers to a source which has not been used in the past. Credibility of data: 1 Confirmed by other sources is applicable when a source different than the originally reporting one confirms the data independently of the first source. 2 Probably true indicates confirmation of essential parts of reported data by another source. Aerial imagery is included in this category. 3 Possibly true means that investigation of a reported fact or action has revealed no further data, however the data is compatible with previous actions or background data available. 4 Doubtful is applicable to an item of data if it tends to conflict with previously reported data. 5 Improbable is applicable if an item of data contradicts previously reported data. 6 Truth cannot be judged is applicable if any freshly reported item of data cannot be compared with data from any other source. It is used when 1-5 cannot be applied. It preferred to use a rating of 6 rather than an inaccurate 1-5 rating. The rating is not a progressive degree of accuracy. It only helps to formalize the credibility of data received. Therefore it is not foolproof. The letters and numerals are independent of each other and give an overall evaluation of the data. For example, a source known to be unreliable (E) might provide accurate data which is confirmed by other sources and therefore given the rating of E1. Additionally, a report evaluated as F6 may be totally accurate and should not be routinely disregarded. (Source: EU - Assessment manual of best practices in various types of emergency) 18 4. DATA ENTRY 4.1 Interagency Database As part of the RC’s continued commitment to standardized accountability, a common online database has been developed for each RC member organization to upload data from the tools described above (URL: http://203.82.57.141/Somalia/main_admin.aspx?user_pass=admin&urole=Admin). This online database allows each organization to upload its own collected data while benefitting from auto-generated analyses and reports drawing on data from all member organizations. The following list outlines the data entry requirements to ensure regularly updated data: 1. Each member organization must identify one focal point within the organization responsible for gathering data from field teams, consolidating it and uploading it to the interagency database. 2. Each member organization is required to upload the data at least on a monthly basis (the end of the month); specific dates to be decided upon within the Consortium. 3. The preferred format for uploading data into the database is Microsoft Excel. The database is organized by data collection tool to make uploading easy. Each section of the database dealing with the standardized data collection tools has the following sub-sections: 1. 2. 3. 4. 5. Add New Record View / Edit Record Upload Multiple Records View Graphs / Reports Download Excel File (For multiple records upload) There are three other sections of the database dealing with: (1) the external feedback and complaints system; (2) global queries and reports (cross-database); and, (3) static and webmaps representing data within the database. 4.2 Steps for Data Entry As stated above, each data collection tool that members of the RC use is represented within the database. Using the paper tools, the data entry focal point enters the collected data into the forms found in the “Add New Record” section for each tool. The steps below give a general overview of how to enter data using the Joint Return Plan as an example. 4.2.1. Login Each user of the database will be assigned a unique username and password to log into the database. Figure 2 shows the login interface. The user enters his specific information for access to the database. 19 Figure 2: Login Interface 4.2.2. Dashboard Once logging in, the user will see a “Dashboard” of options. Figure 3 below illustrates the options available to the user on the Dashboard. Each RC standardized tool has its own section and discreet options within it. This serves as the main navigation page for the database. Figure 3: Database Dashboard 4.2.3. Add New Record Each standardized tool has its own web form that is based upon the paper form used in the field. By clicking on “Add New Record,” an individual questionnaire can be entered into the database. Figure 4 below shows the Add New Record form for the Joint Return Plan section of the database. Each agency is responsible for entering the information found on this page according to the specified data entry schedule. 20 Figure 4: Add New Record 4.2.4. View/Edit Record There may be instances in which a user would want to view an existing record or edit an existing record. This is possible through the “View/Edit Record” function under each data collection tool section. Figure 5 shows the View/Edit page for the Joint Return Plan in which a user can click on “View/Edit” next to one of the records shown in the table and gain access to that record. Figure 5: View/Edit Record 4.2.5. Upload Multiple Records This option makes it possible to upload multiple records at once using an Excel spreadsheet. A customized Excel spreadsheet can be downloaded under each tool section under “Download Excel File (for multiple records upload)”. The user then enters the data from multiple data collection tools into the Excel spreadsheet for upload to the system. Figure 6 shows the Upload Multiple Records page for the Joint Return Plan. The user then saves that spreadsheet on his computer, clicks on the dropdown to indicate which form he is uploading and then chooses the file name under “Browse”. By clicking “Upload”, the records are then incorporated into the system. 21 Figure 6: Upload Multiple Records 4.3 Data Verification As with data collection, data verification during the data entry phase of each element of the M&E system is critical. The RC M&E system focal point for each organization will be responsible for managing this. If an organization is using data entry staff, the suggestions below may provide some guidance: Data entry phase: Data entry staff should be experienced in “cleaning” data. They should be able to critically question each piece of data and verify whether it is a valid response to the question being asked and follows the logic of the tool that was used to collect the data. It is helpful to have a double-cleaning in which data entry staff rotate to check each other’s work. CHECKLIST The focal point for each organization should be responsible for checking the work of each data entry staff at the end of each day. A sample of forms should be compared with their respective entries to ensure accuracy. When only the focal point is entering data, the focal point’s supervisor should check a sample of the entered forms. 5. DATA ANALYSIS Once data is entered into the database, it must be analyzed. The RC M&E database allows for automatic analysis within the online system as well as externally through an interactive webmap. The database allows for the following analysis features: 1. Real-time Joint-Return Plan with planned and achieved goals 2. Queries of data collected within and across data collection tools 3. Automatically generated data visualizations with charts and graphs 4. Customizable analyses with geographic information with static maps and an online interactive webmap 5.1 Reports & Graphs Under each data collection tool section, there is a ‘Reports & Graphs” section that provides a number of select analyses in chart and graph form. Figure 7 shows one example of an analysis of planned returns by organization for 2013 and 2014. 22 Figure 7: Reports & Graphs 5.2 Global Queries & Reports TO BE COMPLETED ONCE DESIGNED IN DATABASE 5.3 Static & Webmaps Another type of analysis is geospatial representation of the data in the database. The section “Maps” includes a listing of static maps as well as a link to an interactive webmap that provides analyses down to the village level. Figures 7 and 8 provide illustrations of these sections. Figure 8: Static Maps 23 Figure 9: Webmap 24 DATA SHARING & PROTECTION Regardless of the type of data being accessed, processed and stored, security is considered of paramount importance to the RC. All data that is held by the RC is held on secure servers, with access restricted to internal use by appropriate members of staff. As data controllers for the data they collect, all sending organizations are expected to treat named data in accordance with the protection protocol, and ensure that sufficient security is in place to protect the data from unauthorized access. Each organization will have information that is for internal management use only and not intended for dissemination outside of the organization. The RC and its members shall take all measures to ensure that the data is only processed by authorized personnel and institutions, whose use and access is warranted by the pursuance of above-mentioned purposes. The RC shall take appropriate measures to protect the personal data shared under this database against accidental or unauthorized destruction, accidental loss, unauthorized access, use, alteration or dissemination, and against all other unauthorized forms of processing. Sharing information between partner organizations is vital to the provision of coordinated and seamless returnee services. In addition, the sharing of information ensures that the sending organization meets the requirements of the RC. Data sharing between agencies gives practitioners access to relevant and accurate information and enables them to provide service users with the best possible care and support. It is important, however, to be aware of highly disaggregated data at the household or individual level, as this may be highly sensitive. Annex 7 outlines the requirements and agreement that all RC members must abide by for use of the database. FAQS 25 ANNEXES Annex 1 Annex 2 Annex 3 Annex 4 Annex 5 Annex 6 Annex 7 RC M&E Framework Intention Survey Beneficiary Registration Form Post Return Monitoring Forms Post Distribution Monitoring Form Post Return Assessment Form Data Protection Protocol 26