Somalia Return Consortium
Monitoring & Evaluation System Manual
Version 7
January 2013
TABLE OF CONTENTS
Figures ................................................................................................................................................................... 3
Acronyms .............................................................................................................................................................. 3
1.
Objective of the Manual ......................................................................................................................... 5
2.
Background ............................................................................................................................................. 5
2.1 Rationale............................................................................................................................................ 5
2.2 Target Audience................................................................................................................................ 5
2.3 Document Structure ......................................................................................................................... 5
3.
Data Collection........................................................................................................................................ 6
3.1 Timeline of Data Collection .............................................................................................................. 7
3.2 Sources of Data ................................................................................................................................ 7
3.3 Data Collection Methodologies ....................................................................................................... 8
3.3.1 Intention Survey................................................................................................................. 8
3.3.2 Beneficiary Registration Form ........................................................................................... 9
3.3.3 Distribution List and Cash Transfer Records ..................................................................... 9
3.3.4 Post Return Monitoring ...................................................................................................... 9
3.3.5 Post Distribution Monitoring (by third party)..................................................................... 11
3.3.6 Post Return Assessment (by third party) ......................................................................... 13
3.3.7 Feedback and Complaint Mechanism ............................................................................. 16
3.4 Training............................................................................................................................................ 16
3.5 Data Verification ............................................................................................................................. 17
4.
Data Entry .............................................................................................................................................. 18
4.1 Interagency Database..................................................................................................................... 19
4.2 Steps for Data Entry ....................................................................................................................... 19
4.3 Data Verification ............................................................................................................................. 22
5.
Data Analysis ........................................................................................................................................ 22
6.
Data Sharing & Protection ................................................................................................................... 25
7.
FAQs ...................................................................................................................................................... 25
8.
Annexes ................................................................................................................................................. 26
2
Tables
Table 1: Specific Objectives & Corresponding Indicators ....................................................................................... 6
Table 2: Tools and Sources of Data........................................................................................................................ 7
Table 3: Post Return Monitoring Frequency Matrix ............................................................................................... 10
Table 4: Sample Size Examples ........................................................................................................................... 12
Table 5: Sample Size Examples ........................................................................................................................... 14
Table 6: Reliability/Credibility Matrix ..................................................................................................................... 17
Figures
Figure 1: Data Collection Timeline .......................................................................................................................... 7
Figure 2: Database Dashboard ............................................................................................................................. 20
Figure 3: Login Interface ....................................................................................................................................... 20
Figure 4: Add New Record .................................................................................................................................... 21
Figure 5: View/Edit Record ................................................................................................................................... 21
Figure 6: Upload Multiple Records ........................................................................................................................ 22
Figure 7: Static Maps ............................................................................................................................................ 23
Figure 8: Webmap................................................................................................................................................. 24
3
Acronyms
DRR
GSV
FGD
HHs
IDP
MoU
M&E
PDM
PRA
RC
SOPs
VoO
Disaster Risk Reduction
Go and See Visit
Focus Group Discussion
Households
Internally Displaced Person
Memorandum of Understanding
Monitoring and Evaluation
Post Distribution Monitoring
Post Returns Assessment
Return Consortium
Standard Operating Procedures
Village of Origin
4
OBJECTIVE OF THE MANUAL
The Somalia Return Consortium Monitoring and Evaluation System Manual is produced in consultation with all
members of the Return Consortium (RC) for the purpose of standardizing indicators as well as data collection,
sharing and analysis procedures. The manual is a key document in the larger body of RC documents intended
to ensure a common voluntary return methodology to leverage synergies and individual agency expertise for
more effective programming and fundraising as well as collective transparency and accountability.
This manual, in a very practical way, expresses the RC’s commitment to delivering quality services and building
accountability at both the individual and institutional level. It builds upon existing work by RC members and is
intended for use as a reference document. As the RC becomes more established and its methodology is further
refined, elements within this manual may need to be changed accordingly.
BACKGROUND
2.1 Rationale
The RC has established a Memorandum of Understanding (MoU) stating that, “The Consortium Members have
come together for the purpose of standard-setting, creating operational synergies, fundraising and advocating
for, and implementing, voluntary return programming with the aim of ensuring voluntariness of return, safety,
dignity and sustainable reintegration for internally displaced person (IDP) households in South and Central
Somalia in accordance with agreed SOPs.” In addition, the RC MoU states in article 4.2 that, “Specific guidelines
for Monitoring and Evaluation (M&E) of the Programme will be developed and endorsed by the Steering group
within three months from the signature date of the MoU, in order to standardize M&E objectives, procedures,
reporting format and dissemination of data.” This manual is a tool for the consortium to develop a more
systematic approach to the collection, management, and reporting of key data in order to inform and improve
operational and strategic decision-making and allow for the evaluation of the outputs and outcomes of its
programming.
The M&E system framework is based on the IASC Framework on Durable Solutions for Internally Displaced
Persons, establishing the context for which indicators will measure the: (1) extent returnee households are
enabled to make an informed choice concerning return; (2) extent returnee households are able to meet basic
human needs and access social services; and, (3) resilience and capacity of returnee households and receiving
communities to mitigate shocks. The system is intended to not only measure impact of RC activities across all
indicators, but also can be used as a project management tool to track progress and gaps. The IASC
Framework represents best practices on IDP returns and is thus used a backbone to the design of the monitoring
and evaluation (M&E) system.
2.2 Target Audience
This manual is targeted toward both remote managers and field staff involved in the standardized returns
methodology employed by the RC. While the manual seeks to present material in a common language that all
readers can follow, it is recommended that the manual be used by staff well-versed in both setting-up and
implementing M&E systems. The manual can also be used to train staff to collect data and contribute to the use
of the M&E system.
2.3 Document Structure
The manual is structured as a practical reference tool to support practitioners in the implementation of the M&E
framework approved by the members of the consortium. With this in mind, each element of the framework is
outlined and operationalized throughout the four stages of the system: (1) data collection (2) data entry; (3) data
5
sharing and protection; and, (4) data analysis. Each section will include a review of the methodologies and tools
used in line with the standard operating procedures (SOPs) established by the RC.
DATA COLLECTION
Each member of the RC will use the M&E Framework found in Annex I as the basis for measuring standard
indicators at the output and outcome levels. These indicators must be measured by each sending organization
in order to provide the most comprehensive data necessary for full RC analysis. Each outcome indicator directly
relates to the three specific objectives of the RC Standard Operating Procedures (SOPs). Table 1 outlines the
standard objectives and indicators for the RC. Each sending organization will be responsible for using the tools
defined in Section 3.2 according to the timeline in Section 3.1 to collect the data for the indicators below.
Sending organizations will then enter this data into the interagency online database, as specified in Section 4.
Table 1: Specific Objectives & Corresponding Indicators
Output Indicator (at result level)
% of HHs surveyed with Intention Survey
# of HHs’ intent to return confirmed via Intention Survey
# of IDP delegates involved in GSV (disaggregated by age, gender)
% of target communities supported with GSV
% of returnee households informed of what items they can take,
available transport and arrangements for those with special needs1
PROXY INDICATORS:
# of returnee HHs assisted with safe, successful transport2 to VoO
% of returnee HHs reporting critical incidents3 during the transport
# of returnee and receiving HHs assisted with RC Standard Minimum
Package
# of litres (average) of potable water available per person per day per
returnee HHs
% returnee-assisted households that have access to emergency shelter
as per Shelter/NFI Cluster Somalia guidelines4
% of returnee HHs with an average food consumption of 2,100 Kcal5 per
person per day
% of returnee children enrolled in primary school
% of returnee HHs benefitting from free health care service in the VoO
or within a certain distance
# of returnee HHs earning incentives / wages
PROTECTION RELATED INDICATORS:
# of reported acts of violence or intimidation targeting returnees on the
basis of their refugee/IDP or minority status in the last 3 months
% of assisted HHs involved in land disputes having access to formal
justice system/ community reconciliation mechanisms
# of assisted HHs involved in land/property disputes
% of Vulnerable6 returnee HHs monitored by sending organizations on
regular basis
# of family reunification mechanisms in place and accessible in the
area of return
# assisted communities where disaster risk reduction (DRR) systems
are in place
# of emergency interventions carried out to respond to shocks
# of feedbacks and complaints acquired by sending organizations
through the F&C mechanism
% of complaints addressed by sending organizations with response
provided to the complainer
Outcome indicators (at
sp. Objective level)
Specific Objectives of RC
Extent returnee
households are
enabled to make an
informed choice
concerning return
To facilitate Informed
and Voluntary
Decision of 20,000
HHs from displaced
communities about
voluntary return in
South Central
Somalia.
1
Extent returnee
households are able
to meet basic human
needs and access
social services
To support a
Sustainable and
Durable Reintegration
of 20,000 returnee
HHs in their villages of
origin.
2
Resilience and
capacity of returnee
households and
receiving
communities to
mitigate shocks
To adapt the Return
Programme through
M&E inputs in order to
mitigate shocks
affecting the returnee
and receiving
communities.
3
Minimum information that should be conveyed, based on IASC guidelines.
“Safe, successful transport” defined as returnees reaching intended destination without injury or death.
3 “Critical incidents” defined as: (1) acts based on IDP status; (2) SGBV acts; (3) inter-communal conflict; (4) armed combatant act. Based on IASC
guidelines.
4 This may refer to emergency shelter kit distribution, shelter voucher distribution or transfer of equivalent amount of cash, based on IASC guidelines.
5 2,100 Kcal per person per day to be linked with Post-Distribution Form
6 “Vulnerable” as defined by registration form and Intention Survey
1
2
6
PROXY INDICATORS:
# of returnee HHs forcibly displaced from the VoO to other location
% of returnee HHs assisted residing in the VoO for more than 3/6/12
months
3.1 Timeline of Data Collection
RC members have agreed to the procedures established in the RC SOPs, setting up a pre-determined data
collection process that will take place in-line with key steps within the SOPs. Figure 1 outlines when specific
tools will be used according to the seven steps laid out in the SOPs. The tools or sources along the bottom of
the timeline correspond to each step before it. The monitoring processes are split into activities that will be
undertaken by RC members and those that will be conducted by external parties. Each of these sources is
explained in more detail in Section 3.2.
Figure 1: Data Collection Timeline
Legend
RC Monitoring
External/Third Party Monitoring
3.2 Sources of Data
The RC M&E System uses pre-defined tools and methodologies for primary and secondary data collection. Each
source defined in Table 2 below directly links with the indicators in the RC M&E framework, providing the
necessary data to measure their progress. All relevant tools are located in the annexes designated in Table 2
and the methodologies for their use can be found in Section 3.3.
Table 2: Tools and Sources of Data
Tool/Source Type
Intention Survey
(Annex 2)
Objective
Type
Target
Used By
Internal Data Sources (directly from sending organizations)
Clarify how return to
survey
household
sending
the place of origin,
level
orgs
local integration and
settlement elsewhere
Frequency
each round
of returns
Data Entry
Mechanism
online
7
Beneficiary
Registration Form
(Annex 3)
Distribution Lists (for
in-kind provision and
voucher)
Cash Transfer
Records
Post Return
Monitoring (FGD, HH
interview with
vulnerable families)
(Annex 4)
Post Distribution
Monitoring by third
party (for in-kind
provision, voucher and
cash)
(Annex 5)
Post Return
Assessment by third
party
(Annex 6)
Satellite imagery
Feedback and
complaint
mechanism
(according to RC SOP)
in the country are
both durable
solutions available for
the IDPs
Summarize key
characteristics of
registered
households in
returning process
Record beneficiary
reception of NFIs and
other goods
To verify reception of
cash by beneficiaries
Gather information
on sustainability after
refugees have
returned
Verify quality of
distributions, impact
and diversion rate.
Improve future
programming.
Assess the process
for returnees’
transport and
integration into
Villages of Origin
Verify data already
collected and provide
additional
authentication source
Ensure high quality
program
implementation and
program adaptation
form
household
level
sending
orgs
each round
of returns
online
list
household
level
sending
orgs
at site
list
household
level
household
level;
village level
sending
orgs
sending
orgs
each
distribution/
intervention
each cash
transfer
once per
month for 6
months
third parties
after each
distribution
offline then
uploaded
online
forms
External Data Sources
survey
household
level
upon
transaction
offline then
uploaded
online
survey
household
level
third parties
month 2, 4
and 6
following
return
online
photograph,
GPS
coordinates
village and
region
levels
sending
orgs, third
parties
ongoing
online
SMS,
phone calls,
emails,
individual
level
sending
orgs
ongoing
online1
1 Initially,
this will be downloaded from the F&C online system and uploaded to the interagency database through Excel spreadsheets. Eventually, the two
systems will be linked to automatically transfer data from the F&C system to the interagency database.
3.3 Data Collection Methodologies
The following section outlines the specific methodologies that are to be used for each tool highlighted in Table 2.
The methodologies for the tools located in the Internal Data Sources section can also be found in the RC SOPs.
3.3.1 Intention Survey
The stated purpose of the intention survey is as follows:
 To verify the genuine intention of the targeted community.
 To know the real intention of the community about voluntary return.
 To know and advocate for durable solutions alternative to return, if requested by part of the targeted
community.
The intention survey is designed to be administered for all potential IDPs wishing to return to their VoO. All
sending organizations are required to implement the intention survey using a sample size of all potential IDP
returnees. The standard intention survey form (Annex 2) is used and interviews are held at the household level
with one individual from the household.
8
 This is linked to Step 2 of the RC SOPs
3.3.2 Beneficiary Registration Form
The stated purpose of the registration form is as follows:
 To have picture, age and gender data about the returning households.
 To acquire written consent by the head of household and the other family members (above 18 only).
 To make sure about the spontaneous and genuine character of the return, double-checking the list from
the intention survey with the actual registration figures.
 To identify persons (adults and children) with special needs in order to plan a customized assistance
package.
 To identify vulnerable families in the villages of origin.
 To provide additional information about the rights IDPs to durable solutions and related implications.
The beneficiary registration form is designed to officially register all households intending to return to their VoO
following their own consultation with leaders and reports from the GSV. Each family that intends to return must
register with a sending organization using the consolidated family registration form (Annex 3). Thus, the sample
size for this tool is all households intending to return.
Pre-registration arrangements and a logistical setting that allows for reduced registration exercise time are
recommended. Registration is a sensitive step of the return process where pressure from the community,
security risks, risk of fraud and/or request to enlarge the target are only some of the challenges. Reduce time and
improve efficiency using experienced and skilled staff (including staff from the areas of return who may support in
the vetting process).
 This is linked to Step 5 of the RC SOPs
3.3.3 Distribution List and Cash Transfer Records
There are not any standardized formats for distribution lists or cash transfer records; these forms are designed
using each sending organization’s own templates. The purpose of these records is to compare what is reported
by each sending organization as being distributed and what each IDP household reports they received during the
third party Post Distribution Monitoring. These forms should, at a minimum, include: (1) the data of distribution;
(2) the distributing agency; (3) the family code and name of the recipient, age and the size of the household; (4)
the number/amount of items distributed; and, (5) the mode of distribution (i.e. in kind, cash or voucher).
 These forms are linked to Step 6 of the RC SOPs
3.3.4 Post Return Monitoring
The stated purpose of post return monitoring is to provide information about to the re-integration of returnees, the
sustainability of the return process and protection risks returnees are facing, with a particular focus on vulnerable
households. This will result in referral, support with family reunification of cases of concern by the monitors and
should lead to the identification of priority issues for advocacy purposes. The results of the monitoring will also
inform planning and implementation of future returnee programs.
The methodology currently will be carried out once per month for six months for vulnerable households in the
VoOs where IDPs have been returned with Focus Group Discussions (FGDs) and meetings with local authorities
three times during the six month period. Monitoring activities for vulnerable households are based on monthly
visits of at least 2 monitoring staff (at least 1 male and 1 female) of the sending organization to the village of
9
origin for 6 months and the permanent availability of monitors for referrals and assistance, if required. The
frequency of the activities during visits will take place according to Table 3 below. It is recommended that the
sending organization organizes a 1-day training session on the SOPs and forms for all their monitoring staff.
The monitoring methodology includes:
 Monitoring of distributions
 Home visits to vulnerable households
 FGDs
 Meetings with local authorities
 Regular phone calls with community focal points and availability of monitors at all times through a “hotline”
Table 3: Post Return Monitoring Frequency Matrix
Month after
return
Frequency and timing of activities
FGD
1
1 x FGD – at location and time of
distribution
2
3
Meeting with local
authorities
1x
1 x at distribution date
1x
1 x FGD – at location and time of
distribution
4
5
6
Home visits to all identified
vulnerable households
1x
1 x at distribution date
1x
1x
1 x FGD – at location and time of
distribution
1x
1 x at distribution date
Distribution Monitoring: The monitors will be present at each distribution of the items of the return package as
well as during each any other distribution (food, voucher, seeds, tools/equipment, livestock, etc.).
The first distribution after arrival provides an opportunity for the monitors to:

Share a list of locations and contact details of all medical services and schools in the proximity as well
as judicial referral mechanisms and family reunification services.

To provide awareness, including distribution of leaflet and other IC materials, to the community about
RC the Feedback and complaint mechanism.

Share at least 2 “hot-line” mobile phone numbers of monitoring staff from the sending organization with
all returnees. In case any urgent issues occur related to access to health, education, land and property
or other protection concerns the monitors are available for referrals. If required services are not in place
or protection issues arise, monitors flag the matter to their main office.

Hold the first FGD as per the SOPs below. Subsequent FGDs take place during third and the sixth
distribution according to the matrix above.
The monitors ensure the distributions are done in an orderly manner and make sure any assistance required to
vulnerable individuals is provided. Any issues arising during distribution are flagged to their main office.
10
Vulnerable Household Visits: When returnees arrive in their village of origin, the sending organization provides
the monitors with a list of the names of heads of vulnerable households and their exact locations (GPS
coordinates or using landmarks/local address systems – whichever is most feasible and appropriate in the
specific location), based on their identification during registration prior to the return. Data related to vulnerable
households is subject to the highest protection measures; circulation is restricted to those conducting the
household visits.
The Post-Return Vulnerable Household Visit Form (Annex 4) will be used for the interview with the head of each
of the vulnerable households and to note the monitor’s observations. In consecutive visits and for some
categories of vulnerable households a few of the questions might become self-evident. In those instances the
monitor is required to be flexible in completing the form.
Vulnerable households must be sensitized before their return that monitors of the sending organization will visit
them regularly after their return and the objective of the monitoring must be clearly explained to them.
Based on the home visits, the monitors refer vulnerable households to medical services, schools, judicial referral
mechanisms and family reunification services, if required. If possible, the monitors accompany the vulnerable
household members to the relevant service.
Focus Group Discussions: FGDs take place at the time and location of the distributions according to Table 3
above. The Post-Return FGD Reporting Form (Annex 4) will be used to record the responses of the FGD. Each
session of FGDs is split into 3 separate groups:
1. 1 FGD with 10-20 youth (16 – 25 years) present at distribution
2. 1 FGD with 10-20 women present at distribution
3. 1 FGD with 10-20 men (> 25 years) present at distribution
Only female staff will facilitate and attend the women’s FGD and only male staff will facilitate and attend the
men’s FGD. The youth FGD can be facilitated by either.
During the first round of FGDs the 3 groups of participants elect a community focal point: 1 from the youth, 1 man
and 1 woman. The focal points are appointed to keep regular communication with monitors and will contact them
by cell phone on a weekly basis to report on any updates on the situation in the area. It is very important to not
expose the focal point to any risk and to keep their engagement low profile and informal. It recommended that
the identified focal points are not community leaders.
 This is linked to Steps 6 and 7 in the RC SOPs
3.3.5 Post Distribution Monitoring (by third party)
The stated purpose of the post distribution monitoring (PDM) is:
 To assess the quality of the distributions
 To evaluate the impact of the goods received on individual households
 To estimate diversion rates of the distributed goods outside the household
The post distribution monitoring process consists of one method of data collection: household surveys with a
random sample of returned IDP households that have received at least one distribution through the RC returns
methodology. The questionnaire used during household interviews for post distribution monitoring is located in
Annex 5. The PDM should be conducted 0-45 days following the distribution.
11
Sampling: Given the large number of potential distribution recipients, it is not possible to survey every
beneficiary because it would be too costly or time consuming. Because of this, third parties contracted by
sending organizations will use the following sampling methodology to draw valid conclusions about the entire
beneficiary population.
Table 4 shows examples of sample sizes for different population sizes. The population size can be defined as
the total number of households from each returnee cohort7 that received a distribution. Sending organizations
can also use the following website to calculate sample sizes when they do not fall within the parameters in Table
4: www.surveysystem.com/sscal.htm.
Table 4: Sample Size Examples
Population Size (HH)
100
200
400
500
1000
5000
10,000
Confidence Interval
5
5
5
5
5
5
5
Confidence Level
95%
95%
95%
95%
95%
95%
95%
Sample Size (HH)
80
132
196
217
278
357
370
Sampling will be done randomly to ensure that each distribution household from each returnee cohort has an
equal chance of being chosen for the sample and thus eliciting representative results. The steps below outline
how to sample from a total population of distribution recipients using simple random sampling:
1. Obtain/create a list of households who received items during the most recent distribution in Excel format
2. Use the RAND() function in Excel to randomize the beneficiary list
a. Insert a column to the left of the distribution list
b. Insert RAND() in the first cell of the new column and copy for all cells in the column to generate
a list of random numbers
c. Highlight the entire table and sort the data from lowest to highest using the new column as the
reference point for the sort function
These steps have now provided the sending organization with a fully randomized list from which to sample the
required sample size. Once selecting the first X number of households according to the required sample size,
generate a back-up list of possible households to interview in case of non-response from the first list. This list
should be the same size as the first and randomized in the same way. When any household from the first list is
not available to respond, the sending organization then pulls from the second list as a replacement.
A NOTE ON SAMPLING
Sampling will only be done within returnee cohorts. Stratification by other population groups within
cohorts will only be done on an ad hoc basis at the request of the sending organization for more
detailed information about a specific population group.
This is done to limit the sample size of each monitoring exercise and is based on the logic that each
returnee cohort is likely fairly homogenous in livelihood strategies, income level, etc.
7
Returnee cohort: Returnee households that returned to the same VoO at the same time.
12
Keep in Mind
The following points should be considered during the post distribution monitoring
 Each sending organization uses different modes (i.e. in-kind, cash, voucher) for distributing required goods to
households. The household questionnaire form (Annex 5) includes options for beneficiaries to respond about all
of these different modes. Sampling, however, will only be done within returnee cohorts and not within modes of
distribution.
 It is important that the post distribution monitoring is conducted about one week after the distribution took place in
order to capture information about household use of the items, but also ensure that the recall period has not been
too long or that other distributions are not happening concurrently.
Once the households that will be sampled are identified, it is then time to conduct the household survey.
Household Questionnaire: The household questionnaire can be found in Annex 5 of this manual. Below
outlines some guidance points for third parties in customizing the questionnaire tool:
1. Section D: Direct Distribution
a. Sub-section 3 (Food Distributions): Customization is required for this section given the different
items distributed by each organization to add up to the 1650 kcal minimum requirement. Third
parties are required to obtain from the sending organization the list of items distributed to be
input under sub-section 3 in the D.2 Item column.
b. Sub-section 4: (Livelihoods Distributions): Customization is required for this section given the
different items distributed by each organization, as appropriate for each community’s
livelihoods practices. Third parties are required to obtain from the sending organization the list
of items distributed to be input under sub-section 4 in the D.2 Item column.
2. Section E: Cash
a. Column E.3 (Amount USD Received): These amounts may vary depending on when the
distribution took place, how often it is to take place, etc. It will be important for all third party
organizations to understand how this works for the sending organization being assessed.
b. Sub-section 3 (Food Distributions): Customization is required for this section given the different
items distributed by each organization to add up to the 1650 kcal minimum requirement. Third
parties are required to obtain from the sending organization the list of items distributed to be
input under sub-section 3 in the E.2 Item column.
c. Sub-section 4: (Livelihoods Distributions): Customization is required for this section given the
different items distributed by each organization, as appropriate for each community’s
livelihoods practices. Third parties are required to obtain from the sending organization the list
of items distributed to be input under sub-section 4 in the E.2 Item column.
 This is linked to Steps 6 and 7 in the RC SOPs
3.3.6 Post Return Assessment (by third party)
The stated purpose of the post returns assessment (PRA) is to assess returnee integration into VoO relative to
agreed-upon RC indicators in the M&E Framework. This is necessarily conducted by a third party to ensure full
transparency and dispel any possibility of a conflict of interest. The PRA is conducted using three discreet
methods: (1) household survey; (2) focus group discussions; and (3) satellite imagery. Sampling for the
household survey is also done according to a simple and widely used methodology, as described below.
13
Sampling (for household survey): Given the large number of returnees, it is not possible to survey every
beneficiary because it would be too costly or time consuming. Because of this, third parties contracted by
sending organizations will use the following sampling methodology to draw valid conclusions about the entire
beneficiary population.
Table 5 shows examples of sample sizes for different population sizes. The population size can be defined as
the total number of households from each returnee cohort. Sending organizations can also use the following
website to calculate sample sizes when they do not fall within the parameters in Table 4:
www.surveysystem.com/sscal.htm.
Table 5: Sample Size Examples
Population Size (HH)
100
200
400
500
1000
5000
10,000
Confidence Interval
5
5
5
5
5
5
5
Confidence Level
95%
95%
95%
95%
95%
95%
95%
Sample Size (HH)
80
132
196
217
278
357
370
Sampling will be done randomly to ensure that each distribution household from each returnee cohort has an
equal chance of being chosen for the sample and thus eliciting representative results. The steps below outline
how to sample from a total population of distribution recipients using simple random sampling:
1. Obtain/create a list of households who received items during the most recent distribution in Excel format
2. Use the RAND() function in Excel to randomize the beneficiary list
a. Insert a column to the left of the distribution list
b. Insert RAND() in the first cell of the new column and copy for all cells in the column to generate
a list of random numbers
c. Highlight the entire table and sort the data from lowest to highest using the new column as the
reference point for the sort function
These steps have now provided the sending organization with a fully randomized list from which to sample the
required sample size. Once selecting the first X number of households according to the required sample size,
generate a back-up list of possible households to interview in case of non-response from the first list. This list
should be the same size as the first and randomized in the same way. When any household from the first list is
not available to respond, the sending organization then pulls from the second list as a replacement.
A NOTE ON SAMPLING
Sampling will only be done within returnee cohorts. Stratification by other population groups within
cohorts will only be done on an ad hoc basis at the request of the sending organization for more
detailed information about a specific population group.
This is done to limit the sample size of each monitoring exercise and is based on the logic that each
returnee cohort is likely fairly homogenous in livelihood strategies, income level, etc.
Once the households that will be sampled are identified, it is then time to conduct the household survey.
14
Household Survey: Based on the sampling strategy explained above, each identified returnee household in the
sample will be interviewed by a third party team. The interview should be conducted with the head of the
household or other adult in the household to ensure accuracy of the information. The third party will use the Post
Return Assessment Questionnaire Form located in Annex 6 to conduct the interview.
Each interview team will consist of one man and one woman (2 total) in order to facilitate comfortable interviews.
During and interview with a female head of the household, the female team member will conduct the interview
while the male team member will record the answers (and vice versa for interviews with male heads of
households).
At the end of each data collection day, the interview teams will return to the base location with all questionnaires
in hand. These will then be entered by the third party data entry team into the interagency database, as outlined
in Section 4 below.
Focus Group Discussions: For the PRA, FGDs will be conducted in order to measure progress on
implementation of community-level disaster risk reduction (DRR) projects and emergency management
measures. The PRA FGD Reporting Form (Annex 6) will be used to record the responses of the FGD. Each
session of FGDs is split into 2 separate groups:
1. 1 FGD with 10-20 women of the community
2. 1 FGD with 10-20 men of the community
Only female staff will facilitate and attend the women’s FGD and only male staff will facilitate and attend the
men’s FGD.
Satellite Imagery: Satellite imagery is an effective method of verifying data that has been collected at the
household level or to assess areas that are currently inaccessible. For the purposes of the RC, it will be highly
useful in analyzing how established the returnees are in their VoO and whether they have been able to establish
livelihood practices that would keep them in their VoO. Linking this with the data that is collected through the
household surveys and FGDs will provide a fully triangulated and highly accurate picture of the situation of
returnees post return.
The following structures or features will be necessary to capture satellite images of immediately after return as
well as 2, 4 and 6 months after return along with the assessment methodology above. The third party will be
responsible for procuring the satellite imagery as well as providing the following analyses per VoO:
1.
2.
3.
4.
5.
6.
Number of shelters
Types of shelters
Hectares of cultivated land
Visible DRR activities/projects
Infrastructure
Progress on the agreed early-recovery/development plan (if any)
 This is linked to Step 7 in the RC SOPs
15
3.3.7 Feedback and Complaint Mechanism
The feedback and complaint system set-up by Danish Refugee Council (DRC) on behalf of the Return
Consortium allows beneficiaries to send Short Message Service (SMS) messages and make calls to share their
feedback, both positive and negative, related to programme implemented by RC members. SMSs and a
summary of phone calls are stored on-line in a RC website. The website has two sections, one private, which is
password-protected and one public. The private section stores the original SMS (or transcript of the phone call)
with details of the sender. The public section shows both the original version in Somali and its translation into
English, but without the contact details to protect the privacy of the senders.
The feedback is grouped around pre-determined categories as follows:
1. Type of Feedback: FAQ/Inquiry/Suggestion/Appreciation/Not related to RC interventions
2. Complaint: General Complaint/Selection of Beneficiaries/Management of funds/Procurement
process/Not related to RC interventions
3. Status: Processed/Pending
4. Return Stages: Pre-departure, travel, security/safety on route, security/safety at the place of origin,
access to basic needs, livelihood, service availability, monitoring,
5. Regions
6. Projects
A NOTE ON DRR
Disaster Risk Reduction (DRR) is an important aspect of the RC methodology. The projects and
awareness raising as part of the DRR interventions, however, are not effectively monitored at the
beneficiary level. With this in mind, project records and consultations with cluster members will be
key in ensuring that DRR projects are indeed being carried out in and around the VoOs. Satellite
imagery will triangulate this information.
The website has been made user-friendly with a map of Somalia and a search function. The website can be
accessed at: http://somcdrd.org/hif/
All the information is processed at the RC level and forwarded to a focal person identified by the concerned RC
member. Feedback that is categorized as “complaint” is then directed to the relevant sending organizations of the
RC. Once the investigation is concluded, the sending organization provides feedback to the RC which will inform
the sender. A summary of this feedback is made public in website of the Beneficiary Feedback System.
A NOTE ON THE FEEDBACK & COMPLAINT SYSTEM
Currently, this data must be downloaded from the separate feedback system and uploaded into the
RC interagency database. Eventually the two systems may be linked to provide for seamless and
automatic data input.
 See instruction manual for feedback and complaint system
3.4 Training
Training is a critical element of any M&E system, as it allows for all team members to have a full understanding
of the purpose of the research at hand, how it will be conducted, the tools that will be used, and what each of
16
their roles will be throughout the timeline. Trainings should be practical and cover every detail of the M&E
system element being implemented. Below is a checklist to use for training field staff on the RC M&E System.
CHECKLIST
Provide each team member with a training covering the following topics





Tools to be used
Plan of action, including methodology to be used and time frame
Working relationships: responsibility of each team member, reporting lines, etc.
Logistical arrangements for the data collection (transport, accommodation, etc.)
Security: existing situation and procedures during data collection
3.5 Data Verification
It is important to verify all data that is collected. Each phase of data collection, outlined in Figure 1 should be
accompanied by data verification processes. Below are some guidelines for the data collection phase:

Data collection phase: Data collection team leaders in the field should check all data that is collected each
day. The most effective method to do this is for the team leader to collect all completed tools at the end of
the day and check for errors or unanswered questions. S/he would then go back to the appropriate
individual and explain how it needs to be changed in subsequent data collection and, if necessary, would
have the individual return to the location in question and collect the correct data. This verification should
take place daily as well as on a randomized basis by someone other than the field data collection leader.
It is also important that all sources used to monitor activities for the RC are verified as reliable and credible. In
order to do this, the following matrix will be used for each source used (e.g. distribution records, transport lists,
etc.). Any documents recorded at a D-F and/or 4-6 will require replacement or secondary documents to ensure
reliability and credibility of the information contained within.
Table 6: Reliability/Credibility Matrix
Reliability of Source
A. Completely reliable
B. Usually reliable
C. Fairly reliable
D. Not usually reliable
E. Unreliable
F. Reliability cannot be judged
Credibility of Data
1. Confirmed by other sources
2. Probably true
3. Possibly true
4. Doubtful
5. Improbable
6. Truth cannot be judged
Reliability of source:
A
Completely reliable refers to a tried and tested source which may be depended upon with
confidence. These are extremely rare and should be kept for special occasions.
B
Usually reliable refers to a source which has been successful in the past but for which there is
still some element of doubt in a particular case. This should be used for sources of known
integrity such as EU and UN agencies, military entities, some major NGOs, etc.
C
Fairly reliable refers to a source which has occasionally been used in the past and upon
which some degree of confidence may be based. Some press sources and
NGOs could fit in here.
D
Not usually reliable refers to a source which has been used in the past but has proved more
often than not to be unreliable. Some press sources and NGOs could fit in here.
17
E
F
Unreliable refers to a source which has been used in the past and has been proven unworthy
of any confidence.
Reliability cannot be judged refers to a source which has not been used in the past.
Credibility of data:
1
Confirmed by other sources is applicable when a source different than the originally reporting
one confirms the data independently of the first source.
2
Probably true indicates confirmation of essential parts of reported data by another source.
Aerial imagery is included in this category.
3
Possibly true means that investigation of a reported fact or action has revealed no further
data, however the data is compatible with previous actions or background data available.
4
Doubtful is applicable to an item of data if it tends to conflict with previously reported data.
5
Improbable is applicable if an item of data contradicts previously reported data.
6
Truth cannot be judged is applicable if any freshly reported item of data cannot be compared
with data from any other source. It is used when 1-5 cannot be applied. It preferred to use a
rating of 6 rather than an inaccurate 1-5 rating.
The rating is not a progressive degree of accuracy. It only helps to formalize the credibility of data received.
Therefore it is not foolproof. The letters and numerals are independent of each other and give an overall
evaluation of the data. For example, a source known to be unreliable (E) might provide accurate data which
is confirmed by other sources and therefore given the rating of E1. Additionally, a report evaluated as F6
may be totally accurate and should not be routinely disregarded.
(Source: EU - Assessment manual of best practices in various types of emergency)
18
4. DATA ENTRY
4.1 Interagency Database
As part of the RC’s continued commitment to standardized accountability, a common online database has been
developed for each RC member organization to upload data from the tools described above (URL:
http://203.82.57.141/Somalia/main_admin.aspx?user_pass=admin&urole=Admin). This online database allows
each organization to upload its own collected data while benefitting from auto-generated analyses and reports
drawing on data from all member organizations. The following list outlines the data entry requirements to ensure
regularly updated data:
1. Each member organization must identify one focal point within the organization responsible for
gathering data from field teams, consolidating it and uploading it to the interagency database.
2. Each member organization is required to upload the data at least on a monthly basis (the end of the
month); specific dates to be decided upon within the Consortium.
3. The preferred format for uploading data into the database is Microsoft Excel.
The database is organized by data collection tool to make uploading easy. Each section of the database dealing
with the standardized data collection tools has the following sub-sections:
1.
2.
3.
4.
5.
Add New Record
View / Edit Record
Upload Multiple Records
View Graphs / Reports
Download Excel File (For multiple records upload)
There are three other sections of the database dealing with: (1) the external feedback and complaints system; (2)
global queries and reports (cross-database); and, (3) static and webmaps representing data within the database.
4.2 Steps for Data Entry
As stated above, each data collection tool that members of the RC use is represented within the database.
Using the paper tools, the data entry focal point enters the collected data into the forms found in the “Add New
Record” section for each tool. The steps below give a general overview of how to enter data using the Joint
Return Plan as an example.
4.2.1. Login
Each user of the database will be assigned a unique username and password to log into the database. Figure 2
shows the login interface. The user enters his specific information for access to the database.
19
Figure 2: Login Interface
4.2.2. Dashboard
Once logging in, the user will see a “Dashboard” of options. Figure 3 below illustrates the options available to
the user on the Dashboard. Each RC standardized tool has its own section and discreet options within it. This
serves as the main navigation page for the database.
Figure 3: Database Dashboard
4.2.3. Add New Record
Each standardized tool has its own web form that is based upon the paper form used in the field. By clicking on
“Add New Record,” an individual questionnaire can be entered into the database. Figure 4 below shows the Add
New Record form for the Joint Return Plan section of the database. Each agency is responsible for entering the
information found on this page according to the specified data entry schedule.
20
Figure 4: Add New Record
4.2.4. View/Edit Record
There may be instances in which a user would want to view an existing record or edit an existing record. This is
possible through the “View/Edit Record” function under each data collection tool section. Figure 5 shows the
View/Edit page for the Joint Return Plan in which a user can click on “View/Edit” next to one of the records
shown in the table and gain access to that record.
Figure 5: View/Edit Record
4.2.5. Upload Multiple Records
This option makes it possible to upload multiple records at once using an Excel spreadsheet. A customized
Excel spreadsheet can be downloaded under each tool section under “Download Excel File (for multiple records
upload)”. The user then enters the data from multiple data collection tools into the Excel spreadsheet for upload
to the system. Figure 6 shows the Upload Multiple Records page for the Joint Return Plan. The user then
saves that spreadsheet on his computer, clicks on the dropdown to indicate which form he is uploading and then
chooses the file name under “Browse”. By clicking “Upload”, the records are then incorporated into the system.
21
Figure 6: Upload Multiple Records
4.3 Data Verification
As with data collection, data verification during the data entry phase of each element of the M&E system is
critical. The RC M&E system focal point for each organization will be responsible for managing this. If an
organization is using data entry staff, the suggestions below may provide some guidance:

Data entry phase: Data entry staff should be experienced in “cleaning” data. They should be able to
critically question each piece of data and verify whether it is a valid response to the question being asked
and follows the logic of the tool that was used to collect the data. It is helpful to have a double-cleaning in
which data entry staff rotate to check each other’s work.
CHECKLIST
 The focal point for each organization should be responsible for checking the work of each data entry staff at the
end of each day. A sample of forms should be compared with their respective entries to ensure accuracy.
 When only the focal point is entering data, the focal point’s supervisor should check a sample of the entered
forms.
5. DATA ANALYSIS
Once data is entered into the database, it must be analyzed. The RC M&E database allows for automatic
analysis within the online system as well as externally through an interactive webmap.
The database allows for the following analysis features:
1. Real-time Joint-Return Plan with planned and achieved goals
2. Queries of data collected within and across data collection tools
3. Automatically generated data visualizations with charts and graphs
4. Customizable analyses with geographic information with static maps and an online interactive webmap
5.1 Reports & Graphs
Under each data collection tool section, there is a ‘Reports & Graphs” section that provides a number of select
analyses in chart and graph form. Figure 7 shows one example of an analysis of planned returns by
organization for 2013 and 2014.
22
Figure 7: Reports & Graphs
5.2 Global Queries & Reports
TO BE COMPLETED ONCE DESIGNED IN DATABASE
5.3 Static & Webmaps
Another type of analysis is geospatial representation of the data in the database. The section “Maps” includes a
listing of static maps as well as a link to an interactive webmap that provides analyses down to the village level.
Figures 7 and 8 provide illustrations of these sections.
Figure 8: Static Maps
23
Figure 9: Webmap
24
DATA SHARING & PROTECTION
Regardless of the type of data being accessed, processed and stored, security is considered of paramount
importance to the RC. All data that is held by the RC is held on secure servers, with access restricted to internal
use by appropriate members of staff. As data controllers for the data they collect, all sending organizations are
expected to treat named data in accordance with the protection protocol, and ensure that sufficient security is in
place to protect the data from unauthorized access. Each organization will have information that is for internal
management use only and not intended for dissemination outside of the organization.
The RC and its members shall take all measures to ensure that the data is only processed by authorized
personnel and institutions, whose use and access is warranted by the pursuance of above-mentioned purposes.
The RC shall take appropriate measures to protect the personal data shared under this database against
accidental or unauthorized destruction, accidental loss, unauthorized access, use, alteration or dissemination,
and against all other unauthorized forms of processing.
Sharing information between partner organizations is vital to the provision of coordinated and seamless returnee
services. In addition, the sharing of information ensures that the sending organization meets the requirements of
the RC. Data sharing between agencies gives practitioners access to relevant and accurate information and
enables them to provide service users with the best possible care and support. It is important, however, to be
aware of highly disaggregated data at the household or individual level, as this may be highly sensitive. Annex 7
outlines the requirements and agreement that all RC members must abide by for use of the database.
FAQS
25
ANNEXES
Annex 1
Annex 2
Annex 3
Annex 4
Annex 5
Annex 6
Annex 7
RC M&E Framework
Intention Survey
Beneficiary Registration Form
Post Return Monitoring Forms
Post Distribution Monitoring Form
Post Return Assessment Form
Data Protection Protocol
26