Terms of Reference for External Contractor
Final evaluation, Camp autonomization
Autonomization of long term camps in Delmas commune, Haïti
March 18th, 2015
Type of evaluation
Expected evaluation
methodologies
Number of evaluators
Expected start/end dates,
number of work days
Deadline for receiving
applications
1.
End-of-project evaluation
The Camp Autonomization project financed by the ARC is ending
(June 2015)
Qualitative methodologies with use of secondary quantitative
data
One evaluator is requested for this evaluation. The evaluator will
be accompanied to sites and assisted by a FRC local facilitator.
Both, individuals or firms can apply for the position.
Expected start date: April 2015
Expected work days for evaluator: 25 days
Expected work days for translator: 10 days
11:59PM EST, 31 March 2015
Description of project/program to be evaluated
1.1. Background and objectives of project/program
On January 12th, 2010, Haiti, the poorest country of the Americas, was hit by a magnitude 7
earthquake which devastated Port-au-Prince, the most densely populated city in the region, and
its surrounding areas, causing more than 220,000 deaths, 300,000 wounded and 1.5 million
displaced people.
In September 2013, the camp population had decreased to an estimated 171,974 individuals (45
280 households) living in a total of 306 IDP sites mostly located in the metropolitan area of Portau-Prince. This large decrease from the previous estimation is mostly due to the withdrawal of 3
large relocation sites (64,378 individuals) from the matrix.
Efforts are still needed to improve IDPs' dire living conditions. Unsanitary environment, poor
hygiene practices, and overcrowding put camp dwellers at risk of water-borne diseases,
including cholera. Income generating activities in camps are limited and populations can hardly
meet their basic needs. In this context, insecurity is widespread in IDPs' sites; violence is
regularly reported while protection solutions are either non-existent or unknown.
Risk of eviction remains a main protection issue for IDPs despite the decreasing number of these
events since mid-2012. Living on private lands for almost 4 years increases the potentiality of
being forced to exit the site. These evictions, when happening, barely leave time to IDPs to seek
other shelter solutions and can be conducted with various levels of threats and violence,
therefore putting them in highly vulnerable situations.
Access to safe water and sanitation facilities remains a major challenge in camps. Recent natural
disasters (storms, floods) also highlighted populations' vulnerability and unpreparedness to face
such events. In order to break the vicious circle of unconditional assistance and to build a sound
American Red Cross – Evaluation TOR template – v13
Page 1 of 13
exit strategy, FRC has focused its last interventions on strengthening the capacities of camps'
committees in these sectors.
Project Goal
The aim of the project is to improve living conditions in targeted camps until a more sustainable
solution can be proposed. More specifically, the project aims at reducing camps dependency on
humanitarian assistance and contributing to the decrease of households’ density in long term
camps.
Objectives
The project’s goal is achieved by improving living conditions in camps by increasing population
resilience, improved knowledge related to DRR, self-management and access to WASH services,
as well as livelihood opportunities for IDPs and their surrounding neighborhoods.
Outcomes
As mentioned above, the camp autonomization project provides an integrated approach (multi
sector) to services provision. The outcomes are disaggregated by sector, as follows:
I.
Camp committees have adequate capacities to self-manage DRR activities and the
targeted groups have increased their knowledge on DRR
II.
Camp committees have increased capacities to self-manage services while IDPs have an
improved access to water and sanitation
III.
IDPs have an improved access to violence prevention support
IV.
IDPs’ income and access to community services are improved through the
implementation of IGA
Activities
The majority of the project’s activities can be grouped in the four following categories:
a) Training and capacity building of committees in self-management of basic services
b) Sensitization of beneficiaries and local authorities (door to door, special events,
awareness campaigns, etc.)
c) Entrepreneurial training and IGAs for targeted IDPs
d) Construction/rehabilitation of WASH infrastructure when a gap is identified
Indicators
The above mentioned activities are gauged either by percentage increase (to calculate
knowledge, active members, etc.) or by a set quantitate target (to determine how many people
reached by sensitization, numbers of infrastructure built or renovated, etc.)
The project is funded by the ARC for a total amounting to 1,529,698.00.USD
1.2. Scope and reach of project/program
The extremely volatile nature of the camp context in Port-au-Prince has significantly influenced
the geographical scope of the project. More specifically, several of the camps target by the FRC
intervention were relocated at some point in time during the implementation of the project.
This subsequently made it difficult to keep contact with the majority of the beneficiaries,
although some specific groups within relocated camps continue to be supported by FRC
activities (such as 50 families from Bill Gates receiving IGAs, or the craftsmen group from Centre
American Red Cross – Evaluation TOR template – v13
Page 2 of 13
KID and Bill Gates, etc.) It is forecasted that the three remaining camps: Kano, Acra nord and
Acra sud will not be relocated before the end of the project.
Site name
Bill Gates
Bonnefil
Acra Nord
Acra Sud
Sinai
Centre KID
Union des Jeunes
Kano
Total
Number of
individuals
590
5 575
8 110
8 473
1 504
1 072
1 156
2 057
Number of HHs
188
1 447
1 846
2 022
441
316
457
493
28 537
Closing date
June/Sept 2014
June/Sept 2014
Still active
Still active
March 2014
Oct/Nov 2014
Jan/Feb 2015
Still active
7 210
1.3. Project/program management
The camp autonomization project is a multi-sector services provision project. As mentioned
above, activities target the four following sectors: DRR, WASH, Protection and livelihoods.
To ensure the proper running of the project, a project coordinator is responsible for overseeing
the overall implementation of the project (management, coordination, strategy, M&E, logistics,
administration, reporting, etc.). There are 4 sector managers; each is responsible for the
implementation of all activities relevant to their specific sector, as well as managing their sector
teams (each team is made up of 1 community agent, 2 technical promoters and 1 training
promoter). There is also a logistician and 4 drivers.
Whenever possible the FRC includes the Haitian Red Cross in the implementation of some of the
activities (especially in the sectors of DRR and Protection). Furthermore, some local partner
organizations were contracted to implement specific trainings.
1.4. Previous evaluation activities
Internally, the FRC bases itself principally on three evaluations to gauge project. The first is an
evaluation that was carried out by FRC staff from HQ in December 2013, which measures the
degree of autonomization of services within camps targeted by FRC past interventions. This
evaluation can be considered as the general baseline, as it provides a strong context analysis.
The two following evaluations are the KAP surveys (initial and final). One is carried out at the
beginning of the project (initial) and the second (final) is carried out towards the end of the
project (March 2015). The purpose of the KAP surveys is to measure behavioral change; more
specifically, to gauge changes in knowledge, attitude and practices of beneficiaries with respect
to WASH and DRR sectors. To increase the reliability of collected data, the methodology and
number of beneficiaries interviewed for both KAP surveys are identical.
2.
Evaluation Overview
2.1. Purpose of evaluation
The final evaluation is in compliance with donor rules and regulations, as stipulated within the
PMP. It is to be carried out towards the end of the project with the purpose of assessing if the
American Red Cross – Evaluation TOR template – v13
Page 3 of 13
project has achieved the outcomes and objectives. Furthermore, the evaluation is intended to
evaluate the impact of the project and to provide lessons learned and recommendations for
future projects.
2.2. Objectives of evaluation
The final evaluation’s principle objective will be:
 to identify the project’s shortcomings;
 to assess the outcomes of the project;
 provide a series of lessons learned and
 to take into consideration the possibility of replicating such an approach in other IDP
camps contexts, following major natural disasters.
The evaluation will mainly focus on the degree of the project acceptance by the beneficiaries, as
well as the self-management of services available in the camps. It is essential to find out if there
have been any positive or negatives changes to the lives and livelihoods of the populations
targeted. The evaluation will take into account the following criteria:
1.
Outcomes
2.
Relevance
3.
Efficiency
4.
Effectiveness
5.
Coherence
6.
Satisfaction
7.
Replicability
8.
Impact
9.
Sustainability
10.
Lessons learned / Capitalization
2.3. Main audience of evaluation
The main audience for the evaluation will be the FRC/ARC Delegation and the FRC/ARC
headquarters. The evaluation’s major emphasis will be on learning objectives.
2.4. Coverage of evaluation
The evaluation is intended to be comprehensive (i.e. to cover all sectors of activities, of
completed and ongoing activities in all of the targeted geographical areas). Please note, as
explained in section 1.2 “scope of project”, the geographical scope of the project is volatile and
has significantly changed over the project’s lifespan due to the relocation process of camps.
Beneficiaries from several camps may be difficult or impossible to locate.
3.
Evaluation criteria and questions
Criteria
Main evaluation questions
American Red Cross – Evaluation TOR template – v13
Sub-questions
Page 4 of 13
Criteria
Outcomes
Main evaluation questions
1.
Did the project achieve its
intended outcomes?
Sub-questions
1.1.1
1.1.2
1.1.3
1.1.4
1.1.5
Relevance
2
How appropriate was
project design?
2.1.1
2.1.2
2.1.3
2.1.4
American Red Cross – Evaluation TOR template – v13
Were there any important
unintended outcomes, either
positive or negative as result of
autonomization process?
What were the main reasons that
determined whether intended
outcomes were or were not
achieved?
What were the main reasons that
determined whether there were
positive or negative unintended
outcomes? Which were under
FRC control and which not?
Were the right indicators selected
to realistically/relevantly gauge
the outcomes and objectives of
the autonomization process by
sector (for example: the ability to
leave the camp, for IGAs)? and
why?
To what degree do the
beneficiaries perceive themselves
as being ‘dependent’ on
humanitarian aid in basic service
provision (WASH).
Was this program design
(integrated services: DRR, WASH,
Protection, Livelihood activities)
the most appropriate way to
achieve intended outcomes?
Were there other, more
appropriate ways (different
services) in which similar
outcomes could have been
achieved?
Which of the activities should be
reduced and which should take a
more important role so as to
better achieve the outcome?
Are the acquired skills and
knowledge relevant in new
relocation sites? (Transferable
skills)?
Was the capacity building
(trainings) components of the
project sufficient as a stepping
stone to autonomization?
Page 5 of 13
Criteria
Main evaluation questions
Sub-questions
Efficiency
3.1 Were objectives achieved on
time?
3.2 Was
the
project
implemented in the most
efficient way compared to
alternatives?
3.1.1 (Per sector of activity: DRR, WASH,
Protection and Livelihood). What were the
main constraints to achieving the desired
results in the given timeframe? and could
these constraints have been forecasted?
3.1.2 What methods could/should have
been used to mitigate the foreseeable
constraints?
3.2.1 What alternative ways could have
been used to implement the activities
(DRR, WASH, Protection, Livelihood) which
could have potentially been more
efficient? Why? (please take into
consideration time as a variable)
4.1.1
Were activities implemented
as planned? What were the
main factors that contributed
to whether activities resulted
in intended outputs and
outcomes?
4.1.2
What were the main factors
influencing whether or not
activities were welcomed or
had
resistance
from
stakeholders (with regards to
the shift of paradigm from
service
provision
to
autonomization)?
Effectiveness
Coherence
4.1 How well were
activities
planned
implemented?
project
and
5.1 How well did the project
adapt its design and
objectives to the prevailing
humanitarian context for the
target population and in the
target sector?
5.1.1
5.1.2
American Red Cross – Evaluation TOR template – v13
To what extent is the
‘autonomization’ process
perceived as relevant by other
stakeholders (beneficiaries,
governmental authorities, CCCM
members, donors, etc.) with
regards to the humanitarian
context?
What strategy would be
considered as more relevant in
regards to the humanitarian
context?
Page 6 of 13
Criteria
Satisfaction
Main evaluation questions
6.1 How satisfied were project
beneficiaries with the
project?
Sub-questions
6.1.1
6.1.2
6.1.3
6.1.4
6.1.5
Replicability
7.1 Is the project replicable in
similar/different settings in
the same country/other
countries?
7.1.1
7.1.2
American Red Cross – Evaluation TOR template – v13
What were the main issues raised
by beneficiaries (can disaggregate
if necessary) concerning their
level of satisfaction with the
project? (Did they properly
understand the ultimate outcome
of French Red Cross’ intervention,
i.e. autonomization?)
How does the evolution of
approaches to services provisions
inside the camps affect the
acceptance level of the project
(shift of paradigm:
autonomization = breaking
dependency on humanitarian
aid)?
What is the government’s
position vis-à-vis the project
strategy? Are the various
government institutions providing
these services within the camp?
What implications does this have
on the project’s implementation?
To what extent were the
beneficiaries meaningfully
involved in, implementation? and
how did this affect the level of
satisfaction?
To what extent did the Red Cross
communicate project progress
and changes in program design to
beneficiaries? What could have
been done to improve this type of
communication?
Can the autonomization approach
be replicated in other camps in
Haiti? If no, why not? If so, which
elements can successfully be
replicated and why?
Can the autonomization approach
be replicated in other countries?
If no, why not? If yes, which
elements and why?
Page 7 of 13
Criteria
Main evaluation questions
Impact
8.1 What has happened as a
result of the programme or
project?
8.2 What real difference has the
activity
made
to
the
beneficiaries?
8.3 How many people have been
reached by the project?
Sub-questions
8.1.1
8.1.2
8.1.3
Sustainability
9.1 How sustainable were
project outcomes?
9.1.1
9.1.2
9.1.3
9.1.4
Lessons learned /
Capitalization
10.1 What lessons can be learned
that would help inform future
projects in the same sector, both
in-country and in other
countries?
10.1.1
10.1.2
10.1.3
8
What were the positive and
negative changes produced by
the intervention, directly or
indirectly, intended or
unintended?
To what degree was the
autonomization process truly
achieved? What are the major
remaining gaps?
Do committee members (inside
the camps) truly represent camp
residents? To what degree are
direct beneficiaries in comparison
to indirect beneficiaries
autonomous? Is there a large
difference? Why?
What are the main factors that
affect, either positively or
negatively, the sustainability of
autonomization program?
Do services (WASH) remain
functional after relocation?
How does the inclusion of
surrounding neighborhoods
affect the sustainability of the
project (WASH service provision)?
What exit strategies were
incorporated into program
design? Were such strategies
implemented and to what extent
did they contribute to
sustainability? Is there any
alternative exit strategy which
could be relevant and not
considered in the project design?
What context specific factors in
each camp influence the success
of autonomization approach?
What are the key conditions
required for the sustainability of
activities?
Which activities can be
autonomized and why? Which
activities can’t be autonomized
and why?
Scope of work and Evaluation design
American Red Cross – Evaluation TOR template – v13
Page 8 of 13
8.1
Scope of work
The evaluator will be responsible for the following:
I.
II.
III.
IV.
V.
To procure the necessary travel documents ,airplane tickets and visa(s) in the country of
origin (including visa fees) and insurance if required (FRC does not pay for insurance)
To bring a working laptop
To print the necessary soft copies of the desk review materials while in the country of
origin
To manage the local facilitator during the evaluation and assign roles and
responsibilities
To arrange accommodation while in Haiti (for international consultants, if requested,
the FRC can assist the evaluator to find suitable accommodation).
The evaluator will not be responsible for the following:
i.
ii.
To hire local assistant or facilitator
To arrange transportation for the field visit and to and from hotel and airport
8.2 Methodologies
The evaluator will develop an inception report in which methodologies for conducting the
evaluation are described in detail. The evaluator will mainly use qualitative and rapid appraisal
techniques. The following is a list of methodologies that are considered applicable; the list,
however, should not be considered definitive and contractor is free to propose additional or
methodologies.
I.
II.
III.
IV.
V.
VI.
VII.
Desk review of key program and strategy documents, monthly reports, etc.
Literature search and review of material on the environment in which the program
operates and national policy documents
Review of evaluations and annual reports of other programs in the same sector and with
the same/similar target population
Interviews with key FRC staff
Interviews of representatives of other program stakeholders, as relevant
Focus groups and key informant interviews of the beneficiary population
Physical site inspections
8.3 Discussion of inception report
Prior to conducting the evaluation, the Lead Evaluator will prepare and submit to the FRC an
inception report detailing the methodologies and work plan for the evaluation. The FRC will
provide an inception report template for this purpose. The inception report will be discussed
with FRC staff and will be subject to approval prior to the start of field activities.
8.4 Reporting relationship
The evaluator will report to Christophe Arnold, Project Coordinator, who is the designated
evaluation manager, for all technical and contractual issues, and Ianis Proal, Deputy Head of
Delegation, for managerial and administrative issues of the evaluation.
American Red Cross – Evaluation TOR template – v13
Page 9 of 13
The evaluator will also work in close cooperation with the following persons:
1. The geographical desk, considered as his immediate superior (n+1)
2. The referent in charge of the evaluation at the headquarter, as much as the different
thematic referents concerned to whom the evaluator has a direct functional link.
8.5 International standards & Presentation of evidence
Standard evaluation and survey methodologies and good practices utilized in the international
humanitarian community should be applied. Such resources should include but are not limited
to those promulgated by the Active Learning Network for Accountability and Performance and
the Organization for Economic Co-operation and Development.
In particular, all findings and conclusions should be based on evidence which is presented in the
evaluation report. For sample surveys, detailed information should be presented on the sample
design (including sample size calculation, stratification, clustering, allocation, selection,
departures from equal selection probability and weighting), the respondent selection
methodology, nonresponse rates, and coefficient of variation, design effect and intra-class
correlation for all variables. For case studies, the criteria and processes for selecting those cases
should be presented.
8.6 Ethical Guidelines
It is expected that the evaluation will adhere to ethical guidelines as outlined in the American
Evaluation Association’s Guiding Principles for Evaluators. A summary of these guidelines is
provided
below,
and
a
more
detailed description
can
be
found
at
www.eval.org/Publications/GuidingPrinciplesPrintable.asp.
1. Informed Consent: All participants are expected to provide informed consent following
standard and pre-agreed upon consent protocols.
2. Systematic Inquiry: Evaluators conduct systematic, data-based inquiries.
3. Competence: Evaluators provide competent performance to stakeholders.
4. Integrity/Honesty: Evaluators display honesty and integrity in their own behavior, and
attempt to ensure the honesty and integrity of the entire evaluation process.
5. Respect for People: Evaluators respect the security, dignity and self-worth of
respondents, program participants, clients, and other evaluation stakeholders. It is
expected that the evaluator will obtain the informed consent of participants to ensure
that they can decide in a conscious, deliberate way whether they want to participate.
6. Responsibilities for General and Public Welfare: Evaluators articulate and take into
account the diversity of general and public interests and values that may be related to
the evaluation.
7. Respect for Red Cross principles and values: The FRC will provide the evaluator with a
contractual document stating the core principles and values of the Red Cross. This
document will need to be read and signed by the evaluator prior to the evaluation.
8.7
Future use of data
American Red Cross – Evaluation TOR template – v13
Page 10 of 13
All collected data will be the property of the French Red Cross and American Red Cross. The
contractor may not use the data for their own research purposes, nor license the data to be
used by others, without the written consent of the American Red Cross.
9
Expected activities and Deliverables
9.1
Expected activities
Activities
Number of days
In/Out Haiti
1. Desk review and literature search
3
Out
2.
3.
4.
5.
Submission of the inception report
Planning and preparatory work (with key FRC staff)
Site visits/Interviews
Preparation and presentation of preliminary findings
to FRC team before departure.
6. Travel to and from country of origine
2
1
7
Out
In
In
1
In
2
In/Out
7. Submission of draft report to FRC for comments
8. Submission of final report to FRC after incorporating
comments
Total expected work days:
8
Out
1
Out
9.2
25
Deliverables
Deliverables
1. Inception report (with data collection tools) in English
Expected deadline
2. Validation of data collection instruments and start of field work
3. Presentation of key finding to senior management of FRC in
English
4. Draft report in English
5. Final report in English
10
Obligations of key participants in the evaluation
10.1 Obligations of the Contractor(s)
a. Inform the evaluation manager in a timely fashion of progress made and of any
problems encountered.
b. Implement the activities as expected, and if modifications are necessary, bring to
the attention of the Evaluation Manager before enacting any changes.
c. Report on a timely basis any possible conflicts of interest.
American Red Cross – Evaluation TOR template – v13
Page 11 of 13
10.2 Obligations of the Evaluation Manager
a. Make sure that the contractor(s) are provided with the specified human resources
and logistical support, and answer any day-to-day enquiries.
b. Facilitate the work of the contractor(s) with beneficiaries and other local
stakeholders.
c. Monitor the daily work of the contractor(s) and flag any concerns.
d. Receive and signoff on deliverables and authorize payment
10.3 Obligations of the NHQ Technical Team
a. Review and approve the proposed methodology.
b. Provide technical oversight in the review of all deliverables.
c. Provide timely comments on the draft report.
d. Receive and signoff on deliverables and authorize payment
11
Required qualifications
The following are the desired qualifications of the Evaluator:
I.
II.
III.
IV.
V.
VI.
VII.
VIII.
IX.
12
Strong analytical thinker and skilled report writer in English
Master or PhD degree in sociology, economics or in relevant field from recognized
university
Demonstrated experience in leading evaluations of integrated large programs primarily
focusing on DRR, livelihoods, water and sanitation, protection and community
development.
Experienced in conducting evaluations in urban camp context
Demonstrated professional experience in post-disaster/humanitarian environments
Demonstrated experience in qualitative data collection and analysis
Demonstrated experience in leading focus group discussions and conducting interviews
with a wide range of stakeholders
Professional work experience in Haiti preferred
Fluency in English and French is required, and knowledge of Creole preferred
Application and selection details
12.1 Application materials
The proposal should include the following five items. Please note that any proposal which does
not contain all five items will be rejected.
1. Summary of experience (1 page maximum)
2. Example of one evaluation report of a similar nature
3. Detailed CVs of all professionals who will work on the evaluation. If there is
more than one contractor on the proposed evaluation team, please attach a
table describing the level of effort (in number of days) of each team member in
each of the evaluation activities.
American Red Cross – Evaluation TOR template – v13
Page 12 of 13
4. Professional references: please provide three references from your previous
clients.
5. Daily rate: please mention the proposed daily rate for each contractor in USD.
The Summary of experience should be no more than one page and should include the following:
Criteria
Details (this column can be deleted
Experience in leading large
scale program/program
evaluations
Number of evaluations led
(with dates, locations and
names of organizations);
number of evaluations served
as team member
Numbers of years of
experience;
Tools/methods used in past
Number of years of experience;
Titles of positions held;
Countries worked in;
Organizations worked for
Experience in qualitative
methods
Experience with integrated
programs, urban program
evaluation with focus on
health, livelihoods, DRR and
construction.
Experience in post-disaster
/ humanitarian context/ IDP
camp context
Professional experience in
Haiti
Language proficiency
for more space)
Evaluator 1
Evaluator 2
(if applicable)
Number of years of experience;
Countries worked in
Number of years of experience;
Organizations worked for
Proficiency in English, French
and/or Haitian Creole
12.2 Application procedures
Applications are to be sent to the following email address: coordo.camp.haiti.frc@gmail.com
The files for the applications will be sent in PDF format in one folder. We ask that the applicant
use the following naming convention for the files (e.g. “lastname_CV”). Incomplete applications
and applications sent after the deadline will not be accepted. Please note that the application is
open to internationals and nationals.
12.3 Deadline for applications
Deadline for email submission: March 31st, 2015 at 23h59 (Port-au-Prince time, GMT-4)
American Red Cross – Evaluation TOR template – v13
Page 13 of 13