Azerbaijan AzRCS / BRCS / ICRC Integrated Program
ToR - Evaluation 2011 & 2012 Program
1. Background
1.1. Context
The Nagorny-Karabakh conflict between Armenia and Azerbaijan (1992-94) resulted in widespread
civilian population displacement, the separation of families and the destruction of infrastructure on both
sides. Although the conflict has been frozen since 1994, 8 districts (13,6% of Azerbaijan territory) remain
under occupation, while the neighbouring villages along the line of contact (LoC) and the international
border (IB) face numerous problems in rebuilding their livelihoods. Around 99'000 persons of which
around 10'500 are IDPs live in 66 villages close to the frontline (line of contact - LoC and International
border - IB).
Civilians living along the frontline therefore face difficulties in terms of their day to day life; including
safe movement within and outside their villages, accessing water (for drinking and irrigation), engaging
in traditional agricultural activities and managing the repair and maintenance of community
infrastructure.
A significant impact of the conflict on the population, stemming from the threat to their physical safety,
is families’ diminished ability to support them-selves and to meet their basic needs. Families living close
to the LoC and IB are economically vulnerable due to several factors:

Impeded access to land, water infrastructures and means of livelihood because of the occupation of
significant part of their lands, military use of the land for the buffer zone, weapon contamination
and risk of shooting.

Lack of economic opportunities due to the reluctance of private and public investors.

There are few other employment opportunities outside of agriculture and animal husbandry and
people are only able to find seasonal employment in agriculture in other villages or more protected
towns and unemployment is widespread.

There are also very few local or international organisations working along the frontline. The only
other international organisations present in Barda are Swiss Disaster Cooperation, Oxfam and Peace
Corps.
The affected population is therefore constrained into a cycle of progressive impoverishment because
their main source of livelihood, livestock ownership and agriculture production, continually decreases to
unsustainable levels. This results in high levels of indebtedness and incapacity to generate sufficient
income.
More positively, the known reluctance of the Government of Azerbaijan to invest properly in the
infrastructure of the areas / villages close to the LoC has significantly improved in 2012, due to various
governmental development projects, improving notably access to water and housing as well as living
conditions. Moreover, security of the civilian population has been improved through the construction of
protective walls.
In spite of these developments, the exposure of civilians has not ceased, be it on the LoC or on the IB,
where the topological conditions (mountainous areas) render their situation even more vulnerable and
largely unsolvable through the construction of a protective wall. The main problems in the villages thus
continue to stem from their proximity to the Azerbaijani and Armenian / NK military positions in
frontline areas which has created a climate of fear and constant psychological pressure on the
communities for almost two decades. The proximity of the frontline further results in specific difficulties
for the population notably in terms of access to land and means of livelihood. Therefore, the focus of
EcoSec remains on those communities living along the LoC and IB.
1.2. The Institutional Partnership between ICRC and BRCS in Azerbaijan
1.2.1 ICRC Mandate and strategy:
Regular visits and assessments of the living situation of the population in villages along the frontline
(Line of Contact/International Border) have allowed the ICRC to continue to increase its understanding
of the problems that this population faces. The population of those 66 villages (24 of which are situated
behind military checkpoints) is estimated at 99'026, among them 10'534 IDPs.
The ICRC addresses the humanitarian needs of the civilian population living close to the LoC and IB
through an integrated strategy aimed at reinforcing the self-protective measures already implemented
by these communities and at increasing their capacity of resilience. The multidisciplinary response
(PROT-ASSIST-PREV) encompasses three main areas:
 risk reduction;
 improved dialogue with authorities
 improved socio-economic situation.
ICRC’s strategy therefore combines the delegation's more traditional authority-centred Protection of
Civilian Population (PCP) activities with an integrated community-based approach in order to effectively
address the needs of the civilian population affected by the conflict. EcoSec programming works at both
household (HH) and community level in the selected communities
1.2.2 Programme Description:
The Ecosec Integrated Partnership (IP) between BRC-ICRC-AzRCS is a three-year project that started in
2011 and is due to finish implementation in 2013. The project sits within the overall ICRC planning for
Azerbaijan and contributes to overall ICRC objectives.
The project aims to

improve the living conditions of the civilian population living in villages close to the Azeri-Armenian
Line of Contact (LoC) and International Border (IB) through a cash grants programme for vulnerable
households

community projects to address socio-economic needs of the selected villages

support to households to implement self-protective measures (SPM) to better protect them from the
risks of shootings resulting from living so close to the LoC or IB

capacity-building for the AzRCS's local branches where the programme is implemented. The AzRCS
and its local branches is a key partner to implementation of the above-mentioned activities.
1.2.2 Evolution of the IP 2010-2013:
2010; An ICRC household survey in August/September confirmed that prevailing insecurity was
impeding access to means of livelihood for people living in communities along the line of
contact, particularly to land and water for irrigation, as well as to public services and adequate
housing1.
2011; Based on the findings of the survey, an Integrated Partnership (IP) with British Red Cross
Society (BRCS) was proposed. With no ICRC EcoSec activities in Azerbaijan for more than a
decade, the EcoSec Department was set up in 2011. Amongst the activities, almost 1'000
households (HHs) were identified to receive livelihood support through provision of capital to
buy and/or upgrade productive assets. This support, given in the form of conditional cash grants,
was disbursed during the last quarter of 2011. By the end of 2011, 945 HHs from communities
along the LoC had received cash grants with the objective to increase their income.
2012; Initial planning in 2012 had contemplated supporting the communities through microcredit, depending on a feasibility assessment, in addition to conditional cash grants. A detailed
assessment was carried out in January 2012 in eight communities along the LoC, revealing that
supporting the most vulnerable families through micro-credit was not viable. Finally the ICRC in
coordination with the BRC took the decision to apply a methodology of conditional cash grants in
all communities. During the household selection process, however, the ICRC observed that
families were facing high level of debt and pressing needs such as health care and improvement
of housing. It was therefore decided that cash grants would have an unconditional component of
30%, in order to tackle those needs. The other 70% had to be used for a productive activity. With
a view to empower these communities to meet their own needs in a sustainable way, the IP
ICRC-BRC-AzRCS, provided cash grants with an unconditional component to the 511 most
vulnerable households in ten villages close to the LoC and one IDP settlement.
2013; Based on the learning process from 2011 and 2012, it was decided to keep the
conditionality of the cash grants in 2013, while allowing for more flexibility in its use, though
keeping it predominantly for projects increasing income, assets or decreasing expenditures,
while making a sustainable change. This was changed from the original PFR objective that
foresaw unconditional grants. The most vulnerable 500 households in 52 villages will be
supported. Through an integrated strategy, the ICRC continued to promote risk reduction /
awareness and self-protective measures at the village level, in view of increasing the capacity of
resilience of communities affected by the conflict and proximity to the frontline. During 2013,
the ICRC will support 5 communities in LoC and IB, enabling them to better fulfil their socioeconomic needs through community projects such as rehabilitation of boreholes for irrigation,
reconstruction of cultural buildings and kindergarten. In addition, 15 households will be
supported to facilitate safer access to their houses through the construction of protective walls,
reconfiguration of entrance points and windows.
2. The Red Cross & Red Crescent Movement in Azerbaijan
The Red Cross Movement is one of only a tiny handful of humanitarian actors operating in the country.
1
REF
The final number of villages assisted will be higher, due to the percentage of households meeting the ICRC
vulnerability criteria being less than foreseen.
2
2.1 AzRCS: The AzRCS was founded in 1920; from 1923 to 1991 it was part of the USSR Red Cross/Red
Crescent Societies Executive Committee, after which it became independent. In 1995 it was
admitted as 169th member to the International Federation of Red Cross and Red and Crescent
Societies.
The specific role for the AzRCS is defined in the Law of the Azerbaijan on Red Crescent Society of
Azerbaijan enacted in May 2007. The AzRCS is the only national Red Crescent Society functioning in
the territory of the Republic of Azerbaijan as a component of the International RC/RC Movement
with the aim to provide humanitarian, medical, social assistance to victims of armed conflicts and
emergency situations, to implement programs for protecting health of the population, and to
promote the international humanitarian law and principles of the International RC/RC Movement.
The AzRCS has signed MoUs with various government ministries and agencies, including the Ministry
of Emergency Situations, the Ministry of Health, and the Azerbaijan National Agency for Mine Action
(ANAMA).
2.2 ICRC: The ICRC has been working in Azerbaijan since 1992, monitoring the needs and concerns of
civilians living along the Line of Contact and the international border with Armenia. In this regard,
ICRC has been helping to address the issue of missing persons (of which there are still 4,600), as well
as providing psychosocial support to families of the missing. It also monitors the respect of IHL in
relation to PoWs/Civilian Internees, takes on the role of a neutral intermediary in case of
repatriation, and visits people detained in connection with the conflict to monitor their conditions
and treatment.
ICRC also supports displaced communities with water projects and the construction of
infrastructure.
Another prime focus for the ICRC is to promote the implementation of international humanitarian
law and its integration into the training of the armed forces as well as into university courses and
school curricula.
ICRC, in collaboration with BRC, provides cash grants to enable conflict-affected civilian communities
to recover or revive their livelihoods or to cover other essential household needs.
Finally, the ICRC works to strengthen the capacities of the Azerbaijan Red Crescent Society,
particularly its tracing and dissemination capacities and its mine action programme. In addition,
through the BRC-supported economic security activities, the ICRC has also been building the capacity
of some Azerbaijan Red Crescent branches in the implementation and monitoring of cash transfer
programming.
2.3 BRCS: The British Red Cross has been supporting the ICRC economic security work in Azerbaijan
since 2010 in the form of an Integrated Partnership (IP) as part of the wider Partnership Framework
Agreement (PFA) with the ICRC. The first step was taken in October 2010 through a joint assessment
with the ICRC to better understand the livelihoods needs of conflict-affected people living close to
the LoC. The objective was to re-establish EcoSec activities after 15 years of inaction. Following this
assessment, BRC and ICRC agreed to work together on a three year ‘Integrated Partnership’ (IP)
economic security programme, to which BRC would contribute financial support and expertise in
livelihoods and the provision of cash grants, to help aid the long-term recovery of people still
suffering from the Nagorny-Karabakh conflict. The terms of this IP (covering the period 2011-13)
were agreed on an MoU signed with the ICRC in March 2011.
BRC staff have visited the programme regularly in 2011-12 to discuss and review progress with ICRC
and AzRCS. During 2012 BRC CEO, Head of DM and the UKO Recovery Programme Officer visited.
Additionally, BRC hired a consultant to conduct a field evaluation of the impact of the 2011 cash
grants (August 2012.) By the end of 2012 BRC recruited a new livelihoods delegate for the
programme. During 2013 two programme visits have been made to the field (January and June.)
3. Terms of Reference of the 2013 evaluation
3.1 Evaluation Objectives
In accordance with the project agreement between BRC, ICRC and AzRCS the programme is
evaluated externally on an annual basis. The programme was previously evaluated in August 2012
(covering the 2011 cash programme)3. The 2013 evaluation will focus on the whole of the Integrated
Programmeduring 2011-12 along the LoC. The principle objectives of the evaluation4 of the 2011 /
2012 IP program are to:
1.
2.
3.
4.
Measure if short / medium term programme outcomes are being achieved (2012 cash grants)
In regards to the 2011 program, to the extent possible, to understand the impact of the program
To review the 2011/12 strategies and to analyse if it requires adapting5
Produce recommendations to inform the on-going cash grant program and the complementing
community projects, protection walls and capacity building in 2014
3.2 Evaluating Programme Outcomes
The principle programme outcomes will differ slightly by program year (2011/2012) but it is anticipated
that the evaluation will address findings, conclusions and recommendations in regards to the following6
components of the IP program (cash grants to individuals, community projects, self-protective
measures and capacity building), unless specified else in the following topics:
3.2.1
Program design:
> Quality of needs assessments across the IP (particularly the 2011 pure conditional cash
grant and 2012 conditional/unconditional).
> Rationale for decision made to change the methodology
> SMARTness of program indicators
> Appropriateness of verification sources
> The extent to which gender or other relevant cross-cutting issues were taken into
account in the design of the project
3.2.2
Community involvement and accountability to beneficiaries (AtB)7:
3
REF
According to the ICRCs methodology, an evaluation is an independent, objective and systematic examination of a
policy, programme/project, support service or emergency operation and its design, implementation and results. In
the ICRC, this is the task of the Institutional Performance Management (IPM) unit. A review is similar to an
evaluation, except that it is led by EcoSec at HQ (HQ-led review) or by the EcoSec team in the field (locally led
review). Given the IP constellation in Azerbaijan, the term evaluation is probably most appropriate.
5
The 2013 program implementation already received some changes compared to 2012, while the PFR 2014
foresees again a slight of the EcoSec strategy. These changes were introduced based on the feedback from the
current EcoSec team and given the fact that the IP comes to an end in 2013.
6
The term objectives in this section refers to the program objectives as detailed in the operational plans for
2011/2012.
7
The evaluation will make use of the British Red Cross Accountability to Beneficiaries (AtB) framework
4
>
>
>
>
Extent to which beneficiaries and local authorities were consulted on and participated in
all the phases of the programme.
Adequacy of communication channels between the team and communities, local
authorities, and beneficiaries
Extent to which communities were kept informed about the project
Effectiveness of complaint/feedback mechanisms
3.2.3
Program efficiency:
> Efficiency of use of national and international ICRC staff and AzRCS volunteers
> Adequacy of administrative and logistic support
> Adequacy of the grant amount
> Extent to which cost and outcome balance (with reference to the results monitoring
framework)
> Whether other approaches could have achieved the same results at a lower cost (e.g.
sectorial agro/livestock intervention) – also see 3.2.5.
> Efficiency of the grant disbursal mechanisms
> Efficiency and effectiveness of coordination with local authorities and other
stakeholders (AzRCS and Bank)
3.2.4
Program effectiveness:
> Extent to which programme objectives were achieved according to the results
monitoring framework
> Identification of major factors influencing the achievement or non-achievement of the
objectives
> Differences in effectiveness of the two grants (2011 and 2012), specifically regarding
their change in value, more focused targeting (fewer beneficiaries), and transition from
conditional/unconditional to unconditional.
3.2.5
Relevance/appropriateness8:
> Current validity of the objectives of the program
> Appropriateness of the programme in relation to the economic security and other
problems faced by the beneficiaries in the context
> Did the targeting criteria enable to reach beneficiaries capable of implementing their
projects ?
> Overall appropriateness of the projects proposed across the integrated programme both
cash grants and the community projects
> Appropriateness of coaching / training / support provided to the AzRCS in enabling
them to meet their role under the IP
> Perception of the support received by AzRCS staff and volunteers
> Extent to which the cash grant approach could be combined with other approaches
(distributions/micro-finance) including community projects
> Beneficiary perception regarding the inputs (quality and quantity)
> Consideration of alternative cash programming designs - specifically whether other
approaches could have achieved the same results at a lower cost (e.g. sectorial
agro/livestock intervention)
> Overall appropriateness of cash grants in support of income generation activities
8
Relevance concerns programme design whereas appropriateness concerns the extent to which the programme is
tailored to beneficiary needs and context.
3.2.6
Coverage:
> Appropriateness of the community targeting criteria and the household targeting
criteria
> Extent to which targeting criteria were applied, including levels of errors of inclusion and
exclusion
> Identification of immediate priorities for as yet untargeted populations
3.2.7
Connectedness/sustainability:
> Extent to which the benefits of the whole programme continued after funding ceased
(where regarding cash grants to focus on 2011 grants)
> Major factors which influenced the achievements of sustainability of the program
> Sustainability of formal and ad-hoc training provided to the AzRCS society volunteers
3.2.8
Impact:
> What happened as a result of the program (impact on whole village level, psychological
effects, security perception, reaction / assistance of/by authorities)?
> What were the intended and unintended (positive or negative) effects, including
protection aspects (Do No Harm)?
In regards to the objectives of the program, this refers to the original or revised Results Monitoring
Framework as detailed in the operational plan of the programme9 for the corresponding years.
3.2 Evaluation Outputs
The principle output of the evaluation should be a report, detailing the approach, methodology and
limitations, findings, conclusions and recommendations, referencing all relevant documents and
intermediary work it is based upon. The report should be no more than 30 pages long plus additional
annexes.
Additional outputs should include:
 A discussion of the findings in Barda, incl. implementing AzRCS HoBs and ICRC Barda SD as
invitees (highly recommended)
 A presentation of the findings in Baku, incl. AzRCS representatives and ICRC HoD as invitees
(highly recommended)
 A draft report prior to the departure of the evaluator to receive feedback upon (or powerpoint
containing the essential information.)
 A presentation of the findings in London, incl. BRCS HQ invitees (Highly recommended)
 A final report within 2 weeks of return of the evaluator (mandatory-date to be agreed)
 An optional conference call / meeting, including all actors involved
4 Methodology
The methodology used should be defined in detail by the evaluator following a literature review and
discussions with the Barda SD and BRCS.
The following points can be understood as guidance for the evaluation:
9
Refer to EcoSec Operational Plan 2011/2012
4.1 Human resources (field team and support staff):




Lead evaluator (should be an independent person without direct involvement in the 2011 / 2012
programme)
Evaluation oversight team (BRC Programme Manager, ICRC Regional Ecosec Coordinator , ICRC
Ecosec delegate, BRC Performance and Accountability Adviser)
Evaluation data collection team: Lead evaluator, 1 IFRC field Officer, AzRC volunteers, BRC
Performance and Accountability Advisor, AzRCS NHQ staff (tbd)
Transport support from the BAR SD (driver)
4.2 Data collection:








Review of secondary information (core list annexed)
Separate analysis of 2011/2012 cash grants, community projects, protective walls and capacity
building. Focus of review should be on 2012 activities but an analysis of 2011 cash grants impact
should also be undertaken.
Quantitative data (structured household interviews  beneficiary households)
Qualitative data (focus group discussion with cash grant beneficiaries, groups tbc, in order
triangulate and confirm information received during the HH survey)
Qualitative data (semi-structured interview with key informants  local / provincial authorities,
AzRCS/BRCS/ICRC staff and volunteers, …; focus group discussion with community members for
community projects)
Comparison to project monitoring data
Use of 2012 HH form to also allow collection of data required for monitoring framework
indicators.
Use of case studies of individual beneficiaries (potentially following up the same beneficiaries as
during the 2012 monitoring).
4.4 Sampling:


The type of sampling used will depend on the human resources available for data collection. If
staff and time permit, probabilistic sampling (for the 2012 cash grant beneficiaries) would allow
for testing of statistically significant changes in key indicators. Depending on the circumstances,
various probabilistic sampling methods could be considered, including simple random sampling,
stratified random sampling and multistage sampling methods.
If staff and time do not permit, purposive sampling with smaller respondent numbers would be
used.
4.3 Data processing:


Quantitative: Excel spread sheets
Qualitative: summary tables
4.5 Data analysis:







The type of data analysis will depend on the sampling used.
Disaggregation of data by type of project, location, household status, gender, family size,
distance to LoC, pre- project debt level
Analysis of results on a household level: income, expenditures, debt, assets
Analysis of data at the community level: project adequacy, process for community based project
identification, community members satisfaction with project and implementation process
Justification of the validity of results for extrapolation
Production of graphs, tables, cross tabulations to illustrate the results and to identify trends for
indicators
Comparison with targets / baseline / general situation / non-assisted individuals
5 Timeframe & Plan of Action:
The fieldwork will take place in January-February 2014 and include 1 week preparation, desk/literature
review, (as well as potentially, key informant interviews and focus group discussion), 2 weeks field work,
1 week analysis, report writing and presentation.
KEY TASKS
Identification of lead evaluator and P&A adviser
Identification of ICRC participants
Finalisation of budget
Flight bookings
Design & finalisation of evaluation methodology
In country logistics planning and bookings, including
local transport
Communicating with stakeholders about the
purpose of the evaluation, and collating feedback on
TOR
Setting up interviews /
confirming availability of stakeholders
Security of the team including briefing and liaison
with local authorities
Data analysis
Drafting the report
Editing the report
Circulating the draft report
Commenting on the report
Finalising the report
6. Budget:
TBD
7. Reference material:
Evaluation Report 2011, toolkit & ToR
ICRC Barda PoAs 2011 & 2012
BRC Monitoring trip reports
ICRC Barda quarterly reports
ICRC Barda project monitoring data
RESPONSIBILTY
BRC
ICRC
BRC
BRC
BRC, ICRC, AzRCS
ICRC
TIMEFRAME
Mid dec. 2013
End Oct 2013
Dec 2013
Dec 2013
Nov-Dec 2013
Jan 2014
ICRC
Dec-Jan 2014?
ICRC
Jan 2014 ?
ICRC
Ongoing
Evaluation team
Evaluation team
leader
Evaluation team
leader
BRC
BRC
technical
advisers,
ICRC
Delegation
Azerbaijan, ICRC
regional
delegation Tblissi,
ICRC HQ, AzRCs
Evaluation team
leader
Jan- Feb 2014
Feb 2014
Feb 2014
Feb 2014
Feb 2014
Feb 2014