Emergencycash­for­worksupporttodrought
impactedfamilies,MurdianandKhojaDoko
Districts,JawzjanProvince,Afghanistan
End­of­projectevaluation
Project implementation
Evaluation dates
Report date
Evaluation consultant
22nd Jan – 21st June 2012.
18th Sept – 1st Oct 20121
This first draft 24th September 2012.
Prof Richard C Carter2.
1
Combined with a second project evaluation in Kandahar province.
Director, Richard Carter and Associates Ltd, Second Floor Rear, The Oxlip, Rear of 2 Church Street, Ampthill,
Bedford, MK45 2LY, UK. Email richard@richard­carter.org Tel +44 (0)1525 405147 Mob +44 (0)7739 184305.
2
1
Acknowledgments
I would like to express my sincere thanks to the community members and representatives of
Jangal Arigh Watani and Chalik Ganj Turkmani villages of Murdian District and the
community members and representatives whom I met in Arab Qarloq village, Khaja Dukoh
District. Thanks to the staff of Tearfund’s office in Sheberghan (Carlos Gomar, Area
Coordinator; Mohammad Shafie, Project Manager; Rahmatullah Samandar, Community
Supervisor; Engineer Hashim, Project Engineer; Seema, Data Entry Operator and Female
Interviewer; and Babur Hamidee (Finance Officer). Thanks also to the Programme Director
Sudarshan Reddy Kodooru and his staff at the Tearfund Operations Office in Kabul. All of
these and Bryony Norman in the Teddington office have made this evaluation possible and
contributed in a large way to its smooth running.
Cover photograph: Project Manager Mohammad Shafie beside recently repaired canal in
Arab Qarloq village, Khaja Dukoh District, Jawzjan Province.
2
Maps
Jawjan Province
3
Contents
Acknowledgments................................................................................................................. 2
Maps ..................................................................................................................................... 3
Glossary ................................................................................................................................ 5
Exchange rates ...................................................................................................................... 5
Executive Summary ............................................................................................................... 6
1.
Introduction .................................................................................................................. 9
This report......................................................................................................................... 9
Background to the evaluation ........................................................................................... 9
Project concept ............................................................................................................... 10
Evaluation questions raised ............................................................................................. 11
Methodology................................................................................................................... 11
Report structure .............................................................................................................. 15
2.
Findings ....................................................................................................................... 15
Introduction .................................................................................................................... 15
Six perspectives ............................................................................................................... 15
The bigger picture ........................................................................................................... 22
3.
Conclusions ................................................................................................................. 23
4.
Recommendations ...................................................................................................... 24
Appendices ......................................................................................................................... 25
Appendix A Terms of reference ........................................................................................... 26
Appendix B Itinerary............................................................................................................ 33
Appendix C Semi­structured interview guidelines ............................................................... 34
Appendix D Consultant’s CV ................................................................................................ 35
4
Glossary
AFN
CDC
cfw
DDA
EFSA
GBP
OFDA
TOR
UNOCHA ERF
USAID
USD
WFP
Afghani (currency)
Community Development Council
Cash for work
District Development Assembly
Emergency Food Security Assessment
Great Britain Pound
Office of Foreign Disaster Assistance
Terms of reference
United Nations Office for the Coordination of Humanitarian
Assistance Emergency Relief Fund
United States Agency for International Development
United States Dollar
World Food Programme
Exchangerates
Approximate rates of exchange at date of report
USD1.00
GBP1.00
USD1.00
GBP1.00
AFN49.82
AFN80.85
GBP0.62
USD1.62
5
ExecutiveSummary
Summary of findings (section 2) and conclusions (section 3)
ES1. This is the final evaluation report of the United States Agency for International
Development (USAID) Office of Foreign Disaster Assistance (OFDA) funded cash­for­
work (cfw) project carried out by Tearfund in Murdian and Khoja Doko Districts of
Jawzjan province between January and June 2012. The evaluation was requested by
Tearfund’s Kabul and UK offices.
ES2. The project was a cash­for­work (cfw) intervention designed to relieve the acute food
insecurity of some of the most vulnerable people in 16 villages of Murdian and Khoja
Doko Districts. There were 1,750 direct cash beneficiaries, of who 438 were non­
working women and vulnerable male community members. Each received
approximately USD135 for 27 days work (or cash grant in the case of non­workers).
Beneficiaries were selected according to a strict and transparent process of poverty
and vulnerability assessment. Work undertaken included road, canal and water well
and reservoir repairs, and wall construction.
ES3. The evaluation was rapid, and hence highly participative in nature. A strong emphasis
was placed on generating discussion between beneficiaries, project staff and the
consultant, in order to reflect on experiences and develop improved approaches for
the future. Female beneficiaries were interviewed by a female interviewer, as is
normal in the culture. Beneficiary interviews were used to stimulate discussion rather
than collect representative data.
ES4. The project had a significant impact on its beneficiaries, within its limited scope and
goals of injecting cash into very vulnerable households and enabling valuable
community work to be undertaken. Longer­term impacts are probably rather limited.
There was some suspicion that the project will have helped to fuel an aid­dependency
culture. There were no other discernible negative impacts. In Tearfund’s rating scale
(1 low, 4 high) impact is rated at 3.
ES5. Stakeholders expressed their satisfaction with the project, and it was clear that
relationships between project staff, local Government, community representatives and
beneficiaries are good. Community members and their representatives did however,
understandably, request more and longer interventions in future. Stakeholder
satisfaction is rated as 4.
ES6. All planned outputs were delivered as set out in the project proposal, with the
exception of the USD level of beneficiary payments, which was a little lower than
planned (presumably because of exchange rate fluctuations, unconfirmed). Output
delivery is rated as 4.
ES7. Project implementation processes were of high quality, particularly emphasising
Tearfund’s standards of Impartiality and Targeting, and Accountability.
Communications and coordination were of a high standard. These processes are rated
6
as 4. Processes around learning and innovation, and influencing of wider practice,
were less strong. These are rated as 2.
ES8. Financial resources were disbursed as planned and beneficiary payments amounted to
60% of total project budget. This is judged acceptable in the context. Human
resources were competent for the task, and time was just sufficient after delays in
contract signing to complete the project on time. Resource aspects are rated as 4.
ES9. Organisational capacity was judged purely on the basis of the achievement of the
project. The fact that the project was delivered as planned to a high standard implies
an adequate level of organisational capacity, and there was no evidence of significant
weakness, save for the observation of weaknesses in information and file­sharing
systems. Organisational capacity is rated at 3.
ES10. The project did achieve its short­term goals. However the situation which gave rise
to the project remains largely unchanged, and the lasting impact is likely to be small. I
believe strong consideration should be given to achieving better integration of
Tearfund’s long­term development, DRR and emergency response work in Jawzjan and
elsewhere in Afghanistan.
Recommendations (section 4)
In future cfw projects:
R1. Types of work. Consider including other forms of work (not only unskilled labour),
including more permanent works involving skilled labour. In this way physical structures
may have longer lives, and beneficiaries may learn new marketable skills. This of course
has budgetary implications.
R2. Disaster risk reduction. Integrate disaster risk reduction training and action planning
with emergency response cfw, devoting sufficient time to this activity to fully embed it
in the communities involved.
R3. Donor-dependence. Reflect on the extent to which projects of this type are
exacerbating donor­dependence. Develop detailed guidance on limiting that negative
impact.
R4. Spreading thin or digging deep? Reflect on the dilemmas about how limited budgets
should be spent in such emergency response interventions. This project assisted 10
villages out of 32 in Murdian and 6 out of 19 in Khaja Dukoh. Should it have gone
deeper in fewer villages, or spread even more thinly? Develop a position on this.
R5. Information systems. Develop systematic information and filing sharing arrangements
so that all who need project information can access it easily and quickly.
R6. Evaluation budget. Budget more for future evaluations – 2­5% of total project budget
for a project of this size, depending on the strategic importance of the evaluation.
7
In regard to future programming in Jawzjan Province
R7. Learning and influencing. Introduce a systematic process for project learning,
documentation of learning and influencing of other stakeholders (Government, donors,
other NGOs). This should include mid­term and end­of­project reviews and exchanges
of experiences within country.
R8. Integration of development and emergency work. Consider restructuring the work in
Afghanistan in order to fully integrate long­term development with short­term
emergency response interventions within limited geographical areas. Address root
causes of food insecurity alongside providing emergency assistance.
8
1. Introduction
Thisreport
This is the final evaluation report of the United States Agency for International Development
(USAID) Office of Foreign Disaster Assistance (OFDA) funded cash­for­work (cfw) project
carried out by Tearfund in Murdian and Khoja Doko Districts of Jawzjan province between
January and June 2012. The evaluation was requested by Tearfund’s Operations Team in
Afghanistan and coordinated by Tearfund’s office in Teddington, UK. Those commissioning
the work were Sudarshan Reddy Kodooru, Programme Director, Tearfund Afghanistan
(Afghan­PD@tearfund.org) and Bryony Norman, Programme Officer, Tearfund UK
(bryony.norman@tearfund.org). The evaluation consultant was Richard Carter
(richard@richard­carter.org). The consultant’s CV is included as Appendix D. The evaluation
was undertaken between 18th September and 1st October 2012, in conjunction with a
second project evaluation in Kandahar, for which a separate report has been submitted.
Backgroundtotheevaluation
The north of Afghanistan was seriously affected by drought in 20113. The drought affected
14 provinces from Herat in the west to Badakhshan in the east. In response, the World Food
Programme (WFP) carried out an Emergency Food Security Assessment (EFSA) between July
and September 20114. This highlighted the food needs of 2.8m people across the northern
provinces. Tearfund carried out its own assessment of the impact of the drought in four
districts of Jawzjan Province in mid­August 2011, using the same methodology as the WFP
EFSA.
In late 2011 Tearfund submitted a proposal to OFDA for a cash­for­work project in Murdian
and Khoja Doko Districts of Jawzjan Province. The final revised proposal is dated 17th
January 2012, and the project commenced shortly thereafter (although it is understood that
the project contract was not signed until May 2012). The project would provide cash for
unskilled work for 1,312 labourers and cash grants (no work) for a further 438 women who
for cultural or other reasons were unable to work. Each individual (both workers and non­
workers) would receive USD145, and the assumption was that a total of 1,750 individuals
would receive cash, contributing to the incomes and food needs of 12,250 households5.
In the two districts targeted by the present project, it was estimated that 3,571 households
were highly impacted by the drought. Out of these, 476 were targeted by another donor­
funded project (UNOCHA ERF), and 576 were notionally targeted by Government food
3
Winter rain usually falls between November and December, with a second period of rain April to June.
Consequently by early to mid­2011 it was clear that the harvest would fail, leaving the poorest households
particularly vulnerable through to the next agricultural season in 2012.
4
World Food Programme / Food Security and Agriculture Cluster (2011) Drought Impact Emergency Food
st
Security Assessment in Fourteen Affected Provinces of Afghanistan. 1 November 2011.
5
The assumption being that the average household size is 7.
9
distribution efforts. Therefore a further 7706 vulnerable households’ needs would remain
unaddressed even with the present project.
The work undertaken in the project was to include rehabilitation of or improvements to
secondary/feeder roads; rehabilitation of water storage points and their catchment areas;
repairing or improving existing wells; repairing irrigation canals; and building fences or
surrounding walls for schools.
Delays in release of funds meant that the initial emphasis in the project was a focus on
‘software’, highlighting disaster risk reduction actions which vulnerable households and
communities could take. Work, and cash payments, only commenced later.
Projectconcept
The project proposal consists of a narrative, Gantt chart, budget and budget narrative.
There was no explicit logframe. The project had three specific intentions:
�
�
�
to relieve short­term acute need due to drought and food insecurity – this was the
cash component, targeted at the most vulnerable;
to leave behind some rehabilitated or repaired physical works which would serve the
community after the intervention – the work component of the project;
to raise awareness of the measures that households and communities could take to
reduce their vulnerability to future adverse events – the DRR component of the
project.
These three components are somewhat intertwined, and the assumption was that carrying
them out together would lead to enhanced outcomes. Making cash payments conditional
on work inputs, and ensuring that work done reflects community needs and desires, is
assumed to empower and impart a certain level of dignity. Linking rehabilitation work – in
most cases – to practical DRR helps to turn DRR from a set of abstract ideas into concrete
reality.
The stated project objective was: “Injection of cash in order to secure emergency food
needs of most vulnerable households.”
The indicators of achievement set out in the project proposal were:
�
�
�
the number of people employed through cash­for­work (cfw) activities;
the average USD amount per person earned through the cfw activities;
the number and percentage of women employed through cfw activities.
6
3,571 minus (476 plus 576 plus the 1,750 in the present project). There is an error of one in these estimates.
10
Evaluationquestionsraised
The project proposal raises a number of questions in the mind of the evaluation consultant.
These include:
�
�
�
�
�
�
�
�
�
Was cfw the most appropriate response to the drought?
Was the beneficiary selection process appropriate?
Did the exclusion of some vulnerable households create any problems?
How efficiently was the cash disbursed?
How was the cash spent?
What short­term and longer­term difference did the cash make?
How useful were the work activities (especially in terms of reducing future
vulnerability?)
How useful were the DRR awareness­raising activities?
What is the lasting value of the project, and could the money have been better
spent?
Some of these questions are more easily addressed than others. Multiple sources of
information and evidence help to give confidence in the conclusions drawn later in this
report (see next section).
Methodology
Generalapproach
There are numerous types of, and approaches to, evaluation. Daniel Stufflebeam sets out
22 types out of which 9 are preferred for various reasons according to circumstances7. The
present evaluation comes closest to Stufflebeam’s category of “Decision/Accountability­
Oriented Studies”, which he describes in the following terms:
“The decision/accountability­oriented approach emphasises that program
evaluation should be used proactively to help improve a program as well as
retroactively to judge its merit and worth.”
“Practically, the approach is oriented to engaging stakeholders in focusing the
evaluation; addressing their most important questions; providing timely, relevant
information to assist decision making; and producing an accountability record.”
This approach scores highest in Stufflebeam’s ratings of overall merit, based on the Joint
Committee on Standards for Educational Evaluation program evaluation standards. The
7
st
Stufflebeam, D (1999) Foundational models for 21 Century Program Evaluation. The Evaluation Center,
Western Michigan University, Occasional Paper Series,
https://www.globalhivmeinfo.org/CapacityBuilding/Occasional%20Papers/16%20Foundational%20Models%20
for%2021st%20Century%20Program%20Evaluation.pdf
11
conclusions drawn apply not only to educational programmes but in general to all types of
project and programme evaluation.
Most evaluations are brief and rapid, but this one was especially so. Rapid evaluations
require very strong collaboration between the external consultant and the project team,
and a high level of trust and honesty. Close rapport of this type risks a loss of objectivity, so
this needs to be guarded against continuously. The pragmatic approach taken by this
consultant is set out in Box 1.
Box 1 Evaluation approach
Purpose of the evaluation. One purpose of the evaluation is to provide accountability to the donors. This is
secondary however to the main purpose, which is about learning. The learning which is generated by the
evaluation is of course partly about looking back (what went well? What could have gone better?), but
importantly too it is about looking forward, and building the lessons into future projects and programme work.
Learning. If everything goes well all the time, there is no opportunity for learning. It is only when we face
difficulties or downright failures that the possibility exists to learn. It would be highly surprising if, in the
context of Afghanistan and these projects, everything went well all the time. Difficulties and failures need to
be turned around into learning, and that is an important, if not the most important aspect of evaluation.
Time, data and rigour. The evaluation is seriously constrained in terms of time available, especially for field
work. It is clearly not possible for the evaluation consultant to collect an amount of data which can in any
sense represent (statistically or in any rigorous manner) the achievements and value of the projects, or the
challenges they have faced.
The importance of trust. In a rapid evaluation of this type it is especially important to quickly establish a level
of rapport and trust between the consultant and project team. The evaluation must be a joint activity of
consultant and project staff, rather than a totally external activity of the consultant alone. The greater the
level of trust and honesty, the greater will be the value of the learning.
Complementing each other. The project team brings to the table their detailed knowledge and understanding
of the context, the achievements and the constraints. The consultant can bring a wider experience of projects
of a similar type elsewhere, knowledge of the thinking in the sector(s) and an ability to ask key questions which
may help the project staff reflect on how to do things differently in future. Each needs the other.
Risks. Too close a rapport between consultant and project team can risk the results of the evaluation being
insufficiently challenging. There is also a risk that the consultant will only be shown the favourable aspects of
the project. Being aware of these risks is the key to minimising their impact.
The place of field work. The main value of the field work in a rapid evaluation is not the collection of new
data, but rather the stimulation of discussion between external consultant, project team and communities,
which can lead to new insights, ideas or proposals for future work.
Evidence
Evidence from which reliable conclusions can be drawn comes from a variety of sources.
The particular conditions of the present evaluation mean that some sources are of higher
12
priority than others. Table 1 lists the various sources of evidence which inform this
evaluation. Note this is not meant to be used in all cases, but has been drawn up specifically
for this evaluation. It therefore represents this consultant’s subjective judgments. From the
table two things are evident: first, because of the time constraints in the evaluation, and
because of the commitment to quality both in monitoring and by the organisation more
generally, the highest weightings are placed on discussions with project managers and
project coordinator, and the monitoring data generated to date in the projects. Secondly
though, it is always important to consider multiple sources of evidence, and to cross­check
or triangulate wherever possible – so even low priority sources of evidence need to be
assessed opportunistically.
Table 1 Sources of evidence considered and assessed for the evaluation.
1
Source of evidence
Beneficiaries
2
Field staff
3
4
5
6
7
Managers & Coordinators
Headquarters senior staff
Community representatives
Local Government
Internal monitoring –
quantitative
Internal monitoring –
narrative
Relevant external
documentation
8
9
Comment
The best­informed and most important in principle, but the
least accessible, especially in terms of numbers (hence the liow
weighting).
High quality source, but may show reporting bias in a rapid
evaluation. Trust and rapport are key.
If well­informed by field experience, very high quality.
Good overview but may be less acquainted with detail.
Good source, if well­informed about their communities.
Useful general source, but risks of misinformation and bias.
Frequent and regular monitoring data of high quality,
complementing 8 and other sources.
Frequent and regular monitoring data of high quality,
complementing 7 and other sources.
Other project experiences in Afghanistan and elsewhere, and
research findings more widely inform the evaluations.
Weighting*
1
2
3
2
2
1
3
3
2
*Note – 1 means low weighting, 2 intermediate, 3 high.
Sampling
Despite the fact that very few beneficiary interviews could be conducted in such a short
evaluation, they are nevertheless very important as they say something about the
experiences of those who were meant to benefit from the project assistance. Choosing
households and household members to interview is a balance of pragmatism with ideals,
the former being dominant in the present context. Box 2 sets out the approach taken at the
outset. As noted later, even this pragmatic approach had to be modified in the field.
Box 2 Household sampling for the evaluation
8
Learning from experience. The evaluation consultant for Tearfund’s BPRM 2 project attempted to select
beneficiaries for interview using a systematic and rigorous process of randomisation. This was a creditable
attempt to achieve impartiality, but it encountered a number of challenges, notably because of insecurity,
access constraints due to road conditions, beneficiary non­availability and communications difficulties.
8
Integrated and Sustainable Services for Returnees and Host Communities in Kandahar, Jawzjan and Faryab
th
th
Provinces, undated report based on field work carried out 12 ­29 November 2011 by Katie Toop.
13
Random selection is easier to do on paper than out in the real world, especially in the context of this project.
In the current evaluation time was severely constrained, and therefore randomisation would have given little
benefit since a random sample would still be too small to be representative of the beneficiary population.
Basis for selection. Field staff members have been visiting beneficiary households regularly over the course of
the project. They therefore have a good understanding of those households which have experienced a good
and positive outcome from the project, and those which have experienced less benefit. The reasons for each
of these may be quite complex, depending on the individuals and households involved, as well as on the way
the project was implemented with them. To achieve any kind of representativeness in a small sample, it is
important to know which category of household is being interviewed, and to interview across the range of
such categories. In this evaluation therefore we have attempted to select households in which the outcome /
impact has been high, those for which it has been intermediate, and those in which it was low – and to try to
find out why the benefits were not evenly experienced by all.
Household categories. Beneficiary households could be usefully categorised according to a number of
different variables such as (a) pre­project poverty (or some form of wealth­ranking), (b) household size, (c)
household land­holding, (d) household social status, (e) response to the assistance, or (f) experience of the
assistance given. We assume that the beneficiary selection criteria have adequately dealt with many of these,
and that the key categorisation is the last one, in other words the extent to which households have benefited
from the project. We propose therefore the three categories as set out below:
Category A Households which have seen very significant change for the better as a result of the project. Such
changes were evident during the project, and there are strong signs that they will be sustained over time.
Category B Households which have experienced some benefit, but where project staff feel it is likely that the
households will be little better off after the project than before (ie no lasting change).
Category C Households which have received the project outputs, but in which little if any change appears
evident, either in the duration of the project or subsequently.
The number of households interviewed in the evaluation was determined according to the time and human
resource constraints imposed by the evaluation budget.
Villagevisits
Two villages in Murdian District (Jangal Arigh Watani and Chalik Ganj Tukmani) and one in
Khaja Dukoh District (Arab Qarloq) were made. In each visit a group meeting was held with
Community Development Committee (CDC) and District Development Assembly (DDA)
representatives and male beneficiaries, separate interviews were held with female (non­
working) beneficiaries, and observations were made of some of works carried out under the
cfw project.
Householdinterviews
Interviews of female grantees of cash (non­workers) were carried out by a female Afghan
interviewer. Interviews of male recipients of cash for work were carried out by the
consultant and project team members, with translation. The interviews were carried out
according to a semi­structured (conversational) style, as indicated in Appendix C. Interviews
with female beneficiaries (14 in total) were carried out, as intended, on a one­to­one basis.
Interviews with male beneficiaries were carried out in group meetings in the presence of
CDC and DDA representatives. These group meetings were an essential part of proper
14
protocol, but by the time they were completed there was insufficient time to carry out
additional household interviews and observe some of the physical works. Emphasis was
therefore put on the latter.
Anoteondocumentation
As evaluation consultant, I received a number of documents from the Project Officer in
Teddington, and others from the Area Coordinator in Sheberghan. The 215 files provided
(excluding photographs) were not well­organised. Specifically they are not all readily
identifiable by file name, they are not held in a systematic file structure, nor are they
available in one place accessible to all who need them. This should be corrected in future
projects.
Reportstructure
The evaluation findings are set out according to the 6 perspectives from which Tearfund
views the performance of projects, namely the perspective of impact, stakeholders, outputs,
process, resources and organisational capacity (see TOR, Appendix 1). A summary of
conclusions follows, with corresponding recommendations.
2. Findings
Introduction
Each of the six perspectives in the following sections is addressed at first from the limited
point of view of the project proposal (narrative and implied logframe to output / outcome
level). After that a single sub­section addresses the bigger question about how assistance
might have been provided more effectively, especially with a view to long­term impact.
This wider question takes us into the realm of the value and benefit of the project at the
higher logframe levels of purpose and goal.
Sixperspectives
Impactperspective
The following comments are drawn from a combination of the monitoring data and project
reports 9 and the field visits including the beneficiary / stakeholder interviews.
There is little doubt that the project achieved its objectives. It brought about an injection of
cash into the communities; it targeted that assistance at the poorest, including those unable
to work; it enabled the construction and rehabilitation of physical works; and it raised
awareness of some of the ideas and possible actions which constitute disaster risk reduction
(DRR).
9
Including a number of beneficiary case study reports and stories of transformation; one quarterly report
st
th
dated 1 June 2012; and the final report dated 11 September.
15
All beneficiary interviews concurred in the view that the right households had been
targeted, and that the poorest and most vulnerable had benefited from the project. No­one
found fault with the beneficiary selection process, although several of those interviewed
commented that the 27 days of cash payments was insufficient. This led to some useful
discussions about the hard decisions to be made when there are limited resources. In most
cases the cash had been spent, as intended, on food items. In a few cases recipients of cash
had spent some or all of it on other items such as medicines, school expenses and house
repairs following flood damage.
Physical works undertaken included canal cleaning, road repairs, repairs to water storage
reservoirs and wells, and construction of school perimeter walls. These activities each have
different impacts on the communities involved. Canal cleaning improves water conveyance
efficiency both for irrigation water and flood drainage. Although the project was designed
as a drought relief intervention, the cleaning of canals turned out to be very beneficial in
relieving flood flows in the rainy season of summer 2012. Road repairs improve access to
and from villages, although this is likely to have only limited impact on market access.
Rehabilitation of domestic water storage structures has immediate benefits to the whole
community. Construction of school walls has little or no discernible benefit in terms of
livelihoods or food security, but it is an activity which communities value, presumably for
reasons of status and pride in their surroundings.
The impact of DRR training was not prominent. Communities are well aware that they can
work to divert flood flows if they are not too sudden or too large. Keeping drainage
structures and flood­protection embankments in a good state of repair can provide some
protection to shelter and fields. In the case of drought – viewed by community members as
far more serious, more frequent, and more an ‘act of God’ – they see little that they can do
in practical terms. It may be that the DRR awareness / training carried out in the early
months of the project has some long­term benefit, but this is difficult to predict.
The greatest negative (unintended) impact of the project potentially is the consequence of
its exclusion of certain districts, villages, households or individuals. In principle this may
have given rise to discontent or conflict. Although this question was explored in beneficiary
interviews, we found no evidence of such negative impacts.
In summary, the project had a significant impact, in its own terms, as a short­duration
emergency response intervention to relieve food insecurity. In terms of the ratings
suggested in the TOR (Appendix A)10, impact is rated at 3.
10
“The Evaluation Team may wish to consider using the following four­point scale to score the project’s
achievements for each of the Key Areas: (1) the project makes no contribution to the aspect; (2) the project
makes a minimal contribution to the aspect; there are major shortcomings that must be addressed; (3) the
project makes an acceptable contribution to the aspect; there are shortcomings that could be addressed; (4)
the project makes a substantial contribution to the aspect.”
16
Stakeholderperspectives
The prevailing culture is characterised by a desire to please and an unwillingness to criticise
or allow loss of face. This makes it difficult for outsiders to get behind the positive remarks
made by beneficiaries and other stakeholders. This is not a situation unique to Afghanistan,
but nevertheless it was a barrier in this evaluation to considered and constructive critique.
Almost every comment made by beneficiaries and community representatives (CDCs and
DDAs) was positive, except when requesting more frequent or longer­duration assistance.
The implied criticism that the help was too little was expressed in the context of sincere
gratitude for the assistance provided.
Specifically, the male beneficiaries and community representatives interviewed requested
that in future interventions of this type a greater number of paid days (two or three times
the 27 days given) should be included, and they asked for the provision of tools, items such
as spinning wheels and protective clothing.
The comments just made, and related observations, lead this consultant to believe that
there is a significant risk of contributing to a culture of aid­dependence. Community
members clearly know what to ask for, and they are aware that other organisations may be
providing items which were not part of this project. Care will need to be taken in future to
avoid exacerbating this risk.
As far as targeting is concerned, Tearfund’s clear beneficiary selection criteria11, and the
transparent process followed in choosing beneficiaries led to acceptance of the decisions
made, at least according to the beneficiaries and the community representatives. Time did
not permit us to interview non­beneficiaries to confirm this point of view.
Tearfund has good relations with the Government­appointed District and Provincial
Governors, and with the community representatives and community members.
Stakeholder perspectives are rated as 4.
Outputperspectives
The outputs of the project were as follows:
�
�
�
cash paid to beneficiaries;
labouring work completed;
provision of training and awareness­raising on DRR.
11
Beneficiary Selection Criteria, no author and undated, and the Beneficiary Selection Criteria Terms and
Definitions, no author, undated.
17
The full extent of outputs delivered is set out in the project final report 12. The totality of the
outputs could not of course be confirmed in a short evaluation, but a sample of the work
done was briefly observed in each of the communities visited.
Table 2 sets out the achievements tabulated in the Final Report alongside the project
indicators from the narrative proposal. As is clear from the table, all three indicators were
substantially delivered. The small inconsistency in indicator 3, and the slightly lower than
planned US dollar payment per beneficiary, represent only minor deviations from the plan.
The non­labouring beneficiaries were mainly women (as intended), but with some males
included for reasons of infirmity, disability or other vulnerability.
Table 2 Project outputs compared to pre-project indicators
Indicator
1. The number of people employed
through cash­for­work (cfw)
activities.
2. The average USD amount per
person earned through the cfw
activities.
3. The number and percentage of
women employed through cfw
activities.
Achievement
1,750 in total of which 1,312 completing
work and 438 non­working.
Achievement %
100%
27 days per worker @ AFN250/d (total
of AFN6,750 or USD135.
93%13
438 non­working beneficiaries of which
328 female.
100%14
All work carried out intentionally employed ‘unskilled’ labour, and therefore used methods
and materials with which the workers were familiar. The consequences of this approach are
that the work can readily be repeated as necessary (eg as canals become silted up again or
roads deteriorate). However the nature of such works is that they lack permanence or near­
permanence. It remains to be seen whether the new structures which have been built and
the existing structures which were repaired will be maintained as needed in future.
The rating of project outputs (against pre­project indicators) is substantial, rating 4.
Processperspectives
Process can be considered in two parts: the processes and procedures followed in project
implementation (standards, communication, accountability, coordination); and those which
help to ensure project design and delivery constantly improves (learning and innovation).
12
Tearfund (2012) Programme Performance ­ Final Report. Emergency cash for work support to drought
impacted families – Jawzjan Province, Murdian and Khoja Doko districts. Award Number: AID­OFDA­G­12­
th
00017. Report date 11 September 2012.
13
Target amount was USD145.
14
The number of women beneficiaries was not explicitly stated in the project proposal, only the number of
non­labour beneficiaries (25% of the total). The male non­workers were disabled or otherwise especially
vulnerable.
18
The project team has worked effectively to deliver the project according to Tearfund’s
quality standards. Comments relating to these are included in Table 3.
Table 3 Delivery to Tearfund’s Quality Standards for Emergency Response
Quality Standards
Values
We are committed to living out our core values through our staff and
with our partners in relationships with all those with whom we interact.
Impartiality & Targeting
We are committed to reaching the most vulnerable and marginalised,
selected on the basis of need alone, regardless of their race, religion or
nationality.
Accountability
We are committed to being fully accountable to project participants,
communities, partners, supporters and donors for the work we
undertake and for the resources entrusted to us. We adhere to a
Supporters' Charter that sets out how we respond to our supporters'
needs, queries and complaints.
Sustainability
We are committed to empowering staff and partners and to seeing that
the work that we support has a lasting impact, being built on local
ownership and using local skills and resources.
Advocacy
We are committed to influencing key decision­makers to make and
implement policies and practices that work in favour of people who are
poor and vulnerable.
Children
We are committed to the development and protection of children,
carrying out activities that are child­sensitive in their planning, design
and implementation.
Gender
We are committed to seeing transformation through restored
relationships between men, women, boys and girls and ensuring equal
value, participation and decision making by all.
HIV
We are committed to working for an HIV free world by reducing the
vulnerability of poor communities to HIV and reducing its subsequent
impact.
Environment
We are committed to reducing our impact on the environment, to
assessing vulnerability to climate change and environmental
degradation and to working with local communities to ensure that we
support, not harm, the natural and socio­economic environment.
Disaster Risk
We are committed to reducing the risk of disaster by strengthening
local capacity and reducing vulnerability to hazards.
Conflict
We are committed to promoting peace and reconciliation, supporting
activities which impact positively upon situations of conflict and that
safeguard staff and project participants.
Technical Quality
We are committed to supporting projects that reflect the priorities of
those we seek to assist and that are guided by relevant technical
standards and good practices.
19
Comments
Tearfund’s values of compassion, truth,
courage and service are evident in the
entirety of the project.
The project deliberately and impartially
focused on the most needy.
Accountability is demonstrated in the
monitoring and evaluation undertaken in
the project, and the good relations with
local community representatives.
In projects of this type, lasting impact is
very difficult to achieve. This is addressed
later in the report.
There is little evidence that the project has
been used as an advocacy vehicle. There is
potential to do more here, especially with
donors.
It can reasonably be inferred that the
project has had direct benefits for
vulnerable children in poor households.
The project was designed and delivered in a
gender­sensitive manner within local
cultural constraints.
This was not explicitly addressed in the
project.
The project was a short­term response to
environmental change and hazard. Little
appears to be going on to understand the
wider processes of environmental change
and their impact on the poor.
A limited amount of DRR training was
carried out, but its impact was probably
insignificant.
There is no evidence that the project has
contributed to conflict, and it may have
delivered a positive impact in this area.
The technical quality of the project was
satisfactory, reflecting current good
practice.
The main standard delivered in this project, and communicated to beneficiaries was that of
Impartiality and Targeting. Accountability, both to donors and to beneficiaries, has also
been prominent. Regarding the latter, Tearfund has established procedures 15 for receiving
comments and requests, logging these and building them into future projects wherever
possible.
Communication and coordination with other stakeholders (local Government, UN agencies,
NGOs) appears to have been exemplary.
In conclusion, the process of project design and implementation appears to have been of a
high standard, rating 4 according to Tearfund’s assessment scale.
The second question however relates to whether the project was a vehicle for learning, and
whether any innovations carried out in the project can point the way to better design of
future projects. To some extent Tearfund is in a cleft stick in this regard. Projects of this
sort are designed to fit the requirements of donors, who may have rather fixed ideas about
what can and cannot be included, and what constitutes ‘good practice’. The power of
donors can tend to stifle innovation and experimentation.
On the other hand, it behoves organisations like Tearfund to deeply reflect on their
experiences in the field, learn from those experiences, and influence the debate about what
really constitutes good practice. It is inevitable that this reflection and learning tends to be
displaced by the urgency of project design, fundraising and implementation. However,
every effort should be made to avoid this. There is not a great deal of evidence that such
reflection and learning has taken place in this project, and there is a danger that the next
project will look rather similar. Learning should be leading to evolutionary (and occasionally
revolutionary) change in the way of working.
The rating of learning and innovation in the project is consequently rather low, at 2.
Resourceperspectives
The project budget and project expenditure were virtually identical. The approved budget
was USD399,696. About USD236,250 (60% of the total) was distributed as cash­for­work,
the remainder (40%) covering the costs of project implementation, management and
coordination. There was no significant deviation between the intended and actual use of
the funds.
It is difficult to benchmark the costs of delivery of this type of assistance, as local and
organisational context is such a strong determinant of operating costs. However, it is clear
from the literature that the costs of cfw generally compare very favourably with those of
15
Beneficiary feedback records
20
direct food distribution (for obvious logistical reasons). Even the otherwise helpful ICRC
Guidelines for Cash Transfer Programming 16 give no indication of reasonable levels of
programme costs in relation to cash disbursed. Is the 60% figure above high, low or
reasonable?
In 2009 GTZ carried out a review17 of a number of cfw interventions in northern
Afghanistan. The consultants involved pointed out that “… cost comparisons of in­kind
assistance (particularly food aid) with cash transfers, is a complicated task … virtually all
project evaluations that have included a cost­efficiency analysis have concluded that
judgment as to the superiority of cash versus in­kind assistance or vice versa depended on
the type of project and the social and economic environment in which it was implemented.”
This consultant draws the tentative conclusion that the present project is acceptably cost­
efficient. Management and administration costs were not excessive.
This short (six month) project was always going to be under time constraints and urgency.
The delays in signing the contract meant that the first four months were devoted to ‘soft’
activities, including beneficiary selection and DRR training. With hindsight, the investment
of significant time, especially in the beneficiary selection, may have been no bad thing.
However, the time then available to undertake the physical works was severely constrained.
In the event though, all planned work is reported to have been carried out – a total of
35,424 man­days of labour.
Project staff members were sufficient in number and competent to deliver the project
outputs.
The resource perspective is rated at 4.
Organisationalcapacityperspectives
It is difficult to point to specific evidence for conclusions about organisational capacity. The
fact that the project was delivered, as planned, in a difficult context which includes some
very remote communities, and despite delays in the signing of the contract, provides
indirect evidence of a strong organisational capacity.
Likewise, the implication of the effective delivery of the project is that the management,
communications and financial systems were adequate for the task.
Organisational capacity is rated 3.
16
International Red Cross and Red Crescent Movement (2007) Guidelines for Cash Transfer Programming.
ICRC http://www.ifrc.org/Global/Publications/disasters/guidelines/guidelines­cash­en.pdf
17
GTZ (2009) Cash for work. A contribution to the international debate on lessons learnt in northern
Afghanistan. http://www2.gtz.de/dokumente/bib­2009/gtz2009­0537en­afghanistan­cash­for­work.pdf
21
Thebiggerpicture
This project had a limited aim of relieving a short­term need for food security. It operated
through an efficient mechanism of cash­for­work, and it was implemented effectively. Most
of the questions listed in section 2 can be answered favourably. However the final question,
about lasting value, and better alternatives, remains.
In discussions with beneficiaries and community representatives it was clear that the pre­
project drought situation faced by those communities is now a regular and frequent
occurrence. Floods may be less frequent, and their severity depends a great deal on
location, but nevertheless they represent a real hazard. Droughts, floods, chronic poverty
and food insecurity now represent the new reality for the Murdian and Khaja Dukoh
Districts, as well as for many others in the Province. It is unlikely that the impact of this
project will be felt a few months from now. How can ‘first aid’ undergo a transition to real
healing?
There must come a time when emergency response is reserved for exactly that – response
to unusual, relatively infrequent events. Perhaps a more fundamental long­term
commitment is needed in order to address chronic poverty and food insecurity – a
programmatic response based on a sound understanding of the root causes of poverty and
food insecurity.
Although this comment lies strictly outwith the TOR of this evaluation, I would like to
suggest the following for Tearfund’s consideration: bring together in the same limited
geographic locations Tearfund’s operational capability in disaster response, its partner­run
long term interventions, and fully mainstreamed DRR work. Establish a long­term
foundation for those short­term emergency interventions which will continue to be needed
from time to time. Fully integrate a ‘menu’ of DRR and community development
interventions 18 with the capacity to change to emergency response mode when necessary –
with strict protocols about how that need is defined, what modalities of response (eg
payments / in­kind) are appropriate, and timing of reversion to development mode. Within
the long­term programming, really try to understand in depth what are the root causes of
the poverty and food insecurity experienced by communities in Jawzjan.
While acknowledging my ignorance of the detailed history of Tearfund’s presence in
Afghanistan, I believe this would represent a somewhat innovative approach, and one which
if well­designed, could effectively integrate long­term development and short­term
emergency responses.
18
Eg drainage and flood protection works, WASH activities, agriculture and livelihoods, off­farm employment.
22
3. Conclusions
C1. Regarding the evaluation and its methodology. The evaluation was very short, because
of budget constraints 19. The methodology was pragmatic and appropriate for a rapid
evaluation. It led to good open discussion, but it was difficult to get community
members and even project staff to think creatively and ‘out­of­the­box’.
C2. Regarding impact. The project made a substantial short­term difference to a significant
number of very vulnerable households. There was no evidence of negative impacts save
the suspicion that the project may have contributed to aid­dependency. Targeting
decisions raise dilemmas about how thinly to spread resources for optimum effect. The
impact of DRR training was probably rather limited in the short­term at least.
C3. Regarding stakeholder perspectives. The project was well­received, and Tearfund
clearly has good relations with local Government, community representatives and
individual beneficiaries. Beneficiaries requested more assistance of this type in future,
but with additional cash and the provision of tools and other goods in­kind.
C4. Regarding outputs. All planned outputs were delivered as planned. Cash payments
were a little lower than planned, for reasons unknown to the consultant. All work
involved unskilled labour, which has advantages in terms of inclusion, but disadvantages
in terms of longevity.
C5. Regarding process. Project implementation was carried out to high standards, with an
emphasis on impartiality, targeting and accountability. Project learning and reflection
was limited, with little likelihood of the project experiences contributing to innovation in
practice.
C6. Regarding resources. The tentative but reasonable conclusion is that the project was
cost­efficient. Time was sufficient, if constrained by delays in contract signature.
Human resources were adequate to the task.
C7. Regarding organisational capacity. The evidence of project delivery according to plan is
that systems were adequate for effective and efficient intervention, with strong
accountability. Work needs to be done on improving information and file­sharing
systems.
C8. Regarding the bigger picture. The project had a valuable short­term impact, but its
effect is unlikely to be sustained. The need which gave rise to the project remains.
Similar emergencies are likely to recur so frequently as to put in question the
appropriateness of the word ‘emergency’. Drought, flood and food insecurity are the
new normality.
19
The evaluation budget was less than 1% of the project budget.
23
4. Recommendations
In line with the findings and conclusions, only a small number of specific recommendations
are made. Six relate to projects of this type which no doubt will continue to be carried out
in future. Two relate to wider issues of programme effectiveness.
In future cfw projects:
R9. Types of work. Consider including other forms of work (not only unskilled labour),
including more permanent works involving skilled labour. In this way physical structures
may have longer lives, and beneficiaries may learn new marketable skills. This of course
has budgetary implications [Ref C4].
R10. Disaster risk reduction. Integrate disaster risk reduction training and action
planning with emergency response cfw, devoting sufficient time to this activity to fully
embed it in the communities involved [Ref C2].
R11. Donor­dependence. Reflect on the extent to which projects of this type are
exacerbating donor­dependence. Develop detailed guidance on limiting that negative
impact [Ref C2].
R12. Spreading thin or digging deep? Reflect on the dilemmas about how limited
budgets should be spent in such emergency response interventions. This project
assisted 10 villages out of 32 in Murdian and 6 out of 19 in Khaja Dukoh. Should it have
gone deeper in fewer villages, or spread even more thinly? Develop a position on this
[Ref C2].
R13. Information systems. Develop systematic information and filing sharing
arrangements so that all who need project information can access it easily and quickly
[Ref C7].
R14. Evaluation budget. Budget more for future evaluations – 2­5% of total project
budget for a project of this size, depending on the strategic importance of the evaluation
[Ref C1].
In regard to future programming in Jawzjan Province
R15. Learning and influencing. Introduce a systematic process for project learning,
documentation of learning and influencing of other stakeholders (Government, donors,
other NGOs). This should include mid­term and end­of­project reviews and exchanges
of experiences within country [Ref C5].
R16. Integration of development and emergency work. Consider restructuring the work
in Afghanistan in order to fully integrate long­term development with short­term
emergency response interventions within limited geographical areas. Address root
causes of food insecurity alongside providing emergency assistance [Ref C8].
24
Appendices
25
AppendixATermsofreference
Project Title:
Country:
Emergency cash­for­work support to drought impacted families: (Murdian and
Khoja Doko districts, Jawzjan Province)
Afghanistan
Consultant:
Prof. Dr. Richard Carter
Programme Officer:
Bryony Norman
Child Protection Clearance
Confirmation:
TBC
Date:
Between 21 September and 07 October 2012 (two evaluations to be
completed within this timeframe by the same consultant – see separate ToR
for second assignment. Total of 14 days will be required for the assignments
to be completed, including travel days and weekends).
BUDGET
�
�
Overall Budget, Payable from which Budget Line/Code: Budget Code to be determined from OFDA
project budget (Evaluation budget line).
Breakdown by Fees, Travel, Subsistence, Other Costs: To be confirmed after discussion with evaluator
and details of daily rate agreed.
Expense Caption
External Evaluator (based on daily consultancy fee of £ 300.00, approximately $475.00
– for 5 days, inclusive of four working days, two weekend days and one travel day)
Ground transport (national flights @ $100.00 each, and vehicle rental at $100.00)
International Travel and Visa Costs
Subsistence (food, accommodation) ($100.00)
National Monitoring Support Staff ($100.00 per day each for x1 staff for x 1 days)
Total budget
Amount $
$ 2,375.00
(£1,500.00)
$ 300.00
$ 600.00
$ 100.00
$ 100.00
$ 3,475.00
� Total amount requested from Tearfund: $ 3,475.00
NB This section above can then be detached from the rest of the TOR when sending it to the consultant if
budget involves financial information that shouldn’t be revealed to the consultant (e.g. payments to other
consultants, contingency amounts etc.)
BACKGROUND
Programme / Project Title: Emergency cash for work support to drought impacted families.
Background to Programme / Project: Afghanistan has been in a rebuilding phase since early 2002, which has
seen significant improvements in the areas of health accessibility, education attendance, and an increasing
GDP. However, the rural poor, more specifically, those that are outside economy created by the military effort
and large­scale development intervention, remain vulnerable to conflict and natural disasters. 2011 saw
significant below average rainfall, which led to a slow­onset drought across 14 provinces in Afghanistan, from
Herat province bordering the Islamic Republic of Iran in the west to Badakhshan province in the east bordering
China. As the large impact of the drought became apparent, the World Food Programme (WFP) organised an
Emergency Food Security Assessment (EFSA) report to take place in the 14 provinces. The general finding of
26
20
this report was that 2.8 million people would require food assistance due to the drought. The background of
the problem is described in the EFSA. “Afghanistan experiences droughts that are cyclical in nature. Hence,
2010/11 agricultural season was one of such drought years. The 2010/11 was characterized by a poor start of
the rainfall season in November and December 2010 for the planting of winter crops, below normal winter
precipitation (winter provides 50% of the annual precipitation) and below normal April to June rainfall. The
poor precipitation amount and distribution resulted in a 17 percent reduction in 2011 cereal production
compared to average, poor water and pastures especially in the 14 provinces in northern, north­eastern,
central highland and north­western areas of the country.”
Main problem: Loss of income from loss of harvest
The lack of rainfall before and during the growing season has led to partial and total loss of harvest in all rain­
fed agricultural land and near total loss of irrigated land in Jawzjan. This means that farmers received minimal
income or food from their fields.
Loss of secondary income sources and livestock
Small­scale farmers (with less than 2 jeribs) and landless households often supplement their income by
working as casual wage labor for larger farmers. However, given the decrease in harvest expected even in
irrigated land, the demand for casual wage labor has also decreased drastically, and wages have dropped as a
result by 30­40%. However, what is even more significant than the lower wages was that the days available to
work dropped to a very low level. Furthermore, livestock have been sold at distressed prices, thereby reducing
income for herders.
As there are few options available to replace the lost work opportunities, the EFSA noted that employment
opportunities were expressed as a high priority.
Together with UNOCHA and OFDA, NGOs identified cash­for­work (CfW) as an appropriate activity to rapidly
and effectively provide life­saving assistance to the most vulnerable households in the most affected regions of
northern Afghanistan. The project, implemented by Tearfund, was initiated in January 2012 with a total budget
of $ 394,687 USD.
Main Objective: Injection of cash in order to secure emergency food and water needs of most vulnerable
households
Activity: Cash­for­work activities
After the selection of communities, projects and households was finalized in a participatory manner
with the communities (see next section), the identified households received 27 days of paid labour on
the identified projects at a rate of $7.17 per day. Of this amount $5.37 was received by the labourer,
while $1.8 was held back and redistributed to households who could not participate in the CfW projects
(such as female headed households, widows and the disabled). 1,312 people have been targeted
through this project intervention. Further details of the project (including specific project intervention
locations, beneficiary numbers per location, and individual project objectives and indicators), can be
found in the project proposal, logical framework and budget (attached: Annexes 1, 2 and 3).
Summary Findings of Previous Reports and Evaluations: To date, one mid­term narrative report (detailing
progress made against each of the individual project objectives and indicators), has been completed and
submitted to OFDA for review (May 2012). This report can be found in the attached annex file (Annexes 4) and
further reports that are due to be completed ahead of the evaluation schedule will be circulated to the
20
nd
EFSA 2 phase, page 6
27
selected evaluator (final narrative and financial reports). Other than this, there have been no prior evaluations
undertaken of this project, though individual examples of impact monitoring stories and reports can be shared
with the selected evaluation consultant upon request.
Partner Profile and History (Tearfund Overview): Tearfund is a humanitarian relief and development
organisation, with approximately 40 years of experience. Tearfund’s vision is to transform the lives of millions
of the world’s poorest people, in a positive and sustainable way. Tearfund works to eradicate poverty through
integrating the approaches of community development, disaster management and advocacy. Tearfund
currently funds projects in over 50 countries worldwide (particularly in Africa, Asia and South America – and
also across Eastern Europe and the UK), tackling the major issues which keep people in poverty. Tearfund aims
to work through local partner organisations, in the first instance, to release people from material poverty but
in some instances a team of highly qualified humanitarian professionals is sent to set up a direct operational
programme in response to acute humanitarian needs. This is the case with Tearfund’s presence in Afghanistan.
Tearfund’s Disaster Management Team has been operational in Afghanistan since 2001, in response to both
natural disasters and conflict. Tearfund first opened a programme office in Afghanistan in Kandahar. Whilst a
project office remains in this province, the programme office was relocated to Kabul. Other project offices
have historically been located in Faryab, Kapisa and Parwan provinces whilst they continue to be located and
manage programmes in Jawzjan and Kandahar.
The major sectoral foci for the Afghanistan programme include water, sanitation and hygiene (WASH), public
health promotion (PHE, including a specific focus on child­focused health education), disaster risk reduction
(DRR), and emergency response. Other sectoral foci include nutrition and livelihoods.
The programme employs approximately 85 staff (including expat and national) and has an annual budget of
roughly £2 million. The programme is currently assisting approximately 6,000 households. Previous projects,
predominantly in Kandahar province, have included winterization projects and distributions, canal rebuilding,
schools reconstruction, water and sanitation and health education delivered via radio. Tearfund’s donor
portfolio within Afghanistan for the period 2011/12 included: BRPM, the UK Department for International
Development (DFID), the Humanitarian Innovations Fund (HIF), Tearfund Trust Fund, the UN Office for
Coordinating Humanitarian Affairs (OCHA), the UN World Food Programme (WFP), and US OFDA (funding this
project).
Current Activities: Cash­for­work initiative will be implemented with 1,312 individuals and will focus on the
following labour activities:
o
o
o
o
o
Rehabilitation of or improvements to secondary/feeder roads
Rehabilitation of water storage points and their catchment areas
Repairing or improving existing wells
Repairing irrigation canals
Building fences or surrounding walls for schools
Households that are not eligible for CfW activities (e.g. female­headed households, child­headed households,
disabled­headed households) will benefit from the cash­grant but will not be required to complete manual
labour.
Summary of Region/Country Strategy: Currently Tearfund is implementing projects in only Jawzjan and
Kandahar. In order to maintain the ability to scale up in times of emergency and to respond to changing needs
in different locations Tearfund would like to be operating from at least three, optimally four bases, that are
different ethnically, geographically and security­wise, enabling us to maintain minimum levels of activities
when any of the above factors require temporary suspension of a particular project or field office. For 2012­13,
therefore, the programme is hoping to establish a base in Bamyan and is currently seeking further funding
28
opportunities to support this. It is also seeking to expand operations into Balkh province, from its base in
Jawzjan.
In 2012­13, Tearfund will continue to focus on strengthening the capacities of communities, various
government departments and other NGOs, primarily in the areas of WASH, DRR, emergency response,
nutrition and livelihoods. Additionally, the programme is committed to maintain high standards in beneficiary
accountability and monitoring, review and evaluation practices. Tearfund will use remote research that has
been undertaken in 2011­12 (which focuses on accountability and monitoring practices for projects in volatile,
highly insecure operating environments) in order to ensure that beneficiary accountability and project
monitoring is promoted and well achieved in its medium­to­high, remotely managed project locations.
How the Need for the Requested Assignment Arose: The proposal for this project (funded by OFDA) stated
that an ‘independent final evaluation’ would take place, costs for which were included in the budget. This
assignment would satisfy this commitment.
PURPOSE
The aim of this assignment is to conduct an evaluation of the project: Emergency cash for work support to
drought impacted families, running from 22 January 2012 – the 21 June 2012.
An evaluation is an integral part of the project cycle. If carried out well, an evaluation should increase
transparency and allow all stakeholders to be able to influence the direction and emphasis of the project.
An evaluation therefore has two main functions: to strengthen accountability and to increase learning.
The focus of DMT Project Evaluations is to make assessments for each of the following Key Areas:
Key areas, scope and questions
N.B. The word project has been used throughout, but could be interchanged for programme.
IMPACT PERSPECTIVES
Goal of the project, e.g. lives saved, morbidity reduced, livelihoods protected
What change has taken place during the project?
Which changes are attributable to the project?
Are the changes in line with the goal of the project?
Were there any unintended changes, positive or negative, because of the project?
STAKEHOLDER PERSPECTIVES
Outcomes of the project; attitudes and behaviour of beneficiaries, partners, churches, staff, local governments,
UN, donors, etc.
What do the beneficiaries think of the project? Its relevance, appropriateness and outcomes? Are the outcomes
sustainable?
Have beneficiaries adopted new or changed any behaviour as a result of the project?
What do other primary and secondary stakeholders (e.g. staff, community leadership, local government officials,
non beneficiaries, etc.) think of the project? Were the most vulnerable reached? Was the targeting appropriate?
How do beneficiaries and other stakeholders describe the quality of relationships with project staff?
(NB Note beneficiary views should be given more prominence than those of other stakeholders)
29
OUTPUT PERSPECTIVES
Quality of outputs completed; Results achieved
To what extent were the planned outputs achieved?
Were outputs / deliverables of an appropriate technical quality?
PROCESS PERSPECTIVES
Needs, vulnerability and capacity assessments completed, conflict assessment, management, coordination,
standards.
To what extent has the project / programme outworked Tearfund’s quality standards? Which quality standards
were prioritised and was this selection appropriate?
How effectively were the standards communicated to the beneficiaries?
Were appropriate systems of downwards accountability (participation, information sharing and feedback), put in
place and used by project participants? Did the feedback received shape project design and implementation?
Was new learning being captured and acted upon during the project implementation? If yes, how and what? If no,
why not?
Were there any identifiable innovations developed that could be repeated in the future?
Were the initial assessments of a good quality and based on strong beneficiary participation?
How has the project been coordinated with the activities and priorities of other agencies and organisations
(including local and national government and UN)?
RESOURCE PERSPECTIVES
Budget, funding, gifts in kind, financial management, cost efficiency, cost control, people and time.
Was the budget and available financial resources realistic for the achievement of the intended objectives?
Were the funds used as stated?
Was the project / programme cost effective?
Was enough time allowed for the achievement of the intended objectives?
Were there enough staff, of appropriate competency, for the achievement of the intended objectives?
ORGANISATIONAL CAPACITY PERSPECTIVES
Leadership, strategy and policy, human resources and training, accountability, knowledge and learning, research
and innovation, capacity development, partnerships and networks.
Was the project design and implementation informed by learning from previous experience?
21
Was there an appropriate system of management and communication in place to support project staff ?
22
Were appropriate financial systems in place ? (e.g. did project management have adequate financial information to
make good decisions?)
23
Was there an appropriate logistics system in place ? (e.g. did the procurement process work in a timely manner? )
21
This does not infer the need for an audit, rather, did the management and communication system have a positive or negative bearing on
the performance of the project /programme
22
This does not infer the need for a financial audit, rather, did the financial systems in place have a positive or negative bearing on the
performance of the project / programme? For example in supplying project managers with accurate and up to date financial information.
30
CONCLUSIONS
Was the project efficient? (comparison of inputs to outputs)
Was the project effective? (comparison of outputs to impacts)
What were the key lessons learned? What should be repeated in similar projects in the future? What should not be
repeated in similar projects in the future/
METHODOLOGY
The methodology for this evaluation will be negotiated between the independent evaluator and Tearfund DMT
Afghanistan. The methodology will include:
�
�
�
�
�
�
A gender­aware, participatory approach.
A review of pertinent documentation, held by Tearfund DMT Afghanistan field staff.
Field visits to the relevant project sites in Kandahar province.
Interviews with project beneficiaries and community representatives / members; relevant field based
staff; key officials in co­ordinating agencies, and; local and/or central government representatives.
Interviews with individuals may be complemented by discussions with groups of beneficiaries (focus group
discussions).
Adherence to the Red Cross/Red Crescent NGO Code of Conduct, SPHERE and HAP standards.
SCHEDULING
The evaluation will take place immediately following the project completion, being initiated on or shortly after
18 September 2012.
It is recommended that a team of national monitoring staff be selected by the evaluator in order to
complement field visits to project implementation areas in Kandahar province.
It is anticipated that a total of 9 working days will be sufficient to carry out this evaluation (two international
travel days):
�
�
�
�
�
�
�
2 days international travel (including any preparation required for the evaluation);
0.5 days preparation (developing specific methodology, tools and templates);
2.5 hours briefing (at Kabul office – including interviews with Kabul­based personnel);
1 full day for travel to and from Kandahar (Kabul to Kandahar return);
2 full days in Kandahar (subject to security – national monitoring team required to support evaluation
visits. Must be a skilled team of monitoring staff selected);
3 full days for data analysis and report writing.
2.5 hours for report finalisation (after feedback from Tearfund).
MANAGEMENT OF VISIT
Tearfund DMT Afghanistan is commissioning and approving the work. The consultant should refer to
Tearfund’s Grants and Information Officer, when recruited, to resolve issues as they arise.
Responsibility for practical arrangements, travel arrangements, hotels etc. This will be confirmed in discussion
and agreement with the evaluator.
23
This does not infer the need for an audit, rather, did the logistics systems in place have a positive or negative bearing on the
performance of the project / programme? For example were materials of the required specification provided in a timely manner?
31
EXPECTED OUTPUT
The expected output of this assignment is a report (in Tearfund standard reporting format: please refer to the
Consultants’ Briefing Pack) with the following sections:
�
�
�
�
�
�
�
Executive Summary (no more than four A4 sides)
Introduction / Background
Methodology
Context Analysis
For each Key Area, a section in the form
o Findings
o Conclusions
o Assessment
Specific Actionable and Prioritised Recommendations
Annexes (indicative)
o Terms of Reference for the Evaluation
o Profile of the Evaluation Team
o Evaluation Schedule
o Protocols for the Evaluation
o Documents consulted during the Evaluation
o Persons participating in the Evaluation
o Field data used during the Evaluation, including baselines
o Bibliography
For each of the Key Areas outlined under ‘Purpose’, the Evaluation Team is required to make a clear statement
of the Team’s assessment of the project’s achievements. The Evaluation Team may wish to consider using the
following four­point scale to score the project’s achievements for each of the Key Areas:
1
2
3
4
the project makes no contribution to the aspect;
the project makes a minimal contribution to the aspect; there are major shortcomings that must
be addressed;
the project makes an acceptable contribution to the aspect; there are shortcomings that could be
addressed; or
the project makes a substantial contribution to the aspect.
INTENDED USE OF THE EXPECTED OUTPUT
An evaluation is not useful if the recommendations and lessons learnt are left on the shelf, not being read or
utilized. To ensure that these recommendations and lessons are not ‘lost’, the programme staff and UK Team
are required to respond to each evaluation. It is the responsibility of the programme to address those
recommendations that relate directly to the programme, by drawing up a management response and action
plan. It is the responsibility of the UK team to ensure transferrable lessons learnt and recommendations are
captured and disseminated as part of Tearfund’s evaluation and learning system.
EVALUATION OF CONSULTANCY
Tearfund’s Programme Director – Sudarshan Reddy Kodooru, Area Coordinator for Kandahar – Mannu Pereira,
Grants and Information Officer – Marjorie Orotin, and Programme Officer – Bryony Norman, will review and
comment on the report within 10 working days of report’s submission.
32
AppendixBItinerary
Date (2012)
Tuesday 18th September
Wednesday 19th September
Thursday 20th September
Friday 21st September
Saturday 22nd September
Sunday 23rd September
Monday 24th September
Tues 25th – Fri 28th September
Sat 29th – Sun 30th September
Monday 1st October
Activity
Travel to Kabul via Dubai.
Arrive in Afghanistan (approximately 5pm). Security
briefing.
OFDA Evaluation Day One
Preparation (developing methodology and tools; briefing
at Kabul office.
OFDA Evaluation Day Two
Continuation of preparation, studying documents, evening
flight to Mazar­e­Sharif. Briefing with Area Coordinator.
OFDA Evaluation Day Three
Full day in the field in Murdian District, Jawzjan Province.
OFDA Evaluation Day Four
Half day in field in Khaja Dukoh District, Jawzjan Province.
Drive back to Mazar in afternoon.
OFDA Evaluation Day Five
Drafting evaluation report. Interview with DRR Adviser.
BPRM 3 Evaluation, Kandahar.
3 full days in Kandahar.
BPRM Evaluation Days Four and Five
Drafting BPRM 3 evaluation report.
Travel back to UK via Dubai.
33
AppendixCSemi-structuredinterviewguidelines
The interviews should be carried out with women who received assistance from the project, either through
cash­for­work or a direct cash grant (with no work). Please try to find examples of both of these categories.
The emphasis in these interviews should be about quality of information, not quantity. A total of 6­8
interviews over two days will be sufficient if the quality is good.
Explain to your interviewee that we are trying to understand what difference, if any, the Cash­for­Work / Cash
Grant project had on those who received money. It is to enable us to do better in future if we engage in
similar activities. Please tell the interviewer that they will not be identified by name in any report.
Please ask the interviewee the following factual questions:
1.
2.
3.
4.
5.
Name
Approximate age
Household members – relationship and age of each one. Are you the head of household?
How is your household provided for – in terms of work, sources of income?
Did you work, or was the cash granted without work?
Please ask the following conversational questions (take time, and ask the interviewee to explain their
responses so that you the interviewer really understand. Try to write down some of the things people tell
you):
6.
Please tell me about your situation prior to receiving this assistance.
� Probe about the interviewee’s poverty / wealth, the drought which led to the assistance, the
consequences of that drought for her and her community, anything else she can say about the
nature and level of her need at the time.
7. Please explain how you were selected for inclusion in this assistance.
� Probe about her understanding of how she was selected and by whom, what was her role in
getting included.
8. Are there others who were in a similar situation to you, who were not included?
� Probe her feelings / perceptions about her need in relation to that of others. Was she one of the
worst off? Were there others who got left out? If so, why?
9. What difference did the money make to you and your household at the time of the project?
� Probe about what she spent it on, the extent to which it met her needs at the time, whether it
saw her through until her next income of cash or food from elsewhere.
10. Has the money made any lasting difference to you? If so what?
� Probe about any investments or savings she has made, any changes in her poverty / wealth.
11. Were other things done in the project which may help you to avoid such need again in future?
� Probe about the work done by men in the project and whether the assets which they worked on
have made any difference, also about any work with the community more generally to limit the
effects of droughts, floods or other disasters.
12. How do you think Tearfund could do this sort of project better in future?
� Probe about timing (was the help too soon or too late, did it finish at the right time), the work
component (was that useful?), the use of cash (as opposed to vouchers, direct food provision, or
other items), the relative importance of short­term relief versus longer­term assistance.
34
AppendixDConsultant’sCV
Profile
Richard Carter is a water sector development professional with over 35 years’ experience, and with significant
expertise in project and programme evaluations. He has extensive knowledge of the natural and social
science, engineering and management of water resources for rural and urban water supply and irrigation. He
has managed and undertaken institutional development and capacity­building activities, consultancies,
research projects, and education and training programmes in many aspects of the water sector. He has acted
as adviser to numerous NGOs, UN agencies, bilateral and multilateral agencies. In his higher education role he
supervised many successful MSc and PhD students. He has published more than 100 papers and reports in the
field of water development in low­income countries.
Richard has a strong appreciation of the place of context and complexity in the achievement of outcomes and
impacts of development interventions. He works in an interdisciplinary style and values multiple methods in
deriving evidence and learning from projects and programmes. His field experience has focused especially on
sub­Saharan Africa.
Richard is currently a consultant and adviser to the DFID­funded SHARE (Sanitation and Hygiene Applied
Research for Equity) project led by the London School of Hygiene and Tropical Medicine, and the Gates
Foundation funded project to deliver Total Sanitation at Scale in Nigeria. He is a retained consultant to the
international NGO WaterAid.
Richard is an academically rigorous practitioner, technical adviser and experienced evaluator with excellent
written and oral communication and presentation skills.
Highlighted Projects
Head of Technical Support, WaterAid (July 2009 – June 2012)
Richard established and guided the technical support function in WaterAid, including the introduction of Joint
(Country) Technical Reviews to complement full Country Programme Evaluations. In this position he led the
delivery of thematic framework documents on Sustainability, Sanitation, Urban WASH, Water Security, and
Hygiene. He prepared internal guidance on post­implementation monitoring of WASH interventions. He
continues as part­time consultant to WaterAid.
Team Leader: Evaluation of WaterAid’s Country Programmes in Malawi (2007), Zambia (2009) and Uganda
(2010)
Led teams comprising international and national consultants and programme staff, evaluating country
programme portfolios of water, sanitation and hygiene promotion work in rural and urban contexts.
Team Leader: Joint Technical Review of WaterAid’s Tanzania Country Programme (2011)
Designed the framework for Joint Technical Reviews and led a joint UK­Tanzanian team to review the WaterAid
Tanzania Country Programme.
Team Leader: Evaluation of Medair’s WASH programme interventions in Pader and Kaabong Districts, Uganda
(2008)
Undertook evaluations of humanitarian relief WASH interventions in IDP camps of Pader District and insecure
chronic poverty contexts in Karamoja.
Team Leader: Evaluation of Uganda Water and Sanitation (NGO) Network (UWASNET)
Led a team of Ugandan consultants in the evaluation of Uganda’s WASH sector NGO network for Government
of Uganda and Water and Sanitation Program of the World Bank.
Project Director: Landscaping of Water and Sanitation Technologies and Approaches (Bill & Melinda Gates
Foundation), 2006­07
Led a team from Cranfield University, IRC and Aguaconsult in identifying technologies and approaches
presenting opportunities for sustainable interventions at scale in water, sanitation and hygiene.
Team Leader: SOCODEP Project Evaluation, Ethiopia, IFAD (2006)
Led multi­disciplinary team of three Ethiopian consultants in the evaluation of the Southern Region
Cooperatives Development and Credit Programme.
35
Team Leader: Study of water well drilling costs, Ethiopia, WSP (2005­06)
Led team of four Ethiopian consultants in an in­depth study of water well drilling costs in Ethiopia, with
comprehensive recommendations on means of cost­saving.
Team Leader: Evaluation of Special Country Programme II, Ethiopia, IFAD (2004­05)
Led international team evaluating the SCP II Project. This project undertook small­scale irrigation
modernisation, catchment conservation and complementary agricultural support activities in pursuit of food
security.
Team Leader: various project and programme evaluations (pre­2004)
Undertook evaluations of WASH and training programmes in Burkina Faso (1985), Ethiopia (1990), India (1985
and 1988), Tanzania (1985 and 1988) and Uganda (1994, 1997, 2001, 2004 and 2004) for British Council,
Busoga Trust, Concern Universal, DanChurch Aid, European Commission, FAO and Tearfund.
Selected Publications
Hunter P R, MacDonald A M, Carter R C (2010) Water Supply and Health. PLoS Med 7(11): e1000361
doi:10.1371/journal.pmed.1000361
http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1000361
Carter R C, Harvey E, Casey V (2010) User financing of rural handpump water services. IRC Symposium: Pumps,
Pipes and Promises, The Hague, November 2010. http://www.washcost.info/page/1066
Carter R C, Danert K, Chilton J and Olschewski A (2010) Siting of Drilled Water Wells: a Guide for Project
Managers. RWSN Field Note 2010­5, June 2010.
http://www.rwsn.ch/documentation/skatdocumentation.2010­12­03.0768183939/file
Danert K, Luutu A, Carter R C and Olschewski A (2010) Costing and Pricing: a Guide for Water Well Drilling
Enterprises. RWSN Field Note 2010­6, June 2010.
http://www.rwsn.ch/documentation/skatdocumentation.2010­07­08.6754105740/file
Carter, R C and Parker, A (2009) Climate change, population trends and groundwater in Africa. Hydrological
Sciences Journal 54 (4), 676­689, August 2009.
http://www.tandfonline.com/doi/abs/10.1623/hysj.54.4.676?journalCode=thsj20
Federal Democratic Republic of Ethiopia (2008) Southern Region Cooperatives Development and Credit
Programme Final Evaluation, Nov 2008, IFAD. Report No. 1909­ET. Carter R C, Assefa M, Asafaw T,
Gebremariam A. http://www.ifad.org/evaluation/public_html/eksyst/doc/prj/region/pf/ethiopia/socodep.pdf
Carter, R.C. (2006) What the Dickens can science and technology offer Africa? A tale of two villages in east
Africa, Science in Parliament, 63, 1, 26­27.
Carter, R.C. (2006) Ten­step guide towards cost­effective boreholes: case study of drilling costs in Ethiopia.
WSP Field Note, October 2006. http://www.rwsn.ch/documentation/skatdocumentation.2007­06­
04.3136351385/file
Federal Democratic Republic of Ethiopia (2005) Special Country Programme Phase II Interim Evaluation, Apr
2005, IFAD. Report No. 1643­ET. Carter R C, Gebremariam A, Danert K, Amede T, Hiwet M.
http://www.ifad.org/evaluation/public_html/eksyst/doc/prj/region/pf/ethiopia/scp.pdf
Further information
Chair, Rural Water Supply Network (RWSN) (2011­date).
Visiting Professor, Cranfield University (2009­date).
External Examiner at Universities of Birmingham, Cambridge, East Anglia, London, Loughborough, Newcastle,
Open University and Southampton (various dates).
Director, DEW Point (DFID Resource Centre for Water and Sanitation, Environment and Climate Change) (2007­
2009).
Trustee, the Humanitarian Centre, Cambridge (2008­date).
Editor, Waterlines Journal (2008­date).
36