External Evaluation of:
Kandahar, Jawzjan and Faryab
Integrated and Sustainable Services
for Returnees and Host Communities
A final and independent project evaluation
Table of Contents
1. Glossary of Acronyms .................................................................................................... 3
2. Executive Summary ....................................................................................................... 4
3. Introduction .................................................................................................................... 5
3.1. Partner and project background.................................................................................. 5
3.2. Purpose of evaluation ................................................................................................. 6
4. Methodology................................................................................................................... 6
4.1. Tools........................................................................................................................... 6
4.2. Beneficiary Selection .................................................................................................. 7
4.3. Record of Stakeholder Participation............................................................................ 9
5. Context Analysis .......................................................................................................... 10
5.1. Security .................................................................................................................... 10
5.2. Cultural ..................................................................................................................... 10
6. Evaluation of each Key Area: Findings, Conclusions and Assessment......................... 11
6.1. Impact Perspectives ................................................................................................. 11
6.1.1. Findings ................................................................................................................ 11
6.1.2. Conclusions .......................................................................................................... 13
6.1.3. Assessment .......................................................................................................... 13
6.2. Stakeholder Perspectives ......................................................................................... 14
6.2.1. Findings ................................................................................................................ 14
6.2.2. Conclusions .......................................................................................................... 15
6.2.3. Assessment .......................................................................................................... 16
6.3. Output Perspectives ................................................................................................. 16
6.3.1. Findings ................................................................................................................ 16
6.3.2. Conclusions .......................................................................................................... 16
6.3.3. Assessment .......................................................................................................... 19
6.4. Process Perspectives ............................................................................................... 19
6.4.1. Findings ................................................................................................................ 20
6.4.2. Conclusions .......................................................................................................... 21
6.4.3. Assessment .......................................................................................................... 21
6.5. Resource Perspectives ............................................................................................. 21
6.5.1. Findings ................................................................................................................ 21
6.5.2. Conclusions .......................................................................................................... 22
6.5.3. Assessment .......................................................................................................... 23
6.6. Organisational Capacity Perspectives....................................................................... 23
6.6.1. Findings ................................................................................................................ 23
6.6.2. Conclusions .......................................................................................................... 24
6.6.3. Assessment .......................................................................................................... 24
7. Specific Actionable Recommendations......................................................................... 24
8. Annexes ....................................................................................................................... 26
8.1. Annex 1: Data and Analysis ...................................................................................... 26
8.2. Annex 2: Evaluation Terms of Reference.................................................................. 27
8.3. Annex 3: Profile of the Evaluation Team ................................................................... 28
8.3.1. Profile of Lead Evaluator ....................................................................................... 28
8.3.2. Profile of National Surveyors ................................................................................. 28
8.4. Annex 4: Evaluation Schedule .................................................................................. 30
8.5. Annex 5: Documents Consulted During the Evaluation ............................................. 31
8.6. Annex 6: Photographic Evidence .............................................................................. 32
8.7. Annex 7: List of Documents Reviewed On-Site......................................................... 33
1. Glossary of Acronyms
Acronym
AC
ACG
ANP
ANSO
AOG
BPRM
BSF
CDC
CLBS
DMT
DoAIL
DoRR
DRR
FGD
HH
HR
IDP
IED
IMF
ISAF
M&E
MoE
MRRD
NGO
ToR
UNHCR
VRF
WASH
3
Abbreviated Word(s)
Area Coordinator
Armed Criminal Group
Afghan National Police
Afghan NGO Safety Office
Armed Opposition Group
US Bureau for Population, Refugees and Migration
Bio Sand Filter
Community Development Council
Community Livestock Banking System
Disaster Management Team
Department of Agriculture, Irrigation and Livestock
Department of Returnees and Refugees
Disaster Risk Reduction
Focus Group Discussion
Household
Human Resources
Internally Displaced Person
Improvised Explosive Device
International Military Forces
International Security Assistance Forces
Monitoring and Evaluation
Ministry of Economy
Ministry of Rural Rehabilitation and Development
Non-Governmental Organization
Terms of Reference
United Nations High Commissioner for Refugees
Voluntary Repatriation Form
Water, Sanitation and Hygiene
2. Executive Summary
(3.1)1 Tearfund’s Disaster Management Team (DMT) Afghanistan, active in the country since
2001 and currently having a particular focus on WASH (water, sanitation and hygiene),
nutrition, and livelihood activities, carried out a one year multi-sector project entitled
Kandahar, Jawzjan and Faryab Integrated and Sustainable Services for Returnees and Host
Communities between September 2010 and September 2011. The project, funded solely by
the US Bureau for Population, Refugees and Migration (BPRM), targeted 2,300 returnees,
IDPs and vulnerable host community members in three provinces (Jawzjan, Faryab and
Kandahar) with a goal of aiding in resettlement and helping to establlish a sustainable
healthy standard of life. Specific project objectives were developed relating to safe drinking
water, basic hygiene knowledge and practice, livelihood opportunities, nutrition through
vegetable gardening and animal husbandry, and community self-reliance.
(3.2) This report is the result of the end-of-project evaluation, funded by BPRM and
commissioned by Tearfund DMT, to assess the effectiveness of this project from
perspectives of impact, stakeholders, output, processes, resources and organisational
capacity.
(4) The methodology employed included tools of individual interview, household interview
and observation, focus group discussion (FGD), material/process review and document
review. Care was taken to maximize randomization with regards to beneficiary selection for
household interviews, although time restraints, security restraints, weather limitations and
other unforseen factors severely affected randomization. A total of 270 community members
(133 male and 137 female), 18 national staff (13 male and 5 female), and 6 international staff
(5 male and 1 female) participated in the evaluation.
(6) In assessing the six key perspectives, a four-point scale was used, with 1 indicating the
project made no contribution to the aspect and 4 indicating substantial contribution.
(6.1) Impact was rated as a 3. Findings revealed that changes related to 4 of the 6 project
objectives were strongly in line with the goals and initial design of the project. These findings
included evidence of improved basic hygiene knowledge and practice in more than 2 areas
by more than 20%; increased yarn production (by nearly 5 times), and therefore increased
income, for beneficiaries who received spinning wheels; and greater than 60% reporting of
increased vegetable and animal product consumption. Shortcomings noted were related to
water supply interventions and plans for sustainability; however, those were seen as issues
of output quality, a separate perspective discussed later in the report.
(6.2) Stakeholder perspectives were rated as a 4. Overall, feedback from stakeholders on all
levels, from government down to non-beneficiary community members, revealed positive
opinions about the project, the project staff and the impact of the project on the communities.
Evidence of behaviour change was noted. Targeting of beneficiaries was appropriate and
successfully done in cooperation with key coordinating agencies.
(6.3) Outputs received a rating of 2. While it was found that deliverables were distributed as
promised, in the areas of WASH and agriculture, there were concerns about the quality of
these outputs. This was particularly the case for biosand filter (BSF) installation, BSF
training, reservoir construction, and reservoir maintenance plans.
(6.4) Process perspectives received an assessment rating of 3. Opportunities for
improvement in baseline surveying were noted, including redesign of some elements of the
survey itself and retraining for the staff on how to conduct surveys. Beneficiary accountability
and coordination with other key agencies were two areas in Process perspectives that were
particularly strong.
1
The numbers preceding paragraphs in the Executive Summary refer to the relevant section in the
main body of the report from which the summary is drawn.
4
(6.5) Resources were another area that received a rating of 3. Budget and time were, on the
whole, adequate for the achievement of intended objectives, though more than 30% of the
staff interviewed replied that, in an ideal situation, the timeline would be longer to encourage
sustainability. Two staffing issues were raised: the capacity of project staff to carry out
effective training and monitoring in multiple sectors that each require expertise, and; the
need for greater technical oversight at the project level.
(6.6) Finally, organisational capacity was rated as a 4. Leadership, communication, financial
management and logistical systems were found to be adequate, appropriate and effectively
functioning. Incorporation of learning from past projects and experiences was evident
throughout the evaluation. It was found that DMT, despite working in an often adverse
environment, remains a healthy and effective team with strong relationships and a
commitment to continued growth and learning.
The overall score for the assessment was 19 out of a possible 24. This represents a strong
project, based on proposal design and DMT mandate, with room for improvement in the
future.
(7) Eighteen specific actionable recommendations were generated from the observations
made throughout the evaluation and evaluation report. These are as follows:
• Retrain project staff on BSF operation and maintenance
• Discontinue practice of having beneficiaries install their own BSF
• Adjust BSF monitoring forms and practices
• Consider design revisions for water reservoirs to reduce contamination risk and increase
sustainability
• Improve reservoir maintenance plans
• In future water quality testing training, include methods of safe disposal of membrane
papers
• Increase presence of technical advisors on the field
• Plant seeds and train beneficiaries to plant seeds according to the agricultural timeline,
not the project timeline
• Discontinue irrelevant training topics where possible
• Consider implementing Community Livestock Banking Systems (CLBS) the same way in
the south as in the north, through CDCs
• Continue spinning wheel interventions
• Increase posters for training in agriculture-related topics
• Increase pictures and decrease text in handbooks distributed to community members
• Include all sectors in baseline survey questions
• Include observation in baseline survey questionnaires
• Use the same baseline survey in different provinces
• Conduct refresher training on surveying
• Increase the time for the final evaluation
3. Introduction
3.1. Partner and project background
Tearfund, a humanitarian relief and development organization established in the UK in 1968,
seeks to assist in the eradication of poverty through community development, disaster
management and advocacy. Tearfund funds programs in more than 50 countries worldwide,
generally working through local partners. In particularly severe or unstable situations,
however, DMTs are set up to respond directly. This is the case in Afghanistan. Tearfund’s
DMT has been active in Afghanistan since 2001, responding to both natural disasters and
conflict. There are approximately 85 staff members country-wide, including expat and
national, operating in three provinces.
5
Current DMT Afghanistan activities are focused on assistance for returnees and IDPs using
an integrated approach that includes WASH, nutrition and livelihoods. This is keeping in line
with DMT’s Afghan country strategy for 2011/12, which is to strengthen the capacity of
communities and government departments in those three sectors. In addition, Tearfund plans
to assist in strengthening capacity of communities and government in the area of disaster
risk reduction (DRR). Internally, Tearfund is aiming to build its own capacity for localized
emergency response and to improve monitoring and beneficiary accountability methods.
The project under review in this evaluation – Kandahar, Jawzjan and Faryab Integrated and
Sustainable Services for Returnees and Host Communities – was implemented by Tearfund
DMT in Afghanistan between September 2010 and September 2011. It was designed as a
response to the dire needs of returnees and particularly vulnerable members of host
communities whose access to basic needs and services is severely impeded due to security,
natural disaster, lack of assets, lack of income and lack of opportunity for income. With a
goal of aiding both in the resettlement and in the development of a sustainable healthy
standard of life for 2,300 beneficiary households, the following six objectives were set out for
the program: (1) increased access to and consumption of safe drinking water; (2) increased
knowledge and practice of basic hygiene; (3) increased income opportunities through use of
spinning wheels; (4) improved nutrition, savings and income through access to and use of
vegetables; (5) improved nutrition, savings and income through access to and use of animal
products, and; (6) community self-reliance strategies.
The project was funded entirely by BPRM with a budget of $1,667,371.
3.2.
Purpose of evaluation
This final evaluation was included in the initial design of the project, as it was specified in the
proposal that an independent evaluator would conduct a review upon completion of the
project activities. The costs of the evaluation are covered by BPRM as part of the project
budget. Tearfund DMT Afghanistan commissioned the evaluation, and DMT’s monitoring and
evaluation (M&E) officer, Bryony Norman, was the representative of Tearfund who liaised
with the evaluator on all matters logistical and otherwise. The purpose of the evaluation, in
addition to fulfilling the requirement of the project proposal, is two-fold and is stated as
follows in the Terms of Reference: “to strengthen accountability and to increase learning.”
No previous external evaluations have taken place for this project. Internal reports, including
the mid-year review conducted by the M&E officer and quarterly reports compiled from
project locations, were consulted during the evaluation and are referenced as necessary
throughout the report. A peer review was conducted by SERVE down in Kandahar; however,
no documentation or report is available.
The following six key areas of assessment were considered in data collection and will be
addressed in this evaluation: (1) impact perspectives; (2) stakeholder perspectives; (3)
output perspectives; (4) process perspectives; (5) resource perspectives, and; (6)
organization capacity perspectives.
4. Methodology
4.1. Tools
Table 1 below gives an overview of the tools used to conduct the evaluation.
Table 1: Methodology Tools
Tool
Individual
Interview
6
Involved
Stakeholder(s)
Beneficiaries
Project staff
Community leaders
Description
Along with direct observation, this tool was relied upon heavily
for providing a comprehensive view of the project. Interviews
with beneficiaries covered each aspect of the project, focusing
Biosand filter (BSF)
technician
Technical advisors
Other staff
Direct
Observation
Beneficiaries
Focus Group
Discussion
(FGD)
Beneficiaries
CDC members
Project Staff
Material/Process
Review
Staff
Document
Review
Staff
4.2.
not only on the impact of the interventions but the quality of the
implementation process, nature of relationship with DMT, etc. As
much as possible, interviews were conducted onsite during
household visits and were accompanied by direct observation
and photographic evidence.
Intentional, guided observation was used to confirm or challenge
not only information offered during interviews but also
information found in key documents such as the Pre- and PostKAP surveys and quarterly progress reports. This observation
took place mainly during household visits but also during visits
to other key project sites such as the water reservoir.
FGD was employed to increase the quantity of the input, given
the limited time period of the evaluation. In general, FGD was
used to get a broader sense of the quality of the process and its
impact. This tool also helped to inform how widespread the
observations at household level are likely to hold true. Greater
weight, however, was given to individual interview and direct
observation when information from FGD conflicted with other
findings.
Key materials and processes used in the project implementation
were reviewed by the lead evaluator, including training material,
water quality testing processes, survey materials and processes,
and beneficiary selection materials and processes.
Document review began prior to and continued during the
evaluation. Onsite document review at each of the field offices
was done in cooperation with the project managers and included
documents mentioned in the “means of verification” portion of
the logical framework. Complete lists of requested documents
can be found in Annex 7.
Beneficiary Selection
In order to produce a truly impartial report, selection of interviewed beneficiaries must be
randomized; however, due to security situations, time constraints and other extenuating
circumstances, true randomization was impossible during this evaluation. Rather, within the
confines of the context, randomization was maximized and selection was carried out as
described below.
Jawzjan
To begin, beneficiaries in Jawzjan were numbered from 1 to 1000. A total of 4 out of the 6
northern field days were allotted to Jawzjan due to the fact that it had twice as many
beneficiaries as Faryab. The process for selecting actual beneficiaries began with selecting
districts. With 4 field days, we could cover villages in 4 districts, as travel time between
project locations made more than one district unfeasible. Of the 5 project districts in Jawzjan,
Mingajik was ultimately eliminated due to roads that were impassably muddy. After district
selection, came village selection. This was done by first eliminating the villages that were too
insecure to visit. Then a random number sequence from 1 to 1000 was produced. When a
beneficiary’s number from a village that had not been eliminated due to security or weather
came up, that village was selected for a visit. Using this method, two villages in each district
were selected. The final step was to produce another random sequence using the numbers
associated with the beneficiaries in the selected villages. The first number in the sequence
became the first beneficiary to be visited for a household interview; the second was the
second, and so on. In the case that a beneficiary was unavailable, the next number in the
sequence was chosen until the maximum number of beneficiary households possible had
been seen in a field day. A summary of the villages visited is shown below, along with
comments as to any exceptions to randomization.
7
Table 2: Selected Jawzjan Villages
District
Village Visited
Remarks
This was not chosen randomly but was selected intentionally by the
evaluator, as it is the site of one of the water reservoirs. Due to the
extensive time needed to travel to this site, it was used for household
interviews as well.
This was the only other village in Khanaqa not eliminated due to
insecurity. Nevertheless, it was a tense visit, as the CDC had been killed
a couple months before. There was a generally tenuous feeling in the
community. Women were not permitted to gather for a women’s FGD.
Only one individual interview could be conducted due to time and
security constraints.
No extenuating circumstances.
Khan Ariq was the second village planned; however, the road conditions
did not allow for the vehicles to pass. Instead, Hindukush village was
visited in order to interview a BSF technician.
No time for individual household visits in this village; had requested to be
taken to Eiraghly, as it had come up in the random selection, and only
had half day for surveying, so tried to go to Eiraghly after focus group
discussions in Eslam Joy.
Again requested to be taken to Eiraghly; by the time it was explained
that this was Jalal Abad, it was too late to travel to another location so
household visits were conducted in Jalal Abad.
Only one village was feasible for Aqcha due to time constraints. Qara
Boian was chosen randomly.
Elak Rabat
Khajambaz
Khanaqa
Sha Mirza
Murdian
Fatee Abad
None
Eslam Joy
Sheberghan
Jalal Abad
Aqcha
Qara Boian
Faryab
The beneficiary selection process for Faryab was much the same as Jawzjan, though there
were a greater number of villages eliminated as survey potentials due to insecurity in the
area. In addition, travel time to Faryab made it possible to see only one village in each of the
2 districts selected.
Table 3: Selected Faryab Villages
District
Village Visited
Andkhoy
Tawachi Khord
Qurgan
Surkh Bazar
Remarks
Randomness was impossible in this village, as most of the village had
left their homes to go to a wedding by the time the surveying began;
therefore, we just had to go to the houses of whoever was home.
Kandahar
Randomization was virtually impossible in Kandahar. This was due in part to security but also
to time constraints and the limitations of the surveyors. Of the 7 project villages in Kandahar,
the project staff selected the two most secure locations for surveying. Based upon this, a
random number sequence for each village was generated to select individual households for
survey; however, the surveyors did not understand the list and did not follow the priorities of
the random sequence. It is therefore assumed that either community development council
(CDC) leaders or Tearfund project staff selected the households for interviews. Because the
evaluator could only be in Kandahar two days for security reasons, there was no time to
correct the error.
Table 4: Selected Kandahar Villages
District
Village Visited Remarks
District 9
Loya Wala
District 3
Shin Ghazi
8
There are no female beneficiaries in this village, so only the male surveyors
conducted interviews here.
Both male and female surveyors worked in this village; however,
randomization did not happen due to a misunderstanding on the part of the
surveyors.
Focus Group Discussions
No attempt at randomization was made for focus group discussions in any of the provinces.
The participants were selected by the CDC. This was due to time constraints; focus groups
had to be assembled before surveyors reached the sites, so it was necessary to rely on local
leadership.
4.3.
Record of Stakeholder Participation
To conduct a review that would be statistically significant was not possible, given limitations
of time, security and finance. Therefore, the approach taken was to maximize the quantity of
input without compromising quality during the time allotted. Table 5 below outlines the
participation of various stakeholders in each of the provinces. It should be noted that
government participation was not part of the survey. To do an interview with one government
official would have required elimination of an entire village visit, and, as beneficiaries were to
be given priority in this evaluation, it was deemed more important to maximize face to face
interaction in villages. Government support and monitoring was instead verified during the
document review conducted with the project managers in each province. This was
satisfactory.
Table 5: Stakeholder Participation
Type of Participation
No. of
Jawzjan
Participants
No. of
Faryab
Participants
No. of
Kandahar
Participants
M
F
M
F
M
F
Household (HH) Interview
7
5
5
1
8
4
Focus Group Discussions
59
73
20
10
9
44
CDC Interview
BSF Technician
Subtotal
Village-level Participants
20
1
87
78
26
Staff Focus Group Discussions
Project Manager Interview
Agriculturalist Interview
Subtotal
National Staff Participants
5
1
1
3
3
7
3
Area Coordinator (AC) Interview
1
Program Director Interview
M&E Officer Interview
1
WASH Advisor Interview
1
Grants & Information Officer
Subtotal
Expat Staff Participants
1
4
1
TOTAL PARTICIPANTS
98
82
9
1
Remarks
Kandahar Female HH
interviews incomplete,
reflected in data
Kandahar Female FGD
and one male FGD (5
men) conducted by
Tearfund staff so not
impartial
3
11
20
48
2
1
2
Based in Kabul
3
0
3
2
Previous Kandahar AC
interviewed via email
Based in Kabul
Based in Kabul
Previous WASH
Advisor interviewed via
email
Based in Kabul
1
1
1
29
11
24
50
In total, 270 community members (133 male and 137 female), 18 national staff (13 male and
5 female), and 6 international staff (5 male and 1 female) participated in the evaluation.
Findings from interviews, FGDs and observation will be discussed in part in this report,
where the evaluator will highlight particularly significant or relevant data. A more complete
set of data and data analysis can be found in Annex 1.
5. Context Analysis
5.1. Security
In considering the context in which the project has been implemented, security is the first
aspect requiring analysis. Afghanistan has been in a perpetual state of conflict and instability
for decades. Political factions, ethnic divisions and criminal and terrorist groups, along with
various external stakeholders, have helped create the fractured situation in which the country
exists. Current key players in the conflict include Armed Opposition Groups (AOGs), who
oppose the sitting government and its Western ties; Armed Criminal Groups (ACGs), who
capitalize on the generally insecure and unstable environment; Afghan National Police
(ANP), who are widely known for their corruption, and; International Security Assistance
Forces (ISAF), a NATO-led military force with a mission of bringing stability to the country.
ISAF and other International Military Forces (IMF) are among the most common targets for
AOGs. In these attacks, the threat to NGO activity is one of collateral damage. ACGs,
however, have been known to directly target International Non-Governmental Organizations
(INGOs) and their staff with theft or abduction.
The Afghan NGO Safety Office (ANSO) has noted a general deterioration countrywide in
terms of both general security and safety for NGOs. Their Quarter 2 report for 2011
(available online at www.afgnso.org) revealed a 42% increase in AOG attacks in 2011 over
the same quarter in 2010 and a 73% increase in NGO incidents as compared to 2010. In the
same report, provinces were given a current security rating based on reported incidents.
Jawzjan was rated “Deteriorating,” Faryab was rated “Highly Insecure,” and Kandahar was
rated “Extremely Insecure.” For both Jawzjan and Faryab, this represented a decline in
security rating since 2009.
The security context significantly affects the way DMT carries out their project activities. Most
notably, the Kandahar team requires remote management, as the threats of abduction and
IED attack remain prohibitively high. Staff visits to the province are rarer than to the north
and include more extreme risk management measures, such as sleeping in different
locations each night and not staying for periods of more than a few days.
Monitoring and evaluation are particularly difficult in the current situation. In both northern
and southern provinces, there are field locations that cannot be visited regularly due to
instability. In those cases, information from DMT local staff must be relied on for assessing
project progress.
In terms of risk mitigation, of utmost importance in all three provinces is the establishment
and maintenance of good relationships with target communities.
5.2.
Cultural
In light of maintaining acceptance in target communities, cultural considerations must also be
given high priority. Conservative manners of dress, particularly for international female staff,
are observed in order to show respect for local customs. Women wear head scarves and,
when security requires, will even use a burqa. Men are also encouraged to adopt local dress
when visiting field locations.
Both international and local staff must demonstrate acceptable behaviour between genders.
Generally, when operating in field locations, local women will not interact with unknown men
10
outside of their family. International women may interact with local men but must always
remain aware of what and how they are communicating. These cultural guidelines can be
limiting in terms of project activities, particularly monitoring and evaluation. If a female
evaluator does not speak one of the local languages, accessing the female beneficiaries will
be much more difficult as female English-speaking translators who have permission to travel
to unstable locations are difficult – and often impossible – to find.
Photography can be another cultural issue. Women in rural communities are often
uncomfortable with being photographed, as it is viewed as inappropriate for their faces to be
seen by men outside their families. When photographic evidence is required by donors, it
must be handled sensitively.
6. Evaluation of each Key Area: Findings, Conclusions and
Assessment
The following section summarizes the findings, conclusions and assessment of the six key
areas outlined for evaluation in the ToR. For each key area, a statement of assessment is
made following a four-point scale. The four points are defined as follows:
1: The project makes no contribution to the aspect.
2: The project makes a minimal contribution to the aspect; there are major shortcomings
that must be addressed.
3: The project makes an acceptable contribution to the aspect; there are shortcomings
that could be addressed.
4: The project makes a substantial contribution to the aspect.
6.1.
Impact Perspectives
The following section will consider the impact of the project, including what changes took
place, whether positive or negative, intentional or unintentional, and to what degree they
were in line with the overall goal of the project.
The stated goal of the project, according to the project logical framework is:
Returnees, IDPs, and other vulnerable groups within the population are able to settle and
integrate, and sustain a basic healthy standard of life within host communities.
6.1.1. Findings
Findings will be organized according to the six project objectives. Baseline data was not
collected uniformly between the provinces (see Section 6.4.1 for a discussion on this) and
was only synthesized and summarized for Jawzjan (as far as the evaluator is aware, given
documents received). It is impossible to consider two completely different sets of data and
three separate provinces in the space allotted; therefore, as Jawzjan represents the province
with the greatest number of beneficiaries, Jawzjan figures for baseline will be used for
comparison, unless otherwise noted. Data in the column labelled impact will represent a
summary of the findings from all three provinces, however, unless otherwise noted.
11
Table 6: Project Impact
Objective
Baseline
No knowledge of,
experience with or
possession of BSF
Raw, untreated
1. 2,300 beneficiary
water with >10
households have
e.coli per 100ml
increased access to and
being used
consumption of safe
regularly for
drinking water, through the drinking
construction of community
water reservoirs, the
installation of household
level water treatment
No water reservoirs
systems (biosand filters),
and training in proper use.
Use of soap:
reported baseline
0%
2. 2,300 beneficiary
households have
increased knowledge and
practice of basic hygiene
techniques that reduce
contamination and spread
of disease.
Knowledge of
handwashing:
reported baseline
0% (same for
Kandahar)
Latrine coverage,
as an indicator of
safe sanitation
practices: reported
baseline 0%
3. 300 beneficiary
households have
increased income
opportunities through the
use of spinning wheels.
37% reported
spinning only 1kg
of wool per day
4. 2,300 beneficiary
households have improved
nutrition, savings, and
income potential through
increased access and
consumption of
vegetables.
3% reported
planting a variety of
vegetables
12
Impact
100% of beneficiaries report and were observed having
and regularly using BSF to treat their drinking water
Quarterly reports stated that water quality sampling was
100% in compliance with <10 e.coli/100ml. This was
unable to be confirmed during the evaluation. Serious
concerns over improper installation and maintenance of
the filters (see Section 6.3.1) lead the evaluator to
question this; however, it can be said with confidence
that beneficiaries are drinking water of a better quality
than before, as even an improperly used BSF will
perform a degree of filtration.
Beneficiaries are collecting drinking water from the
reservoirs. This is more convenient and easier for them
than collection from an open source. No comment can be
made on the impact to water quality or, therefore, to
health as a result of this because reports of water quality
during open storage vs. water quality in the reservoirs are
unavailable.
Reported impact from DMT: 100%
The findings of this evaluation were different. 100%
reported using soap now while 90% actually had soap
available, but only 45% acknowledged this was a new
behaviour in the last year. This is still a significant impact
and is in line with project proposal goals, but it is not
100%.
Reported impact from DMT: 85%
Using the same question as DMT used for evaluating this
indicator, the evaluator found a 71% post-project
understanding of the relationship between handwashing
and diarrheal disease. Using a question about critical
handwashing times, 65% of respondents were found able
to name the two most critical times – after defecating and
before eating.
Reported impact from DMT: 100%
Findings for this were vastly different. 69% of
respondents reported having had a latrine since before
the project. Of the 31% that remained, 78% reported
having constructed a latrine after what they learned. This
represents an overall increase in latrine coverage of
24%. Again, this is significant and in line with project
goals but is much less than 100%.
Findings of the evaluator showed that, on average, prior
to the project, women could spin 0.6kg, which brought
them an average of 17 Afs per day of profit. After
receiving and being trained on spinning wheels, the
women reported being able to spin an average of 2.8kg
of wool per day, bringing in 81 Afs. This is an overall
increase of nearly 5 times and represents significant
benefits in the lives of the women, who use the money
“…for rice, bread, life…”
100% planted 6 varieties of seeds
Of those, 90% harvested vegetables
Of those, 100% ate the vegetables
56% of respondents in the south reported selling veg.
0% in the north reported selling (water problem)
73% overall reported “eating of more veg” as a benefit
48% overall reported health benefits as a result
5. 2,300 beneficiary
households have improved
nutrition, savings, and
income potential through
increased access and
consumption of animal
products.
6. Communities develop
self-reliance strategies for
Objectives 1 to 5.
14% reported
raising their own
animals for milk;
3% reported raising
their own chickens
100% received and are raising goats and chickens, either
themselves or in cooperation with another family or
community member.
90% report consuming eggs from their chickens
70% report consuming milk from their goats
34% report selling some portion of eggs
14% report selling some portion of milk
See Section 6.3.1 below for a detailed discussion on the outputs named under
this objective. There are two aspects not touched on in that section: the impact
of spinning wheels on beneficiary income (covered in this table above) and the
project’s effect on the incidence of communicable disease. Sufficient
communicable disease data was not collected or available at project locations
to assess this. Health data collected during the evaluation showed that:
• 73% of respondents noted improved health as a benefit of BSF; 23%
identified a reduction in diarrhoea specifically
• 56% of respondents noted improved health as a benefit of hygiene
education
Two other changes in the target communities were reported, relating to the Community
Livestock Banking System (CLBS) introduced during this project. In the north, CLBS was
received with gratitude and enthusiasm. CDCs chose secondary beneficiaries. There were
plans in place for tertiary beneficiaries and so on. It was noted that this was promoting unity
in the community. In the south, however, CLBS operated differently, without CDC control.
This resulted in lots of arguments and disunity in the communities.
6.1.2. Conclusions
For objectives 2 through 5 in the table above, impact goals were achieved as outlined in the
project proposal. At least 2 hygiene behaviours were increased by at least 20% (and, indeed,
other behaviours could have been reported if not for space limitations); beneficiary income
was increased dramatically through introduction of the spinning wheel and exceeded the goal
of doubling production; more than 60% of beneficiaries reported an increase in vegetable
consumption (though, in the case of the north, this was true only for a short period of time,
due to water problems resulting in poor harvest), and; more than 60% of beneficiaries
reported increased consumption of milk and eggs.
Concerns regarding the 1st and 6th objectives are covered largely in Section 6.4.1 so will not
be discussed at length here.
The only unintentional negative community change noted was the arguing and fighting that
began as a result of the way CLBS was conducted in the south.
The activities that seemed the strongest and led to consistently significant impact were
spinning wheels and CLBS as conducted in the north.
6.1.3. Assessment
Generally speaking, the project impact was seen to the extent and in the way that it was
intended and designed in the project proposal. There are concerns about not achieving
maximum impact with water supply interventions as well as concerns about sustainability, but
these are more directly related to the quality of outputs and, as such, will be discussed in the
sections that follow. On the whole, from the perspective of impact on the communities, the
project activities have been largely successful. Thus, the evaluator’s overall assessment of
this perspective, according to the scale defined above, is: 3.
13
6.2.
Stakeholder Perspectives
In this section, the perspectives, attitudes, behaviours and opinions of various project
stakeholders will be looked at with regards to the relevance and outcomes of the project.
Beneficiary targeting will be considered as well as the relationships between stakeholders.
6.2.1. Findings
Beneficiary Perspectives
In order to evaluate beneficiary perspectives and
attitudes, household interviews and focus group
discussions included questions about beneficiaries’
feelings about the project, their changes in behaviour,
their plans to continue or discontinue new behaviours and
their relationships with DMT staff.
When asked if they would recommend this project to
another village, the response was 100% positive. Also
100% positive were the responses to questions about the
quality of DMT’s work and the behaviour of DMT staff in
the villages.
Quotes from Beneficiaries
“Tearfund staff act as a
brother would with us.”
-- Beneficiary, Jawzjan
“Even if they have a choice of
building roads or clinics or
wells, this project is better
because it impacts personal
life.” – Beneficiary response
when questioned about whether
or not he would recommend this
project to another village, Faryab
Indirect beneficiaries, such as family members and other
“We are praying for them.” –
community members, were also part of FGDs in the
Beneficiary when questioned how
northern provinces, where such secondary beneficiaries
they felt about DMT staff,
were often included in trainings. There was no difference
Kandahar
noted between the perspectives of direct and indirect
beneficiaries as far as opinions that were voiced. No
objections were raised to the evaluator or survey team;
rather, secondary beneficiaries supported the claims of
direct beneficiaries and were grateful for the project and project staff.
In terms of behaviour change, the following key target behaviour areas were evaluated:
personal hygiene, home vegetable gardening and treatment of drinking water with biosand
filters
Table 7: Findings for key target behaviour change
Behaviour Sector
Findings
45% of respondents reported adoption of new handwashing behaviour; 41% both
reported adoption of the behaviour and demonstrated presence of handwashing
Personal hygiene
device, water and soap
28% of respondents reported adoption of tooth-brushing behaviour
31% of respondents reported generation of seeds for next season’s planting; this
Home vegetable
varied widely between the north and the south, as the north showed only 11%
gardening
generation of seeds (due to drought) while the south showed 76% generation
100% of respondents reported regular use of BSF for treatment of drinking water; this
was confirmed by observation. However, knowledge of proper use was lacking. 55%
Proper use of biosand
reported cleaning that was too frequent (daily, weekly or monthly) and not based on
filters
water flow while 81% were unable to correctly explain or demonstrate the swirl and
dump cleaning method.
Other Stakeholder Perspectives
Other stakeholders who were asked about their perspective on the project and stakeholder
relationships included community leaders, local staff and international staff. Local
government was unable to be interviewed directly due to time and security constraints;
however, government participation and opinion was verified through document review. In
both the north and the south, strong support of the project was exhibited not only through
14
signed agreements and monitoring documents but through letters of appreciation and
handover documents.
Community leaders
As with project beneficiaries, community leaders (CDC and village leaders) gave 100%
positive feedback in terms of their opinion about the project itself and relationship with project
staff. No complaints were voiced regarding quality of work, staff behaviour or communication.
Responses revealed that leaders felt the needs being addressed were relevant to their
communities’ needs. In two instances, CDC members, who were not beneficiaries of the
project, noted that they had purchased a BSF after seeing its impact.
Staff
Table 8 summarizes staff sentiments regarding the relevance of project activities and overall
impressions of the project. In general, staff members were eager to discuss the positive
aspects of design and implementation, noting that communities are requesting continuation
of this project. This was confirmed in HH interviews and FGDs.
Table 8: Overall project impression by staff
Subject
Responses
Very successful and designed according to community needs; in particular, the
Overall impression
integration of sectors was viewed as a positive and effective approach. 70% of
the respondents indicated they felt every aspect was a success.
Most commonly noted by local and international staff were the elements of BSF,
Most successful elements
spinning wheels and goat distribution
Staff questioned the successful of vegetable gardening in terms of sustainability
(due mainly to water problems, though also due to questions of land access in
Least successful elements
kuchi communities). Also in question was the likelihood of long-term successful
management of water reservoirs
Staff were asked what changes they would have introduced to the project. Responses that
came up on more than one occasion included the following: (1) increase the number of goats
and chickens distributed, and; (2) increase spinning wheel input, both through increased
number of machines and increased amount of initial wool distributed.
Beneficiary Targeting
According to the project proposal, the targeted populations were returnees, IDPs and the
most vulnerable in the host communities. In FGDs and individual interviews, staff expressed
that, from their perspective, targeted populations were successfully reached.
In leading up to this project, DMT outlined their beneficiary selection criteria and defined key
terms for use in identifying populations. These were reviewed by the evaluator. Key
coordinating agencies in beneficiary targeting included Department of Returnees and
Refugees (DoRR), United Nations High Commissioner for Refugees (UNHCR) and CDCs.
Voluntary Repatriation Forms (VRF) were used to verify the status of all returnees. In
identifying “most vulnerable” populations in the host communities, the community members
themselves were involved. A list of the most vulnerable was produced by the CDC and then
posted in a public place where community members could contest the choices and suggest
alternate beneficiaries who were in greater need. A number of instances were cited by staff
in which this process was carried out successfully.
6.2.2. Conclusions
It was clear to the evaluator that stakeholders on every level – from local government (as
verified through document review) to community leadership to beneficiaries (both direct and
indirect) to local and international staff – had a strongly positive overall impression of the
project. Additionally, there was strong evidence of open, trusted communication between
15
DMT project staff and other stakeholders, particularly beneficiaries and community leaders.
No concerns were raised in these areas.
Behaviour change goals stated in the project proposal referred specifically to hygiene
behaviours (20% increase in 2 hygiene practices), vegetable gardening (30% generation of
seeds for future planting) and BSF (85% knowledge and practice to maintain filters).
Evidence was found during this evaluation to support completion of the hygiene behaviour
and vegetable gardening goals (though sustainability of such adopted behaviours must be
discussed separately). However, with regards to BSF, it was found that 81% of the
beneficiaries interviewed were unable to describe proper cleaning methods. Thus, although
the behaviour of using BSF has been adopted, it has been adopted incorrectly.
In reviewing targeting tools and processes, it was determined that targeting was carried out
effectively, efficiently, with strong government and key agency coordination, and with the
participation of targeted populations.
6.2.3. Assessment
On the whole, Stakeholder Perspectives were strongly positive. The main concern was
incorrect adoption of the BSF technology. The evaluator’s overall assessment of this
perspective, according to the scale defined above, is: 4.
6.3.
Output Perspectives
The following section will answer the question of whether or not outputs were achieved and
whether or not those outputs were of appropriate technical quality.
6.3.1. Findings
Using the project proposal and logical framework as guides, project outputs were evaluated
with the findings summarized in Table 9.
6.3.2. Conclusions
Evaluation findings confirmed that outputs had been delivered as outlined in the project
proposal and logical framework.
The quality of outputs varied. Of particular concern was the quality of BSF. Although the
body construction was correct, there were significant issues of improper installation and
improper maintenance knowledge and practice. Even with improper use, BSF will result in
cleaner drinking water, as the filtration of the sand is still effective to an extent; however,
simple filtration could be achieved using much easier and cheaper methods. The design of
BSF is to allow for 4 different means of pathogen removal: predation, adsorption, physical
filtration and natural die-off. To that end, BSF is designed with particular parameters which,
when not observed, will result in lower rates of pathogen removal. Project staff reported that
BSF installation was carried out by the beneficiaries after receiving training in washing gravel
and sand; however, such activities should only be carried out by a BSF technician. Sand
washing, in particular, is a critical step and must be neither overdone nor underdone. Project
staff also reported that, in addition to swirl and dump methods of cleaning, beneficiaries were
advised to empty out, wash, and replace filter media if outlet pipes became clogged. The
clogging of outlet pipes is a symptom of improper installation. Once a filter is properly
installed and has been tested, the media should never need removal.
Secondly, concern over the quality of the water reservoir in Murdian was noted. Not all
measures were taken to mitigate the risk of contamination of the stored water. Additionally, a
vague maintenance plan and an observed lack of understanding on the part of the CDC as to
how regular maintenance should be carried out lead the evaluator to question the
sustainability of the reservoir as an improved water storage and water collection option.
16
Table 9: Outputs and output quality
Output
Findings
Distribution registers and direct observation
By Nov. 2010, BSF are
confirmed the distribution of BSF to beneficiary
being produced and sold
homes. Workshops could not be observed directly
by at least 3 new local
due to scheduling conflicts. One BSF technician
workshops and delivered
was interviewed who was contracted for 700 of the
to 30% of beneficiary
2,300 total filters and reported filling the contract
households.
100%.
By February 2011, at
HH interviews and observation revealed that 100%
least 60% of beneficiary
of the beneficiary households were regularly using
households are using
their BSFs. Issues of maintenance were discussed
and maintaining a BSF
in section 6.2.1
in their homes.
Water quality tests were not conducted during this
evaluation. Quarterly reports were reviewed,
indicating that the requirement had been met;
however, it was noted that one set of water quality
By May 2011, 85% of
tests reviewed in the north was not in compliance.
BSF that are being
The tests were conducted in March 2011. Results
properly used and
showed that 14 of 20 tests of BSF water were out
maintained are
of compliance. The range of e.coli/100ml was
producing water with
between 3 and 42. The evaluator does not place as
less than 10 e.coli per
much weight on the result of water quality tests as
100ml.
on the likelihood of BSF not functioning to
maximum capacity due to issues of installation or
construction; hence the concerns noted to the right
under “quality comments.”
One of the two reservoirs was visited during this
evaluation. The second could not be accessed due
to road conditions; however, it had been visited
during the mid-year review. 100% of this output
has been completed.
By May 2011, 2
underground water
reservoirs have been
built.
Quality Comments
Positive observations about BSF quality include:
• No leaks.
• Sand size is not too large, as only 2 filters – 8% - recorded a flow rate of more than 0.5
litres/minute, which is the maximum recommended flow rate.
• 94% of users were filtering into containers that were both different from the collection
containers and appeared clean. This indicates effective training on this topic and
successful adoption of this behaviour.
The following concerns were noted during this evaluation:
• Improper maintenance – see section 6.2.1 for details. This is the greatest cause for
concern.
• 55% of observed filters had outlet spouts longer than 2cm, which will prevent the biolayer from forming on the face of the sand. This issue was raised in an earlier evaluation
report by CAWST.
• Photographic evidence from Kandahar showed at least 58% of the homes (7 of 12)
practicing the plugging of the outlet spout to keep water from dripping. This is another
behaviour that will prevent formation of the bio-layer.
• 54% of observed filters were placed outdoors, increasing the risk of contamination of the
water during collection from the filter.
• Of lesser concern but still notable, 30% of the lids were noted as ill-fitting. This issue had
been raised during the mid-year review.
• When reviewing water quality testing, consider safe disposal of used membrane filtration
tests and proper storage of materials. It was observed that old filter papers had been left
in all of the petri dishes, which is not recommended as filter papers contain active
bacterial cultures and present risk of infection.
A sanitary survey was used to determine the risks of contamination of the water in the
reservoirs. Areas of concern are highlighted below.
• Large, unscreened openings both at the inlet and the outlet of the reservoir – these
present opportunities for mice, rats and other animals to enter, as well as debris. The
inlet screen had been removed entirely and was being stored somewhere else. The
overflow screen was in place but ill-fitted.
• Lack of proper drainage around the handpumps or the reservoir itself – all spilt water
collects on the surface of the reservoir, presenting health risks during water collection
and greater risk of contamination of stored water.
• Ground around the reservoir was not levelled or mounded in such a way that runoff
would flow away; rather, runoff will flow on to the surface of the reservoir. Additionally,
the mud and other debris was noted falling directly into the silting basins for the same
reason. Photographic review of the second reservoir showed this issue was handled
more successfully in Mingajik.
Handpumps were already very loose at the point of connection.
Exposed rebar was noted at the inlet.
More minor concerns included: lack of locks or gasket seals on the hatches and lack of a
fence.
Without direct observation, it is difficult to comment on the quality of this output. A strong
emphasis on education through pictures was noted, which is appropriate for the target
audience. One recommendation would be the narrowing of topics and targeting of 1 or 2 key
hygiene behaviours. The evaluator noted, for example, that in the south, one topic apparently
introduced, at least to some extent, was solar disinfection as an option for water treatment
(according to Pre- and Post-KAP survey results). In a project where BSF is a main focus and
where the volume of information being communicated to uneducated beneficiaries is already
very high, this topic could have been eliminated.
Beneficiaries reported 100% satisfaction rate with the quality of spinning wheels and wool
distributed, as well as a 100% response rate for the planned on-going use of spinning
wheels. The only difficulty noted was the lack of a stool or chair to sit on while using the
machine.
Training could not be directly observed. Materials were reviewed and were of good quality.
Respondents were able to identify a few new concepts they had learned, which included the
organization of small plots and land levelling/ preparation. One point of concern was noted:
• The practice of planting seeds according to the project cycle schedule rather than the
plant schedule, led to a higher rate of plant failure than necessary and to a
misunderstanding on the part of the beneficiaries as to when various vegetables should
be planted and harvested. In particular, in the north, spinach, which should have been in
harvest stage during this evaluation, was seen growing in only 2 households and was
reported to be not in season.
A second point – not of concern but perhaps for conversation regarding future training – is
this: the continued use of ditch irrigation, even in small plots, as observed and photographed
during the survey, may not be conducive to home gardening in water-scarce parts of the
country.
This varied widely from northern to southern regions. The north saw high rates of harvesting
– over 70% - for 3 crops (spinach, tomato and lettuce), while the south saw over 70% harvest
rates for all 6 crops. The crop with the combined overall highest success rate was spinach
(92%).
There was again variance in this output between north and south. In the north, CDCs were
given greater control of the CLBS, as they were responsible for designating secondary
beneficiaries (with opportunities for community member approval). In the south, CDCs were
given no control of CLBS; they were asked to agree to the process, but secondary
beneficiary selection and follow-up was left to direct beneficiaries. The approach in the north
seemed to produce greater results in terms of community relationships and sustainability of
the system. In the south, it appeared unlikely that the system will continue beyond the first
•
•
•
By February 2011, at
least 90 community
mobilisers have
completed training and
are able to implement
education sessions on
health and hygiene.
By May 2011, [spinning
wheel] beneficiaries
have increased
production by 100%.
By Nov. 2010, all
targeted households
have received basic
vegetable cultivation
training.
By May 2011, at least
40% of the targeted
households grow their
own vegetables.
By Nov. 2010, CLBS
have been agreed with
all CDCs in target
communities.
18
Community mobiliser lists and training reports were
viewed by the evaluator to confirm this activity.
Direct observation was not possible, as the activity
has been completed. 42 mobilisers in Faryab and
76 in Jawzjan were reportedly trained.
Through FGDs, it was found that beneficiaries are
producing, on average, 4.5 times as much yarn as
they did before receiving the spinning wheel.
100% of household interview respondents
indicated that they had received basic vegetable
garden training. In addition, facilitator reports of the
trainings were seen. Attendance records were
reviewed in the south, though in the north they had
already been destroyed.
90% of respondents reported harvesting their own
vegetables during the course of the project.
100% of CDC members interviewed confirmed the
agreement of CLBS. Supporting documents –
signed agreements – were reviewed by the
evaluator in each province.
By Nov. 2010, all
targeted households
have completed basic
animal husbandry
training.
By May 2011, at least
40% of targeted
beneficiary households
continue to practice
animal husbandry
sustainably.
By February 2011, the
two communities with
underground reservoirs
have maintenance plans
written by the water
management
committees.
100% of household interview respondents
indicated that they had received basic animal
husbandry training. In addition, facilitator reports of
the trainings were seen. Attendance records were
reviewed in the south, though in the north they had
already been destroyed.
At the time of the evaluation, 100% of households
surveyed had received the animals and reported
acceptable quality. Of these, only 3 reported that
the animals were no longer at their house. In 2 of
these cases, the goat had been given to someone
else to be cared for due to lack of space and/or
money to feed the animal. In the third case, the
chickens had been given away because the
beneficiary couldn’t afford to feed them; her brother
was raising them.
A basic maintenance plan was on file at DMT’s
office in the north. The water management
committee seemed to know little about what the
plan included. When asked to see the plan, they
replied that they didn’t have a copy but that
Tearfund could show it to us at the office.
generation of kids.
Training could not be directly observed, as the activities were over. Training materials were
reviewed and were of good quality, though could be improved by decreasing text and
increasing pictures, as also noted by local and international staff. Topics were diverse.
Concern was raised over the teaching of structures for chickens and goats that were too
expensive for beneficiaries to build. Project staff raised a request for more use of posters in
both animal husbandry and vegetable training, as are used in hygiene training.
• One element being considered in this project was improvement on the disposal of the
waste of the animals and control of their movements in the yard. Little evidence of
chicken control was seen. Goats were largely limited by a rope or a simple structure.
• An improved practice was the use of vaccinations to control disease. One beneficiary in
Jawzjan noted that he had raised chickens for many years, but they always used to die.
These ones are still alive.
• Animals remained in good condition, producing eggs and milk. 90% of respondents
continue to consume eggs from the chickens, while 70% consume milk from the goat.
The maintenance plan was more basic than anticipated and did not include a schedule or
plan of activities, particularly sediment removal. Nor did it assign responsibilities. These
facts, coupled with the fact that the water committee did not have a copy of the plan in their
possession, make sustainable operation and maintenance unlikely.
Thirdly, in regards to home vegetable gardening training, it was observed and reported that distributed seeds were planted according to the timeline of the
project, not according to best agricultural practice. This led to higher rates of crop failure than necessary and to misunderstanding of proper practice
6.3.3. Assessment
Output Perspectives were strong in terms of meeting goals of deliverables. However, significant quality issues were observed and raised by the evaluator.
Therefore, the overall assessment of this perspective, according to the scale defined above, is: 2.
6.4.
Process Perspectives
In evaluating the process of this BPRM-funded project, the following elements will be discussed: baseline survey, quality standards, beneficiary
accountability and project coordination.
19
6.4.1. Findings
Baseline Survey
Baseline surveys were conducted by first identifying beneficiaries and then choosing a
random sample to participate in household interviews. A total of 495 surveys were
completed, which represents 22% of the total beneficiaries, a strong level of participation.
The surveys in the northern and southern provinces were different. Below are observations
about the surveys themselves and how they were carried out.
Table 10: Baseline survey observations
Province of
Strengths
Survey
• Covered all sectors
• Questions designed to be answered
Jawzjan
by every respondent (e.g., “Don’t
and
know” is an option on knowledge
Faryab
questions
• Covered knowledge and practice
• Questions relevant to the project
• Included elements of observation to
confirm or deny responses given by
interviewees
Challenges
•
•
•
•
•
•
Kandahar
•
No observation built in
Evidence that staff didn’t fully understand
process; for example, Q6 should only be asked
of respondents who said yes to Q5, but that
was not the case
To determine rates of diarrheal disease, must
also record family size
Many irrelevant questions (e.g., AIDS)
Covered only one sector
Many questions didn’t offer options for every
respondent, making it impossible to tell if the
survey was not fully completed or if the
respondent gave an answer that wasn’t listed
Some questions unlikely to be accurately
answered; for example, respondents are not
likely to know how many times they’ve had
diarrhoea in the past 3 months
Quality Standards
The Tearfund quality standards reported most frequently by staff as being a priority during
this project were beneficiary accountability and gender, followed by values, and conflict.
Evidence was seen to support the prioritisation of beneficiary accountability and will be
discussed in the following paragraph. In terms of gender, there was strong participation of
both men and women, as is reflected not only in the beneficiary lists but in the makeup of the
facilitation teams. Additionally, the introduction of the spinning wheel component of the
project promoted gender issues. Minimizing conflict was a consideration in the design of the
project and informed the process of beneficiary targeting by assuring that vulnerable host
community members were selected in addition to the returnees and IDPs. Finally, the values
of encouragement and servanthood were reported as being a focus of the teams and were
seen demonstrated between and among DMT staff, both local and international.
Beneficiary Accountability
Significant work has been done to boost beneficiary accountability during the course of the
project. Monitoring and accountability tools have been developed, including monthly
beneficiary feedback systems and methods for gathering stories of transformation. In
addition, project staff were trained by the M&E Officer not only on use of the tools but on the
principles of accountability and monitoring. Further practical training happened while
conducting the mid-year review, which was designed to intentionally build staff capacity in
this area. Upon interviewing staff, they noted the following key actions they took in remaining
accountable to the beneficiary communities:
• Gave answers to all questions posed by the beneficiaries
• Delivered what was promised
• Gave beneficiaries choices
•
•
Carried out activities and shared information in the presence of many stakeholders
Established communication methods – beneficiaries had staff phone numbers and vice
versa
Coordination
As discussed in Section 6.2.1, coordination with key agencies took place in the project
throughout, beginning with project design. The project was directly in line with two of the
UNHCR’s priorities for Afghan returnees in their 2010-2011 Global Appeal. Following project
design, beneficiary targeting was done in close cooperation with UNHCR and DoRR. During
implementation, line departments and ministries were involved in monitoring project activities
and processes, including tender ceremonies, distributions, animal vaccinations, etc. The key
line departments were the Department of Agriculture, Irrigation and Livestock (DoAIL) and
the Ministry of Rural Rehabilitation and Development (MRRD). Together with the Ministry of
Economy (MoE), these government offices had regular and close involvement in the project.
The governors’ offices were also involved. Their participation is documented.
In the majority of the project areas, it was confirmed that no other NGOs were carrying out
similar types of projects. German Agro Action (GAA) was involved in water supply activities
but not BSF, so there was no duplication. In order to encourage cooperation and mutual
learning, a peer monitoring activity was carried out with another international NGO, SERVE.
6.4.2. Conclusions
Interviews, observation and analysis revealed that both beneficiary accountability and
coordination were aspects of the process that were particularly strong and continuing to
develop. Beneficiary feedback was not only solicited and effectively collected and
documented but was used to inform the design of future projects. An area that remains to be
further developed is how feedback can be used to influence implementation during the
course of the project itself. This is difficult, given time constraints and project design, in which
all outputs are determined during the proposal stage.
Baseline surveys were found to include adequate participation on the part of beneficiaries.
Improvements could be made in survey design and staff could benefit from further training on
survey goals and methods.
Staff were aware of quality standards, though there was not a collective understanding of
which standards were being prioritised in this particular project.
6.4.3. Assessment
Overall, Process Perspectives were positive, particularly with respect to beneficiary
accountability and coordination. There is room for development in the quality of baseline
surveying. The evaluator’s assessment of this perspective, according to the scale defined
above, is: 3.
6.5.
Resource Perspectives
In this section, issues of budget, project timeline and staffing will be reviewed and assessed.
6.5.1. Findings
Budget
Interviews with local and international staff revealed that all felt the budget was adequate for
carrying out the project activities. The only area of lack mentioned was in the initial budget for
this external evaluation. A review of the actual expenditures showed an overall underspent
budget of around $30,000. Variance, at the point in time of this evaluation, was
21
approximately 8%, reportedly within the 10% allowance by BPRM. The area of greatest
variance by percentage was in personnel, where there was an overall overspending.
In terms of directly verifying whether or not funds were used as stated, only documents
specifically mentioned in the logical framework as “Means of Verification” were reviewed by
the evaluator. These included: distribution registers for seeds, animals, BSF and spinning
wheels; purchase receipts for animals and spinning wheels, and; tender documents for
animals and spinning wheels. All were found to be in order.
Time
Responses from staff regarding project timeline varied. Fifteen staff responded that they felt
there was just the right amount of time for the project activities and objectives. Seven
indicated in their responses that the timeline was short. Reasons reported for wanting to
increase the time were: (1) there was a lot of information to learn in just 12 months, and
beneficiaries indicated to staff that training/teaching time was not enough; (2) to see
sustainable behaviour change, more than one year is needed, and; (3) interventions like
spinning wheels could include activities further along the marketing chain if time allowed,
thus increasing both impact and sustainability.
Staff
From the perspective of those in direct management, there was enough project staff of
adequate capacity to carry out the activities satisfactorily. Two strengths of project staff that
were reported were their previous experience with similar types of interventions in the past
and the fact that they were most often from the area in which they were working.
Two areas of concern were noted by the evaluator.
Table 11: Staffing concerns
Area of Concern
Observations
Staff capacity
Staff seemed unaware of basic problems with certain project activities, such as BSF.
Monitoring forms had been filled out and submitted but did not reflect the actual
situation. For example, on none of the BSF monitoring forms reviewed by the evaluator
did staff comment on outlet pipes that were too long or beneficiaries unable to describe
the “swirl and dump” cleaning method. All flow rates for BSF were recorded as exactly
the same in one province (0.4 litres/minute, the optimum clean filter rate), though
clearly that is impossible.
Technical oversight
There was an acknowledged lack of project-level technical officers. Instead, technical
oversight was provided by centralised advisors in agriculture and WASH. With
technically sensitive interventions in both of these sectors, consistent, on-site training,
monitoring and follow-up is needed. Though project staff were trained by technical
advisors, it is unrealistic to expect either supervisors or facilitators to become experts
in the fields of agriculture and WASH through quarterly – or less frequent – trainings.
6.5.2. Conclusions
Overall, resources were adequate for delivering the outputs outlined in the project proposal.
The budget was adequate and followed according to BPRM guidelines.
The timeline was adequate for deliverables and debatable for sustainability. Given that one
of the project objectives dealt directly with sustainability, it is a subject worth further
conversation. However, it was also clearly noted that DMT’s mandate does not include
having long-term presence in communities. They are a disaster management team, not a
development team. This mandate will continue to inform project timelines.
A few concerns with staffing were found. The conclusion of these is that, at a minimum, the
presence of technical advisors at project locations needs to be increased. Project staff are
22
not and should not be expected to be technical experts. Close, careful monitoring of their
activities is needed. When technical oversight is lacking, project staff effectively pass on their
misunderstandings to beneficiaries, compromising the quality of the outputs and reducing the
overall impact of the project.
6.5.3. Assessment
In general, Resource Perspectives were positive. Budget and timing reflected the stated
objectives and DMT mandate, while in staffing, there was an observed need for greater
technical advising. Thus, the overall assessment for this perspective, according to the scale
defined above, is: 3.
6.6.
Organisational Capacity Perspectives
The following section will evaluate the organizational capacity of the implementing agency,
looking at management and communication systems and knowledge and learning within
Tearfund.
6.6.1. Findings
Management and Communication
Table 12 below summarizes the findings of staff interviews and focus group discussions as
they relate to issues of management and communication.
Table 12: Staff feedback on management and communication
Coordinator/ Advisor Level
Field Project, HR and Logistic Level
Area of Assessment
Responses
Responses
Relationship to
Positive; communication systems
Very positive; close relationship with
Leadership
between Kabul and field offices were
leaders; daily communication with
adequate; relationships were open and
remote managers; clear pathways of
free; no hesitation in voicing thoughts or communication
opinions
Financial Management Positive; financial communication
Positive; field level managers received
generally on time, though noted some
timely information; budget revisions
delays after departure of the Finance
were smooth; everything was carried
Manager; external USG audit gave
out according to the timeline
DMT Afghanistan 100% success
Logistics
Mixed responses; central HR, logistics
Positive; security information always
and transport were all named
timely and up to date; Kabul office seen
specifically as being helpful and
as being “fully supportive”
competent; procurement processes
(and thus distribution processes) were
noted as being protracted and delayed
at times, to the detriment of the outputs
Knowledge and Learning
As noted above in Section 6.4.2, systems for allowing beneficiary feedback to inform project
implementation beyond the proposal development stage have yet to be developed. There is,
however, strong evidence to support that learning and experience from previous projects
directly influenced design and implementation of the project under review. The following are
specific examples of this influence:
• CLBS: In a previous BPRM-funded project, CLBS was introduced. A learning point from
that experience was that the project cycle was not long enough to ensure monitoring of
the handing over of the first generation of kids to secondary beneficiaries. Thus, in this
project, goats were selected who had either just delivered their first kid or who were just
months away from delivery.
23
Spinning wheels were introduced in this project due to both beneficiary requests and staff
observations during previous projects.
• After conducting seed distribution in a previously BPRM-funded project, it was found that
beneficiaries were planting them in scattered, disorganized places in their yards; thus,
land management training was introduced.
• Agriculture and animal husbandry training materials – in particular, the handbook
distributed to beneficiaries for their reference – were adjusted following previous projects.
It was found that the amount of text was too great for largely uneducated audiences, so
pictures and photos were increased.
• Per a concern raised in the external review of the first BPRM-funded project regarding
animal waste in yards, training was given pertaining to proper waste disposal and the
necessary distance between animal structures and water points.
These are a few of the many examples reported during the review of the way knowledge and
learning informed project development and implementation.
•
6.6.2. Conclusions
Upon reviewing the data collected during the survey, it is concluded that the management
and communication systems, the financial systems, and the logistical systems put in place by
DMT Afghanistan were appropriate and adequate for this project. Though one concern was
raised about procurement processes and other examples of activity delays were reported,
these are to be expected in the implementation of an integrated project in this context and do
not, in the opinion of the evaluator, represent an inherent fault in management or
communication systems.
In terms of knowledge and learning, numerous specific examples of incorporating learning
from past experience were shared and observed. In addition, it was clear to the evaluator
that staff remained alert to opportunities for future learning during the implementation of this
project. The learning process was seen to be part of the fabric of DMT’s strategy for working
effectively in Afghanistan.
6.6.3. Assessment
The review of Organisational Capacity Perspectives turned up strongly positive findings.
There were no major concerns raised. The evaluator’s overall assessment of this
perspective, according to the scale defined above, is: 4.
7. Specific Actionable Recommendations
Below is a summary of actionable recommendations based on analysis and observations
covered in this report. Each recommendation is cross-referenced to the relevant section in
the body of the text.
Table 13: Specific actionable recommendations
CrossSector/Intervention
Recommendation
Reference
6.2.1
Retrain project staff on operation and maintenance, with a particular
6.3.1
emphasis on cleaning practices (swirl and dump) and cleaning
6.3.2
frequency (when water flow is too slow). Have them discontinue the
teaching of washing the gravel and sand to communities. Community
members only need to know swirl and dump.
Biosand filters
6.3.2
Discontinue the practice of having beneficiaries install their own filters.
This should be done by a trained technician, and the filter should be
tested upon installation.
6.3.1
Adjust BSF monitoring practice. Updated BSF monitoring forms
improve upon the previous forms by adding detail, but they don’t
include checking for outlet spout length, which has proven to be a major
24
6.3.1
Water reservoirs
6.3.1
Water quality testing
6.3.1
6.5.1
Technical oversight
6.3.2
Vegetable gardening
training
6.3.1
Animal husbandry
training
CLBS
6.1.1
6.3.1
Spinning wheels
6.1.2
6.3.1
Training materials
6.3.1
6.6.1
Baseline survey
6.4.1
6.4.1
6.4.1
6.4.1
6.5.1
4.2
Project evaluation
25
concern. This parameter should be added back in to the new forms.
Flow rate measurement may be made easier as well by having
facilitators record the “Time to fill 0.4 litres” instead of the flow rate.
Times less than 1 minute can be flagged as problematic.
Review the design and construction. Specifically, improve silting and/or
filtration at the inlet, screening of openings, drainage, levelling of the
ground around, and installation of handpumps. Consultation with ZOA
regarding their design, which reportedly involves a multi-stage gravel
filter at the inlet, could prove beneficial.
Improve maintenance plans and increase community ownership of
them, making sure CDCs retain copies of documentation as well.
Specifically, outline plans, schedules and responsibilities for sediment
removal and handpump repair.
When the next refresher training or field testing is carried out, include a
section on proper, safe disposal of used membrane papers, which
present a risk of infection.
Maximize the presence of technical advisors in the field, to help
mitigate against quality control issues such as those noted in this
report. Ensure that their involvement is not limited to training but
includes side-by-side field work with project staff, where security allows.
Do not allow project timelines to dictate when planting takes place. This
results in failed crops and general confusion on the part of the
beneficiaries as to which seasons are optimal for which vegetables to
be planted. If seed procurement timing is off, save the seeds to be
planted the next season, maximizing their potential yield.
Discontinue the teaching of any irrelevant topics, given that
beneficiaries are receiving copious amounts of information in a short
period of time. Specifically, if the structures for chickens and goats that
have been taught are consistently too expensive for beneficiaries to
construct and are producing no results, find an alternative. The principle
of streamlining topics can be applied to the other sectors as well. A
specific example noted in the text was the irrelevance of solar water
disinfection to this project.
If attempting CLBS in the south again, try implementing it using the
system that the north is using, whereby CDCs identify secondary
beneficiaries and monitor the handover process.
Continue this activity, as able
Increase the use of posters for both vegetable gardening and animal
husbandry training, per project staff requests
As happened in the first revision of the vegetable gardening and animal
husbandry manual distributed to beneficiaries, continue to decrease
text and increase pictures to suit the needs of the audience.
Include all sectors in baseline surveys for all provinces
Include observation in baseline surveys
Make baseline surveys the same across the provinces for any given
project if interventions and goals are the same
Conduct refresher training on survey techniques. Include side-by-side
surveying in the field as a significant portion of the training.
Increase the time allotted for final project evaluation. When time
constraints prevent travel to more remote areas, a significant bias is
introduced.
8. Annexes
8.1.
Annex 1: Data and Analysis
See Electronic Attachment
26
8.2.
Annex 2: Evaluation Terms of Reference
See Electronic Attachment
27
8.3.
Annex 3: Profile of the Evaluation Team
This evaluation was conducted by Katie Toop, with the assistance of nationals.
8.3.1. Profile of Lead Evaluator
Katie is a resident of Kenmore, Washington, USA. As a
graduate of the Masters program at the Water, Engineering
and Development Centre (WEDC) of Loughborough
University, UK, Katie entered the relief and development
world as a water and sanitation specialist in 2005 and has
been active on the field since then. She served one year in
Sri Lanka (2005-2006) following the tsunami, helping to
build the capacity of a local Sri Lankan NGO to design and
sustain a water/sanitation program. After Sri Lanka, Katie
served in northern Afghanistan – Balkh province – from
2006-2011, first working with Afghanistan American
Friendship Foundation for two and a half years as a WASH
specialist and then working another 2 years with Joint
Development Associates, Int’l as WASH Program Director and later, Acting Country Director.
She left the field just 6 months prior to this evaluation.
Evaluation Experience
In terms of monitoring and evaluation experience, Katie’s strengths are in the field of WASH.
During her service in Sri Lanka, she conducted water quality monitoring visits for local and
international NGOs, surveyed and made recommendations about water supply options for
temporary and permanent housing projects and designed survey and monitoring forms for
WASH-related activities. In Afghanistan, Katie’s experience was with baseline and follow-up
surveys for the Joint Development Associate’s WASH program as well as project monitoring.
Educational Background
Oct., 2004
Loughborough University: Water, Engineering and Development Centre
Loughborough, Leicestershire, UK
Master of Science, Water and Environmental Management
Degree with Distinction
May, 2000
Bethel College
St. Paul, Minnesota, USA
Bachelor of Science, Physics
Summa Cum Laude
8.3.2. Profile of National Surveyors
To assist Katie in evaluation, two teams of surveyors were assembled by Tearfund – one in
the north, one in the south. A brief biography of each is below.
Northern Provinces – Jawzjan and Faryab
A team of three people – one male and two female – was put together.
Male Surveyor: Shamsuddin, son of Noor Mohammad
Shamsuddin is a resident of Sheberghan. He studied in the faculty of Law and Political
Science at Balkh University and is currently employed as Admin and Finance Manager for
STEP Health and Development Organization in Jawzjan. He is fluent in Dari, Pashto, Uzbeki
and Turkmani and has an intermediate level of English. His previous survey experience
includes two health sector surveys for USAID-funded projects. Shamsuddin’s other previous
work experience was in the areas of logistics and accounting.
28
Female Surveyor: Aziza, daughter of Noor Mohammad
Aziza is Shamsuddin’s sister and is a new graduate of 12th grade. She is fluent in Dari and
Uzbeki and has a beginning level of English (10 months of study). Aziza’s current job is as a
teacher for an English course in Sheberghan. She has no prior experience of surveying or
working in NGOs.
Female Surveyor: Marina, daughter of Mohammad Ibrahim
Marina is also a resident of Sheberghan and graduated 5 years ago from university. After
graduating, she worked 4 years with Save the Children and currently works as a teacher both
in a school and in a course. She is fluent in Dari and Uzbeki with a beginning level of English
(10 months of study). She has no prior experience with surveying.
Southern Province – Kandahar
In Kandahar, due to security and cultural restrictions, a family was employed to conduct the
surveys. Profiles of the surveyors were not completed or submitted in time for this report. The
family consisted of two men – one of whom demonstrated strong English skills, and his son,
who was also apparently an English-speaker – and three women. None of the women spoke
English.
29
8.4.
Annex 4: Evaluation Schedule
Prior to the evaluation, a number of activity schedules were proposed and accepted by both
Tearfund and the evaluator. Due to extenuating circumstances (visa issues, security,
weather, etc.), those schedules were altered several times during the course of the
evaluation. The table below reflects the actual timeline of events during the evaluation.
Date
Location
Activity
Persons Involved
Nov. 12
Nov. 13
Nov. 14
USA to UAE
USA to UAE
UAE to Kabul
Evaluator
Evaluator
Evaluator
M&E Officer
Security Officer
Nov. 15
Kabul to Mazar
Mazar to
Jawzjan
International Travel
International Travel
International Travel
Arrival to Tearfund DMT
Briefing with Monitoring & Evaluation Officer
Briefing with Security Officer
Domestic Travel
Domestic Travel – Road
Document preparation (print/copy) for survey
Surveyor training
Nov. 16
Faryab
Survey day
Nov. 19
Jawzjan
Survey day
Nov. 20
Jawzjan
Survey day
Nov. 21
Jawzjan
Interview with Project Manager, Jawzjan
Half day survey
Half day focus group discussion with staff
Interview with AC
Nov. 22
Faryab
Survey day
Nov. 23
Jawzjan to
Mazar
Half survey day
Travel day
Nov. 25
Nov. 26
Mazar to Kabul
Kabul
Nov. 27
Kabul to
Kandahar
Travel
Interview with Program Director
Kandahar briefing
Material preparation for Kandahar evaluation
Travel
Surveyor training
Nov. 28
Kandahar
Nov. 29
Kandahar to
Kabul
30
Survey day – Survey team only
Staff focus group discussion
Village elder focus group discussion
Beneficiary focus group discussion
Interview with Project Manager
Document Review
Travel to Kabul
Interview with Agriculturalist
Interview with M&E Officer
Evaluator
AC
Jawzjan Logistics
Jawzjan Surveyors
Evaluator
Jawzjan Surveyors
Project Staff
Evaluator
Jawzjan Surveyors
Project Staff
Evaluator
Jawzjan Surveyors
Project Staff
Evaluator
Project Manager
Jawzjan Surveyors
Project Staff
Area Coordinator
Evaluator
Jawzjan Surveyors
Project Staff
Evaluator
Jawzjan Surveyors
Project Staff
Area Coordinator
Evaluator
Evaluator
Program Director
M&E Officer
Evaluator
Area Coordinator
Kandahar Surveyors
Evaluator
Kandahar Surveyors
Project staff
Logistic staff
Evaluator
KDH Project Manager
Area Coordinator
Agriculturalist
M&E Officer
8.5.
Annex 5: Documents Consulted During the Evaluation
See electronic attachments for the following documents:
Annex 5A:
Annex 5B:
Annex 5C:
Annex 5D:
Annex 5E:
Annex 5F:
Annex 5G:
Annex 5H:
Annex 5K:
Annex 5L:
Annex 5M:
Annex 5N:
31
Project Proposal
Project Logical Framework
Beneficiary Selection Criteria
Jawzjan Pre-KAP Survey
Faryab Pre-KAP Survey
Kandahar Pre-KAP Survey
Jawzjan Post-KAP Survey
Faryab Post-KAP Survey
Kandahar Post-KAP Survey
Jawzjan Survey Summary
Mid-year Review
Quarter 4 Report
8.6.
Annex 6: Photographic Evidence
See Electronic Attachment
32
8.7.
Annex 7: List of Documents Reviewed On-Site
See Electronic Attachment
33