Looking Back
Participatory Impact Assessment of Older Projects
Linkage diagram, Panjapatti community, Panjapatti Panchayat, Karur District, Tamil Nadu, India
A revised final document for peer review by study team members
July 2000
Looking Back Study: Final report (draft)
1
The principal authors of this report include:
Aderie Adugna
Martin Dery
Joe Gomme
A Kalimuthu
Herbert J Kashililah
Meaza Kebede
Evans Sasu Owusu
N Radha
Susan Maganga
Darren Saywell
Looking Back Study: Final report (draft)
EOC, Ethiopia (withdrawn from study, November 1999)
ProNet, Ghana
WaterAid, Madagascar
WaterAid, India
Water Department – Dodoma, Tanzania
WaterAction, Ethiopia
ORAP, Ghana
LEAD, India
WaterAid, Tanzania
Water Engineering and Development Centre, UK
2
Executive Summary
Introduction
This document reports findings from a WaterAid funded project (‘Looking Back’) concerning
the longer term impact on communities of project interventions. Results arising from
fieldwork in the four study countries (Ethiopia, Ghana, India and Tanzania) are discussed. In
addition to serving as a resource for WaterAid and its partner organisation staff, this report
has been written for various external audiences – decision makers and professional staff in
development agencies, NGO’s and government.
The purpose of the report is not only to detail the outcomes from the Looking Back study so
that others can learn lessons arising from it, but to document the process, and comment on the
methods employed during the research. In this way, the material presented in the document
can help to inform future strategic thinking and messages for advocacy campaigns and at the
same time act as a reference point for those planning similar studies in the water and
sanitation sector.
Impact assessments have gained in popularity with donor agencies in recent years and have
subsequently become an increasingly important activity for recipient organisations. This
preoccupation reflects a shift in thinking away from the inputs and outputs of conventional
evaluations to a consideration of the outcomes of interventions.
WaterAid carries out periodic evaluations of the work it supports overseas. Before this
project, however, it had sponsored no studies that directly examined the long-term impact of
its partners’ water and sanitation projects on the communities they serve. On the basis of
measures of its outputs, WaterAid has assumed that projects led to ‘sustainable
improvements’, but felt that this assumption needed to be tested. WaterAid therefore decided
to initiate assessment of the impact of its programmes on the communities it affects. The
Looking Back study differs from previous WaterAid evaluations in that rather than
emphasising the effects or outputs of a project (“is the water still flowing?”), its focus is
clearly on the impact it has (“what changes in community life can be attributed to the flowing
water?”).
Hypotheses, study objectives, and work schedule were all discussed in detail with project
partners prior to and during an initial workshop. Four study teams worked on the Looking
Back project, with members drawn from WaterAid and its partner organisations in Ethiopia,
Ghana, India and Tanzania. For each country, two individuals comprised the study team and
in all cases field assistants (typically four in number) were employed to facilitate fieldwork.
An external consultant (supported by a wider team) from the Water Engineering and
Development Centre (WEDC) at Loughborough University in the UK acted as a process
facilitator for the study. Project management was lead by a WaterAid staff member based
overseas, with e-mail and fax being used to provide feedback and guidance throughout the
research process. In general, a decentralised approach to project management was adopted,
with study teams being free to develop ideas in their own ways, as they judged appropriate for
their study communities. The selection of communities for inclusion in the Looking Back
study began with project partners drawing up a shortlist for further consideration. Loose,
‘purposive’ criteria for selection were subsequently adopted.
Looking Back Study: Final report (draft)
3
Methodology
In attempting to develop a methodological approach that best met the needs of impact
assessments of this kind, the Looking Back study has not tried to identify the optimal method,
but a mix of methods that are appropriately combined. Thus, key elements from the scientific
method (which focuses on representativeness, quantification and attribution) and the
humanities or participatory approaches (which focus on ability to uncover process, capture
perceptions, unexpected impacts) have been married together.
In this study, several general observations can be made about the methodological approach:
 The study involved a pre-test of the methodology.
 The study focused on individual, household, community, enterprise and institutional levels
of analysis.
 A key consideration in project design was that community members, rather than project
team members, identify indicators of impact and describe the outcomes of project
interventions from their own perspective, free from the filtering interpretation of an
external agent.
 One of the key assumptions underpinning the study was that the assessment would be
participatory.
 Gender and social grouping perspectives on impact assessment were emphasised, and the
selection and conduct of participatory techniques was geared to achieve this objective.
 The study teams identified non-beneficiary communities similar in characteristic to
reference communities. Both were then asked to recall past and present situations (before
the time of intervention), and analyse the types and extent of change.
More specific methodological issues included:
 In order to minimise costs, staff inputs and logistics, a sampling strategy was followed for
the Looking Back study. This operated on two levels: selection between communities and
selection of individuals/groups within communities. The key point to stress is that the
study aimed to understand in-depth what, if any, impact communities identified and to
explore the process of gathering impact indicators. Small sample sizes and (typically) nonrandom sampling methods were therefore adopted.
 Where possible, study teams sought and analysed available baseline data for the selected
projects. These included reports, surveys and file notes (from WaterAid, partner
organisations, health centres, water committees, etc) on the villages that would provide
adequate data for comparison purposes. In most cases, study teams were unsuccessful in
gathering information of significant value.
 It was agreed amongst the study teams that the study would not attempt to prove causation,
rather to look at credible causation based on community understanding of the relationship
between cause and effect. General indications on the qualitative importance of different
interventions were therefore examined.
 A series of measures were employed to improve the reliability and validity of the data
collection process, including triangulation, multidisciplinary teams, mixed methods and
feedback sessions with community members.
 A pretest of the methodology prior to the main research fieldwork stage was conducted as a
way to trial the draft methodology, to trial the use of particular tools at community level
Looking Back Study: Final report (draft)
4
and to provide study teams with the confidence required in conducting this type of
assessment.
 The key focus in this study has been the process of facilitating communities to identify,
discuss and reach consensus on those indicators (expressed, proxy, process, etc) which
reliably measure impacts which the community perceive to be significant. Indicators from
different groups within the community were sought and recorded, as were negative impacts
arising from project interventions. Tools used to initiate discussion about general changes
in the community over time (e.g., history line) were combined with more specific
approaches to probe for detail on impact and indicators of impact. In this way, a picture of
what the community considered important change, and how they knew these changes had
occurred (indicators), emerged during the study. It was noticeable that many of the impacts
identified have only been revealed through exploring issues in a participatory way with
community members.
Impact assessment
This section of the report is divided into two parts – an assessment of impact within study
countries and an assessment of impact across countries (by theme).
Each country case study includes three common elements. A background section provides an
overview of the country programme and sets the broad framework in which the findings can
be placed. A description of how the research was conducted follows, which focuses on
location of reference communities, review of tools employed and any constraints study teams
may have faced. The third component is the description and analysis of the impacts expressed
by community members. This includes an appraisal of the impact indicators identified and
relevant field insights (quotations, case histories) where appropriate.
The thematic impact section aggregates case study findings from the four countries. The
range of themes identified includes livelihoods, socio-cultural, health and hygiene,
psychological, education, management, gender, empowerment and sustainability issues.
Discussion under each of these headings is organised into sub-themes for more focused
analysis. Examples of selected findings are listed below:
 Reallocation of time (particularly women’s time) away from water collection to existing or
new activities, such as livelihood activities, observance of social obligations or attendance
at school
 Reduced incidences of morbidity and mortality (self-reported), with particular impact on
women's and children’s health
 Provision and easy access to safe water played a significant role in reduction of tension and
anxiety (for both men and women) regarding the physical availability of water, or concern
about the welfare of women and girls searching for water far from a village
 Rates of school enrolment, attendance and absenteeism improved in many reference
communities as a result of reduced need for children to search for water. In some
instances, academic performance improved because children stayed longer in classes
during the day, and teachers were able to focus on the curricula rather than fetching water
for students
 Gender roles were observed to change in some communities, with men and women’s role
becoming interchangeable. This was particularly noted in those communities where strong
female empowerment processes had been experienced.
Looking Back Study: Final report (draft)
5
The preliminary findings from the study demonstrate the capacity that communities have to
undertake complex analytical tasks to arrive at their own conceptualisation of impact. An
interesting exercise has been for the study teams to compare their own perceptions of impact,
with the indicators of change and impact actually recorded by communities.
It became clear during the course of the study that there have been impacts which could not
have been foreseen by the study teams, and which have only been revealed by communities
identifying impact themselves. It is not only the identification of a category; it is the extent
and scale of impact that was unforeseen by the study team. The range of socio-cultural
impacts and their penetration into all aspects of community life has been particularly
informative.
Lessons learned
The study disaggregated lessons learned by those relevant to the hypotheses for the study, to
water and sanitation programmes, those applicable for participatory impact assessments and
lessons from managing a study of this kind. Key findings from each of these categories are
included below.
Hypotheses for the study
Hypothesis 1: Projects constructed and managed by communities have a positive impact on
the living standards of those communities, particularly in the areas of health (especially of
children), economic status (especially of women) and school attendance
Findings from the study indicate that this hypothesis is proven. Health impacts are
notoriously difficult to attribute to specific interventions. However, consensus emerged
across the study teams that the health impacts identified by community members focused on
the incidence of disease, improvement in women’s and children’s health, personal cleanliness
practices, household cleanliness and increased health/hygiene awareness. In all of the
reference communities studied, there were clear signs of improvement to livelihoods,
particularly in asset ownership, growth in new livelihood activities, changes in purchasing
power and changes to income. In terms of attribution, community members across the
country case studies appeared confident in linking drinking water interventions to
improvements in economic status. A strong feature of the study findings is the impact that
time and labour saved from water collection has on the potential to change economic status,
especially for women. In a number of communities, an improvement has been observed in the
numbers attending community schools, in child retention when at school and subsequent
levels of absenteeism. The relationship between this impact and the availability of drinking
water is strongly identified by communities themselves.
Hypothesis 2: Project impact is less for the poorer sections of the community, but greater for
women and children than for men
Findings from the study suggest that this hypothesis is not proven. Across the case studies,
the study teams found that the distribution of impacts between different groups in the
community varied, but that it was difficult to arrive at the conclusion that poorer households
and men benefited less than women and children.
Hypothesis 3: Beyond the immediate, positive effects of education on improving sanitation,
the environmental impact of projects on their communities is negligible.
Looking Back Study: Final report (draft)
6
The study findings indicate that this hypothesis is not proven. It is evident that the
environmental impact in several projects has been both positive and significant, leading to for
example, a greening of community surroundings and generally cleaner household/community
surroundings.
Hypothesis 4: Impact depends more on effective management than on technical quality of
works.
Findings from the study indicate that this hypothesis is not proven. In general, the team
member’s interpretation is that impact depends equally on both technical quality of works and
effective management.
Hypothesis 5: Impact of projects is not associated with a longer period of provision of support
to community organisations.
Findings from the study indicate that the hypothesis is not proven. The general perspective of
the study teams is that continued and on-going support to community organisations facilitated
sustained development within the communities. Projects that have been in existence for a
longer period tend to have made a greater impact on the people than those that are relatively
new. However, it should be noted that there were no prescribed parameters applied to define
the time duration implied by a ‘longer period’ in the study. The phrase longer period was
undefined because it merely indicated relatively. The suggestion is that one project, with a
longer period of support, will have greater impact than another with a shorter period.
Water and sanitation (‘watsan’) programmes
 If WaterAid and its partner organisations continue to work in remote and isolated regions,
they need to see their interventions in relation to other (local) development organisations
in order to optimise their impact on the livelihood of the target communities.
 The impact of the interventions turned out to be farther reaching than initial project
objectives. The socio-economic dimension to impact was particularly noted, and from the
beginning of projects, WaterAid and its partner organisations should be planning for a
wide variety of impacts.
 In the absence of alternative water sources, rural communities use domestic water supply
for watering livestock. Consequently, watsan programmes need to take into account the
demand for water for the livestock population associated with the project.
 The development of long term relationships between WaterAid and partner organisations,
and between partners and communities was a significant factor in sustaining the drinking
water facility created by watsan programmes.
 Community members are capable of introducing their own management system(s) for a
sustained operation of facilities, although they do not necessarily follow established rules
pertaining to the management of water points, e.g. opening of bank accounts.
 Communities are capable of evolving their own internal structures to ensure proper
management and sustainability of water and sanitation facilities. Straight-jacket
prescriptions for the establishment of particular structures at community level may be
counter-productive. Communities clearly need to be more involved in decision-making
processes in the planning and implementation of projects.
Looking Back Study: Final report (draft)
7
Participatory impact assessments
 It is feasible and practical for communities to generate indicators to assess the impact of
projects on their daily lives without assessors identifying pre-determined indicators. In
many instances, project teams found that indicators of impact were spontaneously
identified. This in turn has led to the explanation of impacts that were farther reaching
than initial project objectives.
 To be effective, this methodological approach requires skilful application of several
participatory tools.
 The length of time to conduct this type of study for the range of hypothesis being
examined needs to be increased. Two weeks was adequate for a limited pretest exercise,
but the length of inputs recommended for the main research fieldwork and report writing
stages needs to be critically reviewed. In this study, as part B highlights, an average
figure closer to 14 weeks continuous input was required.
 Some project teams found that expressed indicators of change were difficult to establish
on the basis of the information collected. Proxy indicators are useful for triangulation and
data checking purposes, although they were not used exclusively for this purpose in the
Looking Back study. The impact was that project teams needed to increase time inputs for
triangulation and data checking activities.
 People’s enthusiasm and expectations can become inflated through the process of
participating in PIA. In this study, project teams acknowledged this risk and planned to
manage these expectations by preparing guidelines for rehabilitation / additional work.
Benefits and challenges of Looking Back style study
 The study design and approach provided for a two-way learning opportunity between the
community and the researchers. It has enhanced project team’s experience with applying
participatory tools in the field. For some of the field assistants, it provided an opportunity
to develop experience with using PRA tools.
 It has also exposed team members to traditional community knowledge and challenged
professional perceptions. New skills and expertise have arisen from the study which are
of benefit beyond the individuals involved to participating organisations as a whole.
 The opportunity for dialogue and collaborative work between partner organisations and
regional WaterAid offices has strengthened their relationship. The experience of visiting
country programmes as part of the process of developing the methodology has enhanced
opportunities for learning across project teams. Additionally, WaterAid UK has
recognised the value of inter-country collaboration as a means for developing new ideas /
studies.
 The study provided an opportunity for partner organisations to look objectively at their
interventions. This facilitated the process of disaggregating impact from watsan
programme and other interventions (within and between agencies).
 The number of hypotheses for the study was considered too ambitious in the time
available. Project team members commented that this type of participatory assessment
generated large quantities of information that needed to synthesised and categorised
appropriately. Completing triangulation of data within the time available proved
problematic.
Looking Back Study: Final report (draft)
8
Contents
3
Executive summary
11
13
14
16
Acknowledgements
List of tables, figures and boxes
Glossary
Acronyms
17
Part A: Introduction
18
18
20
21
22
25
27
Overview to Part A
Introduction
Hypotheses and objectives
Study work schedule
Study teams
Study communities
Structure to the report
29
Part B: Methodological approach
30
30
36
38
41
43
Overview to Part B
Review of methodological approach
Sequencing of methodological steps
Pre-testing and main research fieldwork phases
Review of tools used in impact assessment
Development of indicators
45
Part C: Impact assessment (country case studies)
46
47
Overview to Part C
Country case studies
47
58
67
80
- Ethiopia
- Ghana
- India
- Tanzania
93
Part C: Impact assessment (Thematic impact)
94
94
94
Overview to part C (ii)
A note in impacts identified
Thematic impacts
94
- Livelihoods
Looking Back Study: Final report (draft)
9
97
99
104
105
107
109
110
- Socio-cultural
- Health and hygiene
- Psychological
- Education
- Gender
- Management
- Sustainability
113
Part D: Lessons learned and discussion
114
114
120
122
124
Overview to Part D
Lessons in relation to Looking Back study hypotheses and objectives
Lessons/observations for water and sanitation programmes
Lessons for participatory impact assessments
Benefits and challenges of Looking Back style study
126
Part E: Discussion and conclusions
130
Annexes
131
133
134
135
136
139
140
142
150
158
1
2
3
4
5
6
7
8
9
10
Looking Back Study: Final report (draft)
List of community members
Provisional agenda for Ethiopian workshop
References
Example of pretest action plan (Ethiopia)
Methodology briefing note
Example of triangulation matrix
Example of guidelines for rehabilitation / additional work
Example of outcomes from tools employed in Looking Back study
Recommendations for further work
Diagram explaining causality (as understood by community)
10
Acknowledgements
The authors would like to thank the following for the significant contribution they made to the
completion of the Looking Back study.
Communities
Members of:
Birshign Village, North Gondar, Ethiopia
Anker & Adeza Village, North Gondar, Ethiopia
Chandiba Village, North Gondar, Ethiopia
Rimecha Michiko Village, Arsi Zone, Oromia region, Ethiopia
Hate Tulu Peasant Association, Hitosa watsan project, Ethiopia
Iteya Shaki Community, Hitosa watsan project, Ethiopia
Songambele Village, Kongwa district, Dodoma region, Tanzania
Berege Village, Mpwapwa District, Dodoma region, Tanzania
Chaludewa Village, Mpwapwa District, Dodoma region, Tanzania
Tandala Village, Kondoa District, Dodoma region, Tanzania
Mpraeso Amanfrom community, Kwahu South District, Eastern region, Ghana
Kwaku Dwira community, west of Nkawkaw, Eastern region, Ghana
Ekawso community, near Nkawkaw, Eastern region, Ghana
Atwedie community, Eastern region, Ghana
Arappalipatti Village, Vayalur Panchayat, Kaur District, Tamil Nadu, India
Aggragampatti community, Pothuravuthanpatti Panchayat, Kaur District, Tamil Nadu, India
Kullampatti community, Pothuravuthanpatti Panchayat, Kaur District, Tamil Nadu, India
Panjapatti community, Panjapatti Panchayat, Kaur District, Tamil Nadu, India
A complete listing of all community members is provided in the Annexes.
Study teams
Meaza Kebede, Planning, Monitoring and Evaluation Expert, WaterAction, Ethiopia (PT)
Aderie Adugna, Monitoring and Evaluation Division Head, EOC, Ethiopia (PT)
(Withdrawn from study, November 1999)
Martin Dery, Project Manager, ProNet, Ghana (PT)
Evans Sasu Owusu, Snr Supervisor for Social Development, ORAP, Ghana (PT)
Mahama A. Tampurie (FA)
Eunice Adade (FA)
N Radha, Executive Director, LEAD, India (PT)
A Kalimuthu, Technical Officer, WaterAid, India (PT)
Herbert J Kashililah, O&M Engineer, Water Department – Dodoma, Tanzania (PT)
Susan Maganga, Monitoring and Information Officer, WaterAid, Tanzania (PT)
Looking Back Study: Final report (draft)
11
James Bujiku, Health Officer (FA)
Farida Msilu, Water Techician (FA)
Mustafa Mbughu, Community development officer (FA)
Thanks to the following WaterAid staff…
Joe Gomme, Country Representative, WaterAid, Madagascar
Habtamu Gessesse, Country Representative, WaterAid, Ethiopia
Jean MacGrory, Evaluations Officer, WaterAid, UK
Dave Mather, Country Representative, WaterAid, Tanzania
Shunmuga Paramasivan, Country Representative, WaterAid, India
Thanks to the following WEDC staff…
Sue Coates
M Sohail Khan
Darren Saywell
Brian Skinner
Ian Smout
Lastly, thanks to all the colleagues of Looking Back team members who took up the slack and
supported the team members during the study period.
Looking Back Study: Final report (draft)
12
List of tables, figures and boxes
Tables
1.
Study work schedule
24
Figures
1.
Structure to Looking Back report
2.
Methodological sequence for Looking Back study
27
37
Boxes
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
20
22
25
26
30
39
42
48
49
60
61
69
72
83
85
150
152
153
154
155
Hypotheses and objectives for Looking Back study
Key criteria for staff selection on Looking Back study
Tanzanian study team community selection criteria
Indian study team community selection criteria
Description of simple approach to impact assessment
Common elements from study team pretest action plans
Review of benefits/limitations of tools used during study
PRA and other tools employed in the Ethiopian case study
Community identified indicators recorded during Ethiopian study
PRA and other tools employed in the Ghanaian case study
Community identified indicators recorded during Ghana case study
PRA and other tools employed in the Indian case study
Community identified indicators recorded during Indian case study
PRA and other tools employed in the Tanzanian case study
Community identified indicators recorded during Tanzanian case study
Possible list of questions posed by community in relation to additional work
Ethiopian case study recommendations for rehabilitation / additional work
Ghanaian case study recommendations for rehabilitation / additional work
Indian case study recommendations for rehabilitation / additional work
Tanzanian case study recommendations for rehabilitation / additional work
Looking Back Study: Final report (draft)
13
Glossary
Baseline study: a collection of data about the characteristics of a population before a
programme or project is established. This data can then be compared with a study of the
characteristics carried out later in order to see what has changed and/or as part of a monitoring
system.
Beneficiaries: the people actually benefiting from, or intended to benefit from, a specific
intervention or project.
Community: all the people living in a specific locality.
Impact: the change, for instance in the lives of the beneficiaries, as a result or a project or
intervention. This impact may be direct or indirect as a result of the intervention.
Impact assessment: study that aims to understand changes (intended or unintended, known
and unknown) experienced by different stakeholders as a result of project interventions and to
capture qualitative as well as quantitative aspects of such changes.
Indicators: ways of measuring (indicating) that progress is being achieved. These must relate
to the aims and objectives of the project or piece of work. Indicators are evidence that
something has happened or an objective has been achieved; they are not proof.


Impact indicators are needed to assess what progress is being made towards reaching
the objectives and what impact the work has had on different groups of people
Proxy indicators measure things that represent (or approximate) changes in areas that
cannot be measured directly
Intervention: an initiative by an outside agency.
Method: manner, technique of doing something, for instance, gathering information on
hygiene practices
Outcome: the actual or visible result of an activity/project.
Output: the specific results of the input, products or systems that a project delivers, that are
expected to produce intended impacts.
Participation: active involvement in an activity or project.
Participatory approaches: approaches involving the participation of staff, partners, people
affected by a piece of work, managers and others in the planning and carrying out of (for
example) an assessment, monitoring, review or evaluation exercise.
Qualitative research: a flexible, open-ended method of building up an in-depth picture of a
situation, community, etc; methods used include observation and discussion.
Looking Back Study: Final report (draft)
14
Quantitative research: research used to collect data that can be analysed in a numerical form;
things are therefore either measured or counted, or questions are asked according to a defined
questionnaire so that the answers can be coded and analysed numerically.
Reference community: those locations where a previous watsan intervention had taken place.
Sustainability: the extent to which either project activities, or the outcomes they are expected
to bring about, continue after a project ceases.
Tool: instrument. For example, an informal interview schedule is a tool for collecting
information, as is a questionnaire. A tool can be applied using a variety of methods or
techniques.
Triangulation: is a method to overcome the problems that stem from studies relying on a
single theory, single method, single set of data and single investigator. It is possible to
achieve triangulation within a qualitative inquiry strategy by combining different kinds of
qualitative methods, mixing purposeful samples and including multiple perspectives. It is also
possible to cut across inquiry approaches and achieve triangulation by combining qualitative
and quantitative methods.
Looking Back Study: Final report (draft)
15
Acronyms
ACORD
Agency for Cooperation and Research in Development
EOC
Ethiopian Orthodox Church
GRAAP
Group de Recherche et d’Appui pour l’Auto-promotion Paysanne
LEAD
League for Education and Development
NGO
Non Governmental Organisation
ORAP
Oboomaa Rural Action Project
PIA
Participatory Impact Assessment
PRA
Participatory Rural Appraisal
RRA
Rapid Rural Appraisal
Looking Back Study: Final report (draft)
16