Using Indicators to Measure Progress
and Performance
Dominique Senn, seecon gmbh
Using Indicators to Measure Progress and Performance
1
Find this presentation and more on www.sswm.info
Copyright & Disclaimer
Copy it, adapt it, use it – but acknowledge the source!
Copyright
Included in the SSWM Toolbox are materials from various organisations and sources. Those materials are open source. Following the opensource concept for capacity building and non-profit use, copying and adapting is allowed provided proper acknowledgement of the source
is made (see below). The publication of these materials in the SSWM Toolbox does not alter any existing copyrights. Material published in
the SSWM Toolbox for the first time follows the same open-source concept, with all rights remaining with the original authors or producing
organisations.
To view an official copy of the the Creative Commons Attribution Works 3.0 Unported License we build upon, visit
http://creativecommons.org/licenses/by/3.0. This agreement officially states that:
You are free to:
• Share - to copy, distribute and transmit this document
• Remix - to adapt this document. We would appreciate receiving a copy of any changes that you have made to improve this
document.
Under the following conditions:
• Attribution: You must always give the original authors or publishing agencies credit for the document or picture you are using.
Disclaimer
The contents of the SSWM Toolbox reflect the opinions of the respective authors and not necessarily the official opinion of the funding or
supporting partner organisations.
Depending on the initial situations and respective local circumstances, there is no guarantee that single measures described in the toolbox
will make the local water and sanitation system more sustainable. The main aim of the SSWM Toolbox is to be a reference tool to provide
ideas for improving the local water and sanitation situation in a sustainable manner. Results depend largely on the respective situation
and the implementation and combination of the measures described. An in-depth analysis of respective advantages and disadvantages and
the suitability of the measure is necessary in every single case. We do not assume any responsibility for and make no warranty with
respect to the results that may be obtained from the use of the information provided.
Using Indicators to Measure Progress and Performance
Find this presentation and more on www.sswm.info
Contents
1. Introduction
2. What Are Indicators
3. The Selection of Indicators
4. Why Use Indicators
5. What to Use Indicators for
6. The Development of Indicators
7. Typical Indicators in SSWM
8. Advantages and Disadvantages
9. References
Using Indicators to Measure Progress and Performance
3
Find this presentation and more on www.sswm.info
1. Introduction
Every project and programme has specific objectives. In order to assess
the progress and performance of the project (e.g. to analyse whether
targets are met, to give account of the progress to donor agencies or to
show accountability to the public, funders, donors and stakeholder
groups), monitoring and evaluations need to be carried out.
Indicators
provide an
effective tool for
measuring
progress and
performance.
Source: http://www.clearhorizon.com.au/wp-content/uploads/2009/02/theproject25percent.jpg [Accessed: 23.10.2013]
Using Indicators to Measure Progress and Performance
4
Find this presentation and more on www.sswm.info
2. What Are Indicators
Definitions
• OECD/DAC: “A quantitative or qualitative factor or variable that
provides a simple and reliable means to measure achievement, to
reflect changes connected to an intervention, or to help assess the
performance of a development actor.”
• USAID: “A variable, which purpose it is to measure change in a
phenomena or process.”
• UKAID & United States Institute of Peace: An Indicator “refers to a
measure used to demonstrate change in a situation, or the progress
in, or results of, an activity, project, or program.”
• European Commission: “A description of the project’s objectives in
terms of quantity, quality, target group(s), time and place.”
Source: MDF 2005, UKAID & UNITED STATES INSTITUTE FOR PEACE n.y.
Using Indicators to Measure Progress and Performance
5
Find this presentation and more on www.sswm.info
2. What Are Indicators
Example
Specific indicators have been developed in order to measure progress
towards the Millennium Development Goals.
Goals and Targets
Indicators
Goal 1: Eradicate extreme poverty and hunger
Target 1.A: Halve, between 1990 and 2015, the proportion of people whose
income is less than one dollar a day
Target 1.B: Achieve full and productive employment and decent work for all,
including women and young people
Target 1.C: Halve, between 1990 and 2015, the proportion of
people who suffer from hunger
1.
2.
3.
1.
2.
3.
4.
1.
2.
Proportion of population below $1 (PPP) per day
Poverty gap ratio
Share of poorest quintile in national consumption
Growth rate of GDP per person employed
Employment-to-population ratio
Proportion of employed people living below $1 (PPP) per day
Proportion of own-account and contributing family workers in total
employment
Prevalence of underweight children under-five years of age
Proportion of population below minimum level of dietary energy
consumption
Goal 2: Achieve universal primary education
Target 2.A: Ensure that, by 2015, children everywhere, boys and girls alike, will 1.
be able to complete a full course of primary schooling
2.
3.
Net enrolment ratio in primary education
Proportion of pupils starting grade 1 who reach last grade of primary
Literacy rate of 15-24 year-olds, women and men
For the full list, see: http://mdgs.un.org/unsd/mdg/Host.aspx?Content=Indicators/OfficialList.htm
Using Indicators to Measure Progress and Performance
6
Find this presentation and more on www.sswm.info
3. The Selection of Indicators
(1/2)
Generally an indicator focuses on a small, manageable set of
information that gives a sense of the bigger picture.
The choice of indicators is important as to whether it gives sufficient
“sense of the bigger picture”.
 Only well selected indicators can assist in maintaining a focus on the
important work areas and take strategic decisions to address problem
areas.
The careful selection of indicators makes a project sustainable and
allows responsible persons to act.
Using Indicators to Measure Progress and Performance
7
Find this presentation and more on www.sswm.info
3. The Selection of Indicators
(2/2)
The term “minimum indicators” is used in recognition that:
• It is better to start with a small set of indicators that are feasible
to monitor and to improve over time.
• There are many other indicators that could be used to measure
progress beyond this basic level.
Using Indicators to Measure Progress and Performance
8
Find this presentation and more on www.sswm.info
4. Why Use Indicators
An indicator is used to answer the question:
How do we know if and how much we have achieved our objective?
Indicators are useful to:
• Measure progress and performance over time against various water
and sanitation objectives to evaluate the effect of policy actions and
plans.
• Provide information on the status, trend or performance of a given
system to inform the project team (responsible organisation),
stakeholders, the public and policy makers in a simplified way.
• Translate data into policy relevant information (describe, show
trends, communicate results of implementing objectives).
• Identify areas for increased attention by an organisation.
Using Indicators to Measure Progress and Performance
9
Find this presentation and more on www.sswm.info
6. The Development of Indicators
Criteria for Developing Indicators
Good indicators should meet certain criteria. Three different concepts
are presented on the following slides: SMART indicators, SPICED
indicators and CREAM indicators.
While SMART describes the properties of the indicators themselves,
SPICED relates more to how indicators should be used.
But keep in mind: the most important point in developing indicators is:
Don’t be too ambitious!  Start with what can be realistically done
or else failure is guaranteed.
Using Indicators to Measure Progress and Performance
10
Find this presentation and more on www.sswm.info
6. The Development of Indicators
SMART Indicators
SPECIFIC
The indicators need to be clearly articulated, well defined
and focused. They should be clear to people with a basic
knowledge of the issue, programme or initiative.
M
MEASURABLE
The indicator should be measurable, that is, it has the
capacity to be counted, observed, analysed, tested, or
challenged. The indicators should be able to determine the
degree of completion or attainment. Using the same
methodology and information, findings should be
repeatable and comparable.
A
ATTAINABLE
The target attached to the indicator should be achievable
within the scopes of the project/ programme.
R
RELEVANT
The indicators should be able to detect change and be
related to the specific situation they are “indicating”
information about. They should be measurable at an
appropriate scale.
T
TIME-BOUND
The indicator should be attached to a time frame. Term
dates for measurement should be included.
S
Source: Save the Children, https://sites.google.com/site/savethechildrendme/Home/smart-indicators [Accessed: 17.12.2013]
Using Indicators to Measure Progress and Performance
11
Find this presentation and more on www.sswm.info
6. The Development of Indicators
SPICED Indicators
S
P
I
C
E
D
SUBJECTIVE
Informants have a special position or experience that gives them unique
insights which may yield a very high return on the investigators time. In
this sense, what may be seen by others as 'anecdotal' becomes critical
data because of the source’s value.
PARTICIPATORY
Indicators should be developed together with those best placed to
assess them. This means involving a project's ultimate beneficiaries, but
it can also mean involving local staff and other stakeholders.
INTERPRETED &
COMMUNICABLE
Locally defined indicators may not mean much to other stakeholders, so
they often need to be explained.
CROSS-CHECKED &
COMPARED
The validity of assessment needs to be cross-checked, by comparing
different indicators and progress, and by using different informants,
methods, and researchers.
EMPOWERING
The process of setting and assessing indicators should be empowering in
itself and allow groups and individuals to reflect critically on their
changing situation.
DIVERSE &
DISAGGREGATED
There should be a deliberate effort to seek out different indicators from
a range of groups, especially men and women. This information needs to
be recorded in such a way that these differences can be assessed over
time.
Source: MDF 2005
Using Indicators to Measure Progress and Performance
12
Find this presentation and more on www.sswm.info
6. The Development of Indicators
CREAM Indicators
Indicators should be...
C
R
E
A
M
CLEAR
...Precise and unambiguous
RELEVANT
...Appropriate to the subject at hand
ECONOMIC
...Available at reasonable cost
ADEQUATE
...Able to provide sufficient basis to assess performance
MONITORABLE
...Amenable to independent validation
Source: UKAID & UNITED STATES INSTITUTE FOR PEACE n.y.
Using Indicators to Measure Progress and Performance
13
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
(Adapted from MUGA and MIHELCIC 2008)
• Functional indicators (e.g. adaptability, durability and reliability of
the system)
• Economic indicators (e.g. capital costs, operation and maintenance
costs and user costs)
• Environmental indicators covering resource use (e.g. nutrient
reuse) and emissions (e.g. emissions of carbon dioxide)
• Social-cultural indicators (e.g. public participation, acceptance,
and stimulation of sustainable behaviour)
Using Indicators to Measure Progress and Performance
14
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
Examples of Indicators for Water Sources Management
• Water withdrawal: absolute or per-person value of yearly water
withdrawal gives a measure of the importance of water in the
country’s economy.
• Ground water level development (rise or fall of groundwater levels,
pollution).
• Water sources quality (e.g. turbidity, minerals content, pathogens,
chemical quality).
Water quality assessment with different indicators (biodiversity, chemical
substances, nutrients, toxics, sediments, etc.).
Source: http://www.epa.gov/region1/lab/reportcardintro.html [Accessed: 10.12.2013]
Using Indicators to Measure Progress and Performance
15
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
Examples of Indicators for Water Purification
• Treatment efficiency indicator (e.g. removal of indicator bacteria)
Removal efficiency of organic
nitrogen in different
treatment methods (1nitrification, 2-oxidation pond,
3-chemical coagulation, 4chlorination, 5-ammonia
removal, 6-filtration, 7reverse osmosis).
Source: http://www.intechopen.com/books/watertreatment/relationship-of-algae-to-water-pollution-andwaste-water-treatment [Accessed: 10.12.2013]
Using Indicators to Measure Progress and Performance
16
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
Examples of Indicators for Water Use
• Percentage or number of people not served with improved drinking
water and extension of piped water supply.
Number of household
connections per 100 residents
in urban sub-Saharan Africa.
Source: WSP & WUP 2003
• Domestic water use efficiency (per capita use of water).
• Agricultural water use efficiency and efficacy (water demand per
irrigated area).
• Industrial water use.
• Awareness and actual use of water-saving appliances.
Using Indicators to Measure Progress and Performance
17
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
Examples of Indicators for Wastewater Collection
• Percentage of wastewater collected to total wastewater
production.
Particularly in peri-urban and
rural areas in developing
countries, wastewater is rarely
collected, representing high
health risks to the people
living in these aras.
Source: http://www.ens-newswire.com/ens/aug2007/200708-07-01.asp
Using Indicators to Measure Progress and Performance
18
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
Examples of Indicators for Wastewater Treatment
• Percentage or number of people not served with basic sanitation.
Percentage of households
having a sewerage connection
in sub-Saharan African cities.
Source: WSP & WUP 2003
• Number of people affected by water borne diseases (e.g.
diarrhoea, cholera).
• Investment in drinking water supply and sanitation.
• Removal of pollutants.
• Awareness and actual use of safe sanitation systems.
Using Indicators to Measure Progress and Performance
19
Find this presentation and more on www.sswm.info
7. Typical Indicators in SSWM
Examples of Indicators for Reuse and Recharge of Nutrients and Water
• Percentage of reclaimed water used in agriculture to total
agricultural water use.
• Percentage of nutrients used from recycled sources.
Proportion of recycled water
to withdrawn water.
Source:
http://www.potashcorp.com/annual_reports/2012/gr
i/environment/#EN8 [Accessed: 10.12.2013]
Using Indicators to Measure Progress and Performance
20
Find this presentation and more on www.sswm.info
8. Advantages and Disadvantages
• Failures can be detected and
eliminated.
• Indicators can be adapted to
the circumstances.
• It is not necessary to measure
every single parameter to get
a “bigger picture” over the
whole progress.
Using Indicators to Measure Progress and Performance
• A set with too many indicators
will tend to clutter the
overview.
• Measuring indicators is a longterm exercise. If the
indicators are not defined
properly, and if measuring is
not done correctly and
conscientiously, it will not
work.
21
Find this presentation and more on www.sswm.info
9. References
Business Dictionary. URL: http://www.businessdictionary.com/definition/indicator.html#ixzz2mVtidEG6 [Accessed:
04.12.2013]
MDF (Editor) (2005): MDF Tool: Indicators. Ede: MDF Training and Consultancy. URL:
http://www.undg.org/docs/11652/MDF-Indicators-Brief-%282005%29.pdf [Accessed: 04.12.2013]
MUGA, H.E.; MIHELCIC, J.R. (2008): Sustainability of Wastewater Treatment Technologies. In: Journal of Environmental
Management 88, 437 - 447. URL: http://www.ncbi.nlm.nih.gov/pubmed/17467148 [Accessed: 23.10.2013]
TAYLOR, P.; LIDEN, R.; NDIRANGU, W.; JIN, L. (2008): Integrated Water Resources Management for River Basin
Organisations – Training Manual. Pretoria: International Network for Capacity Building in Integrated Water Resources
Management (Cap-Net). URL: http://www.cap-net.org/sites/cap-net.org/files/RBO%20Manual.pdf [Accessed:
23.10.2013]
UKAID & UNITED STATES INSTITUTE FOR PEACE (n.y.): Indicator Module. URL:
http://dmeforpeace.org/sites/default/files/3.9%20Indicators.pdf [Accessed: 10.12.2013]
WUP and WSP (2003): Better Water and Sanitation for the Urban Poor. Good Practice from Sub-Saharan Africa. Kenya:
Water Utility Partnership (WUP) and The World Bank Water and Sanitation Program (WSP). URL:
http://www.wsp.org/sites/wsp.org/files/publications/330200725049_afBetterWaterandSanitationForTheUrbanPoorG
oodPracticeFromSSA.pdf [Accessed: 13.09.2013]
ZEWO (Editor) (n.y.): Outcome and Impact Assessment in International Development. Zewo Guidelines for Projects and
Programmes. Zurich: Schweizerische Zertifizierungsstelle fuer gemeinnuetzige, Spenden sammelnde Organisationen
(ZEWO). URL: http://www.zewo.ch/impact/en/impact [Accessed: 23.10.2013]
Using Indicators to Measure Progress and Performance
22
“Linking up Sustainable Sanitation,
Water Management & Agriculture”
SSWM is an
initiative
supported by:
Created
by:
Using Indicators to Measure Progress and Performance
23