Module 1:
An Overview of Important
Aspects of Evaluation
- What to Evaluate?
Central American Regional Training Workshop on Indoor
Air Pollution and Household Energy Monitoring, Antigua,
Guatemala, 2-6 May 2005
Department for the Protection of the Human Environment
World Health Organization
Acknowledgement: This presentation was prepared by Eva Rehfuess, WHO and
Jonathan Rouse, seconded to WHO by the University of Loughborough in collaboration
with Nigel Bruce and Kirstie Jagoe from the University of Liverpool, and David Pennise
from the Center for Entrepreneurship in International Health and Development.
• Multiple impacts of household energy
• Focus on indoor air pollution and health
• How to decide on what to monitor?
Multiple Impacts of Household Energy
Multiple Impacts of Household Energy
POVERTY
POVERTY
•• Reduces
Reducesability
abilityto
toswitch
switchto
tocleaner
cleanerfuels
fuels
•• Inefficient
Inefficientuse
useof
ofpolluting
pollutingsolid
solidfuels
fuels
restricts
restrictseconomic
economicdevelopment
development
INCOME
INCOME
4
5
• • Simple
Simplefuels
fuels
restrict
restrict
opportunities
opportunities
• • Lack
Lackof
oftime
time
• • Poor
lighting
Poor lightingand
and
home
environment
home environment
WOMEN
WOMEN
3
5
• • Opportunity
Opportunitycost
costof
of
lost
time
lost time
•• Injuries
Injuriesand
andassault
assault
during
fuel
during fuel
collection
collection
•• Decision
Decisionmaking
making
HEALTH
HEALTH
Household
Energy
•• ARI,
ARI,COPD,
COPD,lung
lung
cancer
(coal)
cancer (coal)
•• TB,
TB,low
lowbirth
birthweight
weight
•• Burns
Burns
•• Poisoning
Poisoning
(kerosene)
(kerosene)
2
3
ENVIRONMENT
ENVIRONMENT
• • Local:
Local:deforestation,
deforestation,
erosion,
erosion,
desertification
desertification
• • Global:
Global:methane
methaneand
and
other
greenhouse
other greenhouse
gas
gasemissions
emissions
2
4
Aspects of Evaluation
–
Adoption
–
Market development
–
Technology performance
–
Pollution and exposure
–
Health and safety
–
Time and socioeconomic impacts
–
Environmental impacts
Adoption
•
What does this aspect of evaluation tell us?
–
–
–
•
basic questions regarding implementation of project
or programme
number/ percentage of individuals or households
reached by a project or programme
results and their explanation will depend on priorities
of beneficiary communities
What are the challenges?
–
–
–
easy to conduct
standard practice in reporting success of a project or
programme
further information: WHO/PCIA Catalogue of Methods
Adoption: Key Questions
and Impact Measures
Key questions
Examples of impact measures
How many households have
adopted the intervention?
# households relying on improved versus
# households relying on traditional
cooking practices
To what extent have
households adopted the
intervention? Do they
continue to use the
intervention
% of time/occasions using improved
versus % of time/occasions using
traditional cooking practices
To what extent did the
intervention reach its target
audience?
% of poor households among total
households that adopted intervention in
village or district
Market Development
•
What does this aspect of evaluation tell us?
–
–
–
•
core questions for sustainability of intervention
creation/existence of a viable, self-sustaining market
with a balance between supply and demand
percentage of market penetration
What are the challenges?
–
–
–
easy to conduct
based on standard market survey tools
further information:
Shell Foundation's Breathe Easy Kit
Market Development: Key
Questions and Impact Measures
Key questions
Examples of impact measures
Which factors constrain
supply?
# manufacturers
Location/delivery mechanisms of manufacturers
Production capacity
Which factors determine Source of information on intervention
demand?
Suitability of intervention
Size of subsidy versus user contribution
Ability and willingness to pay
What is the growth rate
of adoption?
# new purchases per month
Change in rate over time
How sustainable is the
intervention?
Consistency of user maintenance
Availability/affordability of spare parts
Technology Performance
4
•
What does this aspect of evaluation tell us?
–
–
–
•
performance of device in laboratory informs
intervention choice
performance of device in the field is related to human
behaviour
performance testing informs necessary design changes
What are the challenges?
–
–
requires specific equipment
requires training
2
Pollution and Exposure
•
What does this aspect of evaluation tell us?
– measure of performance of intervention in a
real-life situation
– reduced indoor air pollution levels are primary
outcome to inform health improvements
•
What are the challenges?
– requires equipment
– requires in-depth training
– very labour- and resource-intensive
3
•
What does this aspect of evaluation tell us?
–
–
•
Health and Safety
documentation of change in reported health symptoms
associated with traditional cooking practices, e.g.
respiratory disease, burns, backache etc.
health is often the main objective of an intervention
What are the challenges?
–
–
–
–
outcomes assessed through questionnaire are not seen
as "gold standard" evidence among medical community
relative easy to conduct
questionnaires need to be adapted to specific setting
requires some training
Time and Socioeconomic Impacts
5
•
What does this aspect of evaluation tell us?
–
–
•
time and financial savings are often a key motivator for
adoption
other broader impacts considered important by the user
should be documented, e.g. cleanliness, status
What are the challenges?
–
–
–
some time and socioeconomic impacts can be assessed
quantitatively through questionnaires
some broader impacts require qualitative methods, e.g. focus
group discussions and open-ended questions
type of qualitative information obtained critically depends on
the choice and skills of the interviewer/facilitator
Environmental Impacts
•
What does this aspect of evaluation tell us?
–
–
–
•
changes in pressures on the local environment, e.g.
deforestation, charcoal production processes
changes in pressures on the global environment, i.e.
reduction in emissions of carbon dioxide, methane
demonstrating positive impacts on the environment is
important for cost-benefit analysis and alternative sources
of funding
What are the challenges?
–
–
–
–
not applicable to all interventions and settings
methods mostly based on technology performance and time
and socioeconomic impacts
may require sophisticated calculations
few specific tools currently available
Environmental Impacts: Key
Questions and Impact Measures
Key questions
Impact measures
To what extent has the
intervention led to reductions
in deforestation, desertification
or erosion pressures?
Changes in fuel source
Reduction in fuel consumption
Changes in watershed
management
To what extent has the
intervention led to reductions
in carbon dioxide, methane
and other greenhouse gas
reductions?
Changes in fuel source
Changes in combustion efficiency
Changes in technology emissions
Focus on Indoor Air Pollution
and Health
PARACELSUS
1493-1541
“Father of Toxicology
and Environmental
Health”
“The Dose Makes
the Poison”
The Environmental Health Pathway
= applies the concept of
"dose makes the
poison" to
environmental health.
Sources
Emissions
Concentrations
Exposure
Dose
Health effects
Environmental Health Pathway
for Indoor Air Pollution (1)
Sources
Emissions Concentrations Exposure
Emissions
into house
Fuel/stove
type
Indoor
concentrations
Dose
Personal
exposure
Health Effects
Pneumonia
Breath CO
Source:
Kirk Smith
Environmental Health Pathway
for Indoor Air Pollution (2)
Sources
Emissions Concentrations Exposure
Dose
Health Effects
Cost, Difficulty, Intrusiveness, Delay
Accuracy
Source:
Kirk Smith
How to Decide on What to Monitor?
How might you decide what
to evaluate?
Evaluation Decision-Making
• Target audience
– Beneficiaries, implementers, donors,
policymakers?
• Objectives
– Community development, project
development, funding, research?
• Priority areas
– Children’s health, market development,
environmental impact etc?
Intervention Characteristics
• Type:
Improved stove, behaviour
change, housing design, other?
• Scale:
How many houses?
• Stage:
Planned/underway/completed?
Feasibility
• Organizational capacity
–
–
–
–
Infrastructure
Planning, evaluation, data analysis and reporting
Staff experience, knowledge and expertise
Access to training
–
–
Access to funding
Partnerships
• Financial and human resources
• Time
– Implications for evaluation design
• Community
– Accessibility and willingness to participate
WHO/PCIA Catalogue of Methods
• A resource to help organizations develop evaluation
strategies for their intervention projects
• Uses a step-by-step process to ensure:
– Appropriate strategies
– Meaningful strategies
• Guide to developing an evaluation strategy
– EITHER: key questions and issues to guide choice of
methods and develop tailor-made evaluation strategy
– OR: ‘set menus’, e.g. basic evaluation using * methods
or advanced evaluation including *** methods
Conclusions
Working Towards an Evaluation Strategy
Many issues …
… many purposes …
… many target audiences for evaluation!
Your evaluation strategy will be driven by:
• Your priority issue of concern
• Your evaluation objectives and target audience
• Feasibility - financial and institutional