Health Impact Evaluation:
An Introduction
Temina Madon, PhD
Executive Director
Center of Evaluation for Global Action
University of California, Berkeley
What is Impact Evaluation?
Identify a health need or
target.
• Proportion of children under 5 sleeping under
ITN is < 6% and target is 60%.
• Health worker absenteeism is reducing
patients’ use of clinics (unpredictable access to
care is a deterrent)
What must be done to
meet needs or reach
targets?
What interventions or
services could be used?
• Increase access to and proper use of ITNs
• Decrease health worker absenteeism
Which is the most
robust program to use?
• Pilot a few programs and measure their
impacts.
• Is program A able to reliably produce desired
• Cost-free distribution of ITNs through schools,
or door-to-door distribution.
• Incentive-based payments to health workers
conditioned on health coverage or on
attendance.
Why Impact Evaluation?
• If you need evidence that a program works
Does the program actually improve health outcomes?
Does it reach the people in need?
• Accountability to civil society
• Accountability to funders
• Ability to target a limited budget
• If you want to improve the program over time
• Results-based management
• Demonstrate cost-effectiveness
• If you want to scale up but need proof of concept
Objectives of this Workshop
• Understand how IE is related to M&E
• Identify questions suitable for impact evaluation
• Determine which indicators and designs to use
• Understand impact evaluation methods
• Develop an impact evaluation concept note for an
intervention of your choice
M&E vs. IE
IMPACT
OUTCOMES
OUTPUTS
INPUTS
Effect on living standards
- infant and child mortality,
- prevalence of specific disease
Access, usage and satisfaction of users
- number of children vaccinated,
- percentage within 5 km of health center
Goods and services generated
- number of nurses
- availability of medicine
Financial and physical resources
- spending in primary health care
M&E vs. IE
• Are deworming treatments being delivered as planned?
M&E
• Does school-based delivery of deworming medication
increase school attendance?
IE
• What is the correlation between access to health clinics and
the proportion of children under 5 receiving routine
M&E
vaccinations?
• Do house-to-house immunizations lead to an increase in the
proportion of children under 5 who are immunized, relative IE
to the level in communities with annual vaccination
campaigns? Which costs less per child vaccinated?
M&E vs. IE
• M&E can often be started well after the
program has been designed and
implemented
• IE generally needs to be considered at the
outset of a program’s implementation or
scale-up, because you need to build the
evaluation into the design of the program
M&E vs. IE
• M&E allows you to guess how the program
may have hurt or helped people.
• IE allows you to measure the program’s
effects directly. You will rule out every
other explanation for the observed effects.
What to Evaluate?
•
•
•
A health service program
– ITN distribution at pre-natal
clinics
– Oral rehydration packets and
education for new mothers
– Deworming medicines in
schools
Health systems project
– Health insurance plans
– Clinic staff retention
Local or federal health policy
– Seatbelts law
– Tobacco tarriffs
•
•
•
Behavior Change
– Patient adherence to treatment
– Improved care provided by
health care provider
– Community-wide adoption of
prevention practices
Services & Systems
– Quality of Services
– Cost-Effectiveness
Health Outcomes
– Maternal mortality
– Child survival
When to use IE?
M&E can be used to fine tune a program’s
design, but IE will:
– Tell you whether or not the program is “worth
the money” (Does it work?)
– Guide results-based management
– Improve communication with the public & civil
society