Program Evaluation
Regional Workshop on the
Monitoring and Evaluation of HIV/AIDS
Programs
February 14 – 24, 2011
New Delhi, India
Objectives of the Session
By the end of this session, participants will be able to:
 Understand the purpose and role of program evaluation
 Distinguish between different evaluation types and
approaches
 Link evaluation designs to the types of decisions that
need to be made
Why Evaluate HIV/AIDS Programs?
 To improve the design an implementation of a
program
 To reach informed decisions on the allocation of
existing limited resources, thereby increasing
program performance and effectiveness
 To identify factors that influence health and social
outcomes
 To generate knowledge, to know what works and
what does not.
 Good evaluations are public goods
Current Challenges in Evaluating
HIV Preventon Programmes
 HIV prevention programmes are increasingly
complex, multi-component and context-specific
 The underlying behavioural theories leading to
multiple behaviour changes and ultimately impact
are difficult to assess;
 Many projects/interventions/services aim to affect
HIV risk factors and/or vulnerabilities rather than
averting HIV infections directly.
Source: Strategic Guidance for Evaluating HIV
Prevention Programmes. UNAIDS 2010
All Programs/Projects have (implicitly
or explicitly):
 Objectives
 Expected outcomes
 Target population
 Mechanism(s) to deliver services (the intervention)
 Criteria for participating in the program
 A conceptual framework that provides rationale for
program existence (sometimes called the Development
Hypothesis)
Monitoring vs. Evaluation
Objectives of Monitoring:

To provide information on the functioning of the program:
a) Is it progressing according to plan?
b) Identify problems for correction

To track key program elements over time (to assess changes)
Characteristics of Monitoring:
•
•
Mostly tracks key quantifiable indicators of key program elements:
• inputs, processes, outputs, and outcomes
Often done on a routine basis
•
Key issue: good measurement using relevant indicators
•
No assessment of what is the cause of the change in the indicators
Monitoring vs. Evaluation

Objectives of Evaluation:
- To determine whether a program achieved its objectives
- To determine the impact of the program on the outcome
intended by the program
- How much of the observed change in the outcome can be
attributed to the program and not to other factors?

Characteristics of Evaluation:
- Key issues: causality, quantification of program effect
- Use of evaluation designs to examine whether an observed change in
outcome can be attributed to the program
Note: Monitoring tells you that a change occurred; Impact Evaluation will tell
you whether it was due to the program
Source: Strategic Guidance for Evaluating HIV
Prevention Programmes. UNAIDS 2010
Deciding Upon An Appropriate
Evaluation Design

Indicators: What do you want to measure?
 Provision
 Utilization
 Coverage
 Impact

Type of inference: How sure to you want to be?
 Adequacy
 Plausibility
 Probability

Other factors
Source: Habicht, Victora and
Vaughan (1999)
Clarification of Terms
Provision
Are the services available?
Are they accessible?
Is their quality adequate?
Utilization
Are the services being used?
Coverage
Is the target population being reached?
Impact
Were there improvements in disease
patterns or health-related behaviors?
Clarification of Terms
Adequacy
assessment
Did the expected changes occur?
Are objectives being met?
Plausibility
assessment
Did the program seem to have an
effect above and beyond other
external influences?
Probability
assessment
Did the program have an effect (P <
x%)
Source: Habicht, Victoria and
Vaughan (1999)
Adequacy Assessment Inferences
 Are objectives being met?
 Compares program performance with previously established
adequacy criteria, e.g. 80% ORT use rate
 No control group
 2+ measurements to assess adequacy of change over time
 Provision, utilization, coverage
 Are activities being performed as planned?
 Impact
 Are observed changes in health or behavior of expected direction and
magnitude?
 Cross-sectional or longitudinal
Source: Habicht, Victora and
Vaughan (1999)
Plausibility Assessment Inferences (1)
 Program appears to have effect above & beyond impact of
non-program influences
 Includes control group
 Historical control group
 Compares changes in community before & after program and attempts to rule out
external factors
 Internal control group
 Compares 3+ groups/individuals with different intensities of exposure to program
(dose-response)
 Compares previous exposure to program between individuals with and without the
disease (case-control)
 External control group
 Compares communities/geographic areas with and without the program
Source: Habicht, Victora and
Vaughan (1999)
Plausibility Assessment Inferences (II)
 Provision, utilization, coverage
 Intervention group appears to have better performance than
control
 Cross-sectional, longitudinal, longitudinal-control
 Impact
 Changes in health/behavior appear to be more beneficial in
intervention than control group
 Cross-sectional, longitudinal, longitudinal-control, casecontrol
Source: Habicht, Victora and
Vaughan (1999)
Probability Assessment Inferences
 There is only a small probability that the differences between
program and control areas were due to chance (P < .05)
 Requires control group
 Requires randomization
 Often not feasible for assessing program effectiveness
 Randomization needed before program starts
 Political factors
 Scale-up
 Inability to generalize results
 Known efficacy of intervention
Source: Habicht, Victoria and
VaughanSource:
(1999) Habicht, Victora and Vaughan (1999)
Evaluation Flow from Simpler to
More Complex Designs
Type of
evaluation Provision
Adequacy
1st
Plausibility
Probability
Source: Habicht, Victoria and
Vaughan (1999)
Utilization
Coverage
Impact
2nd
3rd
4th (b)
4th (a)
5th
Possible Areas of Concern to
Different Decision Makers
Type of
evaluation Provision
Adequacy
Plausibility
Utilization
Health center manager
International Agencies
Coverage
District health managers
International Agencies
International Agencies
Probability
Source: Habicht, Victora and
Vaughan (1999)
Impact
Donor Agencies & Scientists
Donor
agencies
Scientists
Process Evaluations
 Assess whether the program was implemented as
intended
 May look at
 Access to services
 Reach and coverage of services
 Quality of services
 Client satisfaction
 May also provide an understanding of cultural, sociopolitical, legal and economic contexts that affect
implementation of the programme.
Outcome/Impact Evaluations
 Assess whether changes in outcome/impacts are
due to the program.
 May look at
 Outcomes such as HIV-related behaviors,
 Health impacts such as HIV status, life expectancy
Evaluating Program Impact
Outcome
The Evaluation
Question:
How much of
this change
is due to the
program?
Program
midpoint or end
Program
start
Time
Evaluating Program Impact
Outcome
With
program
Net Program
Impact
Without
program
Program
midpoint or end
Program
start
Time
Features of Different Study Designs
True experiment Quasi-experiment
Non-experimental
Partial coverage/
new programs
Partial coverage/ new
programs
Full coverage programs
Control group
Comparison group
--
Strongest design
Weaker than
experimental design
Weakest design
Most expensive
Less expensive
Least expensive
Readiness criteria for Outcome &
Impact Evaluation
 The program
 is implemented with sufficient quality
 has achieved adequate coverage
 is of long enough duration that expected change in
the specified outcomes for the evaluation has had
time to occur
When to use an experimental or
quasi-experimental design
 The program has unknown
effectiveness
 There is the potential for negative
effects
 The program is politically or otherwise
risky
Source: Strategic Guidance for Evaluating HIV
Prevention Programmes. UNAIDS 2010
Who should plan for Evaluation?
 All programs should conduct basic monitoring
 Most programs should conduct process evaluations
 Implementation assessments
 Quality assessments
 Coverage assessments
 Some programs should conduct outcome evaluation
when evidence is needed as to whether the program
is effective
References
 Adamchak S et al. (2000). A Guide to Monitoring and Evaluating
Adolescent Reproductive Health Programs. Focus on Young Adults, Tool
Series 5. Washington D.C.: Focus on Young Adults.
 Fisher A et al. (2002). Designing HIV/AIDS Intervention Studies. An
Operations Research Handbook. New York: The Population Council.
 Habicht JP et al. (1999) Evaluation designs for adequacy, plausibility and
probability of public health programme performance and impact.
International Journal of Epidemiology, 28: 10-18.
 Rossi P et al. (1999). Evaluation. A systematic Approach. Thousand
Oaks: Sage Publications.
 UNAIDS (2010). Strategic Guidance for Evaluating HIV Prevention
Programmes.
MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.