Selecting the Right
Evaluation Method
Objectives
•
Why should we evaluate?
•
Which activities should we evaluate?
•
When should we evaluate?
•
How should we evaluate (methods)?
Why Do We Evaluate?
•
Design a program. (or item; e.g., curriculum, poster)
•
Determine if objectives were met. (best practices)
•
Re-design or refine a program. (lessons learned)
 Baseline, formative
 Outcome, process, impact
 Build an evidence base for ACSM
 Process, outcome
Which Activities Should We Evaluate?

Is this a new activity?

Is this an activity that may be expanded?

Have we invested a lot in this activity?

Is something unexpected happening?

Does the funder want an evaluation?
IF YES ….. EVALUATE!
Case Example
Is this a priority program for evaluation?
Why or why not?
Which objectives or activities are most
important to evaluate?
IMPACT
OUTCOME
PROCESS
FORMATIVE
BASELINE
Design
Implementation
End
When do we evaluate?
Key Informant
FORMATIVE
Interviews
EVALUATION
Client Exit
Interviews
OUTCOME EVALUATION
KAP Surveys
Media Scans
PROCESS
EVALUATION
Analysis of
Program Data
IMPACT
EVALUATION
Focus Groups
Secondary
Analysis
ACSM
ASSESSMENT
Simple Surveys
When Can We Use Quantitative Methods?
BASELINE
FORMATIVE
PROCESS
OUTCOME
IMPACT
Routine Surveillance
Analysis of NTP Data
Analysis of Project Data
Populationbased
Surveys
Population-Based Surveys
Simple Surveys
Use a QUANTITATIVE Method to:
•
Count things or people.
•
Measure your program against a benchmark.
•
Monitor trends over time.
•
Statistically correlate factors: Is there a connection
between variables?
When Can We Use Qualitative Methods?
BASELINE
FORMATIVE
PROCESS
OUTCOME
Focus Groups
Informant Interviews
Exit
Interviews
Media Scans
Exit Interviews
Media Scans
IMPACT
Use a QUALITATIVE Method to:
•
Find out overall concerns, opinions.
•
Gather initial information or sense of direction.
•
Get details of/reasons for any problems.
•
Test communication materials.
Combine BOTH for Best Results:
•
Identify key issues with a focus group or interviews.
•
Analyze NTP data to look for correlates or to obtain a
baseline measurement.
•
Use surveys to measure before and after results.
•
Conduct interviews to assess how your process is going.
•
Link your final outcome data to surveillance or NTP data.
Which Method and When?
BASELINE
FORMATIVE
PROCESS
OUTCOME
Analysis of Routine Surveillance
Focus Groups and In-Depth Interviews
KAP
Exit
Interview
KAP
Exit Interview
Simple Survey
Media
Scan
Media
Scan
IMPACT
Factors to
Consider
MY ACTIVITY OR PROJECT
Scale
Small,
local
Large,
multi-region
National
Cost,
Generalizability,
Rigor
Low
Medium
High
Key
informant
interviews
KAP surveys
Experimental
design
(research)
Methods
Focus
groups
Simple
surveys
Exit
interviews
NTP/Program
data analysis
KAP
surveys
What is the purpose of the evaluation?
I am at the end of a three-year, district-level ACSM project.
Did we produce the expected outputs and short-term
outcomes?
• Analysis of project data, focus group, or KAP survey?
I am designing a media campaign to inform my workers
that TB treatment is free. What is the best way to get this
message to them? What do they already know?
• Routine surveillance, KAP survey, or focus group?
What is the most rigorous method to answer
the evaluation question?
The NTP wants an impact evaluation of ACSM because
these new strategies have never been used at the national
level.
•
Experimental design, secondary data analysis, exit interviews?
I am looking for a successful ACSM model that I can
replicate in my district.
•
Case study, KAP survey, experimental design?
We trained 50 DOTS nurses on interpersonal
communication and counseling skills and we want to know
if they are using them correctly.
•
Time location survey, exit interviews, secondary data analysis?
How quickly do you need the results?
The NTP just announced funding for ACSM to reduce default. I
have to design an activity in six weeks! I need to know why
defaults are high in my district.
• KAP survey, simple survey, analyze surveillance data?
We have Global Fund money to implement a whole
“package” of ACSM interventions to improve case detection
over five years. We really need to know if the activities are
effective!
• Research study, informant interviews, KAP survey?
What financial and human resources
do I have?
I am SO lucky! My donor wants a formal evaluation of
ACSM and gave us a lot of money for it. AND we have
partners at the School of Public Health to help us!
• KAP survey, routine surveillance, research study?
My donor thinks that radio/TV/newspapers are the best way
to reach our population and is giving me a lot of money to
improve coverage of TB issues. But they want all of the
money to go toward implementation. How can I know if it is
effective?
• KAP survey, media scan, focus group?
Evaluation Template
ACSM Objective:
ACSM Activity:
Evaluation
Question
Evaluation
Category/Method
When or
How Often
Sampling
Resources
Needed
Plan for the Results