Evaluation Methods
Objectives
•
Describe the most common quantitative and qualitative
methods used to evaluate ACSM.
•
Demonstrate the role of these methods in supporting
ACSM evaluation.
Evaluation of Treatment Support Program
Quantitative
Qualitative
Patient data from
10 TB clinics.
_____________________________
Interviews with 20 patients
who left treatment early.
_____________________________
•
How many patients left
treatment early.
•
What they experienced
during treatment.
•
How many weeks of
treatment were completed.
•
Why they left treatment.
•
Ideas for improving the
treatment support system.
•
% default by gender,
ethnicity, age.
What are Quantitative Methods?
•
Analyze numerical data.
•
Use standard tools (e.g., questionnaires).
•
Need precision and reliability!
•
Often use large sample sizes with sophisticated methods
to select participants.
•
Results are generalized to a population.
Example: The TB prevalence survey planned for Nigeria
will include 49,000 people selected from 700 clusters
representing six zones.
What are Qualitative Methods?
•
Provide in-depth, detailed information on behavior,
opinions, ideas, etc.
•
Analyze text, words, pictures, or interview transcripts
instead of numbers.
•
Use open-ended data collection tools.
•
Usually smaller sample sizes.
Quantitative Methods
•
Analysis of routine surveillance and NTP data.
•
Analysis of program or project data.
•
Simple surveys.
•
Population-based surveys:
−
KAP, prevalence.
Method: Analysis of Routine
Surveillance and NTP Data
•
Existing Ministry of Health reporting system data:
−
−
−
Vital registration data.
Routine reports from disease-specific programs.
Hospital data.
•
Data usually available; no special data collection needed.
•
Quality may be a concern: completeness, timeliness,
accuracy may not be good.
Examples: Routine Surveillance
•
Number of deaths due to TB in State B in 2010.
•
Prevalence of TB/HIV co-infection by age and gender.
•
Number of TB cases reported in 2012.
•
Percentage of MDR-TB cases among new reported
cases in 2012.
Using NTP Data to Prioritize ACSM
Quarterly review of SS+ case notification
•
•
Which districts report an increase in SS+ case notification?
•
Where is TB/HIV co-infection the highest?
Which district reported the highest number of retreatment
cases?
Method: Analysis of ACSM Program
Data
•
Collect and analyze key output and outcome indicators
over the life of the project.
•
Data come directly from your project.
•
Quantitative indicators and sources:
− Advocacy.
− Communication.
− Social mobilization.
Activity
Expected
Outcomes
Analysis
Community health workers identify and
refer persons with presumptive TB.
Increased
number of
people arriving
at DOTS center
for screening
Increased
number of TB
cases
diagnosed
Increased
number
starting
treatment
% of
referrals
received
at DOTS
center
% of
referred
persons
diagnosed
with TB
% of TB
cases
starting
treatment
TB Screening Referrals from Community
Health Workers, District X, 2011
140
120
100
80
55%
75%
57%
40%
60
40
8%
18%
20%
15%
20
0
Q1
Q2
Q3
Q4
Referrals made to
DOTS
Referrals received at
DOTS
TB cases diagnosed
TB cases starting
treatment
Contribution of Community Health Workers
to TB Case Notification in District X, 2011
Quarter
Number of TB
cases
diagnosed
after referral
Total number
of TB cases
notified in
district
Contribution
of referral
network to
case
notification
Jan – Mar
8
60
13%
Apr – Jun
9
73
12%
Jul – Sept
12
85
14%
Oct – Dec
10
93
11%
Give Your Data Meaning and Importance
•
Interpret results within a larger context.
•
Link project data with routine/NTP data to show
contribution.
•
Monitoring: How many did we refer and how many
were diagnosed with TB?
•
Evaluation: How much did our referrals ultimately
contribute to overall case notification in the district?
Method: Population-Based Surveys
•
Provide data on priority indicators that can be
generalized to a population.
−
•
% of adults 15 years and older who correctly
identify chronic cough as a TB symptom.
Usually requires a large research team, specialized
expertise, and a full-time survey manager.
−
Nigeria 2011/2012 prevalence survey includes
49,000!!!
What is a KAP Survey?
•
KAP = knowledge, attitudes, and practices
•
Population-based survey—very large sample size!
•
Face-to-face interviews with standard tool.
•
Provides summary data on KAP.
−
Averages, percentages, totals, ranges, etc.
What does a KAP survey tell us?
Knowledge, attitudes, and practices
•
What does the population know about TB?
•
What myths or misinformation are common?
•
What stigmas are related to TB?
•
Where do people seek care?
•
What media channels are used by this population?
Essential Components: KAP Survey
•
•
•
•
•
•
•
Appropriate sampling strategy
Standardized, pre-tested questionnaire
Field protocol and manual
Trained, UNBIASED data collection team
Data analysis plan
Data use strategy
BUDGET!
Method: Simple Surveys
•
Information from a smaller, targeted sub-group:
− Village surveys.
− Basic questionnaires.
− Polls.
•
Faster and easier, but less scientifically rigorous than
KAP surveys.
•
Cannot generalize the results to a larger population.
Qualitative Methods
•
•
•
•
Focus group discussions
Key informant interviews
Exit interviews
Media scans
Method: Focus Group Discussions
•
Small group (6–10 people).
•
Gather opinions, insights, thoughts, and feelings about
a topic.
•
Led by a moderator; assisted by a notetaker.
•
•
Semi-structured guide with key questions.
•
Useful for pre-testing communication materials.
May be recorded, transcribed, formally analyzed with
software.
Method: Key Informant Interviews
•
Face-to-face interview with critical individuals:
− Policymakers, opinion leaders.
− Those who represent key stakeholders (e.g., NGO
directors, association leaders).
− Marginalized, vulnerable populations who do not want
to participate in groups.
•
Good for collecting sensitive,
personal, or confidential
information.
•
Limited perspectives.
Interviews or Focus Groups?
•
What do patients think of a new brochure on treatment
adherence?
Focus Group
•
How can we get the directors of TB clinics and HIV
centers to collaborate more?
Interviews
•
What do HIV-positive injection drugs users know about
TB symptoms and diagnosis?
Focus Group or Interviews
Exit Interviews
•
Evaluate short-term outcomes of changes to service
delivery.
•
Standardized instrument used to gather data on patient
experiences with a clinic or specific provider.
•
Can measure patient satisfaction
and monitor provider behavior
after training.
Media Scans
•
Search media for how a specific topic is covered:
− What are the most common messages
about TB?
− How often is TB discussed in the
media? Is it a “hot” topic?
− Is the information about TB accurate?
− Are the messages stigmatizing?
•
Useful to evaluate interventions with journalists.
•
Formal or informal.
Q&A/Resources