Program Evaluation
• A systematic effort to describe the
status of a program
• Extent to which program objectives
achieved
Uses of Health Program Evaluation
Insight
- Needs
- Barriers
- Activities
Improvement
- Social mobilization
- Inter- sectoral Coordination
- Implementation
- Client Conveniences
Affordability
Accessibility
Availability
- Cost - Benefit
Outcomes / Impact
- Skills development
- Behavioral change
- Level of success in achieving objectives
- Accountability
Types of Evaluation
Needs Assessment
To identify
•
Goals
•
Products
•
Problems
•
Conditions
Types of Evaluation Contd…
Formative (Process) Evaluation
To improve developing or ongoing program
Role as helper/advisor/planner
Progress in achievements
Potential problems/needs for improvements in
•
Program Management
•
Inter-sectoral coordination
•
Social mobilization

Implementation

Outcomes
Types of Evaluation Contd…
Summative (Coverage) Evaluation
(To help decide ultimate fate)
Summary statement about
Program’s achievements
Unanticipated Out comes
Comparison with other programs
Sample Size
Factors
- Purpose of study
- Population size
- Level of precision (sampling error)
- The confidence / Risk level
- Degree of variability
- Appropriate for the analysis
- Appropriate for comparative analysis of sub groups
- Add for non respondents
Sample Size Contd…
Strategies
- Using a census - small population
- Using sample size of a similar study
- Using published tables / software
- Using formulas
Health Program Evaluation - Quantitative
Research Methods
Approach
-Measures the reaction of a great many
people to a limited set of questions
-Comparison and statistical aggregation of
the data
-Broad, generalizable set of findings
presented succinctly and parsimoniously.
Health Program Evaluation Qualitative Research
Methods
•
•
•
•
•
Participant Observation
Key Informant Interviews
Open ended Interview
Focus Group Discussions
Pile sort
Health Program Evaluation - Qualitative Research
Methods Contd…
Findings
- Presented alone / in combination with quantitative data
- Validity and reliability depends on methodological skills,
sensitivity, integrity of the researchers
- Skillful interviewing - more than just asking questions.
- Content analysis - more than just reading to see what’s
there.
- Generate useful and credible findings through
observation, interviewing and content analysis
How?
- Discipline, knowledge, training, practice, creativity,
hardwork
Data Processing
•
Raw field notes should be
corrected edited and typed
•
Tape recordings need to be
transcribed and corrected
•
Texts by field worker should not be
changed to make it ‘writable’ or
‘readable’
Data Reduction
•
Process of selecting, focussing,
simplifying, abstracting and
transforming data from field notes
and transcripts
•
Researcher retains some data
chunks, pulls out some and gets an
idea of story to tell
Analysis Steps
•
•
•
•
•
Free listing
Domain Evolution
Coding
Tabulation
Summarizing
Quotable Quotes
•
Give a vivid, meaningful flavor
which is far more convincing than
pages of summarized numbers
-
These should not be distracters
Should not take the reader away
from the real issues in hand
“This is an unprecedented event where all people
irrespective of caste, creed and religion take part in PPI
program on the same day (NID) throughout the country”
•Health worker (150): Burdwan
“He (my husband) told me that everybody is going for polio
drops. Then why should we be left out ? After all, everybody
is not a fool”
•Utilizer (1422): Delhi
“We have not at all immunized our son. My husband was very
stubborn. He said ‘those who are immunized are also getting
this disease (polio) and whatever happens let it happen’. He
has not allowed me to get the child immunized”
•Non utilizer (630): Hyderabad
Data Display
•
This is an organized, compressed
assembly of information that
permits conclusion drawing and
action
•
Matrices, graphs, charts and
networks are used
Qualitative Vs Quantitative Research
- the dichotomy
Qualitative
Quantitative
Social theory
Action
Structure
Methods
Observation, interview
Experiment, survey
Question
What is X?
(classification)
How many Xs?
(enumeration)
Reasoning
Inductive
Deductive
Sampling method
Theoretical
Statistical
Strength
Validity
Reliability
Qualitative Vs. Quantitative
• It is not qualitative Vs. quantitative but
qualitative and quantitative
• Qualitative methods are rapid, exploratory
and hypothesis generating
• Can be used as Impact evaluation
research
• Allow the researcher to palpate the unique
cross-cultural features
Multi-Centric Evaluation Studies
Issues
• Common understanding of the program
• Common understanding of aims and
objectives of evaluation exercise
• Standardization of research instruments
• Standardization of protocol implementation at
various sites
• Regional variation in program implementation
Multi-Centric Evaluation Studies Contd…
Steps
• Cohesive network of partners
• Multi-disciplinary team of investigators
• Piloting of instruments at different sites
• Development of common understanding
• Training of research teams
• Multiple layers of quality assurance measures
Network Dynamics
Sustaining the Network
During active period:
• Recognized individual efforts to excel
- Co-opted as extended CCT members
• Recognized PMC efforts to excel
- Made regional centers
• Communication Channels (phone, fax, e-mail)
• Pilot of instruments (ownership)
During interface:
• IndiaCLEN / INCLEN meetings, workshops
• Feedback on completed reports
Network Dynamics Contd…
Quality Assurance Measures
• National orientation workshop (PI s)
• Regional orientation workshops (PI s, RA s)
• PMCs, Regional Centers:
Quality checks of interviews, schedules, tapes
• Central Coordinating Office:
Random checks of recordings, interviewing procedure,
transcripts, translations
• Regional Coordinators, CCT members:
Site visits / Supervision of FGDs
• Method triangulation using
In-depth interviews, FGDs
Quality Assurance Measures
• Development of Interview Schedules
• Consistency Checks / VALIDITY
Method Triangulation
Data Triangulation
Data Interpretation
• Partner Medical Colleges
• Regional CEUs
• Central Coordinating Office
Capacity Building
National Level
•
Leadership transfer to coordinate MI project
- Thiruvananthapuram
State Level
•
PMCs upgraded to Regional Centers
•
Upgrading of physical facilities
•
Ten investigators INDEPENDENTLY took up
evaluation of national programs at state/district
level
Network Partners
•
Research - individual/collaborative
•
Resource persons - local/regional/national
•
Extended CCT members
Interaction with Program Managers /
Policy Makers
Program Evaluation: A partnership exercise
• Developing objectives, instruments
• Dissemination of Findings
• Support other program related activities
Dynamics of Establishing Partnerships with
Policy Makers
Evaluators express their opinions explicitly
- based on evidence gathered
- consistent quality assurance measures
- limitations of study accepted up front
- politics of evaluation
- remains a scientific endeavor
Dynamics of Establishing Partnerships
with Policy Makers
• Results to be presented in a manner
which are perceived as
VALID, RELIABLE & FEASIBLE TO IMPLEMENT
• Working in a strict TIME SCHEDULE for
timely fine-tuning of strategies
• Program Evaluators have to establish
CREDIBILITY with Program Managers
How can Evaluation Data be Used?
Program managers
–
–
–
–
–
–
Redefining aims objectives
Modifying or fine tuning strategies (process)
Sustainability (including fatigue factor)
Judge the worth (impact)
Expense / cost
Interaction with other activities
How can Evaluation Data be Used?
Education
– Generalisability
– Unique features (success/failure)
– Determinants of provider and client
behavior
IndiaCLEN
Program Evaluation Network
Activities
Studies Completed (1997-2001)
Title
Year
Funding
Source
Pulse Polio Immunization
Program (PPI-1)
1997- 98
USAID/INCLEN
Pulse Polio Immunization
Program (PPI-2)
1997- 98
USAID/INCLEN
Intensive Pulse Polio
Immunization Program (IPPI-3)
2000
USAID/INCLEN
Family Health Awareness
Campaign (FHAC-1)
1999
USAID/INCLEN
Barriers in Polio Eradication
(Moradabad, UP)
2000
WHO/SEARO
Family Health Awareness
Campaign (FHAC-2)
2000
USAID/INCLEN
(Coverage & Process)
Forthcoming Studies
Title
Year
Funding
Source
Integrated Disease Surveillance Program:
Barriers in Surveillance Activities in three
States (pilot-FGDs)
2001
World Bank
Vitamin-A and Iron Folate
Supplementation Program
2001- 02 MI/IDRC
Safe Injection Practices
(Coverage & Process)
2001- 02 World Bank
Evaluation of AFP Surveillance
2002
USAID/INCLEN
Family Health Awareness Campaign
(Coverage)
2002
USAID/INCLEN
IndiaCLEN Members -1996
Delhi
Lucknow


Nagpur
Chennai

Vellore

Thiruvananthapuram
•
Srinagar
Delhi

Ballabhgarh
• Agra
Jodhpur
•
•
Ahmedabad

•
Panaji
•

•
Patna
Burdwan
•
Bilaspur
Gulbarga
Tirupati
•
•
Chennai

•
Vellore
Calicut• • Madurai
Bangalore


Dibrugarh
Lucknow
Bhopal

Nagpur
Mumbai
IndiaCLEN Program Evaluation
Network - 1997
Thiruvananthapuram
•
Berhampur
•
Srinagar
•Kangra
••
IndiaCLEN Program Evaluation
Network - 2000
Delhi
Rohtak
Ballabhgarh
Agra
Jodhpur
Dibrugarh
Lucknow
Bikaner
Jaipur
Darbhanga
Kohima
Guwahati
Kota
Gwalior Kanpur
Agartala
Patna
Imphal
Jamnagar
Bhopal
Burdwan
Aizwal
Ahmedabad
Kolkata
Bilaspur
 •
• •
•
•
Mumbai
•
Panaji
• 
• •
••

••
• • Cuttack

Sambhalpur
Nagpur
•
Berhampur
•
• Vijayawada
•Tirupati
Chennai

•
Vellore
• Madurai
Hyderabad
Gulbarga
•

••
Kannur •
Calicut•
Bangalore

Thiruvananthapuram
• •
•
• •
Agenda Item No.14- Conduction of Family Health
Awareness Campaign
A brief (15 minutes) presentation on evaluation of
FHAC round 2000 was made by Dr. N.K. Arora,
IndiaCLEN, AIIMS.
“Addl. Secretary & Project Director (NACO) said
that the short comings observed in evaluation of
the campaign should be taken into consideration
while preparing action plans for the next round of
FHAC in the year 2001. After discussion (one hour
15 minutes) with the State Project Directors, it was
decided that…”
Letter No.T.11014/2/2001-NACO dated 05.07.2001
IndiaCLEN
Program Evaluation Network
VISION
Facilitate development
and implementation of
people friendly, effective
IndiaCLEN Program Evaluation Network
Investigators
CCT Members
N. K. Arora
M. Lakshman
Kiran Goswami
Sneh Rewal
R.M. Pandey
K. Anand
K.K. Ganguly
Naveet Wig
Leena Sinha
S. Vivek Adhish
N. Chaudhuri
H.K. Kumbnani
Thomas Mathew
Sandip K. Ray
S.L. Chadha
Rema Devi
K.C. Malhotra
R. Sankar
Sunita Shanbhag
Ballabhgarh
S.K. Kapoor
Praveen Kumar
M.K. Taneja
R.C. Agarwal
Rohtak
A.D. Tiwari
B.K. Nagla
Mohinder Singh
Kangra
T.D. Sharma
Swaran Lata
K.L. Ghai
G.L. Jaryal
Lucknow
R.C. Ahuja
Vikas Chandra
J.V. Singh
A.K. Srivastava
Kanpur
V.N. Tripathi
Joginder Singh
R.P. Singh
Agra
Deoki Nandan
S.K. Mishra
S.P. Agnihotri
Dibrugarh
Faruque Ahmed
Swapna D. Kakoty
Mayashree Konwar
Bijit Bhattacharya
Mir Shahadat Ali
Pranab Jyoti Bhuyan
Guwahati
Chiranjeeb Kakoty
Sajjad Ahmed
Alaka Bhattacharyya
Imphal
E. Yaima Singh
T.H. Achouba Singh
R.K. Narendra
Tiasunup Pongener
Umatula
Agartala
Partha Bhattacharjea
Aizwal
L. Lalhrekima
Jodhpur
S.L. Solanki
Suman Bhansali
Afzal Hakim
Y.R. Joshi
Kota
Raghuveer Singh
Gopal Bunkar
Hans Raj
Jaipur
Anurag Sarna
Rajesh Jain
Hemant Jain
Bhopal
Sheela S. Bhambal
A.K. Upadhyaya
R.K.S. Kushwaha
U.K. Dubey
Bilaspur
Vijay Tiwari
D.N. Mishra
Ajay Gurudiwan
Ashok Tiwari
Gwalior
A.G. Shingeweker
A. Shingeweker
Berhampur
B.C. Das
D.M. Satapathy
G.S. Patnaik
T. Sahu
Cuttack
S.C. Jena
S.K. Sahu
K. Misra
Sambhalpur
O.P. Panigarhi
H.P. Acharya
S.C. Panda
Nagpur
A.K. Niswade
Sanjay Zodpey
Sanjay Deshpande
Suresh Ughade
Prashant Langewar
Mumbai
Alka Jadhav
Nagaonkar
Nitin Deshpande
Shubhangi Upadhye
Chitra Nayak
Vijayawada
S. Narasimha Reddy
T.S.S. Manidhar
A. Rama Prasad
C. Usha Rani
Hyderabad
B.V.N. B. Rao
C. Bala Krishna
J. Ravi Kumar
Tirupati
K. Raghava Prasad
N.A. Chetty
G. Raviprabhu
Gulbarga
B. Mallikarjun
R.R. Rampure
B.N. Patil
Shreeshail Ghooli
Bangalore
Shivananda
Gopal
Premalata
Bangalore
A.S. Mohammad
Lalita Bhatti
R.M. Christopher
Chennai
R. Sathianathan
A. Vengatesan
R.K. Padmanaban
S. Karthikeyan
Vellore
Kurien Thomas
O.C. Abraham
Mary Kurien
Madurai
C. Kamaraj
M. Eswaran
T. Rajagopal
Thiruvananthapuram
M. Narendranathan
P.S. Indu
J. Padmamohan
S.M. Nair
Kannur
Jeesha C. Haran
T.P. Mubarack Sani
M. Jayakumary
Calicut
M. Ramla Beegum
C.R. Saju
N.M. Sebastien
Program Evaluations - Relevance to Policy
• ACADEMIA can play an important role in
influencing the National Policy
- multi disciplinary teams
• Evaluations are not done in VACUUM, should be
Policy Relevant
- central, state, district level
• RECOGNIZE Policy Makers & Other Stakeholders
as partners
Models of Program Evaluation
• Goal oriented evaluation
Aimed to assess the progress and the effectiveness of innovations/
interventions.
• Decision oriented evaluation
Aimed to facilitate intelligent judgements by decision makers.
• Responsive evaluation
Aimed to depict program process and the value perspectives of key people.
• Evaluation Research
Focused on explaining effects, identifying causes of effects, and
generating generalizations about program effectiveness.
• Goal free evaluation
To assess program effects based on criteria apart from the program’s own
conceptual framework, especially on the extent to which real client needs
are met.
• Advocacy - adversary evaluation
Evaluation should derive from the argumentation of contrasting points of
view.
• Utilization - oriented evaluation
Structured to maximize the utilization of its findings by specific
stakeholders and users.
Design Effect
Ratio of variance with cluster sampling to variance
with simple random sampling
Var simple random sampling
=
p(1 - p)
n
Var cluster sampling
=
(pi - p)2
K(k-1)
Design effect
=
 (pi - p)2 n
k(k-1) p(1-p)
Health Program Evaluation - Quantitative
Research Methods
Approach
- Measures the reaction of a great many
people to a limited set of questions
- Comparison and statistical aggregation of
the data
- Broad, generalizable set of findings
presented succinctly and
parsimoniously.
Summary
Qualitative methods aim to make sense of, or
interpret, phenomena in terms of the meanings
people bring to them
Qualitative research may define preliminary
questions which can then be addressed in
quantitative studies
A good qualitative study will address a clinical
problem through a clearly formulated question and
using
more
than
one
research
method
(triangulation)
Analysis of qualitative data can and should be done
using explicit, systematic, and reproducible
methods
Development of Program Objectives
(Program Evaluators)
• Lessons of success & failure
• Wider application of program strategies
• Determinants of client behavior
• Impact on other health systems
[national & international interest in later part]