presented at
Evaluation Considerations:
Measures & Methods
Shrikant I. Bangdiwala, PhD
Professor of Research in Biostatistics
Injury Prevention Research Center
University of North Carolina at Chapel Hill, USA
1
WHO VIP Webinar 2011
Content
Purpose of evaluation
 Cycle of program planning & evaluation
 Indicators
 Study designs
 Statistical modeling
 Challenges

2
WHO VIP Webinar 2011
What are we ‘evaluating’?
Actions, programs, activities
 Conducted in a community setting, over a
period of time
 Aimed at reducing deaths, injuries, and/or
events and behaviors that cause injuries

3
WHO
Safety
VIP2010
Webinar
London
2011
Example:
Suwon,
South
Korea
 area of
‘safety
promotion’
http://www.phs.ki.se/csp/safecom/suwon2.htm
4
WHO VIP Webinar 2011
Why do we ‘evaluate’?

To know ourselves what works and if we are doing
some good
In performing some activity
 In the community
 In the country

To convince funders and supporters that their
investment is worthwhile
 To convince the community about the benefits of the
multiple activities and actions carried out

5
WHO VIP Webinar 2011
Main purposes of evaluation

Evaluation helps determine:
How well a program/policy works relative to its
goals & objectives
 Why a program/policy did or didn’t work, relative
to planned process
 How to restructure a program/policy to make it
work, or work better
 Whether to change funding for a program

6
WHOChicago
VIP Webinar
NSC
20102011
6
Methodological complications

Multiplicities
Multiple components of a program
 Multiple populations at risk
 Multiple study designs
 Multiple types of effects/impacts/outcomes &
severities
 Multiple audiences/objectives of ‘evaluation’
 Multiple methods for conducting evaluation

7
WHO VIP Webinar 2011
When should evaluation be
considered?
Evaluation needs to begin in, and be part of,
the planning process…
 Otherwise, “if you do not know where you are
going, it does not matter which way you go,
and you will never know if you got there or
not!”

Lewis Carroll (1872)
Alice in Wonderland
8
Adapted from M. Garrettson
WHO VIP Webinar 2011
8
Types of evaluation depending
on program phase
Program
Planning
Phase
Program
Implementation
Phase
Post
Program
Phase
9
Formative Evaluation
How can the program activities be improved before
implementation?
Process Evaluation
How is/was the program (being) implemented?
Impact / Outcome
Did the program succeed in achieving the intended
impact or outcome?
WHOChicago
VIP Webinar
NSC
20102011
9
Cycle of program planning and
evaluation
10
Adapted from C Runyan
WHO VIP Webinar 2011
Identify population & problem
Surveillance data
 Other needs assessment strategies

key informant interviews
 focus groups
 surveys
 evaluations of past programs
 literature
 consultation with peers
 other info…

11
WHO VIP Webinar 2011
11
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
12
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
12
Define target audience

To whom is the program directed?
Whose injuries need to be reduced?
 Who is the target of the program?

•
•
•
•
•
13
at risk persons
care givers (e.g. parents)
general public
media
decision makers
WHO VIP Webinar 2011
13
Understand target audience

What are their characteristics?
Special needs (e.g. literacy)
 Interests, concerns, priorities
 Attitudes & beliefs re: problem & solutions to
problem
 Cultural issues

14
WHO VIP Webinar 2011
14
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
15
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHO VIP Webinar 2011
Set goals/
objectives
15
Identify resources

Community partners
interest in topic
 working on similar projects

On-going activities
 Sources of financial support
 Interests in community

16
WHO VIP Webinar 2011
16
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
17
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHO VIP Webinar 2011
Set goals/
objectives
17
Set goals & objectives

Goal


broad statement of what program is trying to
accomplish
Objectives
Specific
 Measurable
 Time-framed

18
WHO VIP Webinar 2011
18
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
19
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHO VIP Webinar 2011
Set goals/
objectives
19
Choose Strategies

Identify existing strategies/programs
Literature: evidence based? promising practice?
 WHO manuals
 Successes from other communities-regionscountries


Develop new strategies:
Logic model (how would it work)
 Haddon matrix

20
WHO VIP Webinar 2011
20
Haddon Matrix
Person
Vehicle/
vector
Physical
Environ.
Social
Environ.
Preevent
Event
Postevent
21
Haddon 1970 Am J Public Health
WHO VIP Webinar 2011
21
3-dimensional Haddon Matrix
Other??
Feasibility
Preferences
Stigmatization
Equity
Freedom
Cost
Event
Post-event
Effectiveness
22
Runyan 1998 Injury Prevention
WHO VIP Webinar 2011
22
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
23
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
23
Formative Evaluation
Why it’s useful
Questions it answers



What is the best way to
influence the target
population?
Will the activities reach the
people intended, be
understood and accepted
by target population?
How can activities be
improved?
24



Improves (pilot-tests)
program activities before
full-scale implementation
May increase likelihood
program or policy will
succeed
May help stretch resources
WHO VIP Webinar 2011
24
* Modified from Thompson & McClintock, 2000
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
25
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
25
Implementation

As planned, with attention to detail

Documented clearly so others can replicate if
appropriate
26
WHO VIP Webinar 2011
26
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
27
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
27
Process evaluation

Purpose is to address:
What was done?
 How was it implemented?
 How well was it implemented?
 Was it implemented as planned?

28
WHO VIP Webinar 2011
28
Process evaluation –
examples of questions
•
•
•
•
Who carried out intervention?
Was this the appropriate person/group?
Who supported and opposed intervention?
What methods/activities were used?
29
WHO VIP Webinar 2011
29
Process evaluation
- why is it useful?
•
•
30
Allows replication of programs that work.
Helps understand why programs fail.
WHO VIP Webinar 2011
30
* Modified from Thompson & McClintock, 2000
The intervention cannot be a
black box…
It must be clearly understood
Idea
31
?
WHO VIP Webinar 2011
Outcome
31
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
32
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
32
Impact evaluation

Purpose is to address changes in:





33
knowledge
attitudes
beliefs/ values
skills
behaviors / practices
WHO VIP Webinar 2011
33
Using impact measures for

Establishing effectiveness
Suppose we have a public safety campaign as our
strategy
 Need to show

Campaign

 Outcome
If we already have demonstrated that
 Behavior

 Behavior
 Outcome
We simply need to show
Campaign
34
 Behavior
WHO VIP Webinar 2011
34
Outcome evaluation

Purpose is to address changes in:
injury events (e.g. frequency, type, pattern)
 morbidity (e.g. frequency, severity, type)
 mortality (e.g. frequency, time to death)
 cost (e.g. direct and indirect)

35
WHO VIP Webinar 2011
35
Example: Bike helmets
Intervention
Physician
counseling
parents
Enforcement of
helmet law
Impacts
Outcomes
Parental
attitudes
toward child
helmet use
Head injury in
bike crashes
Purchase of
helmets
Deaths from head
injury in crashes
Media campaign
Use of helmets
by children
36
WHOChicago
VIP Webinar
NSC
20102011
36
Evaluation examples of questions
for local policy of smoke alarms
Did the local policy of smoke alarms in apartments…






Get passed
Where people aware of it?
Did people have access to smoke alarms?
Did people get them installed properly?
Do people keep them maintained?
Lead to a reduction in the number or rates of:
events (e.g. apartment fires)
 injuries
 deaths
 37
costs (e.g. burn centerWHOcosts,
family
burden, property loss)
VIP Webinar
2011

37
Evaluation – selection of
measures
‘Quantitative Indicators’
Process
 Impact
 Outcome

Health related
 Financial

38
WHO VIP Webinar 2011
38
Choice of measure or indicator
We need to choose appropriate impact and outcome measures



‘Soft’ (more difficult to measure) outcomes –

Perceptions constructs: fear, insecurity, wellbeing, quality of life

Knowledge, attitudes and behaviors constructs
Hard outcomes –

Deaths, hospitalizations, disabilities due to injuries and violence

Societal impacts – local development indicators
Economics outcomes –

Direct $/€/£/¥, indirect DALYs, QALYs, opportunities lost, burdens
39
WHO VIP Webinar 2011
Evidence of effectiveness

Obtain qualitative ‘evidence’ to complement
the quantitative ‘evidence’

Ex. Are “multisectorial collaborations and
partnerships” friendly and functioning well?

Ex. Is “community participation” optimal?
Incorporate process indicators
 Incorporate narratives & testimonials

40
WHO VIP Webinar 2011
Disseminate
Identify
problem
& population
Evaluation:
•Process
•Impact
•Outcome
Identify
resources
Implement
Test, Refine,
Implement
Test & refine
implementation
41
Define target
audience
Evaluation:
•Formative
Choose
strategies
WHOChicago
VIP Webinar
NSC
20102011
Set goals/
objectives
41
Dissemination

Dissemination not done well
Not attempted
 Not based on research about how to disseminate
information to intended audience


Dissemination done well
Defining audience
 How to access audience
 How best to communicate change message to them
 Presentation of clear, straightforward messages

42
WHO VIP Webinar 2011
42
Evaluation measures
Lead to evidence of effectiveness
 But only if the research and study
methodologies, and the statistical analyses
methodologies, are appropriate to convince
the funders and supporters, the skeptics,
the stakeholders, the community


43
and understandable
WHO VIP Webinar 2011
43
Research methodology approach:
Evidence of effectiveness


Obtain quantitative ‘evidence’ that favors the hypothesis that
the intervention is effective as opposed to the (null) hypothesis
that the intervention is not effective.
How?
 Experimental study designs - randomized clinical trials,
grouped randomized experiments, community-randomized
studies
 Quasi-experimental study designs - non-randomized
comparative studies, before-after studies
 Observational studies - cohort studies, case-control studies
and comparative cross-sectional studies
44
WHO VIP Webinar 2011
Randomized controlled trial
(RCT) / Experiment
‘strongest’ evidence
Intervention
Group
O
X
O
O
X’
O
Randomize
Control
Group
45
WHO VIP Webinar 2011
45
Quasi-experimental designs
‘qualified’ evidence
Intervention Group
Comparison Group
46
O
O
X
O
O
WHO VIP Webinar 2011
46
One group pre/post
‘weak’ evidence
Intervention
Group
47
O
X
O
WHO VIP Webinar 2011
47
One group –
multiple pre / multiple post
better ‘weak’ evidence
Intervention
Group
48
O
O
O
X
O
O
O
O
O
WHO VIP Webinar 2011
48
One group, post only
‘basically ignorable’ evidence
Intervention
Group
49
X
O
WHO VIP Webinar 2011
49
Observational designs
- cohort study
evidence?
Self-chosen Intervention Group
Self-chosen Non-intervention Group
50
X
X
X
O
O
O
WHO
Safety
VIP2010
Webinar
London
2011
50
Observational designs
- case-control study
evidence?
X
Cases
O
X
Controls
O
51
WHO
Safety
VIP2010
Webinar
London
2011
51
Observational designs
- cross-sectional study
evidence?
X
X
O Injured X
O
O
X
Non-injured O
O
O
52
WHO
Safety
VIP2010
Webinar
London
2011
52
Statistical analysis
methodologies

Choice - often guided by what has been done
previously, or what is feasible to do, or easy to
explain

Choice should be tailored to the audience &
their ability to understand results; but also on
the ability of the presenter to explain the
methodologies
53
WHO VIP Webinar 2011
Statistical analysis


Determined by research question(s)
Guided by study design – experimental or observational





Guided by whether outcome is studied at a single time point or
multiple time points


Group randomized controlled experiment
Non-randomized comparison study
Single site pre/post; surveillance study
Retrospective or cross-sectional
Time series analyses
Guided by audience

VIP Webinar 2011
Visual
and descriptiveWHO
appreciation
54
Visual and descriptive analysis
– longitudinal time series
Example:
Espitia et al (2008)
Salud Pública Mexico
55
WHO VIP Webinar 2011
Visual and descriptive analysis
– comparisons over time
Example:
www.gapminder.org
56
WHO
Safety
VIP2010
Webinar
London
2011
Statistical analysis - challenge
But what we as a field have not done as well as other
fields, is to draw strength from numbers  develop
collective evidence
 Combine results from multiple studies

Systematic reviews (of observational studies)
 Meta analysis (of experimental & observational studies)
 Meta regression (of heterogeneous studies)
 Mixed treatment meta regression (for indirect
comparisons)

57
WHO VIP Webinar 2011
Systematic reviews

A protocol driven comprehensive review and
synthesis of data focusing on a topic or on related key
questions







58
formulate specific key questions
developing a protocol
refining the questions of interest
conducting a literature search for evidence
selecting studies that meet the inclusion criteria
appraising the studies critically
synthesizing and interpreting the results
WHO VIP Webinar 2011
Example –
Systematic review
Shults et al (2001) Amer J Prev Med
59
WHO VIP Webinar 2011
Systematic reviews

Of particular value in bringing together a number
of separately conducted studies, sometimes with
conflicting findings, and synthesizing their results.
Zaza et al (2001) Amer. J Preventive Medicine – motor vehicle

To this end, systematic reviews may or may not
include a statistical synthesis called meta-analysis,
depending on whether the studies are similar
enough so that combining their results is
Green (2005) Singapore Medical Journal
meaningful
60
WHO VIP Webinar 2011
Meta analysis

A method of combining the results of studies
quantitatively to obtain a summary estimate of the
effect of an intervention


61
Often restricted to randomized controlled trials
Recently, the Cochrane Collaboration is
‘branching out’ to include both experimental and
observational studies in meta analyses
WHO VIP Webinar 2011
Meta analysis

e.g. Liu et al
(2008)
Cochrane
Collaboration
62
WHO VIP Webinar 2011
Meta analysis

The combining of results should take into account:

the ‘quality’ of the studies
• Assessed by the reciprocal of the variance

the ‘heterogeneity’ among the studies
• Assessed by the variance between studies
63
WHO VIP Webinar 2011
Meta analysis
– estimation of effect


The estimate is a weighted average, where the weight of a study
is the reciprocal of its variance
In order to calculate the variance of a study, one can use either a
‘fixed’ effects model or a ‘mixed’/’random’ effects model


Fixed effects model:
utilizes no information from other studies  var( Yi )  var( ei )  VYi
Random effects model:
considers variance among and within studies  Yi     i  ei
var( )   2
var(Yi )   2  VY*i
WHO VIP Webinar 2011
64
Borenstein et al (2009) Introduction to Meta Analysis
Meta analysis & meta regression

Dealing with ‘heterogeneity’ among the studies - 2


Decompose the total variance into among and within
components  using mixed effects models for
getting a more precise estimate of the intervention
effect
If there is still residual heterogeneity

65
Expand the mixed effects model to include study-level
covariates that may explain some of the residual
variability among studies  meta regression
WHO VIP Webinar 2011
Meta regression

e.g.
Yi    1 X 1i   2 X 2i   i  ei
random error
study random effect
Overall mean
X1 study variable – EU/USA
X2 study variable – population type
66
WHO VIP Webinar 2011
Meta analysis




Standard meta-analytical methods are typically restricted
to comparisons of 2 interventions using direct, head-tohead evidence alone.
So, for example, if we are interested in the Intervention A
vs Intervention B comparison, we would include only
studies that compare Intervention A versus Intervention B
directly.
Many times we have multiple types of interventions for
the same type of problem, and we hardly have head-tohead comparisons
We may also have multiple component interventions
67
WHO VIP Webinar 2011
Mixed treatment meta analysis

Let the outcome variable be a binary response




1 = positive response
0 = negative response
We can calculate the binomial counts r j:k out of a total
number at risk n j:k on the kth intervention in the jth study
rj:k
We can then calculate
p j:k 
n j:k
the estimated probability of the outcome (risk of response)
for the kth intervention in the jth study
68
Welton et al 2009 Amer J Epid
WHO VIP Webinar 2011
Mixed treatment meta analysis


Let each study have a reference ‘‘standard’’ intervention arm,
sj, with study-specific ‘‘standard’’ log odds of outcome, j .
The log odds ratio, j:k, of outcome for intervention k, relative
to standard sj, is assumed to come from a random effects
model with


mean log odds ratio (d k  d s j ) , and
between-study standard deviation 
where dk is the mean log odds ratio of outcome for
intervention k relative to control (so that d1 = 0).
69
Welton et al 2009 Amer J Epid
WHO VIP Webinar 2011
Mixed treatment meta analysis

This leads to the following logistic regression model:
ln
where
70
Welton et al 2009 Amer J Epid
p j:k
1  p j:k
 j

 j   j:k
 j:k ~ N[(dk  d s ), ]
j
WHO VIP Webinar 2011
int  s j
int  k
Mixed treatment meta analysis
- multiple-methods interventions

If we have multiple methods in the ith intervention
M1i , M 2i , M 3i ,...

Plus we have multiple times when the outcome is
assessed
Yit     i  1t   2 M1it  3 M 2it   4 X i  eit
Study effect
Components 1 & 2 effects
Time effect
71
WHO VIP Webinar 2011
Study covariable
Error term
Statistical analysis


Methodology does exist for developing stronger
collective evidence, evaluating the effectiveness of
community based interventions, using different types
of study designs and interventions
Developing “practice-based evidence”
72
WHO VIP Webinar 2011
Dissemination



We should not stop at developing the evidence
We must work alongside economists in developing
ways to effectively communicate ‘what works’ 
methodology and cost models do exist for estimating
the “return on investment”
Money talks !!
73
WHO
Safety
VIP2010
Webinar
London
2011
Challenges – Evaluation requires







Integration of evaluation from the beginning
Appropriate measures, possible to be collected objectively,
unbiasedly, easily and with completeness
Appropriate qualitative and process information, to complement
the quantitative information
Concrete and convincing evidence of what aspects work in
individual communities
Formal methodological statistical evaluation of specific elements
of programs
Collective evidence of what common elements of programs work
Effective dissemination strategies – “return on investment”
74
WHO VIP Webinar 2011
75
WHO VIP Webinar 2011
75