Monitoring and evaluation for
social and behavioural change
communication
Dr Renay Weiner
Soul City: Institute for Health and
Development Communication
Outline
Soul City: background
What do we measure?
Selecting indicators
Approach to evaluation
Issues and challenges
Soul City: mission
Mission is to make an improvement in
people's health and quality of life by:
• Harnessing the power of the mass
media
• Developing high quality education
material through thorough research
and evaluation involving the
communities we serve
• While remaining an organisation not
for gain, ensure the sustainability of
our effort through sound business
principles and practices
• Being strong advocates for health and
development
• Measuring impact on health and
development
SC model of SBCC
• Effect change at individual, community
and societal levels
• Methods: MEDIA; SOCIAL
MOBILISATION and ADVOCACY
• Learning and Reflection; self-efficacy
• Draw on different theories of SBCC
5 Pillars of Health
Promotion(Ottawa Charter):
1.Create enabling environments
2.Develop personal skills
3.Advocate for healthy public policy
4.Re-orientate services
5.Stimulate community action
Social Cognitive Theory
Social
Learning
A person’s
behaviour isTheory
influenced by :
•Personal factors (cognitive, affective, biological events)
Modelling
•Environmental
factors
Self
efficacy
In many instances, without social change, individual behaviour change,
People
learnimpossible
by observing behaviour of others and the outcomes
is almost
thereof
Soul City aims to impact on individual, community and societal levels
People
adopt those
behaviours
if they
believe
theychange,
are capable.
to enhance
things that
help
behaviour
and decrease
those things that hinder.
Important theory specifically for edutainment because it includes
concept of traveling on a journey with characters and seeing the
results of risky / negative behaviour and can role model positive
behaviours and outcomes)
Changing behaviour is a complex endeavour
In many instances, without social change, individual
behaviour change, is almost impossible
Soul City aims to impact on individual, community
and societal levels to enhance those things that help
behaviour change, and decrease those things that
hinder.
Soul City Institute for Health and
Development Communication
2008-2012, Soul City is
tackling two important
social and public health
issues in South Africa:
HIV/AIDS and violence.
Campaign approach
• Onelove campaign: SA
and regional
• PhuzaWize: SA
Campaign model
Advocacy to create a
supportive environment
Mass media
Reflects
Social change
Stimulates
Outreach /social
mobilisation
example of regional level OL
Soul City: interventions and
activities
MEDIA
• Soul City TV drama: 13
episodes
• Soul Buddyz TV drama
• Soul City radio drama:
translated into all languages
• Other: Kwanda; Love Stories,
BOM
• Print booklets
SOCIAL MOBILISATION
• Soul Buddyz programme
• Adult training and community
dialogues
ADVOCACY
What do we monitor and
evaluate?
Need to consider objectives of programme/project
REACH OF Onelove
REACH
numbers and %
by age and sex
geographical location
province and geotype
Selecting indicators:
deciding what to measure
MEDIA
SOCIAL MOBILISATION
ADVOCACY
Social
Knowledge
Community---Intervention --- Attitudes ---Behaviours---- Prevalence
Individual
Self-efficacy
Incidence
INPUT
PROCESS
OUTPUT
OUTCOME
OUTCOME
IMPACT
IMPACT
Selecting indicators
Example: HIV prevention
Intervention
(output)
Knowledge
Attitudes
Self efficacy
Behaviours
Prevalence/Incidence
% of adults exposed
to the TV/radio/print
intervention
% of adults with
accurate HIV
prevention
knowledge
% of
respondents
who used a
condom at last
sex
HIV prevalence amongst
pregnant women aged
15-24 years
% of respondents
who agree that a
woman can refuse
to have unprotected
sex with her
husband if he has
an STI
Approach to evaluation:
quantitative and qualitative
approaches
Quantitative questions
Qualitative questions
Relate to `quantities’
Count or measure phenomena
How many?
What proportion?
How often?
Relate to `quality’ of
variations in experience or the
meaning of experience for different
people
Classify events or phenomena
How?
Why?
eg % of youth that abstained in past year
Reasons for abstaining---lack of opportunities
versus choice to self-protect
What do we monitor and
evaluate?
PROCESS EVALUATION
• How was the intervention implemented?
• Was the intervention implemented
according to plan?
• Under what conditions was
implementation successful?
What do we monitor and
evaluate?
IMPACT/OUTCOMES
Individual level
knowledge
attitudes/beliefs
self-efficacy
behaviours
Community level
-Social norms
-Community
cohesion/social capital
-Civic action
-Sustainability
Evaluation: quantitative and
qualitative approaches
Quantitative
•
Aims to quantify research findings using
numbers, %, ratios etc
•
Quantify frequency of disease/health
events, risks, measures associations
70
60
Standardised procedures decided in
advance
•
Samples used to approximate status in the 30
population
•
•
Measures are estimates, with defined
levels of certainty eg HIV prevalence:
10% (95% CI:8-12%)
Random sampling allows generalisability
to population
Sample sizes large enough to ensure
precision of estimates
51
51
50
•
•
61
40
20
20
12
10
0
OneLove
Multimedia
Soul City TV
OneLove campaign
elements
Booklets
Soul City Radio
Impact evaluations of SBCC
require household surveys
Evaluation: quantitative and
qualitative approaches
Qualitative
• Concepts presented in written words
and images
•
Good for explaining how and why;
good for exploring beliefs and
behaviours
•
Methods and Qs can change as data
collected; can measure impacts that
have not been predicted and generate
new ideas
•
Sampling is purposive---data not
statistically generalisable
•
Sample sizes small
•
Analysis: extract themes, organise
data into coherent findings
• “Yes, Desert Soul has
made a difference to my
life because I used to
have many boyfriends.
When I read this book I
have changed. That
means that I have
dumped all my
boyfriends because I
read that it is
dangerous to have
many partners that
increases the risk of
being infected by HIV”.
(Rural female, 26-35; FG 8)
Results: Impact - Behaviour
Consistent condom use with regular partner among adults (>17yrs)
80
70
73
60
60
45
50
36
40
30
23
20
Exposed (%)
32 35
23
21 20
13
10
8
13
Not exposed (%)
18
6
3
0
Bots
Les
Mal
Moz
Nam
Swaz
Zam
Zim
Results: Impact - Behaviour
• “I have realised the importance of protected sex
and have since refrained from sex without a
condom. The booklet also influenced me to go
for voluntary counselling and find out my HIV
status” – (Lesotho)
• “I never used to like condoms but now with my
girlfriend we buy and use” (Zambia)
Quantitative approaches:
RCT and evaluating mass media
intervention
• Cannot allocate intervention- implemented via
national public media
• Cannot restrict intervention-aimed for maximum
reach of largest possible audience
• Intervention tested in development phase through
target audience research
• Recent systematic review effect of communication
programs on HIV/AIDS—no randomised studies for
full coverage programmes (Bertrand et al)
Evaluation methodologies: quantitative
approaches
What study design to use to evaluate impact of
health communication interventions?
Observational
Cross-sectional, case-control, cohort
Intervention
Quasi-experimental (pre-post),
RCT (individual/cluster)
`GOLD STANDARD’
randomisation
Evaluation methodologies: quantitative
approaches
What study design to use to evaluate impact of
health communication interventions?
Consider: level of implementation of the
intervention
SC intervention
Level of implementation
MEDIA- TV
Population
MEDIA-radio
Population
MEDIA-print
Individual (support other media)
SOCMOB-Soul Buddyz;
community training
Individual/family/schools
ADVOCACY
Population
Quantitative approaches:
alternative study designs
1. Cross-sectional analytic
exposure to SC measured
impact compared in exposed and non exposed;
dose response relationships measured
reverse causality, need to control confounders
2. Cohort
follow-up same group over time
impact compared in exposed and non exposed
3. Pre-post/quasi-experimental
baseline for comparison
Evaluation: qualitative methods
examples
• Focus Group Discussions: facilitator guides group;
participants have similar characteristics
• Interviews in-depth: guided by the respondent
semi-structured: loose set of questions
• Ethnographic observation: immersion in another’s
culture
• Most Significant Change methodology
Participatory M and E
AIMS
• To generate data on activities being
carried out at the local community level
• To increase beneficiaries commitment and
understanding in planning and
implementing community based
development projects
Examples: Community Dialogues
Evaluation: triangulation
Refers to the use of two or more different research
methods in combination, principally as a check on
validity
SC series 7 and 8
• Quantitative data
HIV/AIDS is the result of sinning
Multimedia exposure
None
Low
High
Agree/strongly agree
47.7%
28.2%
19.1%
• Qualitative data
`To me, to watch SC changed my whole life because I was affected
with HIV and AIDS .I was able to teach others to disclose to their
partners….’ (female KZN, rural 16-25)
Evaluation: triangulation
Routine data:
Television Audience Measurement Surveys
Helpline data
AIDS helpline
Graph 2: Presented Calls Breakdow n by Half-Hour Intervals (23-10-07)
300
Number of Calls
2.
National media monitoring
250
200
Presented Calls
150
100
50
0
00
:0
0
01
:0
0
02
:0
0
03
:0
0
04
:0
0
05
:0
0
06
:0
0
07
:0
0
08
:0
0
09
:0
0
10
:0
0
11
:0
0
12
:0
0
13
:0
0
14
:0
0
15
:0
0
16
:0
0
17
:0
0
18
:0
0
19
:0
0
20
:0
0
21
:0
0
22
:0
0
23
:0
0
1.
Time of Day
Number of Calls
200
150
100
Time of Day
00
:00
01
:00
02
:00
03
:00
04
:00
05
:00
06
:00
07
:00
08
:00
09
:00
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
Number of Calls
150
Presented Calls
100
50
300
250
Presented Calls
Number of Calls
200
Number of Calls
00
:00
01
:00
02
:00
03
:00
04
:00
05
:00
06
:00
07
:00
08
:00
09
:00
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
250
0
350
00
:00
01
:00
02
:00
03
:00
04
:00
05
:00
06
:00
07
:00
08
:00
09
:00
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
00
:00
01
:00
02
:00
03
:00
04
:00
05
:00
06
:00
07
:00
08
:00
09
:00
10
:00
11
:00
12
:00
13
:00
14
:00
15
:00
16
:00
17
:00
18
:00
19
:00
20
:00
21
:00
22
:00
23
:00
Graph 2: Presented Calls Breakdown by Half-Hour Intervals (13/11/07)
Graph 2: Presented Calls Breakdown by Half-Hour Intervals (06/11/07)
160
140
120
100
80
60
Presented Calls
40
20
0
Time of Day
Time of Day
Graph 2: Presented Calls Breakdown by Half-Hour Intervals (30-10-07)
Graph 2: Presented Calls Breakdown by Half-Hour Intervals (20/11/07)
160
140
120
100
80
60
50
20
0
0
Presented Calls
40
Time of Day
Community responses
Letters and newspaper articles
Issues and challenges
Multilevel intervention—multiple sources of
data
New activities---new indicators:
dynamic M and E plan
Reach
• Media monitoring: data limited especially
radio; cannot tell if same or new viewers
• Survey data better but infrequent
Monitoring media: special issues
• Need ongoing
monitoring data to
estimate reach
• Two sources of mass
media monitoring data:
1074
1000
979
848
800
Number of sms'
SAARF media monitoring
Response to advice lines
Total number of sms' sent by viewers per episode
1200
766
600
545
560
541
497
453
452
400
200
38
21
Total
l
13
:
12
:
Li
ve
R
Fi
na
ev
is
it
d
ka
st
a
de
de
Ep
is
o
de
Ep
is
o
Ep
is
o
11
:
Tj
a
Tj
a
Ep
is
o
de
10
:
M
9:
u
Episode
ka
st
ad
um
e
al
th
w
al
um
e
ne
th
w
M
8:
u
Ep
is
od
e
ne
ep
ep
ha
7:
L
de
Ep
is
o
Ep
is
od
e
e
ep
ep
ha
rv
ill
6:
L
de
Ep
is
o
Ep
is
o
de
5:
P
ef
fe
rv
ill
e
si
ef
fe
4:
P
de
Ep
is
o
Ep
is
o
de
3:
K
w
ak
w
at
w
ak
w
at
si
2
2:
K
de
Ep
is
o
de
1:
L
ea
rn
in
g
ca
m
p
0
Ep
is
o
• Print media monitoring:
distribution data
Media surveys: SAARF
South African Advertising Research Foundation
1.AMPS: All Media Products Survey
Annual national household survey; F2F interviews
sample of 25 000 adults
TV viewing information: past 4 weeks; past 7 days, yesterday
2. TAMS: TV Audience measurement survey
`Peoplemeters’ attached to TV equipment of sampled
households to monitor what is being watched
Every household member must log onto the system when
viewing
Measures viewership patterns at home only
Daily reports provided
Issues and challenges
REACH
• Print distribution as a proxy for booklet
reach—survey data better
• Training data: from partners —m and e
support/quality audits; feedback
IMPACT
• For mass media cannot do RCT
• Costs of surveys
Conclusion
• Monitoring and evaluating social change
communication complex
• Quantitative and qualitative approaches
complementary
• Alternative study designs to RCT needed
for evaluating media interventions