Developing a participatory impact
assessment approach and action
research culture within a
communication for social change
organisation in Nepal
June Lennie, Andrew Skuse, Bikash Koirala,
Nirmal Rijal and Michael Wilmore
Paper presented at the Perspectives on Impact Evaluation
conference, Cairo, Egypt, 2 April 2009
Acknowledgement
The 1st Chief Investigator of the Assessing
Communication for Social Change project is
Assoc Prof Jo Tacchi from Queensland University
of Technology, Brisbane, Australia
Outline of presentation



Benefits of a participatory
approach to evaluation capacity
building (ECB)
Potential of ethnographic action
research (EAR) for creating a
research culture
Overview of the Assessing
Communication for Social Change
(AC4SC) project
Outline of presentation (cont.)




Context of the AC4SC project
Strategies used to develop
stronger M&E systems
Challenges and issues that
emerged
Key outcomes and learnings
Benefits of a participatory approach to ECB





Process is flexible and
responsive to change
Uses inclusive processes that
aim to be empowering
Can generate trust, and a
shared vision and objectives
Can enable better decision
making, program improvement
and sustainability
Can be a cost-effective method
Potential of ethnographic action
research (EAR)



Used to build the research and
evaluation capacity of media
initiatives
Aims to be a rigorous, holistic
approach
Combines participatory techniques
with an ethnographic approach in an
ongoing action research framework
Potential of EAR (cont.)



Uses methods such as
participant observation, in-depth
interviews, and immersion in
research sites
Focus is on community
capacities to engage and take
action to facilitate social change
EAR researchers aim to foster a
research culture
The Assessing Communication for
Social Change project: partners and
funders
Overview of AC4SC



Aims to design and trial a participatory
methodology for evaluating the impacts
of two popular Communication for
Social Change (CFSC) radio programs
made by Equal Access – Nepal (EAN),
and related outreach activities
Aims to develop locally-defined
indicators of social change
Is developing and adapting the EAR
approach and setting up systems that
integrate an EAR-like approach into the
operations of EAN
Radio programs the impact assessment
is focussed on
Saathi Sanga Man Ka Kura
(SSMK) (Chatting with my best
friend) - youth-oriented; mixes
drama, information and
listeners feedback. Aims to
improve life skills and empower
young people
Naya Nepal (New Nepal) - aimed
at peace-building and
reconciliation
SSMK listener club members at
World AIDS Day rally
Evaluation context in Nepal




Lack of community of practice in
evaluation
Lack of M&E understanding at all
levels of a program
Data manipulated to please donors
Lack of understanding by donors of
the complexity of measuring social
change impacts, and the time and
resources required
Contextual challenges and issues
in Nepal





Extreme poverty
Communication and travel
problems
Gender and power issues – low
status of women, caste-based
culture
Sensitivity of issues such as
sexuality & reproductive health
Political instability – frequent
strikes
Maoist demonstration in Bharatphur, Central Terai,
during AC4SC field visit, September 2007
Context in Equal Access – Nepal before
AC4SC project






Lack of leadership in evaluation
Poor feedback systems; use of unrealistic indicators
of program effectiveness & impact; uncoordinated
approach to data analysis
Reliance on ‘success stories’ from listener letters and
other feedback
Limited training in all aspects of M&E and EAR
Unable to demonstrate the usefulness of EAR
Staff wanted to improve their skills in a wide range
of M&E methods
Strategies to develop stronger M&E
systems



Critically reviewed existing M&E
capacities, needs, systems and
practices
Conducted relationship and
team building exercises
Developed better data
management, reporting and
feedback systems
Strategies to develop stronger M&E
systems (cont.)



Held capacity-building
workshops with EAN staff
and practiced skills with
community groups
Identified and trained eight
community researchers
(CRs)
Developed basic manual and
toolkit for CRs
Some key challenges and issues



Initial methods and framework seen as too
complex - led to confusion; problems dealing
with large volumes of qualitative data
Resistance from SSMK team to changing its
M&E systems; lack of cooperation with M&E
team
Human resources and implementation
problems
Key project outcomes




Impact assessment
methodology streamlined and
immediate re outcomes
More effective M&E systems
Collaborative relationships
between M&E and content
team improved
Content team more aware of
importance of M&E and
enthusiastic about learning
from CR data
Some learnings from AC4SC project




Building capacities and setting up
effective M&E systems takes time and
effort but is essential before good
quality data can emerge
Need to understand organisational
culture before collaborative
relationships can be improved
Can train community members to
collect participatory data but they
need continuous mentoring & good
feedback systems
Regular critical reviews needed to
identify & address problems &
encourage critical approach
Some learnings about creating an evaluation
culture in development organisations





Involve all relevant staff in ECB activities
ECB initiatives need to be flexible and open to change
Communication and feedback systems need to be
effective and timely
Benefits of evaluation needs to be actively
demonstrated to program staff
Keep participatory methodologies simple and practical,
esp. in initial phase, but ensure that methods and data
are rigorous and trustworthy
Thank you!
Questions?
Comments?
For more information contact
June Lennie at j.lennie@bigpond.net.au or
Bikash Koirala at bikashkoirala@gmail.com
For more information about Equal Access go to
http://www.equalaccess.org