Evaluating the Impact of CBR:
work in progress
on developing an innovative participatory approach
to impact evaluation in Uganda and Malawi
Mary Wickenden and Lyness Manduwa
with
Huib Cornielje, Rachel Gondwe, Steven Msowoya, Anneke
Maarse, Marguerite Schneider, Erik Post
& colleagues from Malawi and Uganda
Introduction
Plan for today
• Brief Overview of the project
• The first round of fieldwork – a review of
some tools
• Lessons learned so far
• Future plans
Background to the study
• There is a gap in theory and practical tools for m&e for CBR
• Aims of this research project:To design a model of impact evaluation for CBR and develop an
evaluation toolkit for use in diverse contexts globally.
In the context of new international and local disability policies, this
participatory project will develop tools to evaluate the impact of CBR on the
wellbeing of disabled people and families; assessing the contribution of both
service delivery and inclusive community support.
The toolkit will be piloted with CBR programmes in Uganda and Malawi in
partnership with local NGOs ,disability experts and local research teams
• Timescale – 2 ½ years ( May 2013- Oct 2015)
• Lead by University College London, funded by Australian Gov Aid prog,
with support from NAD and with local partners MACOHA and NUDIPU
Practical Wish list for CBR M&E
What the evaluation model and tools need to do ?
•
deal with multiple types of data
•
be applicable a variety of contexts and types of
CBR programme
•
have a basic common structure but also flexibility
•
enable some comparisons across CBR progs in diverse
settings and across time
•
be do-able by busy CBR staff and evaluators with only 2-4
weeks for the process
•
bring out the real impact of CBR on PWDs’ lives
•
generate recommendations for changes/future planning
Theoretical ingredients/influences
• CBR matrix
• CBR principles
(Inclusion, Equal access to services and resources, Participation, Creating
equal opportunities, Meeting basic needs, Improved quality of life,
Empowerment, Sustainability)
• Outcome mapping (looks at contribution of different actors
bringing about change)
• Looking at organisational capability (5Cs
evaluation)(of ‘CBR team’)
• Participatory approaches (eg MSC , PADEV)
CBR Matrix (WHO 2004)
Outcome mapping? What’s that?
• An approach used increasingly in a range of
development sectors but not yet much in
disability
• A qualitative and participatory approach to
planning and for evaluation
• Focuses on change and change agents
Changes in behavior as ‘outcomes’
Inputs,
activities,
outputs
Sphere of
control
Outcomes:
Changes in
behavior
Sphere of
influence
Impact:
Changes in
state
Sphere of interest
Who are your boundary partners?
Programme
Beneficiaries
Outcomes:
behaviour change of boundary partners!!!
Stakeholders
Boundary Partners
What to focus on in the evaluation?
We decided on 3 key aspects?
• Access
• Quality
• Impact
6 Stages in our evaluation process
(using a combination of tools)
1
2
3
4
5
6
• Situational analysis – Collecting contextual info and overview of CBR
• CBR programme core team capability and performance review
• Listening to groups of stakeholders
• Listening to people with disabilities/carers, individually and in groups
• Community meeting to validate findings and look forward
• Analysing & summarising findings, reporting and dissemination
Overall structure of the draft toolkit
(used in fieldwork 1)
Stage
Tools
1.
Big mapping, stakeholder mapping, timeline
Document review
2.
CBR team 5Cs evaluation (5 organisational capacities)
CBR manager interview
3.
Focus Group Discussions with
Boundary Partners
Community Stakeholders
DPOs/Parents of CWD
Children with Disabilities
4.
Individual Interviews with People with Disabilities
5.
Community meeting: validation and forward planning
A reflection on some of the tools
Mapping
the district
Stakeholder mapping
Timeline
Stage 3 . Listening to groups of people
Focus group discussions with:•
•
•
•
Boundary Partners
Community Stakeholders
People with disabilities / parents of CWDs
Children with disabilities
Positive aspects of FGDs
Challenges with FGDs
Stage 4. Listening to Individuals
• Individual interviews with people with
disabilities
Positive aspects
Challenges
Most Significant Change Stories
Were collected at the end of individual
interviews or at the end of focus groups
Positive aspects
Challenges
Challenges with Evaluation Process
• People didn’t always know what ‘CBR’ is or who is doing it –
though they can describe activities
• Not being able to identify impact on their life specifically
• Terms like ‘social’ and ‘empowerment’ are difficult to
translate
• People found evaluating quality difficult – no benchmark for
this (ie often something is better than nothing)
• Monitoring data & Documentary evidence quite weak/absent
• Sampling to get really inclusive overview very difficult – some
people left out
• Power issues within structures and within disability
community
• Difficult to get evidence re coverage and those not reached
• How to capture change over time?
Refinements needed
• Need for much better monitoring data to feed into evaluation
• Usefulness of impairment data about individuals?
• More specific ?s re impact – need better probes and better
recording of responses
• Better way to summarise/analyse data
• ?CBR Matrix can be the framework for analysis for the
evaluation – but doesn’t need to be explicit to the participants
• More specific focus on the CBR principles?
• More visual methods – eg symbols/visual analogues
• Better explanation of the use of Outcome mapping and
feeding into planning in the evaluation instructions
Overview of some key findings of the evaluations
 Lack of awareness of CBR as a concept and of the matrix but plenty of
evidence of activity and of impact
 Lack of clarity about who is doing what , who is responsible in CBR
 Exposed lack of monitoring and of documentation
 Gaps in perceptions between the CBR team and the beneficiaries
 Evaluation tools do tell us what is going on on the ground
 The tools collected a mass of rich group and individual data
 The process precipitated linkages and conversations between
organisations locally
 Impact of health, education and livelihoods work is easier to capture than
social and empowerment, but there is activity in all 5 components
 The more mainstreamed CBR is, the more difficult it is to evaluate!
Next stages in the research project
(1 year left!)
• A second round of fieldwork to try out a refined
version of the tools
• Consultation with a wider group of CBR practitioners
and INGOs
• Expected outputs – some kind of manual/toolkit –
format as yet undecided!
• Further trials ? And dissemination
• Links with WHO CBR m&e initiatives
Many Thanks
To all those who have contributed to the project so far…….




The 2 incountry Technical Advisory Committees & consultants
Our local logistics partners NUDIPU/Min of Gender and MACOHA
NAD
The two districts who hosted the 1st round of fieldwork
Machinga in Malawi and Kayunga in Uganda
 The 2 incountry research teams
Uganda: Priscilla Nkwenge, Joel Kawanguzi, Kristie Oroma, (Collins
Kafeero)
Malawi: Aron Mapondera, Lyness Manduwa, George Chiusiwa (Sebastian
Katamula)
And most importantly! Thanks to…….
 People with disabilities and their families, boundary partner
organisations and other community stakeholders who contributed
their valuable views to the evaluation process
References and resources on CBR evaluation
.
•
Cornielje H, Velema JP, Finkenflugel H. (2008) Community based rehabilitation programmes: monitoring
and evaluation in order to measure results. Lepr Rev; 79:36–49.
•
Earl S, Carden F, Smutylo T. (2010) Outcome Mapping Facilitation manual.
http://outcomemapping.ca/resource/resource.php?id=269
•
Finkenflugel H, Wolffers I, Huijsman R (2005) The evidence base for community-based rehabilitation: a
literature review. Int J Rehabil Res 28:187–201.
•
Hartley S (ed) (2002) CBR- A Participatory Strategy in Africa. UCL, Bangalore: India
•
Helander E, Mendis P, Nelson G (1980). Training Disabled People in the Community. An Experimental
Manual on Rehabilitation for Developing Countries. Geneva: WHO.
•
Mont D (2007) Measuring Disability Prevalence, Human Development Network Social Protection,
(Disability and Development Team) World Bank
•
United Nations (2006). Convention on the rights of persons with disabilities. http://
www.un.org/disabilities/convention/conventionfull.shtml
•
Wirz S L and Thomas M (2002) Evaluation of Community-based rehabilitation programmes: a search for
appropriate indicators. International journal of rehabilitation research, 25, 163-171.
•
WHO (2002) International Classification of Functioning, Disability and Health. Geneva. WHO
•
WHO/ILO/UNESCO (2010). Community Based Rehabilitation Guidelines. Geneva. WHO
5C organisational capability evaluation
(Uganda example)
overview of the 5 capabilities
Comparison of performance on the 5CBR components
(Malawi)
Comparison of the 5CBR components between
BPs and CSs
BP
3
health
3
CS
2.5 2
2
1.5
3
Empowerment
1
2
2
0.5
0
2
Social
3
2
livelihoods
3
education
• The study revealed
that both BPs and CSs
are involved in all the 5
CBR components.
• The extent of their
involvement was
somewhat different.
Stage 5. Community meetings:
Validating findings and Looking Forward
Analysing and summarising data
Draft report
Validation meeting with stakeholders
Selection of Most significant change stories
Discussion re community recommendations and future
actions
Positive aspects of CBR
Challenges for CBR
Final report edited in the light of the community meeting
In Theory – in designing our evaluation
we aimed for these important aspects










An equity focus
Inclusion of a wide view of perspectives
Involvement of communities as a whole not just PWDs
breadth of info and depth of information
to aid learning within the CBR programme as much as to evaluate
performance
to know IF change has happened and also HOW it has happened
to know about unsuccessful/negative changes as well as positive
unexpected/ unpredicted changes
to know what the barriers and constraints to change are
to collect useful info for a purpose not just facts for their own sake
to know about changes in ‘informal’ aspects such as people’s
behaviour(showing change of attitudes) and also changes in formal
structures – such as the way of change of law has been implemented
or how a service works differently now
A look at Outcome Mapping
• A participatory method for planning,
monitoring and evaluation
• Focused on changes in behaviour of those
with whom the programme works
• Oriented towards social & organizational
learning
• Looks at the ‘bigger picture’
• Recognizes changes as: continuous, multidimensional, complex, non-linear
• Learning as you go – iterative/flexible
• Looks for contribution rather than attribution
Abstract for AfriNEAD 2014
SYMPOSIUM IN MALAWI
Authors/presenters
Mary Wickenden, Huib Cornielje, Marguerite Schneider, Erik Post, Rachel Gondwe, Steven Msowoya and colleagues from
Malawi and Uganda
Research title
Evaluating the impact of CBR: progress report on a study developing an innovative participatory approach to impact evaluation in
Uganda and Malawi
Background
Community based rehabilitation is promoted by WHO as the intervention model of choice to achieve increased wellbeing,
participation and access to rights and equality of disabled people, particularly in under-resourced settings. This strategy for multisectoral inclusive development has been reconceptualised during the last decade, as described in the WHO CBR guidelines (2010).
CBR is widely used throughout Africa, although in different forms. However evaluation of the impact of CBR is problematic.
There is a lack of consistency of approach which makes assessment of the value of CBR and comparison across programmes
challenging.
Objective(s)
The paper will present the findings so far from an on-going study in Uganda and Malawi. The research aims to develop an
innovative and flexible model for outcome and impact evaluation of CBR, which will be usable in a variety of contexts and types
of CBR programme.
Methodology
The research team have developed a model and toolkit for impact evaluation, based principally on Outcome Mapping. This mixed
methods approach recognises the nonlinear and complex nature of community interventions and provides ways of making
processes participatory. This has not previously been used in the disability arena and we are combining it with other approaches,
such as Most Significant Change, PADEV, Sensemaker and the 5Cs method. Initial pilots in Malawi and Uganda will yield
preliminary data on which to base a more refined and finalised version of the tool.
Results
This paper will present the analysis from the initial fieldwork. It will discuss the successes and challenges experienced so far and
the proposed adaptations to be trialled in the second stage of the study.
Conclusion
The potential of Outcome Mapping alongside other methods to provide a new participatory approach to impact evaluation of CBR
as a strategy for inclusive development will be described and critiqued.