Using a realist evaluation framework
for a learning-focused evaluation of
the Cambodian Initiative for
Disability Inclusion: lessons and
challenges.
Dr Robin Vincent robvconsult@gmail.com
(Independent Consultant and Visiting Fellow, Durham School
of Applied Social Sciences)
CARES Conference Liverpool October 29th
Overview
 Why a realist evaluation framework? – challenging dominant
evaluation practice in international development
 Cambodian Initiative for Disability Inclusion – background
 Components of the evaluation
 Theory of change of the CIDI programme
 Key thematic findings
 Reflections on the method and process – value for ‘learning’ and
ability to tackle complexity
 Reflections on presentation of findings
 Reflections on theory - complex causality and mechanisms
Why realist evaluation?
Engagements with complexity theory and complex social change - attempts to put into
practice insights for evaluation of development practice

Panos Institute, evaluation of ‘Communication for Social Change’ in HIV, health and
development using insights from complexity (Vincent 2011)

UNAIDS, HIV prevention Think Tank 2009 – assessing the ‘social drivers’ of HIV
infection (Vincent 2009)

DFID Uganda HIV prevention strategy using Realist Evaluation and Qualitative
Comparative Analysis (late 00’s)

Collaborations with Durham School of Applied Social Sciences, Liverpool School of
Tropical Medicine.

Intransigence around ‘traditional’ experimental methods in range of proposals reviews
processes

Evaluation of community/public engagement with international health research
(programmes supported by Wellcome Trust)
Cambodian Initiative for Disability
Inclusion - CIDI

Complex programme with national reach
 38 partner organisations, 55 projects over 2.5 years
 aiming ‘to improve the quality of life of people living with disabilities by
supporting national efforts to address the risks, causes and consequences
of disability’

Variety of projects - disability awareness, physical rehabilitation, inclusive
education, psychosocial support, accessibility, livelihood security including self-help
and savings groups, mine risk reduction, road safety….

Focus on ‘influencing strategies’ rather than more direct ‘outputs’
 facilitating networking among organisations in disability sector (including
disabled people’s organisations and ‘mainstream’ organisations)
 Capacity development of organisations in disability sector

Wanted a learning focused evaluation to address complexity and link to
participatory methods of the project
Evaluation elements
 Negotiating with commissioning team and advisory
board - adapted Realist Evaluation framework
 Evaluation team of 2 Cambodian based programme
managers, one Australian Red Cross quality team member
and lead evaluator
 25 days of lead evaluator overall
 10 days in Cambodia visiting projects and doing interviews,
and 3 day participatory workshop
 Range of existing sources of data
 Additional data gathering
Existing data
 Existing eclectic mix of programme monitoring data, including a summary
report of outputs across all projects
 Review of role of partnerships in catalysing change in disability inclusion by
‘Enable’, organisation of people with disabilities conducted just before the
evaluation (seen as part of accountability)
 CIDI on-line forum for organisations involved in the programme.
 Documentation of regular all-partner participatory networking and learning
workshops
 Existing participatory self-assessment/evaluation exercises undertaken by
CIDI participants (capacity development/empowerment dimension):
 ‘peer to peer’ evaluation and exchange visits
 ‘most significant change’ stories
 regular review meetings with CIDI programme team
 video accounts of where projects made a difference
additional elements

Distilling ‘theory of change’ from secondary research and programme documentation

‘Realist’ interviews and group discussions with range of stakeholders with different
relationships to programme and ‘division of expertise’ – (Pawson & Tilly 2006, Pawson 2013)

Quantitative survey on networking and capacity development related outcomes and ‘how’
programme had helped (both as a data source and to focus participatory evaluation exercises).

Participatory evaluation exercises at 3 day workshop with 32 out of 38 organisations
(drawing ‘journey’s of change’, network outcomes mapping, group discussion/analysis)

Visits to projects, observation at events

Ongoing reflection with key programme managers on emerging findings

Integration and analysis of data sources along realist evaluation lines and writing report
In search of the theory of change…
 CIDI programme documentation
 a ‘people- centred programme logic’ drawing on network
perspectives in project/programme design
 Unelaborated theory of change on paper
 Review of secondary literature on networking and capacity
development in development contexts
 Key informant interviews with programme managers during
planning/inception phase of the evaluation
Limitations of evaluation
 Volume of documentation – review was selective and
focused by theory of change and pragmatics of time
constraints
 Language:
 some interviews remained at the ‘general’ level
 Translation in participatory workshop exercises – time and
comprehension
 Some partner organisation ‘peer’ evaluation exercises had a
different focus from the current evaluation
 Focus on ‘what worked’, less on where things didn’t work
Evaluation findings
Evaluation report available (in English and Khmer)
Key concrete things that make for effective networking
 Consistent use of participatory methods underpinning
programme partner meetings AND personal style of facilitators
(former less recognised before).
 Multiple and overlapping methods for communication among
partners and bringing people together to share experience and
reflect
 Encouragement of horizontal connections, exchanges and
collaboration among partners
 Involvement of ‘local champions’ with recognised commitment
to the issue
Evaluation findings
Key concrete things that make for effective capacity
development
 Focus on identified needs of partner organisations
 Practical action learning focus of training
 Promotion of ongoing reflection on practice and action
learning
 development of personal relationships, trust and respect
(underpinned ownership and responsibility for learning and
change)
 flexibility in project management
Evaluation findings
Programme design and management
 Variety of ‘ways of working’ in the programme:




flexibility with partner organisations fostered respect
regular review and reflection
Creativity, having fun and practical at meetings
pragmatic orientation of programme staff on helping partner
organisations to achieve ‘their’ objectives
 layering of networking, capacity development and communication
activities reinforcing each other
 Open, flexible orientation consistent across layers of programme
management and staff
 Programme model emphasising relationships
Evaluation findings
Challenges identified
 Language – working more consistently across English
and Khmer demands more resources and facilitators
 Tools for capacity assessment over time may help
for more systematic attention in identified areas of need
(with caveats)
 Consistent long-term funding (experience of various
changes, relatively short programme, being wound up
at time of evaluation)
Reflections on process
 Project staff had a clearer idea of how and why the
programme was working (where it was). Sense of
being successful made more concrete and ‘tangible’ –
validated.
 Partner organisations keen to feed into their learning
and future work – specific meeting to discuss and
unpack the findings (consistent with learning focus of
work and evaluation)
 Australian Red Cross used evaluation as a basis for
regional reflections on processes of supporting
networking and capacity development
Reflections on process
Personal reflection that core elements of the Realist evaluation
framework valuable for:
 Integrating data of a variety of types
 Integrating input from varied stakeholders
 the focus on theory of change as the ‘adjudicator’ (Pawson
2013) helping to navigate complex and social processes that
are often poorly understood/articulated
 Good for learning AND accountability and integrating
quantitative and qualitative, participatory and ‘outsider’
perspectives
 Perceived ‘authority’ of method as scientific AND learning
focus
Reflections on process
 Importance of advocates within organisation for evaluation approach
and willingness to engage with complexity
 Time constraints and data overload – here TOC helps, providing some
‘boundaries’ of the inquiry.
 lack of detailed engagement with context both as ‘setting’ and ‘type of
participant’
 First articulation and refining of the core theory (s) of change around
networking and capacity development that fitted the data gathered.
 Humility Realist evaluation really a temporary full stop - needs to be
positioned within changing realities of programme AND wider body of
cumulative learning.
 Need to make the case for the necessary secondary review of
literature/practice – relied on existing knowledge of evaluator
Presentation of findings
Findings of the evaluation were presented as a set of short mutually
reinforcing sentences aiming to explicitly articulate links between
aspects of interventions and the ‘conditions’ that made for success
Networking:
“A combination of platforms for sharing information and knowledge
among CIDI partners, including face-to-face and on-line forums mutually
reinforced each other to build communication and relationships
among CIDI partners”
“Quality and intensity of interactions facilitated the establishment of
trust among CIDI partners, providing a foundation for more substantive
relationships, joint working and mutual learning among a core of CIDI
partner organisations”
Presentation of findings
Capacity development:
“training in core areas of organisational functioning and
technical areas of disability inclusion programming responded
to the identified needs of partners, which increased its
perceived relevance and uptake”
“trained was delivered using participatory learning and practice
oriented approaches so that it could be applied to the
immediate needs of organisations to embed the learning”
Context, mechanism, outcome configurations..
 Realist evaluations often present tables of context, mechanism,
outcome configurations
 Importance of accounting for articulation of these (context,
mechanism and outcomes) in any particular instance (‘Realist
Diagnostic workshop’, Pawson 2013 - risk of lack of articulation).
 Presentation of CIDI findings showing combination of conditions or
‘active ingredients’(though less attention to context)
 Resonance with insights from complexity theory - parallels with
notion of ‘configuration of conditions’ and causal factors that
lead to outcome patterns in case-based comparative work (Ragin and
Rihoux 2009)
 Notion of ‘conditions’ rather than ‘context’ (Tilly 1996)
Reflections on causality

Causality in social practice as ‘complex and contingent’ (Byrne 1998, 2002, 2005) and
emergent - in nested levels of social reality - with causality flowing in many directions (‘up’
and ‘down’, ‘across’)

Challenge of range of different levels and processes that need to be simultaneously taken
account of in understanding social practice

Complexity consistent theory can usefully help to understand and evaluate the nested levels
of reality in social change (Westhorpe 2012, 2013, Byrne and Callaghan 2014)

Granularity or level of focus is key for what bodies of theory are relevant, which data to gather
and which mechanisms, contexts and outcomes are relevant

Theory (s) of change under primary focus is key for ‘managing’ this, putting boundaries
around the inquiry.

‘boundaries of inquiry’ for systems and complexity theorists (Williams and Imam, 2007, Byrne
and Vincent 2011, Byrne and Callaghan 2014)
One persons outcome is another’s
context….or do I mean mechanism…

Mechanisms of ‘critical realist’ philosophy may operate at a range of different levels

Some generate outcomes that become the ‘context’ for other mechanisms operating at
different scales (gender difference generated as an emergent outcome from a range of
mechanisms but also acts as a contextual factor)

In the CIDI case the establishing of trust through respectful interactions, was both


an important outcome

also in a sense, a key part of how the mechanism of collaboration and mutual learning
worked.
and a key condition or contextual factor for practitioners taking ownership of learning
and responsibility for organisational change

Where causality is complex and emergent, distinctions between context, mechanism,
outcome may depend on focus of inquiry (or action) – valuable heuristic for analysis

Layers and combinations of cause also important, with feedback and circularity among
them in social practice (theories of practice - Bourdieu 1977, Crossley 2011)
Conclusions

Realist Evaluation has clear core principles of inquiry useful in time/resource
constrained settings

useful for learning AND summative focused evaluation (if by summative mean
interpretation of what the evidence is telling us about outcomes)

Humility about partial picture, but confidence about findings within the focus of the
evaluation

Reminder of the need for cumulative learning among body of evaluations – steps
towards ‘portable programme theories/reusable conceptual platforms’ useful

Link between Realist informed systematic reviews and evaluations clear (with
caveats)

Complementary with complexity theory worth developing further
 understandings of emergence, recursive causality and complexity consistent theory
 Complementarity with complex case based methods and ‘configurational’ approach of
Qualitative Comparative Analysis QCA, (Ragin and Rihoux 2009, Byrne and Ragin
2009).
Thanks!
robvconsult@gmail.com
Input/review from David Curtis, Australian Red Cross
CIDI Evaluation report: http://www.redcross.org.au/files/CIDI_REFLECTION_REPORT-%28En%29_indd.pdf
Vincent (2011) Insights from Complexity theory for the evaluation of development practice
http://wiki.ikmemergent.net/files/1203-IKM_Emergent_Working_Paper_14-Complexity_Theory-March_2012.pdf
Vincent (2009) Assessing social and structural change for HIV Prevention: http://e-mops.ning.com/page/insights-fromcomplexity-theory-for-evaluation
Byrne and Vincent (2011) Evaluating Social Change Communication for HIV/AIDS: New directions. UNAIDS discussion
paperhttp://www.unaids.org/en/media/unaids/contentassets/documents/unaidspublication/2009/20111101_JC2250_evaluating-social-change_en.pdf