“There’s nothing so practical as good
theory”
Evaluating ‘Wicked’ Problems
Bob Gardner
July 2011
Clear Starting Points
• English evaluation leader Ray Pawson quotes sociologist Kurt Lewin that
“there’s nothing so practical as good theory”
• Pawson wasn’t arguing for abstract theory, but for ensuring we are
always clear about
• the assumptions we are making that underpin our work – whether planning a
specific service initiative or developing a broad multi-sectoral strategy
• the pathways and factors that we assume will lead from the planning through
service delivery to the hoped-for impact
• how we think all of this will vary depending upon the organizational, social or
policy context
• there are many approaches and as many terms – theory of change,
program theory, framework for change
• but the basic idea is to be really clear about starting points and premises
when planning any initiative
• and it’s this set of assumptions, pathways and objectives in our theory of
change that we evaluate
April-13-15 | www.wellesleyinstitute.com
2
The Problem to Solve = Systemic
Disparities in Ontario
•there is a clear gradient in health in
which people with lower income,
education or other indicators of
social inequality and exclusion tend
to have poorer health
+ major differences between
women and men
•the gap between the health of the
best off and most disadvantaged
can be huge – and damaging
•in addition, there are systemic
disparities in access to and quality
of care within the healthcare system
April-13-15
3
‘Wicked’ Policy Problems
•
•
health inequities and their underlying social determinants of health are classic
‘wicked’ policy problems:
• shaped by many inter-related and inter-dependent factors
• in constantly changing social, economic, community and policy environments
• action has to be taken at multiple levels -- by many levels of government,
service providers, other stakeholders and communities
• solutions are not always clear and policy agreement can be difficult to achieve
• effects take years to show up – far beyond any electoral cycle
have to be able to understand and navigate this complexity to develop solutions
• need comprehensive strategy to tackle the roots of health inequality in the
wider social determinants of health
• from high-level national social and policy change to reduce inequality through
community-based innovation, cross-sectoral collaboration and mobilization
April-13-15
4
…Need Theory
• for complex social problems – from health inequities, through poverty
reduction, to neighbourhood renewal and comprehensive community
initiatives -- we need to be able to:
• identify the connections and causal pathways between multiple factors
• articulate the mechanisms that we assume drive change in these factors and
the problem as a whole
• within that, identify the crucial leverage points that will drive significant
change in the problem
• develop strategies for moving these leverage points
• specify the short, intermediate and long-term outcomes expected and the
preconditions for achieving them.
that is our theory of change
April-13-15
5
… And Here’s How
how to develop theory of change:
• basic principle is to pull group working on the issue together to answer those
questions
• to clearly articulate our starting points – what we think are the key causal
pathways, mechanisms for change and crucial leverage points
• these are our assumptions and goals for how we think program, community
initiative or policy direction will work
• based upon best available evidence, learning and experience
and some wrinkles
• can’t have all the answers and can’t be too prescriptive – don’t wait for
‘perfect’ theory
• be as participatory as possible – especially if community-based
evaluation then tests these assumptions about change mechanisms
and outcomes → contributes to further learning and development
of the initiative
April-13-15
6
Theories of Change for ‘Wicked’ Problem I:
Health Inequities
complex multi-level strategies to tackle
health gaps:
•
•
•
international frameworks such as WHO
Commission on Social Determents
European Union and other international
initiatives
many individual countries
Theory of change = all about reducing
structured inequality :
•
•
•
key features in all:
•
•
•
•
April-13-15
focus on inclusive labour market, antidiscrimination, childcare, affordable
housing, social security and other macro
policies
targets and deliverables for relevant
departments
equitable access to quality health care is
just one part of this broader package
emphasized partnerships with community
service providers and organizations – in
both policy development and service
delivery
•
•
•
health inequities arise out of wider social
determinants of health
won’t solve through health reform alone
underlying structures of social and
economic inequality need to be addressed
by new policies
policies need to be aligned with the
incentives and processes that drive
government
key levers will vary – e.g. could be reducing
prevalence and impact of precarious
employment
has to involve collaboration and
coordination across governments and with
many community and non-government
stakeholders
7
Theories of Change for ‘Wicked’ Problem II:
Community Level Action
Comprehensive community initiatives:
•
•
•
•
April-13-15
broad partnerships of local residents,
community organizations,
governments, business, labour and
other stakeholders
coming together to address deeprooted local problems – poverty,
neighbourhood deterioration, health
inequities, racism
collaborative cross-sectoral efforts –
not just for better coordinated
service provision but for community
development and capacity building
Vibrant Communities – 14
communities across the country
working to build individual and
community capacities to reduce
poverty
Theory of change:
•
•
•
•
•
poverty, social exclusion, etc are not
individual problems but are deeply rooted
in fundamental social and economic
inequality → even the best programs
cannot get at these underlying structures
underlying structures of inequality need to
be addressed by new policy directions
building community capacities is another
direction to address structural inequalities:
• drilling down – Vibrant Communities
works to build up individual skills and
other assets as building block for
community change
broad collaborations are key to both
enhancing community capacities and
implementing policy changes locally
broad coalitions also can highlight the
deep-seated nature of problems → build
community and public awareness and
focus political advocacy
8
Evaluating Complex Policy and
Community Initiatives
•
far too complex to pick apart all the causal relations and patterns of influence:
•
•
•
•
impact can take many years to show up
•
•
•
very difficult to attribute particular changes to particular components of the overall
initiative
will never meet RCT gold standard of proof – that approach can’t capture complexity
but that doesn't mean initiative is ineffective
and that doesn’t mean nothing is happening
traditional evaluation of one program in isolation or of a particular population
among many will not capture this complexity
so we evaluate our theory of change
•
•
•
•
April-13-15
Tamarack talks of framework for change – to highlight the breadth of activities and
stakeholders involved in comprehensive initiatives
we identified levers in our strategy – did they prove to be important in practice?
looking for indications that the change mechanisms unfold as we expected, that the
direction of causal influence and impact is as we expected ,etc
looking for evidence that outcomes anticipated are being achieved, and the change
mechanisms we anticipated have been important
9
Build Use and Users Into Evaluation
• crucial to identify users of evaluation
• what they need
• the context in which they will interpret and use the results of evaluation
• policy makers need to:
•
•
•
•
solve social problems as they see them
determine where to invest scarce resources for maximum impact
avoid mixing up accountability requirements with evaluation
get beyond the assumption that randomly controlled trials are the ‘gold
standard’ of evidence
• community leaders and members are interested in:
• the same – how to solve social problems as they see them – and live them
• how to make case for reform/spending to policy makers
• how to ensure programs and services meet the needs of community members
– and how to continually improve service delivery
• both need to be able to determine – and demonstrate – what types of
initiatives work, for whom and in what contexts
April-13-15
10
Case Study: Health Equity Strategy Into
Action
• goal is to ensure equitable access to high quality healthcare
regardless of social position
• can do this through a multi-pronged strategy:
1.
building health equity into all health care planning and delivery
•
•
2.
3.
aligning equity with system drivers and priorities
embedding equity in provider organizations’ deliverables, incentives
and performance management
targeting some resources or programs specifically to addressing
disadvantaged populations or key access barriers
4.
•
5.
April-13-15
doesn’t mean all programs are all about equity
but all take equity into account in planning their services and outreach
looking for investments and interventions that will have the highest impact
on reducing health disparities or enhancing the opportunities for good
health of the most vulnerable
while thinking up-stream to health promotion and addressing the
underlying determinants of health
11
Equity-Focused Planning
• a critical component of all these strategic directions is good planning
• addressing health inequities requires a solid understanding of:
• key barriers to equitable access to high quality care
• the specific needs of health-disadvantaged populations
• gaps in available services for these populations
• great deal of emphasis on tools such as Health Equity Impact Assessment
• but can’t see particular tool as ‘magic bullet
• to develop effective planning, we need:
•
•
•
•
•
clear strategy
a coherent approach
a repertoire of effective tools and techniques
support for planners and practitioners to effectively use them
good actionable information
• and drilling down: what is our ‘theory’ of how equity-focused planning
works?
April-13-15
12
taking account
of social
constraints &
conditions
April-13-15
not just
individual
programs but
coordination,
partnerships &
collaboration
13
enhanced
access to
health
promotion for
most
disadvantaged
April-13-15
up-stream heath
conditions &
opportunities
improve fastest
for those in
greatest need
14
Complexities At All Stages
• processes and constraints are complex, and outcomes
uncertain and unpredictable, at each of these junctures
• and all of this varies by context:
• particular communities or neighbourhoods – with their different
health challenges and needs
• particular population health and service landscape in specific areas
• community resilience, connectedness, organizing and traditions
• further specified by health condition or concern (e.g. the determinants
and dynamics of mental health are different)
• we need to know what works best at each of these junctures
(and cumulatively), what programs and frameworks work for
particular populations, and what works in varying contexts
→ need to build evaluation in from the start to learn
April-13-15
15