Measuring policy influence:
like measuring thin air?
John Young: j.young@odi.org.uk
The policy cycle
Cabinet
Donors
Policy
Formulation
Agenda
Setting
Parliament
Decision
Making
Civil Society
Monitoring and
Evaluation
2
Private
Sector
Ministries
Policy
Implementation
Research plays a minor role
Kate Bird et al, Fracture Points in Social Policies for Chronic Poverty Reduction, ODI WP242, 2004
(http://www.odi.org.uk/publications/working_papers/wp242.pdf)
There are many other factors
Experience
& Expertise
Pragmatics &
Contingencies
Lobbyists &
Pressure
Groups
Scientific
Evidence
Habits &
Tradition
Judgement
Resources
Values
and Policy
Context
Source: Phil Davies Impact to Insight Meeting, ODI, 2005
Example: health care in Tanzania
“The results of household
disease surveys informed
processes of health service
reform which contributed to a
43 and 46 per cent reduction
in infant mortality between
2000 and 2003 in two districts
in rural Tanzania.”
TEHIP Project, Tanzania: www.idrc.ca/tehip
What is policy change?
• Discursive: Client-focused
services
• Attitudinal: Farmers have
good ideas
• Procedural: Participatory
approaches to service development
• Content: UU20, UU25. New guidelines
and programmes
6
• Behavioural: Approach being applied in
practice
Level of Activity
It’s all about behaviour change
By the
Project
By other
Actors
Inputs
7
Activities
Outputs
Outcomes
Outcome
Outcomes
Behaviour
Change
Impact
Impact
Impact
Animal Healthcare in Kenya
1970s
Professionalisation of Public
Services.
International
Research
Structural Adjustment → collapse
of services.
1980s
Paravet projects emerge.
projects.– Action-research.
ITDG projects
1990s
Privatisation
Privatisation.
ITDG Paravet network
network.change of DVS.
Rapid spread in North. The Hubl Study
Dr Kajume
KVB letter (January 1998).
2000s
Multistakeholder WSs → new policies.
Still not approved / passed!
Lots of methods
• Classical case studies (IDRC, IFPRI)
• Episode studies (ODI/RAPID)
• Stories of Change (Denning)
• Most Significant Change (Davies)
• Micro-Narratives / Sensemaker (Snowden)
• Outcome Mapping (IDRC)
• Impact matrices (Davies)
• Peer evaluations (CHSRF)
• HERG Payback Framework (Brunel)
• Systematic reviews (DFID)
• RCTs (IDS)
9
A systematic approach
1. Strategy and direction –are you doing
the right thing?
2. Management –are you doing what you
planned to do?
3. Outputs – are the outputs appropriate
for the audience?
4. Uptake – are people aware of your
work?
5. Outcomes and impacts –are you having
any impact?
10
A systematic approach
1. Strategy and direction - Logframes; Social
Network Analysis; Impact Pathways etc
2. Management –‘Fit for Purpose’ Reviews;
Quality Audits; Horizontal Evaluation;
3. Outputs – Peer review; Evaluating websites;
Evaluating networks; After Action Reviews
4. Uptake – Impact Logs; New Areas for
Citation Analysis; User Surveys
5. Outcomes and impacts –Outcome Mapping;
RAPID Outcome Assessment; Most Significant
Change; Innovation Histories; Episode
Studies
11
Logical frameworks
Goal
Indicator
MOV
Purpose
Indicator
MOV
Assumptions/Risks
Output 1
Indicator
MOV
Assumptions/Risks
Output 2
Indicator
MOV
Output 3
Indicator
MOV
Output 4
Indicator
MOV
√
√
Theories of change
• Causal Chain - succession of
elements with logical links (eg logframe approach)
• Dimensions of influence overlapping domains which
interact, where it is possible to
influence (eg RAPID CEL
Framework)
• Actor-centred theories - where the
behaviour of actors can be
influenced (eg Outcome Mapping)
13
Theories of change
14
After Action Review
• What was supposed to
happen?
• What actually happened?
• Why was there a difference?
• What can we learn from it?
15
15 minute team
debrief, conducted
in a “rank-free”
environment.
ODI CommStats
16
Stories of change
1. Essential elements:
•
•
•
•
Situation before
Context
Situation after
What changed and why
2. Most Significant Change (Davies)
• Stories of change from
different stakeholders
• Systematic analysis of significance.
3. Micro-narratives (Snowden)
17
http://www.mande.co.uk/docs/MSCGuide.pdf
RAPID Outcome Assessment
18
www.odi.org.uk/RAPID/Publications/RAPID_WP_266.html
M&E in ODI
Think Tank Initiative evaluation
Theory
ofof
Change
and
Assumptions
Theory
Change
and
Assumptions
Theory
Management
of
20
Theory ofUptake
Change and Ass
Theory ofImpact
Change and Ass
Think Tank Initiative evaluation
• Review of ToC: testing the assumptions
– Literature review
– (Desk-based) political economy analysis
– Interviews
• Assessment of capacity change
– Quantitative analysis of M&E data
– Interviews and focus-groups
– Stories of change
• Policy impact
– (In country) political economy analysis
– Case studies
– Stories of change
– Interviews and focus-groups
21