Intervention Logic
A Presentation to
the Pathfinder Project
Karen Baehler
Victoria University of Wellington
463 5711
karen.baehler@vuw.ac.nz
The problem
Citizens want to know if government is making a
difference. Are we getting results in return for our
taxes?
Ministers want better advice: “The most common
problems which can be discerned in recent
experience are:
• overstatement of what will be achieved;
• under-explanation of how policy actions will achieve the
claimed outcomes.”
From Improving Policy Advice (1993) by G. R. Hawke, p. 27
Victoria University of Wellington
2
The solution
•
•
•
•
•
•
•
Identify goals (outcomes)
Chart a course to those goals
Measure current progress
Stop things that don’t work
Alter things that sort of work
Keep improving things that do work
Discover/invent new avenues to success
Victoria University of Wellington
3
Outline
•
•
•
•
•
•
What is intervention logic?
What are its prerequisites?
What are its uses?
What are its blind spots?
How do we minimise the blind spots?
How do we know if an IL is working?
Victoria University of Wellington
4
What is intervention logic?
• A testable theory of causation
– Linked “if-then” statements
– Action/reaction pairs
• A chain of conditions to be achieved
• Ultimate/end outcome
= policy goal
• Intermediate outcomes and immediate impacts
– Lead to the end outcome
– But are not ends themselves
• A basis for confirming performance
Victoria University of Wellington
5
Start with a backbone
• The vertical dimension of IL
• Outcomes logic, not processes or activities
– Outcome grammar
– Connectedness
• The necessary but not sufficient rule
• Advisor’s mindset
– Optimistic
– Skeptical
Victoria University of Wellington
6
The “If you build it, they will
come” backbone
End
outcome
Reduce
traffic
congestion
People
drive
on it
Immediate
impact
Output
Build
bypass
• Political benefits of
simplicity
• Analytical pitfalls
• What’s wrong with
this backbone?
• Can the matrix fill in
the gaps?
Victoria University of Wellington
7
The more complex backbone
• Add intermediate outcomes
• More assumptions about indirect
causation
Victoria University of Wellington
8
Ultimate outcome: Increased educational achievement/smarter kids
“Good” schools
get even better
“Good” schools
gain pupils
Some “bad”
schools fold
New “good”
schools come
on line
Some “bad”
schools lift
their game
“Bad” schools
lose pupils
Parents choose “best” schools for their children
Parents possess “appropriate” information about schools
Parents aware of & understand program
Victoria University
of Wellington
Output:
School
vouchers
9
The black box at the top of the
backbone
Social policy: Note the
large leaps in logic
that often occur at
the top
Ultimate outcome
realised
Client’s behaviour changes
(How? Why?)
Client responds well
to services
Victoria University of Wellington
10
Plot twists
• When is an intermediate outcome also
an end outcome?
• Can one agency’s / department’s
intermediate outcome be another
agency’s end outcome?
Victoria University of Wellington
11
The backbone as a management tool: Links below outputs
Im m ed ia te im p a cts
A ctivity 1
Inp ut 1
Inp ut 2
O utp ut 1
O utp ut 2
A ctivity 2
A ctivity 3
O utp ut 3
Inp ut 3
Source: R Waite
Victoria University of Wellington
12
The backbone as a risk ID tool:
Collateral outcomes
A c tu a l risk s (p = ?)
U ltim a te o u tco m e
(u n inte nd e d)
U ltim a te o u tco m e
(in te nd e d)
In term ed iate ou tco m e
(u n inte nd e d)
In term ed iate ou tco m e
(in te nd e d)
Im m e d iate im p a ct
(u n inte nd e d)
Im m e d iate im p a ct
(in te nd e d)
O u tp ut
Victoria University of Wellington
13
What are IL’s prerequisites?
• Agreed outcomes for the top row
– Sources
• Statement of intent
• Agency/departmental mission
– The importance of first principles review
– The role of problem definition
• Outcomes (goals) are the flipside of problems
• The “problem logic” model and the black box
• Intervention option(s) for the bottom row
• Common sense
Victoria University of Wellington
14
Group Exercise 1
• Work in pairs
• Choose a familiar policy/output from
your work or from the news
• Produce a backbone linking the output
to intermediate and ultimate outcomes
• Identify strong and weak links
Victoria University of Wellington
15
Move to a matrix (Funnell 1997)
1
Outcomes
Hierarchy
Ultimate
outcome
Intermediate
outcomes
Immediate
outcomes
Output
2
3
Success Factors
Criteria Within
Control
4
5
6
Factors Activities & PerforOutside Resources mance
Control
What are its uses?
•
•
•
•
Conventional uses
Testing existing
policy hypotheses
Testing performance
Improving impacts
through design &
management of risk
Making better use of
existing data
Unconventional uses
• Comparing policy
options
• Identifying generic
intervention
templates for a
department
• Discovering/
inventing new
interventions
Victoria University of Wellington
17
Testing existing policy
hypotheses
• If X, then Y
• Y = f (X)
– Does the raw logic hold? (ex ante)
– Does the available evidence support the logic? (ex
ante and ex post)
– What additional evidence is needed to test the
logic?
• IL breaks an impact evaluation into chunks.
Victoria University of Wellington
18
Testing/confirming
performance
Column 6 in the IL matrix allows us to
• disaggregate performance into chunks
• distinguish chunks that are working well
from those working less well
– based on achievements compared agains
success criteria/targets
Victoria University of Wellington
19
Improving impacts
• Identify conceptual and operational
gaps in existing policy
• Target issues for review (weak links)
• Monitor
– Internal and external risks
– Counter-intuitive causes and effects
• Revise design
Victoria University of Wellington
20
Making better use of data
• Evidence need not relate to ultimate/end
outcomes to be useful
• Findings to date (from NZ or international)
may shed light on immediate and
intermediate links in the chain
• Role of research in the “problem logic”
• Examples
– School choice research and its place in the IL
Victoria University of Wellington
21
Comparing policy options
(via the conventional matrix)
Multiple outcomes = unlinked “criteria”
Criteria
Option A
Option B
Option C
Life years
saved
New
incidents
prevented
Equity
SA1
SB1
SC1
SA2
SB2
SC2
SA3
SB3
SC3
Victoria University of Wellington
22
Using IL to compare options
Step 1: Prepare a backbone for each option
• Compare #’s of links
– More links = more
chances to stuff it
up/more resources
required?
– More links = less
uncertainty, more
robust theory?
– Fewer links =
political plus?
• Compare #’s and
magnitude of weak
links
• Compare #’s and
magnitude of
possible unintended
outcomes
Victoria University of Wellington
23
Using IL to compare options
Step 2: Prepare an IL matrix for each option
Victoria University of Wellington
24
Using IL to compare options
Step 3: Compare across IL matrices
See next slide
• Compare risks across
A, B, C
• Compare resources
needed A, B, C
• Compare performance
contract possibilities
• Compare evaluability
IL sets up more
accurate costeffectiveness
analysis
– Remove
unnecessary steps
before costing
– Identify possible
sources of extra
costs
Victoria University of Wellington
25
A cross-cutting matrix
Option A
Option B …
Outcomes (1)*
Weak logic
Strong logic
Success criteria (2)
Measurable
Unmeasurable
Internal control (3)
High
Low
External risks (4)
Low
High
Costs (5)
$ per X
$ per X
Institutional
capacity (5)
Management (2-6)
High
Low
?
?
Victoria University
*Numbers in parentheses
refer oftoWellington
columns in the IL matrix 26
Identifying IL templates
• The case management model
– Generic steps (slide 28)
– Early intervention example
• The information campaign model
– Generic steps (slide 29)
• The deterrence model
– Mandatory sentencing laws example (slide 30)
• The pollution permits model
– GHGs example (slide 31)
Victoria University of Wellington
27
Reduced long-term costs and/or increased long-term benefits
to the community
Life circumstances/chances of individual are improved;
long-term objectives are achieved
Short-term objectives for individual progressively achieved
Individualised programme put in place to meet objectives
Realistic objectives set for (and with) the individual
Individual’s needs & prospects assessed accurately
Output: Target
group enters
programme
Victoria University
of Wellington
28
Behaviour change leads to improved outcomes
Readers influence others to change opinions/behaviour
Readers change their behaviour
Readers change their opinions
Readers learn the facts
Audience reads literature
Appropriate audience receives literature
Literature passes pretest for readability, etc.
Output: Educational
literature
Victoria University
of Wellingtonproduced
29
Rates of crime X
X offenders work
harder to avoid
apprehension
Potential offenders
avoid crime X
Fewer X
offenders on
the street
Past & potential offenders include
new X sentencing risk in their
personal decision making
Past & potential offenders
aware of sentencing
More X
offenders
jailed longer
Judges understand and apply them
Output: Mandatory
Victoriaprison
Universitysentences
of Wellington for crime X
30
Regulators
sanction
Some plants
emit GHGs
above permit
GHGs
& costs
Some plants
buy add’l
permits
Plants calculate costs & benefits
of investing in cleaner technology
v buying add’l permits v paying
fines for excessive GHGs
Innovations in clean
technology diffuse
“Clean”
firms
profit
Some plants
invest in
clean R & D and
technology
Govt invests
auction revenue
in clean R&D
Permits auctioned to bidders (or other allocation made)
Output: Tradable emissions
permits
created for GHGs
Victoria University
of Wellington
31
Discovering/inventing new
interventions
• The brainstorming approach
– Pick generic policy instruments
– Apply to the problem at hand, using quick,
back-of-the-envelope backbones
• The engineering approach
– Start with the “problem logic”
– Find the entry points in the model
– Fashion interventions for the entry points
Victoria University of Wellington
32
Group exercise 2
• Same pairs
• Choose an end/ultimate outcome and make it
the top “vertebra” of a backbone
• Work down to identify intermediate outcomes
that might lead to that end outcome (based
on your knowledge of how that outcome is
“naturally” produced)
• What interventions suggest themselves as
you move down?
Victoria University of Wellington
33
What are IL’s blind spots?
Equity
• Might the chain of outcomes look
different for different population groups
of interest?
• Might risk factors differ across groups?
• Might different groups need different
activities and resources to reach each
intermediate outcome?
Victoria University of Wellington
34
What are IL’s blind spots?
• Hidden portions of the backbone
– Inputs and activities (below)
– Collateral outcomes (beside)
– Program/theory assumptions (beside)
• Getting trapped in a paradigm
– Tikanga v cognitive-behavioural paradigms for
explaining crime
• Focusing on the lower levels of the hierarchy,
where managers have more control
Victoria University of Wellington
35
How do we minimise
the blind spots?
• Research that contributes to robust problem
logics
– The poverty example
– The drug harms example
• Evaluation that contributes to robust
intervention logics
– The welfare to work example
• Outcomes that reflect actual results in the
community/real consequences
Victoria University of Wellington
36
How do we know when an IL
is working (or not)?
• Does it help us distinguish between
apparently more and less promising
interventions?
• If it just rationalises everything, not robust
• Does it systematically favour some types of interventions
over others? Why? Is this warranted (cross check)?
• Does it help us make better use of existing
evidence?
• Does it help us generate a research agenda?
Victoria University of Wellington
37
Ex ante criteria for a good IL
• Proper “grammar” in the backbone
– Outcomes, not processes or activities
• Each intermediate outcome represents a
necessary but not sufficient cause of the next
outcome
• Success criteria are measurable and lend
themselves to targets
• Activities and resources cover all of the key
factors within the programme’s control
• Activities and resources supply what is
needed to get from one outcome to the next
Victoria University of Wellington
38
Ex post criteria for a good IL
• Are outcomes being produced more cost
effectively than prior to use of IL?
• Are intended and unintended outcomes
predicted more accurately?
• Are there fewer unintended outcomes?
• Are there fewer unexpected
outcomes/surprises?
• Is the department accumulating better
information about its own performance?
Victoria University of Wellington
39
How can IL evolve?
Challenges to be met
• Through multiple
– Accounting for new
applications,
sets of surprises
learning by doing
– Facilitating cross• Through crossdepartmental
thinking on
breeding with other
partnerships for
soft & hard systems
particular outcomes
approaches
– Facilitating equity
analysis
• Through peer review
– Other
Victoria University of Wellington
40