3D impact analysis
A new tool to approach impact evaluations
April 23, 2015
Rob D. van den Berg
Visiting Fellow, IDS
CDI is a joint initiative between:
and
and
For more information: www.ids.ac/cdi or email: cdi@ids.ac.uk
1
Overview
•
•
•
•
•
•
•
What is impact?
What is evidence?
What is causality?
What is attribution/contribution?
Time
Space
Scale
2
Impact
• Impact is an ordinary word in the English
language
– “the effective action of one thing or person on
another; the effect of such action; influence;
impression”
• Its meaning cannot be scientifically claimed
• Demand for impact evidence can refer to a
wide variety of effects, influences and
impressions
3
Evidence
• Evidence is an ordinary word in the English
language
– “the quality or condition of being evident;
clearness; evidentness”
• Its meaning cannot be scientifically claimed
• Demand for impact evidence can refer to a
wide variety of qualities or conditions
4
Causality
• The word “cause” is an ordinary word in the
English language
– “A person or thing that gives rise to an action,
phenomenon, or condition”
• Its meaning cannot be scientifically claimed
• Demand for evidence of cause and effect can
refer to a wide variety of actions, phenomena
and conditions
5
Attribution / Contribution
• Both words are ordinary words in the English
language, with great variety in meaning
– Attribution: in copyright law, requiring an author
to be credited; in marketing, assigning a value to a
marketing activity based on desired outcome;
journalism, practice of attributing information to
its source
– Contribution: donation, sharing, payment,
publication, a song by Mica Paris
6
Impact Evaluation
• Focus of Impact Evaluations:
– Impact = evidence of causality between an
intervention and the desired effect by establishing
a counterfactual through controlled
experimentation, which attributes part of the
effect to the intervention
• This partially meets the demand for impact
evidence in politics, the media and society
• So what to do with other demands?
7
Meeting impact demand
• Broaden the concepts of impact and causality
• Broaden the range of scientific methods and
tools
• Develop a framework for understanding
demand for impact evidence
• Incorporate issues of time, space and scale
• This is urgent, given the adoption of the
Sustainable Development Goals in September
2015
8
Understanding causality
• Schaffer (2013) proposes two kinds of causality:
“difference” and “production”
– Difference: with/without (counterfactual) analysis
– Production: A “produces” B (natural systems, physics &
technology)
• Concepts that include causality:
– Catalytic roles (the change agent speeds up change but is
not involved in the change itself)
– Dynamic and chaotic systems (feedback loops, iterative
processes, Fibonacci sequences)
• Terry Pratchett: “hardly anything important has a single
cause”
9
Issues of time, space and scale
• Some changes can be observed immediately – others take
decades
– Short-term results: vaccinations, technology transfer, new
livelihood approaches etc.
– Either short- or long-term: market transformations, societal
change
– Long-term results: health trends, ecosystem services, ozone
layer
• Some changes are local, other regional, national or even
global
• Some changes concern one actor, intervention or
institution, others involve multiple actors or institutions,
and multiple sectors
• Sustainable development involves longer time horizons,
overlapping locations and many scales
Matrix of evaluable impact
• Impact can be evaluated at different moments in time: ex
ante, in real time and ex post
– These can be refined: ex ante tends to be done once, but real
time and ex post have many possibilities
– Different processes tend to have different time horizons
• Geographical space runs from local to global
– These can also be refined: the boundaries of societies,
economies and natural systems are different from each other
and may overlap
• Scales of involvement can go from one actor to a
multiplicity, from one market to a full economic system,
from one governance level (or sector) to many
– Actors, markets and governance may not fully overlap
Appraisal
Inception
Implementation
End-of-project
2 years ex-post
5-8 years ex-post
12
Local
Regional
National
Global
Multisector
Multiactor
One
13
Matrix dimensions space and time
Ex ante
Local
Inception
Mid-term
End of
project
Ex post <
2 years
Ex post 58 years
Experimentation
National
Regional
Global
Ecosystem
Monitoring and data
analysis (including “big
data”)
Mixed methods /
theory of change
approaches
(overlap with
other rows)
14
Matrix dimensions space and scale
One inter- Multiple
vention
interventions
Enabling
environment
Market
change
Market
transform
-ation
Climate
change
Local
RCTs
National
Regional
Global
Mixed methods / theory of change
Monitoring
Data
analysis
Ecosystem
(overlap with
other rows)
Double evaluand evaluations
15
Matrix dimensions scale and time
One inter- Multiple
vention
interventions
Enabling
environment
Market
change
Market
transform
-ation
Climate
change
Ex-ante
Inception
Real-time
End-ofproject
RCTs and
quasiexperimental
Monitoring
Data
analysis
Ex-post
Mixed methods / theory of change
16
Counterfactual analysis
One inter- Multiple
vention
interventions
Local
National
Regional
Global
RCTs
Enabling
environment
Quasiexperimental
and QCA
Social
Network
analysis
Market
change
Market
transform
-ation
Climate
change
Delphi
Modelling of data
and
experimentation
(quasi- and natural)
Ecosystem
(overlap with
other rows)
17
Production causality
One inter- Multiple
vention
interventions
Enabling
environment
Market
change
Market
transform
-ation
Climate
change
Local
National
Regional
Inspection, validation
before/after data
Systems evaluation
Global
Ecosystem
(overlap with
other rows)
Verification of data, trend analysis
18
Thank you!
rdwinterberg@gmail.com
For more information: www.ids.ac/cdi or email: cdi@ids.ac.uk
19