Useful Impact Assessments in A
Climate of Donor Driven Demand
Maureen O’Flynn
IA the way I see it..
Monitoring
Measures ongoing
activities
Evaluation
Measures performance
against objectives
Main work during
implementation
Main work in middle or
at end of
project/programme
cycle
Focus on interventions Focus on affected
Focus on interventions
Focus on outputs
Focus on outcomes
“What is being done?”
“What has happened?
Did we achieve what
we set out to achieve?”
Impact Assessment
Assesses change in peoples
lives: positive or negative,
intended or not
Can be included at all
stages and/or can be used
specifically after the end of
programme/project
populations
Focus on impacts
“What has changed? For
Whom? How significant is it
for them?”
Impact – So What?
Impact Assessment addresses the “So What”
question...
Thousands of “successful” projects and
programmes make no lasting difference to
people’s lives
Make Impact Assessments more
Useful: 4 Key Messages
• Be honest
• Be realistic
• Be clear
• Be practical
Be Honest
• When was the last time you told a lie, or were
economical with the truth?
• Why did you do this?
• So what’s this got to do with M&E and
Assessing Impact?
Conditions needed to enable honest
impact assessments
•Trust
•
•
•
•
Understanding of what’s needed
Capacity and support
Time
Belief that the exercise will lead to positive
results
Be Realistic
• Two parts:
• In terms of understanding contributions
• In terms of scale and reach..
In terms of contributions....Danny
Who might claim contribution to Dan’s
quality of life? Are they conflicting?
What does Dan think?
The
music
industry
His
idols
gvt
school
Dan
Parents
Designer
clothes
businesses
friends
Sports
club
Transpose this thinking to Tanzania and a
14 year old male student, Bahati..
INGO
The
gvt
Partner sch
music
indust organisationsool
ry
His
idols
Designe
r
cloth
es
busines
ses
Local
Pare
environment
Dan
nts
and
institutions
frien
ds
Spo
rts
club
Bahati
........................... And
how realistic is attribution?
• So many “successful projects”.. So little change
in people’s lives...
• How to realistically understand
organisational efforts in relation to change?
• Understand your organisational Spheres of
Influence and plan your assessment of impact
accordingly
VSO Theory of Change for Education Programme ..
Assess impact for direct spheres of influence;
illustrate impact for indirect spheres
Be Realistic in Terms of Reach
• Its no good claiming impressive scale and
reach or
• giving too many figures in an attempt to
please.. or to be seen to contributing to
MDGs....
• Different orgs are using and therefore
repeating nos ( e.g. HIV in Nepal)
Be Clear:
• Staff and partners need to understand
difference and complementarities between
M&E and impact
• Linking reasons for Impact to approaches..
• Plan for impact from the start of interventions
Linking reasons for Impact to
approaches
What’s the main driver?
• Demonstrate success?
• Improve performance?
• Be accountable to the people (stakeholders)
for whom we are working?
• Use the findings from impact assessments to
advocate for changes in policy and legislation?
Three Possibilities: Approach 1
A retrospective study of a project which has, typically, been
planned using a linear approach to change
The main purpose is to test or verify whether the logic of the
project was correct: did it achieve the changes that it set out
to?
Approach 2:
• More process driven – and less judgemental –
than the first.
• Stakeholders are involved in all stages
• Makes use of the concept of emerging change.
• Purpose:
– Work with stakeholders to visualise changes they
want to see in their lives and support them to
track progress
– Empower stakeholders
– Contribute to organisational learning.
Approach 3:
• Similar to second but “turns the telescope round”. Takes place
some years after the completion of the project or programme
• Identifies changes that have taken place within the
community, and then to consider to what extent they can
have contributed to change in defined areas
• Purpose
– understand to what extent organisational efforts are making a
difference to the lives of the people they claim to be working with and
for.
All organisations need to consider carrying out from time to
time.
Develop a Theory of Change (ToC)
What are they?
• They differ from (but can complement) logical
framework thinking in that they:
– describe a causal pathway by specifying what is
needed for goals to be achieved
– require the articulation of underlying assumptions
which can be tested and measured.
– change the way of thinking about implementation
strategies: the focus is not so much on what will be
achieved, but rather what will change?.
• They can be set at different levels - organisation,
programme and/or project
Climate Change and Development Knowledge
Network (CDKN) ToC Framework
1
Problems CDKN
aims to address
2
Underlyi
ng
causes
of these
problem
s
3
4
Vision
5
Principles of
engagement
6
Who
we
work
with
(targe
t
group
s)
7
How we
work
with
them
(strategi
es)
8
To achieve
(medium and
long term
outcomes)
CDKN
Goal(s)
Dimensions of Change
• Organisational ToCs can be further developed by identifying
“Dimensions of Change” (DoC) to which all of their efforts will
contribute.
• These DoC form the backbone of their impact assessment
process.
– They are inter-related: any one project or programme may be supporting
changes in one or more of these dimensions.
– All organisational interventions should be explicitly working towards changes
in one or more of these dimensions.
– They are usually designed to be interpreted according to regional and national
contexts, but broad enough to be able to facilitate comparability across
different regions and countries.
• Progress against these dimensions is assessed through
monitoring, evaluation and impact assessment processes.
CDKN works in all of these areas
in order to achieve:
Slide 23
Be Prepared
• Plan for impact at the start of the
project/programme
• Be clear about who is responsible for what
• Build IA processes into existing systems and
structures
• Use a few tried and tested tools over complex
and clever ones
• Use findings widely and creatively for
increased impact of impact assessments
Plan Impact from the start:
CDKN Impact Assessment Plan
• Four distinct but complementary strategies, based on
DoC:
•
– Ongoing monitoring of impact built into many of the M&E
reporting formats, and impact data w gathered routinely
alongside other M&E data.
– Part of the mid-term review and evaluation process
– Tracing study carried out over the life span of the project.
This will focus on how identified sample groups are
changing and developing as a result of on-going CDKN
efforts, in relation to the dimensions of change.
– Retrospective study will take place some time after the
completion of the project. Will assess the changes relation
to the dimensions of change, and to what extent CDKN
was able to contribute to these changes.
I N G O
partner
Be clear about who is responsible for
what: Assessment? Or Illustration?
Build IA processes into existing
systems and structures
Add this set of questions:
• In relation to Dimensions of Change:
– What’s changing
– For whom
– How significant is it?
– Will it last?
Tools
Think time and capacity and resources:
• Use a few tried and tested methods (which allow for
triangulation) over complex or innovative and untested
ones that are not clearly understood at field level.
• Involve stakeholders in gathering information. Build their
capacity to do this effectively
• Remember:
–
–
–
–
MSC takes time and can be used very inefficiently
Old favourites are good (FGD, SSI, Interviews)
Appreciative enquiry is empowering
PRA tools are good
Use findings widely and creatively for improved
impact of impact assessments!
• For Planning ( based on findings) : scenario
workshops, appreciative enquiry
• For reporting/ advocacy: variety of media and
other communication tools - youtube, video
etc
Make Impact Assessments more
Useful: 4 Key Messages
• Be honest
• Be realistic
• Be clear
• Be practical
Last Tip.. Don’t give up or get cynical!
• Impact assessment can be rewarding and inspirational,
especially when there is evidence that efforts have led
to improved programmes and improved relationships
with stakeholders.
• Depending on the approach, it can either be complex
and frustrating exercise, or an essential part of
organisational planning and learning.
• As development workers, we need to take the
exploration of changes that we influence in people
lives seriously.