Annex 16: Our impact measurement day – full notes
Tracking changes toward impact for our impact groups – Ghana
Testing our theory of change – Bangladesh
Continuous analysis of context – Niger
Testing our theory of change – Bangladesh
Identified the hypotheses in the theory of change
Prioritise the hypotheses
Propose a methodology to test the hypothesis.
Time was limited. But we learnt

Identify an analyze assumption (test of the ToC) during the program design
process

Make sure we have a good communication strategy around our analysis (test
of ToC) with the stakeholders

Build staff and partners capacity on the test ToC (testing assumptions,
selecting tools)

The test of ToC is an opportunity to consider risks and challenges and
emergency issues
Assumptions
Tested?
Yes
Important for CARE and stakeholders?
Yes
Feasible?
Yes
To
consider
for the
ToC test
and
document
No
No
No
Document
and
explore
other
options
Overall in looking at the program we need to see that the different elements come
together in cumulative ways. If we look at the different processes, we need to measure
our progress towards our impact goal. We ask the questions there about whether
something has happened and that it is leading to something else. There is a need to look
at assumptions at different levels, pathways, domains, between domains and the overall
goal. If we get a level of achievement in the domains, then we ask what the effect has
been in terms of impact.
I am just wondering, I listened to that discussion – there were many thoughts about that.
This is one of the critical changes of understanding particularly for log frames.
Assumptions were ‘outside’ the project, where as here this is something that we will
work on. We need to be careful not to fall back on log frames when we talk about
assumptions. We need to work on bringing assumptions INTO the program – they are
part of what we will work on.
My confusion is about the word assumption. The Theory of Change is an assumption – it
is a hypothesis. Other assumptions related to program (e.g. in Liberia, if there is no
conflict our activities will continue smoothly and we will achieve our goal).
Ghana – Tracking changes towards the impact goal.
We need to:






Clarify the impact group – too many groups, hard to see the relationship
between them.
Reexamine the domains of change
Reexamine the pathways
Do research on sources of data
Select a human condition impact indicator
Other indicators to see if they match the theory of change and lead to human
condition indicator.
Who should do this? Program Leadership Team or Program Coordinator.
We did not look at how and who to include around research partnerships?
We have a working draft; we agreed that we will come annually together to review –
using this type of feedback and the PQAT.
How do we actually manage the iterations? Not fundamental changes necessary
(enhanced rather than real changes). Unless we find that the ToC is invalid and then we
might need to shift something.
How we relate the projects to the programs? How do donor relations fit? How do we
relate them to each other?
In Niger: with CARE Norway, we designed the program with consultants paid for by
CARE Norway. We developed ToC with their investment. Last year a Farmers
Confederation in Norway decided that not engaging men (working only with women)
would lead to a problem. So Norad told CARE Norway this and gave feedback. There
was a demand to engage men. CARE Norway – you have to add men in the program;
now it is about tobacco. So, you have to be an artist.
In Mali: We also have a CARE Norway program, it supported the program design and the
strategic plan was also influenced by this. It was also a good moment for the impact
measurement design. These iterative processes, they happen and they have to happen and
one of the most difficult things for us is to document them and realize that you are at a
point when you are making a fundamental change to our theory of change. It is hard to
define the moment at which the change in realization happened. The Iterative Process
happens at different times and levels, for the impact groups for example our discussion
with communities, our baseline. Underlying causes of poverty, we are using external
resources etc. Then on the domains of change, because we are using the same theory of
change for all programs – there are good opportunities between programs and across
programs to do the testing together (synergy workshops – 2-3 programs come together
and look at a domain of change together across the three programs). We haven’t the
resources to document.
Assumptions and theories of change: high level assumption related to the enabling
environment. We are looking at political unrest – now we have to revisit our theory of
change because the enabling environment has shifted considerably.
Continuous analysis of the context – Niger
Very rich and interesting discussion. We talked about what they are actually doing.
Some examples of what they use, one of the key things is that they have forms of
information that can be used for multiple purposes; in particular an Early Warning
System that had been built up over time, it is a community based system using indicators
defined by the community (food stocks, selling female animals to the market). This
system reports to the commune level. 15,000 communities to 265 communes to 36
departments to 8 regions to 1 nation. This system was being developed over 4 years.
The government has taken over responsibility for it (from CARE project).
There are four main indicators:
Food security
Social cohesion
Environment
Health and nutrition
They include: numbers of conflicts, access to water sources, animal destruction of fields
etc.
You have a committee with equal numbers of men and women who meet with
communities, they pass information up to commune (mobile or physically). This
information communications around trends. At the commune level if there is information
that looks serious, it is validated by NGOS or authorities.
That one system serves as a method to update you on contextual analysis, it also provides
early warning, disaster risk reduction. It can contribute to both developmental and trigger
emergency programming.
The way the validation works provides explanations of why the trends are moving.
The Emergency Preparedness Plan
It is reviewed annually. We use the EWS as well as information from Government and
NGOs.
Thematic Studies
Different kinds of specialized studies in different parts of the country.
Government Poverty Reduction Strategy Information
These four methods are used to update themselves regularly on the context. How does
this information feed into program advocacy strategies? How does this information get
used actively to influence government?
Validation
You were relating this to the early warning system, my question is what kind of good
enough validation standards do they have? To ensure quick responsiveness.
The validation takes a day; decision making on the situation takes place at the commune
level. It isn’t long.
Coup d’etat
This is really good, there are good systems. I wonder whether you have seen the coup
d’etat coming? We set up security systems in advance.
The information
EWS – started for food security, but there was a need to start to understand various issues
(gender, social cohesion). It was tested as an interesting process. It is a very good
system. As we start to see signs we capture it and can respond – even on issues of
marginalization (political and social).
We don’t always use the information strategically. It often depends on the levels of risk
aversion of Country Director.
Global Impact Measurement
The Directory
Replaces the c-pin. It has been overhauled. We have surveyed users and reviewed the
content to ensure data is relevant and focused and manageable.
IT WILL GO OUT IN MARCH, it is on line; it is available in multiple languages. We
can access data on line. Next year the Directory will also include outcome and impact
data (over time).
New types of information – program data, and project data too. Clustered according to
MDGs. We will collect information on partnerships and advocacy. The other big change
is that it is accessible, you can fill it out on or off line and access it to get filtered
information.
CO impact measurement guidance
This year, we are going to different regional PQ meetings and COs to work with them
and documenting the work. We are hoping to have basic CO impact measurement
guidance ready by END FY 10. Then in FY11 further guidance will be sent out.
Setting up systems
Working with some COs to develop the guidance through learning by doing. Also
looking at what it takes to set up an IM system. We spoke with DRDs about this. There
are different challenges in different places – context, connectivity or experience. We
chose three levels of COs – advanced (75% complete), intermediary (35-50%) and
challenging (assess the needs). We will help them and also assess what the investment
needs are in CARE for all COs.
The two COs in WA are Mali (advanced) and Ivory Coast (challenging). Mali and the
Ivory Coast will be visited in mid-May.
Business plan for impact measurement for capacity development
In order to present a plan to the organization systematically to define what it will take to
get a good impact measurement system. There is also a need to convince them of
investment needs (financially). ECARMU has started this process, they have extended
this to two other COs in WARMU (Mali and Niger) and one in SARMU (Malawi), in
collaboration with CARE Norway.
There will be questionnaires sent out, they are not required, but they will help get a good
sense of what the needs are.
Do we wait? Or do we move forward anyway? How do we manage the process? Don’t
wait – the guidance will be based on what we have been talking about. We are trying to
do this collaboratively.