Chair: David Peretz
Secretariat c/o
Evaluation Department
Department for International Development
Abercrombie House
Eaglesham Road
East Kilbride
Glasgow
G75 8EA
mail@iacdi.independent.gov.uk
12th November 2010
The Rt Hon Andrew Mitchell MP
Secretary of State for International Development
Independent Advisory Committee on Development Impact (IACDI)
The Committee’s Terms of Reference require that I send you an open letter once a
year.
This will be my third and last such letter as IACDI winds down its work following
your decision to create an Independent Commission for Aid Impact (ICAI).
My annual letters of 1st December 2008 and 2nd December 2009 focused on ways to
strengthen the independence and effectiveness of DFID’s central evaluation function
and the quality of DFID’s evaluations and assurance systems. Some, but not all, of
our recommendations were accepted, and of those that were accepted some have yet
to be implemented. The Committee believes that most of them as well as those
contained in this letter remain relevant.
IACDI was created with three central aims: to strengthen the independence of
evaluations of development impact carried out for DFID; to improve the quality of
assessments made; and to seek to ensure that lessons from evaluations are learned and
used to inform policy. The following paragraphs address these themes in turn.
Independence
This was the main focus of the Committee’s work in 2008 and we made a set of
recommendations designed to strengthen the independence and effectiveness of
DFID’s internal evaluation function. Some of these recommendations were
implemented, others were not. We particularly welcomed DFID’s new evaluation
policy adopted in 2009, which gave effect to some of our recommendations.
Although this now needs revision, it is important for DFID to continue to set and
1
implement an explicit policy covering evaluation of its development interventions.
Your decision to create the ICAI is very much in line with IACDI’s views about the
need to strengthen independence. We welcome the move, and in particular the
decision to give Parliament and the International Development Committee (IDC) the
role in overseeing the Commission.
The creation of the ICAI will however also bring new challenges. In particular, it
highlights the parallel need to introduce step changes in current practice to enhance
the quality and quantity of internal monitoring and evaluation in DFID. In addition,
as discussed further below, experience suggests that extra effort will be needed to
ensure lessons from external evaluations conducted for the ICAI are learned and acted
on. Members of IACDI have had several discussions with you and your officials
about the remit for and work of the ICAI and the implications for DFID, and a list of
the points we have made is attached as Annex 1. I commend them to you and your
officials, to the ICAI and to the IDC which will oversee its work.
Quality
In 2009, following an in depth review, we made a series of recommendations aimed
at improving the quality of DFID’s evaluations and assurance systems. While these
recommendations were all accepted by your department, resulting action remains
very much work in progress. In particular I would highlight the following areas where
more still needs to be done:

DFID senior management needs to deliver on its commitment to reshape
defensive attitudes into a culture where both external independent evaluation
and internal evaluation are welcomed and championed as essential
contributions to enhancing the effectiveness of the Department’s work.

All major new development interventions should have clear objectives against
which they can be evaluated, with performance and monitoring frameworks
set at the planning stage.

DFID should carry through on its commitment that a senior management
committee will assure the quality of the department’s interactions with,
response to and lesson learning from all evaluations, both those carried out
internally and those undertaken under the auspices of the ICAI.

The completion reporting system that generates ratings of development results
is critical to the tracking of DFID’s performance and should be reviewed:
IACDI had intended to carry out such a review but has not had the time to do
so.
Lesson learning
In 2008, following an early IACDI recommendation, DFID agreed to produce and
publish an annual review of follow up to recently completed evaluations. The last
stock take and annex are published on the IACDI website and, like previous reports,
shows mixed results.
IACDI therefore embarked in 2010 on an in-depth
investigation to identify sources of weakness and better ways to ensure that lessons
2
drawn from evaluations are acted on, based on a detailed ODI Study, a workshop held
with DFID officials and outside experts and our own deliberations.
Annex 2 to this letter sets out IACDI’s findings and recommendations about how to
improve lesson learning from evaluations. We find that DFID falls well short of
having the coherent lesson-learning strategy that it needs. Accordingly, our central
recommendation is that the Department should move to develop such a strategy,
recognising the key importance of face-to-face and informal styles of learning. This
would complement and reinforce the strategy DFID is already developing to improve
the gathering and use of evidence.
There is always a tension between the use of evaluation for accountability and its use
for lesson-learning. The former emphasises independence of the evaluation function;
the latter points to the need for an evaluation culture embedded within the
organisation. However, experience elsewhere suggests that, if managed well, the
tension can be a constructive one, contributing to greater development effectiveness.
The creation of the ICAI and the parallel actions being taken to strengthen DFID’s
internal evaluation function thus provide both a challenge and an opportunity.
The opportunity will be to create strong ownership of DFID’s embedded evaluations
and to build stronger formal and informal mechanisms to make use of the evidence
they create in policy making and project and programme design. The challenge is
twofold: first to enhance and sustain the quality of internal evaluation (the existence
of the ICAI will inevitably require this); and second for DFID to set up the learning
infrastructure needed to react constructively to ICAI’s independent evaluations.
Sustained leadership from DFID senior management, as well as from Ministers, will
be needed on both aspects.
Finally, we are glad that the Head of DFID’s Evaluation Department will again be
publishing an annual review of DFID’s evaluation work and its findings. We
recommend that this practice should continue in future in respect of DFID’s internal
evaluation work.
The International Dimension
Development assistance is increasingly a cooperative activity among donors and
between donors and recipient countries, and the same should be true of efforts to
assess its impact. There is much going on internationally including pooled efforts,
such as 3ie, and joint evaluations. The Committee believes these efforts are important
for the future of development cooperation and hopes that DFID will continue to play
a leadership role. In our own work we have benefitted greatly from attendance at
several of our meetings of heads of other major agencies’ evaluation functions. I
thank them for their assistance in sharing their experience with us and hope that the
ICAI will continue a similar practice.
As noted in Annex 1, we also hope the ICAI will be ready to join with other agencies
in joint evaluations. And as we recommended a year ago, in line with the Paris
Declaration and Accra Agenda for Action DFID should step up its support for
strengthening and, if necessary, helping to build evaluation capacity in recipient
countries so that they can take more responsibility for evaluating the impact of
DFID’s and other donors’ assistance projects and programmes.
3
I should like to thank you and your predecessor for the courtesy, attention and support
you have given to IACDI and to me as its Chair. I would also like to thank the DFID
officials both senior and junior who have provided considerable support and
assistance to the work of the Committee and consistently reacted positively to our
suggestions. Finally, I would also like to place on record my thanks as Chair to my
fellow members of IACDI who have all used their varied expertise and experience to
contribute hugely to the Committee’s work.
David Peretz
Chair, Independent Advisory Committee on Development Impact
4
Annex 1
IACDI comments on the creation of the ICAI, its work and the implications for
DFID

IACDI members strongly welcome the creation of the ICAI as a fully
independent agency. DFID had previously accepted some but not all of
IACDI’s recommendations intended to enhance the autonomy and
independence of DFID’s internal evaluation department. Creation of the
ICAI, reporting to Parliament through the IDC, will lay the independence
issue to rest. It also highlights the parallel need to enhance the quality and
quantity of internal monitoring and evaluation in DFID.

The Commission should focus on evaluating the development impact of UK
aid and development policy; and on providing hard hitting, but constructive
suggestions on how this might be enhanced. This is a considerably wider
remit than simply accounting for whether money is spent appropriately and
effectively. This means the Commissioners and their service provider should
have or develop knowledge and skills about key issues in international
development and how aid works, the risks involved, and the variety of
techniques that are and can be used for assessing the impacts of development
spending and development policy. IACDI hopes the International
Development Committee will support this broader approach.

Independence should not mean isolation. All evaluations should have a
lesson-learning as well as an accountability function. There is always a
tension between the two, though if managed well it can be a productive one. It
will be important that recommendations and lessons learned from ICAI
evaluations are acted on. Experience is that the more independent an
evaluation agency the greater the effort required to ensure that lessons from its
work are learned and used in policy development. Part of the answer will lie
in a constructive – not defensive - attitude by DFID to criticisms and
recommendations from the ICAI. The ICAI should also devote effort to
tracking and encouraging follow up to its reports.

IACDI has strongly welcomed DFID’s intention to step up the internal DFID
evaluation effort in parallel with the creation of the ICAI and in line with past
IACDI recommendations. All major new development interventions should
have clear objectives against which they can be evaluated, with their results
monitored and evaluated to good professional standards. The head of DFID’s
internal evaluation department and the head of the new division promoting
better use of evidence in DFID will need to have the status and stature to
ensure such standards are met. The ICAI should expect to be able to draw on
an effective and high quality internal monitoring and evaluation effort in its
work and should also be ready to comment on the quality of DFID’s use of
evidence including in its monitoring, evaluation and measurement of results
and to encourage further enhancement of DFID’s evaluation practices and
5
methods. The existence of the ICAI will provide a useful incentive for DFID
to maintain quality.

It will be important, at the outset, to be realistic about what the ICAI will be
able to show in its evaluations of aid impact. Often impacts of aid take place
over a long timeframe, and so may not be captured by early evaluations.
Development is a risky business and those providing support for it should take
risks – so some interventions can be expected to fail (if they do not, it would
indicate that DFID is not ready to take sufficient risk). And with aid an
increasingly cooperative business involving several donors and the recipient
country there are the well known problems of attributing specific immediate
and wider outcomes of initiatives to particular players. With ICAI’s emphasis
on accountability to the general public this will require a strong
communications effort from the Commission. Its reports should be written in
accessible language. And there may be a place for surveys to identify what are
the key public concerns to be addressed.

The Commission needs to be adequately resourced for its work. Based in part
on IACDI’s own experience, the envisaged time commitment of
Commissioners and proposed level of secretariat support seem relatively low
in relation both to their task in the early months, when they will be setting
their workplan, and later on when, among other activities, they will be
launching and signing off on some 20 evaluations a year. The IDC and
Commissioners should consider whether extra resources and time
commitment will be needed in the set up phase, and then review the issue
again prior to ICAI’s establishment as a permanent body next year.

It increasingly makes sense to assess the impact of aid in cooperation with
other donors and – ideally – also with recipient countries. DFID has played a
leadership role in promoting such cooperation and it is important that the
UK’s contribution to this area of evaluation continues. IACDI hopes that the
ICAI will be ready to follow the lead of other independent aid evaluation
agencies and take part in joint evaluations where that seems a sensible
approach. It will also be important for all ICAI evaluations to take account of
policies and actions of other donors and recipient countries, and to recognize
that recommendations will often be as relevant to them as to DFID. There is
an accountability function for UK aid to recipient countries and their people as
well as to the UK public.

It would be helpful for the Commission to draw together its views and
findings once a year in an annual report giving an overview of its assessment
of the impact of UK aid.

The IDC might like to consider specifying at the outset that after a period of
say four years there should be an external review of the ICAI and its service
provider, looking for example at the quality of their reports and success or
6
otherwise in getting lessons from their evaluations learned by policy makers,
and considering whether any adjustments should be made.
7
Annex 2
IACDI’s findings and recommendations on how to ensure better lesson
learning from evaluations
Research commissioned by IACDI (ODI Report) suggests DFID falls well
short of having a coherent lesson-learning strategy. The record of acting upon
the specific findings and recommendations of evaluation studies is mixed,
though better for ‘decentralised’ evaluations where there is greater staff
ownership. DFID appears to be less good at using objective evidence to feed
into policy decision-making, and to be weak in sustaining an institutional
memory and developing this over time from lessons of experience.
The chief source of weakness is that DFID’s emphasis on formal mechanisms
does not sufficiently recognise how effective learning most commonly occurs
within the Department, largely through informal and inter-personal sources of
advice and information. Other sources of weakness include insufficient
attention to evidence in all aspects of DFID’s work, excessive work pressures
which do not provide the space needed for reflection and learning, and high
staff turnover resulting in serious loss of institutional memory. An
excessively centralised application of evaluation know-how, inadequate
approaches to the dissemination of knowledge and evaluation findings, and a
reluctance to learn (or acknowledge) lessons from weaknesses and failures
were identified as further shortcomings.
IACDI welcomes steps now being taken in DFID to promote greater use of
evidence in policy making. As part of this effort DFID need to improve its
processes for learning lessons from evaluations, addressing the weaknesses
identified. Although new formal systems are needed, they should be tempered
by realism about how lesson-learning occurs within the Department, and
should thus build upon the face-to-face and informal styles of learning to
which the ODI study has drawn attention. IACDI therefore recommends
preparation and implementation of a lesson-learning strategy taking account
of the following considerations:

Staff ownership of the internal evaluation process should be the
guiding principle. Staff should be involved in the planning, selection,
design and guidance of decentralized evaluations.
Evaluation
outcomes, and consideration of these, should be designed as a learning
process. The decision to strengthen DFID’s embedded evaluations
provides an opportunity to create strong ownership of such
evaluations, without undermining standards, and to build stronger
formal and informal mechanisms to use the evidence they create to
inform and shape policy and project design.
8



DFID’s professional advisers and their networks already provide a
good mechanism for lesson learning, and one which could be exploited
more systematically.
Dissemination of results needs to be re-thought. DFID staff stress the
lesson-learning value of one-to-one briefings, workshops and short
briefing papers.
Reducing staff turnover would help and there is also a need for more
systematic handover and induction processes so that institutional
memory is not lost.
For the ICAI, independence should not mean isolation. Experience shows that
the more independent an agency is the greater the effort needed to ensure that
lessons are learned. Both the ICAI and DFID will therefore need to make a
conscious effort to ensure that the lessons from ICAI’s work, which may often
be challenging, are used to inform policy, by managing their relationship so
that their interactions are constructive, rather than confrontational.
Above all, DFID management needs to champion and promote a culture of
learning and the constructive use of evidence across the department. This will
take strong, sustained leadership from senior management as well as from
Ministers.
9