Type of Review: Annual Review
Project Title: Estimating maternal and newborn lives saved using
Lives Saved Tool (LiST)
Date started: Oct. 2012
Date review undertaken: Oct. 2013
Instructions to help complete this template:
Before commencing the annual review you should have to hand:






the Business Case or earlier project documentation.
the Logframe
the detailed guidance (How to Note)- Reviewing and Scoring Projects
the most recent annual review (where appropriate) and other related monitoring reports
key data from ARIES, including the risk rating
the separate project scoring calculation sheet (pending access to ARIES)
You should assess and rate the individual outputs using the following rating scale and
description. ARIES and the separate project scoring calculation sheet will calculate the overall
output score taking account of the weightings and individual outputs scores:
Description
Outputs substantially exceeded expectation
Outputs moderately exceeded expectation
Outputs met expectation
Outputs moderately did not meet expectation
Outputs substantially did not meet expectation
Scale
A++
A+
A
B
C
Introduction and Context
What support is the UK providing?
The UK (DFID) will provide up to £267,393 (October 2012 – April 2017) for the development,
implementation and dissemination of a model (LiST – Lives Saved Tool) to estimate the number of
maternal and newborn lives saved through DFID support. The model has been adapted for DFID use
from the original LiST computer model developed by John Hopkins University. The LiST work is to
enable DFID to calculate and report our results. This is an important component of our commitment to
monitoring and reporting progress made and in delivering results.
1
What are the expected results?
The expected results are:
 a working computerized LiST model (adapted from original LiST model to meet DFID needs) to
generate lives saved estimates;
 produce a detailed methodological note to guide the collation of input data for the model;
 produce annual estimates of maternal and newborn lives saved for all countries where DFID has a
bilateral programme – both country level and aggregate results;
 produce final cumulative estimates of maternal and newborn lives saved through DFID support
during the Reproductive and Maternal and Newborn Health (RMNH) Framework for Results (FfR)
period (2010 – 2015).
What is the context in which UK support is provided?
Context and need for a DFID intervention
Context
The UK Government is committed to reduce poverty in poorer countries, in particular through achieving
the Millennium Development Goals (MDGs), including MDG 4 (to reduce child mortality) and MDG 5 (to
improve maternal health). While progress is being made towards attainment of the MDGs, the rate of
progress is insufficient to attain the MDG targets, with particular challenges in reaching the poorest and
most disadvantaged as described below:
MDG 4: Reduce child mortality
Globally, nearly 19 000 children under five years die every day, amounting to 6.9 million deaths a year.
Although mortality rates for under-fives dropped by 41% between 1990 and 2011, these rates are still
insufficient to achieve MDG 4 by 2015, with only 6 out of 10 regions on track. A large share of child
mortality (40%) is attributable to deaths in the neonatal period with approximately 3 million newborns
dying in the first month of life1.
MDG 5: Improve maternal health
Worldwide, 287 000 women die each year from complications arising during pregnancy and childbirth,
representing a 47% decrease globally between 1990 and 2010. Over 99% of maternal deaths
occurring in developing countries2. For every woman who dies during pregnancy or childbirth an
additional 20 or 30 suffer complications such as such as severe anaemia, incontinence, damage to the
reproductive organs or nervous system, chronic pain, and infertility3.
To support attainment of the MDGs, in 2010 DFID published a Framework for Results for improving
reproductive, maternal and newborn health in the developing world (FfR) setting out how the UK will
support action to:

save the lives of at least 50,000 women during pregnancy and childbirth and 250,000 newborn
babies by 2015;

ensure at least 2m safe deliveries, particularly for the poorest 40% by 2015;

provide access to modern methods of family planning for 10m additional women, including 1m
young women aged 15 – 19 by 2015; and

prevent more than 5m unintended pregnancies by 2015.
1
Level and trends in child mortality. Report 2012. Estimates developed by the UN Inter-Agency Group for Child Mortality Estimation.
WHO, UNICEF, UNFPA, World Bank (2012). Trends in Maternal Mortality: 1990 to 2010. Geneva: World Health Organization.
3 C Murray and A Lopez, eds. Health Dimensions of Sex and Reproduction, Vol. 3, Global Burden of Disease and Injury Series. Boston:
Harvard University Press, 2008.
2
2
The Government is also committed to making British aid more effective by improving transparency and
value for money4. This includes monitoring and reporting progress made in delivering results
commitments.
In 2011, DFID published a Results Framework outlining four levels of indicators that capture each main
stage through which inputs are transformed into developing country results:5
 Level 1 indicators focus on progress against the MDGs, outcomes that cannot be attributed to DFID
alone but result from the collective action of countries and diverse development partners;
 Level 2 indicators measure outputs and intermediate outcomes that can be directly linked to DFID
interventions. Level 2 indicators relevant for RMNH include:
a) number of births delivered with the help of a midwife, nurse or doctor through DFID support;
b) number of additional women using a modern method of family planning through DFID support,
c) number of maternal lives saved through DFID support; and
d) number of newborn lives saved through DFID support.;
 Level 3 and 4 indicators measure the operational and organisational effectiveness of DFID, to
assess how effectively and efficiently DFID conducts its business.
In June 2012, DFID published its first Annual Report, as the main vehicle for reporting DFID results and
confirmed the commitment of the Government to publish progress against all indicators at all levels on
an annual basis.
In addition to the measurement of results, the RMNH FfR outline 4 additional processes for the overall
monitoring and evaluation of DFID’s contribution towards improving maternal and newborn health:
1. Routine project and programme level evaluation;
2. Mid-term review (2013);
3. Global and partner monitoring and evaluating processes; and
4. Final evaluation (2015-16).
The Level 2 RMNH indicators, reported annually, will feed into and inform the above processes,
providing essential input data for the evaluation of programmes, the mid-term review and final
evaluation of the FfR as a whole.
Need for the intervention
DFID already has established internal processes to measure and report the number of safe deliveries
and number of additional users of modern methods of family planning through DFID support. Standard
methodology and supporting guidance notes have been developed and disseminated to relevant
cadres within DFID, including results and health advisors with responsibility for DFID programmes. The
first set of results were reported by Country Offices and multilateral programmes accordingly and
published in the 2012 Annual Report.
In contrast to the above, DFID needed to establish a methodology to measure maternal and newborn
lives saved. The 2012 Annual Report did not provide results on these indicators, but reported instead
that DFID was working with partners to establish an internationally agreed methodology. An adaption of
the existing LiST model for DFID is the process by which this data will be collected. It is anticipated
that this LiST amendment to estimate the number of maternal and newborn lives saved will eventually
be accepted as the internationally agreed methodology to do this (a global public good as other donors
also face this issue around measurement of results – maternal and newborn lives saved) It is
anticipated that a measure of maternal and newborn lives saved will be included in the next DFID
Annual Report (2013).
4
5
DFID Business Plan 2012 – 15. DFID, May 2012
DFID’s Results Framework. Managing and reporting DFID results. DFID, March 2011.
3
Section A: Detailed Output Scoring
Output 1: Modified LiST statistical tool developed from original modelling tool
Output 1 score and performance description:
Modified LiST statistical tool developed: A
1. Output Indicator 1.1: Number of workshops conducted. Milestone 3, Achieved (3).
2. Output Indicator 1.2: Milestone: Statistical tool finalised. Achieved
Progress against expected results:
The statistical (LiST) tool was developed to meet DFID needs. 3 workshops were organized and they
were attended by more than 40 DFID advisors dealing with Health and Results. This was to allow the
development of the tool to take into account comments and concerns raised by DFID advisers with the
tools designers but also to obtain the required buy-in from DFID priority countries for the need of this
work. 14 DFID priority countries were represented at least in one of the three workshops in developing
the agreed statistical tool.
Recommendations:
The current version of the log frame does not contain a milestone for indicator 1.2 for next couple of
years. Revised log frame attached which includes milestones for forthcoming years.
Impact Weighting (%): 25%
Revised since last Annual Review? Y/N No
Risk: Low/Medium/High: Low
Revised since last Annual Review? Y/N No
Output 2: Standard template of maternal and newborn lives saved developed for DFID use:
Output 2 score and performance description: A
Standard template of maternal and newborn lives saved developed:
1. Output Indicator 2.1: Milestone: development and dissemination of template and guidance.
Achieved.
2. Output Indicator 2.2: Number of DFID Country Offices submitting completed input data
template.
Milestone 27 countries, but only 19 countries have programmes that have an impact on lives
saved and 19 is the absolute maximum that the programme could have achieved . 19 countries
have provided input data.
4
Progress against expected results:
On-track for year one milestone. The process need to be repeated every year. The system of data
collection is now in place.
Recommendations:
The current version of the log frame needs to be updated to include milestones for future years.
Revised log frame attached which includes milestones for forthcoming years.
Impact Weighting (%): 25%
Revised since last Annual Review? Y/N No
Risk: Low/Medium/High: Low
Revised since last Annual Review? Y/N No
Output 3: Results estimates of maternal and newborn lives saved generated
Output 3 score and performance description: A
Results estimates generated of maternal and newborn lives saved.
1. Output Indicator 3.1: Milestone: Report on number of maternal and newborn lives saved
with DFID support during FrR period. Achieved.
2. Final cumulative report on number of maternal and newborn lives saved with DFID support
during FfR period – Report to be completed by April 2016. Not completed yet.
Progress against expected results:
In March 2013, John Hopkins University (JHU) generated results for 19 countries. 15 countries
validated the results before March 2012 publication. During the summer 2013 an additional 3 countries
validated their results. We are currently improving the data collection of the input data for Pakistan.
JHU will repeat the analysis for the Pakistan results in October 2013. This delay was due to the inability
of Health Advisor in Pakistan to participate at the Nepal workshop. The results for Pakistan will be
updated in October 2013 and the programme is on track to deliver the final report by April 2016
Recommendations:
None
Impact Weighting (%): 25%
Revised since last Annual Review? Y/N No
Risk: Low/Medium/High: Low
Revised since last Annual Review? Y/N No
5
Output 4: Dissemination of modified LiSt and evidence as a global public good
Output 4 score and performance description: B
Dissemination of modified LiST and evidence as a global public good.
1. Indicator 4.1: Milestone: Number (2) of peer reviewed publications or presentations at
regional/global events on the use of modified LiST. Achieved 1. A paper on the LIST
methodology was submitted and presented at the International Family Planning Conference
in Ethiopia (12-15 November, 2013).
Progress against expected results:
JHU/Health Advisor and Senior Research Fellow are currently drafting a scientific publication on the
development and implementation of the LiST tool (amended to undertake this task). The global push to
demonstrate results by donors means that the LiST tool as developed for DFID use, if accepted
globally, will be a global public good. Although DFID has paid for the adaption for its own use the
intention is to make this available generally.
Recommendations:
We recommend adding this activity as a milestone for next year. By 2014, JHU should submit an
annual report and a methodological note for internal and external circulation and should be written to a
publishable standard (Output Indicator 4.2). The log-frame was updated and two additional indicators
were incorporated in Output 4 (see attached revised log-frame).
Impact Weighting (%): 25%
Revised since last Annual Review? Y/N No
Risk: Low/Medium/High: Moderate
Revised since last Annual Review? Y/N No
Section B: Results and Value for Money.
1. Progress and results
Value for money through procurement
DFID needed to engage with a partner familiar with the overall LiSt methodology/computer modelling
and who was able to adapt it to our requirements. John Hopkins had developed original LiST tool in
collaboration with Gates Foundation. We also required a partner organisation with the ability to
demonstrate and to generate the required lives saved estimates with the independence and authority
to disseminate the modified LiSt model and evidence as a global public good. It was thought that this
work could be procured at lower cost via direct engagement with JHU the original LiST developers and
who had been closely working with DFID on the concept of a DFID tailored model, rather than open
competition. JHU’s technical and commercial capabilities were assessed to ensure that it could offer
sustainable quality representing value for money throughout the life of the project and a reduction was
obtained in the original estimate of £30,000 after negotiation.
1.1. Has the logframe been updated since last review? Y/N
No – However it is recognised that this is now required. Revised logframe attached.
1.2 Overall Output Score and Description:
A
1.3 Direct feedback from beneficiaries
6
In August 2013, the Health Advisor in ARH team organized a series of VC meetings to gather views
and concern from country advisors involved in the LiST estimation process.
The advisors were generally happy with the process of estimation and results BUT they express
concern that LiST estimation might underestimate the true impact of DFID programmes (Yemen,
Zambia, Ethiopia, etc). They would like to have more information about risks associated with estimation
process in the report. There were some specific challenges to model important RMHN programmes –
i.e. Tanzania (maternal health commodities). In August 2013, DFID health adviser (AB) organised a
series of VCs with all country offices to discuss the risks associated with the LIST estimation and to
gather views/concerns with the country result. In September 2013, the aggregate LIST results were
presented at the Health Advisers meeting in WH.
1.4 Summary of overall progress
Overall progress is going according plan. Data collection continues apace with outstanding countries
such as Pakistan completing their inputs. Another LiST sensitisation workshop is being planned for the
end of this year (2013) to ensure that DFID advisers/country officers have support and continue to
provide the data required. Another will be considered in 2014 if needed.
1.5 Key challenges;
o Making sure that DFID advisors/country offices continue to provide input data to the Lives
Saved work;
o JHU does not/cannot generate results as scheduled. Insufficient information is available for
annual reports;
o The results generated are not credible with the global community and are not considered a
“global public good”:
o Update Section 4.2 - reviewing actions taken to minimise these risks.
How we could improve this estimation for next year?




Need to ensure that new programmes are included as this will contribute to results;
Strive for better data by improving the format of the statistical template;
Continue to run workshops and to offer country advisers support;.
Assess how best to model programme that have not been modelled to make data collection easier.
1.6 Annual Outcome Assessment
The estimation of maternal and neonatal lives saved from LiST for DFID funded investments was
delivered on time and according to planned timescale and budget by John Hopkins. All 19 countries
identified as requiring to provide input data have now done so. The overall programme scores A.
7
2. Costs and timescale
2.1 Is the project on-track against financial forecasts: Y/N
No – the project is actually slightly behind its financial forecasts but has delivered the required results.
2.2 Key cost drivers
Key cost drivers are JHU staff inputs and international travel costs. For DFID the key cost driver is core
HDD advisory time and the availability of DFID country advisers to complete the data collection
required.
2.3 Is the project on-track against original timescale: Y/N
Yes
3. Evidence and Evaluation
3.1 Assess any changes in evidence and implications for the project
No
3.2 Where an evaluation is planned what progress has been made?
The mid-term review of the RMNH Framework for Results (FfR) is currently underway and it will
provide opportunity for independent external review of the approach taken with regard to data
collection and LiST, with recommendations made to DFID as necessary.
4. Risk
4.1 Output Risk Rating: Low/Medium/High
The overall risk is low.
4.2 Assessment of the risk level
The main risks and actions undertaken to mitigate the risks are illustrated in the table below:
Table: Risks and mitigation strategies
Risk
Failure to determine input
data and to modify LiST
model to generate lives saved
estimates
Probability
Low
Impact
High
Risk Mitigation Strategy
Three workshops were held with
DFID staff to develop and refine the
statistical template for LiST tool.
Additional workshops to be
conducted.
8
The DFID Maternal and Newborn
Senior Research Fellows peer
reviewed the model and
methodology during development
and prior to implementation.
DFID country offices do not
provide input data to JHU for
the generation of annual
estimates
Low
High
DFID staff were involved in the
development of the tool and
awareness was raised through
health advisor and statistical
advisor cadre meetings.
To assure annual data collection –
additional workshops to be
conducted and awareness continue
to be raised via health adviser and
statistical adviser cadre meetings.
Standard methodology was
developed, including a table for
input data and guidance will be
prepared on how to complete the
table. This continues to be refined
JHU does not generate results Low
as scheduled. Insufficient
information is available for
annual reports
Results generated are not
Low
credible with the global
community
Moderate
Low
JHU provided on-going support to
country offices to ensure that input
data is complete and accurate. To
be publicised again.
JHU generated the results in time
for the annual report.
JHU/Health Advisor and Senior
Research Fellow are currently
drafting a scientific publication on
the development and
implementation of the tool.
A paper on the LIST methodology
was submitted and accepted for
oral presentation at the
International FP Conference in
Ethiopia (12-15 November).
The mid- term review of the FfR is
underway and it will provide
opportunity for independent
external review of the approach
taken with recommendations made
to DFID as necessary
4.3 Risk of funds not being used as intended
The risk is considered low. Funds are only directed to JHU, an international organisation of repute,
against invoices received at quarterly intervals and according to the workplan. .
4.4 Climate and Environment Risk
N/A
9
5. Value for Money
5.1 Performance on VfM measures
Early negotiation on the proposal enabled a £30,000 saving to be made. Proactive project
management and transparency about DFID approach to consultancy fee rates, travel costs and
incidentals has raised awareness of the need to demonstrate VfM and to reduce unnecessary costs.
Early planning has also allowed the required workshops to be held at minimal expense to both the
project and DFID, by tagging them on annual adviser retreats. The required data collection has also
been made as simple as possible. This has been achieved whilst maintaining progress and standards.
5.2 Commercial Improvement and Value for Money
Much of the scope for commercial improvement and Vfm has already been pursued during initial
contractual negotiations. However the programme team will continue to be alert to future opportunities,
particularly during forthcoming workshop (ie: using DFID premises and low cost accommodation)
5.3 Role of project partners
John Hopkins University is the implementing partner for this project. They are aware of the need to
demonstrate VfM and to reduce unnecessary costs particularly around air travel and accommodation
costs for planned workshops.
5.4 Does the project still represent Value for Money : Y/N
Yes
5.5 If not, what action will you take?
None proposed at this time
6. Conditionality
6.1 Update on specific conditions
No conditionality
7. Conclusions and actions



The logframe has been modified and improved as part of this review to more accurately reflect
the work undertaken by JHU and the results to be expected. The risks section has been
updated to take into account comments made from the HDD programme Review Group;
Another LiST workshop is planned for late 2013 and another in 2014 to continue support to
country offices in completing data input requirements which are also being simplified where
possible. DFID project management team will support JHU with the logistics and JHU will
deliver the workshop;
Following the findings of the mid- term review of the FfR (an opportunity for independent
10
external review of the LiST approach with recommendations), will be an opportunity to consider
further our strategy in promoting the acceptance of the LiST model as a global good. Results of
the MTR are expected to receive from the independent external review team in early December
2013.
8. Review Process
This Annual Review was conducted by Angela Baschieri (Health Adviser) and Scott Hardie (Policy
Officer for Maternal Health issues) and includes comments/suggestions from the HDD Programme
Review Group.
11