Pilot Project of Results Based Aid (RBA) in the Education
Sector in Ethiopia
Business Case
October 2011
CONTENTS
Intervention summary………………………………………………………3
Strategic Case………………………………………………………………5
Appraisal Case…………………………………………………………….13
Commercial Case………………………………………………………….30
Financial Case …………………………………………………………….32
Management Case ………………………………………………………. 35
Annexes
Annex A: Logframe
Annex B: Independent verification and evaluation Business Case
Annex C: RBA model
Annex D: RBA Economic Appraisal Spreadsheet
Acronyms
BC
BoFED
CGD
CoDA
CPAR
CSC
CTE
DRS
EC
EMIS
EPRDF
ESDP
ETP
FPPA
FRA
GDP
GEQIP
GER
GOE
GoE
GPOBA
GTP
HRITF
MDG
MIS
MoE
MoFED
MoU
NER
NLA
OFAG
PBS
PDP
PEFA
PFM
PISA
PTA
QESSP
RBA
RBF
REB
TOC
TVET
VfM
WOFED
Woreda
Business Case
Bureau of Finance and Economic Development
Centre for Global Development
Cash on Delivery Aid
Country Procurement Assessment Review
Critical Success Criteria
College of Teachers Education
Developing Regional States
European Commission
Education Management Information System
Ethiopian People’s Revolutionary Democratic Front
Education Sector Development Plan
Education and Training Policy
Federal Public procurement Agency
Fiduciary Risk Assessment
Gross Domestic Product
General Education Quality Improvement Program
Gross Enrolment Ratio
Government of Ethiopia
Government of Ethiopia
Global Partnership for Output Based Aid
Growth and Transformation Plan
Health Results Innovation Trust Fund
Millennium Development Goal
Management Information System
Ministry of Education
Ministry of Finance and Economic Development
Memorandum of Understanding
Net Enrolment Ratio
National learning Assessment
Office of Federal Auditor General
Protection of Basic Services
Peace and Development Program
Public Expenditure and Financial Accountability
Public Financial Management
Program for International Student Assessment
Parent Teacher Association
Quality Education Strategic Support Programme
Results Based Aid
Results Based Financing
Regional Education Bureau
Theory of Change
Technical, Vocational Education and Training
Value for Money
Woreda Office of Finance and Economic development
An administration division of Ethiopia managed by local
Government/ equivalent to District
Intervention Summary
Project title: Pilot Project of Results Based Aid (RBA) in the Education Sector in Ethiopia
What support will the UK provide?
The UK will provide up to £30 million over four years 2011/12 – 2014/15 depending on results
achieved and a further £1.5 million to cover the costs of evaluation and independent results
verification.
Why is UK support required?
The DFID Business Case includes a commitment to pilot RBA in at least three developing
countries. This is one of those pilots. In this pilot, we are proposing to pay incentives to the
Ministry of Education (MoE) in order to increase the number of students, especially girls and
students in Developing Regional States (DRS)1, who sit and pass the national grade 10
examinations (the equivalent of lower secondary school). It is hypothesised that the additional
results -based financing will act as an incentive to government to direct its efforts and
resources on achieving additional results in a cost effective and efficient way (see Theory of
Change).
The pilot project will be implemented by MoE and will potentially use structures at federal,
regional, woreda (district) and school levels. Funding through this project will be additional to
DFID’s scaled up support to the education sector in Ethiopia through existing instruments - the
General Education Quality Improvement Programme (GEQIP) and the Protection of Basic
Services Grant (PBS) - but will complement these programmes. The Ministry of Education has
expressed a desire to test RBA approaches at the highest levels, and results in the education
sector lend themselves well to this approach. DFID is at the forefront of efforts to improve aid
effectiveness in Ethiopia and is well-placed to drive this pilot forward. There are high levels of
support from both the UK and Ethiopian governments to trying out this approach in the
education sector.
There are a range of RBA models, but the approach taken in the pilot builds on the work of the
Centre for Global Development (CGD) and shares many features of their Cash on Delivery Aid
(CoDA) modeli. A price per student who sits/passes the grade 10 examination over an agreed
baseline will be established – with a premium paid for girls and students in the less stable
peripheral regions - and payment made to the MoE once the results have been independently
verified. The Ministry will be free to use this additional money in any way it sees fit to achieve
the additional results. IN practise, the programmatic and policy response will be carefully
monitored through an independent evaluation process and by a DFID adviser seconded to
MoE.
The project will focus on improving access to, and performance in, the grade 10 examinations,
especially for girls and students in the DRS. This is because, despite impressive progress in
expanding access to primary education over the past 10 years, access to lower secondary
education remains extremely limited. There is robust international evidence to support the
case for investing in girls’ secondary education because of its social and economic benefits,
including lower fertility rates, delayed marriage and increased earning potentialii. This is why an
1
Developing Regional States also called the emerging regions are the four most underdeveloped regions of
Ethiopia – Somali, Afar, Gambella, and Benishangul Gomuz
additional premium will be paid for every girl who sits and passes the examination. An
additional incentive will also be paid to MoE for progress in the four most under-developed
regions in order to accelerate progress in these regions.
In Ethiopia, there are no other donors working on an RBA2 approach, but there have been
discussions with donors about their possible involvement in the future. The US and German
governments have both expressed an interest in the pilot. The World Bank, who manage our
multi-donor instruments, are interested in learning lessons from the pilot for some of the
existing instruments, including GEQIP. The GoE itself has been exploring how to increase the
results focus of its block grant financing to regions.
What are the expected results?
The overall impact of the project will be improved access to, and quality of, lower secondary
schooling. This will be measured by the grade 9 and 10 gross enrolment rate and the
percentage of girls and boys passing the grade 10 examination (see logframe at Annex A).
The outcome will be an increase in grade 10 students sitting and passing the grade 10
examination, especially for girls and in emerging regions. The modelling predicts the following
additional results incentivised by RBA over and above a rising baseline:
o
o
o
o
129,000 more girls and 55,000 more boys sitting the grade 10 examination in non DRS
100,000 more girls and 70,000 more boys passing the grade 10 examination in non DRS
3,500 more girls and 3,200 more boys sitting the examination in DRS
2,600 more girls and 4,500 more boys passing the examination in DRS
The outputs are derived from the theory of change:
o Government responds to incentives with improved policies and programmes that lead to
increased enrolment and retention of students in lower secondary school – percentage of
regional governments responding to RBA incentives with new policies/programmes
o
Incentives lead to more targeted and efficient use of existing resources- proportion of
sector financing allocated to secondary schooling annually
o
Stronger aid relationship between donor and Governments –– percentage of stakeholders
in Ethiopia who perceive RBA to have less conditions than other forms of aid in the sector
An independent evaluation and a verification of results will be critical parts of this programme.
The purpose of this evaluation will be to assess whether RBA is an effective use of foreign aid
to achieve development goals. Answering this question will require evaluation of the
relationship between the RBA incentive, government and donor behaviour, and how these
impact on results (see Theory of Change below) The purpose of the verification is to ensure
that the results are actually achieved and that any manipulation of results is exposed. Given
that this is a pilot project and the importance of the evaluation and the independent
verification means that significant financing has been allocated to them in the BC.
A separate Business Case has been developed for the RBA verification and evaluation in order
to allow enough time to internationally tender and contract firms, and start data collection
before the start of the next Ethiopian academic year in September 2011.
2
Unless otherwise specified, the term RBA will be used to refer to the CoDA type approach
1. Strategic Case
A. Context and need for DFID intervention
A1. The Ethiopian Contextiii
Ethiopia matters to the UK for a range of development, foreign policy and security priorities. It
is populous, poor, vulnerable but comparatively stable in the Horn of Africa. From a low base,
Ethiopia’s growth and expansion of basic services in recent years have been among the most
impressive in Africa. The UK Government has an opportunity to make our support more
transformational and accelerate Ethiopia’s graduation from aid dependency. The Government of
Ethiopia (GoE) is capable and committed to growth and development, and is a proven partner in
making rapid progress towards the Millennium Development Goals (MDGs). But its approach to
political governance presents both substantive challenges to sustainable development and
reputational risks to partners.
Ethiopia lies at the heart of an unstable region that has experienced almost continuous conflict
and environmental shocks in recent decades. Ethiopia and its neighbours – including Somalia,
Sudan and Eritrea – languish at the bottom of the Human Development Index. Poverty and
instability in the Horn of Africa are among the drivers of migration to Europe, and also
contribute to an environment in which fundamentalism and radicalisation can prosper. UK
interests in the region include progress towards the MDGs, resolving conflict, bolstering
stability, accelerating sustainable growth and development, mitigating the impact of climate
change, tackling migration, and countering terrorism. A stable, secure and prosperous Ethiopia
is critical to UK interests.
Ethiopia has come a long way in a short time, and has achieved stability through decentralised
regional government. But Ethiopia has yet to successfully manage its democratic transition. The
Ethiopian People’s Revolutionary Democratic Front (EPRDF) took power in 1991 and have held it
since then. They have made progress towards a functioning democracy and respect for human
rights, but there is still a long way to go. Ethiopia has a capable government that is
demonstrably committed to addressing poverty, with an impressive record of pro-poor
spending, sound financial management and relatively little corruption. Prime Minister Meles
Zenawi and others in GoE play a role on global issues, including climate change, reform of the
international financial architecture, and global health.
Ethiopia has made impressive progress towards the MDGs. In the last five years, with
substantial support from the UK and others, Ethiopia has: halved the incidence of malaria;
deployed 32,000 more health extension workers; doubled the immunisation rateiv; rolled out an
innovative social safety net to protect almost 8 million of the most vulnerable peoplev; and put 4
million more children in primary schoolvi. Strong macroeconomic leadership has helped Ethiopia
achieve annual growth of over 7 per cent for the last decade. GoE’s new Growth and
Transformation Plan (GTP) targets a doubling of the economy and achievement of the MDGs by
2015, and a greater (if still limited) role for the private sector and accelerated industrialisation.
The GTP provides a platform to align UK support with GoE’s ambitions, make it more
transformational, and accelerate Ethiopia’s graduation from aid dependency.
Despite recent progress, Ethiopia remains one of the world’s poorest countries, with more than
30 million people living in extreme poverty. It is comparatively under-aided, receiving less than
the African average per capita aid (still ranking in the bottom quartile). Aid per capita levels are
the fifth smallest among the 17 current DFID priority countries in Africa.vii. Strong progress
towards some of the MDGs is from a very low base, and will be difficult to maintain as the needs
of harder to reach populations are prioritised. Population momentum will see the current
estimated population of 83 million people rise to around 120 million by 2030viii, which is likely to
be accompanied by rapid urban growth. Ethnic nationalism and underdevelopment fuel
instability and insurgency in parts of the Ethiopian periphery, threatening the delivery of
Ethiopia’s development objectives. External shocks, including climate change and fluctuating
commodity prices, threaten growth. Ethiopia was badly affected by the oil and food price shocks
in 2008, and will find it difficult to avoid price rises as global prices rise again in early 2011.
Ethiopia can absorb more aid and use it well. DFID is a leader within the development
community, championing results, aid effectiveness and transparency. DFID works closely with
UK Government partners in pursuit of shared objectives for a stable, secure and prosperous
Ethiopia.
A2. Education and RBA context
The GoE prepared the National Education and Training Policy (ETP) in 1994, and within this
framework of the ETP launched the first five year Education Sector Development Program (ESDP
I) in 1997 as part of a twenty-year education sector plan. As a result of a series of important
organisational, financial and programmatic measures, the target set for ESDP I of raising primary
enrollment from 3.7 million to 7 million was surpassed with enrollment reaching 8.1 million in
2000/01.
By 2009/10 primary enrolment had reached 15.7 million (82% Net Enrolment Rate - NER). The
number of out of school children dropped from 6.5 million to around 3 million between 1999
and 2008ix. Enrolment of girls has shown a steady increase with the gender parity index (girl to
boy ratio) now standing at 0.93 in primary. First cycle secondary enrollment trends show
significant increases (Gross Enrolment Ratio – GER - from 17.1% in 2001/02 to 39% overall and
35% for girls in 2009/10) and although second cycle secondary enrolment is low (GER of 7%
overall and 5% for girls in 2009/10), it is increasing. These achievements have been remarkable
and have occurred at the same time as a major expansion of both the Technical and Vocational
Education and Training (TVET) and higher education sub-sectors.
As part of the UK’s broader programme of transformative support for Ethiopia, DFID Ethiopia
has agreed with MoE to pilot a Results Based Aid project. The pilot will support the GoE’s
ambitious planx to increase lower secondary enrolment from 39% to 62% from 2011 – 2015 and
put an additional 700,000 boys and 800,000 girls in lower secondary school by 2015. The targets
are ambitious since lower secondary access has to date not kept pace with primary enrolment
as the most recent statistics from GoE showxi:


In 2009/10, the national net enrolment ratio for lower secondary school was only 16% as
compared to 82% for primary.
Few of those who enter lower secondary school pass the grade 10 examinations – in 2008/9,


849,432 students entered grade 9, but only 525,908 students took the grade 10 examination
in 2009/10 and only 327,501 students passed (about 60% of grade 9 enrolment).
Girls are less likely to enter secondary school, are less likely to take the grade 10
examination and are less likely to pass than boys – in 2008/9, girls comprised 43% of
students entering grade 9, in 2009/10 they comprised 44% of students taking the grade 10
examination but only comprised 37% of students passing the grade 10 examination
The situation is particularly serious in the Developing Regional States (DRS).which are less
developed than other regions in Ethiopia, have a high proportion of the pastoralist
population, and in some cases are conflict prone. Although these four regions account for
10% of lower secondary age children, they account for less than 4% of the students passing
grade 10 examinations in 2008/9. Students in the DRS are also less likely to pass the grade
10 examination – in 2008/9, the national pass rate was 43% but it was only 34% in the four
emerging regions. Children are less likely to attend secondary school in DRS. In 2008/9, the
net enrolment ratio for lower secondary school in the four regions was only 2.7%.
There are a range of results-based approaches (see evidence section). In the Ethiopia context
DFID will pilot an approach where the partner government must arrange finance for the cost of
all up-front inputs. Donor funding is only provided once results have been achieved and verified.
(CGD’s Cash on Delivery Aid model is a good example of this approach). CGD’s Cash on Delivery
Aid (CoD-Aid) model features arexii:



CoD-Aid is a contract between the funding agency and the recipient government based on a
mutually desired outcome and a fixed payment for each unit of confirmed progress.
The purpose of CoD-Aid is to empower partner governments to achieve results however they
see fit, thereby strengthening a government’s accountability to citizens, rather than donors.
Donors also take a ‘hands-off’ approach, while partner countries decide on the path to
achieving the agreed results
Developing countries may need to finance the up-front investments as necessary. This is
because payment is linked to the independent verification of results after they have been
achieved.
One of the strengths of CoDA-type RBA schemes is the use of country systems to achieve and
gather data on results, thereby strengthening those systems for delivery of services and
measurement of results. The intention during the pilot phase is not to move all financing to a
results based approach, but to use the RBA financing as an additional and complementary
instrument to generate additional incentives, to accelerate results, and to attract additional
financing to Ethiopia through this modality. CGD visited Ethiopia in 2009 to discuss the
approachxiii and had dialogue at the highest levels. The Prime Minister himself was reported to
be interested, although GoE did not wish to pursue a pilot in only one or two regions as was
proposed.
The approach in the pilot builds on the work of the Centre for Global Development (CGD) and
shares many features of their Cash on Delivery Aid (CoDA) modelxiv. A price per student who
sits/passes the grade 10 examination over an agreed baseline will be established – with a
premium paid for girls and students in the less stable peripheral regions - and payment made to
the MoE once the results have been verified. Feedback received from the MoE during the design
phase indicates that they would allocate the additional funding to regions on the basis of results
and regions would allocate to districts and schools also based on results achieved. Additional
financing to schools would be additional but complementary to per capita grants provided
through GEQIP. DFID will carefully monitor the response both through the evaluation process
and through the secondment of a senior education adviser to the Ministry of Education.
A3. The policy context
Achieving the millennium development goal targets Given the size of the country and scale of
the problem, achieving the education related MDG targets in Ethiopia will make a significant
contribution to achieving the MDG targets in sub-Saharan Africa and globally. Ethiopia accounts
for about 3 million (of whom 55% are girls) of the estimated 32 million children out of school in
sub Saharan Africaxv.
DFID’s Business Plan commitments - In its Business Plan (2011-2015) DFID is committed to
conducting at least three RBA pilots (deliverable under value for money and transparency):
“Pilot Results-Based Aid and cash on delivery contracts in three developing countries by
November 2011”xvi. This intervention will inform, through lesson learning, DFID’s policy and
guidance on results based aid and the possible roll out of this approach to other programmes in
Ethiopia and elsewhere.
DFID-E Operation Plan - This intervention will contribute to the following DFID-E objectives in the
2010 – 2015 Ethiopia Operational Planxvii. This has been accompanied by a commitment to
champion results, transparency, independent scrutiny and a focus on girls. This pilot project will
contribute to all of these objectives as follows:
 Results – strong financial incentives for GoE to deliver and accurately measure results
because payments will only be made on the basis of verification and agreement of the
additional results actually achieved
 Transparency – the approach will establish clearly the basis upon which payments will be
made
 Independent scrutiny – at two levels: (i) independent verification of results, that will form
the basis for our release of funds; and (ii) an independent evaluation of the pilot
 Focus on girls – the premium for girls should create incentives for the government to put
in place policies and programmes to accelerate progress in girl’s learning outcomes, to
help unleash the transformational potential of girls.
National policy context The Government of Ethiopia has ambitious goals as set out in the Growth
and Transformation Plan (2010 – 2015)xviii. This document includes very stretching targets for
almost full primary enrolment (98% for both boys and girls) and an increase in lower secondary
education gross enrolment rate from 39% in 2009/10 to 62% by 2015. The ambition of the
government, the progress it has been able to make in the past and its focus on results provide a
strong foundation for DFID’s efforts to increase the focus on results through an RBA project.
Under ESDP IV the main goals of GoE for general education are to improve access to quality basic
education in order to make sure that all children, youngsters and adults, with particular emphasis
on females, acquire the competencies, skills, values and attitudes enabling them to participate
fully in the social, economic and political development of Ethiopia and to sustain equitable access
to quality secondary educationxix. GoE also recognises the challenges faced by girls, and students
in the DRS, and has laid out strategies in ESDP IV that seek to promote their participation and
performance.
A4. Other interventions in education in Ethiopia
In Ethiopia, there are no other examples of RBA of the type that will be tested by the pilot. The
RBA pilot is expected to generate additional results in secondary education, thus maximising the
impact of our other sector investments. Complementary education support from DFID is
summarised in the table below:
Table 2: DFID Ethiopia – planned support for education: 2011 – 2015
No
Programme/instrument
What it is supporting/contributing to
1
Protection of Basic
Services (PBS – 40%
spend on education)
Recurrent expenditure mainly salaries of
teachers (access)
2
General
Education
Quality Improvement
Programme
(GEQIP)
2009 – 2013
Quality inputs and non-salary recurrent
expenditure for general education (quality)
3
Basic services in Somali
Region (new – being
designed) 2011 – 2014
Improving access to quality education in
Somali region (equity in access)
e
Quality
Education
Strategic Sector Support
Programme (QESSPP)
2010 – 2012
Strategic support to MoE through a
combination of advisery secondment and
Technical Assistance
4
End Early
(inception)
provide school materials as an incentive to
keep girls at risk of early marriage in school
in Amhara Region (equity in access)
Marriage
A5. Feasibility of intervening
DFID is committed through the BAR and OP to scale up results in this area. It has experience in
this sector and the trust it has established with the Government of Ethiopia and other
stakeholders. The RBA model allows the focus of DFID’s intervention to be shifted to
achievement of priority results and it is anticipated that this will influence the way other donors
and the Government of Ethiopia operate. This approach is strongly-aligned with the UK
Government’s development aid agenda focused on results, value for money and accountability.
In addition, the Ethiopian context meets all of the criteria identified in DFID’s Results Based Aid
Primerxx.
1. There is interest from the partner government in piloting this type of RBA: RBA has
attracted high level interest from policy makers in Ethiopia, including the Prime Minister,
after a visit from CGD here in 2008. The Minister of Education has expressed a keen interest
in piloting the approach in the education sector.
2. There is a degree of confidence that the particular results are achievable: the track record
of Ethiopia in promoting improved education outcomes is impressive and it is therefore likely
that the approach can work. For example, in the last five years enrolment in primary
education has increased by over 2,000,000 and lower secondary by over 500,000xxi This is a
good platform for further accelerating progress at lower secondary education level through
the pilot.
3. There are robust data collection systems in place. Measurement of results is at the heart of
the RBA model and national systems are used for data collection. As such, DFID-E has
considered the following:
a. Do management information systems (MIS) exist and do they work properly? A
comprehensive abstract is produced annually based on routine administrative data.
This includes data on most key indicators of interest to stakeholders. The result
selected, the grade 10 examination, is set and marked federally and results are
published annually.
b. What is the quality of the data? (relevance, accuracy, timeliness, accessibility,
comparability and coherence) : The Ministry has been working hard to improve the
timeliness of the availability of EMIS data and the report was available within three
months of the end of the academic year last year. Data quality is used for reporting
purposes by development partners, although capacity building to improve accuracy
and timeliness is being carried out through other channels. The grade 10
examination data is robust with respect to sitters, although there are concerns about
cheating which may contaminate the results relating to passing. This will be
monitored closely through the independent verification process.
c. Is there potential for independent verification of the data? Agreement to the
verification process will be included in a Memorandum of Understanding with the
Ministry of Education. Terms of Reference for the independent verification have
been developed as part of the separate Business Case for evaluation and verification.
A6 Consequences of not intervening
Not intervening would significantly impact on DFID-E’ BAR offer results. Based on modelled
estimates of additonality attributable to the pilot (see model at Annex C), it could mean:
o
o
o
o
129,000 fewer girls and 55,000 fewer boys sitting the grade 10 examination in non DRS
100,000 fewer girls and 70,000 fewer boys passing the grade 10 examination in on DRS
3,500 fewer girls and 3,200 fewer boys sitting the examination in DRS
2,600 fewer girls and 4,500 fewer boys passing the examination in DRS
In addition, not intervening would leave a significant gap in our knowledge of how RBA
approaches impact on the aid relationships and results and lose the opportunity to substantially
impact on aid effectiveness in Ethiopia.
A7 Sustainability
Ethiopia spends a relatively high share of its Gross Domestic Product (GDP) on education by
international standards relative to it’s level of per-capita income of $344xxii it spends significantly
more than India, (whose per-capita income of $1192 is almost four times as much), and only a
little less than South Africa (whose per-capita income of $5786 is more than 16 times that of
Ethiopia).
Despite this, the overall funding gap in ESDP IV is estimated at £1.2 billion as indicated above.
Much of this funding gap is expected to be filled by external partners. With the GoE already
spending around 4% of GDP on education and over 20% of its budget, there is little likelihood
that the GoE will be able to mobilise additional domestic resources to tackle the challenge of
improving quality of education in the next five years. The RBA pilot is designed to demonstrate
to partners how a different approach to aid can deliver results effectively and efficiently,
thereby mobilising additional funding for the sector.
As discussed above, improving education quality and access, especially in secondary schools, will
contribute to both economic growth and improved social outcomes. Amongst countries that
have participated in the Programme for International Student Assessment (PISA - an
internationally benchmarked measure of learning) shows that test scores of one standard
deviation above the average are associated with an average annual growth rate in GDP per
capita of two percentage points higher over 40-yearsxxiii
B. Impact and Outcome
The overall impact of the project will be improved access to, and quality of, lower secondary
schooling.
The outcome will be an increase in grade 10 students sitting and passing the grade 10
examination over an adjusting baseline, especially for girls and in emerging regions.
The outputs are derived from the theory of change:
o
Government responds to incentives with policies and programmes that lead to increased
enrolment and retention of students in lower secondary school – % of regional governments
responding to RBA incentives with new policies/programmes
o
Incentives lead to more targeted and efficient use of existing resources- proportion of
sector financing allocated to secondary schooling annually
o
Stronger aid relationship between donor and Governments –– percentage of stakeholders
who perceive RBA to have less conditions than other forms of aid in the sector
B1. Political and Institutional impact
There is strong political commitment to scaling up access to and quality of services from the
GoE including education services. This will build on the GoE record of rapid scale up and delivery
of results in recent years. The GoE’s Growth and Transformation Plan (GTP) reflects the
ambitious targets of the education sector development programme to 2015 which focuses,
among other objectives, on the achievement of the MDGs. This is a politically high profile
intervention and has interest at the highest levels in both the UK and Ethiopia. For the UK, the
objectives are around transparency and results, while the government of Ethiopia see this as a
potential way to receive aid with fewer conditions. DFID will need to be clear that its policy
conditions continue to apply even when using this instrument. There are also reputational risks
to DFID if the approach does not yield predictable and timely aid flows. This is why the support
has to be additional to existing financing for the sector during the pilot phase.
It has been recognised that good governance requires state capability, responsiveness, and
accountability, and that all three elements are needed to make states more effective, to tackle
poverty and to improve people’s lives. There is no guarantee, for example, that a more capable
MoE will automatically focus on reaching marginalised communities unless it is responsive and
accountable. The MoE continues to explore alternative modes of education delivery
considering the challenges and living styles of populations. The RBA pilot is an innovative
approach to delivering aid against the achievement of result in the education sector.
Supporting the GoE to expand access to and quality of services and make better development
progress will inherently be strengthening the GoE’s legitimacy. In Ethiopia, given the
overwhelming majority of seats won and held by the ruling political party in the 2010 election, it
may be considered hard to clearly distinguish between supporting the GoE and supporting the
ruling party. In line with DFID’s approach to state building for security and stability, DFID
considers that providing support for an increasingly transformational programme in Ethiopia is
the best way to help transform Ethiopia into an even more prosperous, stable and secure
country. DFID has judged that a productive approach to working in Ethiopia is to combine work
to improve transparency and accountability, and create a healthier, more educated population
with growing expectations of economic opportunity and political space. Enhancing the quality of
education will contribute to the creation of well educated citizens and communities able to hold
the government to account.
The pilot is intended to have far reaching institutional impacts, both with respect to the aid
relationship and in terms of the way in which institutions at different levels respond to the
incentives. It has also been claimed that RBA approaches can strengthen the social contract
between the citizen and the state. The pilot is also expected to impact on other donors’
institutional approach to the delivery of aid. These impacts will also be carefully monitored
through the evaluation process. A steering committee will be set up, including participation from
GoE and donors, to provide oversight to the evaluation and help to create buy in.
B2. Social impact
Addressing barriers to educating girls and students in the Developing Regional States (DRS) were
recently indentified as priorities in in the GEQIP social assessment. xxiv The project has been
designed to maximise its social impact by providing additional incentives for girls and for
students in the four most under-served regions. The focus on lower secondary and girls is based
on evidence that investing in secondary education for girls has powerful effects on fertility, age
of marriage, and earning potentialii. There are risks that rewarding grade 10 performance could
divert resources away from primary level and may not be pro-poor as students who attend
secondary tend to be relatively well off. A key assumption is that GoE commitment to pro-poor
and equitable provision of education services will not be undermined by the pilot. For example,
in the DRS a focus on lower secondary may draw resources away from remote pastoralist
communities. The social impacts of the project, including financial flows, will be monitored
through the independent evaluation.
2. Appraisal Case
A. Determining Critical Success Criteria (CSC)
The success criteria for the pilot have been derived from the theory of change in the first
instance. Three additional criteria have also been included which relate to equity, accountability,
and broader buy in from stakeholders.
Table 1: Success criteria
CSC
Description
Weighting (1-5)
There is a demonstrable impact on results, especially
1
5
for girls and in emerging regions
The pilot generates incentives to implement policies
and programmes that increase grade 10 participation
2
5
and performance, especially for girls and students in
DRS
The additional financing is not earmarked to specific
3
4
policies and programmes
The pilot increases the efficiency of funding to the
4
4
sector
The pilot incentivises equity in the distribution of
5
3
resources and incentives
Civil society bodies, such as parent-teacher
6
associations, are more involved in financial decision3
making and data verification
Other stakeholders buy in to the results based aid
7
2
approach
B. Feasible options
The following two options, that are variants of the CoD-Aid approach, are identified for appraisal.
Option A - CoDA tied to the education sector
Option B - CoDA tied to individual schools
While the Centre for Global Development’s (CGD’s)xxv proposed CoD-Aid approach (described in
detail below) has provided a basis for dialogue with the Ministry of Education with respect to
approach, coverage and pricing, it is not included as one of the options for appraisal. This is
because it has been agreed that a) the incentive will be paid for both passing and sitting (see
below) and b) additional payments will be channelled through MoE rather than through MofED
as this was judged to be more likely to generate incentives for action in the education sector.
The concept behind results-based aid is that the prospect of increased financial resources for a
particular result will cause the recipient, in this case the Ministry of Education, to make to use
existing resources in a more focused way to achieve this result. In this case, it is possible that
additional resources and/or change in use of existing resources could occur at a number of levels
including schools, the regions and the federal government, depending on the actions of the
Ministry. The approach is also intended to reduce the reporting and policy conditions that are
frequently applied in existing aid instruments. The theory of change is presented
diagrammatically below.
Key assumptions underpinning the theory of change are:
a) That a relatively small amount of additional financing will incentivise the Ministry of Education
to improve policies and programmes and use finance better
b) Stakeholders regard RBA as an effective aid instrument
c) Outcomes can be verified in order to trigger payment
d) Sufficient results can be achieved to generate payment that will incentivise government
Figure 2: Proposed theory of change
RBA incentivises govt to
Improve policies and
programmes and
use finance better
Outcomes can be
verified
in order to
trigger payment
Government responds to incentives
with improved
policies and programmes
RBA payment
based on results
Stronger aid relationship between
donor and Government
Improved
learning
outcomes and
staying on rates
for boys and
girls
More targeted use of existing
resources
Sufficient results can
be achieved to
generate payment that
will Incentivise govt
Stakeholders regard
RBA as an
effective aid instrument
Additional financing
Evidence for impact of Results Based Aid
Results-based approaches can broadly be categorised as Results Based Aid (RBA) or Results
Based Financing (RBF), although hybrids also existxxvi.
RBA is an aid partnership between a donor and a partner government. It departs from inputbased approaches where funds are made available for inputs such as development of policies,
procurement of services, work and supplies, and payments for recurrent expenditure such as
salaries. Instead, RBA introduces a new concept of conditionality whereby disbursement is tied
to results, but where the donor otherwise takes a more ‘hands-off’ approach.
Examples of variants of RBA: Global Fund for HIV/AIDS, TB and Malaria, Gavi Immunization
Services Support, EC MDG Contract (a type of general Budget Support implemented by the EC),
and the so far untested CoDA approach.
RBF is an approach to contracting a service provider or incentivising a beneficiary of services.
Unlike RBA, it is not an aid relationship with the partner government. Voucher schemes, cash
transfers and output-based contracts with service providers are examples of RBF schemes that
DFID already has experience with. RBF schemes are funded from either domestic government
resources or aid (or both), with payments made to beneficiaries or service providers, usually
through a third party. RBF schemes that make payments to beneficiaries are aimed at increasing
access to services, while those that make payments to service providers such as teachers and
health workers aim improve the provision of services. Some schemes provide funding to both
beneficiaries and service providers.
Examples of RBF: Global Partnership for Output Based Aid (GPOBA), Health results Innovation
Trust Fund (HRITF), vouchers, and cash transfers.
There is evidence to show, especially in the health sector, that RBA/RBF approaches can deliver
resultsxxvii. For example, a recent World Bank literature reviewxxviii found that results-based
financing mechanisms appear to increase utilisation of priority maternal and child health
services, and that conditional cash transfers have been shown to have positive effects on child
health outcomes.
However, the COD-Aid model is untested, especially with respect to the hypothesised causal
chain between RBA incentives, improved aid relationships, more effective programming and
improved results (see Theory of Change below). Experience with results-based aid schemes in the
education sector is very limited, particularly when compared to the health sector. However, the
need for such approaches has been recognisedxxix. In the absence of a robust evidence base, the
DFID RBA Primer was used to identify a number of key design issues and these have been taken
into account in the options for appraisal below. In brief the key design issues as they relate to
this pilot are:

Results: (i) choosing results that have broad-based country ownership – GoE places a very
high priority on expanding quality secondary education. (ii) outcome vs. output: the level of
results to be targeted is important. The intention is to choose results that are linked as
closely as possible to the desired outcomes and that can be measured without incurring in
significant disbursement delays. Grade 10 sitters and/or passers are both closely associated
with the desired outcome and (iii) additionality is important: the results chosen should be
able to be identified as additional to results that would have been achieved with any other
government or donor funding. This is a key issue for the design of the evaluation of the pilot.

Unit of reward: It is important to negotiate a ‘unit price’ that represents value for money for
both DFID and the partner government/service provider, reflects the degree of difficulty of
achieving the results, and addresses incentives. The unit of reward should not be too high to
negatively affect value for money and not too low or the incentive might be reduced to zero.
The price will be subject to negotiation with government and based on the average cost of
educating a secondary student for one year.

Measuring and verifying results: there is a need to identify an acceptable and feasible
process for verifying results that addresses weaknesses in statistical systems and negative
incentives e.g. for data fraud – independent evaluation of results has been agreed in the
design phase.

Conditionality: RBA proposes a different approach to conditionality: no policy conditions are
being attached to the financing.

Evaluability: Building in evaluation from the outset, to inform DFID’s longer-term policy on
results-based approaches.

Avoiding negative unintended consequences:
distortion and “cherry picking” : we need to be aware that strong incentives to achieve
results in one programme might distort the allocation of resources away from other
programmes that might be equally important. This can be a particular problem if the
neglected activities are around systems strengthening and quality of services.
“gaming”: data or reporting fraud to receive higher rewards/payments is a potentially
major problem especially if the financial pay off is large enough. Rigorous independent
verification of the results is planned that should provide valuable insights into these
possible unintended consequence.
Based on these principles, two options were actively developed and considered in order to
deliver on this theory of change. A ‘do nothing option’ has not been formally appraised because
of the high level appetite for this pilot. In both the options considered, the result selected is the
grade 10 examination. This is because this result is judged to be the most robust and accurate
with respect to its measurement3. The pilot will also explore whether other results (such as the
grade 8 examination results) could be introduced subsequently.
For ease of comparison, the key features of CGD’s Cod-Aid approach and the two main options
are briefly described and compared below:
The key features of CGD’s COD-Aid approach would be (from www.cgdev.org):






DFID would pay based on a unit of reward for each assessed completer (sitter).
Unit of reward would be set at £123 (or $200) per student sitting the Grade 10 exam and
there would be no ceiling on the payment.
GoE has full responsibility for and discretion in using funds – funds are totally unearmarked and go to Treasury/Ministry of Finance. This is based on the concept that
because payments would only be released once results are achieved, it does not really
matter how the GoE would spend the funds ex-post. Under this approach, it is assumed
that the results will be sufficient to incentivise government to focus finance on the
priority sector.
One national baseline above which payments will be calculated:
The outcome measure is verified by an independent agent. The independent verification
would be contracted out to a third party.
The contract, outcomes and other information must be disseminated publicly to assure
transparency.
Option A (CoDA tied to the education sector): this is similar to CoDA with respect to the last two
bullets above. It has the following key differences:
 Payment based on a unit of reward for each girl and boy that sits and passes the Grade
10 exam (pass rates are published annually in the Annual Abstract).
3
The concern with the grade 10 examination is that the additional incentive for passing the examination (which is
relatively high stakes) could increase the amount of cheating, which is already a problem. The MoE are planning
to tackle this problem by strengthening invigilation. The issue will be closely monitored through the verification
and evaluation processes. This would not affect the sitters.




The unit of reward includes incentives for girls and students in emerging regions as
follows: £50 for boys and £85 for girls in the major regions and £75 for boys and £100 for
girls in the emerging regions.
Funds when released are given to Ministry of Education rather than paid into the general
budget and therefore they are notionally earmarked for the sector – MoE has full
discretion over how it uses the funds;
It has 4 baselines above which additional results will be calculated for payment: girls in
emerging regions; boys in emerging regions; girls in non-emerging regions; and boys in
non-emerging regions.
There is a financial ceiling of £10 million per annum to protect DFID-E from unlimited
liability
Option B (CoDA tied to individual schools): this is the same as option B in all respects except
bullet three:
 Rewards are paid to each individual lower secondary school rather than to the federal
ministry; so the more results a school achieves the more it gets in funds;
 A baseline and results would be developed for each school providing lower secondary
education.
Environmental and climate change effects
We anticipate that the additional financing will mainly be spent on soft inputs at Woreda and
school level, much of it in the form of enhanced school grants. As a result, environmental impact
will be minimal. Any infrastructure development would follow GoE environmental guidelines
which are set out in standards documents that have been agreed with regional governments. If
there is significant capital investment, then this will be monitored through the independent
evaluation and the quality of the construction, including its potential environmental impact, will
be monitored through this process. Any deficiencies identified in the pilot phase will be used to
(1) inform any future scale-up of the intervention (2) inform efforts to strengthen environmental
assessment in infrastructure developments (e.g. through the Local Investment Grant
programme).
In the table below:
 The quality of evidence for each option is rated as either Strong, Medium or Limited
 The likely impact on climate change and environment is categorised as A, high potential
risk/opportunity; B, medium/manageable potential risk/opportunity; C, low/no
rjsk/opportunity; or D, core contribution to a multilateral organisation
Option
Evidence rating (A, B, C, D)
A
B
C
Limited
Limited
Limited
Climate change and
category (A, B, C, D)
C
C
C
environment
C. Appraisal of options
The following appraisal examines each of the options against the critical success criteria
presented in table 1 above.
1. Demonstrating results
Which result to focus on?
The CoDA-Aid approach proposes that the most robust indicator is sitting (i.e. paying for each
student who sits the exam, irrespective of their performance). The advantage of focusing only on
sitters is that it is relatively easy to verify and is not as prone to dispute as passing. The
disadvantage is that it may rapidly increase participation, but with no corresponding focus on
performance. While acknowledging the methodological problems posed with rewarding passing,
MoE was clear that the pilot should pay not only for the number of additional students who
enrolled or the examination, but also for those who actually gained a pass mark.
Therefore both options A and B propose to reward improvements in sitting and passing. The
government is paid an incentive for every additional student who sits the examination. The
advantage of including an indicator for both passing and sitting is that this meets GoE
expectations for a focus on quality, minimises the risk that incentives will be generated to only
focus on students who are likely to pass the examination, and also reduces the risk that
participation will be expanded at the expense of quality.
Rewarding the number of students who pass the examination rather than the number who sit is
in line with the GoE focus on quality improvement. However, at least three problems were
identified with a focus on paying for passers only. The first is that it may incentivise the system to
focus only on those students who are likely to pass at the expense of low achievers. Secondly, it
may encourage cheating at decentralised levels. Thirdly, in the case of Ethiopia the pass rate is
partially norm referenced and has been subject to considerable variation over the past five years,
the reasons for which are not well understood (see table 2 below).
Table 2: Grade 10 examination results (GoE Annual Statistics Abstract, 209/10)
Year
% males passing
% females passing
Overall %
2005/06
54.7
36.9
48
2006/07
56.1
39.5
49.8
2007/08
44.6
28.6
38.4
2008/09
49.9
32.2
42.6
2009/10
69.7
52.7
62.3
Tables 3 and 4 show how the payments for both options work for results in non DRS and in the
DRS The tables show that the reward is higher for girls in all regions and for students in the DRS.
Table 3: Payments for sitters and passers (non DRS)
Category
Option A and B
Boy who sits
£50
Boy who sits and passes
£100
Girl who sits
£85
Girl who sits and passes
£170
Table 4: Payments of sitters and passers (DRS)
Category
Option A and B
Boy who sits
£75
Boy who sits and passes
£150
Girl who sits
£100
Girl who sits and passes
£200
Demonstrating impact
The focus of RBA is achieving additional results over a baseline in order to generate incentives.
The options need to show that results will be achieved with a particular incentive package over a
verified baseline. Actual results with the RBA approach are very difficult to predict.
Demonstrating that results are attributable to the RBA incentive is even more challenging. The
independent evaluation will seek to do this.
Option A proposes earmarking to the education sector but with no further policy or
programmatic conditions. We judge that, given our current dialogue in the sector and the
strategic secondment in the MoE of a senior DFID education adviser, the uses to which our
additional financing would be put would be relatively easy to track (although attributing changes
in results to our funding would be challenging).
Option C earmarks financing to the school level. We judge that this would be the most
straightforward to track and attribute to results over time.
2. Generating incentives
This is a core assumption in the ToC and it will be carefully evaluated. Under option A, the sector
ministry has oversight of the additional financing and will develop criteria for how it is allocated
to regions, and broad guidelines for how regions should use the additional money. We judge that
this will provide enable incentives to be generated at both regional and woreda level to develop
innovative approaches to improving grade 10 examination results.
Option B would earmark funds to the lowest level of the system - the school – and would
therefore be most visible to end users. However, many of the barriers to improved participation
in secondary schooling are not necessarily that schools can tackle themselves (for example,
provision of safe boarding close to the school or improvements in teacher training). We
therefore judge that this option will be less effective that option A in generating incentives at
appropriate levels of the system.
The question of whether the payment is set at the right level has been keenly debated. The
payment is actually based on the average costs of educating a secondary school student. The
evaluation will seek to answer the question of whether this incentive has been set at the
appropriate level.
3. Earmarking to specific policies and programmes
This critical success criteria is concerned with the extent to which the recipient has ownership
and control of the inputs
Neither of the options specify what type of policy response would be appropriate in order to
achieve the results – this is fully in line with CGD’s CoD-Aid approach. However, the two options
can be placed on a continuum, where option A provides better discretion for MoE to flexibility
use the budget with in the education sector, option B provides no discretion to MoE (as this
option earmarks additional resources to schools based on a pre-agreed funding formula). There
are advantages and disadvantages to both approaches as shown in table 5 below:
Table 5: Advantages and disadvantages of earmarking in the two options
Option
Advantages
Disadvantages
Option A Builds on existing education sector Possible risks that additional
dialogue and policies but retains a financing may not reach the point of
high degree of discretion in the service delivery i.e the school
sector
Option B
Ensures that the money flows down Reduces
discretion
that
the
to school level where it may generate government has over how it allocates
significant incentives
the additional money – not in line
with the Co- Aid philosophy
4, Impact on efficiency
The CoD-Aid approach is predicated on an assumption that the incentives created will lead to
increased value for money with respect to achieving the specified result.
The World Bank’s GEQIP Project Appraisal Document stated that “the Public Expenditure Review
(PER) in 2004 showed that the composition of spending by level of education appears
appropriate.
During the past few years, there has been an increase in public spending towards higher
education with the expansion of the tertiary education system, financed mostly by the federal
Government.” However there seems to be two challenges: (i) insufficient spending in education
despite the education budget being 19.5% of total federal budget (or 4.8% of GDP); and (ii) a high
share of expenditure allocated to salaries. These challenges were confirmed by the 2010 Public
Expenditure Reviewxxx.
RBA schemes in general (including CGD’s COD-Aid) are modalities that make payment conditional
on achieved results. The funds are released once results have been achieved. Therefore there is
no injection of funds ex-ante.
The GoE is incentivised to achieve results with existing resources (own and donors’) with the
expectation of a reward (ex-post). That implies that the government needs to make use of the
existing resources in an efficient and effective way to achieve additional results over the
baselines. Efficiency gains are integral to the RBA approach as is reflected in the Theory of
Change.
Efficiency gains will be a key component of evaluation, as this is one of the major lessons to be
learned from RBA – including a test of whether the size of the incentive is sufficient to generate
change (Ethiopia’s education budget is over $1 billion/annum).
Both options entail paying ex-post, i.e. after achievement of results. So both options potentially
trigger efficiency gains.
Option B that earmarks funds at the lowest level, i.e. at the schools, and makes a very strong
assumption that the schools are key institutions in the system that could improve efficiency.
Schools are very important in the delivery of education services, but the education system is
more than just schools. Efficiency gains could be found at different levels from federal to
decentralised regions and below, and the pilot should allow the system to test these efficiency
gains. Therefore, option A provides more leeway to the government to pursue efficiency gains.
5. Equity/not diverting resources away from deprived groups:
Equity is not a primary concern for CoDA, but in the Ethiopian context, there is no question that
equity is an important consideration both in terms of gender and geography. Girls’ gross
enrolment in lower secondary schools is 35% and it is even lower in Somali and Afar regions at
around 6%.xxxi.
Both options directly address gender and geographical equity by building in financial incentives
as Tables 3 and 4 above show. Option B further addresses equity by earmarking funds to all 1,300
secondary schools (although the performance based component could penalise poor performing
schools with lower achieving student populations); however, this would significantly reduce MoE
discretion over how the additional money would be used.
Both options entail some risk that financing will be diverted from primary schools to lower
secondary, undermining equity objectives, as lower secondary students tend to be drawn from
higher income families. This possible unintended consequence will need to be tracked through
the evaluation process.
6. Involvement of civil society
One of the impacts of RBA approaches is that they should strengthen a government’s
accountability to citizens rather than donors.
None of the proposed options has a specific focus on building civil society engagement in
budgeting and verification of results. Option B brings the additional financing very close to the
end users (children and their parents). Option A is within the education sector where GEQIP is
already providing a platform for dialogue on school level accountability and there is a strong
possibility that at least some of the incentive would flow down to school level and be subject to
scrutiny by Parent Teacher Associations as in option B.
7. Other stakeholders buy in to the approach
Ethiopia is a context with multiple donors, often with competing priorities, and some
fragmentation of aid. An important outcome of the pilot is that other donors learn from and
apply the CoDA approach (whether in the education sector or elsewhere).
There has already been considerable consultation in Ethiopia and more broadly on the proposed
pilot and there is interest in the general principles. We judge that a sector based pilot will be
more visible and understandable to stakeholders. CGD critiqued option B as it ties the funds to
schools and was regarded as excessive earmarking. Option A is both visible and relatively hands
off so should generate reasonably good initial buy in and interest. .
D. Comparison of options
Table 6 compares the three options against the six critical success criteria (CSC) identified under
A and using the analysis presented in the appraisal section. Wt is the weight of each CSC – see A.
S is score and WS is weighted score.
Table 6: Comparison of options
CSC
1. There is a demonstrable impact on
results, especially for girls and in
emerging regions
2 Incentives generate to implement
policies and programmes that increase
grade 10 participation and performance,
especially for girls and students in DRS
Incentives generated to
3. The additional financing is not
Wt
5
A: Sector CoD
S
WS
4
20
B: School CoD
S
WS
5
25
5
4
20
3
15
4
4
16
2
8
earmarked to specific policies and
programmes
4. The pilot increases the efficiency of
funding to the sector
5. The pilot does not lead to a diversion
of resources away from deprived groups
6: Civil society bodies, such as parentteacher associations, are more involved
in financial decision-making and data
verification
7: Other donors buy in to the results
based aid idea
TOTALS
4
3
3
4
3
2
12
2
8
12
4
12
3
9
4
12
4
8
3
6
97
86
As the total scores suggest, option A, CoDA tied to the education sector, has come out as the
better option with respect to the critical success criteria.
E. Measures to be used or developed to assess value for money
We applied DFID’s 3Es (Economy, Efficiency and Effectiveness) VfM framework to analyse
how we maximise the impact of each pound to be spent on the Result Based Aid (RBA)
pilot project.
E1 Economy
This relates to how our money buys key inputs of the appropriate quality at the right price. In
Ethiopia it costs approximately £50 to fund a student for two years of lower secondary
educationxxxii. Additional costs of passing the examination, of educating students in the emerging
regions, or of educating a girl are not fully known.
In our appraisal, both options use the following pricing:
Table 4 - Pricing for Options A and B
Passer
Emrg
Non-emrg
Boys
£75
£50
Girls
£100
£85
Sitter
Emrg
Non-emrg
£75
£50
£100
£85
The rationale for the additional cost (over the average cost of £50) is based on the hypothesis
that a financial incentive above cost and paid ex-post encourages the education system to deliver
more results, in excess of what it would otherwise have delivered (without the incentive).
In the case of the education sector, the reward could be considered to be related more to the
marginal cost of reaching marginalised students, such as girls and students in emerging regions,
than with the average cost across the whole of Ethiopia. But how much more should the reward
be above the average cost? Because RBA is meant to improve results and not to fund huge
expansion of services, they act as an incentive for the public administration to be more efficient.
That means that an RBA scheme creates an incentive for the service provider to move toward a
situation where inputs are used most efficiently to provide a (higher) level of services without
changing the technology or without large capital investment.
CGD in their simulation model for education in Ethiopia suggests $200 (approx £123) per student
completing education. The $200 does not seem to be related to costs and it is hypothesised to be
large enough to trigger a response by the education sector to achieve more results (the
hypothesis is untested).
By comparison, both options A and B both use a base price that is tied to the costs of educating a
student for two years in secondary school and apply that as an incentive for both sitting and
passing. Additional costs are hypothesised for emerging regions (relating to under-development
and difficulties of reaching marginalised populations) and girls (socio-economic challenges to
attracting and retaining girls in secondary school). The independent evaluation will test.
Through the independent evaluation we intend to: (i) test whether the pricing structure is
optimal to trigger the expected increase in sitters and passers the impact of the RBA that pricing;
and (ii) to track the value for money over the life of the pilot. One measure of this will be the per
pupil cost of educating students in lower secondary school.
The model used to project additional sitters and passers for the both options, allowed a
calculation of the average costs (investment) per girl sitting or passing Grade 10. An analysis of
the CGD’s pure CoDA is also included in the model spreadsheet just for comparison only (See
Annex C).
The model relating to passers for both options assumes an 80% and 75% increase in passing rate
for girls in emerging and non-emerging regions respectively over an average increase in the rate
without RBA. For boys, the figure is 50% and 40% respectively. Accordingly the model predicts an
additional 178,000 passers would be incentivised. The model for sitters under both options
assumes an 8% and 7.5% increase in sitting rate for girls in emerging and non-emerging regions
respectively over an average increase in the rate without RBA. For boys, the figure is 5% and 4%
respectively predicting that an additional total 191,000 sitters would be incentivised under both
options.
Table 5 - VfM measures –sitters
Option
Million £
Additional sitters
over three years
Same for
both A
and B
£26. 9
191,000 at a total
cost of £13,943,000
Additional passers
over three years
178,000 at a total
cost of £12,994,000
Av. Cost per
student
sitting/passing
£73
Based on the modelling, therefore, both options offer the same better value for money per sitter
and passer. However it is also worth noting that Option B entails additional transaction costs in
order to earmark and channels the incentives all the way down to the 1,300 secondary schools,
so although the average cost per girl passing Grade 10 is the same as Option A, the additional
transaction costs decrease its value for money. Using this measure of VfM, option A is the
preferred option.
E2 Efficiency
The efficiency analysis relates to how well our partners (in this case the GoE) convert inputs into
outputs. The fact that the pilot pays on the achievement of results is anticipated to leverage
greater efficiency from the education system.
In the literature, educational efficiency is reported to have a high correlation to quality and the
commonly used measure of educational efficiency is student’s drop-out rates and repetition
rates. As highlighted in the recently published Ethiopian Economic Association (2011)
publication on the Ethiopian economy, high drop-outs and repetition rates imply inefficiency
and wastage of resourcesxxxiii. Drop out is not a major problem in secondary schooling, but as
noted above, many students who sit the grade 10 examination fail it. If the pilot is able to
incentivise the system to increase the grade 10 pass rate then this will result in efficiency gains.
E3 Effectiveness
The potential results benefits of a programme focusing on improving secondary education
access and quality include:
Increased earnings potential – In economics literature, education is viewed as an investment
that turns unskilled labour into skilled labour which increases returns in the labour market. As a
result, education is expected to be positively related to labour market outcomes (Malhotra
et.al., 2003). For example, one estimatexxxiv suggests that the rates of return to one additional
year of schooling, averaged over 100 countries, is 10%. Figure 4 below shows rates of return
analysis for education in Ethiopia conducted in 2005xxxv. This shows significant and increasing
returns to higher levels of education. Another model based analysis by Tilahun (2005)xxxvi
revealed that extra year of schooling for an Ethiopian manufacturing worker would generate
10% rate of return for both men and women.
Figure 4: Monthly mean earnings per level of education: Ethiopia 2001 (from World Bank, 2005)
This analysis draws on other studies in Ethiopia that have found consistently found positive
effects on investments in education for earnings. These include:
o
Tilahun (2005)xxxvii - an extra year of schooling for an Ethiopian manufacturing worker would
generate 10% rate of return for both men and women,
o
Verwimp (1996)xxxviii - based on a small sample of 422 male wage earners employed in the
public and private sectors in 1994 revealed that an extra year of schooling yields an
estimated return of 15 percent in urban Ethiopia.
o
Wolday’s (1997)xxxix - Using data for 1996 for 843 workers in a state owned enterprise (Edget
Cotton Factory) and 170 workers in a formal sector private enterprise (MOENCO), found a
return of 5 percent in the public sector and 8 percent in the private sector for an additional
year of schooling.
Krishnan, Selassie, and Dercon (1998)xl - using data from the 1997 Ethiopian Urban Household
Survey, revealed varied rates of return to education for public and private sector employment.
According to the result, an extra year of schooling would yield a return of 10.6 percent for
primary education, and 15 percent for both secondary and higher education for men employed
in the public sector. On the other hand, for those working in the private sector, study came out
with 0 rate of return for primary education, 8.2 percent return for secondary education and 21.5
percent return for higher education.
The demographic bonus If the pilot succeeds in increasing the number of girls who successfully
complete primary and secondary education there will be positive externalities related to lower
fertility rates and reductions in early marriage. According to a 2005 study, raising education
levels for women is a key factor in accelerating demographic transition in Ethiopia. The same
study stresses the importance of girls’ education in reducing early marriage and fertility and
increasing economic growthxli.
Health benefits Improved levels of education have been associated with better health of
children through parental education; increased life expectancy; improvement in spouse’s health
and lowered mortality; and improved contraceptive efficiencyxlii. Education leads to reduced
fertility by enhancing women’s knowledge, decision making power, confidence in interacting
with the outside world, closeness to their husband and economic and social self reliance.xliii
Maternal education is a key factor in improved child survival rates, and even low levels of
education increase child survival.xliv Maternal education is also positively related to knowledge
of immunisation and may be expected to influence accessing preventative health servicesxlv.
Child nutrition is positively and independently associated with mothers’, fathers’ and
grandmothers’ education.xlvi
Improved equity – poor children, those living in rural areas, and pastoralists are amongst the
groups who are less likely to benefit from improved education opportunities. By providing
additional incentives to the DRS, the RBA pilot is expected to have a disproportionate benefit on
the poor, rural children
Aid effectiveness – the hypothesis to be tested is that the approach will have a significant impact
on aid effectiveness with respect to reduced transaction costs and conditionality and increased
government ownership.
E4 Cost effectiveness
Cost benefit analyses was carried out for both options to measure the net benefits attributed to
each option and the impact on poverty reduction of the Result Based Aid (RBA) programme
relative to the additional inputs (£27 million) to be allocated (see annex D). The cost benefit
analysis focuses on two benefit streams: a) public expenditure efficiency gains associated with
system strengthening and b) the impact on growth (using a proxy of private returns to
secondary education) associated with the increase in the number of semi-skilled workers in the
economy as a result of the investment in education. We employed two methods to this effect:
one based on the sum of the efficiency gain and the private return to investment in secondary
education and another based on efficiency gain only.
Impact on Growth: Considering the global and Ethiopia specific evidence regarding return to
investment to secondary education as summarized in the effectiveness section above, we
assumed 20% return to investment in secondary education which translates to £32.3 million
monetized benefits using method 1 for both options. This is because in both options we
assumed a similar number of exam sitters and passers which benefits from this scheme. Under
method 2 we did not assume impact on growth of the RBA.
Efficiency Gain in Public Expenditure: General or sector budget support will generate efficiency
gains in public expenditure. However, as indicated in a Joint Evaluation of Budget Supportxlvii, the
magnitude of these effects depend on whether an increase in discretionary funds is actually
made available to the government budget. It was reported that in some cases general budget
support clearly improved allocative efficiency by enabling the governments to complement
earmarked resources. The same evaluation also highlighted the positive contribution of General
Budget Support to operational efficiency of public expenditures by facilitating a better balance
between capital investment and recurrent spending in government budgets. The proposed
Result Based Aid programme, which a sector budget support type of instrument, is thus
envisaged to deliver efficiency gain to public expenditure in education.
But, we do not have empirical evidence on efficiency benefits of sector budget support
programme in Ethiopia. So, in the absence we have looked in to evidences from budget support
programmes in other countries. One notable experience is the Malawi’s General budget support
programme economic appraisal work. The Pearmanxlviii paper for Malawi’s GBS Economic
Appraisal mentioned about the possible indirect benefits from Poverty Reduction Budget
Support (PRBS) programmes in the form of GDP growth and labour productivity enhancements.
They assumed 0.4% efficiency gain benefits in their economic appraisal of Malawi’s General
Budget support.
So, taking note of such country experience and since the RBA investment is assumed to be
channelled directly to the education sector with a greater incentive for the government to
design better systems for nationwide application we have assumed a relatively higher efficiency
gain of 0.6% for option A in our RBA cost benefit calculation. For option B, on the other hand,
since the investment would be directed to the school, the incentive to design and implement
better systems would be limited to the schools. So, we assumed a relatively lower efficiency gain
of 0.4% which is based on assumptions from a direct budget support programme in Malawi. This
delivers discounted efficiency gain of £40.9 million for option A and £26.7 million for option B
under both methods 1 & 2. We assumed that the efficiency benefits will be higher during the
intervention period and then will diminish thereafter.
Overall Economic Appraisal: Over a 14 year period, the cost benefit analysis using Method 1
(our preferred approach) resulted in a Net Present Value (NPV)xlix of £50.25 million and a Benefit
Cost Ratio (BCR) of 3.19 at 12% discount rate for option A (Table 6). This implies that over £3 of
benefits is delivered to Ethiopia for every £1 that is spent in the education sector through
Result-Based Aid (RBA) approach i.e. the incremental benefits of the programme exceed the
incremental costs by three times (see the attached spreadsheet).
For option B, method 1 resulted in Net Present Value (NPV) of £36.06 million and a Benefit Cost
Ratio (BCR) of 2.57 at 12% discount rate. Our preference in using the 12% average lending rate
as discount rate is based on the consideration of the fact that Ethiopia is a highly credit
constrained economy. This is in line with the recent discount rate guideline which highlights the
fact that the SOC is more appropriate for making shorter-term trade-offs in economies marked
by a severe shortage of resources. Also, the World Bank used 12% discount rate for Ethiopia in
its economic appraisal of the Agricultural Growth Programme (AGP).
It is not possible to calculate an IRR for this method because of difficulties in generating a multiyear benefit stream. Also, it should be noted that even with out considering efficiency gain
benefits, this method generates NPV amounting to £9.37 million and Benefit-cost ratio of 1.41
for both options which again justifies the RBA investment from the VfM perspective.
Table 6: Economic Appraisal on RBA: Rate of Return + Efficiency Gain (Method 1)
Particulars
Option A
Option B
0.20
0.20
396000
396000
73
73
87.6
87.6
Total Benefits (£)
32,324,400.00
32,324,400.00
Total Discounted costs (£)
22,956,247.12
22,956,247.12
Net present value (£) (excluding
efficiency gain)
9,368,152.88
9,368,152.88
BCR (excluding efficiency gain)
1.41
1.41
Discounted efficiency gain (£)
40,886,543.48
26,696,293.48
Total discounted benefits (£)
(including efficiency gain)
73,210,943.48
59,020,693.48
Net present value (£) (including
efficiency gain)
50,254,696.4
36,064,446.4
3.19
2.57
Private Return
education
to
secondary
Number of Pupils (beneficiaries)
passing the grade 10 exam over
four years
Cost per pupil (£)
Value per beneficiary (£)
BCR (including efficiency gain)
So, based on the estimated results using method 1, option A delivers better VfM from a costbenefit perspective as it yielded greater NPV of £50.25 million and benefit cost ratio of 3.19
based on method 2 compared to option B.
We have also employed a second methodology that only considers the efficiency gain benefits
of the RBA investment. This method generates a lower NPV and benefit cost ratio estimates
compared to method 1. That is, for option A the NPV amounted to £18 million with an Internal
Rate of Return (IRR)l of 12% and a benefit cost ratio of 2. For option B, on the other hand, th
estimated NPV using this method is quite low (£3.7 million) with an Internal Rate of Return4
(IRR) of 3% and a benefit cost ratio of 1.2. In both cases, these benefit streams would have been
higher had we have included the growth effect.
It is worth mentioning the caveats of this economic appraisal on RBA investment. First, this
result is based on some strong assumptions regarding the possible efficiency enhancement
contributions of a four year RBA programme far in to the future. So, it might be worth
undertaking a research to understand the real contribution of education related expenditure
towards public expenditure efficiency enhancement. Also, the estimation depends to a certain
extent on the assumed rate of return to investment in secondary education.
To test whether the estimated results are sensitive to the assumed parameters, sensitivity
analysis was made on the estimated cost-benefit results for option 1 under method 1. This was
done by varying the private return to investment in secondary education and public expenditure
efficiency-improvement parameters in three different scenarios.
First, if private return to investment in secondary education was reduced to 10% while keeping
the same public expenditure efficiency improvement parameter, the NPV becomes £47.6 million
and BCR of 3.07. Even at this level, the appraisal revealed positive NPV which shows good value
of money of the RBA support programme.
Second, if we reduce the private return to investment in secondary education parameter to 5%
while keeping the same public expenditure efficiency-improvement, the NPV becomes £46.2
million and BCR of 3.01. Again, the appraisal revealed good value of money of the RBA support
programme and we would accept the envisaged RBA intervention.
Finally, if the private return to investment in secondary education parameter was kept at the
same level while reducing the public expenditure efficiency-improvement to 0.2%, the NPV
becomes £22.2 million and BCR of 2.0. This appraisal result is again in favour of the decision to
accept the envisaged RBA intervention.
In general, the above appraisal fully supports the case for a) providing financing to RBA and b)
channelling that financing through the education sector budget. RBA should be seen as part of
the existing package of DFID’s support to basic services in Ethiopia including through the
Protection of Basic Services (PBS) programme which is complementary to the sector specific
programmes in education. Work is also underway to explore complementary and focussed
programmes to improve equity across the regions.
4
IRR is the internal rate of return on an investment. IRR finds the discount rate that makes the NPV equal to zero.
The discount rate is the cost of borrowing or using money for investments. The decision to accept or reject the
purchase depends on the whether the internal rate of return is higher than the discount rate. The decision criteria is
that, accept the project if the IRR is higher than the discount rate or the cost of borrowing.
3. Commercial Case
Clearly state the procurement/commercial requirements for intervention (distinguishing
between direct and indirect procurement)
All procurement undertaken with financing from the RBA pilot will all be indirect and will be
undertaken by the Federal Ministry of Education, universities, regional governments, woredas,
and schools. At the federal level, the procurement Department in the MoE is responsible for the
oversight of any procurement activities.
DFID will undertake direct procurement of firms to conduct the independent verification and the
evaluation (see separate Business Case).
A. Why is the proposed funding mechanism/ form of agreement the right one for this
intervention, with this development partner?
One of the core objectives of RBA is to give more ownership to partner governments with
respect to how they use aid. A Memorandum of Understanding with the government, which
guides the principles on which direct payments will be made (i.e. on the basis of results with an
agreed price), is considered the most appropriate choice of funding mechanism. As part of the
MoU the Ministry will be required to transparently communicate decisions to stakeholders on
how the money is allocated. This will be communicated at the annual education conference (a
multi stakeholder event). Indirect support also potentially reduces costs and is likely to offer
good value for money as financing will use government’s own systems and guidelines with no
additional costs of administration or oversight.
The actual expenditure items in the project cannot be identified upfront with this kind of
approach. Preliminary discussions with the Ministry have indicated that they will likely reward
regions on the basis of their school performance in the examination. In their turn, regional
governments have suggested that they would reward districts and schools in the form of
enhanced grants. Spending on a range of inputs could take place at any of these levels.
B. What has been done to assess whether the third party organisation has the necessary
capability and capacity to obtain best value for money from the funds they are spending on
behalf of DFID? Where improvements in capability or capacity have been identified how are
these being taken forward?
Assessment of national procurement systems
The recent World Bank Country Procurement Assessment Review (CPAR)li at Federal level found
procurement systems to be acceptable overall when benchmarked against international good
practice. The CPAR did however identify a gap in procurement audit capacity as the Auditor
General function is not equipped to focus specifically on procurement. The MoE has
commissioned independent consultancy services for the procurement audit of GEQIP. Although
substantial improvements to the legal framework for public procurement have been made in
Ethiopia in the last few years, procurement capacity remains a significant challenge at all levels of
government. The Federal Public Procurement Agency (FPPA) has currently embarked on a
multifaceted approach to improve the procurement capacity of the GoE including strengthening
the capacity of staff and oversight functions.
Assessment of education sector procurement systems
A procurement capacity assessment of the education sector was carried out by the World Bank
during appraisal of the GEQIP project in 2008. This focused on the Federal MoE and
implementing entities in the project. Risks were assessed as substantial with respect to the legal
framework and high with respect to structures for procurement management. The following
were among the mitigation measures that were agreed during appraisal and where capacity
continues to be built:
o
Conduct regular procurement supervision and an annual procurement audit
o
Recruit additional staff at federal and regional levels to assist with procurement
o
Ensure procurement training for key staff
Procurement for the RBA pilot under the country’s financial management and
procurement systems
Under an RBA approach, it is not possible to identify upfront which, if any, goods and services will
be procured, and who would procure them. Given the amounts of money involved and the way
in which the MoE is proposing to allocate them, it is likely that most of the expenditure will be on
softer inputs such as training, or small purchases of goods below the World Bank international
procurement thresholds. The evaluation will track and feedback on how the money is used and
this will feed into the lesson learning.
B. Value for money through procurement
Any procurement undertaken through the pilot will use GoE procurement guidelines and
process. These guidelines include measures to ensure value for money in both the procurement
of goods and services
4. Financial Case
A. How much it will cost
The expected cost of this programme is up to £30 million over three years.
The approximate breakdown of these costs by year (generated by the modelled estimates) is as
follows:
2011/12
Nil
2012/13
£8,000,000
2013/14
£9,000,000
2014/15
£10,000,000
Specific issues exist around predicting payments based on an RBA approach. Some work has
been done to model projections related to the price and likely results in order to reduce the risk.
However, it is conceivable that results could either exceed or fall short of expectations resulting
in either significant overspend or underspend. DFID is proposing to set an annual financial ceiling
for the project of £10 million. However, there is a risk that in year 3 the results may exceed the
financial ceiling. If this were to happen, we would seek to secure additional resources from
within the DFID Ethiopia Programme budget. In the event of an under spend, we would reallocate funding to another sector or programme within the DFID Ethiopia programme portfolio.
Additional funds of up to £1.5 million will be required for data verification and impact evaluation
(see Business Case at annex B).
B. How it will be funded: capital/programme/admin
The programme will be funded from programme resources, and has been budgeted for in the
Operational Plan for DFID Ethiopia (2011 – 2015). The planned resources for the next three
years are within the current spending round period (to 2015).
There are no contingent or actual liabilities.
C. How funds will be paid out
Funds will be paid to the Federal Ministry of Education in arrears based on annually verified
results. Payments will be governed by the terms of the formal aid exchange (MoU) between DFID
and the Ministry
Payments for directly procured services (evaluation and data verification) will be made directly to
the service providers based on agreed milestones and will be governed by the terms of the
relevant contracts. These are covered by the evaluation and verification business case.
D. How expenditure will be monitored, reported, and accounted for
D1 Assessment of national financial risk and fraud
Results rather than finances are the main subject of monitoring, reporting and audit under the
proposed RBA pilot. This is because payment to the MoE can only be made once reported results
are verified through an independent process. The flow and use of funds is of interest to the
independent evaluation of the project rather than as a condition for payment. Nevertheless, we
judge the countries public financial management systems to be good. The 2010 Ethiopia Public
Expenditure and Financial Accountabilitylii (PEFA) review concluded that overall the public
financial management systems and processes were functioning well by international standards.
Several weaknesses were identified, but the conclusion was that these areas are being addressed
adequately by the Government’s Expenditure Management and Control Programme.
Similarly, the DFID 2010 Fiduciary Risk Assessment (FRA)liii assessed the overall fiduciary risk as
moderate with a positive trajectory of change and a credible programme of financial
management reform currently in place. Fiduciary risk in general and corruption risk in particular
is considered lower in Ethiopia compared with other countries at a similar stage of development.
A recent study of corruption in Ethiopialiv concludes that perceived corruption risks in Ethiopia’s
health and education sectors are lower than those typically encountered in many developing
countries as well as in other sectors within the Ethiopian public administration. A range of
accountability systems are in place in the sectors, which are particularly strong on the supply
side (e.g. internal audit systems and vertical accountability to the various levels of the
administration). Although RBA finds will be difficult to track once mixed with other MoE funds,
DFID will have access to MoE ‘s internal audit reports.
D2 Monitoring
MoE will announce the results and the regional allocation at the Annual Education Conference.
MoE will also develop guidelines for how the additional money should be utilised at regional
level. Additional financing will be transferred from MoE to the regional bureaus through existing
Bank accounts. The MoE is planning to ask regions to report on how the money was used at the
subsequent Annual l Education conference. Financial accounting, reporting and auditing will
follow government guidelines as described below. In line with RBA principles, DFID-E will not ask
GoE to track the additional money separately within the system. The evaluation will seek to
understand how the additional money affects resource allocations and priorities at lower levels.
D3 Accounting
The GoE’s accounting policies and procedures will be used for accounting for the project. The
GoE follows a double entry bookkeeping system and modified cash basis of accounting. This is
documented in the GoE’s accounting manual. This has been implemented at the federal level
and in many regions. The main elements of the ongoing accounting reform are the adoption of
(i) a comprehensive Chart of Accounts consistent with the budget classification; (ii) a system of
ledgers accommodating all types of accounts (including transfers, assets, liabilities and fund
balance in addition to revenues and expenditures); (iii) double entry book-keeping (thus, self
balancing set of accounts); (iv) a system of control of budgetary commitments (recording
commitments as well as actual payments); (v) modified cash basis transaction accounting; and
(vi) revised monthly report formats to accommodate double-entry book-keeping and
commitment control and permit better cash control. The GoE’s accounting manual provides
detailed information on the major accounting procedures, including controls.
If regions use additional money to enhance school grants, a guideline has been developed and
rolled out to all schools and ABE Centres. A simple columnar cash book in a standardised format
should be kept and supporting documents should be maintained for receipts collected and
payments made by schools for all their funds (i.e. school grants provided by GoE and funds
raised by the community). The PTA/Community exercises oversight over school funds. Schools
should submit a copy of their cashbook and supporting documents quarterly to the Woreda
Office of Finance and Economic Department (WoFED).
D4 Financial reporting
Existing financial reporting according to GoE procedures will apply along the following lines:

MoE and Universities report monthly to MoFED

Regional Bureaus and Colleges of Teacher Education report monthly to the regional
bureaus of finance; and

Woreda finance departments report monthly to zones who in turn report quarterly
to regional Bureaus of finance, on all their expenditures.
These reports include information such as trial balance; expenditures by line items (based on
the GoE’s functional classification system), with a break-down by economic classification for
each line item; details of cash and bank balances, and break-down of receivables and payables.
A distinctive feature of results-based aid is that financing will be released based on verified
results achieved each year. No additional financial reporting beyond regular reports is envisaged
although how funds are spent will be monitored for learning purposes as part of the impact
evaluation. The government’s financial management systems were assessed by the World bank
as part of the GEQIP Appraisal process and judged to be robust enough to use for GEQIP
financing (financing channeled through the sector is subject to Ministry of Finance oversight and
reporting requirements)
D5 Auditing
The external audit is carried out by the Office of Federal Auditor General (OFAG). Internal audit
are carried out by the internal audit departments of the respective entities (MOE, Universities,
REBs, CTEs). In addition, MoFED, BoFEDs and WoFEDs perform internal audit including an
assessment of whether the budget utilisation is in line with the intended purposes.
Furthermore there are inspection departments at the MoFED and BoFEDs with a role of
ensuring quality of internal audits at the Ministries at Federal level and Sector bureaus at region
level, following up on the audit recommendations noted by audit reports and providing training
and improving manuals.
D6 Costs related to financial management
All costs related to financial management will be the responsibility of the GoE as per their
regular system of accounting and reporting.
D7 Asset Management
Assets under this programme can not be attributed to DFID, nor can DFID track or monitor them
given that funds will be mixed with government financing. Therefore, the assets will be owned,
monitored and controlled by the various GoE implementing entities involved in this programme.
5. Management Case
A. Oversight
Overall oversight of the programme will be the responsibility of the Planning and Resource
Mobilisation Process Owner in the Ministry of Education. The Minister of Education himself has taken a
direct interest in the programme. The Ministry of Finance are aware of the pilot and interested in
tracking the impact on aid effectiveness.
At decentralised levels of the systems, all eleven regions of Ethiopia will have responsibility for
overseeing resources transferred through the RBA programme (if the federal ministry decides to
allocate to that level). Regions may further allocate funds to woredas and schools to generate
incentives at local levels.
It is envisaged that all these stakeholders will play a key role in ensuring that results are achieved and
in deciding how funds are allocated. A key aim of results-based aid is to increase local accountability,
e.g. to citizens. It is envisaged that parent-teacher associations will also play a role in this regard,
including in the verification of data, if additional funds reach schools.
The RBA will take account of transparency in the sector and will include clear statements in the
Memorandum of Understanding with the Ministry of Education about how they will communicate
access criteria to the education institutions at various tiers of government. The independent evaluation
will undertake an ex-post assessment to verify the effectiveness of the system and to ensure value for
money.
B. Management
B1 Management processes and structure within DFID
This programme will be managed by a DFID Education Adviser based in the DFID Ethiopia. He/she will
meet on a periodic basis with the Ministry of Education’s Planning and Resource Mobilisation Owner to
receive updates on progress of the project. He/she will also attend the Annual Education conference
where it is anticipated that the Minister of Education will make a public announcement about the
regional allocation from the RBA. The adviser will lead on the dialogue with MoE on the results of the
annual verification process. A senior education adviser, partially seconded to the MoE as part of QESSP
will provide additional advice and support.
An advisory group will be established, including participation from the MoE, in order to oversee the
conduct of the independent evaluation, ensure technical quality, and disseminate results as seen fit.
B2 GoE management
MoE, in collaboration with MoFED, will be responsible for managing the allocation of the additional
resources to the eleven regions and to universities who deliver training for secondary teachers.
The regional bureaus of education will manage further transfers to woredas and schools through
BoFEDs and WoFEDs and will also be responsible for any regional expenditure for activities such as
capacity building.
C. Conditionality
Although RBA pays by results, the UK’s overall conditionality will still apply. If any of the following are
breached, then the payment could be suspended even if the results are met:
(i)
poverty reduction and the millennium development goals
(ii)
respecting human rights and other international obligations and
(iii)
improving public financial management, promoting good governance and transparency and
fighting corruption; and
(iv)
strengthening domestic accountability
The disbursement of RBA funds will vary: the more results achieved by GoE the more funds will be
released (up to a ceiling of £10m per year). We have provided some estimates of disbursements in the
appraisal of option 3. Any reduction in annual disbursements from those estimates will not constitute a
breach of conditionality.
E. Monitoring and Evaluation
The separate business case (see Annex B) for independent verification and evaluation of this pilot sets
out plans for monitoring and evaluation and presents the appraisal of different option considered for
this purpose. In summary, this project will be monitored and evaluated in three main ways.
First, it will be monitored through the GoE regular system of data collection, which includes federal
scoring and publication of grade 10 examination results annually. These results are published in the
Annual Statistical Abstract. It is anticipated that the project will help to strengthen these systems.
The main outcome and output indicators to be tracked are set out in the logical framework. At the
outcome level the key indicators of interest are the numbers of pupils sitting and passing the grade 10
examination. The pilot will monitor changes in the number of pupils (dis-aggregated by sex) sitting and
passing in each regional state from the 2011 baseline.
The nature of the pilot will make it difficult to determine from this information alone the specific
impact on sitting and pass rates of results based payments. As such monitoring of the selected output
indicators is also critical. A key measure of the incentive effects of the pilot will be to monitor the
extent to which government and regions introduce new policies or programmes in response. In
addition to monitoring the extent of the introduction of new policies it will also be of interest to
examine any differences in approach between regions and/or whether Government seeks to influence
approaches at the federal level.
At the output level the pilot will monitor the extent to which an incentive based funding instrument
can drive more efficient use of existing resource. A key indicator of process will be the extent to which
the overall proportion of secondary education funding is not significantly altered from the current
position.
Finally a measure of the pilot’s overall success will be the extent to which other stakeholders, including
GoE and other donors, perceive RBA as being an effective instrument and whether it leads to similar
arrangements either in the education sector or beyond.
All of these questions will be examined more fully by the planned full-scale evaluation (see below) but
the log-frame identifies a number of high level indicators which will provide a measure of real-time
progress during the pilot life-cycle.
Second, the government’s reported results will be independently verified. This work will involve
verifying baseline data and conducting annual verification of results as a basis for accurate calculation
of the aid to be paid to the MoE.The scope of the independent verification will not extend to reexamining students, but it will at a minimum include a cross-check of students who sit the examination
against school records in a sample of schools and a review of the cut off point for passing the
examination (the pass mark is norm referenced and has been subject to fluctuation over the past five
years, so it will be important to agree this with respect to verifying passers).
Thirdly, given that it is an innovative pilot the whole project will be subject to a rigorous evaluation
that will test the assumptions and linkages on the project’s Theory of Change presented in the
appraisal case above. The evaluation will seek to generate evidence by answering the following four
broad questions:
1. Did the RBA pilot increase educational results? This will require robust demonstration of
additional results achieved. This is at the heart of the impact evaluation and requires the
identification of a plausible and credible counterfactual. This differs significantly from the
independent verification of results which is focused on determining the extent to which the
results reported by government are accurate and reliable. This element of the evaluation is
focused on establishing rigorously that the results achieved as a consequence of the RBA pilot
were greater than would have been achieved without the RBA pilot.
2. What were the processes that led to these results? This will require a qualitative assessment of
the incentives created at various levels of government including federal ministries, regional
bureaus, Woredas and schools and how the different actors react to these. Specific focus should
be placed on how a national-level incentive has ‘trickled down’ to regional, Woreda and school
levels. It would be useful to examine if these incentives extended beyond the formal state sector,
e.g. to alternative basic education and non-governmental education providers. It is expected that
the evaluation should identify and document best practices in terms of school initiatives and
policy responses. The contractor will be expected to assess the level at which incentives were set.
This should include analysis of what evidence there is that higher incentives, e.g. for girls and in
emerging regions produced better results. The contractor will be expected to present and
analyse any evidence of the optimum level of incentive in terms of both increasing results and
ensuring value for money.
3. How did the RBA pilot impact on aid relationships? This element of the evaluation should cover
the expected effects of the RBA pilot on relationships between GoE and DFID, and between GoE,
DFID and other development partners. The contractor should focus particularly on interactions
between the RBA pilot and DFID’s policy dialogue with GoE. In what way did DFID’s policy
dialogue contribute to or hinder results achieved? In what way did the RBA pilot increase or
reduce DFID’s policy dialogue with GoE?
4. What were the unintended consequences of the RBA pilot? Unintended consequences, both
positive and negative including effects on equity, diversion of existing financial resources away
from other sectors various forms of ‘gaming’, including ‘cheating’ in the Grade 10 examinations.
Cheating should be identified through the data verification processes but will be reviewed as part
of the evaluation.
F. Risk Assessment
The key risks associated with the proposed pilot are identified and analysed in Table 7 (where 1 is low
and 5 is high).
Table 7: Identified risks associated with results-based aid pilot in education sector in Ethiopia
Risk
Probability
1–5
Implementation/operational risks
The project does
not generate
sufficient
incentives to drive
up results
The baseline is set
too low and the
project pays for
results that would
have been
achieved anyway
Overspend or
underspend
Political risks
Unearmarked
funds paid on the
basis of results
achieved could be
perceived as being
used for ‘political’
purposes
Diversion of efforts
from other sectors
or from other subsectors in
5
Impact
1-5
5
5
4
5
5
3
5
2
5
Mitigation
Residual
Probability
Residual
Impact
Including incentives for both
sitters and passers, focusing on
the education sector only, and
using an adjusting baseline have
all been used to ensure that
incentives are generated.
Medium
Medium
Medium
Medium
DFID is proposing to set an annual
financial ceiling for the project of
£10 million. However, there is a
risk that in year 3 the results may
exceed the financial ceiling. If this
were to happen, we would seek to
secure additional resources from
within
the
DFID
Ethiopia
Programme budget. In the event
of an under spend, we would reallocate funding to another sector
or programme within the DFID
Ethiopia programme portfolio.
High
Medium
Making an agreement with the
Ministry of Education, where
DFID-E has existing dialogue and
tracking how the additional
financing is used by the
Government and the types of
programmes and policies which it
generates through the evaluation.
The pilot is being delivered within
the framework of the government
of Ethiopia’s broad based poverty
reduction plan and sectoral plan
Low
Medium
Low
Medium
Modelling of the results and the
use of an adjusting baseline are
expected to minimise this risk
education
with a clear framework and
targets. This will be monitored
through the evaluation process.
Fiduciary risks
Fiduciary risk
5
3
Corruption
2
5
Overspend or
underspend
5
5
RBA does not place any additional
financial reporting demands on
GoE. The financing will flow
through regular government
financial systems within the
education sector. Risks are
assessed as medium with respect
to use of government systems and
the evaluation will include a
review of the financial
management systems.
There are no additional
monitoring requirements for the
additional financing, but the
independent evaluation will
monitor how funds are used. The
MoU will include commitments to
transparently communicate how
incentives will be allocated to
education institutions
DFID is proposing to set an annual
financial ceiling for the project of
£10 million. However, there is a
risk that in year 3 the results may
exceed the financial ceiling. If this
were to happen, we would seek to
secure additional resources from
within
the
DFID
Ethiopia
Programme budget. In the event
of an under spend, we would reallocate funding to another sector
or programme within the DFID
Ethiopia programme portfolio.
High
Medium
Low
Medium
High
Medium
G. Results and Benefits Management
The outcome indicators for the programme have been modelled based on past performance (based on
data on for grade 10 sitting and passing over the past three years that are presented in the annual
statistical abstract). The model estimated a year on year increase in passers and sitters with and
without RBA as follows:
Average increase in sitters and passers with and without RBA
Passers
Without RBA With RBA
Emerging
Male
11.4
17.1
Female
13.1
23.6
Non emerging
Male
7.7
10.8
Female
12.9
22.6
Sitters
Without RBA
2.7
7,4
1.9
7.7
With RBA
7.7
15.4
5.9
15.2
This model then yielded estimates for the number of additional passers and sitters that would be
generated by the results based financing. The assumptions for the size of the effect were estimated
based on a combination of the known results and the amount of the additional financing. Please refer
to the model which is annexed (Annex C) to the Business Case.
The projections are therefore not real targets in the conventional sense of the term. These are results
that are expected to be incentivised by the project. There is a possibility that these could be over or
under estimates and consequently DFID has set an upper limit for payment in any one year of £10
million. The actual results achieved will be managed and monitored through the verification process.
This will seek to ensure that DFID only pays for actual passers and sitters
Similarly, the output indicators are not directly attributable to DFID spending. It is anticipated that
these actions will occur as a response to the incentive to reward the government for verified increases
in grade 10 sitting and passing. The outputs identified are derived from the Theory of Change.
The baselines in the log frame are all based on projections and numbers will need to be updated once
we have actual data and results for 2011.
The independent evaluation of the project will monitor the extent to which the project is incentivising
the government to increase passers and sitters, including its impact on equity, and will make
recommendations to government and to DFID based on the findings.
References
i
Birdsall, N. and Savedoff, W. (2010) Cash on Delivery: A new approach to foreign aid. Centre for Global Development,
Washington DC
ii Bongaarts, J. (Nov. 2003) Completing the Fertility Transition in the Developing World: The Role of Educational Differences
and Fertility Preferences in Population Studies, Vol. 57, No. 3, pp. 321-335,
iii DFID Ethiopia (2011) Operational Plan 2011 – 2015, DFID Ethiopia, Addis Ababa
iv Federal Democratic Republic of Ethiopia (2010) Health Sector Development Programme III, Annual Performance Report: EFY
2002 (2009/10). Ministry of Health, Addis Ababa
v Devereux et al (2006) Ethiopia’s Productive Safety Nets Programme. Trends in PSNP Households with Targeted Transfers.
Institute of Development Studies, Sussex.
vi Federal Democratic Republic of Ethiopia (October 2010) Education Statistics Annual Abstract: 2009 - 2010. Ministry of
Education, Addis Ababa
vii OECD.STAT: data extracted on 20 Jul 2011 07:17 UTC (GMT)
viii Population Division of the Department of Economic and Social Affairs of the United Nations Secretariat, World Population
Prospects: The 2010 Revision, http://esa.un.org/undp/wpp/index.htm, Wednesday July 20, 2011; 4:33:42 AM
ix Education For All (2011) Global Monitoring Report, Statistical Tables 2011
http://www.unesco.org/new/en/education/themes/leading-the-international-agenda/efareport/statistics/statisticaltables/,Accessed 27 July 2011 11:00
x ibid
xi Federal Democratic Republic of Ethiopia (2010) Education Statistics Abstract 2009 – 2010. Ministry of Education, Addis
Ababa, Ethiopia
xii Birdsall, N. and Savedoff, W. (2010) Cash on Delivery: A new approach to foreign aid. Centre for Global Development,
Washington DC
xiii Birdsall, N. Cash on Delivery Aid: Exploration of Feasibility in Ethiopia. Based on Addis Ababa Visit, December 11, 2009
xiv Birdsall, N. and Savedoff, W. (2010) Cash on Delivery: A new approach to foreign aid. Centre for Global Development,
Washington DC
xv UNESCO (2010) EFA global Monitoring Report 2010 Reaching the Marginalised. UNESCO, Paris
xvi DFID (April 2011) Structural Reform Plan, 2011 – 2015 http://www.dfid.gov.uk/About-DFID/Finance-andperformance/DFID-Business-plan-2011---2015/
xvii DFID-Ethiopia Operational Plan 2011 – 2015 (unpublished)
xviii Federal Democratic Republic of Ethiopia (November 2010) Growth and Transformation Plan 2011 – 2015. Minsitry of
Finance and Economic Development, Addis Ababa
xix Federal Ministry of Education (2011) Education Sector Development Programme (ESDP) IV 2010/11 – 2014/15, Addis Ababa
xx DFID (2010) Primer on Results-based Aid and Results-based Financing. DFID London
xxi Federal Democratic Republic of Ethiopia (2010) op cit
xxii http://data.worldbank.org/indicator/NY.GDP.PCAP.CD/countries
xxiii Hanushek, E. and Wosman, L. (2007) Education Quality and Economic Growth. The World Bank, Washington DC
xxiv Jennings, M., Mekonnen, E., and Gudissa, D. (April 2011) Social Assessment for the Educaiton Sector, Ethiopia Social
Developmetn Direct, London
xxv Birdsall and Savedoff (2010) ibid
xxvi
Pearson, M., Johnson, M. and Ellison, R. (2010) Review of Major Results-based Aid (RBA) and Results-based Financing
(RBF) Schemes.
xxvii
xxviii
Ibid
Brenzel, 2009 Taking Stock: World Bank Experience with Results-Based Financing, (RBF) for Health. World Bank,
Washington DC.
xxix Steer, L. and Baudienville, G. (2010) What Drives Donor Financing of Basic Education? Overseas Development Institute,
Project Briefing 39
xxx Ravishankar, V.J. (2010) Ethiopia Education Public Expenditure Review. Federal Democratic Republic of Ethiopia
xxxi Federal Democratic Republic of Ethiopia (2010) op cit
xxxii Ibid
xxxiii Ethiopian Economics Association (2011), Report on the Ethiopian Economy: Financial Sector Developments in Ethiopia –
Performance, Challenges and Policy Issues.
xxxiv Psacharopoulos, G. and Patrinos, H.A. (2002) Returns to investment in education: A further update. World Bank Policy
Research Working Paper No. 2881
xxxv World Bank (2005). Education Ethiopia: Strengthening the Foundation for Sustainable Development. Washington DC.
World Bank
xxxvi Tilahun Temesgen (2005), Determinants of Wage Structure and returns to education in Developing Countries: Evidence
from Linked Employer- Employee Manufactiuring Survey Data for Ethiopia, Seoul Journal of Economics 2005, Vol. 18, No. 4.
xxxvii Tilahun Temesgen (2005), Determinants of Wage Structure and returns to education in Developing Countries: Evidence
from Linked Employer- Employee Manufactiuring Survey Data for Ethiopia, Seoul Journal of Economics 2005, Vol. 18, No. 4.
xxxviii Verwimp, P. (1996) Estimating returns to education in off-farm activities in rural Ethiopia. Ethiopian Journal of
Economics 5(2): 27 - 56
xxxix Wodhay, A. (1998) Returns to schooling in Ethiopia. The case of the formal sector. Human Resource Development in
Ethiopia, EEA Addis Ababa
xl Krishnan, Pramila, Tesfaye Gebre Selassie, and Stefan Dercon. The Urban Labour Market During the Structural Adjustment:
Ethiopia 1990-1997. Working Paper 98-9,Center for the Study of African Economies, Oxford University, April 1998.
xli World Bank (2005) Ethiopia Well-being and Poverty in Ethiopia: The role of agriculture and Agency. Washington D.C., World
Bank.
xlii Jimenez, E. and Patrinos, H. (2008) Can cost benefit analysis guide education policy in developing countries? World bank
Policy Research Working Paper No. 4568 Washington DC: World Bank
xliii Jejeebhoy, S. (1995) ‘Women's Education, Autonomy, and Reproductive Behaviour: Experience from Developing
Countries.’ New York: Ox ford University Press
xliv Basu, A., and Stephenson, R. (2005) ‘Low levels of maternal education and the proximate determinants of childhood
mortality: a little learning is not a dangerous thing.’ Social Science & Medicine 60(9): 2,011-2,023.
xlv Streatfield, K., Singarimbun, M., Diamond, I. (1990) ‘Maternal education and child immunization.’ Demography, Vol. 27, No.
3, pp. 447-455
xlvi Moestue, H., Huttly, S. (2007) ‘Adult education and child nutrition: the role of family and community.’ Journal of
Epidemiology and Community Health 2008; 62:153–159.
xlvii . Joint Evaluation of General Budget Support 1994-2004 – Effects Of General Budget Support (2007) IDD, Birmingham.
xlviii Pearman R. (2010), Economic Appraisal on General Budget support, CBA report V.44.doc, DFID
xlix NPV is the net present value. It is the difference between the present value of cash inflows and cash outflows. NPV
compares the value of a dollar today to the value of that same dollar in the future, taking inflation and returns into account. If
the NPV of a prospective project is positive, it should be accepted. However, if NPV is negative, the project should probably
be rejected because cash flows will also be negative. The NPV calculation finds the net present value using a predefined
discount rate.
l IRR is the internal rate of return on an investment. IRR finds the discount rate that makes the NPV equal to zero. The
discount rate is the cost of borrowing or using money for investments. The decision to accept or reject the purchase depends
on the whether the internal rate of return is higher than the discount rate. The decision criteria for these projects is to accept
the project if the IRR is higher than the discount rate or the cost of borrowing.
li World Bank (November 2010) Ethiopia Country Procurement Assessment Draft Report. World Bank, Addis Ababa, Ethiopia
lii EU PEFA (September 2010) Public Expenditure and Financial Accountability Review, Quality Assured by PEFA secretariat.
liii In draft Hawkins and Giorgis (2010) Fiduciary Risk of the Federal Democratic Government of Ethiopia, DFID, Addis Ababa
liv In draft Hawkins and Giorgis (2010) Fiduciary Risk of the Federal Democratic Government of Ethiopia, DFID, Addis Ababa