ODE Work Plan: 2016
The Office of Development Effectiveness (ODE) undertakes independent evaluations of the Australian aid program.
Its work is overseen by the Independent Evaluation Committee (IEC), an external advisory body that endorses the
ODE work plan and provides quality assurance of ODE’s evaluations and other products. The IEC strengthens the
quality, credibility, influence and independence of ODE’s work through the provision of independent strategic and
technical advice.
This document is ODE’s work plan for calendar year 2016 and is divided into four key areas: (1) strategic
evaluations; (2) operational evaluations; (3) performance and quality analysis; and (4) evaluation policy
development and advisory support. It reflects ODE’s expanded focus on operational evaluations of specific
investments and a reduction in larger scale strategic evaluations, an approach approved by DFAT’s Development
Policy Committee (DPC) in May 2015. It also reflects ODE’s role in supporting DFAT’s internal performance
assessment and evaluation processes, and in building evaluation capacity within the Australian aid program.
The work plan includes activities which are currently underway, evaluations which are expected to commence in
2016, and a reserve list of evaluation topics which may be progressed in 2016 or 2017. The priority of these
reserve topics will be reviewed against emerging priorities when formulating the 2017 work plan.
The following principles guide the selection of evaluation topics:
a. Policy relevance: assessing progress on implementation of the Australian Government’s policy priorities.
b. Potential learning benefits: focusing on areas where there is a strong demand for insight, information or
guidance.
c. Coverage and materiality: ODE’s evaluations should provide appropriate coverage of expenditure by sector,
theme, country and region, over time.
d. Risks to aid effectiveness: examination of issues which pose a challenge to the Australian aid program and
its reputation.
ODE’s involvement in operational evaluations is guided by the above principles in addition to the following
considerations:
a. Readiness of program areas to collaborate with ODE.
b. Clarity of evaluation purpose: accountability, learning or both.
c. Feasibility of the evaluation timing and the availability of human and financial resources.
d. Opportunity to model and promote ‘good evaluation practice’ within DFAT.
1. Strategic evaluations
Strategic evaluations are broad assessments of Australian aid that focus on key policy directions, specific
development themes and sectors, or large programs. These evaluations are independently initiated, managed and
conducted by ODE. They typically examine a number of investments, often across multiple countries, regions or
sectors. By virtue of their scope, strategic evaluations tend to be time and resource intensive.
Strategic evaluations help provide independent oversight of Australian aid. They inform change in both strategy
setting and operational practice and they assist in identifying new and innovative ways to deliver Australian aid.
www.ode.dfat.gov.au
1
Strategic evaluations already underway
Evaluation
Description
Trade facilitation
ODE is conducting an evaluation of the 'trade policy and regulation' element of aid
for trade. The Government's aid policy includes a target of increasing aid for trade
investments to at least 20 per cent of Australia's annual aid expenditure by 2020.
The purpose of this evaluation is to assess the effectiveness of DFAT’s investments
in this area and to elicit lessons that will inform the design and implementation of
future activities. A range of DFAT investments focussing on improving trade policy
and regulation are being assessed.
Investing in teachers
Education continues to be a significant sector for the aid program. In 2015-16
around 19.4 per cent of Australian aid will be spent on education, with a key target
being to improve learning outcomes for 20 million children. This evaluation is
analysing the relationship between Australian assistance for teacher development
and the expected outcomes of improved teacher effectiveness and student learning.
The findings are expected to contribute to refinement of the education performance
assessment framework, and to strengthen evidence of the causal links between
improved teacher quality and better learning outcomes.
Strategic evaluations expected to commence in 2016
Evaluation
Description
Influence of Australian
aid on teacher
development
The ODE report Investing in Teachers (forthcoming) has confirmed DFAT needs to
improve monitoring and evaluation (M&E) of investments in teacher development, in
particular to obtain evidence of aid intervention effects on teacher knowledge and
practices, and pupils' learning outcomes. In this new evaluation (2016–2019), ODE
will work with up to three interested programs to conduct high quality evaluations,
building on M&E strengths and addressing weaknesses identified in the Investing in
Teachers report. Detailed evaluation designs will be developed in collaboration with
program managers, implementing partners and education ministries.
Disability-inclusive
development
Australia is an international leader in disability-inclusive development. Enabling
people with disabilities to find pathways out of poverty and realise their full potential
is integral to achieving the purpose of the aid program. The new ‘Development for
All’ strategy which was launched in 2015 identifies key opportunities to strengthen
disability-inclusive development. The evaluation will identify lessons to help ensure
the new strategy is successfully implemented.
Combatting pandemics &
emerging infectious
diseases in Asia and the
Pacific
Strengthening regional preparedness and capacity to respond to emerging health
threats that cross national borders is a strategic priority for DFAT’s Health for
Development Strategy 2015-2020. Asia is an epicentre for emerging zoonotic
diseases such as Severe Acute Respiratory Syndrome (SARS) and pandemic
influenza, which are derived from animals. From 2006-2015, the Australian
Government invested more than $200 million in programs to combat emerging
disease threats in the region. ODE’s evaluation will help to build the evidence base
to support implementation of a new regional health security strategy and investment
plan, both currently being developed.
Economic partnerships in
Asia and contribution to
partner government
economic reforms
A number of programs in South-East Asia are transitioning to economic partnerships
involving Australia’s diplomatic, trade and aid resources. There is much to be learnt
about how we can engage effectively in partner government policy and budget cycles
and about the effectiveness of other tools which can be used in policy reform. In
www.ode.dfat.gov.au
2
Evaluation
Description
particular, the proposed evaluation will focus on how DFAT monitors and
communicates its contribution to partner government economic policy reforms.
Electoral systems
strengthening
Australia has a long history of support for electoral systems reform and
administration of elections, including in Indonesia, Timor-Leste, Papua New Guinea
and the Pacific. A multi–country evaluation of this support is proposed which would
draw on existing reviews and evaluations of this assistance and consider lessons,
knowledge and experience which can be shared between Australian country
programs.
Reserve topics for 2016 and 2017
Evaluation
Description
Effective facilities for
effective aid (2016)
Significant aid funds are being expensed through ‘facilities’, broadly defined as an
overarching program run by a service provider which manages smaller,
complementary activities on DFAT’s behalf. This evaluation would examine the
effectiveness and efficiency of facilities in delivering outcomes. A better
understanding of the features of effective facilities will contribute to the design of
better, more effective facilities in future.
Technical & vocational
education & training
(2016)
A number of DFAT country programs are investing in skills training systems to
improve labour market outcomes and contribute to economic growth. This
evaluation would consider the degree to which Australian Government partnerships
are supporting quality assured and inclusive skills training systems that enable
participation, including by women, the poorest, people with disabilities; deliver
training which meets the needs of employers and employees; and are relevant and
appropriate to the economic context.
Climate change (2017)
In November 2015, the Australian government announced a commitment of $1
billion over five years to help Pacific island countries build resilience against climate
change and cut emissions. The effectiveness of climate change initiatives funded
over the last five years will be evaluated to identify lessons to inform the design and
implementation of future investments.
Private sector (2017)
A Strategy for Australia’s Aid Investments in Private Sector Development was
released in November 2015. The strategy reflects the Government’s priority of
growing the size and inclusiveness of the private sector in partner countries in order
to drive economic growth and reduce poverty. Its objectives are to build better
business and investment environments; support growth in specific markets; and
maximise the development impact of businesses. An ODE evaluation would examine
the effectiveness of initiatives funded and the extent to which the strategy’s
objectives are being achieved.
Health systems
strengthening in the
Pacific (2017)
Most of the constraints to health systems development in the Pacific relate to
upstream issues of financing and good governance. Two recent bilateral evaluations
have found significant scope for closer working between DFAT’s health and
governance programs. This evaluation would potentially look at the extent to which
Australia’s health investments in the Pacific are coordinated with investments in
governance and Public Financial Management in the Pacific.
Capacity building
programs including
contribution to women’s
leadership (2017)
A significant proportion of Australia’s aid program supports capacity building and
technical assistance. The evaluation would examine the effectiveness of different
approaches to capacity building including the extent to which selected programs
have supported improvements in women’s participation in decision-making and
women’s leadership. This would inform design and implementation of future
capacity building programs.
www.ode.dfat.gov.au
3
Evaluation
Description
Innovation (2017)
DFAT’s approach to innovation is to both promote a more innovative institutional
culture and to fund innovative aid investments. Strategies for encouraging
innovation include funding a range of innovative investments through the
InnovationXchange with a view to scaling up successful approaches, and
encouraging country programs to think more about innovative approaches. The
proposed evaluation would consider the influence of the innovation agenda on the
effectiveness of the aid program.
2. Operational evaluations
Operational evaluations are independent assessments of the performance of individual Australian aid investments.
They provide evidence to inform program management decisions, contribute to broader learning and help to
enhance the accountability of Australian aid. Operational evaluations are generally commissioned and managed by
program areas and may be conducted at whichever point in time is of most value from a management perspective.
Consistent with its role in providing expanded support for operational evaluations, and the selection principles
outlined above, ODE sometimes funds and leads or otherwise participates in a limited number of operational
evaluations.
Operational evaluations already underway
Evaluation
Description
Empowering Indonesian
Women for Poverty
Reduction
The Empowering Indonesian Women for Poverty Reduction program (MAMPU) aims
to improve the lives of poor Indonesian women through increasing their access to
jobs and removing workplace discrimination; improving women’s access to
government social protection programs; and improving conditions for women’s
overseas labour migration. It is also working to strengthen women’s leadership for
better maternal and reproductive health and to reduce violence against women.
ODE is participating in a mid-term evaluation and providing independent verification
of the achievements of the program. The findings from the evaluation will be used to
demonstrate MAMPU’s achievements to date, inform the second phase of the
program, and provide lessons to all DFAT staff on gender equality programs.
Eastern Indonesia
National Roads
Improvement Program
Australia is committed to tackling infrastructure bottlenecks in the region to help
create the right conditions for the private sector and to expand trade. Increasing the
aid program’s infrastructure portfolio over the coming years is a key plank of the
Government’s commitment to scale-up the aid for trade portfolio. This evaluation of
the Eastern Indonesia National Roads Improvement Program (a $330 million
investment over 10 years) is one of two operational evaluations of the economic
infrastructure sector endorsed by the IEC. These evaluations include some common
questions to facilitate generalizability of findings for the sector.
Vanuatu Cyclone PAM
emergency response
Cyclone PAM hit Vanuatu in March 2015. DFAT has provided $15 million in
emergency humanitarian and early recovery assistance to Vanuatu in 2015 and has
committed $35 million to early and long term recovery. Humanitarian Response
Branch (HRB) has asked ODE to lead on evaluating the $10 million initial emergency
response funding.
Australia’s partnership
with the Secretariat of
the Pacific Community
The Pacific region is a high priority for the aid program with estimated funding of
$1.12 billion in 2015-16. It is the only region not to have had budget reductions.
Along with the operational evaluations of Australia’s partnerships with the University
of the South Pacific and the Secretariat to the Pacific Islands Forum in 2016, this
evaluation is focussing on a key Pacific regional organisation which is central to
Australia’s current and future engagement in the region.
www.ode.dfat.gov.au
4
Operational evaluations expected to commence 2016
Evaluation
Description
PNG Transport Sector
Support Program
This evaluation of the PNG Transport Sector Support Program Phase 2
(a$400 million investment over 6 years) is one of two operational evaluations of the
economic infrastructure sector endorsed by the IEC. Substantive work on these
evaluations will be undertaken in the first half of 2016 and will include some
common questions to facilitate generalizability of findings for the sector.
Australia’s partnerships
with the University of the
South Pacific and the
Pacific Islands Forum
Secretariat
Along with the operational evaluation of the Secretariat of the Pacific Community,
these two operational evaluations will focus on Australia’s partnerships with key
Pacific regional organisations which are central to Australia’s current and future
Pacific engagement.
Other operational evaluation topics under discussion with program owners
Evaluation
Description
Civil Society Water
Sanitation and Hygiene
Fund
The Civil Society Water, Sanitation and Hygiene (WASH) Fund is a global grant
program with funding of $102.6 million over 7 years (2011-2018) which provides
grants to civil society organisations to deliver water, sanitation and hygiene
activities. It aims to reach 1.8 million people across Africa, Asia and the Pacific. A
contracted Monitoring, Evaluation and Review Panel (MERP) provides ongoing
monitoring and a mid-term review is planned for 2016. As part of this review, there
is a need for an independent evaluator to consider the Fund’s efficiency in terms of
management arrangements and functioning of the MERP.
3. Performance and quality analysis
ODE’s performance and quality analysis (PQA) work monitors DFAT’s aid performance management and reporting
systems on an annual basis, and independently quality assures the assessments they produce. These are
undertaken within set timeframes, while other activities are one-offs or undertaken within a timeframe controlled
by ODE. There are a number of components to this work, as indicated below.
This work also contributes to ODE and the IEC’s role in quality assurance of the annual Performance of Australian
Aid Report.
Performance and quality analysis work underway
Evaluation
Description
Review of Aid Program
Performance Reports
(APPRs)
ODE assesses the quality of annual APPRs which report on the performance of
country and regional aid programs, highlighting areas of strength and opportunities
for improvement. APPR quality reviews have been carried out every year since APPRs
were introduced in 2008 and have been published on the ODE website since 2013.
Annual spot check of Aid ODE quality assures DFAT’s self-assessments of the performance of individual aid
Quality Checks (AQCs)
investments, which are made through annual AQCs. This quality assurance work
involves checking a statistically significant sample of AQC reports each year, to
assess whether the quality ratings within them are robust.
Review of uptake of ODE The management responses to ODE’s strategic evaluations are cleared at a high
recommendations
level and become a public commitment to implement the accepted
recommendations. ODE reviews DFAT’s management responses regarding
evaluation recommendations around twelve to eighteen months after each
www.ode.dfat.gov.au
5
Evaluation
Description
evaluation and reports on progress to the IEC and the DFAT Executive.
Review of Operational
Evaluations
This review seeks to better understand the practices related to, and the quality of,
independent operational evaluations across DFAT and how these have changed
since 2012. It will provide information to support good quality, independent
evaluations across the department, and it will promote better use of evaluations
across the department and the aid community by facilitating opportunities for
learning.
New performance and quality analysis for 2016
Evaluation
DFAT’s approach to
investment monitoring
Description
ODE will make a detailed assessment of investment monitoring and evaluation
which has been consistently identified as a weakness in DFAT aid investments
(through AQCs since 2008). This study will explore the different types of aid
monitoring, and assess the quality of monitoring systems used across the aid
programs. Factors enabling and/or inhibiting high quality monitoring of
investments will be identified.
4. Evaluation policy development and advisory support
ODE acts as a support service to DFAT staff managing evaluations, either by providing advice; reviewing evaluation
terms of reference, plans and reports; participating in reference groups; or connecting program areas to resources
that support capacity. Key elements of this work for 2016 are shown below.
» Update of DFAT evaluation policy and associated guidance documents.
» Provision of advisory support to program areas.
» Delivery of specialist training on evaluations methods and approaches.
» Corporate training on DFAT monitoring and evaluation – policies, guidelines and implementation.
www.ode.dfat.gov.au
6
5. Published evaluation reports
The following table identifies evaluation reports published in 2015.
Evaluation report
Publication Date
Evaluation of the contribution of Australia Award Scholarships to women’s leadership
December 2015
Banking our aid: Australia’s non-core funding to the Asian Development Bank and World
Bank
September 2015
Evaluation of the Australian NGO Cooperation Program
August 2015
Evaluation of the Australia-Vietnam Country Strategy 2010-15
May 2015
A window of opportunity: Australian aid and child undernutrition
April 2015
Evaluation of Australia’s humanitarian response to the Syria Crisis
March 2015
Research for better aid: an evaluation of DFAT’s investments
February 2015
Working in decentralised service systems: challenges and choices for the Australian aid
program
January 2015
www.ode.dfat.gov.au
7