Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Using Evaluation
to strengthen
results orientation and reporting
with a focus on fragile contexts
- from a donor’s perspective
Mathias Rickli
Swiss Agency for Development and Cooepration
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Framework of Cooperation
International Framework: MDG, Paris Declaration, AAA, MfDR
Switzerland: Federal Constitution,
Law and Message 2013-16
Cooperation Strategies
•Time frame: 4 years
•Indicative budget according to
Federal Council Dispatch
•Monitoring Framework
Partner Country: National Strategies
and Priorities
Projects / Programmes
•Planning, monitoring and reporting according
to PCM, credit proposal
Intenal evaluations
• Project / programme evaluations
• Beneficiary assessments
Intenal evaluations
Annual Reporting
• Peer process
and Planning
Independent evaluation, joint evaluations
• Sector program evaluation, reports on effectiveness
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
2
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Framework of Cooperation
Message on International Cooperation 2013-16
Humanitarian Aid
1. reduced human suffering
(natural disasters,
conflicts)/protection of
civilians
(conflicts/violence)
2. DRR, contributing to
national capacities and
reduction of future
human losses
3. strengthening the
humanitarian multilateral
system (reform
processes, coordination,
performance)
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
3
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Framework of Cooperation (continued)
Message on International Cooperation 2013-16
Regional cooperation:
4. sustainable poverty
reduction (basic
social services,
employment, access
to resources and
political decision)
5. reducing fragility and
its causes, improved
living conditions of
population based on
human rights
Global cooperation:
ca. 20% increase of
6. contributing to improved international frameworks
resources
in issues
fragile
for dealing jointly
with global
(clima, water,
contexts
food security, migration,
health) based on
innovation and policy influencing
7. supporting the multilateral system regarding
improved performance of specific MO and their
contribution to MDG and other international goals
Cooperation with East of Europe, Central Asia:
8. support to sustainable transition towards
democratic and free market systems along specific
sector reforms
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
4
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Purpose of evaluation function within SDC
steering
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
5
Last update: 12.06.2013
Rolling planning: centralized evaluation function
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Nothilfe
(RVL)
Thematic sector
program evaluatiosn
(1-2 / year)
Reports on
Effectiveness
(biennial)
Gouvernanz
(BSL)
IKRK (offen)
BerufsBildung (RIK)
PPDP
( RIK)
Klima
( EKM)
gem. rollender Planung
Landwirtschaft
( RIK)
Gesundheit
Bangladesh
Pilot (RVL)
Country programme
evaluations
(2 / year)
Case 2
Ressortforschung (BSL)
Tanzania
( RVL)
Serbien
( RIK/RVL)
Fragile
Staaten (BSL)
Personal
(EKM)
Beobachtungsperiode
2006-10 (SIE)
+ 100 decentralized internal
project evaluations / year
Mali
(geplant)
Naher Osten Zentralasien
( SHE)
(geplant)
Nachhaltigkeit
EFK (RIK)
Results report on
Message
gem. roll.
Planung
M4P (geplant)
Case 1
institutional
evaluations
(1 / year)
gem. roll.
Planung
Migration joint EDA,
BFM, VBS (geplant)
Kommunik.
(offen)
g. roll. Plan.
g. roll. Plan.
g. roll. Plan.
g. roll. Plan.
g. roll. Plan.
g. roll. Plan.
Globalprog.
(geplant)
Netzwerke
( SHE)
gem. roll.
Planung
gem. roll.
Planung
Case 3
Beobachtungsperiode
2011-15 (SOM/MTD )
2012
2013
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
2014
heute
2015
2016
2017
2018
Message periods
Time
6
Federal Department of Foreign Affairs FDFA
Case 1
Swiss Agency for Development and Cooperation
Staff of Directorate
Reports on Effectiveness
Purpose and set-up
 Information of the public and member of parliament
focus on result achievements of Swiss international
cooperation
promote Swiss international cooperation as transparent
and effective instrument of Swiss foreign policy
 Contribute to institutional learning within SDC
 Thematic sector programme evaluations
-> focus on effectiveness
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
7
Case 1
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Report on Effectiveness Agriculture 2010
Portfolio selection
20
5
23
1
14
21
19
12
22
8
4
13
3
17
11
15
16
18
6
10
9
field study
Random selection of 27 intervention
desk study
incl. 2 global and 4 multilateral interventions
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
8
Case 1
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Report on Effectiveness Agriculture 2010
Ex-post results framework
(according to Mayne, 1999)
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
9
Federal Department of Foreign Affairs FDFA
Case 1
Swiss Agency for Development and Cooperation
Staff of Directorate
Report on Effectiveness Agriculture 2010
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
Source: http://www.deza.admin.ch/de/Home/Wirksamkeit/ressources/resource_en_197093.pdf
10
Federal Department of Foreign Affairs FDFA
Case 1
Swiss Agency for Development and Cooperation
Staff of Directorate
Use of evaluation Case 1:
Report on Effectiveness Agriculture 2010
 Well received by parliament, however few reactions
----------------------------------- Improve quality of and access to project evaluations
 Foresee ways to collect and aggregate qualitative and quantitative
results meaningfully
=> introduce standard indicators for aggregation where feasible
 Be clear on your main target audience (public, parliamentarians)
 Report on Effectiveness is an accountability tool, do not expect
institutional learning
 Foresee clear communication concept
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
11
independent evaluation:
Federal Department of Foreign Affairs FDFA
Case 2
Swiss Agency for Development and Cooperation
Staff of Directorate
Lewis Sida (team leader),
Frauke de Weijer,
William Frej,
Ross Mountain,
Ben Ramalingam
May 2012
Methodology:
Interviews including SDC, FDFA,
SECO, ICRC, UN, NGOs, national
government and other donors
Desk reviews for Nepal, Hindu Kush,
OPT, Haiti, Global – 20 interviews
Institutional evaluation
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
Case studies Tajikistan and Great
Lakes
12
Federal Department of Foreign Affairs FDFA
Case 2
Swiss Agency for Development and Cooperation
Staff of Directorate
Main findings
 SDC well positioned to work in fragile states
 right mixture of approaches, good instincts operationally
 can play an important role in the wider international system
 individual projects and programmes can perform well, and
achieve clear results
 little evidence that they make a difference to the root causes of
fragility.
=> to achieve this potential
 needs to become more flexible
 needs to compliment its technical efforts with political
understanding and positioning.
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
13
Federal Department of Foreign Affairs FDFA
Case 2
Swiss Agency for Development and Cooperation
Staff of Directorate
Measures induced by the evaluation
Implementation plan for fragile states
 Programming: Country strategies include clear outcomes
addressing causes of fragility and conflict, hypothesis of
change, scenarios, CSPM, risk management strategy
 Cooperation: integrated whole-of-government strategies,
coordination with other donors – New Deal. Trust Fund to support
joint UN and WB
 Human resources: SDCs staffing rules and conditions allow for
SDC to have enough and available competent staff
 Security: security strategy with clear roles and responsibilities for
the whole foreign ministry developed by end of 2013
 Communication: SDC communicates the rationale and goals for
engaging in fragile states, how we work with internal and external
stakeholders, successes and problems
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
14
Federal Department of Foreign Affairs FDFA
Case 2
Swiss Agency for Development and Cooperation
Staff of Directorate
Use of evaluation Case 2:
Institutional evaluation
 Engaging more in fragile contexts requires institutional fitness
to do so
=> promote culture of internal evaluation focusing on verifying the
theory of change
---------------------------------- Timely supporting ongoing internal discussion
 High involvement of SDC stakeholders (participative process)
 Convincing evaluation team
=> Strong influence on institutional steering and learning
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
15
Federal Department of Foreign Affairs FDFA
Case 3
Swiss Agency for Development and Cooperation
Staff of Directorate
Monitoring through annual reporting
Cooperation strategy Afghanistan 2012-14
-> extract from result framework
Domain of intervention: Resilience of Rural Livelihood (Provinces: Bamyan, Samangan and Takhar)
Domain goal: Improve the resilience of rural population and especially women to natural and conflict related
stress through improved water management, economic activities and basic education.
(1) Swiss portfolio
outcomes
(2) Contribution of Swiss
Programme
(3) Country development or
humanitarian outcomes
Outcome statement 1:
Education
An increasing share of children and
young women benefit from a
sustainably improved learning
environment and better quality of
education with particular emphasis
on increasing access of girls to
school.
Indicators:
• 70% of schools have a good level of
sustainability of an education friendly
environment through community
ownership and quality assurance by the
Ministry of Education
1 area based and 1 national level
project achieve the following:
Key results of projects:
• Parent/teacher associations are set-up
in 80% of the targeted locations and
become functional.
• The implementation of a
comprehensive gender strategy
increases gradually the
access of girls to basic education
Outcome statement 1
Provide all school-age children in
Afghanistan with equitable access,
without discrimination, to
quality education
Indicators:
• MoE provides adequate funding to
sustain teacher salaries and school
infrastructure
• Girls enrolment in primary and
secondary schools reaches 7.8 million
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
16
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Results Monitoring and Reporting Message 2013-16
SDC is reporting..
2012
Reporting to
Politics
2014
Reporting to
Public
Annual Report SDC-SECO
Communication
Parl. Commissions
Monitoring Finance,
Personnel,
Efficiency
(multiple sources)
Departments
Results; Personnel; Finance;
Plg./Expenditure; Efficiency
Results Report
Outcome Goals Parl. Bill
(Pilot 2012)
Results from Annual
Reports Countries
Global Programmes,
CCM‘s
OMR
2016
FPR 2014
Dashboard
FPR 2015
Accountability
Report 2012-15
FPR 2016
Parliam. Bill 2017-20
AR
Comm.
......
Results Report
2012-15
SDC - SECO
....
......
......
Management Report
Strategic Steering
Management Goals
Divisions,
Cooperation
Offices
2015
Foreign Policy Report 2013
Dashboard (MFA)
Board of
Directors,
Staff
SDC is managing..
2013
PortfolioProject Cycle Mgt.
Results
Report2013
Results
Report 2014
......
Contributes to ..
Key source for ..
Internal Reports
Published Reports
Office Management
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
17
Federal Department of Foreign Affairs FDFA
Case 3
Swiss Agency for Development and Cooperation
Staff of Directorate
Use of evaluation Case 3:
Results Monitoring of cooperation strategies and
Reporting on Message 2013-16
 Result-based Management RBM (log frame)
 Link monitoring, internal evaluation and country strategy
evaluations to verify the results framework
-------------------------------------- Focus on Managing for Development Results in-country
 Trade-off for accountability purpose by reporting on Message
 Challenge:
Robustness of Theory of Chain to be strengthened through good
quality project evaluation
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
18
Federal Department of Foreign Affairs FDFA
Swiss Agency for Development and Cooperation
Staff of Directorate
Ways forward
 Assure good use of evaluation by considering
 timeliness
 ownership of evaluee and
 necessary robust enough methodology
 Introduce theories of changes consequently in fragile context
programming
 Strengthen evaluation capacities in-country among SDC staff
and partners
=> Continue to support evaluation capacity building facilities
such as
IPDET 2013 – Speaker’s Lunch - June 28 - Mathias Rickli
19