CIDA’s REVIEW of PROGRAMBASED APPROACHES
Evaluation Directorate
March 2010
OBJECTIVES
1. PRIMARY:
Review PBAs, including budget support, to
assess performance: TBS, March 2007
2. Analyze alternative delivery systems – situate
PBAs within Country Programs (respond to
FedAA req’ts)
3. Review international experience with PBAs
Policy Context for CIDA
• 2002: Strengthening Aid Effectiveness
• 2007: Treasury Board Terms & Conditions –
Review
PBAs’ performance, report March 31/10
• 2008: Operational Guide to PBAs
• 2009: Policy on PBAs
• 2009: Aid Effectiveness Action Plan, 2009 – 2012
• 2009: Report of Auditor General – Implementing aid
effectiveness agenda
What are
What
arePBAs?
PBAs?
• As defined in the Paris
Declaration on Aid Effectiveness
(March 2005), PBAs are:
“a way of engaging in development
cooperation, based on the
principles of co-ordinated
support for a locally owned
programme of development—
such as a national development
strategy, a sector programme, a
thematic programme or a
programme of a specific
organization.”
Programme based approaches share
the following features:
• Leadership by the host country
or organisation;
• A single comprehensive
programme and budget
framework;
• A formalised process for
coordination and harmonisation
of donor procedures for
reporting, budgeting, financial
management and procurement;
and,
• Efforts to increase the use of local
systems for programme design
and implementation, financial
management, monitoring and
evaluation.
Four types
Types of
Four
ofPBAs
PBAs
• Through budget support, a donor
helps to finance the budget of a
partner country by transferring
funds to the national treasury.
• General budget support (GBS) is
a contribution to the recipient
country’s overall budget. It
includes policy dialogue between
donors and the host government,
focused on overall policy and
budget priorities.
• Sector budget support (SBS) is
earmarked to a particular sector,
with policy dialogue on sectoral
issues.
• Pooled Funds (PF) are negotiated
between donors and the host
government, often an account
which is separate from a
government’s budget process. A
Pooled Fund includes some donor
oversight, as reporting is
expected to show how the funds
were used.
• Projects and technical assistance
may be part of a PBA, marked by
efforts to align donors’ assistance
with recipient country priorities
and systems.
METHODOLOGY: 4 PHASES
1. Design methodology & review international
experience (Sept – Dec 2008)
2. Five Country Program Evaluations (Bangladesh,
Ethiopia, Malawi, Mozambique, Vietnam) Plus Ghana,
but not Honduras. (Fall ‘08 – Fall ‘09)
3. OECD-DAC multi-donor Evaluation of Budget Support;
Mali & Tanzania studies (2008 to late 2010)
4. Synthesis Report: policy; CPEs, international
experience (March 2010)
METHODOLOGY: Criteria & Ratings
EIGHT CRITERIA: (DAC/CIDA)
•
Relevance
•
Effectiveness/results
RATINGS: (WB/CIDA)
Evaluators rated PBAs on each
criterion, using 5-band scale:
•
Highly satisfactory (4.1- 5.0)
•
Satisfactory (3.1 to 4.0)
•
Sustainability
•
Coherence
•
Efficiency
•
Moderately satisfactory (2.1 to 3.0)
•
Management principles (Paris Decl)
•
Moderately unsatisfactory (1.1 to
2.0)
•
Cross—cutting issues
•
Highly unsatisfactory ( 0 to 1.0)
•
Performance management /
Monitoring & evaluation
METHODOLOGY:
Limits & Compensation
1. Secondary data from implementing org’ns; common &
accepted condition:
i) interviews: elaborate, check probls
ii) other reviews: triangulate
2. Comparability within & among CPEs:
i) Tested methodology
ii) Ratings: tendencies, not measures
iii) Patterns & similarities: consistency
3. No country in crisis/fragile states; no country of
modest presence. (Malawi status changed.)
CIDA’s PBAs: Budgets & Annual Disbursement
by Mode, 1999-2008 ($ mn)
MODE
Budgs
1999 –
2008
Budg as
% of
Total
Disb
99 /
00
Disb
00 /
01
Disb
01 /
02
Disb
02 /
03
Disb
03 /
04
Disb
04 /
05
Disb
05 /
06
Disb
06 /
07
Disb
07 /
08
Disb
08 /
09
Disb
TOT
GBS
768
16%
1
1
21
48
40
62
57
102
64
74
469
SBS
1,584
32%
0
0
0
13
54
60
71
135
235
237
805
PF
1,461
30%
2
11
15
33
116
60
107
200
157
162
863
Other
1,118
23%
14
8
6
8
31
66
58
84
123
160
557
TOT
4,931
100%
16
19
43
101
242
248
293
521
578
633
2,694
CIDA’s PBAs: Budgets ($ mn)
by Mode & Geographic Branch
Africa
Asia
Americas
Afghanistan/Paki
stan, Iraq
CIDA TOTALS
GBS
439
39
23
268
768
SBS
996
356
54
499
1,584
PF
1,028
137
173
122
1,461
806
264
6
42
1,118
PBA TOTALS
3,268
477
256
931
4,931
Branch PBA Budget as
% of CIDA PBAs
66.3%
9.7%
5.2%
18.9%
100%
Total Branch Pgm
Budget
4,082
2,215
1,906
1,298
9,950
PBAs as % of Branch
Pgm Budget
80%
22%
13%
72%
50%
MODE
Other
Donor Performance:
Using PBAs to deliver aid
Donor
Budget
Support
(US$ mn)
Other PBAs
(US$ mn)
Total PBAs
(US$ mn)
Total Aid
Disbursed
(US$ mn)
PBAs as % of
Aid Disb.,
2007
Canada
265.6
244.3
509.9
914.1
56%
Germany
109.2
214.6
323.9
940.4
34%
Netherlands
437.5
253.1
690.6
969.3
71%
Sweden
181.8
152.0
333.9
718.6
46%
Utd Kingdom
775.6
373.2
1,148.8
1,598.6
72%
United States
59.0
1,554.2
1,613.2
4,391.6
37%
Totals, 23
donors
3,368
5,074
8,441
18,195
46%
Multilaterals’ Performance:
Using PBAs to deliver aid
Multilateral
Org’n.
Budget Support
(US$ mn)
Other PBAs
(US$ mn)
Total PBAs
(US$ mn)
Total Aid
Disbursed (US$
mn)
PBAs as % of
Aid Disb., 2007
Asian Dev. Bank
206.7
108.6
315.3
915.5
34%
IDB
77.9
70.3
148.1
384.1
39%
Afr. Dev. Bank
250.1
128.3
378.4
1182.7
32%
Global Fund
0.0
486.5
486.5
643.5
76%
United Nations
75.0
509.7
584.8
1,713.2
34%
World Bank
2,199.9
1,089.3
3,289.2
5,887.0
56%
Totals, 8
multilaterals
2,812
2 448
5,261
10,898
42%
International Experience of PBAs:
Reviews & Evaluations
Category
1.0 Implementing
PBAs: What
happened?
Responsible Institution & Nature of Review
1.1 OECD/DAC review of implementation of GBS and SWAps, & donors’
harmonization. (2002)
1.2 Evaluation of “Education for All” multi-country, multi-donor PBA
Netherlands Min. of For. Affairs, 2003.
1.3 World Bank review of Poverty Reduction Support Credits, budget
support for countries’ poverty reduction programs. (2006)
2.0 An Evaluation
Framework for
PBAs?
2.1 OECD/DAC: evaluation framework for country-level General Budget
Support. (2004)
2.2 OECD/DAC evaluation of GBS in seven countries, 1994-2004. 19 donors
(incl CIDA), 5 multilaterals, &7 country gvts. (2006)
2.3 EuropeAid & DAC Evaluation Network: a Comprehensive Eval’n Framework
for GBS & SBS. 2008 to late 2010.
3.0 Evaluations of
PBAs
3.1 Evaluations: Ghana multi-donor GBS, 2007; Tanzania, 1995-2005.
3.2 World Bank evaluation: assistance to Ethiopia, 1998-2006.
3.3 Tanzania Independent Monitoring Group: effective aid relationships,
including PBAs. (2005)
CIDA’s Country Program Evaluations:
Samples by Delivery Mode
Delivery
Mode
B1
Bilat Dir
B2
Bilat
Resp
B3
GBS
B4
SBS
B5
PF
P1
P/ship
M1
M/lat
Totals
Country
Program
#
#
#
#
#
#
#
#
Bangladesh
6
16
3
13
1
1
40
Ethiopia
8
4
2
4
18
Malawi
4
1
Mozambique
3
6
2
5
1
2
Vietnam
13
6
2
3
1
6
Totals
34
29
4
11
21
11
2
7
3
22
31
8
118
PBAs by Performance Criteria
1.
Relevance:
 Def’n: PBA objectives fit country priorities; CDPF
strategy; CIDA policies on poverty & sustainability
 PBAs highly satisfactory – as are other delivery modes
 International experience: Relevance includes
conditionality and ownership
 Political & administrative factors shape PBA choices
 Recommend: More analysis to set threshold for
governance criteria (especially for countries in crisis).
ED to pursue Review re: PBAs in fragile states,
countries in crisis & countries of modest presence.
PBAs by Performance Criteria
2. Effectiveness & results
 Def’n: achieve intended objectives; importance of
results for poverty, sectoral objectives.
 PBAs’ average: satisfactory – varying, high to
unsatisfactory PBAs similar to other modes
 Raises issues of intermediate outcomes & impact
 International: Effects of scale – institutional
development. No strong poverty link.
 Recommend: Develop new methodologies and invest
in national capacity to evaluate PBAs’ design &
effectiveness (see Performance Management)
PBAs by Performance Criteria
3. Sustainability
 Def’n: financial, technical, institutional capacity of host
gov’t / organization to sustain PBA results
 Aid dependency—national & sectoral
 PBAs lower than on any other criterion —moderately
satisfactory.
 Similar to other modes.
 International: potential longer-term institutional benefits.
Tanzania: how to end aid dependency?
 Recommend:
 Analyze aid dependency & exit options;
 Invest in national financial-tech-inst’l capacity;
 Broaden “ownership” for political sustainability
PBAs by Performance Criteria
4. Coherence
 Def’n: Consistency of CIDA support for PBAs; external—coordinate PBAs with other actors; internal—PBAs within
country program.
 PBAs: high end of satisfactory—stronger than other modes.
CIDA well regarded
 International: GBS very well co-ordinated with national
priorities; components integrated.
 Internal coherence raises concerns: Bi/Multi/Partnership;
 Recommend:
 External: dialogue on division of labour
 Internal: plan country programs with all channels –
Bilateral / Multilateral / CPB
PBAs by Performance Criteria
5. Efficiency
 Def’n: Convert human & financial resources
economically into results
 Demand and supply sides of transaction costs
 PBAs satisfactory – GBS & PF strongest
 Requires technical / sectoral expertise in field and
decision-making authority (improve corporate services)
 Internat’l: mixed evidence on transaction costs
 Recommend:
 Analyze transaction costs, with counterparts
 Boost technical / sectoral expertise in field
 Pilot faster PBA approvals
PBAs by Performance Criteria
6.
Paris Decl’n Management Principles
 Def’n: PBAs show ownership, harmonization & alignment.
 PBAs satisfactory – similar to other modes: spread effects
of AE agenda.
 CIDA & DAC define PBAs as approaches, not funding
modalities (budget support): Contentious debate
 International: Both definitions used; Paris indicator (66%)
controversial
 Recommend:
 Clarify ambiguities in CIDA’s PBA definition
 Clarify interpretation of PD indicator
 Simplify data categories; one overall dataset (14
categories)
PBAs by Performance Criteria
7. Cross-cutting issues
 Def’n: PBAs address cross-cutting issues—gender
equality & the environment
 PBAs moderate to satisfactory—similar to other
modes
 International: Cross-cutting issues lost in policy
dialogue—not a priority for GBS
 Recommend: Within PBAs, CIDA commit people &
money to issues of priority, especially for early
baseline data.
PBAs by Performance Criteria
8.
Performance Management / Monitoring &
Evaluation
 Def’n: Quality of frameworks for performance
management, monitoring & evaluation
 PBAs average: satisfactory, slightly better than other
modes (major joint review processes).
 International: Evaluation of PD implementation shows
least attention so far, together with mutual accountability
 Recommend: Invest in public capacity to monitor and
evaluate PBAs, and aid in general. Provide new tools to
CIDA staff.
PBAs’ Performance:
Summary Strengths & Weaknesses
• Strongest on relevance to host country priorities—highly
satisfactory (issue of appropriateness) .
• Satisfactory: effectiveness and results; coherence;
efficiency; fit with PD management principles; performance
management, monitoring & evaluation
• Weakest on sustainability—moderately satisfactory. Crosscutting issues: barely satisfactory
• Pooled-Funding: strongest among PBA modalities—high end
of satisfactory
• No single delivery mode shows clear advantage across all
criteria: Balance portfolio for context, maximize strengths &
minimize weaknesses.
PBAs’ Performance: Summary Scores
by Delivery Mode & Criteria
Dlvry
Mode
Count
Relevance
Effect/
Result
Sustain’y
Coherence
Effic’y
PD Mgt
Crosscutt’g
M&E
Avg
B1
34
4.3
3.3
2.7
3.3
2.8
3.4
3.2
3.5
3.3
B2
29
4.3
3.8
2.7
3.5
3.2
3.6
3.4
3.5
3.5
B3
4
4.5
3.3
3.0
4.5
4.0
4.0
3.0
3.8
3.8
B4
11
4.3
3.3
2.6
3.6
3.2
3.5
3.1
2.9
3.3
B5
21
4.5
3.5
3.0
3.6
3.8
4.0
3.3
3.8
3.7
M1
8
4.9
3.6
2.9
3.7
2.9
3.7
3.5
3.2
3.5
P1
11
4.2
3.6
3.5
3.1
3.3
3.5
3.8
3.2
3.5
Avg.
118
4.4
3.5
2.9
3.6
3.3
3.7
3.3
3.4
3.5
Key for Delivery Modes: B-1: Bilateral directive; B-2: Bilateral responsive; B-3:
General Budget Support; B-4: Sector Budget Support; B-5: Pooled Funds; M-1:
Multilateral responsive; P-1: Partnership responsive.
Review of PBAs:
Outstanding Issues
1. Mutual Accountability
 Mandatory Canadian requirements: TBS, FedAA, ODAAA
with PD principles
 International commitments: Broad, complex, to be
defined—ex: policy dialogue
 Recommend: Manual on M & E for mutual accountability;
systematize & share knowledge
2. Civil Society:
 Accra: Recognized CSOs – broad coalition
 CIDA: Experience; leadership for Accra
 Recommend: Strengthen CSOs within PBAs – ref. AEAP
Review of PBAs: Process
• Management response
• Evaluation Committee: Feb. 23
• Treasury Board Sec’t.: by March 31
Recommend:
Organizational learning on PBAs:
 Internal Audit recommendation
 CIDA & counterparts – literature
 Continue DAC-Network