1
WHAT CAN WE LEARN FROM
SUCCESSFUL COUNTRY SYSTEMS?
PHILIPP KRAUSE
PRMPR
Overview
2




Why bother?
A selection of country cases
How do differences matter?
What do “successful” systems have in common?
Why bother?
3




Two schools of thought
 The “best practice” school
 The “It’s all about the context” school
The truth lies somewhere in between – there are important
characteristics shared by successful cases, but differences between
countries matter tremendously
Most important reason: Countries do M&E for different reasons and
different users – what you need depends on what you want to do
with it
Our definition of success:



High degree of utilization
M&E information meets quality standards and is reliable
System is sustainable over time
Overview
4




Why bother?
A selection of country cases
How do differences matter?
What do “successful” systems have in common?
Australia: Main Features
5






Mandate to evaluate each program every 3-5 years
Portfolio Evaluation Plans to be prepared annually for
the following three years
Department of Finance: Steering & quality control
Sector Departments: planning, implementation
Evaluation results primarily used for budgetary
decisions: allocations of funds for new policies and
reallocation of savings (i.e. the discretionary part of
annual budget process)
System lasted from 1987-1996 – sustainability?
Australia: How does it work?
6
Department
of Finance
Department of
Prime Minister
and Cabinet
Citizens
Sector
Departments
(and outrider
agencies)
Parliament
Treasury
Source: Mackay 2011
United Kingdom: Main Features
7





Comprehensive system of performance monitoring and
targets: Spending Reviews (multiannual budgets) and
Public Service Agreements (“resources for delivery”) in
return for (relative) managerial flexibility
Oversight and leadership from central executive (Prime
Minister, PM’s Delivery Unit, Treasury)
No systematic evaluations in the executive government
Problem: “Gaming in Targetworld”
Value-for-Money Audits: 60 per year from National
Audit Office
UK: How does it work?
8
HM Treasury
Negotiations over PSAs
Prime Minister’s Office
Delivery Unit*
is accountable to
Parliament
monitors priority areas
Ministers
is accountable to
* Moved to HM Treasury in 2007, abolished 2010
reports to
Spending Unit
audits
National Audit
Office
Mexico: Main Features
9




M&E of social policies delegated to a specialized
technical agency – CONEVAL
CONEVAL in charge of evaluation portfolio,
development of methods, dissemination
Key oversight decisions by inter-ministerial
committees – system involves many stakeholders
(Congress, Finance, Presidency, Public Admin
Ministry, Sector Ministries, CONEVAL)
Implementation of evaluation and day-to-day
operation of monitoring done by ministries
Mexico: How does it work?
10
1.
Inter-ministerial Committee
2.
CONEVAL
Sector
Ministry
Sector
Ministry
•Coordinate and steer evaluation system
•Maintain quality control
•Define monitoring parameters
3.
Sector
Ministry
Programs are evaluated by
ministries
4.
CONEVAL
Congress, Presidency,
Sector Ministries, Finance
•Define the evaluation plan
•Decide what and how to evaluate
5.
•Select and hire evaluators
•Supervise evaluation implementation
•Monitor indicators and targets
•Collection of evaluation results
•Dissemination of evaluation results
•Utilization of evaluation results
Chile: Main Features
11





System is highly centralized – closely linked to and based on
a highly centralized, top-down budget process
Budget office manages most details, and is main user of
information – utilization fostered by close link to budget
process
Management control by hierarchical oversight
Little buy-in from other (potential) stakeholders
Some impact on allocations and program management
Chile: Utilization of Government Evaluations—2000–05
Effect on
program
Programs
affected
Minor
adjustment
Major adjustment
of program, for
example, improved
processes or
information systems
of management
processes, for example,
changed targeting
criteria, or new
information systems
24%
38%
Percentage of all evaluated programs. Source: Guzman 2007
Substantial
redesign
Institutional
relocation
of program or
organizational
structure
of program
25%
5%
Program
termination
8%
Chile: How does it work?
12
1.
2.
•Define the evaluation plan
•Decide what and how to evaluate
Budget Office
•Coordinate and steer evaluation system
Budget Office
•Maintain quality control
•Define monitoring parameters
•Select and hire evaluators
3. •Supervise evaluation implementation
•Monitor indicators and targets
4.
•Collection of evaluation results
•Dissemination of evaluation results
Programs
Budget
Office
Programs are evaluated directly
by Budget Office
Budget Office
Budget
Office
5.
•Utilization of evaluation results
Congress
Centralization vs. Delegation
13
Mexico
1.
Inter-ministerial Committee
2.
CONEVAL
Sector
Ministry
Sector
Ministry
Sector
Ministry
Programs are evaluated by
ministries
•Define the evaluation plan
•Decide what and how to evaluate
Chile
Budget Office
•Coordinate and steer evaluation system
Budget Office
•Maintain quality control
•Define monitoring parameters
3.
4.
CONEVAL
•Select and hire evaluators
•Supervise evaluation implementation
•Monitor indicators and targets
•Collection of evaluation results
•Dissemination of evaluation results
Budget
Office
Programs
Programs are evaluated directly
by Budget Office
Budget Office
Budget
Office
Congress, Presidency,
Sector Ministries, Finance
5.
•Utilization of evaluation results
Congress
Overview
14




Why bother?
A selection of country cases
How do differences matter?
What do “successful” systems have in common?
Chile
Institutional Differences
Matter: Index of
Legislative Budgetary
Powers
Mexico
Source: Wehner 2007
Difference in Purpose
16
 Budgetary:
To inform budgetary decisions, best
allocation of resources between sectors and programs,
also to enforce operational savings in annual budget
 Accountability: External accountability towards
legislature, stakeholders and public. To make systematic
information on performance available and strengthen
the public evidence base of policy decisions
 Control: To develop better central government
information on implementation and service delivery of
public programs as tools to hold managers to account
Matrix of Country Comparison
17
Country
Purpose
Leadership
Operation
Users
Chile
Budget/Control
Direct
Centralized
Single
Mexico
Budget/Accountability
Delegated
Decentralized
Multiple
Australia
Budget
Direct
Decentralized
Single
Canada
Budget
Direct
Decentralized
Single
UK/NAO*
Accountability
Direct
Centralized
Multiple
* Supreme Auditor
Features that Matter
18





Centralization requires the right institutional structure – a
centralized M&E system design in a fragmented public sector will
fail
In a system with multiple stakeholders delegation to an impartial
agency might be a viable option – but beware of over-engineering
and objective overload
Who is to gain and who has to worry about buying into a M&E
system: The senior civil service, the legislature, ministries, service
delivery units, the finance ministry, the head of government (PM or
President)?
Staying in control of overall steering and quality of outputs does
not equal having to internalize all aspects of M&E implementation –
strategic delegation might be smart for buy-in and workload
Is it possible to imagine a long-term sustainable, well utilized M&E
system that does not have a stable link to budgetary decisions?
Overview
19




Why bother?
A selection of country cases
How do differences matter?
What do “successful” systems have in common?
Lessons Start M&E Systems Successfully*
(1)
20
 Somewhere in government is substantive demand for
M&E information. This is necessary to start and sustain
an M&E system
 Actors need to have incentives to engage with M&E.
They are key for M&E to be conducted and for the
information produced to be utilized
 Simple is better – successful M&E systems tend to
deliver just what users want, not more. They also serve
only those objectives that result in utilization
 Success is more likely with a powerful champion(s) to
lead the push for institutionalization of M&E – rather
than a legislative or technical exercise
*very liberally adapted from Mackay 2010
Lessons Start M&E Systems Successfully (2)
21
 It is important to have the stewardship of a
central, capable ministry that can design,
develop, and manage the system
 Some reforms may start with a bang, but it
requires patience, determination, and a long-term
effort to build an effective M&E system
 For donors: It helps the process to start with a
diagnosis of what M&E functions already exist in
the country (and why other M&E functions do not
exist – they usually don’t for a reason)
22
THANKS!
For further information, and to access the sources cited
here, please visit:
http://go.worldbank.org/7MZRWD6K50