Preconditions and institutionalising evaluation
Evaluation for Policymaking
Suzhou, China, October 2014
Material prepared by:
Professor Patricia Rogers
RMIT University (Royal Melbourne Institute of Technology)
Australia
Patricia.Rogers@rmit.edu.au
Overview
• What we mean by evaluation capacity
• Strategies for developing evaluation capacity in a single department or agency
• Strategies for developing whole of government strategies
2
A common mental map of evaluation capacity
Skills
to enable
individuals
organisations
to
design
discrete evaluations
conduct
monitoring
More than skills are needed
To use a metaphor.
Think of a water tank whose purpose is to store water for irrigation purposes.
The capacity for irrigating the land is stored in that tank. The fuller it is the
more capacity it has for providing sustenance.
However, the ability to irrigate successfully depends on more than the storage
of water. It depends on climatic conditions, historical rainfall, the presence of
something to turn the tap on and off at the right moment, the quality of the tap
and water distribution networks.
That's what affects the capability of that amount of water to achieve its
objectives.
Bob Williams 2001 Building Evaluation Capability
http://www.bobwilliams.co.nz/Works_in_Progress_files/capability%233.pdf
4
Capacity or capability?
All the skills, knowledge, technical expertise and
experience in the world won't help … if the … program,
community, organisation or environment cannot sustain
and nurture those skills, and abilities.
What I generally see the moment, is capacity building
without a great deal of capability building. If this
continues, I believe we are in danger of being all dressed
up and nowhere to go.
Williams (2002)
More than just doing evaluations is needed
“You can lead a horse to water –
but you can’t make it drink”
[Proverb]`
An enlarged map of evaluation capacity
Skills
Resources
individuals
Structures
Processes
Incentives
to enable
organisations
public sector
systems
Culture
communities
to
frame
design
discrete evaluations
conduct
ongoing evaluative
activity
use
manage
monitoring
Evaluation capacity at the department or agency level
What would you expect to see
in a department with strong evaluation capacity?
What would you expect to see in a department with strong
evaluation capacity?
• Demonstrated use of evaluative activity to improve performance – either
through fixing problems, strengthening good practice, or better identifying
need
• Evaluation not seen as a separate adjunct, but integrated into the core
work, and seen as useful in achieving Key Result Areas.
• Evaluation not just used for justification
• And….?
Davidson (2001) Mainstreaming Evaluation into an Organization’s ‘Learning Culture’
Davidson (2001) Mainstreaming Evaluation into an Organization’s ‘Learning Culture’
Davidson (2001) Mainstreaming Evaluation into an Organization’s ‘Learning Culture’
Elements needed for behaviour change
Michie et al (2011) The COM-B system - a framework for understanding behaviour
13
. Implementation Science 2011 6:42
EXAMPLE: Staged approach in Victorian Department of
Natural Resources and Environment (Agriculture)
1.
Working with one project to respond to a demand for increased
evaluation (18 months)
One internal evaluator largely using existing data
Evaluation and re-planning
2.
Experimenting with 10 volunteer projects (18 months)
2 internal evaluators and consultants using different methods
Evaluation and re-planning
3.
Mandatory evaluation for all new projects (1 year)
Formal training by internal evaluators and consultants
Evaluation and re-planning
4.
Expansion and consolidation (ongoing)
Recommendations from the Victorian Agriculture example
1.
Address all types of evaluative activity – discrete evaluations,
monitoring and ongoing evaluation
2.
Strengthen and build different types of capacity – human capital
(skills), social capital (supportive networks), organisational capital (shared
resources)
3.
Develop common frameworks, structures and processes but allow
flexibility
4.
Address both capacity and opportunity to apply that capacity
(supply and demand)
5.
Build knowledge about what works in evaluation within your
context
6.
Systematically and visibly evaluate each stage.
7.
Think big, but start small, using a staged approach
McDonald, Rogers & Kefford (2003) Teaching People to Fish? Building the
Evaluation Capability of Public Sector Organizations. Evaluation. 9 (1), 5-25.
Impact of building capability for evaluative activity
“.”
It effected cultural change.
Only now it gelled with me that I operate differently,
Now we can question what we do as a matter of course.
We have an atmosphere of critique.
We are actively seeking how we can do better.
We didn’t have that before
Program manager, Victorian Agriculture
Possibly transferable lessons from capacity building programs
• Address different types of capital – including
–human – skills, knowledge
–economic – money and other resources
–social – norms of trust and reciprocity
–institutional – organisational structures and processes
• Take a strengths-based approach – identifying and
building on existing capital
• Also develop opportunities to use the capacity
How can a central agency assist organisations to
develop their capability for evaluative activity?
• Resources – guidelines, examples, co-ordination and facilitation of training
• Co-ordination and support
• Setting standards and guidelines
• Incentives for evaluative activity
Example of assessment of country context: the Philippines
Tungpalan, R. (2013) The National Evaluation Policy Framework: Engendering an
Evaluation Culture in the Philippines. 3rd M & E Network Forum.
19
Example of national evaluation policy: the Philippines
Tungpalan, R. (2013) The National Evaluation Policy Framework: Engendering an
Evaluation Culture in the Philippines. 3rd M & E Network Forum.
20
Elements of South Africa approach – embedding in planning
Element
Agencies involved
Details
Legal & constitutional
mandates for M & E
Dept Performance M&E, Treasury, Dept Public
Service & Admin, Auditor-General, Dept Cooperative Governance, Statistics SA, Public
Service Commission
p.6
Evaluation units in the tiers
of government
Office of the Presidency, Office of the Premier in
each province, local government offices
p.8
Focus on 12 outcomes
Performance agreements with Cabinet ministers,
implementation forums with delivery agreements,
public reporting of Programme of Action
p.9
Five year strategic plans
and annual performance
plans
Treasury developed system, national and
provincial departments use
p.9
Management Performance
Assessment Tool
Annual self-assessment with evidence by head of
department and senior management in national
and provincial departments
p.10
Management Response and DPME, relevant Department, Cabinet Committee
Improvement Plan
and Cabinet
p.16
21
Elements of South Africa approach – quality of service delivery
Element
Agencies involved
Details
Frontline Service Delivery
Monitoring
Unannounced visits to government service sites
(eg schools, clinics, police stations) by DPME
staff and M & E staff in Office of the Premier
p.11
Citizen Based Monitoring
Piloting methods to bring views of citizens into M
&E
p.11
Presidential Hotline
14,000 calls per month logged and followed up
with departments and provinces
p. 12
Municipal assessment tool
Covers service delivery, and also planning,
human resources, community engagement and
governance
p. 13
22
Elements of South Africa approach – National Evaluation system
Element
Agencies involved
Details
National Evaluation Policy
Developed by DPME
p.12
Evaluation guidelines
Developed by DPME for 6 different types of
evaluation – diagnostic, design, implementation,
impact, economic, synthesis
p.12
23
Elements of South Africa approach – Professional Capacity for M & E
Element
Agencies involved
Details
Post-graduate evaluation
degrees
Universities, with support from World Bank and
DfID; CLEAR centre
p.14
M & E courses for
government managers
DPME and Public Administration Leadership and
Management Academy (PALAMA)
p. 14
In-service courses
DPME and CLEAR
p. 14
Co-operation with national
evaluation association
MoU between DPME and South African
Monitoring & Evaluation Association (SAMEA)
p.15
National M & E forum
DPME and Heads of M & E in national
departments
p.15
Provincial M & E forum
DPME and Heads of M & E in Office of the
Premier
p.15
Repository of evaluation
reports
DPME – 83 previous evaluations added (after
quality) review
p.15
24
Map of South African national M & E system
25
Mapping a national M& E system
Ministry of
Finance
Dept of Prime
Minister’s
Office
State Audit
Office/Financial Audit
Dept/ Auditor-General
State Budget
and Strategic
Planning Dept
Public
Service
Commission
Dept of Public
Service
Administration
Ministry of National
Economy/ Econmiic
Development/Economy &
Industry
Bureau of
Statistics
Ministry of
Energy/Energy
& Water
Resources
Universities
and Think
Tanks
NGOs, Civil
Society
Ministry of
Population
Development
and Social
Protection
Evaluation
Associations
and other
Associations
Ministry of
Foreign
Economic
Relations and
Investments
Preconditions and institutionalising evaluation
• capacity to conduct and use evaluation
• strategies for developing skills – courses, mentoring, peer to peer learning, elearning materials
• Address capacity, motivation and opportunity
• embedding evaluation in planning and implementation processes
• complementing local M & E systems with
central system
27