Problematizing Evidence-based
Policy-making
Mark Evans
Director,
Institute for Governance
and Policy Analysis
What is evidence-based
policy making?
• The latest rediscovery of evidencebased policy-making may be viewed
as part of a longer historical search
for usable and relevant knowledge
generated through rational scientific
methods to help address and resolve
social problems.
• This quest dates back to the
enlightenment but finds its modern
expression in the work of the Webbs
at the turn of the 19th and 20th
centuries and in the rational model of
decision-making developed by
Herbert Simon (1945) in the seminal
Administrative Behavior (New York:
Macmillan).
...rational decision-making
involves the selection of the
alternative which will maximise
the decision maker’s values, the
selection being made following a
comprehensive analysis of
alternatives and their
consequences (Simon 1945: 1).
The “What Works” Movement
• The Blair government in the
UK’s (2001) Better Policy
making mapped out an evidence
based approach to policy driven
by Health Sciences.
• Led to the establishment of the
ESRC UK Centre for Evidence
Based Policy-making and
Practice at Queen Mary College,
University of London and even
an academic journal (Evidence
and Policy).
• Blair’s approach clearly
influenced Rudd; ‘Evidencebased policy making is at the
heart of being a reformist
government’ (Rudd, 2007).
The aspiration for evidence-based
policy-making in Australia is ‘to
produce the knowledge required
for fine-tuning programs and
constructing guidelines and ‘toolkits’ for dealing with known
problems’ (Head, 2008). Hence,
the currently famous phrase that
defines much of the movement –
‘what works?’
ANZSOG has recently followed
suit with the announcement of the
launch of a new journal, Evidence
Base, under the editorship of
George Argyrous at the University
of New South Wales.
Institute’s research in this area
• Seeks to problematize the
quest for evidence-based
policy making by bringing
the politics back in through
the study of:
• Critically Examining the
notion of Policy Success
• Critically assessing attempts
to enhance strategic policy
capability (Edwards, Evans
and Scott)
Policy Success?
• Given importance of this issue, under-studied.
• Almost exclusively focuses on the
programmatic dimension – this is the
dimension behind the mantra of evidencebased policy making – need to assess whether
policy does (if assessment of existing policy) or
is likely to (if proposed policy) achieve its
aims.
• However, there are other dimensions of policy
success – and significant problems of
interpretation
Dimensions of policy success:
an heuristic
• PROCESS – Did it pass with limited
opposition/amendment?
• PROGRAMATIC – Did it achieve its aims?
• POLITICAL – Did it contribute to the
Government’s electoral popularity/re-election?
• Obviously may be major contradictions between
outcomes/success on each dimension
• Unintended consequences if all dimensions not
considered
Complexity factors
• Ontology and epistemology, as always!
• Success for whom?
• The temporal dimension – different time-scales for
different actors – different time-scales related to
different dimensions of success
Assessing policy success:
using the Heuristic 1
• J. Rutter, S. Sims and E. Marshall (2012), The 'S'
Factors: Lessons from IFG's Policy Success
reunionshttp://www.instituteforgovernment.org.uk/publicat
ions/s-factors
• Identified 6 UK Policy Successes over previous 30
years
• Policy Success Reunions – Brought together all key
actors to discuss why they were successful
Assessing policy success:
using the heuristic 2
7 key lessons
1. Understand the past and learn from failure
2. Open up the policy process
3. Be rigorous in analysis and use of evidence
4. Take time and build in scope for iteration and adaptation
5. Recognise the importance of individual leadership and
strong personal relationships
6. Create new institutions to overcome policy inertia
7. Build a wider constituency of support
On strategic policy capability
• Comparative project
on strategic policy
capability in Australia
and New Zealand
• Seeks to evaluate the
impact of various
learning interventions
on policy capability
Project aims
• To assemble an information base to allow crossjurisdictional comparisons of policy advising
frameworks and capability requirements for strategic
policy development and advisory work.
• To identify areas where there are specific deficits
and to design and evaluate different initiatives to
address them drawing on international as well as
local initiatives within some ANZSOG jurisdictions.
• To promote wider discussion, learning and reflection
by practitioners and academics on the diverse
approaches used by individual agencies and
departments, Public Service Commissions/State
Service Authorities.
Learning interventions
Type of intervention
Cases
Strategic interventions attempts to
introduce new service-wide missions that
impact directly on the nature of policymaking
The Reform of Australian Government
Administration (The Strategic Policy Network,
Capability Review, The APS 200 Public Sector
Innovation Project
NZ Government Better Public Services Initiatives
NZ Performance Improvement Framework
Developments
Delivery interventions via new system
interactions involving new or altered ways
of solving policy problems
ACT Learning and Development Framework;
Queensland Policy Capability and Development
Framework; Coombs Forum; Centre for
Excellence in Public Sector Design (ThinkPlace)
Process interventions refer to the
introduction of new internal procedures,
and policies for augmenting strategic policy
capability
ACT Government’s Triple-bottom line Assessment
for Cabinet Submissions
Capability development interventions for
enhancing skills development
The APS 200 Public Sector Innovation Project ;
The ACT Senior Policy Forum
Because what
you
told me is
Yes,
absolutely
how
did
Youcorrect
must
but
Where
you
be
a
completely
am
I?
know?
researcher
useless
Because you
don’t know where
you are,
you 30
You’re
You must
Yes.
don’t know
where
metres
be
a
policy
How
you’re going,
and
above
the
maker
did
you
now you’re
ground
in a
know?
blamingballoon
me
Key questions
1. What are the drivers of
strategic policy capability
internationally and
domestically?
2. Is there a problem?
3. If there is – what is the
nature of the problem?
4. Is it systems-based?
5. Is it competency based?
6. Is it a problem of politics?
Or a virtue of liberal
democracy?
7. Can learning interventions
make a difference?
Evidence-based (rational) versus policybased (real world) policy-making
• Notwithstanding that evidence will always be contested in a
survey of Commonwealth and State SES (ANZSOG, 2012;
N=120) we posed four sets of questions using survey and
workshop methods:
1. Questions crystallized around whether evidence was a
sufficient criterion for winning the war of ideas (Survey).
2. Questions centering on the barriers to evidence-based policymaking (Workshop).
3. Questions relating to perceived qualities of policy leadership
(Survey)
4. Questions relating to perceptions of the ingredients of better
policy-making (Workshop)
Is evidence a sufficient criterion for
winning the war of ideas?
• 94% recognised the importance of evidence-based policymaking
• 84% identified an ongoing tension between short-term
imperative and evidence-based policy-making
• 62% noted “Ministerial indifference over the facts”
• All respondents (N=48) who had been in the service for 10
years or more states that the use of evidence in policy-making
was in ‘dramatic decline’
% time spent on developing new policy,
programmes or interventions through a “rational
process of learning”
Male
Female
Commonwealth
24
20
State
12
9
% time spent on retrofitting evidence to
decisions that have already been taken
Male
Female
Commonwealth
76
80
State
88
91
Is evidence a sufficient criterion for
winning the war of ideas?
• Breakdown by gender.
Marginal difference.
• Breakdown by age. More
significant difference.
• Breakdown by jurisdiction.
Predictable difference due to
greater delivery function at
the state level.
What are the major barriers to getting evidence
into policy-making?
•
•
•
•
•
•
•
Behavioural/Conceptual
Competing understanding of its merits
(political versus bureaucratic) reflected in
an anti-evidence culture
Culture of risk aversion
Poor commissioning focus on interpersonal
linkages
Institutional resources/constraints
Absence of clear roles and responsibilities
Special advisors
Poor engagement capacity
Limited skills in active risk or change
management to create opportunity
structures for evidence-based policymaking
•
•
•
•
•
•
•
•
•
Environmental constraints
Crowded policy spaces (institutional
layering)
Public expectations for quick fixes
Prevailing socio-economic conditions
Problems inherent in multi-level
governance (less evident in NZ&Taz) e.g.
poor strategic alignment cross government
Institutional resources/constraints
Lack of support from politicians
Short-term budgets and planning horizons
Delivery pressures and administrative
burdens
Poor rewards and incentives
Capability deficit in political awareness
What are the most important
qualities of politically aware policy
leadership?
1. Broad role (rather than maturity) experience
2. Policy-makers who define politics purely as the
means for pursuing self-interest score poorly; those
that focus on empowerment score highly
3. Cross sectoral/overseas experience
4. Strong advocacy skills
5. Ability to read people and situations
6. Ability to build teams around weaknesses
7. Ability to focus on the work & not get dragged into
the game-laying
What are your principles of
policy leadership for the Twenty-First Century?
Be forward
Looking
EvidenceBased
Be politically
aware
Be outward
looking
Be inclusive
Be outcomes
Driven
Be innovative
and Creative
Join up
upwards,
outwards and
inwards
Learn lessons,
review and
evaluate
22