Dr. Dave O’Brien,
Leeds Metropolitan University and
ESRC/AHRC Placement Fellow at DCMS
Cultural policy and
measurement: context,
challenge and change
Overview




Evaluation in the cultural sector
How central government thinks about evaluation
The Green Book
Essential questions for today
General problems in cultural policy



Data collection, statistics and evidence are a
longstanding issue in the cultural sector (Selwood
2002, 2010)
Reflects the problems of defining culture (e.g.
Gray 2006, Miles and Sullivan forthcoming)
And defining what cultural policy is for
What is cultural investment for?
'Mozart is Mozart because of his music and not
because he created a tourist industry in Salzburg.....
Picasso is important because he taught a century
new ways of looking at objects and not because his
painting in the Bilbao Guggenheim Museum are
regenerating an otherwise derelict northern Spanish
port...... Absolute quality is paramount in attempting
a valuation of the arts; all other factors are
interesting, useful but secondary.‘
Tusa 1999 cited in Reeves 2002:36
.....And can cultural programmes be
evaluated?
‘The concept of the arts itself is indefinable, and
any attempt to measure it cannot begin to
represent its essential quality’
(Missel 1983, cited in Hewison 2002:85)
Understanding impact





Narratives of the value of DCMS’ sectors e.g. Arts
Council England’s Public Value Programme
Economic impact of DCMS’ sectors e.g. Impacts08’s
work
Social Impact e.g. MLA’s Generic Social Outcome
indicators
Educational Impact e.g. MLA’s Generic Learning
Outcome indicators
Intrinsic Impact work e.g. Nesta (2010), Brown &
Novak (2007)
However....
‘the sector is hindered by its failure to clearly
articulate its value in a cohesive and meaningful
way, as well as by its neglect of the compelling
need to establish a system for collecting evidence
around a set of agreed indicators that
substantiate value claims’
Scott (2009:198)
What is policy evaluation?

Policy evaluation uses a range of research methods to systematically
investigate the effectiveness of policy interventions, implementation and
processes, and to determine their merit, worth, or value in terms of
improving the social and economic conditions of different stakeholders.
Policy evaluation uses quantitative and qualitative methods, experimental
and non-experimental designs, descriptive and experiential methods,
theory based approaches, research synthesis methods, and economic
evaluation methods. Policy evaluation for government privileges no
single method of inquiry and acknowledges the complementary
potential of different research methods. (Magenta Book
http://www.nationalschool.gov.uk/policyhub/evaluating_policy/magenta
_book/chapter1.asp)
And why are we interested in it?

“To be effective policy making must be a learning process
which involves finding out from experience what works and
what does not and making sure that others can learn from it
too. This means that new policies must have evaluation of
their effectiveness built into them from the start; established
policies must be reviewed regularly to ensure that they are
still delivering the desired outcome; and the lessons learned
from evaluation must be available and accessible to other
policy makers.”
(Cabinet Office 2000)
Evaluation should......




Be systematic
Be analytical
Study actual effects
Judge success
(Cabinet Office 2000)
Evaluation should aim to.......




Improve decision making
Help resource allocation
Enhance accountability
Bring organisational learning
(Cabinet Office 2000)
Judging success, studying actual effects
and helping resource allocation?
‘It is difficult to decide whether Shakespeare’s
Hamlet is better than his King Lear and even harder
to persuade others of our decision or define what
such a ranking would mean. How many Gershwin
songs sum up to a Shostakovich symphony? Is a
Haydn string quartet better than a Hemmingway
short story? How does a Blake poem compare to a
modern ballet performance?’
Cowen (2006:6)
HMT’s understanding of value






Green Book
Cost Benefit Analysis
Net Present Value (NPV): value of costs of a policy minus
value of benefits, discounted over time
Recommends economic techniques to value ‘nonmarket’ goods
Works well for some forms of government policy, but not
for others
We know the costs of culture- how do we value the
benefits?
Concluding Questions


How to we close the gap between the Green
Book and the Cultural Sector?
At a time of funding cuts, how do DCMS’ sectors
best demonstrate:
‘their merit, worth, or value in terms of improving the
social and economic conditions of different
stakeholders?’