Manchester Cultural Impacts
Toolkit
Final Report
January 2008
www.bop.co.uk
Contents
1
INTRODUCTION ........................................................................................................... 1
2
BACKGROUND ............................................................................................................ 1
2.1
2.2
2.3
2.4
2.5
Why has the toolkit been developed? ............................................................................................1
What does the toolkit do? ...............................................................................................................2
Who will the main users of the toolkit be? ......................................................................................2
What data and information will the toolkit help to collect? ............................................................2
How has the toolkit been developed? ............................................................................................3
3
EXISTING MODELS AND FRAMEWORKS.................................................................. 3
3.1
3.2
3.3
3.4
Learning impact ..............................................................................................................................4
Social impact ..................................................................................................................................4
Economic impact ............................................................................................................................5
Impact, performance management, and programme evaluation ...................................................6
4
TOOLKIT DEVELOPMENT .......................................................................................... 8
4.1
4.2
4.3
Cultural impacts model ...................................................................................................................9
Cultural impacts framework ......................................................................................................... 10
Development of prototype ........................................................................................................... 15
5
CONCLUSIONS AND RECOMMENDATIONS ........................................................... 18
BIBLIOGRAPHY ...................................................................................................................... 20
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
i
www.bop.co.uk
1
Introduction
Manchester City Council Cultural Strategy Team commissioned Burns Owens
Partnership, Ltd. to develop a toolkit for measuring the impact of culture in Manchester.
This report summarises the process undertaken to develop the Manchester Cultural
Impacts Toolkit. It explains the purpose of the Manchester Impacts Toolkit and the basis
on which the toolkit has been designed. The report outlines the purposes of the toolkit,
defines key terms for the project; establishes the steps undertaken to develop the toolkit;
discusses what emerging models and frameworks have been built upon, and presents a
framework for understanding the linkages of cultural projects and programmes to
Manchester City Council priorities.
The Background section seeks to address the potential questions of the main
stakeholders, which include:





Why has the toolkit been developed?
What does the toolkit do?
Who will the main users of the toolkit be?
What data and information help to toolkit collect?
How has the toolkit been developed?
The Existing models and frameworks section examines a selection of the models,
frameworks, and theories considered in developing the Toolkit.
The Toolkit development section:

describes the model that BOP developed for understanding culture’s contribution to
socio-economic outcomes in Manchester.

presents the framework developed by BOP for considering the social outcomes in
relation to Manchester City Council priorities.

provides additional background information from the prototype development and
testing process.
2
Background
2.1
Why has the toolkit been developed?
The Cultural Impacts Toolkit has been developed to improve the collection and general
quality of data and information on how culture makes a contribution to a range of social
and economic outcomes in Manchester.
Specifically, it is hoped that the toolkit will aid:
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
1
www.bop.co.uk
2.2

policy-making by improving the data and information on culture’s cross-cutting
contribution to a number of socio-economic policy priorities, and

service delivery by improving the data and information on how specific cultural
organisations are contributing to the achievement of a range of socio-economic
outcomes

management of investment by providing evidence of the effectiveness of existing
investment and establishing new investment opportunities.
What does the toolkit do?
The toolkit will allow a range of users to demonstrate how cultural activity in Manchester
contributes to current, shared local and national government socio-economic policy
priorities. It provides guidance on how to carry out routine research and evaluation,
particularly addressing the needs of social research and issues of capacity. It also
provides ‘tools’, that is, resources and reference material for use by cultural
organisations in carrying out monitoring and evaluation of their projects and
programmes.
2.3
Who will the main users of the toolkit be?
Direct users will include Manchester City Council (various services) and individual
cultural organisations in Manchester. There is also the potential to share experience with
regional cultural NDPBs, other local authorities and to collaborate with national agencies
and other interested parties.
2.4
What data and information will the toolkit help to
collect?
Monitoring and evaluation is moving away from outputs towards outcomes as
increasingly the view across government is that, “simply doing an activity is not enough –
you have to make a difference”. But this means that the process of monitoring and
evaluation has become more difficult.
Therefore, the toolkit is to be used to collect data and information on the cultural activities
being delivered in Manchester (outputs) and the short and medium term effects of
applying these outputs (outcomes).
Outputs are what organisations produce, for instance, the number of performances,
reference questions answered or people participating in education and outreach
programmes. Very often measurement and evaluation focuses exclusively on counting
the number of outputs produced (as these are the things that are easiest to count).
Outcomes/impact is the short to long term results of applying outputs. Short term
(sometimes referred to as ‘intermediate’ or ‘soft’ outcomes) include satisfaction and trust
in a service/event/experience, increased motivation to learn more as a result of
participating in literacy projects or reduced feelings of social isolation arising from home
library visits. Long term outcomes (sometimes referred to as ‘hard’ outcomes) include an
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
2
www.bop.co.uk
increase in the number of old people able to live independent lives or a reduction in
obesity rates for under-11s.
2.5
How has the toolkit been developed?
In order to develop the toolkit, the BOP team:
3

developed a model for how culture contributes to socio-economic outcomes based
on research findings that classifies the range of social outcomes within the context of
current, shared local and national government policy priorities

developed a framework providing a number of indicators that allows this contribution
to be monitored and evaluated

consulted project and programme staff and partners and reviewed evaluation
material to identify challenges



developed a prototype toolkit that addresses the challenges identified

conducted workshops with Manchester City Council staff, particularly data
officers and service heads to test the framework and toolkit and the process used to
develop it and consider the ways in which it can be integrated into existing processes

reviewed and revised the framework and toolkit in light of issues that emerged in
consultation.
tested the toolkit with a range of cultural projects and programmes
conducted briefing and knowledge exchange sessions with representatives of
regional and national cultural NDPBs, DCMS, and IDEA Culture Unit, and selected
cultural organisations for intelligence on similar work being conducted and changes
to local government performance management frameworks
Existing models and frameworks
The sections below examine a selection of the theories, models, and frameworks that
were considered in the process of developing the Cultural Impacts Toolkit. The
approaches examined have been developed for understanding and analysing the
learning, social and economic impacts and/or managing the performance of various
elements of the cultural sector and include:





Inspiring Learning for All General Learning Outcomes (MLA)

Sport Playing Its Part: the contribution of sport to community priorities and the
improvement agenda (Sport England).
Generic Social Outcomes (MLA)
Use or Ornament? The social impact of the arts (Francois Matarasso)
A New Libraries Performance Management Framework (MLA)
Building active and cohesive communities through culture and sport: an evaluation of
the London Borough of Newham’s cultural and sporting programme 2004/06
(ECOTEC/London Borough of Newham)
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
3
www.bop.co.uk
These theories, models, and frameworks were reviewed as they represent prominent
examples of capturing particular types of impacts, particularly learning impacts, social
impacts, and economic impacts and/or provide insights into the evolving landscape of
impact measurement and performance management in local government and the cultural
sector.
3.1
Learning impact
Despite the close involvement of many elements of the cultural sector in work which has
a direct educational focus as well as those that have a more informal educational role,
few attempts have been made to formalise the way in which such learning activities are
evaluated.
The resulting framework, known as Inspiring Learning for All (ILFA), utilises a set of
‘Generic Learning Outcomes’ (GLOs) that are used to measure the learning that takes
place within museums, galleries, libraries and archives. ILFA is based on a model where
the emphasis on the experience and context of learning is important as one of the unique
contributions that the cultural sector makes in this area. The ILFA framework has five top
level generic learning outcomes: knowledge and understanding; skills; attitudes and
values; enjoyment, inspiration and creativity; action, behaviour and progression. Since
2004 onwards the framework has been rolled out nationally to galleries, museums,
libraries and archives sector. It has been used to evaluate national programmes. The
IDeA Culture Unit has suggested that the GLOs framework can be a ‘validated self
assessment’ method for measuring outcomes across the whole of the cultural sector1.
What the ILFA/GLOs framework lacks is an adequate way of framing and capturing wider
social outcomes. In part, this is because the framework is explicitly educational in nature
and therefore is constructed to track educational processes or the generic acquisition of
knowledge, skills, values rather than the ‘content’ of the learning. In addition to difficulties
in capturing what might be termed ‘socially relevant’ learning by individuals, the
methodology has more fundamental problems in explaining how individual outcomes
build/combine into wider social impact.
3.2
Social impact
There are, however, more longstanding attempts to understand and measure the specific
social impact of elements of the cultural sector, particularly within the arts. Francois
Matarasso’s 1997 report Use or ornament? The social impact of participation in the arts,
is a milestone in this respect. Although it is a research report, based predominantly on a
review of evaluations undertaken across 60 different participatory arts projects, the
research applies a deductive framework to analyse the material across the projects
based on six themes: personal development; social cohesion; community empowerment
and self determination; local image and identity; imagination and vision; health and wellbeing. These six themes are then populated with 50 individual ‘social impacts’ that have
arisen across the participatory arts projects. This implies more systematization in the
research findings than is actually the case as a range of social impacts are reported,
featuring quotations and case study material as well as some limited quantitative
evidence.
1
IDeA (2006) A passion for excellence: an emerging self-improvement strategy for cultural services.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
4
www.bop.co.uk
From the perspective of the present project, the value of Use or Ornament is as a
‘ground clearing’ exercise. It does not actually make a claim to having identified a robust
methodology for measuring social impact; but to have made a start in this process.
Importantly, Matarasso notes that, ‘the [social] benefits are integral to the act of
participation’, they are ‘complex but understandable’ and that they ‘can be assessed and
planned for’ (1997). Many other studies covered by the literature on the social impacts of
culture also point to the importance of participation. The main reason that the evidence
base for the cultural sector suggests as to why cultural participation is beneficial and
something to be encouraged, is that it builds social capital.
Social capital
Social capital is allied with, but not reducible to, ‘trust’, ‘social cohesion’ and ‘civil society’.
It encompasses the web of trust, associations, everyday contacts, cooperative skills and
networking that acts as ‘social glue’. Social capital as a concept has an appeal to the
cultural sector for a variety of reasons:

Alignment with ‘core purpose’: culture and sports ‘core purpose’ regularly deals
with bringing people together from both similar and different groups; challenges
people’s ideas about both their own and other cultures; and supports informed
democracy and access to services.

Theoretical: it provides a theory for how to understand how participation and
individual learning has a wider social influence and a model for thinking about what
happens when cultural institutions come into contact with people.

Methodological: in the short term, it is easier to demonstrate a contribution towards
the formation of social networks, relationships and links to resources, than to longer
term socio-economic outcomes and targets.

Tactical: social capital is broadly accepted as a factor that underpins a range of
socio-economic goals and policies and is now used across the public and voluntary
sectors.
The MLA’s GSOs framework (developed by BOP in 2005) explicitly draws on theories of
social capital to explain how culture contributes to a range of socio-economic outcomes.
3.3
Economic impact
There is no standardisation in terms of what activities and indicators are used in
assessing the economic impact of the arts and cultural sector2. However, fundamentally
the variety of methods boil down to one of two general approaches to looking at the
economic impact of the sector:

Demand-side impacts: providing employment, generating wealth through the sale
of products and services, and buying materials and services from a range of other
2
C.f. Johnson and Thomas (2001) ‘Assessing the Economic Impact of the Arts’, in Ed. Selwood, S. The UK Cultural
Sector: Profile and Policy Issues. London: PSI. As Johnson and Thomas make clear, many economic impact studies of the
arts also include museums and galleries and, while libraries, archives and sport are generally not included in these
studies, many of the methodological issues are common across the cultural sector.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
5
www.bop.co.uk
industrial sectors (e.g. telecoms, IT and media services) are all ways in which the
cultural sector itself contributes to the wider economy.

Supply-side impacts: there are also a range of ways in which the rest of the
economy benefits from the use and consumption of products and services produced
by the sector.
Both demand and supply-side impacts can be, and are, measured using a variety of
different approaches and indicators. While there are a number of practical limitations to
assessing demand-side impacts, it is a more straightforward process than assessing the
supply-side impacts. This is partly because the conceptual understanding of precisely
what supply-side impacts the cultural sector has (and exactly how these might be
measured), is still being developed.
3.4
Impact, performance management, and
programme evaluation
There are a number of frameworks that have been/are being developed with regards to
measuring the impact of culture within the context of local government performance
management. Such frameworks have been/are being developed by cultural NDPBs,
such as Sport England and MLA, while others are being developed by local authorities.
The Government’s desire to rationalise (simplify) local government performance
management, which was set out in the November 2006 local government White Paper,
has provided an impetus for this process. Under the new proposals, the CPA process will
finish in the current financial year 2007-8. After this time, local government performance
in England will be assed using a single set of national indicators, the first draft of which
was recently published by CLG ‘The New Performance Framework for Local Authorities
and Local Authority Partnerships’ (October 2007). This consists of 198 indicators that
cover the full range of local authority responsibilities, as well as those covered by local
authority partners, such as PCTs and the police.
In each area, targets against the set of national indicators will be negotiated through new
Local Area Agreements (LAAs). Each LAA will include up to 35 targets from among the
national indicators, complemented by 17 statutory targets on educational attainment and
early years. Among the 198 indicators are five that relate directly to culture and sport,
based primarily on DCMS’ Departmental Strategic Objectives (DSOs):





NI 8 Adult participation in sport – DCMS DSO
NI 9 Use of public libraries – DCMS DSO
NI 10 Visits to museums or galleries – DCMS DSO
NI 11 Engagement in the arts – DCMS DSO
NI 57 Children and young people’s participation in high-quality PE and sport – DCSF
DSO
In addition, there are a further number of other outcomes and indicators in which culture
and sport should play a prominent role, such as:


NI 3 Civic participation in the local area – PSA 15
NI 6 Participation in regular volunteering – CO DSO
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
6
www.bop.co.uk



NI 50 Emotional health of children – PSA 12
NI 110 Young people’s participation in positive activities – PSA 14
NI 119 Self-reported measure of people’s overall health and wellbeing – DH DSO
The new national indicator set means that the few specific performance indicators that
did exist within the previous CPA regime will no longer be relevant. Similarly, culturespecific standards and frameworks that fed into the CPA process, such as the Public
Library Service Standards will be redundant within the local authority performance
management process. For this reason, DCMS and MLA have developed a New
Libraries Performance Management Framework. This new framework is more in-line
with the outcome-based approach of the new national set of indicators, choosing to focus
on “the key aspects of service performance which lead to community and personal
wellbeing” (DCMS, MLA 2007: 18), which include access, resources, quality, efficiency,
and range of services.
The public library performance framework, as proposed, focuses on key aspects of
library services rather than community outcomes or social outcomes, but the MLA is also
in the process of considering a more rounded response to the proposed changes that will
come about as a result of the local government White Paper. The MLA’s proposed
overarching performance management framework for museums, libraries, and
archives, which is set out in Securing excellence; delivering for communities (MLA 2007:
9) “links sector standards and performance management schemes to the achievement of
outcomes.” Beyond providing sector-specific performance management frameworks for
museums, libraries, and archives, the overarching framework, includes information on
participation and satisfaction and an outcomes framework, which is to be linked to LAA
outcomes frameworks.
Other cultural NDPBs have also developed frameworks for how their area of cultural
activity relates to local government objectives. In 2005 Sport England published Sport
Playing Its Part: the contribution of sport to community priorities and the improvement
agenda, which describes the contribution of sport across the key areas of local
government concern: (i) safer and stronger communities, (ii) children and young people,
(iii) economic development and enterprise, and (iv) health and well-being. Until recently,
sports were at the forefront of considering their role in community and social outcomes,
and remain better able than other aspects of the cultural sector to articulate their impact.
In some cases, cultural programme evaluations have incorporated social outcomes
frameworks, such as the evaluation of London Borough of Newham’s cultural and
sporting programme. The Newham-ECOTEC Assessment Technique (NEAT),
identifies a number of impact areas, such as offending levels or perceived levels of
health, and associates these with more specific outcomes and indicators. The evaluation
framework involves seven stages:






Measure the baseline
Develop indicators
Select projects
Assemble evaluation toolkit
Train and sustain
Evaluate projects
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
7
www.bop.co.uk

Analyse and report
The Newham-ECOTEC framework, in particular, has been a useful reference point for
developing the Manchester Cultural Impacts Toolkit. There are many similarities in
approach, although the contexts in which these frameworks have been developed are
different.
4
Toolkit development
The following sections describe the process of developing the toolkit, from the
development of the cultural impacts model and framework to the development and
piloting of the prototype toolkit.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
8
www.bop.co.uk
4.1
Cultural impacts model
Figure 1 Model for how culture contributes to wider socio-economic outcomes and policy
priorities
What the cultural
sector provides
ENJOYMENT
PARTICIPATION
That contributes
to …
PHYSICAL &
MENTAL WELL
BEING
LEARNING
EMPLOYMENT
&
EXPENDITURE
E
SOCIAL CAPITAL
HUMAN CAPITAL
aka SOCIAL
NETWORKS
aka PERSONAL
(LIFE) SKILLS &
COMPETENCIES
ECONOMIC
CAPITAL
aka MONEY
… Factors that
underpin shared
socio-economic
policy priorities
Source: BOP (2007)
Based on previous work and a review of existing theories, models, and frameworks, BOP
developed a conceptual model of how culture contributes to the attainment of socioeconomic outcomes in Manchester. Figure 1 presents this model which links the cultural
sector’s provision of learning, participation, enjoyment, and employment and expenditure
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
9
www.bop.co.uk
to various forms of capital and wellbeing. These forms of capital (human, social, and
economic) and health and wellbeing are in turn linked to the realisation of Manchester’s
strategic objectives as articulated in the three spines of the Manchester Community
Strategy: (i) reaching full potential in education and employment, (ii) individual and
collective self-esteem and mutual respect, and (iii) neighbourhoods of choice. This model
was used by BOP in the development of social outcome indicators that were integrated
into a cultural impact framework.
4.2
Cultural impacts framework
Having arrived at a model of how culture contributes to socio-economic priorities in
Manchester, BOP examined the relationships between various forms of evaluation,
monitoring, and review, in particular identifying those responsible for collecting
information, the various purposes for conducting evaluation, and the varying forms of the
information collected.
Figure 2 Links between project/programme evaluation and strategic review
INFORMATION COLLECTED
D
A
T
A
PURPOSE
LONGER TERM SOCIOECONOMIC OUTCOMES
CORPORATE & MULTIAGENCY STRATEGIC
REVIEW
CULTURAL SERVICE
OUTPUTS
/ SOME ‘PROXY’
OUTCOMES
PERFORMANCE
MANAGEMENT &
INSPECTION
DATA COLLECTORS
COUNCIL/LSP/PSBWIDE
&
I
N
F
O
R
M
A
T
I
O
N
F
L
O
W
SPECIFIC OUTPUTS
‘PROXY’ & SHORT TERM
CULTURAL OUTCOMES
ORGANISATIONAL
REVIEW
CULTURAL
PROJECTS,
PROGRAMMES, &
ORGANISATIONS
PROJECT/PROGRAMME
EVALUATION
CULTURAL
PROJECTS,
PROGRAMMES, &
ORGANISATIONS
SPECIFIC OUTPUTS
SHORT TERM
CULTURAL OUTCOMES
COUNCIL-WIDE
Source: BOP (2007)
By establishing what information is needed, for what purpose, and by whom, BOP was
able to construct a cultural impacts framework capable of translating information
obtained at a project and programme level for use at a strategic level. The development
of this framework began with the development of a group of social outcome indicators
that relate to the activity of cultural projects, programmes, and organisations. These
social outcomes were in turn associated with the three spines of the Manchester
Community Strategy and with national policy areas. BOP also identified indicators and
information that is currently collected that would provide evidence of outcomes. The
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
10
www.bop.co.uk
framework provides an example of how social outcomes can be mapped to specific and
generic strategic aims and priorities (particularly the Three Spines). It was refined a
number of times during the development and testing of the prototype toolkit. The
framework is intended to be flexible and can be modified to reflect strategic changes as
they arise.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
11
www.bop.co.uk
Figure 3 below presents the cultural impacts framework.
Figure 3 Manchester Cultural Impacts Framework
BVPI areas
Safer and
stronger
communities
Project aims and
objectives (or
generic outcomes)
1. Improve inter-group
understanding and
support cultural
diversity
Manchester three spines
areas
Manchester three spines
determinants
Manchester three spines
actions
Neighbourhoods of choice
Sense of community and
neighbourhood pride
Engaging with
others/understanding each
other
Sense of community and
neighbourhood pride
Engaging with
others/understanding each
other
Intergenerational projects
Individual/collective selfesteem - mutual respect
2. Encourage a sense
of place and
neighbourhood pride
Neighbourhoods of choice
Sense of community and
neighbourhood pride
3. Contribute to crime
prevention and/or
reduce the fear of
crime
Neighbourhoods of choice
Safe: perception and reality
of anti-social behaviour and
crime
Safe: perception and reality
of anti-social behaviour and
crime
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
11
Increase community cohesion
between people of different
ethnicity and backgrounds
Celebration of Manchester's
diversity
Using Mancunian Agreement to
empower people to contribute to
creating the city they want, & to
develop common values &
standards of behaviour
Build social capital:
strengthening social networks,
fostering trust, and building
community spirit
Address perceptions of crime by
engaging local communities in
crime and disorder activity and
communicating activity
Tackling alcohol and drug
related crime; alcohol and drug
treatment services and early
intervention
www.bop.co.uk
Safe: perception and reality
of anti-social behaviour and
crime
4. Encourage
participation in local
decision-making
Neighbourhoods of choice
7. Build the capacity of
community and
voluntary groups
Individual/collective selfesteem - mutual respect
Build social capital
8. Provide safe,
inclusive and trusted
public spaces and
services
Neighbourhoods of choice
Quality district
centres/neighbourhood
centres
Neighbourhoods of choice
Sense of community and
neighbourhood pride
Quality physical environment
and sense of place
9. Improve the quality
of the physical
environment
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
Individual/collective selfesteem - mutual respect
Neighbourhoods of choice
Sense of community and
neighbourhood pride
Personal responsibility/
individual self-esteem
Reinforcement,
responsiveness, and reward
12
Deploying the Safer
Neighbourhoods approach to
intelligence and tasking, reduce
key crime and anti-social
behaviour in n'hoods across the
city
Involve people in developing
their districts/ neighbourhoods
Empowering people to influence
how things are done locally
Strengthen community
engagement and consultation
Services communicating
opportunities for residents to
contribute and results of actions
Capacity building community
organisations and voluntary
groups
Enabling the third sector to
commission services
Providing a cluster of accessible
services & facilities, of a
reasonable quality, that serve
the neighbourhoods needs:
shops; leisure; transport; public
services; quality provision for
parents of very young children
Community pride events
Maintaining & developing high
quality community spaces:
quality design
www.bop.co.uk
Healthier
communities
and older
people
Children and
young
people
1. Encourage healthy
lifestyles an/or
increase participation
in sport and exercise
2. Support older
people's
independence
1. Increase
educational attainment
2. Encourage children
and young people to
make a positive
contribution
3. Provide children
with creative and
cultural opportunities
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
Individual/collective selfesteem - mutual respect
Personal responsibility/
individual self-esteem
Reaching full potential in
education and employment
Good basic health to enable
access to E&E for all
communities
Reaching full potential in
education and employment
Homelfie
Reaching full potential in
education and employment
Quality education
Reaching full potential in
education and employment
Reaching full potential in
education and employment
Quality of early years
experience
Achieving skills
Individual/collective selfesteem - mutual respect
Reaching full potential in
education and employment
Reaching full potential in
education and employment
Reaching full potential in
education and employment
Personal responsibility/
individual self-esteem
Achieving skills
Achieving skills
Quality of early years
experience
13
Develop resilience in children
and young people and a sense
of well being in adults through
effective public health
programmes
Develop and enhance primary
care programmes for people with
mild to moderate mental health
problems
Implement healthy lifestyle
programme over the course of
the lifecycle
Supporting vulnerable people to
live in their own homes
Build mental health and
resilience of pupils
Improve attendance
High quality curriculum that
facilitates personal learning
Increased vocation pathways
Sure Start/Children's Services
Ensuring the achievement of
good basic skills and high
aspirations for young people and
adults
Promoting and facilitating
volunteering
Mentoring
Access to creative learning in
non-traditional environments
The Play Strategy
www.bop.co.uk
Economic
development
and
enterprise
1. Develop business
assistance/skills/empl
oyment opportunities
for the creative
industries
Sustainable economic growth
Reaching full potential in
education and employment
2. Provide basic skills
and/or skills and
training for
employment
3. Promote
Manchester as a
cultural city/Pride in
the city of Manchester
Reaching full potential in
education and employment
Increase competitiveness
and performance of
businesses
Increase competitiveness
and performance of
businesses
Increase competitiveness
and performance of
businesses
Growing the labour force
Achieving skills
Reducing worklessness by
improving access to
employment and retention
Achieving skills
Source: BOP (2007)
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
14
Improving productivity and
competitiveness of existing
businesses
Increasing level of start-up and
self-employment
Growing creative, cultural, and
media sector as a key sector
Increase relevance of skills
supply to Manchester employers
Improving the labour market
relevance of training and the
coherence of delivery
Better targetting of employment
support
Increasing employer and
individual commitmetn to training
and lifelong learning for all
www.bop.co.uk
4.3
Development of prototype
After developing a model and framework to provide a conceptual underpinning to the
work, the development of the prototype toolkit involved the following steps:





consultation and information gathering with projects and programmes
identifying challenges and setting objectives for the toolkit
developing an outline toolkit
developing prototype toolkit
testing prototype with project and programme staff, data officers and service heads.
Consultation and information gathering
The BOP team began the process by consulting with and gathering information from a
group of pilot projects selected by Manchester City Council Cultural Strategy Team. Pilot
projects included:

Gorton Visual Arts Group – a community-led arts group supported by a Cultural
Regeneration Officer. Participants included older people and people with disabilities.

Summer Reading Challenge – a national programme delivered locally in
Manchester Libraries. Participants included children and young people.

Halle Education – education department of a classical music venue. Variety of
projects with various participant groups.

RE:SPECT – Wythenshawe community project involving children and young people
in participatory cultural activity culminating in a talent show. Supported by a Cultural
Regeneration Officer.

Library Theatre Looked After Children Project – collaboration between Library
Theatre and the Lookedl After Children Partnership to find ways of opening services
to looked after children.
BOP conducted multiple interviews with Cultural Regeneration Officers, project and
programme managers, and partners, observed activity and planning sessions, and
reviewed evaluation and project background documentation.
Identifying challenges and setting objectives for the toolkit
Consultation and information gathering identified the three major challenges that are
impeding the collection of evidence of the impact of cultural projects and services in
Manchester:



inconsistent monitoring/evaluation processes
lack of capacity in basic evaluation (and social research) techniques
lack of resources (human and financial) for highly sophisticated and detailed
research and evaluation.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
15
www.bop.co.uk
Beyond identifying challenges, the consultation and information gathering phase also
revealed that:

Projects are broadly interested in learning more about impacts, but lack an
understanding of how to do so

There are a number of projects that are using innovative approaches to improve
service provision, but these innovations may not be reported to or understood by
decision-makers

There is a great deal of information that exists, is being collected or could be easily
collected, but there is no system that consistently captures and presents this
evidence.
Having identified the challenges associated with the collection of evidence, the toolkit
was devised to:


provide a consistent process of monitoring and evaluation across cultural services


imbed monitoring and evaluation in the project planning process
link the monitoring and evaluation to Manchester City Council strategic priorities (e.g.
Three Spines)
integrate basic social research methods into the monitoring and evaluation process
(to better capture social outcomes).
Developing outline toolkit
With these objectives in mind, the BOP team developed a draft toolkit outline, which was
refined in later consultation with project and programme staff and Manchester City
Council staff.
The draft toolkit enabled the pilot projects to easily identify their top three social
outcomes. It then presented them with a series of questions about the project that would
enable them to identify basic social research techniques alongside guidance on how to
tailor and use the tools identified in order to capture information about their particular
project. The analysis of the information collected as a result of using that tool would
enable them to measure the impact of their projects consistently.
Figure 4 on the following page presents the outline structure of the toolkit, which includes
the following sections:





Introduction
Project information and background
Identifying respondents and tools
Developing and using tools
Monitoring and reporting.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
16
www.bop.co.uk
Figure 4 Outline structure of the toolkit
INTRODUCTION
1. PROJECT INFO & BACKGROUND
2. ID RESPONDENTS & TOOLS
3. DEVELOP & USE TOOLS
4. MONITORING & REPORTING
Source: BOP (2007)
Developing and testing the prototype
The toolkit was further developed in a number of brainstorming sessions in which the
BOP team developed each section of the toolkit in turn. The sections on project
information and background and identifying respondents and tools were developed first
and tested with project and programme staff. Feedback from project and programme
staff involved the revision of these sections and the development of the remaining outline
sections. The toolkit was further tested, in its full form, with project and programme staff
and Manchester City Council data officers and service heads. Ongoing testing and
consultation has been critical for the refinement of the toolkit and has insured that the
prototype is fit-for-purpose and user friendly.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
17
www.bop.co.uk
5
Conclusions and recommendations
Developing the Manchester Cultural Impacts Toolkit has been a challenging but
rewarding project. The process of translating theory to conceptual models and
frameworks and these models and frameworks into a useable tool has involved a
number of stages of inquiry, consultation, and refinement. Establishing rapport with
project and programme staff and Manchester City Council staff has been integral to the
success of this project. The prototype toolkit provides a working framework that is
accessible, while addressing the issues of process and capacity that were identified as
key challenges to the collection and presentation of valuable evidence to policy-makers
on the contribution of culture to socio-economic priorities in Manchester. The process of
developing and testing the toolkit has revealed that:

The toolkit must be embedded in the project management process, how to do this
effectively should be covered in a short training session conducted by MCC CST.

The toolkit is useful for collecting evidence that would be applicable to a wide range
of parties beside Manchester City Council, the terms of which will be laid out in any
legal contract/guidelines (see below).

The toolkit may be useful for providing Manchester City Council with an
understanding of resource and capacity implications of an evaluation from the outset
of a project or programme. This will allow the project manager and MCC CST (and
other relevant parties) to provide sufficient resources and funding and/or understand
the limitations of any evaluation that can be undertaken within the budgetary limits.

Using the toolkit could provide a range of opportunities for cross-funding with noncultural departments and organizations, this will be identified by completing the first
step of the toolkit, particularly when identifying the social outcomes.
Despite the progress that has been made, there are key issues that should be
considered during the future piloting stages to be undertaken by Manchester City Council
Cultural Strategy team. Recommendations for piloting and roll-out are as follows (each
are subject to funding and capacity available and should be discussed and possibly be
the subject of an internal workshop as part of the next stage in the development of the
toolkit):

In terms of Copyright and Intellectual Property (IP) MCC CST should liaise and be
advised by their internal legal department. The toolkit is flexible and, in conjunction
with a short training session, should be used as MCC CST and project managers
see fit within the guidance provided. The toolkit allows the project manager to use
their own discretion in devising the tool(s) however it should be made clear that they
should keep within the guidance provided to ensure the questions consistently relate
to the social outcome and the impact they want to measure. If the toolkit is made
available externally copyright and IP will reside with Manchester City Council and
should be used to reflect the terms laid out in any legal contract/guidelines drawn up
between the two parties.

Training and capacity-building for Manchester City Council staff and pilot project staff
will be necessary to effectively embed the toolkit.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
18
www.bop.co.uk

Selection of pilot projects needs to be well-managed in terms of the number of
projects undertaking piloting and their general level of capacity and resource.

Piloting with a small number of external projects to test the fit outside of MCC should
be considered.

Cultural NDPBs and other interested parties should be consulted on opportunities for
sharing the toolkit and/or for influencing the development of other toolkits,
frameworks, and approaches (and financial and copyright implications).

An evaluation plan should be developed for the piloting phase, which makes clear
the commitment from pilot projects and from Manchester City Council and outlines
how the piloting process will be structured.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
19
www.bop.co.uk
Bibliography
1. AEA (2005) Tyne & Wear Museums and Bristol’s Museums, Galleries & Archives:
social impact programme assessment.
2. Barker, Yvonne. Sport Playing Its Part: the contribution of sport to community
priorities and the improvement agenda. Sport England.
3. Belfiore, E (2002) ‘Art as a means of alleviating social exclusion: does it really work?
A critique of instrumental cultural policies and social impact studies in the UK’,
International Journal of Cultural Policy, 8 (2002), 91-106.
4. BOP (2005) New Directions in social policy: developing the evidence base for
museums, libraries and archives in England, report for the Museums, Libraries and
Archives Council for England.
5. BOP (2006) Review of museums, libraries and archives activity with children and
young people, report for Museums, Libraries and Archives (MLA) Council for
England, MLA North West and Renaissance North West.
6. BOP, MIPC and NESTA (2006) Creating growth: how the UK can develop world
class creative businesses.
7. Cambridge Econometrics (2003) The value of the sports economy in England, report
for Sport England.
8. Coalter, F (2001) Cultural services: realising the potential, report for the Local
Government Association.
9. DCMS & MLA (2007). A New Libraries Performance Management Framework.
10. ECOTEC & London Borough of Newham (2006). Building active and cohesive
communities through culture and sport: an evaluation of the London Borough of
Newham’s cultural and sporting programme 2004/06.
11. Elstein, D (2004) Building Public Value: the BBC’s New Philosophy, 19th IEA Current
Controversies Paper.
12. Holden, J (2004) Capturing cultural value: how culture has become a tool of
government policy. Demos.
13. Hooper-Greenhill, E and Moussouri, T (2002) Researching learning in museums and
galleries 1990-1999: a bibliographic review, Research Centre for Museums and
Galleries, University of Leicester, UK.
14. IDeA (2006) A passion for excellence: an emerging self-improvement strategy for
cultural services.
15. IPPR (2006) Sticking together: social capital and local government. London: IPPR.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
20
www.bop.co.uk
16. Johnson and Thomas (2001) ‘Assessing the economic impact of the arts’, in Ed.
Selwood, S. The UK Cultural Sector: Profile and Policy Issues. London: PSI.
17. Jura Consultants (2005) Bolton’s museums, libraries and archives services: an
economic valuation, report for MLA and Bolton Metropolitan Borough Council.
18. Matarasso, F (1997) Use or ornament? The social impact of participation in the arts.
Comedia.
19. Merli, P (2002) 'Evaluating the social impact of participation in arts activities: a critical
review of Francois Matarasso's Use or Ornament?', International Journal of Cultural
Policy, Vol. 8(1), pp.107-118.
20. Middleton, V (2002) Measuring the Local Impact of Tourism: A Review, report for the
British Resorts Association and the Local Government Association.
21. MLA (2007). Securing excellence; delivering for communities: museums, libraries,
and archives and the local government White Paper.
22. Moussouri (2002) A context for the development of learning outcomes in museums,
libraries and archives; report for Re:source.
23. Myerscough, J (1988) The economic importance of the arts in Britain . London:
Policy Studies Institute.
24. Pung C, Clarke, A and Patten, L. (2004) ‘Measuring the economic impact of the
British Library’. New Review of Academic Librarianship (Vol. 10, No1)
25. Putnam (2002) Bowling Alone: The Collapse and Revival of American Community.
New York: Simon & Schuster.
26. Reeves, M (2002) Measuring the economic and social impact of the arts: a review,
report for the Arts Council of England.
27. Ruiz, J (2004) A literature review of the evidence base for culture, the arts and sport
policy, Research and Economic Unit Scottish Executive Education Department.
28. Sport England (2001) The social landscape of sport England: a review of the
research evidence and public policy implications.
29. Substance (2006) Knowing the score: Positive Futures case study research – Final
Report, report for the Home Office.
30. Wainwright, S (2002) Measuring impact: a guide to resources, report for the National
Council of Voluntary Organisations.
Manchester Cultural Impacts Toolkit
Burns Owens Partnership Ltd 2007
21