Methologie

advertisement
1
Management Control in Museums and Cultural Organizations:
Performance Measurement, Metrics, and Genuine Judgements
Martin Piber
Thomas Gstraunthaler
Work in progress, please do not quote.
Abstract
This paper describes and analyses various performance measurement and
evaluation practices in use in three European museums of modern and contemporary
art. As they are the basis for fundamental decisions concerning the overall strategic
orientation, the exhibition programme, and the perception of the museum in the
general public, their design and implementation is an important element for the
success of the whole organization. Herewith different constructions of “performance”
and “effectiveness” are studied. The paper starts with a literature overview in the area
of the management of cultural organizations, followed by three in-depth case studies
of museums of contemporary art in three European countries. The discussed
performance measurement systems are analyzed against the concept of the 4
generations of evaluation (Guba & Lincoln 1989). On this basis a general reflection of
the impact and the perception of performance measurement and evaluation systems
in museums is conducted. The paper concludes with a proposal to open up
performance measurement and evaluation practices in museums towards the 4th
generation of responsive constructivist evaluation and gives suggestions for further
research.
2
Keywords
management
practices,
performance
measurement,
evaluation,
cultural
organizations, museum;
Introduction
In recent years, Art and Culture became important economic factors. The economic
paradigm more and more permeates the cultural sector as well as cultural
organizations. With that we realize a decreased attention paid to the artistic work,
what frequently causes frustration for professionals (e.g. artists or curators) feeling a
detachment of the full amplitude of their artistic work (Cohen & Pate 2000). In parallel
major companies got financially involved in cultural activities. In the literature a broad
band width of books and papers added bits and pieces to this development. But is it
really that easy to combine both, Arts and Business?
The value of cultural activities can not be measured in financial terms alone.
Admittedly, cultural activities do have clear economic effects: The term “Cultural
industries” (Jacobius 2004, Throsby 2004) understand Arts and Culture as economic
fields of action. From this perspective the field is seen as an industrial sector, whose
influence on national and regional products is investigated. This economic value
becomes measurable in metric terms. On an institutional level, we look at additionally
generated visitor numbers or sales figures in museum shops, cafés, restaurants and
hotels. On the other hand, every cultural performance bears an arstic value, not
expressible in economic terms. Its value is reflected by the feelings of the beholder or
those involved in its creation. It is also reflected by a certain group of informed
individuals like artists, art critiques or curators. The judgement of the artistic value is
done by experts in various artistic professions. Therefore this ‘inner value’ of art and
culture remains hidden well behind the economic representation of “success” or
effectiveness. On the main stage visitor numbers, sales figures in museum shops,
cafés, restaurants as well as certain quantities of sold publications are considered.
Referring to this arena the question arises how museums and other cultural
organizations can a) communicate their performance to outsiders and b) how they
can evaluate and interpret it within its own boundaries. The decisions in cultural
3
organizations are based on different – and sometimes even conflicting target
systems: Primarily we find economic and artistic target systems: “We have to balance
economic and artistic interests in our organization.” Director, Center Museum.
In a today’s museum, both approaches, the artistic and the economic values, have to
be aligned. Therefore, a museum-wide performance measurement system has to
compromise between both value systems. This dichotomy turns museums into a very
promising field of research. The museums we have studied are all caught between
the demands and politics of different stakeholders, reaching from public funders to
business-based board of trustees. Therefore, the performance measurements
systems have a strong bias towards external communication, as the museum has to
legitimize its own actions. The internal performance measurement practices in use
are designed to give feedback to their own activities and to provide learning
opportunities. Thus, they are sometimes loosely coupled or even decoupled from
those, which are in place for communication purposes.
Qualitative inquiry
In terms of scientific methods, this study is based on a qualitative inquiry. Using
techniques like participant observation, document analyses, and in-depth interviews
with representatives of the museums, we collected a broad variety of data. The fieldnotes and other material resulted in a “thick description” (Geertz 1973) of the
organizations - a detailed account of the researchers observations in the specific
context. The aim of the method is to allow the description and analysis of the specific
‘culture’ of museums in order to unfold the very specifity of the cases by interactively
creating circumscriptions valued as adequate and plausible by the persons involved.
According to the critique of positivism and post-positivism (Guba & Lincoln 1994)
human inquiry aims at understanding by inscription (Geertz 1973) and interpretation
(Schwandt 1994).
Studying the museums in different countries enabled us to link certain differences
and particularities of their business to their innate cultural and institutional context.
We used the method of semi-structured interviews with an open time frame,
recording questions and answers, and interpreting it in order to generate a
4
sensemaking-process by differentiation. In each museum a couple of interviews with
responsible managers and professionals have been conducted. In detail, we
interviewed managers and professionals out of different departments:

director

creative director (artistic director)

executive director

(chief) curator

mediation/communication

external relations/marketing

collection/research
We have started our analysis with a map of relevant stakeholders to trace back their
expectations and targets, which they impose on the institutions. Investigating the
meanings people attach to their social environment is helpful in understanding the
performance and cultural changes within the organization (Nahapiet 1988).
Herewith a narratological approach complements the methodology. The analysis of
narratives provides insight in basic assumptions, judgements, prejudices and the use
of certain symbols. „[I]f we want to understand a society, or some part of a society,
we have to discover its repertoire of legitimate stories and find out how this evolved
[…]” (Czarniawska 1997:16). The narratives and the way they are enacted are
supposed to shed light on the relevant characters, processes, and norms which
structure and guide the organization. Thereby, we managed to draw a picture of the
self-understanding of different institutions and their representatives, which is vital to
understand their own sense-making. By pursuing this path of research we intend to
gain a profound knowledge as well as empirically grounded understanding of the
organizational behavior of museums and its stakeholders.
At the beginning of our interviews we asked about how a ‘good’ museum would look
like and how the particular museum would turn this vision into reality. When
knowledge and experience is put into daily practices, presumably different strata of
knowledge and experience come into play. With the interviews it was tried to
participate in the ‘mental world’ of the interviewees. In total 17 semi-structured
5
interviews in the three museums mentioned have been done. In addition, internal and
external reports and other documents and software facilities of the museums have
been studied.
Literature review
Previous research has shed light on the issues of the organization and the
governance of museums out of different perspectives. Quite a good deal of the
literature stresses the functions and dysfunctions of management practices
implemented in cultural organizations. Currently non-profit organizations like
museums are changing dramatically in the ways they are financed. Especially private
sponsors and/or donors become more and more important. In the last decades
operational structures and processes of museums have been adjusted to businesslike organizations. However, the implementation of various management practices in
a “New Public Management frame” is also critically discussed (Anheier 1998, Hood
1995).
One of the reasons for the ‘managerialization’ of museums is the ongoing drive to
economize public institutions and to measure their performance also in business
terms. In this context a shift of cultural organizations from public governance to more
or less independent private corporations can be observed in many countries. This
development is part of the spirit of efficiency which has overrun many countries of the
1st, 2nd, and 3rd world during the last 20 years.
Caused by financial crises and collapses of seemingly stable cultural organizations a
new interest in accountability came to the fore. Thus many public organizations
introduced performance measurement systems in the belief that they will lead to a
transparent organization and a better performance. However, the notions of efficiency
and effectiveness are not new for the public sector. Nevertheless there has been a
fundamental shift in the manner in which services are provided. One objective of the
government is to provide services at lower costs, whilst maintaining or even
improving service quality. Although it is no love-relationship, the reality for cultural
organizations around the world is infused with an intensified use of accounting
techniques and performance measurement practices – also in museums. When
6
administrators or political bodies try to enforce the implementation of new
management control systems in organizations, various forms of conflicts can be
observed in the implementation process. Examples are well documented in the
literature (Christensen & Mohr 1995; Carnegie & Wolnizer 1996; Rentschler & Potter
1996; Christiansen & Skærbæk 1997; Zan et. al. 2000; Bruijn 2002; Basso 2004;
Sciulli 2004 and Mottner & Ford 2005).
A second body of literature is placed around the issues of management styles and
organizational change. Museums have changed from being predominantly custodial
institutions to becoming increasingly focused on audience attraction. New emphasis
is placed on museum-audience interactions and relationships. Also this change in the
purpose and the priorities of museums has impacted upon the nature of museum
management. The increasing focus on audience attraction, the drive for efficiency
and effectiveness, and efforts to legitimize former public institutions in the face of
growing public interest have an enormous impact on the nature of managing
museums (Abraham & Griffin 1999; Abraham/Sullivan/Griffin 2002; Gilmore &
Rentschler 2002).
Another focus of the published work addresses marketing issues and the branding of
cultural institutions. Measurement of visitor satisfaction and motivation as well as the
resulting brand strength of cultural institutions is a relatively new concern for
museums (Caldwell & Coshall 2002, Schroeder 2005). In the huge world of events,
leisure, and tourist attractions also museums compete for attention. Thereby a strong
brand and a thorough eye on customer satisfaction help to attract enough visitors.
Indeed there has been a significant focus on customer care and customer service in
museums in recent years (Rowley 1999). On the other hand visitors have more
choices in a world of growing possibilities. Finally better marketing and public
relations can tear down the prevalent perception that nonprofit institutions are
commonly mismanaged (McLean 1995 and 1997, Kotler & Kotler 1998, McNichol
2005, Camarero Izquierdo & Garrido Samaniego 2007).
Further streams of literature concern the political economy of cultural institutions (e.g.
Feldstein 1991, Frey 1995, Johnson 1995), financial issues (e.g. Jacobius 2004) as
well as ethical issues. As regards the latter three main themes can be distinguished:
7
Legal significance, ethical codes, and organizational ethics (Wood & Rentschler
2003). Finally, the regional economic impact of museums and other cultural activities
regarding production (value added) and employment is in the center of a longrunning debate in cultural economics.
The basic framework of this paper is built by the NPM-literature. Management
practices, especially concerning performance measurement and budgeting are
considered as crucial tools to achieve cost control and to efficiently run organizations.
The NPM-movement developed out of the belief that bureaucracy was inefficient and
new paradigms were needed. The labels for this stream of thought were plentifold,
such as “post-bureaucratic”, “entrepreneurial government”, “market-based public
administration” or “managerialism” and, probably the best known of all, “New Public
Management”. Along came a shift of focus, expressed in new vocabulary like
‘empowerment’, ‘client’, ‘customer’ or ‘performance’. Now the public administration
should start to earn money, not spending any more (Savoie 2006). The bureaucratic
government was expected to turn into an entrepreneurial government, both
competitive and customer driven (Osborne & Gaebler 1992). This proliferation of
management techniques lead to a different understanding and legitimation of
administration. Corruption, waste and incompetence had to be minimized (Watkins &
Arrington 2007). The new accountability program questioned the clear separation of
the public and the private sector, and set out to fight managerial discretion,
favouritism and corruption by an elaborated body of procedural rules (Hood 1995).
Accounting as a tool was understood to incorporate a strong orientation towards
markets as it derived out of the business world. The deployment of accounting helped
to overcome the separation between private and public sector and replaced the
process-driven paradigm with a more visible focus on results and performance.
Nowadays, organizations with cultural tasks have to be generally supported with
public funding. Only a small range of ‘mass culture’ can survive when left alone in the
market economy. Therefore the value of the services offered by cultural and artistic
organizations has – among others – a professional and an economic dimension. This
has to be taken into account, when strategic decisions are taken. So far this
dichotomy is not considered in the literature.
8
Accounting took a prominent role in this economization-development and significantly
shaped the understanding of public governance. Its rising importance and its
encroachment into public fields are part of the ‘increasing accountingization’ (Power
& Laughlin 1992) of the public world. It prompts an ‘endless list’ of financial
management techniques to pursue the reforms (Guthrie et al. 1999). Scholars
critically reflected on the permeate role that accounting takes in this process. Hoskin
(1994) stressed out the new role of accounting: “[…] first as an economic practice at
the heart of everyday business management and in the global requirements of
financial reporting and analysis, but also beyond these traditional spheres. […]
Indeed, accounting even arguably penetrates within the self, as individuals discover
that from birth to death they are now accountable, known and evaluated through their
financial and non-financial performance figures.” (Hoskin 1994: 57f). The general
crisis of management accounting was displayed by Johnson & Kaplan with their book
on “relevance lost”. However, the critique was not understood: As a reaction to the
failures of accounting, scholars and practitioners tried to fill the gap with even more
accounting. In parallel to the financial accounting regime, a second system was
introduced – apparently with good intentions: Non-financial performance measures
should cure the lack of flexibility in the accounting systems. Multideminsaional
management tools like the Balanced Scorecard or the Performance Prism
(Neely/Adams/Kennerly 2002) as well as strategic conceptualizations like Intellectual
Capital or Human Resource Accounting have been established in this period. Now
we face a huge swap of these ideas to the former public sector – also to cultural
organizations.
The main idea of multidimensional performance measurement systems can be boiled
down to the idea, that non financial dimensions of performance and the
organizational potential should be made commensurable and accessible for
‘calcutative practices’ (Latour 1987). Also these seemingly comprehensive
accounting tools have reached cultural organizations like museums or theatres.
However, problems occur to seize the artistic performance with the NPM-terminology
and the NPM-tools of organizational governance. Certainly this perspective produces
new information and important insights for cultural organizations. Therefore we
analyze the impact of these governance tools on organizational decisions on the
basis of three case studies.
9
1st case: Center Museum1
The Center Museum, the most prominent museum of modern and contemporary art
in the country, was founded in the 60ties as a 'museum of the 20 th Century’. Now the
collection of the Center Museum covers approximately 9000 works: Paintings,
sculptures, installations, design, photos, videos, films, architecture models and
furniture from the first half of the 20th century. A substantial task of the museum is to
administer the collection and to develop it further to keep it a leading collection in
international terms. Additionally, the museum acts as a vehicle to promote national
and international artists.
The museum understands itself as an actor in the current social and political
discourse and also engages in educational programs. Its mission statement defines
the tasks of the museum, the structure of the organization and the definition of its
rights and obligations. Interestingly enough, the mission statement also specifies the
obligatory function of a management accountant.
The museum receives most of its funds from the national government. To ensure its
efficient use, the museum has to account for the spending of the funds and the
performance the museum achieves. This control on the distance fosters a pure
economic view, which focuses on admittance numbers. The artistic value of
exhibitions in turn is left to professional circles which restrain completely from
numbers. As a national museum the museum is formally responsible to the Ministry,
although the minister interferes only rarely in management issues. In order to
supervise the management a supervisory board with 9 members has been
implemented. Its chairperson is nominated by the ministry. The board is primarily an
economic institution and its decisions have mostly financial consequences.
The exhibition program is agreed on by a board consisting of curators, the marketing
department and the director. The conceptualization of an exhibition is a long-term
process which starts with informal conversations and runs through a complex
evaluation process, with planning times up to 2-3 years. Usually one big spring
1
The name has been changed.
10
exhibition, a small summer exhibition and again a big autumn exhibition as the
highlight of the year take are scheduled, with different target groups.
The different value systems, the economic measurement with the focus on the
visitors and the artistic measurement with the focus on the inner quality of an
exhibition are very hard to align. The director and its staff are facing a trade-off
between necessary numbers of visitors to justify the investment of public money and
the demands of the curators, driven by artistic and scientific perfection. “The required
quota is merely achievable with mainstream exhibitions among which the quality
suffers.” Director, Center Museum. Thought, mainstream and artistically regarded
events do not exclude each other. Rather, the cash-flow that is generated by one
blockbuster is used to finance smaller art projects with a more exclusive approach.
Therefore, the curators are obliged to balance the program for a wide spread public
and the program designed for a professional audience. The budgeting process is run
by the curator and the marketing department. On the other hand, their main
supervisory board, the board of trustees, is strongly biased when defining what
quality is. “The stakeholders of the museum expect an annual increase of the visitor
figure and the financial returns. They don’t care for artistic excellence”. Curator,
Center Museum.
The reporting of the museum contains quarterly reports for the ministry, which merely
consists out of financial figures. The heads of department report their activities to the
board of trusties on a regular base, and once a year a comprehensive museum
report is issued.
The strategy of the museum is lined out in an activity plan for the next 3 years which
is due to acceptance of the board of trustees. If they disagree, a cut down of the
museums budget is possible. To be comparable on a larger scale, the ministry
publishes a benchmarking report every 5 years including all major national museums.
“Summing up, the key figures – especially the number of visitors and the success at
the counter play an important role in the press.” Director, Center Museum.
2nd case: Art museum2
2
The name has been changed.
11
The second case is an art museum in a small European country. It is the main
museum for modern art in the country and the government sees it as a vital element
of self-identification for the inhabitants of the region. It is designed to be a place of
encounter with art and a place to share experiences and of living culture. However,
also touristic reasons complemented the arguments to finance the museum: “[… The
museum is o]ne of the buildings, with which the touristical market value of the city
has been raised in the recent past.” (History of the museum). Nevertheless there are
some particularities that make the museum less prone to commercial interests than
other institutions in the country. The public stake in the founding process was
comparably small, as the building was mainly financed by private initiatives. The
running costs are covered by the government. The collection, on the other hand, is a
donation of the state collection.
The exhibition program is planned for 6 years in advance. Thereby, the museum
emphasizes small niches and elaborates special facets of the artists it puts on
display: If the spot-light is on Pop-Art, then they focus on Warhols late work rather
than the often exhibited work he did with Marilyn Monroe. It is definitely not the target
of the museum to produce blockbuster exhibitions, so the strategy is designed for
regular visitors, coming back again and again. “Visitor figures are relative for us.”
says the executive director of the museum. “In some cases we feel free to say that
we want to exhibit a unique artist – being fully aware that we can’t attract the crowds.
It is not the major task of a museum to generate only visitor figures. And our
stakeholders share this understanding” Director, Art Museum.
The museum works for a better understanding of culture and society. It is seen as a
place to learn and to train social competence. “[…] what is important is the impulse to
create something, which is coming from the artwork. Visual answers for visual
statements.” Communication Manager. The museum contains packed experience
and reality of mankind.
One of the main targets of the museum is that its activities produce an “echo” in the
region. For example there was an exhibition concerning the topic “deceleration” and
shortly after the start of the show other cultural initiatives picked up the topic and
12
lectures on deceleration were given in the city. Furthermore a big trust assigned the
founder of the slow-food movement with an important award. “Good art is not
political. It is much more than that, because political means left-wing or right-wing,
and then you are categorized. A good piece of art still works after 300 years, when
the circumstances will have changed completely.” Curator. These are (qualitative)
signs, that an exhibition and the museum as a whole have an impact on society.
The museum is accountable to various stakeholders: The performance of Curators is
judged by their colleagues and the informed public. Via these channels they try to
establish their reputation. For the museum the success and a positive echo is very
important, if not the most prominent of all measurements.
Concerning visitors it is also the impact which is in the focus of the senior staff: “If you
started something new [an exhibition], there are days, when the sun rises for a
second time and it opens up a new world. […] If you realize, that something has
arisen, somebody leaves the building and has got to know a new world. These are
enriching moments in the museum.” Nevertheless the communication process works
with figures: “We think that you cannot communicate with figures. But the problem is
that our board of trustees believes in figures.” Director.
The strategy has to work for the artistic targets: “Contents are sacred.” Managing
Director. The museum has very stable visitor figures because of a main north-southtransit-route. The museum has a purchasing commission with 5 members including
the director. The person in charge of the collection is without vote in this commission.
The commission is appointed by board of the trust. Every two months a meeting with
the board of trustees with a written report from the director takes place.
3rd case: Tree Gallery3
The third case is a Museum of Modern and Contemporary Art in a major city in the
country. As a public institution the permanent exhibition can be visited free of charge.
The museum “offers something for everyone, whether you want to find out more
about an artist, come with your class to see how we used to live, see the latest in
3
The name has been changed.
13
contemporary art, or simply meet a friend for coffee. Tree City has a museum for you!
All our museums are free, most have café facilities, and we warmly welcome
children.”4 Mission Statement of the museum.
The museum is one of the most visited contemporary art museums in the UK outside
London. Thereby, the museum is directed to display the work of local and
international artists as well as addressing current social issues through its projects. It
opened in 1996 and is housed in an elegant, neo-classical building in the heart of the
city centre. Together with the library it offers a meeting place for all people of the City.
With a focus upon the work of artists from the 1960s until the present day, the
museum displays works out of its own collection and invites young artists to display
their work, respectively to work in the museum, which offers the unique chance to
witness the making of a piece of art. The museum puts a strong emphasis on efforts
concerning education and access programs, ensuring that they address different
social and demographic target groups.
The Tree Gallery is part of the city, which funds most of its activities. As here the
main funding institution is a smaller, more regional community, the museum now
competes for funding against other community projects like the hospital or the oldage pensioner home.
As the city has a very tight financial situation, keeping its budget is an imperative for
the museum as no further help will be available. The core instrument is a comparison
of the planned and finally executed exhibitions and other activities. On an operative
basis a system of cost control is implemented. The department in charge compares
the schedule and its implementation on a basis.
For the city council, the most important measurement is the number of people who
have profited from the spending, as the city council sees itself as a servant for the
democratic principle. Therefore, their main interest rests with the number of visitors.
The museum designs its activities also for an intended focus group that has
previously not been attracted by museums (this activity is summarized under the term
of “social inclusion”). Summing up, it can be stated that decisions are mainly taken on
4
Citation taken from the mission statement of the museum.
14
a qualitative basis. Key figures are used in order to present the fulfillment of the
targets regarding to officials from outside – especially from the City Council.
Performance measurement and evaluation: Discussion
All three cases have in common, that the performance of the museum is measured
and evaluated. However, we found different strategies and methods to present the
achievement of targets and the fulfilling of stakeholder expectations. To analyze the
different performance measurement and evaluation practices in use, we build on
Guba & Lincoln (1989) and their concept of 4 generations of evaluation. Concerning
the empirical data, the display of effectiveness and the performance measurement
practices can be easily connected to the four generations of evaluation.
The first generation of evaluation is described as the “measurement paradigm”. The
role of the evaluator is exclusively technical. It is expected that the evaluator knows
the full panoply of available instruments, in order to measure any variable within the
spectrum of the targets. His ‘superior knowledge’ about the targets to be achieved
and the measures to be used, enables a seemingly clear reproduction of ‘the
performance’ on different scales.
The second generation was triggered by serious deficiencies of the first generation,
as the measurement process takes place without reference to any previously agreed
targets. Performance measurement and evaluation in the second generation
compares the results with the objectives or the mission statement of the organization.
The role of the evaluator is thus to describe patterns of strength and weakness with
reference to clear-cut targets. Measurement is no longer treated as the equivalent of
evaluation but is redefined as one of several tools that might be used in its service
(Guba/Lincoln 1989, p. 28).
In the third generation judgement and expertise come to the fore. A judgement
requires a decision, which is based at least on two different value systems. In
contrast to this a simple choice would be a mere application of clear-cut and
predefined rules without room for special expertise and interpretation. Usually a
decision is taken on the basis of a verdict given by an individual who holds the
15
authority to do so. Authority stems from expertise, which is a base of knowledge that
is recognized as suitable to give a judgement (Madsen & Polesie 1981). In some
cases power issues complement the expertise. With the assignment of a judgement
to an individual, evaluation leaves the ground of objectivity and gets more and more
subjective and – in consequence, political. As any evaluation has a political
dimension, which cannot be eliminated a conscious handling and critical reflection on
judgements become an imperative. Starting from here, judgement is considered as
an integral part of evaluation and the evaluator acts as a judge (Guba/Lincoln 1989,
p. 30).
That clear and easy link between a mission and breaking it down into manageable
figures is sometimes misleading. In contrast to profit-orientated companies which
pursue the maximization of their earnings, museums face multifaceted targets. We
can illustrate this problem upon the Mission statement of the Museum of Applied Arts
in Vienna: “The museum must continue to develop as a place of awareness, free of
external influences, by advancing the discourse between the interplay of experience
and perception. Since it navigates along the borders that separate art and awareness
from innumerable forms of fashionable consumption, the museum's articulation of
qualitative assessment makes it a central forum for resistance against the
widespread loss of meaning pandemic in contemporary popular culture.” Herewith it
becomes obvious, that any evaluation or measurement of the targets mentioned in
this statement cannot be done without various judgements.
Comparing the performance measurement and evaluation of the three cases
described, the following conclusions can be drawn.: Firstly, in all our cases so-called
operational performance measures are in use (e.g. visitors per day, the use of audio
guides per exhibition, the number of catalogues sold, or the marketing expenses per
visitor). These numbers, together with cost figures taken from cost accounting
systems, provide the starting point of performance measurement in all museums
studied. Still, there are substantial differences in how much attention these numbers
receive. In the case of the Center Museum, a decline in visitor numbers would easily
lead to a cut-back on public money. The Art Museum, on the other hand, uses
operative performance measures in order to analyze differences between exhibitions
and also to compare the organization to other museums.
16
Secondly, some institutions link these measurements to their targets in order to
realize to which extent their mission has been fulfilled in a given period. To do so
demands a high degree of freedom in structuring one’s own evaluation system. For
the Center Museum, the ministry requires all federal museums within the country to
report the same set of key figures to ensure comparability between different
institutions. Generally speaking the Tree Gallery makes the most intensive use of this
kind of measures. It is their objective to attract as many inhabitants of the town as
possible. Thirdly, there are very few practices in use to give insight into quality and
subtle results like the artistic or scientific quality of exhibitions or the broader
relevance of the institutions for society. The Art Museum even tries to use metric
measures like the number of articles published in highly regarded newspapers and
magazines about the museum or the contents and the amount of positive feedback
from visitors”, which have been cited by the curators of the Art Museum.
Nevertheless, they see only little informational value in them.
Contrastingly much attention is paid to the comments of experts, which are included
in the official annual report of the museum. In another approach one of the museums
invites artists and other curators to comment on exhibitions and to share these
comments with relevant stakeholders.
Judgement and the use of expertise beyond the realms of key figures are primarily in
use in the Art Museum. Although the key figures are part of the reported performance
measures, it is not the target to maximize the number of visitors. The executive team
of the Art Museum adheres to the expertise of artists, curators, and various experts
within and outside the museum. Judgement as integral part of evaluation in this
context is an attempt to interpret and to communicate the value of the work.
Measuring quality in metric terms bears the danger of over-simplification. Although
tempting, a focus on visitor figures leaves many questions unanswered: “You can’t
measure the success of a museum on a number of people, who walk in and out.”
Curator, Tree Gallery. If we observe decisions about strategy, the general mission,
the exhibition program, as well as the funding of different tasks, it can be seen that
experts from the art field don’t believe in measurements belonging to the first or
second generation. They decide upon their intuition: “I think this is a really good
17
understanding of the senior management in terms of the quality of work at Tree
Gallery and the level of provision that we provide. I think they are having a very, very
good understanding of that.” Director of Tree Gallery. In organizational terms the
main decisions are taken by a team (two cases) or by the executive director, who
consults his experts up front (one case). However, in team meetings especially the
managers try to back their arguments with metric performance measures. On the
other hand these metric performance measures are met with resistance by the
experts, although they are used to present performance issues to external
stakeholders. On an overall basis these performance measures are used to legitimize
the museum to outside constituencies.
In general, the presence of operational performance measures can ‘hide’ the
absence of indicators for outcome and quality. Although the experts highlight the
need for genuine judgements, they are reluctant to argue on the basis of their expert
judgements. External stakeholders who desperately want to get the ‘naked truth’ of
the performance of a museum and who therefore have pressured towards the
introduction of these numbers are left alone with tons of metric performance
measures with little meaning. As they feel that their desire is not met by the inflow of
information, they tend to develop even more measurements with the same outcome.
A full comparability of museums on the basis of benchmarking figures is probably not
achievable. Thus it can be concluded, that the performance and the quality of
museums can only be measured on the surface. In the end expert judgements from
different stakeholders are necessary.
Different stakeholder groups and various interests within the museums and
concerning their external constituencies frequently trigger dilemma-situations: In two
of the museums studied the interviewees reported, that there are frequently
situations, where they have to decide between blockbuster-programs on the one
hand and artistic quality on the other. Other dilemmas touch the acquisition policy or
the trade-off between marketing and educational issues. In such a complex and
partly contradicting environment the conceptualization of the 4th generation
evaluation provides a useful framework for a profound dialogue of the targets and
their evaluation together with the relevant stakeholders. Guba & Lincoln (1989)
developed the responsive constructivist evaluation with clear reference to the
18
deficiencies of mere measurement processes for seemingly clear-cut objects.
Therefore we see an obvious potential for the improvement of performance
measurement and evaluation practices – especially in cultural organizations.
Opening up metric measurement towards the 4th generation of evaluation
Firstly the 4th generation of evaluation takes into account that reality is socially
constructed – also for museums. Therefore it reflects the insight, that an evaluation
process can not verify ontologically given targets. In fact the targets have to be
negotiated with several stakeholders - primarily the funding institutions, the general
public, the professionals involved and the visitors. Also these stakeholders construct
their realities and their target systems – both challenging the strategic alignment of
the museums. Secondly a clear need for responsible expert judgements is articulated
and it becomes evident that there is no ‘objective’ and ‘true’ performance in an
organization. Nevertheless there is a vested interest within and outside the museum
to present its achievements and its relevance as well as its performance to various
constituencies. At this point the 4th generation evaluation establishes a transparent,
responsible and credible process. To summarize the outcomes, it can be concluded
that the credibility and with that the usefulness of evaluation practices for cultural
organizations only can emerge out of a mutual understanding and a profound
examination of the whole cultural field.
On this basis, the evaluation process will challenge an ongoing self reflection within
the museum/cultural organization as well as a fruitful debate on strategy with relevant
constituencies. Further research could align these
conclusions with other
conceptualizations of evaluation. For example social audits or narrative approaches
of accountability are to be considered in this context. Finally further empirical studies
in different cultural fields could enlighten other characteristics of performance
measurement and evaluation practices.
References
19
Abraham M., Griffin D. 1999, Organisation change and management decision in
museums, Management Decision, Vol. 37, no 9/10.
Abraham M., Sullivan T. and D. Griffin, 2002, Implementing NAGPRA: the effective
management of legislated change in museums, Management Decision, Vol. 40, no
1, p. 35-49.
Ames P.J. 1994, A challenge to modern museum management: meshing mission and
market, Moore K. 1994, museum management, London: Routledge, p. 15-21.
Anderson R.G.W. 2005, To thrive or survive? The state and status of research in
museums, Museum Management and Curatorship, Vol. 20, p. 297-311.
Anheier H. 1998, Commerce and the muse: are art museums becoming commercial?
To profit or not to profit. – Cambridge [u.a.]: Cambridge Univ. Press, p. 33-248.
Basso A. 2004, A quantitative approach to evaluate the relative efficiency of
museums, Journal of cultural economics, Vol. 28, no 3, p. 195-216.
Bruijn H. 2002, Performance measurement in the public sector: strategies to cope
with the risks of performance measurement, International Journal of Public Sector
Management, Vol. 15, no 7, p. 578-594.
Caldwell N., Coshall J. 2002, Measuring brand associations for museums and
galleries using repertory grid analysis, Management Decision, Vol. 40, no 4, p.
383-392.
Camarero Izquierdo C., Garrido Samaniego M. 2007, How alternative marketing
strategies impact the performance of Spanish museums, Journal of Management
Development, Vol. 26, no 9, p. 809-931.
Carnegie G., Wolnizer P. 1996, Enabling accountability in Museums, Accounting,
Auditing & Accountability Journal, Vol. 9, no 5, p. 84-99.
Christensen A., Mohr R. 1995, Testing a positive theory model of museum
accounting practices, Financial Accountability & Management, Vol. 11, no 4, p.
317.
Christiansen J., Skærbæk P. 1997, Implementing budgetary control in the performing
arts: games in the organizational theatre, Management Accounting Research, Vol.
8, no 4, p. 405-438.
Cohen C., Pate M. 2000, Making a meal of arts evaluation: can social audit offer a
more balanced approach? Managing Leisure, Vol. 5, no 3, p. 103-120.
Czarniawska B. 1997, Narrating the Organization – Dramas of Institutional Identity.
Chicago and London: The University of Chicago Press.
20
Feldstein M. 1991, The economics of art museums. University of Chicago Press,
Chicago.
Frey B. 1995, Cultural economics and museum behaviour: a reply, Scottish journal of
political economy, Vol. 42, no 4, p. 467-468.
Geertz C. 1973, Thick Description: Toward an Interpretative Theory of Culture, The
Interpretation of Cultures. New York: Basic Books.
Getzner M. 2003, Small museums and the regional economy: an Austrian case
study, Wirtschaftspolitische Blätter. – Wien: Österr. Wirtschaftsverl., Vol. 50, no 2,
p. 271-284.
Gilmore A., Rentschler R. 2002, Changes in museum management, Journal of
Management Development, Vol. 21, no 10, p. 745-760.
Guba E.G., Lincoln Y.S. 1989, Forth Generation Evaluation, London: Sage.
Guba E.G., Lincoln Y.S. 1994, Competing paradigms in qualitative research, Denzin
N.K., Lincoln Y.S. Editors, 1994, Handbook of qualitative research Sage
Publications, Thousand Oaks, CA, p. 105–117.
Guthrie J., Petty R., Ferrier F. and R. Wells, 1999, ”There is no accounting for
intellectual capital in Australia: a review of annual reporting practices and the
internal measurement of intangibles within Australian organisations'', paper
presented at the International Symposium Measuring and Reporting Intellectual
Capital: Experiences, Issues and Prospects, OECD, Amsterdam.
Hatton A. 1999, Organizational Cultures in a Museum and Art Gallery, Moore K.
1999, Management in Museums, Continuum International Publishing Group, p.
208-232.
Hood C. 1995, The `new public management` in the 1980s: Variations on a theme,
Accounting, Organization and Society, Vol. 20, no 2/3, p. 93-109.
Hoskin K. W. 1994, Boxing clever: for, against and beyond Foucault in the battle for
accounting theory, Critical Perspectives on Accounting, Vol. 5, p. 25–56.
Jacobius A. 2004, Museum financing comes together, Pensions & Investment, Vol.
32, no 20, p. 8-8.
James K. 2008, A Critical Theory and Postmodernist approach to the teaching of
accounting theory, Critical Perspectives on Accounting, Vol. 19, n° 5, p. 643-676.
Johnson P. 1995, Cultural economics and museum behaviour: a comment, Scottish
journal of political economy, Vol. 42, no 4, p. 465-466.
21
Johnson H.T., Kaplan R.S. 1987, Relevance Lost – The Rise and Fall of
Management Accounting. Boston: Harvard Business School Press.
Kotler N., Kotler P. 1998, Museum strategy and marketing designing mission,
building audiences, generating revenue and resources, The Jossey-Bass nonprofit
an public management series, XXVIII.
Latour B. 1987, Science in Action: How to follow scientists and engineers through
society. Cambridge: Harvard University Press.
Madsen V., Polesie T. 1981, Human Factors in Budgeting: Judgement and
Evaluation. London.
McLean F. 1995, A Marketing Revolution in Museums? Journal of Marketing
Management, Vol. 11, no 6, p. 601-616.
McLean F. 1997, Marketing the museum. London: Routledge.
McNichol T. 2005, Creative marketing strategies in small museums: up close an
innovative, International Journal of Nonprofit & Voluntary Sector Marketing, Vol.
10, no 4, p. 239- 247.
Mottner S., Ford J. 2005, Measuring nonprofit marketing strategy performance: the
case of museum stores, Journal of Business Research, Vol. 6, no 58, p. 829-840.
Nahapiet J. 1988, The Rhetoric and Reality of an Accounting Change, Accounting
Organizations and Society, Vol. 13, no 4, p. 333-58.
Neely A., Adams C. and M. Kennerly, 2002, Performance Prism: The Scorecard for
Measuring and Managing Stakeholder Relationships. London.
Osborne D., Gaebler T. 1992, Reinventing government: How the entrepreneurial
spirit is transforming the public sector/Reinventing government: How the
entrepreneurial spirit is transforming the public sector. Addison-Wesley Publishing.
Power M., Laughlin R. 1992, Critical theory and accounting, Alvesson M., Wilmott H.
(Eds),Critical Management Studies, Sage, London, p.113-135.
Raymond T., Greyser S. 1978, The business of managing the arts, Harvard Business
Review, Vol. 56, no 4, p. 123-132.
Rentschler R., Potter B. 1996, Accountability versus artistic development. The case
for non-profit museums and performing arts organizations, Accounting, Auditing &
Accountability Journal, Vol. 9, no 5, p. 100-113.
Rowley J. 1999, Measuring total customer experience in museums, International
Journal of Contemporary Hospitality Management, Vol. 11, no 6, p. 303-308.
22
Savoie D.J. 2006, What is wrong with the new public management? Comparative
Public Administration, titled Research in Public Policy Analysis and Management,
Vol. 15, p. 593-602.
Schroeder J. 2005, The artist and the brand, European Journal of Marketing, Vol. 39,
no 11/12, p. 1291-1305.
Schwandt T.A. 1994, Constructivist, interpretive approaches to human inquiry,
Denzin N.K., Lincoln Y.S., Editors, 1994, Handbook of qualitative research Sage
Publications, Thousand Oaks, CA, p. 118-137.
Sciulli N. 2004, The use of management accounting information to support
contracting out decision making in the public sector, Qualitative Research in
Accounting & Management, Vol. 1, no 2, p. 43-67.
Shubik M. 1999, Culture and Commerce, Journal of Cultural Economics, Vol. 23, n°
1/2, p. 13-30.
Stake R. 2003, Case Studies, Denzin N.K., Lincoln Y.S. (Hrsg.), Strategies of
Qualitative Inquiry, 2nd ed., Thousand Oaks et al.: Sage.
Throsby D. 2001, Defining the artistic workforce: The Australian experience, Poetics,
Vol. 28, p. 255-271.
Throsby D. 2004, Assessing the Impacts of a Cultural Industry, The Journal of Arts
Management, Law, and Society, Vol. 34, no 3, p. 188-204.
Watkins A., Arrington C. 2007, Accounting, New Public Management and American
Politics: Theoretical Insights into the National Performance Review, Critical
Perspectives on Accounting, Vol. 18, no 1, p. 33-58.
Wood G., Rentschler R. 2003, Ethical behaviour: the means for creating and
maintaining better reputations in arts organisation, Management Decision, Vol. 41,
no 6, p. 528-537.
Zan L. 2002, Renewing pompeii, year zero. Promises and expectations from new
approaches to museum management and accountability, Critical Perspectives on
Accounting, Vol. 13, p. 89-137.
Zan L., Blackstock A., Cerutti G. and C. Maestro Mayer, 2000, Accounting for Art,
Scandinavian Journal of Management, Vol. 16, p. 335-347.
Download