1 Management Control in Museums and Cultural Organizations: Performance Measurement, Metrics, and Genuine Judgements Martin Piber Thomas Gstraunthaler Work in progress, please do not quote. Abstract This paper describes and analyses various performance measurement and evaluation practices in use in three European museums of modern and contemporary art. As they are the basis for fundamental decisions concerning the overall strategic orientation, the exhibition programme, and the perception of the museum in the general public, their design and implementation is an important element for the success of the whole organization. Herewith different constructions of “performance” and “effectiveness” are studied. The paper starts with a literature overview in the area of the management of cultural organizations, followed by three in-depth case studies of museums of contemporary art in three European countries. The discussed performance measurement systems are analyzed against the concept of the 4 generations of evaluation (Guba & Lincoln 1989). On this basis a general reflection of the impact and the perception of performance measurement and evaluation systems in museums is conducted. The paper concludes with a proposal to open up performance measurement and evaluation practices in museums towards the 4th generation of responsive constructivist evaluation and gives suggestions for further research. 2 Keywords management practices, performance measurement, evaluation, cultural organizations, museum; Introduction In recent years, Art and Culture became important economic factors. The economic paradigm more and more permeates the cultural sector as well as cultural organizations. With that we realize a decreased attention paid to the artistic work, what frequently causes frustration for professionals (e.g. artists or curators) feeling a detachment of the full amplitude of their artistic work (Cohen & Pate 2000). In parallel major companies got financially involved in cultural activities. In the literature a broad band width of books and papers added bits and pieces to this development. But is it really that easy to combine both, Arts and Business? The value of cultural activities can not be measured in financial terms alone. Admittedly, cultural activities do have clear economic effects: The term “Cultural industries” (Jacobius 2004, Throsby 2004) understand Arts and Culture as economic fields of action. From this perspective the field is seen as an industrial sector, whose influence on national and regional products is investigated. This economic value becomes measurable in metric terms. On an institutional level, we look at additionally generated visitor numbers or sales figures in museum shops, cafés, restaurants and hotels. On the other hand, every cultural performance bears an arstic value, not expressible in economic terms. Its value is reflected by the feelings of the beholder or those involved in its creation. It is also reflected by a certain group of informed individuals like artists, art critiques or curators. The judgement of the artistic value is done by experts in various artistic professions. Therefore this ‘inner value’ of art and culture remains hidden well behind the economic representation of “success” or effectiveness. On the main stage visitor numbers, sales figures in museum shops, cafés, restaurants as well as certain quantities of sold publications are considered. Referring to this arena the question arises how museums and other cultural organizations can a) communicate their performance to outsiders and b) how they can evaluate and interpret it within its own boundaries. The decisions in cultural 3 organizations are based on different – and sometimes even conflicting target systems: Primarily we find economic and artistic target systems: “We have to balance economic and artistic interests in our organization.” Director, Center Museum. In a today’s museum, both approaches, the artistic and the economic values, have to be aligned. Therefore, a museum-wide performance measurement system has to compromise between both value systems. This dichotomy turns museums into a very promising field of research. The museums we have studied are all caught between the demands and politics of different stakeholders, reaching from public funders to business-based board of trustees. Therefore, the performance measurements systems have a strong bias towards external communication, as the museum has to legitimize its own actions. The internal performance measurement practices in use are designed to give feedback to their own activities and to provide learning opportunities. Thus, they are sometimes loosely coupled or even decoupled from those, which are in place for communication purposes. Qualitative inquiry In terms of scientific methods, this study is based on a qualitative inquiry. Using techniques like participant observation, document analyses, and in-depth interviews with representatives of the museums, we collected a broad variety of data. The fieldnotes and other material resulted in a “thick description” (Geertz 1973) of the organizations - a detailed account of the researchers observations in the specific context. The aim of the method is to allow the description and analysis of the specific ‘culture’ of museums in order to unfold the very specifity of the cases by interactively creating circumscriptions valued as adequate and plausible by the persons involved. According to the critique of positivism and post-positivism (Guba & Lincoln 1994) human inquiry aims at understanding by inscription (Geertz 1973) and interpretation (Schwandt 1994). Studying the museums in different countries enabled us to link certain differences and particularities of their business to their innate cultural and institutional context. We used the method of semi-structured interviews with an open time frame, recording questions and answers, and interpreting it in order to generate a 4 sensemaking-process by differentiation. In each museum a couple of interviews with responsible managers and professionals have been conducted. In detail, we interviewed managers and professionals out of different departments: director creative director (artistic director) executive director (chief) curator mediation/communication external relations/marketing collection/research We have started our analysis with a map of relevant stakeholders to trace back their expectations and targets, which they impose on the institutions. Investigating the meanings people attach to their social environment is helpful in understanding the performance and cultural changes within the organization (Nahapiet 1988). Herewith a narratological approach complements the methodology. The analysis of narratives provides insight in basic assumptions, judgements, prejudices and the use of certain symbols. „[I]f we want to understand a society, or some part of a society, we have to discover its repertoire of legitimate stories and find out how this evolved […]” (Czarniawska 1997:16). The narratives and the way they are enacted are supposed to shed light on the relevant characters, processes, and norms which structure and guide the organization. Thereby, we managed to draw a picture of the self-understanding of different institutions and their representatives, which is vital to understand their own sense-making. By pursuing this path of research we intend to gain a profound knowledge as well as empirically grounded understanding of the organizational behavior of museums and its stakeholders. At the beginning of our interviews we asked about how a ‘good’ museum would look like and how the particular museum would turn this vision into reality. When knowledge and experience is put into daily practices, presumably different strata of knowledge and experience come into play. With the interviews it was tried to participate in the ‘mental world’ of the interviewees. In total 17 semi-structured 5 interviews in the three museums mentioned have been done. In addition, internal and external reports and other documents and software facilities of the museums have been studied. Literature review Previous research has shed light on the issues of the organization and the governance of museums out of different perspectives. Quite a good deal of the literature stresses the functions and dysfunctions of management practices implemented in cultural organizations. Currently non-profit organizations like museums are changing dramatically in the ways they are financed. Especially private sponsors and/or donors become more and more important. In the last decades operational structures and processes of museums have been adjusted to businesslike organizations. However, the implementation of various management practices in a “New Public Management frame” is also critically discussed (Anheier 1998, Hood 1995). One of the reasons for the ‘managerialization’ of museums is the ongoing drive to economize public institutions and to measure their performance also in business terms. In this context a shift of cultural organizations from public governance to more or less independent private corporations can be observed in many countries. This development is part of the spirit of efficiency which has overrun many countries of the 1st, 2nd, and 3rd world during the last 20 years. Caused by financial crises and collapses of seemingly stable cultural organizations a new interest in accountability came to the fore. Thus many public organizations introduced performance measurement systems in the belief that they will lead to a transparent organization and a better performance. However, the notions of efficiency and effectiveness are not new for the public sector. Nevertheless there has been a fundamental shift in the manner in which services are provided. One objective of the government is to provide services at lower costs, whilst maintaining or even improving service quality. Although it is no love-relationship, the reality for cultural organizations around the world is infused with an intensified use of accounting techniques and performance measurement practices – also in museums. When 6 administrators or political bodies try to enforce the implementation of new management control systems in organizations, various forms of conflicts can be observed in the implementation process. Examples are well documented in the literature (Christensen & Mohr 1995; Carnegie & Wolnizer 1996; Rentschler & Potter 1996; Christiansen & Skærbæk 1997; Zan et. al. 2000; Bruijn 2002; Basso 2004; Sciulli 2004 and Mottner & Ford 2005). A second body of literature is placed around the issues of management styles and organizational change. Museums have changed from being predominantly custodial institutions to becoming increasingly focused on audience attraction. New emphasis is placed on museum-audience interactions and relationships. Also this change in the purpose and the priorities of museums has impacted upon the nature of museum management. The increasing focus on audience attraction, the drive for efficiency and effectiveness, and efforts to legitimize former public institutions in the face of growing public interest have an enormous impact on the nature of managing museums (Abraham & Griffin 1999; Abraham/Sullivan/Griffin 2002; Gilmore & Rentschler 2002). Another focus of the published work addresses marketing issues and the branding of cultural institutions. Measurement of visitor satisfaction and motivation as well as the resulting brand strength of cultural institutions is a relatively new concern for museums (Caldwell & Coshall 2002, Schroeder 2005). In the huge world of events, leisure, and tourist attractions also museums compete for attention. Thereby a strong brand and a thorough eye on customer satisfaction help to attract enough visitors. Indeed there has been a significant focus on customer care and customer service in museums in recent years (Rowley 1999). On the other hand visitors have more choices in a world of growing possibilities. Finally better marketing and public relations can tear down the prevalent perception that nonprofit institutions are commonly mismanaged (McLean 1995 and 1997, Kotler & Kotler 1998, McNichol 2005, Camarero Izquierdo & Garrido Samaniego 2007). Further streams of literature concern the political economy of cultural institutions (e.g. Feldstein 1991, Frey 1995, Johnson 1995), financial issues (e.g. Jacobius 2004) as well as ethical issues. As regards the latter three main themes can be distinguished: 7 Legal significance, ethical codes, and organizational ethics (Wood & Rentschler 2003). Finally, the regional economic impact of museums and other cultural activities regarding production (value added) and employment is in the center of a longrunning debate in cultural economics. The basic framework of this paper is built by the NPM-literature. Management practices, especially concerning performance measurement and budgeting are considered as crucial tools to achieve cost control and to efficiently run organizations. The NPM-movement developed out of the belief that bureaucracy was inefficient and new paradigms were needed. The labels for this stream of thought were plentifold, such as “post-bureaucratic”, “entrepreneurial government”, “market-based public administration” or “managerialism” and, probably the best known of all, “New Public Management”. Along came a shift of focus, expressed in new vocabulary like ‘empowerment’, ‘client’, ‘customer’ or ‘performance’. Now the public administration should start to earn money, not spending any more (Savoie 2006). The bureaucratic government was expected to turn into an entrepreneurial government, both competitive and customer driven (Osborne & Gaebler 1992). This proliferation of management techniques lead to a different understanding and legitimation of administration. Corruption, waste and incompetence had to be minimized (Watkins & Arrington 2007). The new accountability program questioned the clear separation of the public and the private sector, and set out to fight managerial discretion, favouritism and corruption by an elaborated body of procedural rules (Hood 1995). Accounting as a tool was understood to incorporate a strong orientation towards markets as it derived out of the business world. The deployment of accounting helped to overcome the separation between private and public sector and replaced the process-driven paradigm with a more visible focus on results and performance. Nowadays, organizations with cultural tasks have to be generally supported with public funding. Only a small range of ‘mass culture’ can survive when left alone in the market economy. Therefore the value of the services offered by cultural and artistic organizations has – among others – a professional and an economic dimension. This has to be taken into account, when strategic decisions are taken. So far this dichotomy is not considered in the literature. 8 Accounting took a prominent role in this economization-development and significantly shaped the understanding of public governance. Its rising importance and its encroachment into public fields are part of the ‘increasing accountingization’ (Power & Laughlin 1992) of the public world. It prompts an ‘endless list’ of financial management techniques to pursue the reforms (Guthrie et al. 1999). Scholars critically reflected on the permeate role that accounting takes in this process. Hoskin (1994) stressed out the new role of accounting: “[…] first as an economic practice at the heart of everyday business management and in the global requirements of financial reporting and analysis, but also beyond these traditional spheres. […] Indeed, accounting even arguably penetrates within the self, as individuals discover that from birth to death they are now accountable, known and evaluated through their financial and non-financial performance figures.” (Hoskin 1994: 57f). The general crisis of management accounting was displayed by Johnson & Kaplan with their book on “relevance lost”. However, the critique was not understood: As a reaction to the failures of accounting, scholars and practitioners tried to fill the gap with even more accounting. In parallel to the financial accounting regime, a second system was introduced – apparently with good intentions: Non-financial performance measures should cure the lack of flexibility in the accounting systems. Multideminsaional management tools like the Balanced Scorecard or the Performance Prism (Neely/Adams/Kennerly 2002) as well as strategic conceptualizations like Intellectual Capital or Human Resource Accounting have been established in this period. Now we face a huge swap of these ideas to the former public sector – also to cultural organizations. The main idea of multidimensional performance measurement systems can be boiled down to the idea, that non financial dimensions of performance and the organizational potential should be made commensurable and accessible for ‘calcutative practices’ (Latour 1987). Also these seemingly comprehensive accounting tools have reached cultural organizations like museums or theatres. However, problems occur to seize the artistic performance with the NPM-terminology and the NPM-tools of organizational governance. Certainly this perspective produces new information and important insights for cultural organizations. Therefore we analyze the impact of these governance tools on organizational decisions on the basis of three case studies. 9 1st case: Center Museum1 The Center Museum, the most prominent museum of modern and contemporary art in the country, was founded in the 60ties as a 'museum of the 20 th Century’. Now the collection of the Center Museum covers approximately 9000 works: Paintings, sculptures, installations, design, photos, videos, films, architecture models and furniture from the first half of the 20th century. A substantial task of the museum is to administer the collection and to develop it further to keep it a leading collection in international terms. Additionally, the museum acts as a vehicle to promote national and international artists. The museum understands itself as an actor in the current social and political discourse and also engages in educational programs. Its mission statement defines the tasks of the museum, the structure of the organization and the definition of its rights and obligations. Interestingly enough, the mission statement also specifies the obligatory function of a management accountant. The museum receives most of its funds from the national government. To ensure its efficient use, the museum has to account for the spending of the funds and the performance the museum achieves. This control on the distance fosters a pure economic view, which focuses on admittance numbers. The artistic value of exhibitions in turn is left to professional circles which restrain completely from numbers. As a national museum the museum is formally responsible to the Ministry, although the minister interferes only rarely in management issues. In order to supervise the management a supervisory board with 9 members has been implemented. Its chairperson is nominated by the ministry. The board is primarily an economic institution and its decisions have mostly financial consequences. The exhibition program is agreed on by a board consisting of curators, the marketing department and the director. The conceptualization of an exhibition is a long-term process which starts with informal conversations and runs through a complex evaluation process, with planning times up to 2-3 years. Usually one big spring 1 The name has been changed. 10 exhibition, a small summer exhibition and again a big autumn exhibition as the highlight of the year take are scheduled, with different target groups. The different value systems, the economic measurement with the focus on the visitors and the artistic measurement with the focus on the inner quality of an exhibition are very hard to align. The director and its staff are facing a trade-off between necessary numbers of visitors to justify the investment of public money and the demands of the curators, driven by artistic and scientific perfection. “The required quota is merely achievable with mainstream exhibitions among which the quality suffers.” Director, Center Museum. Thought, mainstream and artistically regarded events do not exclude each other. Rather, the cash-flow that is generated by one blockbuster is used to finance smaller art projects with a more exclusive approach. Therefore, the curators are obliged to balance the program for a wide spread public and the program designed for a professional audience. The budgeting process is run by the curator and the marketing department. On the other hand, their main supervisory board, the board of trustees, is strongly biased when defining what quality is. “The stakeholders of the museum expect an annual increase of the visitor figure and the financial returns. They don’t care for artistic excellence”. Curator, Center Museum. The reporting of the museum contains quarterly reports for the ministry, which merely consists out of financial figures. The heads of department report their activities to the board of trusties on a regular base, and once a year a comprehensive museum report is issued. The strategy of the museum is lined out in an activity plan for the next 3 years which is due to acceptance of the board of trustees. If they disagree, a cut down of the museums budget is possible. To be comparable on a larger scale, the ministry publishes a benchmarking report every 5 years including all major national museums. “Summing up, the key figures – especially the number of visitors and the success at the counter play an important role in the press.” Director, Center Museum. 2nd case: Art museum2 2 The name has been changed. 11 The second case is an art museum in a small European country. It is the main museum for modern art in the country and the government sees it as a vital element of self-identification for the inhabitants of the region. It is designed to be a place of encounter with art and a place to share experiences and of living culture. However, also touristic reasons complemented the arguments to finance the museum: “[… The museum is o]ne of the buildings, with which the touristical market value of the city has been raised in the recent past.” (History of the museum). Nevertheless there are some particularities that make the museum less prone to commercial interests than other institutions in the country. The public stake in the founding process was comparably small, as the building was mainly financed by private initiatives. The running costs are covered by the government. The collection, on the other hand, is a donation of the state collection. The exhibition program is planned for 6 years in advance. Thereby, the museum emphasizes small niches and elaborates special facets of the artists it puts on display: If the spot-light is on Pop-Art, then they focus on Warhols late work rather than the often exhibited work he did with Marilyn Monroe. It is definitely not the target of the museum to produce blockbuster exhibitions, so the strategy is designed for regular visitors, coming back again and again. “Visitor figures are relative for us.” says the executive director of the museum. “In some cases we feel free to say that we want to exhibit a unique artist – being fully aware that we can’t attract the crowds. It is not the major task of a museum to generate only visitor figures. And our stakeholders share this understanding” Director, Art Museum. The museum works for a better understanding of culture and society. It is seen as a place to learn and to train social competence. “[…] what is important is the impulse to create something, which is coming from the artwork. Visual answers for visual statements.” Communication Manager. The museum contains packed experience and reality of mankind. One of the main targets of the museum is that its activities produce an “echo” in the region. For example there was an exhibition concerning the topic “deceleration” and shortly after the start of the show other cultural initiatives picked up the topic and 12 lectures on deceleration were given in the city. Furthermore a big trust assigned the founder of the slow-food movement with an important award. “Good art is not political. It is much more than that, because political means left-wing or right-wing, and then you are categorized. A good piece of art still works after 300 years, when the circumstances will have changed completely.” Curator. These are (qualitative) signs, that an exhibition and the museum as a whole have an impact on society. The museum is accountable to various stakeholders: The performance of Curators is judged by their colleagues and the informed public. Via these channels they try to establish their reputation. For the museum the success and a positive echo is very important, if not the most prominent of all measurements. Concerning visitors it is also the impact which is in the focus of the senior staff: “If you started something new [an exhibition], there are days, when the sun rises for a second time and it opens up a new world. […] If you realize, that something has arisen, somebody leaves the building and has got to know a new world. These are enriching moments in the museum.” Nevertheless the communication process works with figures: “We think that you cannot communicate with figures. But the problem is that our board of trustees believes in figures.” Director. The strategy has to work for the artistic targets: “Contents are sacred.” Managing Director. The museum has very stable visitor figures because of a main north-southtransit-route. The museum has a purchasing commission with 5 members including the director. The person in charge of the collection is without vote in this commission. The commission is appointed by board of the trust. Every two months a meeting with the board of trustees with a written report from the director takes place. 3rd case: Tree Gallery3 The third case is a Museum of Modern and Contemporary Art in a major city in the country. As a public institution the permanent exhibition can be visited free of charge. The museum “offers something for everyone, whether you want to find out more about an artist, come with your class to see how we used to live, see the latest in 3 The name has been changed. 13 contemporary art, or simply meet a friend for coffee. Tree City has a museum for you! All our museums are free, most have café facilities, and we warmly welcome children.”4 Mission Statement of the museum. The museum is one of the most visited contemporary art museums in the UK outside London. Thereby, the museum is directed to display the work of local and international artists as well as addressing current social issues through its projects. It opened in 1996 and is housed in an elegant, neo-classical building in the heart of the city centre. Together with the library it offers a meeting place for all people of the City. With a focus upon the work of artists from the 1960s until the present day, the museum displays works out of its own collection and invites young artists to display their work, respectively to work in the museum, which offers the unique chance to witness the making of a piece of art. The museum puts a strong emphasis on efforts concerning education and access programs, ensuring that they address different social and demographic target groups. The Tree Gallery is part of the city, which funds most of its activities. As here the main funding institution is a smaller, more regional community, the museum now competes for funding against other community projects like the hospital or the oldage pensioner home. As the city has a very tight financial situation, keeping its budget is an imperative for the museum as no further help will be available. The core instrument is a comparison of the planned and finally executed exhibitions and other activities. On an operative basis a system of cost control is implemented. The department in charge compares the schedule and its implementation on a basis. For the city council, the most important measurement is the number of people who have profited from the spending, as the city council sees itself as a servant for the democratic principle. Therefore, their main interest rests with the number of visitors. The museum designs its activities also for an intended focus group that has previously not been attracted by museums (this activity is summarized under the term of “social inclusion”). Summing up, it can be stated that decisions are mainly taken on 4 Citation taken from the mission statement of the museum. 14 a qualitative basis. Key figures are used in order to present the fulfillment of the targets regarding to officials from outside – especially from the City Council. Performance measurement and evaluation: Discussion All three cases have in common, that the performance of the museum is measured and evaluated. However, we found different strategies and methods to present the achievement of targets and the fulfilling of stakeholder expectations. To analyze the different performance measurement and evaluation practices in use, we build on Guba & Lincoln (1989) and their concept of 4 generations of evaluation. Concerning the empirical data, the display of effectiveness and the performance measurement practices can be easily connected to the four generations of evaluation. The first generation of evaluation is described as the “measurement paradigm”. The role of the evaluator is exclusively technical. It is expected that the evaluator knows the full panoply of available instruments, in order to measure any variable within the spectrum of the targets. His ‘superior knowledge’ about the targets to be achieved and the measures to be used, enables a seemingly clear reproduction of ‘the performance’ on different scales. The second generation was triggered by serious deficiencies of the first generation, as the measurement process takes place without reference to any previously agreed targets. Performance measurement and evaluation in the second generation compares the results with the objectives or the mission statement of the organization. The role of the evaluator is thus to describe patterns of strength and weakness with reference to clear-cut targets. Measurement is no longer treated as the equivalent of evaluation but is redefined as one of several tools that might be used in its service (Guba/Lincoln 1989, p. 28). In the third generation judgement and expertise come to the fore. A judgement requires a decision, which is based at least on two different value systems. In contrast to this a simple choice would be a mere application of clear-cut and predefined rules without room for special expertise and interpretation. Usually a decision is taken on the basis of a verdict given by an individual who holds the 15 authority to do so. Authority stems from expertise, which is a base of knowledge that is recognized as suitable to give a judgement (Madsen & Polesie 1981). In some cases power issues complement the expertise. With the assignment of a judgement to an individual, evaluation leaves the ground of objectivity and gets more and more subjective and – in consequence, political. As any evaluation has a political dimension, which cannot be eliminated a conscious handling and critical reflection on judgements become an imperative. Starting from here, judgement is considered as an integral part of evaluation and the evaluator acts as a judge (Guba/Lincoln 1989, p. 30). That clear and easy link between a mission and breaking it down into manageable figures is sometimes misleading. In contrast to profit-orientated companies which pursue the maximization of their earnings, museums face multifaceted targets. We can illustrate this problem upon the Mission statement of the Museum of Applied Arts in Vienna: “The museum must continue to develop as a place of awareness, free of external influences, by advancing the discourse between the interplay of experience and perception. Since it navigates along the borders that separate art and awareness from innumerable forms of fashionable consumption, the museum's articulation of qualitative assessment makes it a central forum for resistance against the widespread loss of meaning pandemic in contemporary popular culture.” Herewith it becomes obvious, that any evaluation or measurement of the targets mentioned in this statement cannot be done without various judgements. Comparing the performance measurement and evaluation of the three cases described, the following conclusions can be drawn.: Firstly, in all our cases so-called operational performance measures are in use (e.g. visitors per day, the use of audio guides per exhibition, the number of catalogues sold, or the marketing expenses per visitor). These numbers, together with cost figures taken from cost accounting systems, provide the starting point of performance measurement in all museums studied. Still, there are substantial differences in how much attention these numbers receive. In the case of the Center Museum, a decline in visitor numbers would easily lead to a cut-back on public money. The Art Museum, on the other hand, uses operative performance measures in order to analyze differences between exhibitions and also to compare the organization to other museums. 16 Secondly, some institutions link these measurements to their targets in order to realize to which extent their mission has been fulfilled in a given period. To do so demands a high degree of freedom in structuring one’s own evaluation system. For the Center Museum, the ministry requires all federal museums within the country to report the same set of key figures to ensure comparability between different institutions. Generally speaking the Tree Gallery makes the most intensive use of this kind of measures. It is their objective to attract as many inhabitants of the town as possible. Thirdly, there are very few practices in use to give insight into quality and subtle results like the artistic or scientific quality of exhibitions or the broader relevance of the institutions for society. The Art Museum even tries to use metric measures like the number of articles published in highly regarded newspapers and magazines about the museum or the contents and the amount of positive feedback from visitors”, which have been cited by the curators of the Art Museum. Nevertheless, they see only little informational value in them. Contrastingly much attention is paid to the comments of experts, which are included in the official annual report of the museum. In another approach one of the museums invites artists and other curators to comment on exhibitions and to share these comments with relevant stakeholders. Judgement and the use of expertise beyond the realms of key figures are primarily in use in the Art Museum. Although the key figures are part of the reported performance measures, it is not the target to maximize the number of visitors. The executive team of the Art Museum adheres to the expertise of artists, curators, and various experts within and outside the museum. Judgement as integral part of evaluation in this context is an attempt to interpret and to communicate the value of the work. Measuring quality in metric terms bears the danger of over-simplification. Although tempting, a focus on visitor figures leaves many questions unanswered: “You can’t measure the success of a museum on a number of people, who walk in and out.” Curator, Tree Gallery. If we observe decisions about strategy, the general mission, the exhibition program, as well as the funding of different tasks, it can be seen that experts from the art field don’t believe in measurements belonging to the first or second generation. They decide upon their intuition: “I think this is a really good 17 understanding of the senior management in terms of the quality of work at Tree Gallery and the level of provision that we provide. I think they are having a very, very good understanding of that.” Director of Tree Gallery. In organizational terms the main decisions are taken by a team (two cases) or by the executive director, who consults his experts up front (one case). However, in team meetings especially the managers try to back their arguments with metric performance measures. On the other hand these metric performance measures are met with resistance by the experts, although they are used to present performance issues to external stakeholders. On an overall basis these performance measures are used to legitimize the museum to outside constituencies. In general, the presence of operational performance measures can ‘hide’ the absence of indicators for outcome and quality. Although the experts highlight the need for genuine judgements, they are reluctant to argue on the basis of their expert judgements. External stakeholders who desperately want to get the ‘naked truth’ of the performance of a museum and who therefore have pressured towards the introduction of these numbers are left alone with tons of metric performance measures with little meaning. As they feel that their desire is not met by the inflow of information, they tend to develop even more measurements with the same outcome. A full comparability of museums on the basis of benchmarking figures is probably not achievable. Thus it can be concluded, that the performance and the quality of museums can only be measured on the surface. In the end expert judgements from different stakeholders are necessary. Different stakeholder groups and various interests within the museums and concerning their external constituencies frequently trigger dilemma-situations: In two of the museums studied the interviewees reported, that there are frequently situations, where they have to decide between blockbuster-programs on the one hand and artistic quality on the other. Other dilemmas touch the acquisition policy or the trade-off between marketing and educational issues. In such a complex and partly contradicting environment the conceptualization of the 4th generation evaluation provides a useful framework for a profound dialogue of the targets and their evaluation together with the relevant stakeholders. Guba & Lincoln (1989) developed the responsive constructivist evaluation with clear reference to the 18 deficiencies of mere measurement processes for seemingly clear-cut objects. Therefore we see an obvious potential for the improvement of performance measurement and evaluation practices – especially in cultural organizations. Opening up metric measurement towards the 4th generation of evaluation Firstly the 4th generation of evaluation takes into account that reality is socially constructed – also for museums. Therefore it reflects the insight, that an evaluation process can not verify ontologically given targets. In fact the targets have to be negotiated with several stakeholders - primarily the funding institutions, the general public, the professionals involved and the visitors. Also these stakeholders construct their realities and their target systems – both challenging the strategic alignment of the museums. Secondly a clear need for responsible expert judgements is articulated and it becomes evident that there is no ‘objective’ and ‘true’ performance in an organization. Nevertheless there is a vested interest within and outside the museum to present its achievements and its relevance as well as its performance to various constituencies. At this point the 4th generation evaluation establishes a transparent, responsible and credible process. To summarize the outcomes, it can be concluded that the credibility and with that the usefulness of evaluation practices for cultural organizations only can emerge out of a mutual understanding and a profound examination of the whole cultural field. On this basis, the evaluation process will challenge an ongoing self reflection within the museum/cultural organization as well as a fruitful debate on strategy with relevant constituencies. Further research could align these conclusions with other conceptualizations of evaluation. For example social audits or narrative approaches of accountability are to be considered in this context. Finally further empirical studies in different cultural fields could enlighten other characteristics of performance measurement and evaluation practices. References 19 Abraham M., Griffin D. 1999, Organisation change and management decision in museums, Management Decision, Vol. 37, no 9/10. Abraham M., Sullivan T. and D. Griffin, 2002, Implementing NAGPRA: the effective management of legislated change in museums, Management Decision, Vol. 40, no 1, p. 35-49. Ames P.J. 1994, A challenge to modern museum management: meshing mission and market, Moore K. 1994, museum management, London: Routledge, p. 15-21. Anderson R.G.W. 2005, To thrive or survive? The state and status of research in museums, Museum Management and Curatorship, Vol. 20, p. 297-311. Anheier H. 1998, Commerce and the muse: are art museums becoming commercial? To profit or not to profit. – Cambridge [u.a.]: Cambridge Univ. Press, p. 33-248. Basso A. 2004, A quantitative approach to evaluate the relative efficiency of museums, Journal of cultural economics, Vol. 28, no 3, p. 195-216. Bruijn H. 2002, Performance measurement in the public sector: strategies to cope with the risks of performance measurement, International Journal of Public Sector Management, Vol. 15, no 7, p. 578-594. Caldwell N., Coshall J. 2002, Measuring brand associations for museums and galleries using repertory grid analysis, Management Decision, Vol. 40, no 4, p. 383-392. Camarero Izquierdo C., Garrido Samaniego M. 2007, How alternative marketing strategies impact the performance of Spanish museums, Journal of Management Development, Vol. 26, no 9, p. 809-931. Carnegie G., Wolnizer P. 1996, Enabling accountability in Museums, Accounting, Auditing & Accountability Journal, Vol. 9, no 5, p. 84-99. Christensen A., Mohr R. 1995, Testing a positive theory model of museum accounting practices, Financial Accountability & Management, Vol. 11, no 4, p. 317. Christiansen J., Skærbæk P. 1997, Implementing budgetary control in the performing arts: games in the organizational theatre, Management Accounting Research, Vol. 8, no 4, p. 405-438. Cohen C., Pate M. 2000, Making a meal of arts evaluation: can social audit offer a more balanced approach? Managing Leisure, Vol. 5, no 3, p. 103-120. Czarniawska B. 1997, Narrating the Organization – Dramas of Institutional Identity. Chicago and London: The University of Chicago Press. 20 Feldstein M. 1991, The economics of art museums. University of Chicago Press, Chicago. Frey B. 1995, Cultural economics and museum behaviour: a reply, Scottish journal of political economy, Vol. 42, no 4, p. 467-468. Geertz C. 1973, Thick Description: Toward an Interpretative Theory of Culture, The Interpretation of Cultures. New York: Basic Books. Getzner M. 2003, Small museums and the regional economy: an Austrian case study, Wirtschaftspolitische Blätter. – Wien: Österr. Wirtschaftsverl., Vol. 50, no 2, p. 271-284. Gilmore A., Rentschler R. 2002, Changes in museum management, Journal of Management Development, Vol. 21, no 10, p. 745-760. Guba E.G., Lincoln Y.S. 1989, Forth Generation Evaluation, London: Sage. Guba E.G., Lincoln Y.S. 1994, Competing paradigms in qualitative research, Denzin N.K., Lincoln Y.S. Editors, 1994, Handbook of qualitative research Sage Publications, Thousand Oaks, CA, p. 105–117. Guthrie J., Petty R., Ferrier F. and R. Wells, 1999, ”There is no accounting for intellectual capital in Australia: a review of annual reporting practices and the internal measurement of intangibles within Australian organisations'', paper presented at the International Symposium Measuring and Reporting Intellectual Capital: Experiences, Issues and Prospects, OECD, Amsterdam. Hatton A. 1999, Organizational Cultures in a Museum and Art Gallery, Moore K. 1999, Management in Museums, Continuum International Publishing Group, p. 208-232. Hood C. 1995, The `new public management` in the 1980s: Variations on a theme, Accounting, Organization and Society, Vol. 20, no 2/3, p. 93-109. Hoskin K. W. 1994, Boxing clever: for, against and beyond Foucault in the battle for accounting theory, Critical Perspectives on Accounting, Vol. 5, p. 25–56. Jacobius A. 2004, Museum financing comes together, Pensions & Investment, Vol. 32, no 20, p. 8-8. James K. 2008, A Critical Theory and Postmodernist approach to the teaching of accounting theory, Critical Perspectives on Accounting, Vol. 19, n° 5, p. 643-676. Johnson P. 1995, Cultural economics and museum behaviour: a comment, Scottish journal of political economy, Vol. 42, no 4, p. 465-466. 21 Johnson H.T., Kaplan R.S. 1987, Relevance Lost – The Rise and Fall of Management Accounting. Boston: Harvard Business School Press. Kotler N., Kotler P. 1998, Museum strategy and marketing designing mission, building audiences, generating revenue and resources, The Jossey-Bass nonprofit an public management series, XXVIII. Latour B. 1987, Science in Action: How to follow scientists and engineers through society. Cambridge: Harvard University Press. Madsen V., Polesie T. 1981, Human Factors in Budgeting: Judgement and Evaluation. London. McLean F. 1995, A Marketing Revolution in Museums? Journal of Marketing Management, Vol. 11, no 6, p. 601-616. McLean F. 1997, Marketing the museum. London: Routledge. McNichol T. 2005, Creative marketing strategies in small museums: up close an innovative, International Journal of Nonprofit & Voluntary Sector Marketing, Vol. 10, no 4, p. 239- 247. Mottner S., Ford J. 2005, Measuring nonprofit marketing strategy performance: the case of museum stores, Journal of Business Research, Vol. 6, no 58, p. 829-840. Nahapiet J. 1988, The Rhetoric and Reality of an Accounting Change, Accounting Organizations and Society, Vol. 13, no 4, p. 333-58. Neely A., Adams C. and M. Kennerly, 2002, Performance Prism: The Scorecard for Measuring and Managing Stakeholder Relationships. London. Osborne D., Gaebler T. 1992, Reinventing government: How the entrepreneurial spirit is transforming the public sector/Reinventing government: How the entrepreneurial spirit is transforming the public sector. Addison-Wesley Publishing. Power M., Laughlin R. 1992, Critical theory and accounting, Alvesson M., Wilmott H. (Eds),Critical Management Studies, Sage, London, p.113-135. Raymond T., Greyser S. 1978, The business of managing the arts, Harvard Business Review, Vol. 56, no 4, p. 123-132. Rentschler R., Potter B. 1996, Accountability versus artistic development. The case for non-profit museums and performing arts organizations, Accounting, Auditing & Accountability Journal, Vol. 9, no 5, p. 100-113. Rowley J. 1999, Measuring total customer experience in museums, International Journal of Contemporary Hospitality Management, Vol. 11, no 6, p. 303-308. 22 Savoie D.J. 2006, What is wrong with the new public management? Comparative Public Administration, titled Research in Public Policy Analysis and Management, Vol. 15, p. 593-602. Schroeder J. 2005, The artist and the brand, European Journal of Marketing, Vol. 39, no 11/12, p. 1291-1305. Schwandt T.A. 1994, Constructivist, interpretive approaches to human inquiry, Denzin N.K., Lincoln Y.S., Editors, 1994, Handbook of qualitative research Sage Publications, Thousand Oaks, CA, p. 118-137. Sciulli N. 2004, The use of management accounting information to support contracting out decision making in the public sector, Qualitative Research in Accounting & Management, Vol. 1, no 2, p. 43-67. Shubik M. 1999, Culture and Commerce, Journal of Cultural Economics, Vol. 23, n° 1/2, p. 13-30. Stake R. 2003, Case Studies, Denzin N.K., Lincoln Y.S. (Hrsg.), Strategies of Qualitative Inquiry, 2nd ed., Thousand Oaks et al.: Sage. Throsby D. 2001, Defining the artistic workforce: The Australian experience, Poetics, Vol. 28, p. 255-271. Throsby D. 2004, Assessing the Impacts of a Cultural Industry, The Journal of Arts Management, Law, and Society, Vol. 34, no 3, p. 188-204. Watkins A., Arrington C. 2007, Accounting, New Public Management and American Politics: Theoretical Insights into the National Performance Review, Critical Perspectives on Accounting, Vol. 18, no 1, p. 33-58. Wood G., Rentschler R. 2003, Ethical behaviour: the means for creating and maintaining better reputations in arts organisation, Management Decision, Vol. 41, no 6, p. 528-537. Zan L. 2002, Renewing pompeii, year zero. Promises and expectations from new approaches to museum management and accountability, Critical Perspectives on Accounting, Vol. 13, p. 89-137. Zan L., Blackstock A., Cerutti G. and C. Maestro Mayer, 2000, Accounting for Art, Scandinavian Journal of Management, Vol. 16, p. 335-347.