Conference Note Perception, Performance, and Politics: Recent Approaches to the Qualitative Comparison of Civil Justice Systems Dr. Christoph Kern, LL.M. (Harvard) Albert-Ludwigs-Universität Freiburg I. Introduction .................................................................................................................... 1 II. Difficulties in Measuring and Comparing Judicial Quality ........................................... 1 1. Performance Data ........................................................................................................... 2 a) b) 2. III. Quality of Results ....................................................................................................... 3 aa) Illustration: Legal Correctness and Accuracy .................................................... 3 bb) Résumé ............................................................................................................... 4 Quality of Process ...................................................................................................... 4 aa) Illustration: Duration .......................................................................................... 5 bb) Résumé ............................................................................................................... 6 Perception Data .............................................................................................................. 6 a) Advantages of Using Perception Data ....................................................................... 6 b) Problems with Perception Data .................................................................................. 7 c) Résumé ....................................................................................................................... 9 Recent Surveys and the Role of Politics ........................................................................ 9 1. The so-called “Lex Mundi Study” on Procedural Formalism ...................................... 10 2. The 2008 Business Survey Conducted by Clifford Chance and the University of Oxford’s Institute of European and Comparative Law ................................................ 11 IV. Conclusion .................................................................................................................... 14 Perception, Performance, and Politics I. Introduction Recent years have seen an increasing importance of cross-country comparisons of the quality of court systems. Be it independent surveys focusing precisely on judicial quality, country rankings gathering diverse other data as well, or econometric research using the quality of court systems as dependent or independent variable: data assessing and comparing the quality of judicial systems is more and more being produced, processed, and used. However, it is not legal scholars, but first and foremost economists who, having rediscovered the importance of legal institutions, apply their statistic tools to various aspects of the world’s civil justice systems. In the following, we will discuss these recent approaches to the qualitative comparison of civil justice systems. Our attention will focus first on the difficulties in measuring the quality of judicial systems and, in particular, the difficulties in gathering data which allow for cross-country comparisons. After that, we will analyze how the recent studies use comparative data on judicial quality. In this context, we will come back to the Lex Mundi Study and the 2008 Business Survey, and see that politics also play an important role. II. Difficulties in Measuring and Comparing Judicial Quality What is judicial quality, and how can it be measured? These questions are neither completely new, nor are they peculiar to cross-country comparisons. While large-scale cross-country comparisons are a recent and still rare phenomenon, assessments of the performance of courts within one and the same jurisdiction are quite common today. Nevertheless, the various approaches to quality differ considerably. In our eyes, two basic distinctions should be made. The first distinction concerns the reference point of quality. The quality of a judicial system may be determined with regard to its results, in particular the correctness and justice of its solutions to disputes, or with regard to the process by which these results are achieved. The second distinction highlights the perspective from which “quality” is judged. It asks whether the quality of the court system is determined subjectively, as perceived by the “users” of the civil justice system and general public, or objectively, 1 Perception, Performance, and Politics as measured by a neutral observer. This is the distinction between subjective perception and actual performance. It will not come as a surprise that regardless of the reference point we choose— that is, regardless of how we determine the quality of court systems—, gathering data about judicial quality is difficult, and gathering data for the purpose of cross-country comparisons is even more difficult. At a closer look, we can identify several reasons for this difficulty. First, with respect to objective or performance data, it is often necessary to have resort to indicators or proxies because the information we seek cannot be measured directly, be it for practical or theoretical reasons. Another problem with objective data appears if researchers rely on external sources, in particular, official data. Very frequently, these data have not been collected in an identical way in all the countries. It is, then, questionable whether they can serve as a basis for cross-country comparisons. Finally, even if the data we possess are measuring exactly the same feature and this in exactly the same way, comparing these data may be doubtful, as external factors of the legal system and culture in a country can considerably affect the relative significance and meaning of the feature at issue. Second, as regards subjective or perception data, difficulties are of another kind. Here, we are concerned with the general problems of subjective evaluations, which are further compounded by the relatively abstract nature of the subject—judicial quality— and the wide range of externalities like culture, ideology, and media coverage that vary from country to country. This paper cannot provide a comprehensive discussion. However, we would like to clarify and illustrate these points, using the distinctions introduced above. Let us first have a look at objective evaluations, in other words, performance data. 1. Performance Data As explained above, the distinction between performance data and perception data intersects with the distinction between result-oriented and process-oriented evaluations of legal systems. As to the quality of a court system, in both, the evaluation of results and the evaluation of processes, several factors play a role. Some of these factors 2 Perception, Performance, and Politics appear to be more open to accurate measurement, while others even at first sight require value judgments. a) Quality of Results The primary and direct results or outcomes of a court system are court decisions. Ignoring ADR and discussing only legal correctness and accuracy of court decisions for the sake of brevity, we should keep in mind that other aspects from which the quality of results could be judged are substantive justice and fairness, the “true” resolution of disputes and, in a most general sense, peace and security, legal certainty and confidence in the legal system. aa) Illustration: Legal Correctness and Accuracy Obviously, the first factors in any qualitative evaluation of the results of a civil justice system are legal correctness and accuracy. A decision is legally correct if the law has been applied in a correct way; a decision is accurate if the court has discovered the truth as to all relevant and disputed facts. The higher the percentage of correct and accurate decisions, the better is the quality of a court system. Thus, legal correctness and accuracy seem to be unambiguous factors of judicial quality, perfectly open to be measured and expressed in numbers. However, in practice, it is difficult to determine the relevant numbers. Theoretically, all decisions could be reviewed and classified as correct or incorrect, accurate or inaccurate by excellent lawyers. Needless to say, though, that such comprehensive review is simply not possible. What is more popular among researchers interested in data on the legal correctness and accuracy of first instance decisions is the use of statistics on the number of successful appeals. The idea behind this method is straightforward. Once an appeal was successful, it goes, the reviewed decision must have been wrong. Therefore, the number of successful appeals is considered a viable proxy for correctness and accuracy. Yet things are not that easy. First, not all types of decisions are subject to review, and not all individual cases fulfill the requirements, if any, for an appeal. In addition, the appellate court may have some discretion over the matters which it will 3 Perception, Performance, and Politics agree to hear. These differences in the availability of appeal considerably affect the usefulness of appeals statistics for measuring first instance quality and in the context of cross-country comparisons are compounded by important differences in the law governing the availability of appeal, the costs of an appeals procedure and, last not least, the willingness of parties to seek an appeal which may be different in different legal cultures and depend on the legal framework. Second, the scope of review on the appeals level can be very different. It may range from a full de novo review of law and facts, which would allow for relatively meaningful conclusions concerning first instance quality, to a very limited review of certain legal questions, so that important aspects of first instance quality are not covered. Third, not all successful appeals really mean that the reviewed decision was incorrect in a sense that would affect judicial quality. Not only in case law systems, but also in systems based on codified law, policy questions do arise which can be decided in different ways without necessarily violating the law. bb) Résumé To sum up, the number or percentage of successful reviews only provide basic information about the correctness and accuracy of decisions in a given legal system. Even more complicated is it to undertake a comparison using such data, as the legal frameworks of different countries regarding the type of decisions subject to review or special requirements for an appeal to be admissible, are usually quite different. With legal correctness and accuracy being the most straightforward factors of the quality of results, it cannot surprise that other factors like substantive fairness and justice or the true resolution of disputes are still more difficult to measure and that the use of possibly available data on these factors in cross-country comparisons is all but easy. We will see now that things do not look much more promising as to the quality of process. b) Quality of Process The word “process” as used here means the procedure which brings about the results of a civil justice system. Practitioners and academics have been discussing about the quality of court proceedings for ages. Although approaches differ over time and 4 Perception, Performance, and Politics among legal cultures and authors, nowadays there is relatively widespread consent about the essential elements. According to a distinguished English scholar, civil procedure must find a balance between the factors truth, time and cost.1 Among the factors relevant for the quality of the process, we shall again limit our discussion to one of them which appears to be particularly suitable for numerical assessment: duration. Needless to say, however, that many other factors like costs, procedural fairness, and legitimacy should also be taken into account and, I may add, have their own difficulties. aa) Illustration: Duration Most observers agree that expeditious proceedings are an important factor of the quality of a civil justice system. Assessing duration seems to be relatively simple. Duration can be measured objectively, can be expressed in numbers, and duration data is officially collected in many countries. It therefore appears to be perfectly suited for cross-country comparison. And indeed, duration plays an important role in most comparative surveys. However, on closer examination, things are not that easy. First, as duration will always be expressed as an average, this average should cover the same type of cases. If, for example, in one country, family law disputes involving children are excluded from the data, while they are included in another, the comparison of the two averages may not be very meaningful. A related question is whether each type of cases should have the same weight, in other words, whether the average duration of a certain type of cases should be multiplied with the same factor in all countries before calculating the overall average of duration, and if so, which is the “correct” factor, or whether each case should have the same weight, regardless of the type of cases it belongs to. None of the possible solutions imposes itself—another source of imprecision. Assessing duration in practice proves difficult as well. Official court statistics in different countries do not necessarily use the same approach with respect to start and end dates. Moreover, official statistics do not always provide all the information about 1 Adrian A.S. Zuckerman, Justice in Crisis: Comparative Dimensions of Civil Procedure, in CIVIL JUSTICE IN CRISIS 3, 48-49 (Arian A.S. Zuckerman ed., 1999). 5 Perception, Performance, and Politics types of cases and other characteristics that might be relevant for comparing duration data. Therefore, they should not be used uncritically. bb) Résumé To conclude, duration no doubt is an important element of the quality of a civil justice system, but one should not be betrayed by the seemingly objective and unambiguous numbers. In theory as well as in practice, measuring duration proves problematic. Talking about the quality of process more generally, not only the factor “duration”, but also the factor “costs”, which is as well a seemingly unambiguous, numerical factor, does not allow for clear-cut answers which would permit a simple and objective comparison. Regarding those factors which are of inherently procedural nature like procedural fairness or legitimacy—factors which we have only mentioned but not discussed—an additional difficulty evolves: Researchers must find a viable proxy which is convenient to assess. In view of these difficulties, it may be better to have resort to subjective evaluations, that is, the way how a civil justice system is being perceived. We will, therefore, now turn our attention to perception data. 2. Perception Data A leading English criminal case2 is the origin of the famous aphorism “justice should not only be done, but should manifestly and undoubtedly be seen to be done.” This aphorism alludes to both the objective and subjective approach to evaluating judicial quality. While the requirement that justice be done is put as an uncontested basis, it emphasizes the importance that this is also perceived to be the case by the general public. a) Advantages of Using Perception Data Perception data had become the preferred source of data for the “growth and institutions literature” in the years around the turn of the millenium. And indeed, using such perception data has important advantages: 2 The King v. Sussex Justices, Ex parte McCarthy, [1924] 1 KB 256, 259 (Lord Hewart, C.J.). 6 Perception, Performance, and Politics First, it relieves from the burden of collecting correct objective data, which may be fraught with difficulty. Whether a civil justice system leaves enough room for party autonomy, or whether the judiciary is neutral and independent is hard to determine from an objective perspective, and any proxy researchers might use is susceptible to strong theoretical criticism. On the other hand, asking a certain number of people about such aspects and aggregating their answers is relatively simple. Once their answers are correctly reported, attacks on the validity of the data must be limited to the formulation of the questions and other attendant circumstances. Second, the questions can be framed in a way that covers more than one aspect, so that potential interdependencies are directly captured. For example, a question can cover fairness and impartiality at the same time, although fairness could be seen to focus more on procedural law, while impartiality could be seen to focus on how the judiciary is acting within the procedural framework. A question covering fairness and impartiality should be able to capture a potential interdependency. Third, perception data is considered to reflect both rules and their implementation, that is, law in action. This is seen as an advantage compared to such objective data which is based on the mere existence or nonexistence of certain rules in a country’s statutory law. Some scholars go so far as to say that perception data is the “best available measure for assessing the quality of a country’s institutional environment.”3 b) Problems with Perception Data As you may have expected, we deem this statement overly optimistic. Only some of the challenges and problems with perception data can be mentioned here. A first and fundamental decision concerns the question whose perceptions should matter at all. Is it the whole population of a country, the business community, or maybe only the actual users of the legal system? Should we prefer the perceptions of insiders or outsiders, or even use both? One would say that this needs to be decided on a case by case basis, depending on the focus of the respective survey. However, in practice it seems that this decision is not always being taken with due regard to the context of the study, but 3 Daniel Berkowitz, Johannes Moenius & Katharina Pistor, Legal Institutions and International Trade Flows, 26 MICH. J. INT’L L. 163, 195 (2004). 7 Perception, Performance, and Politics rather pragmatically, that is to say depending on what data is easily accessible or available. Just to mention some other difficulties in collecting perception data, the number of respondents should be high enough, the questions should be clear and unambiguous and—what is particularly relevant in cross-country surveys—correctly translated. Not only the collection, but also the interpretation of perception data is all but plain sailing. The first point to be made is that the subjective impressions contained in these data do not necessarily reflect actual realities. Perception data should be taken for what they are, that is, information about what people perceive to be the case, and not information about how something really is. More important, respondents’ answers may always be influenced by other factors which are hard to identify and isolate. Imagine, on one hand, the loyal citizen who does not want to see his or her country among the “losers” and therefore describes the situation in too positive a way, on the other hand, the political activist who uses every means to press for change and therefore describes the situation in too negative a way. But even apart from such cases, external factors can heavily influence the results, and this becomes particularly important in cross-country comparisons: The subjective impression of respondents about a certain aspect clearly depends on the amount and quality of information which is available. Availability of information, in turn, depends a lot on the efficiency of public agencies and media coverage. It cannot be seriously contested that countries differ greatly as to these respects. Finally, it is not only the availability of “neutral” information, but also the opinion making by diverse institutions pursuing their own interest, which may bear heavily on the perceptions of respondents. Experts’ views become principally more important when it is difficult for non-experts to evaluate a complex situation on their own. One field in which nonexperts have to rely on external advice is just the law. Whether a certain clause in a contract, a certain legal rule or a complete contract or even a legal system as a whole are advantageous, is often beyond the everyday experience of non-lawyers and not seldom difficult to notice even for those in the legal profession. Under such circumstances, powerful marketing and lobbying can strongly influence respondents’ perceptions, possibly in such a subtle way that they do not even recognize it themselves. 8 Perception, Performance, and Politics c) Résumé As a conclusion from our discussion of perception data, it seems fair to say that these data have certain advantages, especially the relative ease of their collection and the fact that doubtful proxies normally need not be found. However, defining the group of respondents, drafting unambiguous questions and interpreting the results in a correct way are a challenge. At the very least, it should have become clear that only a limited degree of accuracy and reliability can be achieved. III. Recent Surveys and the Role of Politics This being said about the difficulties of performance and perception data in crosscountry comparisons of judicial quality, let us in the remainder of this paper discuss more directly the recent surveys and the role of politics. It is important to note that the recent surveys have mostly been conducted by economists in the institutionalist tradition. These economists conceive civil justice systems as part of a country’s institutional environment. The institutional environment, in turn, is seen as a very important, if not the most important, factor for a country’s political and economic development. Under this approach, economists are interested in the quality of a civil justice system because the better the functioning of that system, the higher the economic prosperity will be. A good civil justice system enhances the confidence in the legal system as a whole, in the respect of the rule of law and the protection of property rights. It thereby stimulates private economic activity, which is considered the motor of economic growth. Against this broader background, what matters is not so much comparative data on judicial quality in isolation. Rather, these data become really interesting only if combined with other data. The ultimate aim is to learn more about causal relationships between institutions and development. Once we know—the optimistic reasoning goes—which institutions have the most beneficial effects, all we need to do is to change our legal framework accordingly. In this perspective, there are three different uses for comparative data on the quality of civil justice systems. First, it can be used to establish that judicial quality is indeed a factor relevant for growth. Basically, the idea is that if we can observe a close “relationship” between data on judicial quality and data on development in most 9 Perception, Performance, and Politics countries, this suggests that there also be a causal link. Second, comparative data on judicial quality can be used in research directed at finding out what level of judicial quality is caused by different types of institutional design. Such research typically concludes in a recommendation for reform, true to the above-mentioned idea that once we know which institutional design generates the best result, all we have to do is to amend our system accordingly. Third, research not focusing on judicial quality but on other elements of the institutional environment may use this information as a control for the interpretation of that other data. In this case, the hypothesis is that a certain characteristic depends on a factor different from judicial quality. Researchers verify this hypothesis by determining whether there is a correlation between that characteristic and judicial quality. 1. The so-called “Lex Mundi Study” on Procedural Formalism The so-called “Lex Mundi Study” on Procedural Formalism4 is an example of a survey of the second type in that it aims at proving a relationship between the level of procedural formalism and the quality of the judicial system. The data on procedural formalism and on duration have been collected from law firms by the researchers themselves. The data on judicial quality, except for duration, stem from various external sources, among them, indices provided by Business Environmental Risk Intelligence and PRS Group, which are private enterprises, and the World Bank’s World Business Environment Survey. There are various points which are problematic with this study. 5 It relies on only two model cases, check collection and eviction of a residential tenant, which cannot be considered representative; it uses a debatable concept of procedural formalism and asks ambiguous questions; it links its results to a questionable classification of legal origins and, above all, it does not fully disclose its weaknesses, but sells its results as objective truth and uses them as a basis for a policy recommendation. Taking all this together and considering the tendency of other studies by the same authors, the Lex 4 Simeon Djankov, Rafael La Porta, Florencio López-de-Silanes & Andrei Shleifer, Courts, 118 QUARTERLY JOURNAL OF ECONOMICS 453 (2003). 5 For a comprehensive discussion, see CHRISTOPH KERN, JUSTICE BETWEEN SIMPLIFICATION AND FORMALISM – A DISCUSSION AND CRITIQUE OF THE WORLD BANK SPONSORED LEX MUNDI PROJECT ON EFFICIENCY OF CIVIL PROCEDURE (2007) (for a short presentation, see the website of the publisher: http://www.mohr.de/cgibin/search.pl?lang=d&feldval=&vg=r&sid={SID}&range=0&detail=4776). 10 Perception, Performance, and Politics Mundi Study is not above suspicion of hidden benchmarking. It is here where politics come into play: The studies sponsored by the World Bank serve as a basis for recommendations to developing countries or, even more directly, as a basis for specific requirements and conditions on how a borrowing country has to reform its legal system. The distinguished audience may itself imagine whose and what kind of politics one can suspect to play a role here. 2. The 2008 Business Survey Conducted by Clifford Chance and the University of Oxford’s Institute of European and Comparative Law The brand new 2008 Business Survey conducted by Clifford Chance and the University of Oxford is a survey which does not fully fit into the three categories of cross-country comparisons introduced above. It is, as we understand it, mainly a survey about causation, more precisely: causation between the civil justice system, the choice of forum, and the choice of law. Do businessmen choose the governing law for their contract because they are convinced of that law’s superior quality? And if so, do they have a preference for one particular country’s contract law? If this were the result of the survey, one would feel tempted to conclude that those countries whose contract law is not so attractive should consider a reform of their law according to the example of the preferred contract law, or that a supranational legislator should take the preferred contract law as a model. If the result of the survey did not fully support the hypothesis that businessmen choose the governing law for its perceived quality, the next question is: Do businessmen choose the governing law not for this law’s attractive characteristics, but for the fact that it is the law of a preferred forum? If this were the result, the choice of a particular contract law would not say much about this contract law’s quality. Rather, it would suggest that it is the forum which matters. The next question then would be: If so, do businessmen prefer a certain forum because of the quality of the civil justice system in this forum, or for other reasons, like the geographic and logistic convenience of this forum, the language spoken in that forum, and so on? If it were the quality of the civil justice system, one would again feel tempted to give a policy recommendation to all the other countries or a supranational legislator, in the sense that they should use 11 Perception, Performance, and Politics the civil justice system of the country which is the preferred international forum as a model for future legislation. Yet is this really the correct way to interpret the survey, whatever its concrete results are? Here again, politics come into play. We have already mentioned that nonexpert respondents may to an important degree be influenced by opinion statements from interested persons or institutions. This is particularly true for the questions the survey is dealing with, that is, choice of law, choice of forum, and judicial quality. Let me illustrate this in somewhat more detail. Remember first that legal issues, in particular such of a more general nature which need a good deal of knowledge and practical experience, are not easily accessible even for those engaged in the business world. Their evaluations must be seen from this background. Under these circumstances, it is not too difficult for interested parties to influence the perception of non-experts. And that is exactly what is being done. A great example is the brochure published by the Law Society of England and Wales, entitled “England and Wales: The jurisdiction of choice”.6 This brochure is a paean of praise for the lawyers, courts, and contract law of England and Wales. It contains various attacks on civil law systems which are not backed by factual information,7 presents legal rules which can be found in many modern legal systems as if they only existed in England8 and relates a handful of anecdotes lauding the English courts, lawyers and arbitrators.9 It concludes with model choice of law and jurisdiction clauses pointing, for sure, to England or Wales. This is not to say that English law and English courts do not deserve highest esteem. We rather cite this brochure as an example of powerful law marketing. Such law marketing would not matter if all countries were engaged in this activity to a similar degree. However, the continental legal tradition has been skeptical towards any 6 Available at http://www.lawsociety.org.uk/documents/downloads/jurisdiction_of_choice_brochure.pdf (last visited February 11, 2008). 7 Id., at 8 (claiming, inter alia, that English law gives guidance “with greater certainty than in many civil law systems”, that “English law is based on the principle of freedom of contract which is more flexible than many civil law systems”—no freedom of contract in the civil law?). 8 Id., at 9 (noting, e.g., that “[a]n English judgment can be easily enforced within the European union by virtue of the Brussels 1 Regulation and the European Enforcement order”). 9 Id., at 9-13, 15-18 (“case studies”). 12 Perception, Performance, and Politics forms of advertising in the legal world, and in particular has not been familiar with the conception of marketing its own law and its own courts as a product in order to generate business for legal professionals. Simplifying a lot, French law has been successful because it was perceived as containing the ideas and the spirit of the French Revolution; Italian law owes its role in Latin America to its scholars; German law has had some influence for its technical sophistication. In none of these countries, the legal profession engaged in means to attract business from abroad. This is completely different in common law countries. Suffice it to mention Delaware! It would be naive to believe that this situation, in the case of England coupled with its historical significance and reputation as a place of international commerce, would not influence perceptions in the business world. Moreover, most global law firms are dominated by English or American partners. These firms took advantage of a liberalization of the legal profession which occurred in continental Europe in the last decades of the 20th century. In Germany, as an example, it was the influence of European Law in concert with the German Constitutional Court, and in particular a judge who owed her post to the social democrats, which opened the legal services market to foreign and international law firms. Liberalization and deregulation may not seem to be characteristic of socialdemocrat politics. However, what only few have recognized to date, this was a very radical enterprise against the establishment. Be this as it may, today many businesses rely on law firms dominated by English or American partners. It cannot surprise that these firms advise their clients to choose the law of a common law country and a common law jurisdiction, thereby complicating business for the remaining national and regional firms and thus securing themselves a competitive advantage. Of course, this is not what counsels tell their clients, and it will rarely be their exclusive motive. But the mere fact of advising to choose such law and jurisdiction may be another reason for clients to believe in the superior quality of this legal system and courts. These two examples should suffice to illustrate how external factors can considerably influence perceptions. As the audience may have guessed, from our point of view one should be careful to interpret a survey like the one we are discussing here in the sense indicated above. 13 Perception, Performance, and Politics IV. Conclusion After these somewhat provocative positions designed to stimulate the debate, let me conclude with the words of three scholars engaged in the field, according to which “[c]ross-national studies … are just a beginning that point us in the right direction”10— probably not less, but also definitely not more, I would like to add. This kind of research should not be blocked. But it should not be considered the Holy Grail either. It is the task of the legal profession to keep an eye on how statistical means are applied to the law, and, even more importantly, on how the results are interpreted. 10 Dani Rodrik, Arvind Subramanian & Francesco Trebbi, Institutions Rule: The Primacy of Institutions over Geography and Integration in Economic Development, 9 J. ECON. GROWTH 131, 158 (2004). 14