THE FRAGILE COMPLEXITY OF
REGULATORY QUALITY:
DEFINITIONS AND
MEASUREMENT
Claudio M. Radaelli
Centre for European Studies
Presentation delivered to CARR,
Centre for the Analysis of Risk and
Regulation, LSE, 16 March 2004
1
IRQ – Indicators of
regulatory quality
Project:
• Stage 1 Review of
initiatives and indicators
used by EU member states
to measure regulatory
quality (plus others with
interesting experiences)
• Stage 2: Elaboration of a
set of indicators to be used
by the European
Commission
• Timing: 14 months, interim
report in May 2004, report
on indicators in September
2004
IRQ Team
• Bradford team (Radaelli,
De Francesco, Cameron,
Jalilian)
• Advisor: Scott Jacobs
• Advisor: Bruno Dente
Support network:
27 national experts on
‘better regulation’ policy
from 16 EU M.S. (incl. 6
CEECs)
2
Aims in this presentation
To discuss regulatory quality, the quality
of indicators, and indicators of quality.
Proliferation of OECD and DBER
initiatives… ‘pie-chart’ mania
To collect feedback, suggestions, and
information
3
Table of contents
1. Getting to grips with the concept: what
is regulatory quality?
2. Complexity of regulatory quality
3. How does one measure quality?
4. Classification of indicators
5. From classification to the analysis of the
performance of indicators and system of
indicators
4
1. Regulatory quality (RQ)
• Governments
• How the EU defines RQ
• Definitions provided by international
organisations
• Academic approaches
5
UK: Better Regulation Task
Force
Proportionality,
accountability,
consistency,
transparency, and
targeting
6
Commission
• RQ is part of the strategy for better regulation
and good governance. This implies that, in
addition to properties such as efficiency,
effectiveness, coherency, simplicity, and clarity,
good regulation should also respect:
• Standards for the re-vitalisation of the acquis (COM71
Final,) Commission 2003
• Standards for the democratic use of expertise
• Standards for consultation
• And most importantly good regulation balances
environmental protection, social conditions, and
economic growth sustainable development
7
Mandelkern Report
(Council’s experts – EU)
Common principles:
•
•
•
•
•
•
•
Necessity
Proportionality
Subsidiarity
Transparency
Accountability
Accessibility
Simplicity
MR is focused also on how to achieve RQ. The main tools are:
•
•
•
•
•
Alternatives to regulation
IA
Consultation
Simplification
Access to regulation
8
OECD: evolution of ideas
on RQ
• 1995: Good practices -> standards -> checklists
• 1997: Three stages of development of regulatory reform: deregulation, regulatory quality (instrument by instrument approach),
and regulatory management (more horizontal)
• 2002: From RQ to regulatory governance (transparency, citizens as
partners, and bringing principles of better regulation across levels
of governance and regulatory actors (legislature, judiciary) 
grounded in the wider approach to good democratic governance 
two main perspectives on regulatory quality: economic and juridical,
both correlated with the design and implementation of policies, tool
and institutions
9
Academics: Baldwin and
Cave
 They derive RQ from the claims to
legitimacy [of regulation]:
Legislative mandate
Accountability
Expertise
Efficiency: the legislative mandate is
implemented correctly
10
Baldwin, Hood, and Rothstein 'Assessing the
Dangerous Dogs Act: When Does a Regulatory
Law Fail?' Public Law Summer (2000)
 Principles do not help much. Paradoxically, one can
demonstrate that even bad legislation can pass the
test of complying with the principles of the Better
Regulation Task Force.
 We need to evaluate the results achieved by
regulation rather than use principles and key-words.
More emphasis on systematic evaluation (ex-ante
and ex-post) and on effective implementation of
proposed regulation
 Indicators, not slogans?
11
“Economists”
They take issue with the narrow notion of
efficiency presented by Baldwin and Cave
They have a different benchmark:
maximisation of the collective welfare
Quality regulation is about the economic
impact of regulation, not about
implementing the mandate
12
Dimensions of quality
 Economists: They do not talk about RQ, they
talk about the efficiency of regulation.
 But efficiency is only one element of quality or
one claim to legitimacy (in Baldwin and Cave’s
typology).
 Another element of RQ is the quality of the
process through which regulations are
produced.
 Doubt: do economists measure quality or
quantity (extent of regulation)?
13
To conclude on RQ:
Advantages and limitations
• RQ tells us something
about quality of
governance
• RQ is neutral with respect
to size of government or
scope of intervention
(claim made by OECD).
Hence it does not express
a judgment on how big
government should be.
Unless one follows the
economists and
‘confuses’ quality and
quantity.
• The notion of RQ is a bit
fuzzy
• There are too many
attributes in the current
definitions of regulatory
quality, but how does one
recognise regulatory
quality in the real world?
What are our empirical
sensors?
14
How our project
approaches RQ
• Design of the process (structures,
inter-organisational linkages, etc.)
• Activities and outputs
• Real-world outcomes
15
The puzzle of real-world
outcome
 Cross-country differences revealed by indicators: do they really
mean differences in policies and their enforcement? Or do they
simply register different models of capitalism (Hall and Soskice)?
 Jaffe et al, JEL 1995: the evidence of the impact of regulation on
competitiveness and locational choices of firms is far from clear.
Same results for environmental regulation. But see Koedijk, K. e J.
Kremers (1996), Market opening, regulation and growth in Europe,
Economic Policy. Regulatory governance matters
 Literature on regulatory competition: evidence tells us that
countries do not compete by using regulation in the way predicted
by simplistic economic models of ‘races’
 OUR SUGGESTION: for us real-world outcome makes more sense if
we look at changes in the behaviour of regulatory stakeholders.
Regulatory governance as learning
16
Tools for RQ
Impact assessment
Consultation
Simplification, codification, recasting
Access to legislation and regulatory
transparency
Ex-post evaluation of regulatory tools and
institutions
17
2. Complexity of RQ
Quality is not a monolithic concept
Quality for whom?
(A) Context shapes RQ
SEE PAPER ‘HOW CONTEXT MATTERS’ (Radaelli)
(B) Models of the policy process
(C) Models of actors
(A), (B) and (C) are often neglected in the
one-size-fit all approach to RQ
18
How different stakeholders look at
RQ
Economist
Civil
Servant
Politician
Firm
Citizen
Criteria
Efficiency
Conformity
to rules
Consensus
Cost
Minimisation
Costeffective
protection
from risk
Success
Goals
achieved
Following
procedures
Outcome of Profit
negotiation
‘Enabling’
regulation
Logic of
action
Logic of
social
sciences
Standard
operating
procedures
Logic of
Logic of
Negotiation influence
Logic of
participation
19
Implicit models of actors and implicit
theories of the policy process
Different stakeholders bring into the
discussion diverse logics and criteria of
quality and quality assurance
mechanisms.
It is not clear what is the model of
stakeholders implicit in RQ programmes.
On theories of the policy process see the
paper HOW CONTEXT MATTERS
20
Implicit theories (2)
 Are these programmes based on the assumption that
politicians are rent-seeking, hence quality assurance
mechanisms should target this problem?
 Or does ‘quality’ mean that a government is trying to curb
bureaucratic power? Is RQ an approach to limit the power of
business in the policy process or an instrument to provide
more systematic access of corporate actors to the regulatory
process?
 What do corporate actors want in a regulatory process?
Efficiency or protection? Profit maximisation or satisfying
behaviour? Are they a unitary actor or in competition among
themselves?
21
Implications
It is important to acknowledge the
complexity of RQ up-front.
It is indispensable to discuss the criteria,
theories, and models of actors upon which
our approach to RQ is based.
22
3. How does one measure
quality?
 Tests
Indicators:
Quality of the RIA process
Quality of the regulations produced via RIA
23
4. Classification of
indicators
 LEVELS
 TYPES OF MEASURE
 TOOLS
 PROVIDERS
 TYPES OF REGULATION
 LEVELS OF
 SECTORS OF
REGULATION
 DEGREES OF
SOPHISTICATION
MEASUREMENT
 PURPOSES
 PRINCIPLES
24
BY LEVEL
Design of the process
Activities and Outputs
Real-World Outcomes
25
BY TOOL
Impact assessment
Consultation
Simplification, codification, recasting
Access to legislation and regulatory
transparency
Ex-post evaluation of regulatory tools and
institutions
26
BY TYPE OF REGULATION
ECONOMIC REGULATION
SOCIAL REGULATION
ADMINISTRATIVE REGULATION
27
BY SECTOR OF
REGULATION
 Regulation of Entry
 Regulation of Labor
 Environmental Regulation
 Product Regulation
 TAX and Social security law
 Planning regulation
 Etc…
28
BY DEGREE OF
SOPHISTICATION
 Simple measures
 Composite measures (Factor analysis, aggregation
of simple indicators on the basis of weighting).
 Note that composite measures are not necessarily
better than simple measures. For example, factor
analysis may not be explicit enough on the variables
that can be controlled by program administrators.
Or it can be the statistical artifact of a questionnaire
with too many questions.
29
BY TYPE OF MEASURE
Objective
Subjective
30
BY PROVIDER
Governments (self-assessed
responses to regulatory quality
questionnaires)
Experts (think tanks responding to
Word Bank questionnaires or
reviewing IA)
Stakeholders
31
BY CHARACTERISTIC
Categorical (ex. Dichotomous
(Yes/No);
Ordinal (ex. 1 to 5 scales)
Continuum (ex. Number -- no. of
procedures, steps, etc.; Ratios;
indexes)
32
BY PURPOSE: INDICATORS
FOR WHAT?
Learning
Explanation
Transparency and accountability
Program management
Participation and ‘good regulatory
governance’
33
By PRINCIPLE
of better regulation policy
 Australia
 Canada
 UK
 US
 Academics
 Mandelkern Report
 OECD
 See Section 1 on the concept of RQ for the list
of principles
34
An example: combining
values and levels
Values
Process
Activity
Output
Real World
Necessity
X
X
Proportionality
X
X
Subsidiarity
X
(X)
Transparency
X
Accountability
X
X
Accessibility
X
X
Simplicity
X
X
Consistency
X
X
(X)
35
5. From classification to the
analysis of performance
•
•
•
•
Assessing quality of individual indicators
Assessing quality of systems of indicators
Controlling for adverse effects
Crucial step: Managing indicators – using
indicators to improve management, to learn,
and to produce better regulation.
36
From classification to the assessment
of quality of each indicator (1)
EASY STEPS
 Validity (relation between indicator and
phenomenon)
 Quantifiable at regular intervals, so that the
indicators can be monitored
 Reliable (two different people produce the same
value given the same conditions)
 Comparability and cross-checkable
 Easy to gather, easy to communicate
37
From classification to the assessment
of quality of each indicator (2.1)
ENTERING COMPLEXITY
 Efficient: a sophisticated indicator (arising out of factor
analysis) contains more information than simple
indicators.
 However, there may be a trade off between the quantity
of information contained in an indicator and how easy it
is to communicate it within the organisation and to the
stakeholders.
 It is important that the variables upon which the
complex measure is built are easy to understand and
coherent (no trade-off between the variables).
38
From classification to the assessment
of quality of each indicator (2.2)
 Relation with principles or norms. This is to avoid
ambiguity. The indicator relates to an outcome that can
be unambiguously judged satisfactory or not.
 Do we measure the quality of the regulatory
environment or the quality of better regulation
programmes? In the long term they should converge.
In the short-term, the environment can improve
independently from the policy. Example: more people
employed to issue licenses, the time needed to get a
licence goes down, but the overall regulatory complexity
is the same.
39
From classification to the assessment
of quality of each indicator (3)
ENTERING PRINCIPLES
 Take the example of principles of better regulation.
Think of participation. Is this a dimension of
quality? It depends on why one is designing an
indicator
Always ask the question: if the indicator is the solution, what is
your problem? Do you want to learn, to manage, to steer, to
empower people,….?
This example tells us that the definition of quality is
sensitive to the main purpose one has in mind when
developing the system of indicators.
40
From classification to the quality of
systems of indicators
• Coverage … but also simple and selective
• Good balance among design, output, and real-world
outcome. Good balance between providers (stakeholders
and governments)
• The system should be targeted and focused on those
themes in regulatory reforms programmes that have
large implications for decision-making
41
Dynamic systems of
indicators
• Start with simple system of indicators, then in year two add more
sophisticated indicators
• Use indicators to allow member states to measure their own
progress year by year and in relation to other member states.
Indicators and scoreboard of regulatory quality? Drawback: learning
may be hampered by the ranking syndrome. This is not a race, but
a mutually beneficial exercise.
• Use two sets of indicators. A simple set of those who use indicators
to monitor complex regulatory reform programmes and another set
of indicators for those who manage individual programmes.
42