Impact Assessments in Europe
Preliminary report
Ioannis Lianos & Mihály Fazekas
&
The Gutenberg project team
http://www.ucl.ac.uk/cles/research_initiatives/impact_assessments_in_eur
ope
Structure of the presentation
• Motivations for the research – broader research programme
• Objectives of the research project and organization of the team
• Theoretical framework and principal hypothesis
• Results on the typology of IAs
• Data
• Regressions
• Clustering
• Conclusions and further work
2
Motivations for the research
Broad research landscape
• George (1972) multiple advocacy framework
• Impact assessment exercises the function of the
“honest broker”…?
• The need for dialectical exploration of Facts and
Values
• Elaboration of rules and procedures governing the
exchange of normative arguments
• Towards uniform rules of evidence?
3
Objectives of the project and organization of the
team
Intermediary objectives
• develop and explain a robust typology of IAs
– Identify characteristic types of IA across 19 jurisdictions
– Explore the determinants of IA quality
• examine the impact of IAs in the decision-making process and the
relation between various actors in the IA process
• the impact of IAs on the quality of normative production
• explore the emergence of policy areas rationalities in IA and how these
could fit into the general framework of RIA
Ultimate objectives
• understand how the wider environment of (scientific) evidence
appraisal in policy or judicial decision making (the “knowledge
ecosystem”) affects the development of the tool of IA.
• understand how the introduction of the tool of IA transforms the role
set of bureaucrats and more generally “governmentality”
4
The Gutenberg project team
5
Theoretical framework
The evidence/science
politics nexus
Four models of interaction between
science/expertise and politics/law
(Stehr, 2012)
•Expertise drives politics
•Politics drive expertise
•Symbiosis of politics and
expertise
•Politics and expertise as separate
domains (indifference)
• Challenging the distinction
between policy-makers and
knowledge producers
• “boundary work” (Halffman, 2003;
Hoppe, 2009)
• The “argumentative policy
analysis” or postempiricist
alternative (Fischer & Forester,
1993; Hoppe, 1999)
• “participatory policy analysis” –
measurement becomes a “social
process” (Fischer, 2003)
• Different models of boundary
arrangements between science
and politics
6
The politics – science/expertise nexus
Hoppe, 2005
+ the importance of “participatory
policy analysis” in Europe
7
The six hypothetical models of IAs
• The “ticking box” or rudimentary model of IA
• The sophisticated rational-instrumental model
of IA – CBA style
• The shallow rational-instrumental model of IA –
CBA style
• The rational-instrumental model of IA –cost
effectiveness style
• The participatory model of IA
• The symbiotic model of IA
8
Hypothesized IA clusters and their key
characteristics
9
Schematic logic model of impact assessment quality and
its effects on the quality of regulatory environment
10
Distribution of IA types per country, 2005-2012
(weighted, up-scaled)
11
Descriptive statistics of country IA production, 2005-2012 (non-weighted)
12
The data
• Time period: 2005Q1 – 2012Q2
• 18 countries + EU
• Data collection about IAs as well as countries’
institutional frameworks
• Data collection templates can be found in the
report’s appendices
13
The sample and weighting
• Stratified random sampling
jurisdiction/year
Bulgaria
Czech Republic
Denmark
EU
France
Germany
Greece
Hungary
Ireland
Italy
Netherlands
Norway
Poland
Romania
Serbia
Slovakia
Spain
Sweden
UK
Total
2005
0
0
0
0
0
15
0
0
0
0
1
1
2
0
0
12
0
0
0
31
2006
0
0
10
76
0
16
0
0
5
13
1
11
18
0
0
8
0
10
17
185
2007
0
3
12
88
0
15
0
0
4
44
2
9
20
22
6
13
0
10
41
289
2008
0
22
24
114
0
18
0
0
7
16
5
12
20
0
12
11
14
11
17
303
2009
14
18
15
77
3
15
0
0
5
23
1
7
20
0
11
13
12
10
20
264
2010
0
23
22
51
14
16
10
2
9
20
2
10
24
0
9
11
10
9
20
262
2011
0
17
20
119
9
15
12
32
7
9
5
10
21
0
10
13
10
10
11
330
2012*
1
20
21
32
2
18
12
68
12
1
3
10
26
0
11
2
10
10
27
286
Total
15
103
124
557
28
128
34
102
49
126
20
70
151
22
59
83
56
70
153
1950 14
• Weighting to reflect production per year, country
Lead indicators: overview
• High degree of complexity: 125 elementary
indicators of IAs
• Need for parsimony
• Development of 5 lead indicators
–
–
–
–
–
Scope
Sophistication
Pseudo-quality
Consultation
accountability
15
Lead indicators: scope
• Scope score = (
budget + business +
SMEs + consumers +
innovation + employment +
health + competition +
environment + society +
geographic equity +
EU institutions +
single market +
acquis communitaire +
corruption +
administrative burdens +
subsidiarity + proportionality +
implementation + transition) / 20.
16
Lead indicators: scope
bulgaria
• Marked cross-country
differences
0.48
eu
0.44
germany
0.37
uk
0.36
denmark
0.36
greece
0.29
netherlands
0.24
spain
0.23
0.22
sweden
france
0.20
Mean scope
poland
0.18
serbia
0.18
ireland
0.17
slovakia
0.17
czechr
0.16
0.13
hungary
norway
0.11
italy
0.10
romania
0.06
0.00
0.10
0.20
0.30
0.40
0.50
0.60
0.70
17
Lead indicators: sophistication
• Sophistication score = (
cost/benefit categories explicitly stated +
total costs/benefits calculated +
scoring of options +
use of net benefit calculations +
use of cost effectiveness calculations +
at least one alternative policy option +
do nothing option included +
comparison of alternatives +
taking account of uncertainty +
doing robustness checks +
use of simple/complex calculations +
comparison with other countries +
explicit reference to non-monetisable factors/methodological limitations +
explicitly stating a discount rate +
using consistent figures)/18
18
Lead indicators: sophistication
•
Large cross-sectoral differences
0.4
0.34
0.4
0.3
0.31
0.27
0.26 0.25
0.3
0.2
0.2
0.23
0.21
0.21 0.21
0.19
0.18
0.18
Mean
sophistication
0.1
0.1
0.0
19
Lead indicators: pseudo-quality
• Pseudo quality =
scope * sophistication * log length
20
Lead indicators: consultation
• Extensiveness of consultation = (
any mention of consultation +
evidence submitted through consultation +
opinions submitted through consultation +
whether consultation took place after the IA completion or not +
comprehensiveness of consultation +
openness of consultation +
consultation shorter than 60 days +
number of public parties consulted +
number of societal parties consulted +
number of external experts consulted)/13
2. czechr
3. denmark
4. eu
5. france
6. germany
7. greece
8. hungary
9. ireland
10. italy
11. netherlands
12. norway
13. poland
14. romania
15. serbia
16. slovakia
17. spain
18. sweden
19. uk
Total
20 40 60 80
0
20 40 60 80
0
20 40 60 80
0
Percent
0
20 40 60 80
1. bulgaria
0
.5
1
0
.5
1
0
.5
1
0
.5
1
0
.5
1
comprehensiveness of consultation
Graphs by cname
21
Lead indicators:
Sophistication versus consultation
22
Lead indicators: accountability
• Degree of accountability = (
overarching goals +
operational objectives +
implementation plan +
monitoring progress +
ex-post evaluation) / 7.
23
Typology of IAs across Europe:
our approach
• Theory driven, IA focused (no countries, no years)
sophisticscope
ation
consultation
alt.
cost
benefit
account policy
figures figures
options
impact
on state/
budget
rudimentary IA
low
low
low
low
no
some
some
no
rat.-instr. IA: cost
effectiveness
style
low
high
low
low
no
many
no
yes
rat.-instr. IA: CBA
style (shallow)
high
low
low
low
yes
many
many
yes
rat.-instr. IA: CBA
style (sophistic.)
high
high
low
low
yes
many
many
yes
participatory
model of IA
high
low
high
high
yes/no
some
some
no
symbiotic model
high
high
high
high
yes
many
many
yes24
Typology of IAs across Europe:
results
• Statistically robust findings: 6 major clusters+other
• Partial confirmation of hypotheses
alt.
bene
impact
polic cost
fit
on
y
figur
figur state/bud
optio
es
es
get
ns
0.64 0.23 0.36
0.39
sco
pe
sophi
stic
consulta
tion
acco
unt
other IAs
0.25
0.18
0.32
0.38
rudimentary IA
rat.-instr. IA: cost
effectiveness style
rat.-instr. IA:CBA
style (shallow)
participation/acco
untability oriented
IA (no alt. p. o.)
participation/acco
untability oriented
IA (with alt. p. o.)
0.23
0.07
0.19
0.14
0
0
0
0.37
0.12
0.11
0.18
0
1
0.33
0.18
0.41
0.23
0
0.19
0.10
0.73
0.68
0.18
0.22
0.78
0.67
symbiotic model
0.43
0.47
0.50
0.45
weighted
Freq.
%
1845
12.15
0.35
1304
8.59
0
0.89
1627
10.71
1
1
0.79
3551
23.38
0
1
0
0.32
1944
12.80
1
1
0
0.38
1322
8.71
3594
25
23.67
1
1
1
0.71
Regressions: theory
• Logic map of causal links
Institutional
environment
Macro-level
Micro-level
Impact
assessment
characteristics
Impact
assessment
quality
26
Regressions: variables
• Dependent variable: pseudo-quality
• IA characteristics:
1.
2.
3.
4.
5.
area of regulation,
type of legal text,
preparing institution,
consultation score,
accountability score.
• Country-level institutional environment characteristics-IA-specific
institutions:
1.
2.
3.
4.
existence of a central IA body checking IA quality,
existence of a national IA research institute,
legal requirements to include certain impacts such as competition, environment,
legal requirements to include at least one alternative option.
• The country-level institutional environment characteristics-general
quality of government variables:
1. World Economic Forum wastefulness of government spending (0-very wasteful, 6not wasteful at all),
2. length of time government is in power.
27
Regressions: Main effects
Level-1: IA
Level-2:
yearcountry
Effect
heterogeneity
Main
effect
consultation index
x
++
consultation index (squared)
x
(--)
accountability index
x
+++
central IA bod (=1)
x
national IA research inst. (=1)
x
required scope index
x
mandatory alternative opt (=1)
x
++
govt. years in power
x
+
WEF: wasteful gov. spending
x
-
WEF: wasteful gov. spending
(sq.)
x
(+)
Controls*
x
x
++
++
x
-
+/28
Regressions-selected predictors: fixed
effects:
WEF wastefulness of gov spending
29
Regressions-selected predictors: random
effects
year-country predictions
EU
BG
UK
FR
NL
CZ
30
Regressions-selected predictors: random
effects
Heterogeneous effect of IA Board-type institutions
31
Detailed results
optimization method: maximum
likelihood, constrained
covariance structure
dep var: pseudoq
model nr.:
Regression coefficeints (fixed
effect)
Constant
Level-1 factor: individual IA
controls
consultation index
consultation index (squared)
accountability index
IA
IA
IA
characteristics
characteristics
empty'
characteristics
+IA institutions
+IA institutions
model
+IA institutions
(random
(random
(random slope)
intercept)
slope)+pol inst
1
0.0346
***
-3.95
2
3
4
0.0304*
0.0329**
0.0364***
-2.29
-2.84
-3.55
0.0320***
-4.07
0.0083
-0.28
0.0754***
-9.96
0.0321***
-4.15
0.00587
-0.2
0.0747***
-9.84
0.0223**
-3.12
-0.0232
(-0.83)
0.0821***
-8.71
32
Detailed results
Level-2 factor: year-country cells
central IA bod (=1)
national IA research inst. (=1)
required scope index
mandatory alternative opt (=1)
government cycles
first 4 years
second 4 years
third 4 years
WEF: wasteful gov spending
0.0139
-1.85
0.0326*
-2.21
-0.00506
(-1.69)
0.0245*
0.00899
-1.04
0.0288*
-2.24
-0.00467
(-1.69)
0.0222*
0.00224
-0.3
0.0261*
-2.57
-0.00594**
(-2.64)
0.0186*
-2.03
-1.98
-2.11
ref.cat
0.00830*
-2.28
-0.00927
(-0.95)
-0.00999*
(-2.03)
33
Detailed results
Variance components (random
effects)
Level-3: country
var(required scope index)
0.00001
0.00000
var(centraly IA bod (=1))
0.00024
0.00014
0.00047
0.00045
0.00001
0.00000
var(_cons)
0.00132
0.00087
Level-2: year-country cells
var(required scope index)
var(_cons)
0.00039
0.00023
0.00000
0.00000
var(Residual)
0.00244
0.00225
0.00224
0.00109
N
1947
1864
1864
980
R2
n.a.
23.4%
23.5%
23.9%
-6012.9
-5937.2
-5966.4
-3829.8
Model summary
Deviance
t statistics in parentheses
* p<0.05, ** p<0.01, *** p<0.001
34