HOW DO TOP PUBLIC OFFICIALS EVALUATE PUBLIC SECTOR REFORMS?
FINDINGS FROM THE COCOPS TOP EXECUTIVE SURVEY1
Draft – 23 January 2014 – very preliminary – do not cite
Steven Van de Walle, Gerhard Hammerschmid, Vid Štimac, Anca Oprisor
The research leading to these results has received funding from the European Union’s Seventh Framework
Programme under grant agreement No. 266887 (Project COCOPS), Socio-economic Sciences and Humanities.
1
COCOPS survey
HOW DO TOP PUBLIC OFFICIALS EVALUATE PUBLIC SECTOR REFORMS?
FINDINGS FROM THE 16-COUNTRY COCOPS SURVEY
Abstract
This paper seeks to comparatively and quantitatively assess the several dimensions of impact
of New Public Management (NPM)-style reforms in European countries as perceived by top
public sector officials. Using the COCOPS Top Executive Survey (16 European countries, N=
5,199), we look at the impact of typical NPM reforms such agencification, contracting out, a
customer orientation, flexibilisation of employment, and downsizing on the perceived
intermediate specific effects and final overall performance of public administration. More
specifically, we distinguish between four dimensions of effects, in order to capture both
intended and unintended effects: cost efficiency, service quality, equal access to services and
policy coherence and coordination. Using structural equation modelling, findings reveal
individual-, organisational, and –country-level factors explaining effects.
Points for practitioners
For practitioners, this paper provides an in-depth perspective on how top public sector
executives themselves perceive the extent of public sector reforms and the effects of such
reforms on a number of dimensions. It allows them to see perceived weaknesses in individual
countries’ reform strategies and outcomes, and allows them to compare their own experiences
to that of top executives in other countries.
Keywords: Public sector reform, top officials
2
COCOPS survey
HOW DO TOP PUBLIC OFFICIALS EVALUATE PUBLIC SECTOR REFORMS?
FINDINGS FROM THE 16-COUNTRY COCOPS SURVEY
Introduction
Because of their multifacetedness, public sector reforms are difficult to evaluate. In addition,
they suffer from incomplete specifications prior on the reform, shifts in the reform goals
during the reform process, an absence of pre- and post-tests design or even comparable data
before and after the reform. For this reason, existing evaluations of public sector reform tend
to be highly impressionistic, or focused on just very limited aspects of these reforms and a
very limited set of outcomes.
Our approach to evaluating the effect of NPM-style reforms is the following. We use the
perceived extent to which certain reforms have happened in the policy area of respondents as
independent variable. Subsequently, public officials have been asked to rate how public
administration has performed in their policy field on a number of dimensions over a five year
period, and more specifically whether they thing it has deteriorated or improved (intermediate
effects). They were also asked to indicate how the overall performance of public
administration in their country has changed compared with five years ago (final effect). This
allows us to do things at the same time. One is to assess the impact of specific types of NPM
reforms on the performance of public administration. The second is to assess which aspect of
performance have improved or deteriorated, and how these elements influence the overall
assessment of changes in public administration.
Effects of reform – preliminary literature review
In a recent meta-study of evaluations of NPM-style reforms in European countries, Pollitt and
Dan have been very critical about the validity and reliability of many of these studies.
Overall, they found ‘there is an ocean of studies of the application of NPM ideas within the
Europe, but only a modest sea of works that offer direct empirical analysis of outputs, and no
more than a small pond that convincingly connect specific reforms to particular
outcomes’(Pollitt & Dan, 2011: 52 – deliverable 1). In order words, most material tends to
look at changes in processes only, and that most statements about outcomes tend to be
generalisations rather than empirically sound statements. This is in a way unavoidable, given
the relatively broad and undetermined nature of NPM-style reforms (See also, Van de Walle
& Hammerschmid, 2011). Theoretically, NPM can be described quite elegantly with the
terminology from principal –agent and public choice frameworks. In practice, we see a
variety of reforms. Almost all of them have been labelled NPM reform, even when they do
not theoretically fall under NPM. Some of them show a clear theoretical affiliation with NPM
3
COCOPS survey
principles but have explicitly not been labelled as such. Furthermore, reform practice and
reform rhetoric remain difficult to distinguish.
Evaluating specific reforms is supposedly easier. Yet, also here, reform effects are influenced
by context factors, and by other reforms that may happen simultaneously. Unlike overall
evaluations of NPM-reform packages, the literature is richer when it comes to specific
evaluations and assessments. As some of the authors of this paper have argued elsewhere
‘Evaluating the impact of NPM-style reforms requires a disaggregation of both reforms and
effects into their constituent parts’ (Pollitt 2002; Andrews & Van de Walle, 2013: pp 767).
Some specific bodies of literature have done so. Examples include Alonso et al.’s (2013)
work on the effects of outsourcing on public sector size and employment, also part of the
COCOPS project. Similarly, in the research on agencification concerns have been raised
about the effects of agencification on vertical and horizontal integration of the public sector
and on overall cohesion in policy and implementation (Schick, 1996; Dan 2014). Evaluations
of agency reform have tended to focus on internal processes, and have only to a limited extent
focused on outcomes and outputs (Dan, 2013). To be expanded….
In evaluations, the outcome dimension of public sector reform is generally covered through a
very limited set of indicators. Cutting costs and improving efficiency has been the most
important objective of NPM reforms (Hood 2011) Many assessments of public sector reforms
have thus tended to look at just this dimension of effects (See Andrews, 2010). There are
fewer studies that have looked at a range of outcomes simultaneously. Such more
comprehensive accounts include Boyne et al.’s (2003) evaluation which has looked at three
dimensions: responsiveness, equity and efficiency; or Andrews & Van de Walle’s (2013)
study on NPM’s effects on efficiency, responsiveness, equity and effectiveness in local
government.
Existing analysis has mainly concentrated on intended effects of reforms, which generally
include dimensions such as cost, efficiency etc. Unintended and non-economic dimensions of
performance have received far less attention (see Pollitt, 2002). The economics literature has
indeed posited the existence of some kind of trade-off between efficiency and equity (Okun
1975). Critical accounts of NPM have raised concerns about potential detrimental effects on
democratic accountability, equity, or access, yet often come with limited systematic empirical
evidence. Examples to the contrary include analyses in behavioural responses to choice (Jilke,
2014).
Likewise, some authors have looked at the effects of reform on employment conditions in the
public sector. In many countries reforms have been oriented towards hollowing out the
4
COCOPS survey
traditional career-based model of public employment, and towards introducing position-based
employment with more flexible employment and a mainstreaming of public sector
employment conditions (Bach et al., 1999). At the same time, marketisation, privatisation and
outsourcing have turned many previously public sector jobs into private sector jobs.
Much of the existing material has concentrated not so much on reforms in the inner core of
government, but on liberalisation and privatisation. Jilke and Van de Walle (2013) assessed
whether public service liberalisation has had an impact on citizens’ complaining behaviour
and whether this was influenced by socio-economic status. Brau et al. (2010) have looked at
effects of utility privatisation on price and consumer satisfaction. Our analysis in this paper
however focuses exclusively on central government.
Data: the COCOPS survey
It presents survey results from the largest comparative public management research project in
Europe funded from the European Commission’s FP7: COCOPS – Coordinating for Cohesion
in the Public Sector of the Future. It is based on an original, large-scale survey of more than
top public sector top executives both in ministries and agencies in 16 European countries
(Austria, Estonia, France, Germany, Hungary, Italy, Lithuania, the Netherlands, Norway,
Portugal, Spain, the UK, Sweden, Serbia, Denmark, and Ireland), exploring executives’
opinions and experiences with regards to public sector reform in general government. An
important aspect, which sets the COCOPS survey apart from most other executives’ surveys
in public administration, is the fact that it represents a full census of the target population
defined, and includes secretary-generals, director-generals and directors. The dataset on
which the paper is built consists of the 5,999 valid responses from these sixteen countries,
equivalent to an overall response rate of approximately 24.30%. The survey cannot claim full
representativeness for the data but can be regarded as a good proxy and by far the largest
comparative dataset for European public administrations collected up till now.
Using perceptions to evaluate reform outcomes
Alternative ways of looking at reform effects are to use subjective evaluations, e.g. citizen
assessments, or assessments by key stakeholders, in this case top public sector executives.
Relying on the views to public official’ and managers to evaluate reforms is common practice
– see e.g. Emery and Giauque 2003; Christensen & Lægreid 1999; Worrall et al. 2000). Yet,
this is not without problems – see e.g. Meier & O’Toole (2013). To be expanded
5
COCOPS survey
Reform trends
Before analysing how senior public sector executives evaluate the overall changes in the
public sector in their countries, we first need to establish what changes have actually
happened. The COCOPS Top Executive Survey asked executives to indicate how important
certain reform trends are in their own policy area, on a scale from 1 (not at all) to 7 (to a large
extent).
Figure 1: Reform trends in 16 countries
Collaboration and cooperation intra-sectorial
Transparency and open government
Focusing on outcomes and results
Digital or e-government
Internal bureaucracy reduction
Treatment of service users as customers
Public sector downsizing
External partnerships and strategic alliances
Flexible employment
Mergers of government org.
Citizen participation methods
Contracting out
Extending state provision into new areas
Creation of autonomous…
Privatization
1
2
3
4
5
6
7
Findings show that privatisation is no longer a major trend. The same is the case for the
creation of autonomous agencies and corporatisation, also a typical NPM-style reform.
Intermediate effects: Dimensions of performance
The paper then looks at how public sector executives assess the impact of the various
managerial reforms on public administration in their countries. It distinguishes between
impacts on a number of dimensions:
6
COCOPS survey
Figure 2: Perceived 5-year changes (country means)
Service quality
Cost and efficiency
Fair treatment of citizens
Innovation
External transparency and openness
Ethical behaviour among public officials
Equal access to services
Policy effectiveness
Policy coherence and coordination
Internal bureaucracy reduction
Citizen participation and involvement
Staff motivation and attitudes towards…
Attractiveness of public sector as…
Social cohesion
Citizen trust in government
1
2
3
4
5
6
7
Further on, we will mainly concentrate on four intermediate effects. These dimensions were
selected because of their

Cost and efficiency

Service quality

Policy coherence and coordination

Equal access to services
We first show country differences.
Figure 3: Perceived 5-year change in cost and efficiency (country means)
Denmark
Ireland
The Netherlands
UK
Sweden
Germany
Norway
Austria
Italy
Portugal
Estonia
Hungary
France
Serbia
Spain
Lithuania
1
2
3
4
5
6
7
Figure 4: Perceived 5-year change in service quality (country means)
7
COCOPS survey
Austria
Norway
The Netherlands
Portugal
Estonia
Germany
Sweden
Ireland
Lithuania
Serbia
Denmark
UK
Hungary
Italy
Spain
France
1
2
3
4
5
6
7
Figure 5: Perceived 5-year change in policy coordination and cohesion (country means)
The Netherlands
Denmark
Ireland
UK
Hungary
Norway
Serbia
Estonia
Lithuania
Spain
Italy
France
Portugal
Germany
Sweden
Austria
1
2
3
4
5
6
7
Figure 6: Perceived 5-year change in equal access to services (country means)
8
COCOPS survey
Lithuania
Estonia
Portugal
Hungary
Norway
Spain
Ireland
The Netherlands
Serbia
Denmark
Germany
UK
Austria
Sweden
Italy
France
1
2
3
4
5
6
7
Final effects: Has the public sector become better?
Top public executives were asked to assess how, compared to five years ago, things have
developed when it comes to the way how public administration runs in their country. Error!
Reference source not found. shows means and standard deviation across the entire sample. It
shows that top public sector executives are most positive in Denmark, Norway, and the
Netherlands, even though the mean score is above seven out of ten in just one of these. In
Spain and Portugal, executives tend to indicate that public administration now runs worse
than five years ago. At the same time, we also see a fairly large standard deviation.
Figure 7 Top public executives assessment of improvements in public administration compared
to five years ago (N=5,160)
Spain
Portugal
France
UK
Austria
Italy
Germany
Serbia
Hungary
Ireland
Lithuania
Estonia
Sweden
The Netherlands
Norway
Denmark
1
2
3
4
5
6
7
8
9
10
Source: COCOPS Top Executive Survey. 1 = worse; 10=better
9
COCOPS survey
One year earlier, citizens in all EU member states were asked in a 2011 Eurobarometer survey
‘Compared with 5 years ago, how would you say things have developed when it comes to the
way public administration runs in your country?’. Error! Reference source not found.
shows the percentage of respondents indicating they have seen an improvement. The number
of citizens who have seen an improvement is the highest in Estonia, Cyprus, Luxemburg and
Malta, whereas in Romania, Portugal, Latvia and Greece, hardly anyone admits to having
noticed improvements over the last five years. In Greece, Romania, Italy and Portugal, the
number of people who have seen deterioration is even more than 50% of the effective sample.
In most other countries, the majority of respondents indicated that things have stayed about
the same.
Figure 8 Citizens’ assessment of public administration compared to 5 years ago, % improved,
N=25,137 (Special Eurobarometer 370, 2011)
EE
CY
LU
MT
AT
BE
PL
SK
SE
BG
HU
NL
IT
DK
CZ
UK
EU27
SI
FR
FI
LT
IE
DE
ES
EL
RO
PT
LV
Improved
Stayed about the same
Got worse
0%
20%
40%
60%
80%
100%
Source: Special Eurobarometer 370. Answering options were: ‘improved’, ‘got worse’, and ‘stayed about the same’. Percentages
calculated on useful answers only.
When we compare the evaluation by citizens and top public officials, general trends appear to
be comparable: In countries where citizens are more positive, top officials are also positive.
There are however some exceptions. In France and the UK, citizens appear to be more
positive about improvements in the public sector than top officials. In Germany, Lithuania,
and especially the Netherlands, citizens are more negative about improvements in public
administration.
10
COCOPS survey
Analysis
Top officials’ assessment of whether public administration has improved varies considerably.
How now to explain this variation? Table 1 shows the findings of a regression analysis with
individual- and organisation-level variables as independent variables.
Table 1: Preliminary regression on PA improvement 5 years, COCOPS Top Executives Survey
Unstandardized
Standardized
Coefficients
Coefficients
B
Std. Error
(Constant)
5.740
.195
resp is female (ref= male)
-.136
.068
sex is missing
-.404
age 46-55 (ref=age is under 45)
age over 55
Beta
t
Sig.
29.513
.000
-.029
-2.007
.045
.239
-.039
-1.689
.091
-.224
.079
-.049
-2.834
.005
-.322
.086
-.065
-3.729
.000
.170
.315
.014
.538
.591
MA level (ref = education BA level)
-.207
.105
-.044
-1.979
.048
PhD level
-.201
.133
-.031
-1.507
.132
-.396
.176
-.047
-2.250
.024
second hierarchical level (ref = 1 )
-.285
.085
-.062
-3.358
.001
third hierarchical level
-.597
.092
-.127
-6.501
.000
1-5 year priv sect exp (ref = 1y or less)
-.055
.078
-.011
-.695
.487
> 5 years private sector experience
-.182
.088
-.032
-2.061
.039
private sector experience not indicated
.019
.115
.003
.164
.870
org size between 100 -499 (ref=size <100)
.261
.089
.055
2.929
.003
org size is between 500 and 999
.047
.112
.007
.416
.678
org size is 1000 or more
.368
.095
.074
3.884
.000
Agency (ref = ministry)
-.003
.073
-.001
-.036
.971
policy field infrastructure, environment,
-.228
.091
-.038
-2.521
.012
policy field finance and economic affairs
.165
.085
.030
1.950
.051
policy field general government, foreign
.108
.086
.019
1.268
.205
-.015
.088
-.003
-.175
.861
-.148
.105
-.021
-1.411
.158
policy field other
-.217
.093
-.034
-2.341
.019
policy field justice, public order, safety,
-.066
.102
-.010
-.647
.518
United Kingdom (ref = UK)
.153
.194
.013
.791
.429
Germany
.317
.160
.037
1.979
.048
-.534
.181
-.052
-2.956
.003
age is missing
no education indicated
st
agriculture, transportation
affairs
policy field employment, health, social
protection and welfare
policy field education, recreation, culture,
religion
defence
Spain
11
COCOPS survey
Italy
.349
.210
.026
1.660
.097
Estonia
1.046
.178
.106
5.887
.000
Norway
1.609
.173
.164
9.274
.000
.608
.143
.096
4.249
.000
1.329
.207
.101
6.404
.000
Hungary
.506
.192
.044
2.632
.009
Austria
.400
.158
.048
2.538
.011
Portugal
-.642
.194
-.062
-3.316
.001
Lithuania
1.228
.165
.143
7.427
.000
Ireland
.621
.170
.064
3.647
.000
Sweden
1.396
.164
.180
8.538
.000
Denmark
2.027
.238
.142
8.511
.000
N
4,906
Serbia
The Netherlands
Adjusted R2
0.089
Some preliminary findings indicate that:
-
Older respondents are more negative about developments
-
Female respondents are more negative
-
Lower hierarchical levels are more negative
-
Respondents who have more than 5 years of private sector experience, are more
negative than respondents with one year or less experience in the private sector
-
Compared to respondents in smaller organisations, respondents in very large
organisations and in organisations in between 100 and 499 employees are more
positive
Analysis – SEM – tbc
Conclusion
References
Alonso, J. M., Clifton, J., & D. Díaz-Fuentes (2013). Did New Public Management matter? An
empirical analysis of the outsourcing and decentralization effects on public sector size. Public
Management Review.
Andrews, R. & Van de Walle, S. (2013). ‘New Public management and citizens’ perceptions of local
service efficiency, responsiveness, equity and effectiveness’. Public Management Review,
15(5): 762-783
12
COCOPS survey
Andrews, R. (2010) ‘New Public Management and the Search for Efficiency’ in T. Christensen and P.
Lægreid (eds) Ashgate Research Companion to the New Public Management. Ashgate Press:
Aldershot.
Bach, S., L. Bordogna, G. Della Rocca and D. Winchester (eds) (1999) Public Service Employment
Relations in Europe: Transformation, Modernization or Inertia? London: Routledge.
Banducci, S.A., Karp, J.A., Thrasher, M., Rallings, C. (2008) Ballot Photographs as Cues in
Low-Information Elections, Political Psychology, 29:6 pp903–917
Boyne, G. A., Farrell, C., Law, J., Powell, M. and Walker, R.M. (2003) Evaluating Public Management
Reforms, Buckingham: Open University Press.
Brau, R., Doronzo, R., Fiorio, C. V. and Florio, M. (2010) EU gas industry reforms and consumers'
prices. The Energy Journal, 31:4 pp163-178.
Christensen, T. and Lægreid, P. (1999) New Public Management: Design, Resistance, or
Transformation? A Study of How Modern Reforms Are Received in a Civil Service System.
Public Productivity & Management Review, 23:2 pp169-193.
Dan, S. (2013). The effects of agency reform in Europe: A review of the evidence. Public Policy and
Administration
Emery, Y. Y. and Giauque, D. (2003) Emergence of Contradictory Injunctions in Swiss NPM Projects.
The International Journal of Public Sector Management, 16:6 pp468-481.
Hood, C. (2011) ‘Book review. The Ashgate Research Companion to New Public Management,
Governance, 24(4), 737-739
Jilke, S., Meuleman, B. & S. Van de Walle (2013). We need to compare, but how? Measurement
equivalence in comparative public administration. Paper presented at the PMR Conference,
Madison, Wisconsin
Meier, K. and O’Toole, L. (2013) ‘I think (I am doing well) therefore I am: assessing the validity of
administrator’s self-assessments of performance’, International Public Management Journal
16:1, pp1-27
Okun, A. M. (1975) Equality and efficiency, the big tradeoff, Washington, DC: Brookings Institution
Press.
Pollitt, C. (2002) ‘The New Public Management in International Perspective: An Analysis of Impacts
and Effects’ in K. McLaughlin, S. P. Osborne and E. Ferlie (eds) New Public Management:
Current Trends and Future Prospects (274-292). London: Routledge.
Pollitt, C. & Dan, S. (2011). The impacts of the new public management in Europe: A meta-analysis.
COCOPS Research Report.
Schick, A. 1996. The Spirit of Reform: Managing the New Zealand State Sector in a Time of Change.
Wellington: State Services Commission.
Van de Walle, S. & Hammerschmid, G. (2011). ‘The impact of the new public management:
Challenges for coordination and cohesion in European public sectors’ (review essay).
Halduskultuur – Administrative Culture, 12(2): 190-209
13
COCOPS survey
Worrall, L., Cooper, C. L. and Campbell-Jamison, F. (2000) The Impact of Organizational Change on
the Work Experiences and Perceptions of Public Sector Managers. Personnel Review, 29:5
pp613-636.
14