Universities in the ‘Performance
Age’
Session two:
Performance Management, Assessment
and Rankings
Maison française d’Oxford, Oxford,
February 17-18th
1
Assessments and Rankings in French
Higher Education: a Case Study
Lise Gastaldi & Caroline Lanciano-Morandat
The Institute of Labour Economics and
Industrial Sociology (LEST)
CNRS and Aix-Marseille University
2
Introduction
 A presentation of the first works we realized within a
larger research programme funded by the French
ANR (National Agency for Research)
 The Prestence project: From Prestige to Excellence,
the fabric of academic quality
 We investigate the consequences of a change in
academic quality judgment devices with the increase
of formal devices in addition to informal ones
 With a focus on the devices dealing with higher
education establishments
3
Introduction
 Informal devices which are older


Produce global judgments about quality in terms of
reputation or prestige
Judgments are formulated


With no formalized list of criteria
By various actors included non expert ones
 Formal devices which are increasing currently


Evaluations are based on formalized criteria
With the aim to formulate more objective judgments by a
more systematic evaluation of quality
4
Introduction
 Informal and formal evaluation devices can produce
different judgments on one establishment
 The Prestence research programme aims to investigate
how an establishment is succeeding in producing
academic quality, given different regimes of quality
evaluation exist

Putting the light on the organisational work of gathering and
combining various resources

And investigating how the change of evaluation regime can
influence the evaluation of the establishments and the way
they react in terms of strategy and organisational choices 5
Introduction
 A comparative research between several establishments
in various countries* and scientific fields*
 I and Caroline focus on the chemistry and physics field

We realized the first case study on an old and prestigious
French institution


Characterising a singular organisational model and a focalisation
strategy with the production of specific outputs
Through this French case study, we address the question of
the sustainability of singularity in higher education when
establishments face more developed formal evaluation
devices
6
Content
1. The French context of evaluation in higher
education
2. The case study: a singular establishment
3. Aims and issues
4. Singularity facing formal evaluation devices
5. From evaluation to change on strategy and
organisational path?
6. Conclusion and research perspectives
7
1. The French context of
evaluation in higher education
 We observe a strengthening of assessment
 We focus on establishments and recurrent
evaluations and we adopt a classification of
these evaluation devices into two main
categories

Evaluations leaded by national public authorities


With a focus on public establishments
Evaluations leaded by other actors (newspapers,
higher education institutions…)

ARWU, THE, QS ranking, SIR…
8
The French context of
evaluation in higher education
 A classification that refers also to two main differences

The imperative nature of the evaluation devices (or not)


Public establishments cannot avoid assessments leaded by national
public authorities
If we consider rankings
 Some needs the establishments collaboration (it is the case of national
ones and of the THE ranking for example)

The establishments can refuse to respond to the ranking producers requests,
even it can be a risky strategy given the growing audience of rankings
 Some doesn’t need the establishments collaboration and consequently the
establishments are evaluated even if they are not agreed

The form and the ‘philosophy’ of the evaluation process and
its result
9
Differences between the two
categories we distinguish
National public authorities
assessments
Evaluators Peers and experts of higher
education and concerned
identity
Rankings
Experts in data mining
scientific fields
Judgment
process
A substantive assessment with
quite in-depth analysis of
activities, organisation,
governance, means, outputs…
Based on quantitative data and
qualitative analysis
‘substantive subjectivity’
A mechanical judgment process
through the automatic compilation of
data dealing with inputs and outputs
(considering the evaluated entity as a
black box)
‘mechanical objectivity’ (Porter, 95)
Judgment
result
A substantive evaluation report
with quite developed analysis
and advices
A very synthetic result: a score and a
rank
10
The public assessment devices
 Before 2007

There were various devices leaded by different
public entities
Some was in charge of assessment of the trainings
 Some was in charge of assessment of the research
activities and laboratories


There was no unified evaluation system
11
The public assessment devices
reforms
 In 2007, the AERES was created and took the place of several
former evaluation devices

Agency for research and higher education assessment
 The AERES has the mission to evaluate all the research and
higher education public institutions that are under a French
Ministry authority

With a three levels evaluation: research laboratories, training
programmes and establishments
 An evaluation each 4/5 years by ad hoc experts committees
compound by peers, higher education experts, industrial
researchers…
12
The AERES assessments
 First the entity evaluated make a report that presents itself, its
activities, productions… with a self-evaluation
 Then the experts committee visits the entity with meetings with
the governance team, the academic and administrative staff and
the students
 They write an evaluation report, and the entity can respond
 These documents are published on the AERES website
 The committees mark laboratories and trainings on a A+, A, B, C
scale
 These evaluations can be a basis for the Ministry Higher
Education and Research decisions as funding, trainings
certification…

With nowadays a direct link with the SYMPA model that determines the
research budget allocated to an university in relation with the marks
obtained by its laboratories
13
National and international
rankings
 National rankings in France


Mainly dedicated to engineering and business schools
For engineering schools (that are part of universities or
not)



Rankings are realized and published by newspapers
There are few old rankings and several new ones
A larger audience
 With an Internet diffusion
 With an increasing attention paid to rankings by students and their
families
 International rankings (Shanghai, THE…)

We also can observe a multiplication of rankings that enjoy
a growing audience
14
A functionalist lens with a reference to socioeconomics works (Karpik, 96; Cochoy,99…)
 Various actors (students and their family, public authorities,
partners, academic staff, employers…) who invest resources in
higher education want to have some references about the quality
and performance of research centres, trainings and establishments



Higher education can be considered as experience goods
It is difficult to have a precise idea of an entity quality when you are not
inside
It is a more and more complex and larger system if we consider the
increasing number of institutions and trainings in each country and if we
consider the global ‘market’ of higher education
 The stakeholders need quality references to do their choices
among all these entities
 And considering the new importance of accountability logics
=> All these facts lead to strengthen evaluation
15
2. The case study: a singular
establishment
A French engineering school located in Paris
 Created in 1882 in a specific context



East French regions, where chemistry schools were concentrated, passed
into the hands of Germany
French chemical industrials wanted to create a new school to train
engineers and researchers for national chemical industry
This project was presented to Paris city council and a new school was
created under its authority
 Nowadays


An engineering school that is a ‘grande école’ with several research labs
on a same campus
A pluridisciplinarity both in teaching (chemistry, physics, mathematics
and biology) and in research (chemistry, physics and biology)
16
Methodology: an in-depth case
study
 A collaborative research
 We collect various data between February 2010 and
November 2011


40 interviews
Public and internal documents
 That we analyzed in a qualitative way and with an
historical perspective
 We organised several restitutions


to some actors in charge of the establishment governance
to our Prestence project colleagues
17
A singular establishment
 Institutional status: one of the two public higher education
institutions that are under a local authority and not under a
Ministry authority
 A small institution



250 students
400 academic and administrative staff
It is a small institution in France, and the gap is greater in comparison
with foreign universities
 A strong autonomy of the heads of laboratories behaving as
CEO of small firms


Autonomy to conduct their research activities and manage their teams
Autonomy to lead valorisation activities with for example the capacity to
take patents in their own name
18
A singular establishment
 An organisational model characterised by strong interactions between
research, teaching and industry



Research and teaching: lot of courses take place in the labs; the students spend 4
months in the school labs during the engineers training; the last year is dedicated
to do a research Master degree; most of 60% of engineers do a PhD and the most
of them will begin their career in industrial or academic research centres
Research and industry: researchers lead fundamental research but with a long term
application preoccupation ; they have lots of research contracts and we can
observe all the form of technology transfer: scientific consulting, patents, spinoffs…
Teaching and industry: with a six month stage in an industrial research department,
with some courses linked to a specific firm needs and with various relations with
an active network of industrial partners
 An establishment that produces specific research and students prepared to
work in the field of research and innovation
 It is a singular establishment in the French context comparing to universities
and also others engineering schools
19
3. Aims and issues
 This establishment presents a singular organisational
model and produces specific outputs
 It is a well-evaluated establishment for a long time

By informal judgment devices




Strong reputation and prestige in the academic world
Strong reputation in the industrial world
Some of its researchers are well-known by the public
By old formal judgment devices


Publications
French research institutions as CNRS or INSERM
 However how is it evaluated by current formal
evaluation devices?
20
Aims and issues
 A main issue: Is singularity in the current higher
education system sustainable?
 Declined in two research questions with management
and public management lens

How does singularity face the current formal evaluation
devices?


Or in other words: how do the current formal evaluation devices deal
with singularity?
Could these evaluations have some consequences on the
singular establishments strategy and path?
We address these questions through one case study
21
4. Singularity facing formal
evaluation devices
 The establishment experiences some
difficulties with current evaluation devices

With the AERES and the CTI


The CTI: the ‘commission for engineer diploma’ – a
specific French instance that evaluates engineering
schools on the training side and does
recommendations to the Ministry that gives the right
for an establishment to deliver engineer diploma
With national rankings and especially the
international ones
22
Confronting singularity to
national public assessments
 Most of the research centres are well-evaluated but we studied
the case of a strong controversy about one laboratory evaluation



It is an historical laboratory with a strong reputation and positive CNRS
evaluations
But the AERES assessed it as a ‘B’ laboratory
All the laboratory members, the establishment directory and the head of
the CNRS chemistry department protested
 A case which illustrates some criticisms about the AERES
evaluations




Difficulty to assess pluridisciplinary works
Subjectivity of the committees members
The short time allocated to visit each laboratory
Some tensions around the evaluation of application-oriented research
23
Confronting singularity to
national public assessments
 Tensions with the CTI

The CTI leads evaluations by comparing training
characteristics with standards


The conformity with these standards is a condition to
obtain accreditation
There are conflicts on
The training length: it is a 4 years training in this
establishment compared to the 3 years standard
 The training content: for the CTI students must follow
courses in management and economics that are really low
developed in this establishment

24
Confronting singularity to
national public assessments
 Due to its status there is no global evaluation of
the establishment and no valorisation of its
specific model with strong interactions between
research, teaching and industry
25
Singularity facing rankings
 First we should think that rankings can be more in favour of
this kind of establishment


With the mechanical objectivity of the evaluation process that only
counts the publications, there is no more tension with experts who can
be subjective or not able to evaluate research in pluridisciplinary fields
Some rankings (some national or the THE ones) assert they evaluate all
the dimensions and activities of an establishment
 Nevertheless…


The establishment is ranked between the 10th and 20th positions in
national rankings, consequently not among the most excellent ones
It is only ranked in the ARWU, between the 200th and 300th positions;
it is not ranked in all the others rankings
26
Singularity facing rankings
 When we compare the establishment profile and the rankings
criteria

Strengths





Weaknesses



An old institution
An historic and strong investment in research
A high ratio staff / students
A high quality of the students professional insertion
The small size (with a lot of criteria in absolute terms, with the importance
of reputation inquiry…)
The low internationalisation with few foreign students and academic staff
Dimensions that are not really included in rankings


The industrial relations and technology transfers
The interactions between research and teaching
27
Singularity facing rankings
 If rankings are objective, they incorporate
normative definitions of higher education and
excellence with usually




A conviction of the superiority of a big size
establishment
A focus on research outputs (evaluated with only
two criteria: number and impact of publications)
A strictly academic definition of excellence
The importance of internationalisation both of
students and scientific staff
28
Singularity facing various
evaluation systems
 Consequently we can ask if we would assist to
a process of singularity reduction that will lead
the establishments to more standard
organisational models producing more
standards outputs
29
5. From evaluation to change on
strategy and organisational path?
 Such a process of ‘desingularisation’ could only occur if formal
evaluations lead some changes in establishments strategy or
organisational model, and this depends on several elements:



The way the establishment obtains crucial resources and the existence or
not of a link between its evaluations and its capacity to gather these
resources
The way the establishment’s members sense the evaluation in general and
the different devices and also their consequences on their access to
resources
The strategy of the establishment in relation to the various evaluation
systems, given it can adopt


An active or passive resistance strategy
Or a strategy of adaptation; and in the case of we have to consider the capacity
of the establishment for engaging reforms in order to improve next evaluation
30
Karl Weick (1977, 1995…)
 Weick and the theory of ‘sensemaking’

The process of sensemaking which is very important to understand how
individual and collective actors in organisation act and react to
environmental change is composed of 4 stages





An environmental change
An enactment process that leads the actors to take aware of the environment,
but the environment is complex and various
The selection is the process by which actors try to reduce the environment
‘equivocity’ constructing interpretations and leading operations to give sense
to the environmental change
The retention/memorisation: the actors retain some interpretations and act in
consequence; and the interpretation of the environment which are memorized
will influence the next processes of sensemaking when other environmental
change will occur
Consequently it is important to adopt an historical perspective to analyse
the actors reaction to a change in their environment
31
For a long time, a low sensibility to evaluation
and especially to formal evaluations
 On the research side


At the beginning: laboratories are not affiliated to external entities
Progressively some had been affiliated to universities or research entities
as CNRS or INSERM


With positive evaluations by CNRS and INSERM
Nowadays it remains few laboratories that are not mixed research units
and consequently that escape to formal evaluations

With the conviction that autonomy is a strength because it allows to lead
application-oriented research and to exercise valorisation activities
 On the teaching side

The establishment was in a good position to bargain with the CTI in order
to maintain its singular characteristics
32
For a long time, a low sensibility
to formal evaluations
 The establishment enjoys positive informal evaluations
by




The academic world, due to its research works and
publications
The industrials which pay for collaborative research
(financing projects and fellowships) and recruit engineers and
doctors coming from the establishment
The students and their family, due to the establishment
reputation and the very good conditions of professional
insertion for the engineers coming out of it
The general public, due to its star scientists, 5 Nobel Prizes
and its story (since Pierre and Marie Curie)
33
For a long time, a low sensibility to
formal evaluations
 Thanks to


These positive evaluations
The autonomy some laboratories enjoy because they are not
affiliated to external institutions that allows them to lead
valorisation activities (patents, spin-offs) without difficulties


Valorisation activities as sources of incomes
The historical support of the main public authority that
allocates every year comfortable means to the establishment
 The establishment did not face problems to access
resources
34
For a long time, a low sensibility to
formal evaluations
 They have enough money and job positions to conduct
their activities in good conditions



With funding of public authorities
With industrial research contracts
With the revenues of valorisation
 They do not have problems to recruit scientific staff
 They recruit high level students
 The students do not experience difficulties for their
professional insertion in higher education or in
research and development departments in the network
of industrial partners
35
Nowadays the context is
changing
 It is the end of the former victorious resistance against CTI
orders: the establishment had to conform its engineering
training to the 3 years standard length
 Some evaluations by the AERES are reserved, with effects on
internal hierarchy between laboratories and researchers
 The fact the establishment does not appear in the international
rankings except Shanghai one
 The main public authority (the Paris city council) now heeds to
these rankings and is concerned by the low international
visibility of the establishment

With a stronger issue in terms of the return on the investment realized in
financing this establishment
36
A progressive evolution of
representations
 A perception influenced by the establishment
path

Some people who continue to refer only to informal
judgments


A reference to few academic colleagues in prestigious
institutions
Few persons mention rankings
37
The reaction to academic
evaluations
 Even the conditions of evaluation had changed with the AERES
creation, this kind of assessment is usual and well-accepted



Peers assessment
Evaluation of the research programmes and the publications
Evaluation leaded by national public authorities (even in this case the
Ministry is nor the main authority neither the main resources supplier)
 Moreover their legitimacy, these evaluations condition a larger
part of the resources (money and jobs position)


Directly in the case of laboratories associated to universities
In a more indirect way with the influence of the AERES marks on the
capacity to obtain some financings on projects
38
The reaction to academic
evaluations
 In fact


Some of the criteria of formal public authorities assessments
had been progressively integrated
More recent evolutions can be analysed as a response to the
AERES evaluation process and norms


Some laboratories mergers to constitute more important units
The research activities and laboratories are now presented in a more
intelligible way for external evaluators (or widely for stakeholders)
 Through of a kind of map with different areas for each discipline that puts
laboratories in relation to them

An internal reorganization of some labs to present a structure easier to
understand in a fast way for external evaluators
39
Public evaluations and future of
singularity
 The evolutions are too recent to conclude but we can
ask if some of them could conduce to a reduction of
singularity
 The aim to normalize the situation in integrating all
the laboratories in research units affiliated to external
academic partners, with an automatic loss of
autonomy for them
 The reorganisation of labs and teams are not neutral
on the scientific activities
40
Public evaluations and future of
singularity
 Moreover public evaluation systems could lead to
deeper transformations if we consider that
pluridisciplinary teams and application-oriented
research are not well evaluated by all the committee
experts
 Then the form of the assessment result can have a
performative effect given the report formulates analysis
and gives advices on the governance, the internal
organization and the scientific field: orientations,
programmes, collaborations, publication strategy…
41
The reactions to rankings
 Few persons inside the establishment are preoccupied
by rankings, except the director and the man in charge
of communication
 A low attention is historically paid to national rankings
due to several elements
1- They are seen as non objective and non reliable because of
their way of collecting data
2- They are considered as non legitimate because they are not
realized by peers or experts and not controlled by academic
community or public authorities
42
The reactions to rankings
3- National rankings have quite low impact on students
recruitment



In the French higher education the engineering schools
recruit by a special way: after a 2 or 3 years special training
the students pass very selective exams
There are various exams to integrate the different engineering
schools, and the establishment we study recruits since few
years on the same exam that the most prestigious French
establishment that is Polytechnic school
Whatever is their rank in rankings they succeed in recruiting
excellent students
43
The reactions to rankings
 When the international rankings appeared,
most of the people in this establishment not
paid attention to them

We may see here an effect of establishment path
Its members always have neglected national rankings
 They have not anticipated the new audience of these
international ones thinking the establishment prestige is
sufficient to assure its durability


Indeed until now the establishment resources are
not linked to its ranks in these rankings
44
The reactions to rankings
 However the directory board in which members of Paris
city council are part of took decisions that can be seen
as reactions face to this growing audience of rankings


An increase of the students numbers and the aim to recruit
foreign ones
A strategy to association with other higher education
institutions


It is membership of ‘ParisTech’ that is a large association mainly of
engineering schools located in Paris and its suburbs
It is also membership of ‘Paris Sciences et Lettres’ which is an
association of institutions in sciences and humanities fields located in
Paris
45
The reactions to rankings
 We can see this strategy of association in the French
context characterized by several mergers of universities
with an evident preoccupation of increasing their size


When the size is a key factor in rankings
Given such mergers respond to Ministry pressure
 So the establishment we study adopts an alternative
strategy compared to merger in order to preserve its
autonomy
 We do not know now if this strategy is durable or if in
the future this establishment will have to merge with
university
46
The rankings and the future of
singularity
 For the moment rankings have low effects
 But


The aim to increase the number of students will ask the
question of the capacity to maintain all the activities on
the same campus while it is a key factor in the high
proximity between students and research
If the establishment will merge with other institution and
especially with a big university, the question of the
capacity to maintain its specific model with the heads of
research centres high autonomy that is a factor of the
establishment attractiveness for academic staff will be
asked
47
The creation of a own
assessment committee
 The directory board decides to create an international
scientific committee in order to evaluate the
establishment in all its activities



This committee is composed of 10 international academic
and industrial star scientists
Each year, they lead an overall evaluation on research,
teaching and valorisation activities
They formulate qualitative assessment and numerous
advices. For example they can highlight research topics to
investigate, strategic scientific field to develop…
48
The creation of a own
assessment committee
 We can interpret this own international committee
creation in two ways


As a way to say to national public authorities that the
establishment acknowledges the importance of evaluation
even if it is not evaluated by the AERES
As a buffer

To help the establishment to better understand the new international
evaluation norms

And to help it to imagine some transformations that could improve its
evaluations but in the respect with its singularity
49
Why this need for such a buffer?
 Rankings only evaluate outputs and in some cases inputs (as
budgets or staff), often with few indicators
 These criteria are not obvious to turn into concrete actions



We have to more publish in higher level journals, ok…
It would be better if we have more money, ok…
We should have Nobel Prizes, ok…
 Contrary to the AERES evaluations that give concrete practical
and more easy-to-implement actions
 The AERES evaluations are less normative but easier to
understand and turn into action, whereas rankings are more
normative and directive as they don’t give obvious ways of
improvement
50
6. Conclusion and research
perspectives
Do formal evaluations lead to a standardisation process?
 It seems to be the case. When evaluation is based on determined
criteria and implicit or explicit definition of academic quality
(which tends more to a excellence and thus a restrictive
definition of quality) it promotes norms and by the fact
marginalises entities out of the line
 But the singularity seems to be more problematical in the
rankings evaluation regime


To some extent national public authorities assessments can take into
account national, disciplinary or specific contexts and the establishments
particularities and story
With the rankings, an establishment that does not fit the criteria is in a
bad position, whatever the reasons that explain its specific characteristics
51
Conclusion
 The sustainability of singularity mainly depends on the central question of
resources access


How the evaluation condition resources access in a direct or more indirect way?
Will an establishment be able to maintain its resources access even if it is not
well-evaluated?
 It depends on national systems, establishments status and stakeholders’
strategies (on the question of the importance they give to various
evaluations and especially to rankings)
 A key question seems to be the permeability between the two regimes of
evaluation when the experts involved in public assessment systems are
influenced by the rankings that modify the established hierarchy
 We can question such a process of ‘desingularisation’ with social value lens:
what is the social value of singularity versus a large standardisation?

When we consider the case we study that produces high level research (it is very
well evaluated on the per capita criteria in the ARWU) with strong impact on
industry and innovation, and that produces specific engineers who are
immediately recruited by national and international firms
52
Research perspectives
 Pursuing the work about these questions
dealing with the future of singularity in the
current higher education system
 Others linked questions about


The appropriation of evaluation devices by the
establishments
The relations between industrial linkages and
excellence as it is defined by current formal
evaluation devices
53
Research perspectives
 New empirical investigations

In the chemistry field
In France by others teams of the Prestence project
 University of Geneva, Switzerland, chemistry department
 Polytechnic of Turin


In other scientific fields
54
Thank you for your attention
lise.gastaldi@univ-amu.fr
caroline.lanciano@univ-amu.fr
55