Impact of Open Access on Evaluation
Questions:
Q1: What different evaluation practices do you know of? Are there different evaluation
practices in different scientific fields? If so, which ones?
[aim: to have a broad picture of the varieties of evaluation practices across disciplines, what is
typical, what are the basic (broad + important) characteristics of different evaluation types
(both of institutions and of individuals), particularly how are publications part of different
research assessment methods . ]
Evaluation practices vary very much depending on the type of evaluation considered. Broadly
speaking evaluation can be distinguished in terms of the level considered within the research
system, as well as the timing in terms of research cycle. In terms of the former, evaluation
can take place at the individual level (e.g. researcher, article, research unit, project). In terms
of the latter, one can distinguish between evaluations performed to allocate resources at the
individual level (sometimes referred to as ‘ex-ante’, or in the case of articles simply editorial
peer review), and evaluations performed after the allocation of resources (interim and ‘expost’ evaluation) for analytical or strategic purposes related to policy decision-making. The
WG Research Policy and Programme Evaluation deals specifically with evaluations at
aggregate, non-individual level. More details on this kind of evaluation can be found in this
ESF MO Forum Report:
http://www.esf.org/index.php?eID=tx_nawsecuredl&u=0&file=fileadmin/be_user/CEO_Unit/
MO_FORA/MOFORUM_Eval_PFR__II_/Publications/mof_evaluation_final.pdf&t=1441304150
&hash=6e0201a9ed192799a57bc6d6bb49206165aaa14f
As explained in this report, evaluation methodologies are indeed fully dependent on the
context (e.g. country, institution, scheme, disciplines) and most importantly on the questions
being addressed by the evaluation. A good evaluation will tailor to the context and
questions. As a result, the role of publications in an evaluation will depend on the type of
evaluation being considered.
Differences in relation to scientific fields are also great, and this is due mainly to institutional
differences: the institutional mix within any given field is specific to the field. Institutional
differences in turn explain different uses of evaluation (rather than an intrinsic need to
evaluate)
Q2: Do you know of any evaluation practices where Open Access plays a role or is taken
into account?
[aim: to be pointed in the direction of types of evaluations where the relation we are
investigating may be most apparent and already taken into account.].
Answer: With regards to individual-level evaluation, within our WG we know about the case
of at least one funding agency (F.N.S.-F.N.R.S. of Belgium) and one research performer (INRA
of France) applying the ‘Liege Model’. INRA will apply the model starting in 2016): in their
case, the INRA database is connected to the French national repository (HAL), to avoid the
duplication of entries (for more information on HAL, see the answer to Q4). In addition,
there are specific scientific communities where OA publications are important for one’s track
record (see answer to Q5).
In terms of aggregate-level evaluation, such as organizational evaluations, we are not aware
of cases where evaluations look specifically into OA to publications take-up as an evaluation
criterion.
Q3: How are metrics & indicators currently used in evaluation procedures affected by Open
Access?
[aim: to test the validity of the assumption that publications and publication behavior are major
elements in the methods of assessing research performance (especially in quantitative
measures) and changes in those areas are likely to affect the measurements.]
The metrics and indicators theoretically affected by OA are the bibliometric ones. A recent
survey of MOs by the WG Evaluation revealed what kind of indicators are used and in what
contexts (for what kind of evaluation): see the table pasted at the end of this document.
However, at least for individual-level evaluation, the impact of OA on indicators does not
necessarily translate on an impact on evaluation, because individual-level evaluations should
not be based on indicators and bibliometrics, regardless of whether these are based on open
or closed access material. Moreover the relevance and meaning of bibliometrics is very
much field-specific. To learn more about the role of metrics in evaluation, a good starting
point could be the recent review on the subject conducted in the United Kingdom by the
Higher Education Funding Council for England. The review can be found here:
http://www.hefce.ac.uk/rsrch/metrics/
More in detail: Publications data are a major data source for evaluation (in particular for
questions related to scientific impact), so the issue here is what indicators are being
constructed out of this data source (e.g. journal impact factor is one such indicator, and OA
journals usually have lower impact factors) and in what contexts these are used. To the
extent that OA journals are less cited and journals have lower impact factors, metrics and
performance indicators may be negatively affected by OA. Metrics are however not
employed in isolation. Any theoretically negative impact should be mitigated by good
evaluation practices which entail not relying on metrics alone (see for example the DORA
manifesto, whose principles are widely shared by the SE MOs). At the individual level, metrics
are simply an empirical input that panels can consider but should by no means be taken at
face value, as they can be highly misleading for many reasons beyond the take-up of OA
publishing.
Q4: What advantages and disadvantages are there in using evaluation practices to promote
(encourage) Open Access, for example the ‘Liege Model’?
[aim: to investigate the inversed relation of how evaluation impacts Open Access: evaluations
for resource allocation purposes set and change incentive structures, therefore potentially
supporting or hindering the aims of specific policies, such as policies that are trying to
encourage Open Access. ]
In general, evaluation practices reflect the institutional set-up and the dominant system of
rules and regulations. So as a rule of thumb, evaluation structures follow institutional set-ups
rather than determine them. As a consequence, changing evaluation practices in the hope
that this will cause a structural change may prove to be a naïve approach. The Liege model
seems more realistic of an approach because it is a case of evaluation adapting to an
institutional decision made a priori. Another model on this line is France’s HAL, which holds
that publicly-funded research should be deposited into a common public publication
repository (HAL). Same as Liege model, but depends on a pool of institutions rather than one


The University of Liège adopted the Immediate-Deposit & Optional-Access (IDOA) mandate “It is compulsory for researchers to add
references for all their papers and academic conferences to the institutional repository, backdated to 2002, and to deposit the full electronic version
of all the articles published since 2002.”
Evaluations, appointments, promotions, and budget allocations decisions take into account only references that are archived in ORBi
(institutional repository) see f.e. http://orbi.ulg.ac.be/news?id=03#Declaration
only. The obligation is not applied uniformly neither institutionally (different institutions have
different publication policies), nor academically (different disciplines have different
recognition and publication models).
The main disadvantage of these models from the point of view of evaluation is that the
manuscript uploaded in the repository is not the document of reference. As an end result, a
researcher may end up with different versions of the same article receiving citation, which
can create noise from the point of view of aggregate level evaluation.
At the individual level, an obvious disadvantage is that proper recognition or rewarding of
scientific quality may be understated because of the OA agenda, which is not intrinsically an
issue of scientific quality. With regards to the impact on bibliometrics, again, at the individual
level, one should not rely on bibliometric indicators (see http://am.ascb.org/dora/), so it is
more appropriate to speak about noise with regards to bibliometric statistics.
Q5: Do you see a need for recommendations on how to take Open Access into account in
evaluations?
[aim: to identify (in a preliminary way) a potential impact of this investigation. A practical
outcome for the Task Force and the WG may be to identify the need and if required then
develop recommendations based on the findings.]
From our standpoint it is very hard to imagine recommendations that would be useful
without being discipline-specific. The role of publishing serves very different functions
depending on the scientific field of communities. Communities that perform sensitive
research for safety, security or privacy reasons may not use publishing at all for
dissemination or knowledge transfer. Communities that seek peer to peer communication
and dissemination, use very different channels: chemistry uses databases and publications
revolve around the control of databases, history and anthropology use books, philosophy
uses monographs and books, mathematics prefer OA publications, computer scientists use
conference proceedings for cutting edge results and journal articles rather to take stock and
consolidate existing knowledge, and so on. Efficient worldwide knowledge networks where
OA infrastructures function efficiently, do so because they are structured around social
communities which share similar research practices and needs (e.g. geophysics,
oceanography, etc.)