Using metrics to your
advantage
Fei Yu and Martin Cvelbar
1
Using metrics to your advantage
Metrics
•
•
•
Things that can be counted
•
number of publications
•
citation counts
Also:
•
Journal quality measures – e.g. Journal impact factor
•
H-index
•
Collaboration metrics
Metrics: must always be considered in context – significant disciplinary differences,
and differences over time.
2
Using metrics to your advantage
Why use publication metrics?
1. Metrics provide supporting evidence of claims individuals/research groups make
•
Track record of individuals, groups of researchers
•
Contribution to the discipline and/or significance of the contribution
•
Evidence of international profile
•
Evidence of capacity to collaborate effectively
2. The current environment …
•
research evaluation now part of the landscape (ERA) – grant application
reviewers, potential employers etc. may expect to see metrics
3
Using metrics to your advantage
Research Quality
•
Determined by peers
•
peer review
AND/OR
•
Inferred by “numbers” – bibliometrics indicators
•
Citation counts etc.
4
Using metrics to your advantage
qual·i·ty/ˈkwälətē/
Noun
•
the standard of something as measured against other things of a similar kind
•
the degree of excellence of something
Given the numbers some meaning!
5
Using metrics to your advantage
For papers published in
journals within the ESI
field of Mathematics in
2002 the average
citation rate in WoS is
4.91
In Molecular Biology
and Genetics it is
37.24!
6
Using metrics to your advantage
Citation counts
As a research quality or performance measure the assumption is:
high citations = high quality
BUT
•
“…not only the content of scientific work, but also other, in part nonscientific, factors play a role in citing behaviour. Citations can therefore be
viewed as a complex, multidimensional…phenomenon”
Lutz Bornmann, Hans-Dieter Daniel, (2008) "What do citation counts measure? A review of studies on citing behavior", Journal of
Documentation, Vol. 64 Iss: 1, pp.45 - 80
7
Using metrics to your advantage
Journal Impact Factor
…..
Flawed for a number of reasons…but it may be a useful tool to profile researcher
output … …
8
Using metrics to your advantage
Example
“My research output is of very high quality, with publications
consistently in good quality journals … “
•
What % of your journal articles are in the top 10% by Impact Factor for the Journal
Citation Reports categories you publish in?
9
Using metrics to your advantage
Metrics: the standard 5
1. Number of publications
2. Career citation count
3. Citations per paper
4. % cited (or % uncited)
5. h index
These numbers tell us something … but there’s a lot missing … …
10
Using metrics to your advantage
How to use metrics to paint a complete picture
11
Using metrics to your advantage
Provide more detail … …
1. Number of publications
•
Profile by document type, subject area, year, journal etc.
2. Number of citations (career citation count)
•
Provide an indication of coverage by citation data provider
•
Are all outputs included or just articles and reviews?
3. Citations per paper
•
I have 35 refereed journal articles, of which 33 are indexed by Web of
Science. These articles have received 230 citations, giving an average
citation per (indexed) paper of 7 (source: WoS, 05/01/15).
4. Percentage not cited
•
Of my 33 indexed journal articles, 3 articles have not been cited by others
(9% not cited), and these were all published in 2014 (WoS, 05/01/15).
12
Using metrics to your advantage
And some additional metrics… …
•
Citations for an individual paper or sub-set o publications: supporting evidence for
outstanding contribution in a particular area.
•
Citations per paper by year of publiation and by journal/subject area/FOR code
•
Can then benchmark
•
E.g. Incites, or use Thomason Reuters Essential Science Indicators for
benchmarking
•
Or create your own benchmarks using refine reature within WoS
13
Using metrics to your advantage
5. h-index
… ….
Productivity + Impact = Influence
14
Using metrics to your advantage
Does it mean anything?
•
•
Use the h-index measure with care. Citation patters vary across disciplines.
•
E.g. h-indices in Medicine are much higher than in Mathematics for example.
•
Researchers in different disciplines can not be compared using the h-index.
Even within the same discipline the h-index should not be used alone as a measure
of research quality.
•
Researchers in the same discipline at different stages of their careers can not
be compared using the h-index.
15
Using metrics to your advantage
Consider the following 3 researchers who have all published 10 papers and all have an h-index of 5
Researcher 1
cites per paper
Researcher 2
cites per paper
Researcher 3
cites per paper
Paper 1
100
50
15
Paper 2
90
50
15
Paper 3
70
45
14
Paper 4
50
45
14
Paper 5
35
5
12
Paper 6
4
4
4
Paper 7
0
2
4
Paper 8
0
0
4
Paper 9
0
0
4
Paper 10
0
0
4
Average cites per paper
35.3
20.1
9.0
16
Using metrics to your advantage
Put it in context … …
•
My h-index based on these indexed papers is 10. I have 4 papers (A, B, C, D) with
more than 20 citations and 1 paper (E) with 29 citations (source: WoS, 05/01/15). I
also have an additional 3 papers not indexed by WoS, with 29 citations based on
Scopus data (05/01/15).
17
Using metrics to your advantage
More examples … …
“My work is multi-disciplinary, spanning biochemistry, biophysics and
oncology …”
Analyse your WoS articles by WoS subject category to see if this is evidenced in
your research output.
“34% of my journal articles are in the WoS Subject Area of Biochemistry
and Molecular Biology, with 29% in Biophysics and 16% in Oncology (WoS
Subject Areas, 05/01/15)”.
18
Using metrics to your advantage
More examples …
“My research has found (widespread) application beyond my discipline … “
Look at the articles which cite your work in WoS and analyse them by WoS
Subject Area.
“Whilst 65% of my articles are published in Physiology (WoS Subject Area),
55% of my citations are from journals with a Subject Area of Cell Biology
(source: WoS, 05/01/15).”
19
Using metrics to your advantage
For Early Career Researchers… with a short track record
•
Look in detail at the quality of the citations your work is receiving
•
What are the impact factors of the journals in which your work is cited
•
•
•
What fields are the journals in?
Who is citing your work?
•
Who are they?
•
Where are they from?
•
Why are they citing your work?
How quickly is your work being cited?
20
Using metrics to your advantage
Beyond bibliometrics…..
Altmetrics is the creation and study of new metrics based
on the Social Web for analyzing and informing
scholarship.
From http://altmetrics.org/,aofesto/
21
Using metrics to your advantage
22
Using metrics to your advantage
Consider other measures of research impact … … for example:
•
Mentions in legislation, government policy etc.
•
Academic library holdings
•
Look at these “56 indicators of impact”. http://cas-csid.cas.unt.edu/?p=4475
•
See also:
23
Using metrics to your advantage
Some questions to think about:
•
Have you viewed your download statistics or abstract views in eSpace? [countries
& dates viewed]
•
How many of your books are held in libraries worldwide? [library can provide data]
•
Have you been invited to speak at conferences, and were the papers peerreviewed?
•
Has any of your work been commissioned?
•
Has your research influenced government or education policy?
•
Has there been any measurable social benefit from your research?
•
Have you collaborated with others domestically or internationally?
•
Have you received any awards?
24
Using metrics to your advantage
Some advice … …
•
Use author disambiguation tools
•
Google Scholar Citation profile
•
ResearcherID
•
Scopus authorID
•
ORCiD (Open Researcher and Contributor ID)
25