Bibliometrics: Measuring the Quality of Research
Professor John Mingers
Kent Business School, May 2013
j.mingers@kent.ac.uk
A. Measuring research quality
1. You have had the idea, done the research, written the
paper, got it published – isn’t that the end?
• No! Now people want to evaluate the quality of the
research. Is it world-class? Internationally good?
Merely average?
2. There are essentially two ways of evaluating the quality
of a paper:
• Peer review – subjective judgements by experts
• Metrics – measuring the reactions of other academics
quantitatively by their behaviour – primarily citations
“Peer review by the whole world”!
3. Why is measuring quality important?
• Jobs and promotion
• Allocating funds, e.g., grants, Government funding
• League tables, e.g., REF
4. It can occur at different levels
– A single output, e.g., a paper
– An academic
– A department
– A journal
– A university/multiversity
5. Peer review
• Seen by many as the “gold standard”, hence the REF is
still based on peer review, but
•
•
•
•
It is extremely time consuming and expensive
Experts can genuinely disagree (cf referees)
There are unconscious and conscious biases
It can be very conservative – the “Golden Triangle” – and
can create an old boys club which can be difficult to enter
6. Metrics (bibliometrics, scientometrics, infometrics)
• Generally, the quantitative study of published or
recorded information. In terms of evaluating
research it takes the number of times a paper
has been cited as a surrogate for its quality.
• Citations really concern impact which is not quite the
same. A paper could be cited because it is wrong!
• Many papers are never cited – are they therefore
worthless?
• There are problems with recording or measuring
citations, especially in the social sciences and
humanities
7. Using the journal as a measure of quality
• It is common, in some subjects, to use the quality
of the journal as a surrogate for the quality of the
paper
• Not all papers in a journal are of similar quality
• How do you measure the quality of the
journal? Peer review or metrics!
B. Where do citations come from?
1. Citation databases – Thompson-ISI Web of Science/Knowledge
(used to be SCI, SSCI) or Elsevier Scopus
•
•
•
•
These record all the references for a given set of journals (WoS 12,000) and then allow them to be searched by key words or cited
authors
They also now do citation analyses for both individuals and journals.
They are rigorous and generally reliable but limited in coverage,
especially in the social sciences (40% - 70%) and humanities (20% - 50%)
They do not cover books at all, and only some conferences
2. Google Scholar – this searches the web, like Google,
looking for references to academic papers and books
• Covers all kinds of outputs other than journals
• Generally finds many more citations
• Is equally good for all disciplines (about 90%)
• But, it is not rigorous picking up references from
teaching material, home pages etc not just
research journals or books
• The data is poor quality – spelling errors etc. – the
same paper may occur many times and the
authors’ names may be wrong
• Best accessed through Publish or Perish
(http://www.harzing.com/pop.htm)
C. Citation behaviour I - Distributions
*JORS
7.3
7.2
11.3
Dec
Sci
11.1
Ops
Res
14.6
Man
Sci
Histogram of JORS2, OPSRES2, EJOR2, MANSCI2, DECSCI2, OMEGA2
0
JORS2
38.6
40
80 120 160 200 240 280
OPSRES2
EJOR2
150
100
Actual sd 17.9
% zero
cites
18
Max
cites
176
15.5
22
19.0
14
14.0
12
28.6
10
42.4
5
50
Frequency
Actual
mean
Omega EJOR
MANSCI2
150
DECSCI2
OMEGA2
100
50
87
140
66
277
181
0
0
40
80 120 160 200 240 280
Citations in 6 good quality journals over a 15
year (1990-2005) period for 600 papers
0
40
80 120 160 200 240 280
0
Citation behaviour II - Dynamics
Number of Citations per Journal
900
800
700
OpsRes
Citations
600
Omega
500
ManSci
400
JORS
DecSci
300
EJOR
200
100
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Periods since Publication
Citations in 6 good quality journals over a
15 year (1990-2005) period for 600 papers
You cannot predict the final number of citations
from what happens in the first few years:
• Shooting stars: papers that gain a lot of citations very quickly,
often because they are empirical reviews of fashionable topics
but then die away
• Sleeping beauties: papers are are not cited initially but then get
recognised and cited for a long time. Often because they are
ahead of their time when first published
D. Basic citation metrics
1. Total number of citations for a person, department etc
Person A: 1 paper, 50 citations; Person B: 10 papers, 5 cites each
• Ignores how many papers have produced the citations
2. Citations per paper (CPP) – average citations per paper
A is still 50, B is only 5
• This is the standard measure in one form or another, but
• It now ignores the quantity of outputs completely, indeed it actually discourages you
from producing more papers as they may bring down your score
C: 1 paper 50 cites – CPP=50; D: 10 papers, 30 cites each – CPP=30
• Citation rates vary significantly across the disciplines so cannot be compared without
some form of normalisation to the disciplinary norms (cf Leiden methodology)
• They also vary with time – the older a paper the more citation it will have
• Can be skewed by some very high citations (cf h-index)
• Depends on robust data (cf h-index)
3. Journal Impact Factors
•
•
•
A 2-year CPP. The number of citations in year n to papers published in years
n-1 and n-2.
e.g., Acad Mgt Review has IF 6.169 so this is the average citations in 2011
to papers published in 2009 and 2010. For management this is very high –
many journals are less than 1.0.
But, in science we can get much higher figures: Ann Rev Immun – 52,
Physiol Rev 36, Nature 31
These are very short term. There is 5-year IF, but in social science citations
may not reach their peak until up to 10 years
4. Normalisation – the Leiden Methodology for a
department
•
•
•
Calculate the CPP for all the citations for all the papers over a fixed time
period (e.g., 5 years), let’s say it is 6.0 cites per paper
Look at all the journals from the field (e.g., Management), as defined in
WoS, and calculate the CPP for all the papers in all these journals over the
same time period, suppose it is 3.0
Then the field normalised CPP is 6.0/3.0 = 2.0. So this department is
getting cites at double the field average
5. The h-index
•
•
•
•
A relatively new measure created by a physicist (Hirsch) in 2005 that is
simple to understand and combines both impact and productivity:
0, 0, 1, 2, 4, 8, 9, 9, 10, 15, 25, 50, 100
There are 8 papers with at least 8 citations
His paper has 2700 citations!
In order to get a high h-index you need both a lot of papers and high
citations
A : 1 B: 5 D: 10
Advantages: it is a measure of overall contribution, not just quality; easy
to understand; robust to poor data; can be used for journals,
departments etc
Disadvantages: it ignores the higher citations; it is not pure quality but
productivity – so Einstein would have a very small h-index; it is very time
dependent and only ever gets bigger and so disadvantages early
researchers – can use the h-rate (h-index/years publishing): anything over
1 is good (in social science)
E. How can you improve your citations?
1. In an empirical analysis, I found the following to be the major
significant factors:
•
•
•
•
Quality of journal. This is problematic – better journals have better papers in the main
- but the effect was very strong
Institutional rank of the lead author (Times Higher) but this could be just a marker for
the quality of the author
Type of paper – review, theoretical, empirical
Size – pages and references
2. In my experience:
•
•
•
Be innovative and ahead of the game (sleeping beauties)
Provide something valuable – datasets, reviews – state of the art, applications,
literature
Go for the best journal you can (5 yr IF)
References
These are my papers but they will provide a way into the literature
Mingers, J., and Q. Burrell. Modelling Citation Behavior in Management Science Journals. Information
Processing and Management 42, no. 6 (2006): 1451-64.
Mingers, J., and A.-W. Harzing. Ranking Journals in Business and Management: A Statistical Analysis of the
Harzing Dataset. European Journal of Information Systems 16, no. 4 (2007): 303-16.
Mingers, J. Exploring the Dynamics of Journal Citations: Modelling with S-Curves. Journal Operational Research
Society 59, no. 8 (2008): 1013-25.
Mingers, J. Measuring the Research Contribution of Management Academics Using the Hirsch-Index. Journal
Operational Research Society 60, no. 8 (2009): 1143-53.
Mingers, J., and L. Lipitakis. Counting the Citations: A Comparison of Web of Science and Google Scholar in the
Field of Management. Scientometrics 85, no. 2 (2010): 613-25.
Mingers, J., and F. Xu. The Drivers of Citations in Management Science Journals. European Journal of
Operational Research 205, no. 2 (2010): 422-30.
Mingers, J., F. Macri, and D. Petrovici. Using the H-Index to Measure the Quality of Journals in the Field of
Business and Management. Information Processing & Management 48, no. 2 (2012): 234-41.
Mingers, J., K. Watson, and M.P. Scaparra. Estimating Business and Management Journal Quality from the 2008
Research Assessment Exercise in the UK. Information Processing and Management 48, no 6 (2012): 10781093
Open Access Publishing
• After the Finch Report the UK is moving rapidly towards open access
publishing. What does this mean for us?
• Forms of OA
• Green – much like now, in repositories such as KAR. Does not cost
anything but there may be delays for embargo period although
increasingly this is ignored
• Gold – the published version of the paper is immediately available to all
In most journals this will cost money
• Article Processing Charge (APC) – up to £3000 per paper! There may
well develop differential charges depending on the quality of the journal
• Who will pay – the universities basically. And for the change-over period they
will pay subscriptions as well.
• RCUK have said that all papers coming out of their funds should be gold
access
• The REF 2020 has said that it will only accept open access papers but not
clear if this includes green