Traditional Bibliometrics and
Alternative Metrics in Research
Program Assessment
LORRIE JOHNSON
U.S. DEPARTMENT OF ENERGY
OFFICE OF SCIENTIFIC AND TECHNICAL
INFORMATION (OSTI)
ICSTI TECHNICAL ACTIVITIES
COORDINATING (TACC) MEETING
OCTOBER 2013
Scientific and Technical Publication Metrics
• Scientific and technical publications, such as
journal articles, are the primary products of
scientific research, and the means by which new
knowledge is created and transferred.
• Publication metrics can provide useful insight into
research productivity, and into the impact and
influence of research within particular fields of
science.
Analysis of Joint Genome Institute Publications
Number of Publications by Year (2002-2012)
(908 articles were identified using Web of Science)
180
165
160
145
140
117
120
111
106
100
78
80
34
40
20
57
54
60
23
18
0
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
Aggregate JGI
Citation Data
Citations per Year
JGI publications (2002-2012)
12000
(from Web of Science)
JGI publications were
cited 51,590 times
(through April 2013)
Average citations per
article = 56
H Index = 113 (113
articles have been cited
more than 113 times)
Average citations per
year = 4,299
Citations for JGI
publications are
continuing to grow over
time.
10506
10000
9297
8000
7592
5686
6000
4457
4000
3336
2491
2619
2005
2006
1675
2000
837
0
33
2002
2003
2004
2007
2008
2009
2010
2011
2012
Top Journals for JGI Publications
Comparison of Citation Rates for JGI Articles versus Other Articles in Journal
Journal Titles
STANDARDS IN GENOMIC SCIENCES
# Articles
% of Total 5 Year Impact
(908)
Factor
Total Number of
Citations for JGI
Publications
Average Number of Average Number of Citations
Citations per JGI for All Articles Published in
Publication
Journal 2002-2012
168
19%
1.621
596
3.5
3.2
JOURNAL OF BACTERIOLOGY
72
8%
3.924
1222
17.0
20.2
PLOS ONE
42
5%
4.537
737
17.5
5.9
PROCEEDINGS OF THE NATIONAL
ACADEMY OF SCIENCES OF THE
UNITED STATES OF AMERICA
41
5%
10.472
3077
75.0
48.6
APPLIED AND ENVIRONMENTAL
MICROBIOLOGY
39
4%
4.453
2443
62.6
24.3
NATURE
36
4%
36.235
12669
351.9
61.7
SCIENCE
36
4%
32.452
9011
250.3
58.4
GENOME RESEARCH
29
3%
12.486
2454
84.6
59.5
BMC GENOMICS
28
3%
4.328
565
20.2
12.8
NUCLEIC ACIDS RESEARCH
26
3%
7.417
2142
82.4
34.9
ENVIRONMENTAL MICROBIOLOGY
20
2%
6.151
530
26.5
24.8
ISME JOURNAL
16
2%
7.850
334
20.9
17.4
MOLECULAR BIOLOGY AND
EVOLUTION
16
2%
9.857
865
54.1
37.5
NATURE BIOTECHNOLOGY
16
2%
28.161
1865
116.6
40.0
Citation Rates for 2008 JGI Articles in Nature
Compared to Average Citation Rate
2008 JGI Nature Articles
120
100
Amphioxus genome...
The Genome of the...
80
Phaeodactylum genome...
Genome of Laccaria
60
Trichoplax genome...
Genomics of cellulosic...
40
Early origins...
Microbiology...
20
2008 Nature Articles Average Citations/Year
0
2008
2009
2010
2011
2012
Citation Rates for 2009 JGI Articles in Science
Compared to Average Citation Rate
2009 JGI Science Articles
50
45
40
Green Evolution...
35
30
Genome Project...
25
20
Metagenome...
15
10
2009 Science Articles Average
Citations/Year
5
0
2009
2010
2011
2012
Analysis of Individual Publication:
The genome of black cottonwood, Populus trichocarpa…
Indicators of Impact and
Influence within Research Field
Paper was published in 2006;
citations peak in 2009-2011.
 Primary citations = 1,218
Citations by Year
250
 Secondary citations = 24,058
234
 Research Areas of Citing
217
206
198
200
163
150
125
100
56
50
12
0
2006
2007
2008
2009
2010
2011
2012
2013
through
4/13
Articles = 40
 Unique Authors of Citing
Articles = 5,149
 Distinct Institutions
Represented in Citing Articles
= 1,112
Analysis of Individual Publication:
The genome of black cottonwood, Populus trichocarpa…
Top 10 Research Areas of
Articles Citing this Article
Top 10 Journals Citing this
Article
Journal Title
# Publications
Research Areas
# Publications
BMC GENOMICS
72
PLANT SCIENCES
542
NEW PHYTOLOGIST
60
GENETICS HEREDITY
303
PLANT PHYSIOLOGY
50
BIOCHEMISTRY MOLECULAR BIOLOGY
265
BMC PLANT BIOLOGY
48
BIOTECHNOLOGY APPLIED MICROBIOLOGY
173
PLOS ONE
35
FORESTRY
81
PLANT JOURNAL
33
EVOLUTIONARY BIOLOGY
73
TREE GENETICS GENOMES
29
AGRICULTURE
72
PLANT CELL
28
CELL BIOLOGY
62
BMC EVOLUTIONARY BIOLOGY
23
SCIENCE TECHNOLOGY OTHER TOPICS
61
JOURNAL OF EXPERIMENTAL BOTANY
21
ENVIRONMENTAL SCIENCES ECOLOGY
48
Alternative Metrics for JGI Article in PLOS One:
“Direct Comparisons of Illumina vs. Roche 454 Sequencing Technologies on the
Same Microbial Community DNA Sample”
Article Metrics provided
by PLOS One:
Other Alternative Metrics
Measurements:
 Total article views = 23,384
 Mendeley readers = 212
 Total PDF downloads =
(89% in the Biological
Sciences; 25% of readers
from the U.S.)
 Cited by 17 articles in
PubMed Central
 46 citations in Google
Scholar
 Altmetric score = 21




4,403
4 Tweets
4 Facebook posts
15 bookmarks on CiteULike
20 cites on CrossRef
NISO Alternative Assessment Metrics
(Altmetrics) Project
 Sloan Foundation grant to undertake two-phase
initiative to explore, identify, and advance standards
and/or best practices for new assessment metrics,
including:
Usage-based metrics
 Social media references
 Network behavioral analysis

 Will also explore potential assessment criteria for non-
traditional research outputs, such as data sets,
visualizations, software, and other applications.
NISO Alternative Assessment Metrics
(Altmetrics) Project
 First phase – in person and virtual meetings to
solicit input and collaboration from community.
 ICSTI members are encouraged to participate:
Washington, DC – December 11, 2013
 Philadelphia, PA – January 23-24, 2014

NISO Alternative Assessment Metrics
(Altmetrics) Project
 Outcome from Phase 1 – white paper describing
outcomes, recommendations, and action items
identified during the meetings.
 Phase 2 and Final Outcome – Published standards or
recommended practices regarding alt metrics.
 For more information, see NISO project website:

http://www.niso.org/topics/tl/altmetrics_initiative/
 Or contact Todd Carpenter:

tcarpenter@niso.org