Empirical analyses of scientific
papers and researchers on Twitter:
Results of two studies
Stefanie Haustein, Timothy D. Bowman, Kim Holmberg,
Vincent Larivière, Isabella Peters, Cassidy R. Sugimoto, & Mike Thelwall
Background
• when Garfield created SCI, sociologists of science
analyzed meaning of publications and citations
(Merton, Zuckerman, Cole & Cole, etc.)
• sociological research
• What is it to publish a paper?
• What are the reasons to cite?
• empirical bibliometric research
• disciplinary differences in publication
•
and citation behavior
delay and obsolescence patterns
Background
• empirical studies helped sociologists to understand
structure and norms of science
• for bibliometricians, studies provided a theoretical
framework and legitimation to use citation analysis
in research evaluation
• knowledge about disciplinary differences and
obsolescence patterns helped to normalize statistics
and create more appropriate indicators
Background
• recently social-media metrics have become
important in the scholarly world
• suggestions to complement (or even replace)
citation analysis by so-called ”altmetrics“
• broader audience (not just citing authors)
• more timely
• however, similar to bibliometrics in the 1960s,
little is known about the actual meaning of various
social-media counts
Research questions
•
•
•
•
•
What is the relationship between social-media and
citation counts?
How do various social-media metrics differ?
Why are papers tweeted, bookmarked, liked…?
Who tweets (bookmarks, likes…) scientific papers?
How do these aspects differ across scientific disciplines?
Two case studies on Twitter
• large-scale analysis of tweets of biomedical papers
• in-depth analysis of astrophysicists on Twitter
Study I: Tweeting biomedicine
Aim of the study
•
large-scale analysis of tweets of biomedical papers
• Twitter coverage
• Twitter citation rates (tweets per paper)
• correlation with citations
• discovering differences between:
• documents
• journals
• disciplines & specialties
 providing empirical framework to understand the extent
to which biomedical journal articles are tweeted
Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting biomedicine: an analysis of tweets and citations
in the biomedical literature. Journal of the American Society for Information Science and Technology, http://arxiv.org/abs/1308.1838.
Study I: Tweeting biomedicine
Data sets & methods
•
1.4 million PubMed papers covered by WoS
• publication years: 2010-2012
• document types: articles & reviews
• matching of WoS and PubMed
• tweet counts collected by Altmetric.com
• collection based on PMID, DOI, URL
• matching WoS via PMID
• journal-based matching of NSF classification
• tweets per article, Twitter coverage and correlation
with citations for:
• journals
• NSF disciplines and specialties
Study I: Tweeting biomedicine
Data sets & methods: framework
Study I: Tweeting biomedicine
Data sets & methods: correlations
PY=2010
PY=2011
PY=2012
Study I: Tweeting biomedicine
Results: documents
•
•
Twitter coverage is quite low but increasing
correlation between tweets and citations is very low
Publication
year
Twitter
coverage
Papers
(T≥1)
Spearman's ρ
Mean
Median
Maximum
T2010
C2010
2.4%
13,763
.104**
2.1
18.3
1
7
237
3,922
T2011
C2011
10.9%
63,801
.183**
2.8
5.7
1
2
963
2,300
T2012
C2012
20.4%
57,365
.110**
2.3
1.3
1
0
477
234
9.4%
134,929
.114**
2.5
5.1
1
1
963
3,922
T2010-2012
C2010-2012
Study I: Tweeting biomedicine
Results: documents
Top 10 tweeted documents:
catastrophe & topical / web & social media / curious story
scientific discovery / health implication / scholarly community
Article
Journal
C
T
Hess et al. (2011). Gain of chromosome band 7q11 in papillary thyroid carcinomas of young patients
is associated with exposure to low-dose irradiation
PNAS
9
963
Yasunari et al. (2011). Cesium-137 deposition and contamination of Japanese soils due to the
Fukushima nuclear accident
PNAS
30
639
Sparrow et al. (2011). Google Effects on Memory: Cognitive Consequences of Having Information at
Our Fingertips
Science
11
558
Onuma et al. (2011). Rebirth of a Dead Belousov–Zhabotinsky Oscillator
Journal of Physical
Chemistry A
--
549
Silverberg (2012). Whey protein precipitating moderate to severe acne flares in 5 teenaged athletes
Cutis
--
477
Wen et al. (2011). Minimum amount of physical activity for reduced mortality and extended life
expectancy: a prospective cohort study
Lancet
51
419
Kramer (2011). Penile Fracture Seems More Likely During Sex Under Stressful Situations
Journal of Sexual
Medicine
--
392
Newman & Feldman (2011). Copyright and Open Access at the Bedside
New England
Journal of Medicine
3
332
Reaves et al. (2012). Absence of Detectable Arsenate in DNA from Arsenate-Grown GFAJ-1 Cells
Science
5
323
Bravo et al. (2011). Ingestion of Lactobacillus strain regulates emotional behavior and central GABA
receptor expression in a mouse via the vagus nerve
PNAS
31
297
Study I: Tweeting biomedicine
Results: journals
• 97.7% of 3,812
journals at least
tweeted once
• two-thirds of
journals have
coverage below
20% and Twitter
citation rate < 2.0
• high Twitter citation
rates often caused
by few papers
• high coverage and
Twitter citation rates
for general journals
Study I: Tweeting biomedicine
Results: disciplines
Study I: Tweeting biomedicine
Results: specialties
• specialties differ in
terms of coverage,
Twitter citation rate
and correlations
with citations
• 47 of 61 specialties
bubble size = Twitter citation rate
show low positive,
3 negative and 13
no correlation
Study II: Astrophysicists on Twitter
Aim of the study
•
in-depth analysis of astrophysicists on Twitter
• number of tweets, followers, retweets
• characteristics of tweets: RTs, @messages,
#hashtags, URLs
•
comparison with scientific output
• publications
• citations
• comparison of tweet and publication content
 provide evidence in how far astrophysicists on Twitter
use Twitter for scholarly communiation
Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists on Twitter: An in-depth analysis of
tweeting and scientific publication behavior. Aslib Proceedings.
Study II: Astrophysicists on Twitter
Data sets & methods
•
•
•
•
•
37 astrophysicists on Twitter identified by
Holmberg & Thelwall (2013)
web searches to identify person behind account
publications in WoS journals
• publication years: 2008-2012
• author disambiguation
Twitter account information
68,232 of 289,368 tweets downloaded and analyzed:
• number of RTs per tweet
• % of tweets that are RTs
• % of tweets containing #hashtags, @usernames, URLs
Holmberg, K., & Thelwall, M. (2013). Disciplinary differences in Twitter scholarly communication. In: Proceedings of ISSI 2013 –
14th International Conference of the International Society for Scientometrics and Informetrics, Vienna, Austria (Vol. 1, pp. 567-582).
Study II: Astrophysicists on Twitter
Data sets & methods
•
•
grouping astrophysicists according to tweeting and
publication behavior
analyzing differences of tweeting characteristics
between user groups
Selected
astrophysicists
tweet rarely
tweet
tweet
(0.0-0.1 tweets occasionally regularly
tweet
frequently
(N=37)
per day)
(3.7-58.2)
do not publish
(0 publications 2008-2012)
publish occasionally
(1-9)
publish regularly
(14-37)
publish frequently
(46-112)
total
(tweeting activity)
(0.1-0.9)
(1.2-2.9)
total
(publishing activity)
--
--
1
5
6
4
3
4
2
13
--
5
5
3
13
1
3
1
--
5
5
11
11
10
37
Study II: Astrophysicists on Twitter
Data sets & methods
•
comparison of tweet and publication content
• extraction of noun phrases from tweets and abstracts
• limited to 18 most frequently publishing astrophysicists
•
•
to ensure certain number of abstracts
analyzing overlap of character strings
calculating similarity with cosine per person and overall
Selected
astrophysicists
tweet rarely
tweet
tweet
(0.0-0.1 tweets occasionally regularly
tweet
frequently
(N=37)
per day)
(3.7-58.2)
publish regularly
(14-37)
publish frequently
(46-112)
total
(tweeting activity)
(0.1-0.9)
(1.2-2.9)
total
(publishing activity)
--
5
5
3
13
1
3
1
--
5
1
8
6
3
18
Study II: Astrophysicists on Twitter
Results: correlations
•
comparison of Twitter and publication activity and impact
• publications and tweets per day: ρ=−0.339*
• citation rate and tweets per day: ρ=−0.457**
• citation rate and RT rate: ρ=0.077
Study II: Astrophysicists on Twitter
Results: characteristics
Mean share of tweets containing at least one user name or
URL per person per group
Study II: Astrophysicists on Twitter
Results: content similarity
•
overall similarity between abstracts and tweets is low
• cosine=0.081
• 4.1% of 50,854 tweet NPs in abstracts
• 16.0% of 12,970 abstract NPs in tweets
• Twitter coverage among most frequent abstract terms is
high, although this differs between users
• 97,1% of 104 most frequent noun phrases on Twitter
Conclusions
•
•
Twitter coverage of biomedical papers is low but increasing
number of tweets per paper varies between journals,
disciplines, specialties and from year to year
 tweet counts need to be normalized accordingly
• correlations between tweet and citation counts are low
(biomedical papers) or even moderately negative
(astrophysicists)
 tweets cannot replace citations as measures of
scientific impact
 challenge is to differentiate between high tweet counts
because of value (to scientists and/or the general public)
and curiosity
Outlook
•
•
user surveys and qualitative research to investigate who is
using scholarly content on social media and why
empirical large-scale studies on other metrics
Haustein, S., Peters, I., Sugimoto, C.R., Thelwall, M., & Larivière, V. (in press). Tweeting
biomedicine: an analysis of tweets and citations in the biomedical literature. Journal of the
American Society for Information Science and Technology.
Haustein, S., Bowman, T.D., Holmberg, K., Larivière, V., & Peters, I., (submitted). Astrophysicists
on Twitter: An in-depth analysis of tweeting and scientific publication behavior. Aslib Proceedings.
Thank you for your attention!
Questions?
Stefanie Haustein
stefanie.haustein@umontreal.ca
@stefhaustein