Here Comes the Sunburst:
Measuring and Visualizing
Scholarly Impact
John Barnett
Scholarly Communications Librarian
Jennifer Chan
Assistant Scholarly Communications Librarian
Office of Scholarly Communication and
Publishing
University Library System
University of Pittsburgh
Here Comes the Sunburst:
Measuring and Visualizing
Scholarly Impact
University of Pittsburgh
 Pittsburgh campus + regional
campuses in Bradford,
Greensburg, Johnstown, and
Titusville
 16 undergraduate, graduate,
and professional schools
 456+ degree programs
 2012: conferred 8,949 degrees
University of Pittsburgh
 Top 10 American higher ed. in
federal funding (NSF)
 Top 5 in annual research support
(NIH)
 5,369 faculty; 4,470 full-time faculty
 Research conducted: more than 300
centers, institutes, laboratories,
clinics
University Library System
 ARL
 22nd largest academic library system in North
America
 25 libraries; 6.6 million volumes
 279,000 current serials
Sum of the Parts
Office of
Scholarly
Communication
and Publishing
ULS
Department of
Information
Technology
Liaison
Librarians
Why Pitt?
 Strategic goal:
Innovation in scholarly communication
 Providing services that scholars understand, need,
and value
 Putting ourselves in faculty “spaces”
 Re-envisioning our librarian liaison program
 Deepening our understanding of scholarly
communications issues
Why PlumX?
 Making research “more assessable
and accessible”
– Gathering information in one place
– Making it intelligible and useful
 Measuring and visualizing research
impact
 Correlating metrics from traditional and new forms of
scholarly communication
 Allowing researchers, labs, departments, institutions to
track real-time scholarly impact
 Promoting research, comparing with peers, connecting
with new research
Altmetrics Project Timeline
Spring
2012:
Fall 2012
• First meeting
with Plum
Analytics
• Gathered data
from pilot
participants
Summer
2012:
• Announcement
of Pitt as Plum
Analytics’ first
partner
Spring
2013
• Faculty
surveyed;
enhancements
made
Winter 2013
Fall 2013
• PlumX pilot
system made
public
• IR widget
launched;
rollout
preparations
Pilot project aims
 Develop a tool for measuring and visualizing
research impact
 Gathering information in one place
 Intelligible and useful
 Impact in social media and other scholarly
communication methods
 Traditional measures counted as well
 See where to disseminate works to increase impact
Traditional vs. new
• Traditional measures are also
counted
• Findings are complementary
to conventional methods of
measuring research impact
(e.g., H-Index)
• Not intended to replace them
New measures
 More comprehensive: Altmetrics = ALL METRICS
–
–
–
–
–
Citations
Usage
Captures
Mentions
Social Media
 Covers impact of online behavior
– Because scholars increasingly work online
 Measures impact immediately
– Because citation counts take years to appear in literature
Pilot Process
Created
Altmetrics
Task Force
CV receipt
and
records
creation
Engaged
Liaison
Librarians
Built
Sharepoint
Site
Pilot
Selection
Pilot Project Participants
• 32 researchers, various
disciplines
• 9 schools
• 18 departments
• 1 complete research group
• Others joined as they
learned about the project
Pilot Project Participants
discipline
school/department
Selected
faculty
participants,
diversified by:
online behavior
level of career
advancement
Technologies
 Internal
–
–
–
–
IR built on Eprints Platform
Sharepoint
Microsoft Office Suite
PMID/DOI data import tool
 External
– PlumX
– DOIs
– PMID
Data collection for pilot project
• Created records in D-Scholarship@Pitt, our
institutional repository
• Focused on articles, books, book chapters,
proceedings
• Scholarly output with standard identifiers
• DOI, ISBN, PubMed
ID, official URL, etc.
• Scholarship
produced since
2000
Other Library work
• Developed guidelines to standardize record creation
• Data entry from faculty CVs into IR (2 to 3 student
workers with QA by librarians)
• Librarian liaisons and other staff trained in record
creation
• SharePoint site used to track work completed
• Coordination with pilot faculty
• Gathered feedback and administered online survey
Sharepoint
 Altmetrics Meetings Minutes
 Faculty CVs
 Excel spreadsheets
 Word docs
External Data Sources
Metadata sources
 Faculty CVs . . . But verify metadata!
 Books: PittCat, WorldCat, Books in Print, publisher
sites, online retailers
 Journals: Serials Solutions list, journal websites,
JournalSeek, UlrichsWeb, DOAJ, PubMed
 Conference presentations: Websites, PittCat,
indexes, WorldCat
PMID Import Tool
 Custom build by SysAdmin for Eprints Platform
 Utilizing PMIDs from PubMed, able to import
records that prepopulate metadata fields
– Item Type, Title, Abstract, Creators, Publication Title, ISSN,
Volume/Issue, Page ranges, Date and Date type, DOI,
MeSH Headings, Grant Information, Keywords, etc.
Data Ingestion
Full-text sources
 DOAJ
 ERIC
 PLOS
 SSRN*
 Other repositories*
 Federal government websites*
 Conference websites*
* Use with caution
Plum Analytics processing activities
Harvest records from Pitt IR for each participant
Build profile for each researcher in PlumX
Harvest additional online artifacts NOT in Pitt IR
Use data mining to harvest publically available
metrics from hundreds of sites on the Web
Create visualizations to display metrics on PlumX
interface
Key features
 Faculty profiles
 Online ‘artifacts’
–
–
–
–
–
Article
Book
Book chapter
Video
Etc.
 Impact graph
 Sunburst
Faculty profile
Online ‘artifact’ display
Impact graph
Sunburst
Feedback
• Solicited via email and online survey
• Generally positive in most cases
• Data corrections
• Errors in profiles
• Links to wrong data
• Quickly corrected by Plum staff
• Requests for results from additional online sources
(Google Scholar, SlideShare, Reddit, etc.)
• PlumX collects data from these but did not gather information
in advance for profiles
The survey says
 Surveyed pilot project faculty in spring 2013
 @ 1/3rd responded to the survey
 Meaning 13 out of 32 participants responded
Accurate and useful data
90
76.92
80
70
60
Strongly disagree
Disagree
Agree
Strongly agree
50
40
30
20
10
15.38
7.69
0
0
Summary profile data page
The bar graph
70
58.33
60
50
40
30
25
20
16.67
10
0
0
Usefulness of interactive bar graph
Strongly disagree
Disagree
Agree
Strongly agree
The sunburst
60
50
50
41.67
40
Strongly disagree
Disagree
Agree
Strongly agree
30
20
8.33
10
0
0
Usefulness of sunburst
Traditional & new measures
60
54.55
50
40
36.36
Strongly disagree
Disagree
Agree
Strongly agree
30
20
9.09
10
0
0
Conveying traditional and new measures
Usefulness of altmetrics
60
54.55
50
40
Strongly disagree
Disagree
Agree
Strongly agree
30
20
18.18
18.18
9.09
10
0
The value of altmetrics
Learning something new
40
36.36
35
30
27.27
27.27
25
Series 1
Series 2
Series 3
Series 4
20
15
10
9.09
5
0
Learning something new about my own research
Comments
 Affiliations/bio inaccurate or has missing
information
 “Mentions” by whom & when?
 Publications misclassified
– Books vs. conference proceedings
 Data not collected
– Google Scholar
– Slideshare
Comments
 Filter out unwanted information
 Data are wrong—and not useful
 Overabundance of information in sunburst
 “I only care what a select group of scholars thinks
of my work”
 “I did not find this useful for my discipline”
Observations
 Lacked information about faculty practices
 Are the results useful to all faculty, all disciplines?
 May appeal more to faculty who are early in their
careers or whose work is more contemporary
 Will the data be used against faculty or programs?
 Labor-intensive strategy
 When it comes down to it . . . Does anyone care?
Embeddable widgets
(in development)
For researchers, to add to:
• their own Web pages
• department directories
• IR researcher profile page
For individual artifacts,
to build article level metrics
for imbedding in:
• IR document abstract page
• Article abstract page for
journals we publish
Roll-out challenges
 Who creates profiles? Who edits?
 What information should be included in profiles?
Who can view them?
 Separate data gathering from D-Scholarship
deposits?
 Who promotes the service? Who trains?
 Timing . . .
Future plans
 Data checking
 Additional data gathering
 Record merging/deduping
 Ability to edit user profiles and artifact records locally
 Open API
 To allow integration with other online systems
 More exhaustive scholarly practices survey for all faculty
 Rollout to all Pitt Researchers
 Will use automatic feed from Pitt IR to PlumX
Discussion
 How would you “sell” PlumX to additional faculty?