Library Services within the Total User
Experience: Discovery and Digital
Scholarship
William Mischo
Grainger Engineering Library Information Center
University of Illinois at Urbana-Champaign
Library Services within the Total User Experience
National Institute of Technology Silchar
January 21, 2015
Core Library Services
• Activities that support the campus research and
instructional experience
– Discovery gateway systems that offer “smart”
search assistance that mimics the actions of a
trained reference librarian
– Dynamic digital environments for active
engagement with students and faculty
Library Services within the Total User Experience
• The role of the academic library; the
importance of establishing key campus and
external partnerships
Role of the Library
• Core values: library’s changing roles in
collection development, discovery and delivery,
and digital scholarship
• Library’s role in end-to-end lifecycle of
knowledge creation and knowledge
management
• Concept of local collection much less
meaningful
• Re-imagineering the library
Discovery and Delivery
• Libraries at a Crossroads
• Roger Schonfeld (Ithaka): “this is a good moment for
academic libraries to step back to reconfirm (or
reconsider) their vision for discovery, to ensure that
their visions connect with information-seeking
practices and preferences…”
• Exploring three technologies:
– Federated search
– Web-Scale Discovery Systems (WSDS)
– Hybrid Bento-style approach
Full Library Discovery
• Overarching issue is how to optimize access within
the complex information landscape we live in: online
catalogs, publisher full-text, institutional repositories,
A&I services, local and multi-institutional digital
content, learning resources, etc.
• Full Library Discovery – Lorcan Dempsey
• Not only collections but services – subject area
specialists, library webpages, smart search assistance,
pathfinders, *personalization, etc.
Search & Discovery
• Gateway function and full-text is King
• Federated search: broadcast and merge over remote
information targets
• Web-Scale Discovery Services: single aggregated
index harvested from content holders
• Hybrid bento-style federated search where the
screen display is partitioned into tile areas
• Numerous libraries investigating Bento
Search & Discovery
• University of Illinois at Urbana-Champaign (UIUC)
Library discovery efforts:
– Easy Search federated search and recommender
system
– WSDS: Primo Central, 2011-2014; WorldCat
Discovery, 2013-; Ebsco Discovery Service API
– Bento-style search and display
Full-Text Linking:
-- CrossRef and DOI
-- Local Link resolver
Areas of Focus:
Discovery and Delivery Environment
Total User Experience Focus
-- A&I via OpenURL local Link
Resolver
--Search Assistance
--Information Content Integration
--Full Library Discovery
-- Google Scholar and Link Resolver
Direct
Links
Subject Area Identification
-Open Worldcat Class Label
-Recommended targets
-Departmental library
-Librarian subject specialist
Federated Search (broadcast)
, Web-Scale Discovery (aggregation)
Hybrid Bento-style Search/Display
Easy Search API
Direct Links (Best Bets)
Libguide Matches
Spelling Suggestions
Subject Area Identification
Author Suggests
Limit/Broaden Suggests
Added Title Search links
Journal AtoZ Matches
DOI matches
Journal & Article Locator
Link Resolver
A to Z Search
“I am looking for a specific item”
“I need articles, books about X”
Information target
HTTP Connectors
Transaction
Logs
LibraryWebsite
Search
Web-Scale Discovery
Systems
Users
Digital Content
Management
Repository
Google Discovery
Services APIs
National/International Digital Platforms
DPLA, Hathi, LOCKKS, CrossRef,
DataCite (datasets)
Multiple Publisher Repositories
e-journals, e-books
Dark Target
Searches
Course
Management
Systems
OPACs
-UIUC VuFind
-State VuFind
-Voyager
-WorldCat Discovery-
A&I article databases
APIs
Scopus
API
Ebsco
EDS API
University of Illinois Library Gateway
• Revised Gateway portal introduced in
September 2007, powered by Easy Search
federated search system:
o Recommender and discovery system
o Employs context-specific and adaptive Search
Assistance mechanisms
o Helps with search strategy modification and
navigation
o Takes users into native interfaces at point of
completed search
o Writes out custom transaction logs
Evidence-Based Services
• We believe in user testing and incorporating
user needs in systems
• User behavior studies and custom transaction
log data and user survey input
• Online Catalog user studies have been
contradictory
• Increasing usage of Gateway as access
mechanism for known-item search
10
UIUC Library Gateway
UIUC Library Gateway
UIUC Library – Bento Display
UIUC Library – Bento Display
Digital Scholarship Centers in Libraries
• Support growing & widespread e-research and digital
scholarship
• Bring together expensive technologies for use by all
campus departments
• Bring together expertise to serve multiple campus
departments
• Support graduate and undergraduate students who
desire to create digital projects but do not have access
to tools and expertise in their department
Library/Campus Partnership
• Library as place, Place as service
• Collaboration, e-scholarship, innovation, experiment
with new forms of scholarship and publishing
• Knowledge visualization, data visualization, data
analytics, next-gen design, user interaction
• Lifecycle of scholarly content. Helping the institution
become a learning organization
Digital Scholarship Center facilities: UCLA
Visualization facility at Hunt Library NCSU
Brown University Library
Brown University Library
Georgia State University Library
Grainger Engineering Library Information Center: Illinois
Data Management Issues
• U.S. federal grant agencies require Data
Management Plans: NSF, NIH, DOE
• The 2013 White House OSTP mandate and
White House memo
• Need to demonstrate importance to campus of
a campus-wide infrastructure for data
management
• Multiple campus stakeholders: library,
academic computing center, research offices,
colleges, departments
UIUC Library Initiative factors
• There is a growing number of disciplinary and
government repositories
• We wanted to lead the efforts for a campuswide program
• Wanted to show upcoming compliance issues,
dangers of current practices
• Approached offices of: Vice-Chancellor for
Research and Provost
NSF Data Management Plans
• Data Management Plans (DMPs): required
element in NSF proposals, January 2011
• July 2011: the Library, working with the campus
Office of Sponsored Programs and Research
Administration (OSPRA) began an analysis of
DMPs in submitted NSF grant proposals
• Currently, looked at 1,600 grants with 1,260 in
the analysis.
Reasons for DMPs
• Make research data available and sharable
• Use of data for verification of results and
reproducibility of research
• Federal agency can show significant return on
investment to justify funding
• Important to know storage venues and
mechanisms for sharing and reuse
• Use of local templates and local campus
resources such as institutional repository
Reasons for Analysis
• What storage venues and mechanisms for
sharing and reuse are being used?
• Are the PI’s using local templates and local
campus resources such as the IDEALS?
• Is the campus in danger of losing data? Of losing
grants because of compliance issues?
Analysis
• Analysis attempts to characterize and classify
DMPs into categories
• DMPs assigned multiple categories
• 1,260 DMPs from July 2011 to November 2013
Data Venue and Risk
Submitted Proposals
Data Location
Funded Proposals
Risk of Loss/Corruption/ Breach
n=298
n=1260
PI Server/Website
Departmental
Server/Website
Campus-Wide Resource
IDEALS (Institutional
Repos.)
NCSA
Disciplinary
Repository/Cloud
Remote Repository
Optical Disk, Specimens,
Analog
64%
High
61%
High
11.2%
Medium to High
7%
Medium to High
52.9%
21.9%
45%
Low
4.3%
19.8%
Low
16.4%
25.8%
Medium to Low
21.4%
Medium to Low
28%
Medium to High
22.8%
Medium to High
19.4%
Out of Scope
11%
Out of Scope
Analysis
• Any differences in storage venue or technologies
between the unfunded proposals and the funded
proposals?
• Any differences between the proposals from the
first year and the more current proposals?
• Other differences in proposal categories
between funded and unfunded
• 734 active NSF awards, $861.8 million
Implications and Conclusions
1. No significant differences between
funded/unfunded proposals in storage venues no advantage in specifying IDEALS or
Disciplinary
2. More recent proposals suggest IDEALS and
disciplinary repositories included at a
significantly higher level
• Questions remain: what is the role of the library?
The campus? The subject discipline?
Linking of Data and Article Literature
• The standards are CrossRef DOIs and DataCite
DOIs: we need bi-directional links
• Role of publishers in data storage and linking
• Linking is of interest to publishers, authors, grant
agancies
• Potential for a Linked Open Data application
• Potential for data repository recommender
service
Linking of Data and Article Literature
• Example is search of Scopus to retrieve an article
by an author or on a topic
• The article DOI (CrossRef DOI) is extracted and a
link to DataCite using this CrossRef DOI is
constructed
• The matches at DataCite are displayed along
with their links from the DataCite DOI to the
dataset results landing page/producer page
Follow-on
• Vice-Chancellor for research and Provost have
funded a campus-wide Research Data Service RDS
• Assist in compliance with federal agencies and
assist researchers with DMPs, DOIs, storage
mechanisms both short-term and preservation
• Develop important partnerships with campus
units and national entities
Illinois Environmental Scan
• Subscribe to 200+ specialized article and report
databases
• 65K online journals
• Collection budget of $15 million
• Online is, in many cases, the exclusive access
point
• Have a legacy online catalog (search over
metadata, holdings of books, journals, +)
• We operate in a distributed, heterogeneous
information environment
35
Evidence-Based Services
• We believe in user testing and incorporating
user needs in systems
• User behavior studies and custom transaction
log data and user survey input
• Online Catalog user studies have been
contradictory
• Increasing usage of Gateway as access
mechanism for known-item search
36
We Do Tools and Services
• Debate over role of Library in “discovery”
• Desire for one-stop-shopping
• We look at custom transaction logs over Gateway and
ES Advisory Committee
• Easy Search recommender and search assistance tool
• Known-item access important
• Problem with blended displays and full-text searching
37
Today’s Library Gateways
• Academic libraries are moving from federated
search systems to web-scale discovery systems
(WDS) with centralized aggregated indexing to
Bento-style hybrid search and discovery
• WDS feature single-entry search boxes with faceted
limiting operations’ is faceting useful?
• As we transition to WDS & Bento, understanding
and supporting user search behaviors is important
Retrieval Issues
• Little use of advanced systems; facet use is
moderate, about 30%
• Full-text search retrieval and relevancy
ranking issues
• Large number of known-item cut-and
paste searches
39
Web-Scale Discovery Features
• Hosted and in the networked Cloud
• Aggregated index of licensed publisher-based
bibliographic content – journal articles, newspaper
articles, book chapters, dissertations, and local
OPAC and digital content
• Single entry search box
• Blended display format with limiting facet
operations & other Web 2.0 features
• Central aggregated index contrasts with the
federated search/broadcast search approach
40
Why are Libraries Embracing WDS?
• Rival to Google; hope to bring back users
• Extend the OPAC and integrate local
digitized content & Institutional Repository
• Negative experiences with federated
search/broadcast search
• One-stop shopping approach attractive
• Next Big Thing – keep up with others
41
The Big Issues
• Understanding and supporting user search
behaviors is important
• The effectiveness of full-text search and
relevancy ranking is under question
• Some modules (e.g. OPAC) may be better
than central index search/discovery
• Important to distinguish between
discovery features and known-item access
mechanisms
42
Easy Search Custom Transaction Log
• We record user actions, system suggestions, search
reformulations, and clickthroughs
• 2010-2011 study of 1.4 million searches and 1.5 million
clickthroughs
• 2014 study of 1 million searches, 1.1 million clickthroughs
over 10 month period May 2013 to March 2014
• Initially supported by NSF and IMLS
• OPAC studies ambiguous results; web search engine
behaviors different
Our Study of Custom Transaction Log Data
• Conducted an evidence-based study of user
behavior in the Easy Search University of Illinois
Gateway
• Looked at 1.4 million searches and 1.5 million
clickthroughs performed by users in Fall 2010
and Spring 2011 semesters; now updating
• Our transaction log studies were supported by
National Science Foundation and Institute of
Museum and Library Studies
UIUC Library Gateway
UIUC Library Gateway
UIUC Library – Bento Display
UIUC Library – Bento Display
Examples
• Competent Jerks, Lovable Fools, and the Formation of Social Networks
• The virtual supermarket: An innovative research tool to study consumer
food purchasing
• Mothers and fathers : a study of the development and negotiation of
parental behaviour
• Matouschek, Kellis, Serrano, Fersht Nature
• Kinsella and Phillips 2005
• Mowen, A. J., Kyle, G. T Jackowski, M. (2007). Citizen preferences for the
corporate sponsorship of public sector park and recreation organizations.
Journal of Nonprofit and Public Sector Marketing, 18(2), 93
• Hemenway, D. (2010). Why We don't Spend Enough on Public Health.
New England Journal of Medicine, 362(18), 1657
• university, need for money
• J. Quant. Spectrosc. Radiat. Transfer.
2014 Transaction Log Analysis
• Breakdown of searches from the main Gateway:
– Easy Search and associated tabs:
902,420
76.5%
• the gateway Easy Search default tab: (55.1%)
– advanced search:
– departmental library page searches:
– myeasysearch:
– search suggestion searches:
• Total:
56,170
158,750
18,159
42,708
4.8%
13.5%
1.5%
3.6%
1178207
50
Search Assistance Features
• Context-specific suggestions and prompts
for:
– Spelling changes
– Redo as author search
– Direct links to frequent or special search terms or
Libguides
– Links to matching e-journal title(s)
– Links to citation linker for full-text retrieval
– Limit results to exact phrase, title word(s), or title
phrase
– Dark background searches displayed if needed
2014 Transaction Log Analysis
• Gateway searches total: 902,420
– Easy Search tab:
– Books search tab:
– Article tab:
– Journal Title tab:
649,320
106,375
76,375
70,350
• 2007: 49.4% of all searches were known item
• 2011: 51.2% of searches in 54% of sessions were
known-item
52
2014 Transaction Log Analysis
• Sample of 974,137 user-entered searches:
– average words per query is 5.11 words
– This is up from earlier studies:
• 3.58 words per query in 2008; 3.76 words per query in
2009; 4.33 words per query in 2011
• 93,642 one word, 158,054 greater than 8 words
• 49.6% were 3 words or less, 29.4% are 6 words or more
• longest search was 299 words
53
2014 Transaction Log Analysis
• 20.98% of all searches in 29.13% of all sessions
used a search assistance suggestion or custom
added search result link
• 57.2% of the 489,272 search sessions were single
query sessions
• 6% of the sessions contain 6 or more queries,
1,157 sessions contain more than 20 queries, and
96 search sessions contain more than 50 queries
54
Primo Clickthrough Study May-June 2014
• Primo clicks 15,068 times between 05-01-2014 and
06-21-2014. Total of 147,326 clickthroughs during
that time
• Looked at a sample of 478 of the 15,068 searches
• 245 known-item searches (51.8%) and 228 topical
searches (48.2%)
• Of the 245 known-item searches, 159 were judged
successful - they brought back the expected results in
the Primo first page or as the top item
55
Primo Clickthrough Study (2)
• Of the 159 ‘successful’ searches, all of them were also
successful in Scopus, Ebsco, CrossRef, WorldCat
Discovery, Web of Knowledge, VuFind, IShare, Google,
or Google Scholar searches
• Removing the Google results left 9 searches that were
successful in Primo but not via Scopus, Ebsco, Web of
Science, CrossRef, VuFind, IShare, or WorldCat
Discovery. Removing CrossRef and WorldCat
Discovery resulted in 35 searches successful in Primo
but not successful in Scopus, Ebsco, Web of Science,
VuFind, or IShare
56
Primo Clickthrough Study (3)
• Found that CrossRef and WorldCat Discovery
good at retrieving successful search results for
full citation searches
• 48 known-item searches that were unsuccessful
in Primo but successful in one of the other Easy
Search targets - Scopus, Ebsco, Web of Science,
CrossRef, VuFind, IShare, WorldCat Discovery, or
Google/GS
57
Primo Clickthrough Study (4)
• Evaluated sample against Arizona State’s
Summon. In one case, a known-item search that
was unsuccessful in Primo yielded a first page
match in the AZU Summon. However, there were
50 known-item searches that were successful in
our Primo that were unsuccessful (item not on
top or first page) in the AZU Summon
58
Easy Search and Bento approach user survey
• 120 survey results that users of Easy Search
provided in October 2014
• 43 were daily users and 53 indicated they were
weekly users
• 99 surveys where users provided us with
comments on their Easy Search experiences. In
77, users made either a very favorable (28) or
favorable comment about Easy Search
59
Easy Search and Bento survey (2)
• 91 surveys with comments about the Bento
approach with 66 of these providing very
favorable or favorable comments
• 40 users with Very favorable/favorable views of
ES who also held Very Favorable/Favorable views
of the Bento search
• 17 user surveys where ES was regarded as Very
Favorable/Favorable and the Bento version was
regarded as Unfavorable or Neutral
60
Easy Search and Bento survey (3)
• 21 surveys where ES was regarded as
Unfavorable or Neutral and the Bento version as
Very favorable/Favorable
• Overall, this shows a significant acceptance of the
Bento search and display approach
• Note that one of the Bento options is to reperform the user search in “classic” Easy Search
61
Bento Table
• Table comparing the feature sets of 17 Bento
search/display implementations deployed by
university libraries, including our version
• Number of Bento versions that do not do spell
checking and do not provide direct links from the
bento display page to full-text
• None of the others employ the one-click to fulltext PDF that we do
62