Discovery or Displacement?
A Major Longitudinal Study of the Effect
of Web-Scale Discovery Services on
Online (Journal) Usage
SCELC Colloquium
March 5, 2014
Michael Levine-Clark, University of Denver
John McDonald, University of Southern California
Jason Price, SCELC Consortium
“…a steep increase in full text downloads
and link resolver click‐throughs suggests
Summon had a dramatic impact on user
behavior and the use of library collections
during this time period.”
The Impact of Web-scale Discovery on the Use of a Library
Collection
Doug Way (2010)
http://scholarworks.gvsu.edu/library_sp/9/
Vendor marketing
http://www.oclc.org/partnerships/econtent/solutions.en.html
Does implementation of a discovery
service impact usage of publisherhosted journal content?
What did we measure?
• Whether there is an
effect
• NOT why that effect
exists (that’s a future
study!)
• “Society will need to shed some of its obsession
for causality in exchange for simple correlations:
not knowing why, but only what”
• (Cukier & Mayer-Schonberger. 2013. Big data: A revolution that will
transform how we live, work, and think.)
Data collection
• List of libraries with discovery services
> Searched on lib-web-cats
• Surveyed Libraries
> Discovery service Implemented
> Implementation Date (month/year)
> Search box location
> Marketing effort
• 149 Libraries Gave Approval
> 33 libraries selected for this phase
> 6 for each of the 4 major discovery services and a
group of 9 libraries with no service
• 33 Libraries
Dataset
– 28 US, 2 CA, 1 each from UK, AUS, NZ
– WorldCat book holdings
> Average: 1,114,193 ; Range: ~300k to ~2.6mil
• Implementation dates (Discovery Libraries):
> 2010 (3), 2011 (19), 2012 (2)
• 6 Publishers
• 9,206 Journals
• 163,545 Usable Observations
Methodology
Compared COUNTER JR1 total full text article views for the
Included implementation month in Year 1 to ensure that
both periods included an entire academic year
End
Year 2
May 2012
Year 1
Implementation
May 2011
Start
June 2010
12 months before vs 12 months after implementation date
Examine Data for Outliers
Observations by Library & Service
Observations by Publisher
Average Usage Change By Discovery & Publisher
Analyzing Usage Change: % vs Total
Use 12
months
before
Use 12
months
after
% Change
Total
Change
Journal A
500
600
20%
100
Journal B
5
15
200%
10
Which is the better measure?
Is it the same for publisher- & journal-level data?
Reducing variation due to institution size
Currently converting to change per FTE
Values are shown as x 1,000 to bring the change
metric back per journal-library combination to a
minimum of 0.1
2013 JISC Discovery study took a similar approach
Average Usage Change By Discovery & Publisher
Per Journal &
Per 10,000 FTE
Full Model
Including Discovery Service, Publisher, and Library
Including Discovery Service, Publisher, and Library
Does the effect of discovery service differ across libraries?
Library 1-9
Library 10-15
Library 16-21
Library 22-27
Library 28-33
Nested ANOVA Model
[all three factors – preliminary results]
Does usage change vary across libraries?
Institution (sorted by Mean Change)
Does usage change vary across publishers?
Publisher (sorted by Mean Change)
Does usage change vary across discovery services?
Does the effect of discovery service differ across publishers?
Publisher
Results
Can we detect differences between Discovery
Services, Publishers, and/or Libraries and/or
their interactions?
• Library – Yes
• Publisher – No
• Discovery Service – Yes
• Differential discovery service effect by
publisher – Yes
Next Steps
• Design & test for effects of:
–
–
–
–
–
Aggregator full text availability
Publisher Size
Journal Subject
Overall usage trends (Requires Disc Srvc ‘control’)
Configuration options in Discovery services
• Expand pool of libraries
• Perhaps explore WHY
Sharing Data
• With participating libraries
– Customized reports for each library
• With participating publishers
– Customized reports for each publisher
– Presentations as requested
• With discovery vendors
– Presentations as requested
• In publications and presentations
– Maintaining anonymity of data
Doing “Resarch”, SCELC Style!
• Why SCELC?
• SCELC Discovery Project
– Partial funding provided by SCELC
• SCELC Participation
– Survey: http://bit.ly/DSparticipation
Past/Future Presentations
•
•
•
•
•
•
•
•
Ithaka Sustainable Scholarship Conference (October 2013)
Charleston Conference (November 2013)
ER&L/Library Journal Webinar (December 2013)
Shangai Jiao Tong Univ / Beijing Univ Forum (Jan 2014)
SCELC Colloquium (March 2014)
ER&L (March 2014)
UKSG (April 2014)
Presentations posted on slideshare; linked from:
– http://visualcv.com/lpq4t1s
michael.levine-clark@du.edu | johndmcd@usc.edu | jason@scelc.org
Sharing SCELC library experiences
with Discovery Systems
Lala Badal, California Lutheran University (WorldCat Local)
Linda Wobbe, Saint Mary's College (Ebsco Discovery Service)
Beth Namei, University of Southern California (Summon)
Panel Moderator: Jason Price (SCELC)
SCELC Colloquium March 5, 2014
Ten Minute Assessments
1) Two institutional goals each
a)
b)
All the same - Percent of content discoverable
All different - Choose one thats interesting and
assessable
1) Impact of your discovery system on instruction
and reference
1) What is the single biggest factor that would
increase use of your institution’s discovery
system?
WorldCat Local Discovery Service at CLU
Lala Badal,
California Lutheran University
March 2014
CALIFORNIA LUTHERAN UNIVERSITY
•35 undergraduate majors and 34 minors.
•Graduate programs – doctorates in educational leadership,
higher education leadership and clinical psychology.
•Master’s degrees in education, psychology, computer science,
business administration, public policy and administration,
information systems and technology, and economics.
•Total enrollment – 4,282 students.
PEARSON LIBRARY
Collection features about 132,000 physical volumes, 130 active
print periodicals, 158 databases, 219,000 eBooks, 65,000 fulltext
journals, and 10,500 streaming online e-videos.
WorldShare Management Services & WorldCat Discovery System
was implemented in September 2011.
Discovery System that
o Provides single point access to many library research resources
- In addition to our print collection, 80% of the library online resources overall are
discoverable and searchable in our WorldCat Local.
o Enables positive end user experience
o Increases usage of the subscribed electronic resources
Percentage of discoverable and searchable resources by material type
Full-text downloads: pre and post discovery system assessment
Online Catalog Usage
Selected Year (January 2013 – December 2013) Total: 472,907
52 Weeks Prior Total: 391,151
Selected Period (January 2014 - February 2014)
Total: 64,159
Use at Reference
Using WorldCat Local for reference assistance is
the “default action” for our
Reference/Information Commons “staff”.
If the discovery service search does not yield
relevant results, then they start using subject
related databases and/or research guides.
Impact on Instruction
The WorldCat Local introduced to all lower–division
classes.
All 4 librarians at CLU teach the discovery layer and if
needed additional resources (specific databases, ejournals, websites, and etc.) introduced as well.
History Subject Guide
Based on the CLU librarians impute and available
research on impact of discovery systems on information
literacy sessions, our Information Literacy Manager
outlined why, when and how we use DS in instruction
Impact on Instruction
Based on the CLU librarians impute and available research on impact of discovery systems on information literacy
sessions, our Information Literacy Manager outlined why, when and how we use DS in instruction:
Pros
Multidisciplinary searching
Intuitive information retrieval; less of a need to teach searching
Cons
Large result sets
Some lack of relevant results
Helps identify useful databases
Teach at the reference/IC desk
Not knowing how results are indexed
Some resistance from instruction librarians. Obviates/renders
unnecessary teaching students search process
Not precise enough for graduate-level research
Items/articles may not be covered in databases
Masks the complexities of the search process; makes it appear easier
than it really is
Less focus on differences in resources found
Good for articles when few are needed
Geared more to undergraduates
Ease of use
One-stop-shopping. Easier to sift through all the database offerings
Audio/video content more visible
Good for wide-range of topics/interdisciplinary
Good for searching a range of formats and materials
Shifts student thinking to critically evaluating what they found rather than
how to search
Shift away from “brand names” to streamlined searching
Instruction approaches
Introduce WCL as a place to launch your research, as one-stop shopping.
Metaphor: A discovery tool is similar to a large department store such as Wal-Mart where everything is available and the quality of the merchandise can vary,
whereas a subject database is more comparable to a boutique store, smaller selection but higher quality. Buck & Steffy (2013)
Class Activities
Have students explore the discovery tool and a more conventional database to discover similarities and differences. Then have a class discussion about the value
of each tool.
Have students work in teams of two to work on different options for refining a search and share the experience with the class.
Have students list the ways they might use a discovery tool in their future work.
Quick Writes (See lesson plan on the Discovery Tool vs. the Web)
WCL Lesson Plan - IL Session
What is the single biggest factor
that would increase use of WCL
discovery system at CLU?
AWARENESS
Unique Features:
- Allows users to discover relevant content beyond the library specific holdings
- Provides librarians with collection development tool and generates
more PDA through ILL
Unique Features:
- Provides patrons with opportunity to save searches and results, create and
share reading lists and bibliography.
Unique Features:
- Provides patrons with opportunity to save searches and results, create and
share reading lists and bibliography.
DISCOVERY SERVICE
ASSESSMENT
Linda Wobbe Saint Mary’s College of CA
2014
SMC Goals
◻
◻
◻
◻
Google-like searching
Access the entire Library collection
Discovery of overlooked resources
Improve cross-disciplinary access
Discovery: search everything?
What is missing?
Reference Desk
◻
Do you use Discovery at the Reference Desk?
YES: 100%
Reference
◻
◻
◻
◻
◻
YES: as a way to retrieve known articles.
YES: when I'm not sure what discipline a topic will
fall into.
YES: when no results are found in the subject
specific database, then I'll show Multisearch to
"discover" things that might be hiding somewhere
else.
YES: students often aren't sure whether they want
a book or article
YES: students at the desk need something NOW
and Multisearch is great for that.
Instruction
Do you teach Multisearch to undergraduates?
◻ YES: 67%
Instruction for undergraduates
◻
◻
◻
◻
YES: one search box is easy for students
YES: perfect for lower division courses
YES: "Google-like" experience that makes
research easy
YES: limit results by full text
Instruction for Undergraduates
◻
◻
NO: lower division classes focus on learning
the database for that discipline.
NO: brings up too much, which is
overwhelming for the lower division students'
needs.
Graduate Instruction
Do you teach Discovery to graduate students
NO: 67%
◻
Instruction for Graduate & Upper
◻
◻
◻
◻
YES: as a supplement to the disciplinary
database particularly with interdisciplinary
topics
NO: they need more options for limiting
NO: their more sophisticated needs are met in
a database that has subject nuanced search
features
NO: The search feature in subject specific
databases is detailed to the particular subjects
Increase use?
Number one:
Present Discovery Search as the default
◻
How to increase use?
◻
Improve precision
⬜
Material type limiters
⬜ Discipline-specific limiters
⬜ Take advantage of catalog scopes
◻
Sign truce with ProQuest
⬜
◻
Would percentage included to 60% from 50%
Enhance reference handbook
and statistical resources content
Multisearch Results
Impact & Assessment of Summon @ USC
Beth Namei,
University of Southern California
3/5/2014
USC’s Two Major Discovery Service
Implementation Goals:
#1:Provide better discoverability of our subscription
and purchased content (via a unified access point)
#2: Provide (more) relevant results
Goal #1: Provide better discoverability of our
subscription and purchased content (via a unified
access point)
757
85% (647)
of our database content is indexed by Summon
62,797
91% (56,980)
of our current journals (with active ISSNs) are
indexed in Summon
OpenURL ClickThroughs to Full-Text
Summon added as
default search option on
the libraries’ homepage
(July 2010)
Summon rolled out
Direct Linking to FullText, bypassing the
OpenURL link resolver
(Nov. 2011)
Top Referring Sources to Full-text Content (via our
OpenURL link-resolver)
In March 2013, 32 A&I databases were added to
USC’s Summon instance.
This led us to revise goal #2:
Provide better discoverability and access to information, via a
single search box, regardless of whether we own or subscribe
to the content.
March 2013:
32 A&I databases
were turned on in
Summon
(over 134 million
records)
New ILL Accounts Created
There was a
25% increase in
new ILL
accounts after
March 2013,
when USC
added an ILL
button to
Summon
Goal #2: Provide (more) relevant results
Typical complaints about our catalog:
Even if you know the title of the book you're looking for, HOMER searches
often result in random/not relevant results. I find myself going to
WorldCat just to locate books in our library! We need a better search
engine.
...say I was trying to find out if the library has "The Name of War" by Jill
Lepore. If I search "Name of War" or "The Name of War" as the title,
HOMER gives me a list of results that are totally wrong. In fact, the first
result is "Domesticating Vigilantism in Africa." What?? On the other
hand, if I go to Google Books or WorldCat and type in "The Name of
War," Lepore's book is the very first result.
April 2010 - Pre-Summon homepage
July 2010 - Summon is launched as the default tab
July 2012 - Summon-centric homepage (Homer tab is
removed)
The catalog is still an option, just a less prominent one:
via a dropdown menu
an icon below the main
Summon search box
Summon added
as default search
tab (July 2010)
Catalog search
tab removed
from homepage
(July 2012)
Over 3 Million Summon Searches in 2013
How Users are Getting to the Catalog (2013)
Success!
We have had no complaints about our
catalog’s (bad) relevancy since July 2012.
But…users are still not completely satisfied with the
results they’re getting from Summon
Usability Study of the Homepage, 2013
The USC’s version of Quick Search just doesn’t generally turn up reliable
results for me. I’ll search a very obvious keyword or a very specific keyword
and it won’t turn up the most relevant results first even though I know the
highly relevant results are in there it won’t bring them up so it’s probably a
backend USC libraries problem with the Quick Search function. But
because it’s difficult to narrow down by content and by type that I want and
because the results just aren’t that relevant or they don’t turn up the best
results first, I tend not to use Quick Search. Unless I’m feeling really, really
lazy and I have to turn something in in 20 minutes and I’m like ‘whatever’s on
top, I’ll take that!
What effect has implementation of Summon had
on library instruction and reference service at
USC?
Results from instruction survey
Surveyed 47 instruction/reference librarians; 25 responded (53%)
14 out of 15 librarians who teach introductory library sessions
for freshman writing courses teach Summon:
•
o 1 does not teach Summon at all
o 3 spend 1/4 of the class session demonstrating it (10-20 minutes), but
also demonstrate 1-2 (or more) other databases/tools.
o 6 briefly demonstrate it (no more than 5 minutes) before moving on to
demonstrating other databases/tools (most of these respondents said
they felt obliged to address it since it was so prominent on our
website)
o 5 spend the majority of their teaching time demonstrating it
Do you teach
Summon in upper
level/subject specific
courses?
Yes: 12
No: 9
n/a: 4
Do you use Summon at
the Reference Desk?
Yes 11
No: 12
n/a: 2
In what ways (if any) has incorporating Summon into
your instruction sessions changed your teaching?
•
•
•
•
hasn’t changed my instruction at all
allows for more time on other resources
I spend less time with other [resources] to incorporate this one
allowed for less lecture time spent on our webpage and what a
database is and more focus on content type, evaluations, building a
search. Also: I have significantly cut down my lecture time to
allow them to figure out these things through hands on activities.
•
I spend more time talking about the information cycle and what the
different source types mean. I think it's allowed me to focus a little
more on the higher-order skills of research because they don't have
to be bogged down in the tool.
What is the single biggest factor that would
increase use of USC’s discovery system?
better relevancy
&
better linking