Usability, the User Experience &
Interface Design:
The Role of Reference
July 30, 2013
Evaluating user behaviors and
experiences
Summon discovery service at Virginia Tech
By
Tracy Gilmore & Lesley Moyo
2
Introduction
• Continual assessment necessary in providing proactive
user support
• At Virginia Tech we endeavor to gather data on user
behaviors every spring
• Study of user experiences with discovery services is our
current focus
• Summon is our choice of Web scale discovery services
3
Collecting Data and Observing Behaviors
• Virginia Tech has adopted a mixed methods approach to
studying user research behaviors
1. Surveys
2. Focus Groups & One-on-one interviews
3. Usability task analysis
• This helps triangulate the data and gives us grounds for
making strong inferences
4
Unveiling the Data
• Using a robust survey tool is very helpful
• We used Qualtrics Research Suite: www.qualtrics.com
• Qualtrics offers strong data analysis features
• We crunched some of the survey data using SPSS to get
more detailed analysis
• NVIVO for analysis of interviews and focus group data
What we learned about user
behavior and preferences
6
Survey Results – User Preferences
User Preference - 2012
Preference for Summon:
• Stronger than Google
Scholar
• Stronger than the Library
Catalog
7
Easy to use
High preference
Moderately easy to use
Moderate preference
Neither easy nor difficult
Neither high nor low preference
User Preference - 2013
Moderately difficult use
Somewhat low preference
Difficult to use
Low preference
2012 Survey results - Predictors of satisfaction
Overall I am very
satisfied with Summon
Facets use & Satisfaction
Overall I am very
satisfied with Summon
Chi Square
229.90
Chi Square
41.40
Degrees of freedom
16
I always use the
facets in Summon
to improve my
search strategy
Degrees of freedom
16
P-value
0.00
P-value
0.00
Ease of use & Satisfaction
My rating of
Ease-of-use of
Summon
Top ranked journals &
Satisfaction
I can find articles
from top ranked
journals in my
field
8
Chi Square
Overall I am very
satisfied with Summon
246.68
Degrees of freedom
16
P-value
0.00
2013 Survey results - Predictors of satisfaction
Overall I am very
satisfied with Summon
Facets use & Satisfaction
Overall I am very
satisfied with Summon
Chi Square
263.08
Chi Square
54.90
Degrees of freedom
16
I always use the
facets in Summon
to improve my
search strategy
Degrees of freedom
16
P-value
0.00
P-value
0.00
Ease of use & Satisfaction
My rating of Easeof-use of Summon
9
Top ranked journals &
Satisfaction
Overall I am very
satisfied with Summon
I can find articles
from top ranked
journals in my
field
Chi Square
436.74
Degrees of freedom
16
P-value
0.00
User voices - Open ended questions
Comments 2012
Comments 2013
• I like that I only have to search in • The faceted searching (I didn’t
one place
know it was called that) is
absolutely essential
• On several occasions Summon
has found some good papers
• Ability to get many different types
that Google Scholar didn't
of sources with one search
• Summon is great if I know the
article I’m looking for
• I found it to be more helpful when
looking for books than articles
• I need more skill at searching
• Even with facets, Summon casts
too broad a net for my needs
• I get so many hits that are not
useful
10
User voices - Interviews/Focus groups
Comments
• Summon is easy to use
• Facets are a great tool
• Too many results!
• Boolean operators please
• Training would be nice
11
Suggestions
• Provide direct linking to
articles
• Provide more instruction on
how to use Summon:
• Using facets
• Using Subject Terms
Task Analysis observations
Strengths
Challenges
• Good results achieved
with advanced search
• Page navigation
• Effective use of
quotations for known
items
• Mixed success rates in task
performance
• Evident lack of familiarity with
Summon features
• Some participants able to
effectively use facets
• Large result sets still an issue
12
• Get VText (360 link resolver)
Enter librarian perspectives
• In spring 2013 - We included librarian voices
• This provided us with comparative perspectives
How do we as librarians interpret user
behaviors, given our own biases of what
research processes should look like?
13
Librarian voices
More students use it as a starting place, so when
troubleshooting that's what's involved more often than not
Summon has been especially helpful with interdisciplinary topics
even in identifying specific databases to more fully explore
Fast way of gathering extensive information and familiar to
the "Google generation”
I am a very happy user of Summon. The interface, the single user
box, and the facets are intuitive and easy to use
14
Changes implemented from findings
• Better integration in the instruction program especially at the
freshman level
• Improvements to the interface - better direct linking to full
text
• Increase in the content index - emphasis on eBooks and
reference sources
• Better visibility on the Library homepage
15
Going forward - Continuing assessment and evaluation
• Implementation of Summon 2.0 (anticipated fall 2013)
• Analysis of usage statistics from Google Analytics added to
Summon in November 2012
• Expansion of Mixed Methods study to academic research
libraries utilizing the Summon discovery service (fall 2013)
Contact information
Lesley Moyo
Director for Research & Instructional Services
University Libraries
Virginia Tech 560 Drillfield Drive
Blacksburg, VA 24061
Tel: 540-231-2708
Mobile: 814-777-7565
E-mail: moyo@vt.edu
Library Web: www.lib.vt.edu
17
Tracy Gilmore
Collections Assessment Librarian
University Libraries
Virginia Tech 560 Drillfield Drive
Blacksburg, VA 24061
Tel: 540-231-4956
Mobile: 540-239-4384
E-mail: tgilmore@vt.edu
Library Web: www.lib.vt.edu