Usability Assessment, Round Two:
Re-Examining the UW Libraries
Information Gateway
Jennifer Ward
University of Washington
jlward@lib.washington.edu
Presentation Overview
Background and History
 Current environment
 Hardware, personnel
 Efforts to date
 Lessons learned and futures

Looking Back
Move to Web interface for most
services – early 1998
 Rapid design, testing, and prototyping
of website
 Only formal usability testing was by
students as a class assignment

Current Environment
Web Steering Committee (WSC)
established August, 2000
 Decentralized website management
 Approximately 170 staff in 40 units with
write permissions
 No testing on current interface

Lab and Support

Usability lab
– Other labs on campus
– Currently in a shared conference room; will
have dedicated space soon
– PC with Camtasia, video camera

Half-time graduate student dedicated to
usability efforts
– Brought a fresh perspective to the process
Work with WSC to determine priorities
 System-wide support for assessment

Summer, 2001 Online Survey
July 23 – August 13, 2001
 49 unique responses; almost 1/3 each
faculty, students, staff
 Goal was to get a sense of what users
thought of the Gateway
 Based on a previous survey
 Used WebQ survey software, linked
from home page

Summer, 2001 Survey Results
Asked a lot of open-ended questions
about navigation and features
 Nearly 80% successfully found what
they wanted

– “Once one becomes familiar with the
site, it’s easy to navigate.”

Library Catalog most important resource
– Most requested new features focused on
catalog functionality
Points to Ponder

Timing is everything!
– Low response rate

Data quality
– Many open-ended questions went
unanswered
– Too focused on navigation and features;
need to broaden it
Autumn, 2001 Online Survey
September 21 – October 20, 2001
 Same goal as before, but do a better job
this time
 Plugged in online campus newsletter
 Incentive was provided, but not
advertised
 131 unique responses

– 10% faculty, 28% staff, 27% grad
students, 27% undergrads, 5%
alumni, and 2% non-affiliates
Autumn, 2001 Survey Structure
Restructured summer survey so we’d
get more answers
 Ran past a couple of local experts for
feedback
 15 questions, multiple choice or openended
 Users could remain anonymous

– Didn’t get affiliation, which would have
been nice for data analysis
Autumn, 2001 Survey Results
Participants for follow up studies
 Part of site that drew the most
comments was our catalog (!!)
 Overwhelming response – it’s not
“broken”

– Almost 80% very satisfied or satisfied
– Some areas need work and we’re
addressing them
– “Micro” level changes
Autumn, 2001 Survey Results

Asked what new feature users wanted:
– Some things were already on our site;
looked at making them more prominent
– Database selection wizard/helper
– Online tutorial for Gateway and catalog
– Better integration with campus
authentication
– $100 put into their bank account
Autumn, 2001 Survey Results
Data clumping
 Wording and Navigation
 Databases and Journals

– Show vendors on database list
– Need an “all-purpose” database instead of
having so many
– No clear winner on hierarchical vs
alphabetical listing of databases and ejournals
Card Sorting

November 13 – December 6, 2001
– Developed and started recruiting while we were
waiting for online survey to conclude
To determine the significance of
features of the Gateway and how users
would organize and name these
features
 Standardized procedures and documentation

– Consent forms, scripts, task lists

Participants
– one undergrad, one grad, one faculty, and
two staff members (not what we wanted)
Card Sorting – Tasks
Pilot Test
 Think aloud protocol, videotape, index
cards
 Two sets of tasks with 46 cards
 Rank according to perceived
importance

– Students more flexible than faculty/staff

Group according to their own
navigational structure
– Change wording if they wanted
Card Sorting – Results

Statistical analysis (dendrogram)
– More significant with more participants
Items in upper right are more important
 Confusing wording (i.e., Cascade,
Connecting, Starting Points)
 Didn’t understand News and Current
Events
 Have not used data in redesign

“By Subject” Pages





One of the “broken” items from the survey
and WSC’s perspective
Politically hot topic
How to help beginning researcher get started,
also try to address the layout
Hierarchical vs alphabetical presentation
Recruitment
– Targeted email, undergrads from library units
http://www.lib.washington.edu/subjects/
“By Subject” Pages – Tasks
Think aloud protocol, captured screen
activity with software
 Write answers to questions as they go
along
 Mockups on a test server
 Undergrad tasks varied – did they use
“Core Resources”?
 Graduate student and faculty tasks
were targeted to their discipline

“By Subject” Pages – Results

Undergraduates
1. Wanted listing by class name, searched
Yahoo! for everything (more comfortable)
– Wanted ranking system (4 stars)
2. Expert researcher who never left the test
pages; completed tasks in 2-3 minutes
– Understood and liked “Core Resources”
3. Stayed with test pages except when he
felt lost, then returned to his favorite
search tool – Google
– Drawn to resources with the folder icon
“By Subject” Pages – Results

Graduate Students
1.Went there once, wasn’t satisfied and
never came back
2.Not inclined to leave the pages; weakest
researcher

Faculty
1.Resources were already bookmarked
“By Subject” – Other Efforts
We’re working on adding a survey to the
bottom of these pages
 Search box on each page
 Simplify the design by removing
explanatory blurbs at top

Lessons Learned





However long you think something will take,
multiply the time by 3 and you might be close
Standardize the process as much as
possible; use templates and set guidelines
Refine tests before the involving users – run
pilot tests or talk with experts
Timing, timing, timing
Try not to put something online unless you
know it works. If users don’t find what they’re
looking for the first time, they don’t come
back.
Questions?

The website:
http://www.lib.washington.edu/usability/

Contact us:
jlward@lib.washington.edu or
libuse@u.washington.edu