Got Data, Now What?
Analyzing Usability Study Results
Lynn Silipigni Connaway
June 26, 2005
Presented at the ALA 2005 Annual Conference
Chicago, IL
LAMA/MAES
Using Measurement Data for Library Planning
and Assessment Committee
Usability Testing: Why?
“Probably the best reason to test for usability is to eliminate
those interminable arguments about the right way to do
something. With human-factors input and testing, however,
you can replace opinion with data. Real data tend to make
arguments evaporate and meeting schedules shrink.”
(Fowler, 1998, Appendix, p. 283)
Usability Testing: Definition
 Degree to which a user can successfully learn and use a
product to achieve a goal
 Research methodology
• Evaluation
• Experimental design
 Observation and analysis of user behavior while users use a
product or product prototype to achieve a goal (Dumas and
Reddish, 1993, p.22)
 “User-centered design” process involving user from initial
design to product upgrade (Norlin and Winters, 2002)
 Approach is to be a servant to the users of a system NOT to
be subservient to technology (Gluck, 1998)
 Goal is to identify usability problems and make
recommendations for fixing and improving the design
(Rubin, 1994)
Usability Testing: Background
 Relatively new methodology (Norlin and Winters, 2002)
• Origins in aircraft design
• Traced back to marketing
• Development of a product
• Popular in 1980s with widespread access to computers
• Initiation of human computer interface usability studies
• Evolved from human ethnographic observation,
ergonomics, and cognitive psychology
• Qualitative and quantitative data
Usability Testing: Purpose
 Evaluation tool
 Identify problem areas
 “Determine the fit of the design to the intended users”
(Norlin and Winters, 2002, p. 5)
Usability Testing: Suitable Questions
 What is the best layout for a web page?
 How can you optimize reading from PDAs and small screen
interfaces?
 Which online fonts are the best?
 What makes an e-commerce site difficult to use?
 Can individual personality or cognitive skills predict Internet
use behavior?
 How can library collection holdings and library data be
represented geographically?
Usability Testing: Principles




Keep the end user in mind
Achieve superiority through simplicity
Improve performance through design
Refine and iterate (Norlin and Winters, 2002, p.10)
Usability Testing: Web Design Criteria









Links must be consistent and predictable
Group like things on the same page
Be consistent with language
Most important information should be on the first screen
Provide keywords for quick reading/scanning
Do not use animation or sounds
Make links look like links
Distinguish text from graphics
Avoid jargon (Spool, 1999)
Usability Testing: Web Design Criteria
 Ten Usability Heuristics (Nielsen)
• Visibility of system status
• Match between system and the real world
• User control and freedom
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from errors
• Help and documentation
Usability Testing: Web Design Criteria
 Goals for user-centered design
• Enable users to
• Achieve their particular goals and meet their needs
• Move quickly and with few errors
• Create a site that users like
• More likely to perform well on a product that provides
satisfaction
Usability Testing: Methodology
 Artificial environment (laboratory)
• Maintain more control
• May provide more specific data on a particular feature
 Natural environment
• Better holistic representation of real people doing real
work
Usability Testing: Methodology
 Four types of usability tests (Rubin, 1994, p. 31-46)
• Exploratory test – early product development
• Assessment test – most typical, either early or midway
in the product development
• Validation test – verification of product’s usability
• Comparison test – compare two or more designs; can be
used with other three types of tests
Usability Testing: Methodology



Develop problem statements, objectives, and/or hypotheses
Identify and select participants who represent target population
• May or may not be randomly selected
Select test monitor/administrator
• Empathetic
• Impartial
• Good communicator
• Good memory
• Able to follow test structure
• Able to react spontaneously to situations that cannot be
anticipated
• Allow user time for task
• Don’t rescue the user
• Continue with the plan if mistakes occur
Usability Testing: Methodology
 Design test materials
• Screening questionnaire
• Provides user profile
• Ascertains pretest attitudes and background
information
• Provides information about participants’ previous
knowledge and experience
• Orientation script
• Describes the test to participants
• Aids in understanding the participants’ performance
• Data logger materials
• Data collection instrument for categorizing
participants’ actions
• Can note time to match with videotape recording
Usability Testing: Methodology
 Design test materials
• Non-disclosure and tape consent forms for legal
protection
• Task list
• List of actions participants will execute
• Desired end results
• Motives for performing task
• Actual observations monitor will record
• State of system
Usability Testing: Methodology
 Design test materials
• Posttest questionnaire
• All participants asked the same questions
• Gather qualitative information and precision
measurements
• Debriefing guide
• Structure and protocols for ending the session
• Participants explain things not apparent in actions
• Motive
• Rationale
• Points of confusion
Usability Testing: Methodology
 Test materials and equipment
 Conduct the test
• Represent the actual work environment
• Users are asked to think aloud
• Observe users while using or reviewing the product
• Probe
• Controlled and extensive questioning
• Collect quantitative and qualitative data and measures
• Record comments or questions about the product
• Observe and document users’ behaviors
Usability Testing: Methodology
 Debrief
 Analyze the data
• Diagnose and recommend corrections
• Categorize and identify problems with the product
• Identify solutions
• Qualitative analysis
• Textual notes from debriefing
• Read responses
• Summarize findings
Usability Testing: Methodology
 Analyze the data
• Quantitative analysis
• Questionnaires
• Screening
• Posttest
• Triangulation to validate findings
• Data from questionnaires, observations, screen
tracking software, comments, and open-ended
questions
Usability Testing: Interpret Data
 Interpret the data
• Five factors for benchmarking the usability of an
interface (Shneiderman and Plaisant, 2004)
• Time to learn
• Speed of performance
• Rate of errors
• Retention over time
• Subjective satisfaction
Usability Testing: Interpret Data
 Interpret the data
• Prioritize severity of problems
• Severity ratings (Zimmerman and Akerelrea, 2004)
• Time required to complete task
• Number of users who encountered problem
• Negative impact on users’ perception of the product
• Difficult if 70% of users cannot perform task
• Error criticality = Severity + Probability of Occurrence
(Rubin, 1994)
Usability Testing: Interpret Data
 Usable Web site: (Rubin, 1994)
• Usefulness
• Establish whether it does what the user needs it to do
• Effectiveness
• Ease of use to achieve the desired task
• Learnability
• Ease of learning application and moving from being a
novice to a skilled user
• User satisfaction
• User’s attitude about the site—how enjoyable it is to
use
Usability Testing: Report Results
 Executive summary
 Report
• Describe methodology
• Who, what, when, where, and how
• Describe how tests were conducted
• Profile users and describe sampling
• Detail data collection methods
• Succinctly explain the analysis
• Provide screen captures
• Include tables and graphs
• Provide examples
• Identify strengths and weaknesses
• Recommend improvements
Usability Testing: Making the Data Work




Read report
Determine what worked and what did not work
Redesign product/system based upon findings
May be necessary to conduct another usability test
Usability Testing: Limitations
 Two major limitations (Wheat)
• Reliability
• Testing of users who are not atypical users
• Individual variation within the test population
• Validity
• Test tasks, scenarios of the search processes, and
testing environment are not accurate
• Results not generalizable to the entire user
population
• Testing is always artificial (Rubin, 1994, p.27)
OCLC WorldMapTM
 Research prototype
• Test geographical representation of WorldCat holdings
• By country and date of publication
• For library collection assessment and comparison
• Complement the AAU/ARL Global Resources Network
project
• Geographically represent library statistical data from
UNESCO, ARL, Bowker, and others
• Number of libraries by type
• Expenditures by library type
• Number of volumes and titles
• Number of librarians
• Number of users
Usability Testing: OCLC WorldMapTM
 Review sample handouts
• Screening questionnaire
• Task list
• Posttest questionnaire
• Executive summary
Usability Testing: OCLC WorldMapTM




Conducted informal usability tests
Currently redesigning the interface
Conduct second group of formal usability tests
Make revisions prior to making publicly available
Questions and Discussion
connawal@oclc.org