Chapter 4 – Expert Reviews,
Usability Testing, Surveys, and
Continuing Assessments
4.1 Introduction
• Designers may fail to evaluate adequately.
• Experienced designers know that extensive testing is
necessary
• Many factors influence the evaluation plan
• Evaluations might range from two-years to a few days
• Range of costs might be 10%  1% of a project
budget
• Customers are more and more expecting usability
4.2 Expert Reviews
• Formal expert reviews have proven to be effective.
• Experts may be available on staff or as consultants
• Expert reviews may take one-half day to one week plus
training
• There are a variety of expert review methods to chose
from
• Expert reviews can be scheduled at several points in the
development process
• Try not to rely on just one expert.
Expert Review Methods
• Heuristic evaluation
• Guidelines review
• Consistency inspection
• Cognitive walkthrough
• Formal usability inspection
Using Expert Reviews
• Danger: Experts may not have an adequate
understanding of the task domain or user
communities.
• Danger: may get conflicting opinions
• Expert reviewers are not typical users, and may
not relate completely.
• Helps to chose experts who are familiar with the
project and the organization.
• Beneficial to do usability testing as well
4.3 Usability Testing and
Laboratories
• There is increasing attention to usability testing
• Has benefits beyond usability
• Not controlled experiments
Usability Laboratories
• Might be set up to allow observation via
one-way mirror
• Staffed by expert in usability testing and
interface design
• IBM was an early leader
• Consultants available
Usability Testing Process
• Plan ahead
• Pilot test
• Choice of participants is important
• Other factors to be controlled
• Participants should be kept informed and respected
• Think Aloud protocols useful
• Videotaping is useful
• Test can be repeated after significant improvements
Field Tests
• Real environments instead of labs
• Still useful to log
• Beta testing is field testing
Paper Prototypes
• Obtain very early feedback, inexpensively
• Person plays the role of the computer,
displaying screens
• Allows capturing difficulties with wording,
layout, and sequences involved in tasks
Competitive Usability Testing
• Closer to controlled experiment
• Compare interface to previous version or
competitor
• Ensure tasks are parallel
• “Within Subjects” recommended
• Counter balance order
Issues with Usability Testing
•
Emphasizes first-time usage
•
Has limited coverage of the interface features.
•
Also use expert reviews
Heuristic Evaluation and
Discount Usability Engineering
Taken from the writings of Jakob
Nielsen – inventor of both
Heuristic Evaluation
• Context – part of iterative design
• Goal – find usability problems
• Who – small set of evaluators
• How – study interface in detail, compare to
small set of principles
Ten Usability Heuristics
•
•
•
•
•
•
•
•
•
Visibility of system status
Match between system and the real world
User control and freedom
Consistency and standards
Error prevention
Recognition rather than recall
Flexibility and efficiency of use
Aesthetic and minimalist design
Help users recognize, diagnose, and recover from
errors
• Help and documentation
How to Conduct a Heuristic
Evaluation
• More than one evaluator to be effective.
• Each evaluator inspects the interface by
themselves
• General heuristics may be supplemented
• Results can be oral or written
• Evaluator spends 1-2 hours with interface
• Evaluator goes through interface > 1 time
• Evaluators may follow typical usage scenarios
• Interface can be paper
Different Evaluators Find
Different Problems
Number of Evaluators
Heuristic Evaluation Results
• List of usability problems
– With principle violated
– With severity
• NOT fixes
• May have debriefing later to aid fixing
• Discount usability
Usability Problem Location
• Single Location
• Two/Several Locations
• Overall Structure
• Something Missing
Severity
• Help focus repair efforts
• Help judge system readiness
• Factors in Severity:
– Frequency
– Impact
– Persistence
– Market impact
• Scale severity to a number
• May wait on severity
H.E. Complementary w/
Usability Testing
• Each will find problems that the other will miss
• H.E. Weakness – finding domain specific
problems
• Don’t H.E. and Usability Test same prototype
version
Discount Usability Engineering
• “It is not necessary to change the fundamental way
that projects are planned or managed in order to
derive substantial benefits from usability inspection”
• 6% of project budget on usability
• 18% of respondents used usability evaluation
methods the way they were taught
More Discount Usability Engineering
• Cost projection to focus on usability may be
reduced
• “Insisting on only the best methods may result
in having no methods used at all”
• 35% of respondents used 3-6 users for
usability testing
• Nielsen and others suggest 50-1 ROI
Elements of Discount Usability
Engineering
• Scenarios
• Simplified Thinking Aloud
• Heuristic Evaluation
Scenarios
• Take prototyping to extreme – reduce functionality
AND number of features
• Small, can afford to change frequently
• Get quick and frequent feedback from users
• Compatible with interface design methods
Simplified Thinking Aloud
• Bring in some users, give them tasks, have
them think out loud
• Fewer users in user testing
Heuristic Evaluation
• Fewer principles etc to apply
• Compare interface to previous version or
competitor
• Ensure tasks are parallel
• “Within Subjects” recommended
• Counter balance order
Stages of Views of Usability in
Organizations
1.
2.
3.
4.
5.
6.
7.
8.
Usability does not matter.
Usability is important, but good interfaces can surely be
designed by the regular development staff as part of their
general system design.
The desire to have the interface blessed by the magic wand of
a usability engineer.
GUI/WWW panic strikes, causing a sudden desire to learn
about user interface issues.
Discount usability engineering sporadically used.
Discount usability engineering systematically used.
Usability group and/or usability lab founded.
Usability permeates lifecycle.
End Nielsen insert for Chapt 4
4.4 Surveys
• Users are familiar with surveys
• Surveys can provide lots of responses
• Surveys can be inexpensive
• Survey results can often be quantified
• Surveys can be complementary to usability
tests and expert reviews.
Successful Use of Surveys
• Clear Goals
• Preparation
• Don’t forget to gather background info
• Other goals concerning learning about the
user …
• Goals concerning the interface …
– reasons for not using an interface
– familiarity with features
– their feeling state after using an interface
Online Surveys
• Online surveys cut cost
• Online surveys may boast response rate
• Online survey may bias survey
Simple Survey
•
Use a simple scale
–
Easy for users
–
Directly quantifiable for use in statistics
•
Ask a few questions addressing goals
•
Few questions lead to higher response rate
•
Low cost, quantifiable results makes survey
repeatable
More Scaling
• Some surveys use bipolar alternatives
• E.g. Error messages were
– Hostile 1 2 3 4 5 6 7 Friendly
Not So Simple Survey
• Shneiderman’s Questionnaire for User Interface
Satisfaction (QUIS)
– Detailed info – gives specific feedback on many things
– Response rate will be lower
– Response bias to those highly motivated to help, very
patient, and/or not that busy
– Short form available for less patient
• IBM’s Post-Study Usability Questionnaire
• Software Usability Measurement Inventory
4.5 Acceptance Tests
• Large (particularly custom) software projects have “acceptance
tests”
• It’s time for something more specific than “user friendly” for
handling of usability
– Time to learn specific functions
– Speed of task performance
– Rate of errors by users
– Human retention of commands over time
– Subjective user satisfaction
• Multiple such tests - different components - different user
communities.
• After acceptance, field testing before full distribution..
• The goal all usability evaluation is to improve interface in the
prerelease phase, when change is relatively easy
4.6 Evaluation During Active Use
• Must continue to evaluate usability under
real use
• Improvements are possible and are worth
pursuing.
4.6.1 Interviews and Focus Groups
• Interviews with individual users
• After individual discussions, group discussions
4.6.2 Continuous User-Performance
Data Logging
• Software should enable collecting data
about system usage
• Logged data provides guidance
• E.g. Most frequent error message
• E.g. Most frequently used capabilities
• Pay attention to user’s privacy
4.6.3 Online or Telephone Consultants
• Online or telephone consultants provide assistance
to users
• Helpful to users when problems arise.
• Consultants can provide info about problems users
are having
4.6.4 Online Suggestion Box
• Provide facility to allow users to send
messages to the maintainers or designers.
• Easy access encourages users to make
productive comments
4.6.5 Online Bulletin Board
• Electronic bulletin board (newsgroups) permit
posting of open messages and questions.
• New items can be added by anyone, but usually
someone monitors the bulletin board
4.6.6 User Newsletters and Conferences
• Newsletters can help users, include requests for
assistance, promote user satisfaction
• Printed newsletters can be carried away from the
workstation and have respectability.
• Online newsletters are less expensive and more
rapidly disseminated
• Conferences allow workers to exchange experiences
with colleagues
• Obtaining feedback in these ways allows gauging
attitudes and gathering suggestions (as well as being
good PR)
4.7 Controlled Psychologically
Oriented Experiments
• Scientific and engineering progress aided by precise
measurement.
• Designs of interfaces will be improved if quality can be
quantified
• Scientific method as applied to HCI:
•
•
•
•
•
•
•
•
Deal with a practical problem – but within a theoretical framework
State a clear and testable hypothesis
Identify a small number of independent variables
Carefully choose the dependent variables
Carefully select subjects and assign to groups
Control for biasing factors
Apply statistical methods to data analysis
Resolve the practical problem, refine the theory, and give advice to future
researchers
• Controlled experiments useful in fine tuning the interface.
End Chapter 4