Chapter 7
Usability and Evaluation
Dov Te’eni
Jane M. Carey
Chapter 7
Usability and Evaluation

Usability and evaluation are closely related
concepts
 Usability means that the HCI design of the
system supports the user’s cognitive and
ergonomic limitations and is easy to use and
learn
 Evaluation means to assess the system for
functionality as well as usability
 We explore how to compare, assess, and
improve interfaces
Chapter 7
Usability and Evaluation

Mini-Case 1984 Olympic message
system demonstrates:
1. You will not get it right the first time
2. You will get different answers when you
use different methods of observation
3. You will need to come up with integrative
solutions to problems
Chapter 7
Usability and Evaluation

Designers of the OMS (Olympic
Message System) followed 3 principles:
– Early focus on user and tasks
– Empirical measure on user’s behavior
– Iterative design
Chapter 7
Usability and Evaluation
Figure 7.2 Attitudes, use, performance, and satisfaction
relationship
Chapter 7
Usability and Evaluation

ISO (International Standards
Organization) defines usability as “ a
concept comprising the effectiveness,
efficiency, and satisfaction with which
specified users can achieve specified
goals in a particular environment”
Chapter 7
Usability and Evaluation
Table 7-1: Usability indicators based on performance
1. Goal achievement indicators (success rate, failure rate,
accuracy, effectiveness).
2. Work rate indicators (speed, completion rate, efficiency,
productivity, productivity gain).
3. Operability indicators of the user's ability to make use of the
systems features (error rate, problem rate, function usage)
4. Knowledge acquisition indicators of the user's ability and
effort in learning to use the system (learnability and learning).
Chapter 7
Usability and Evaluation
Table 7-2: Usability measures based on performance and HCI
1. Time to complete a task
2. Number of user commands to complete task
3. Fraction of task completed
4. Fraction of task completed in a given time
5. Number of errors
6. Time spent on errors
7. Frequency of online help used
8. Number of available commands not used
9. When task is repeated, ratio of successes to failures
10. Fraction of positive comments made by user
11. Fraction of good to bad features recalled by user
12. Number of expressions of frustration and satisfaction
13. Number of times user loses control over system
14. Number of times the user needs to devise a way of working around the
problem/system.
Chapter 7
Usability and Evaluation
Table 7-3: The usability engineering life cycle
· Know the user
· Analyze competing products
· Set usability goals
· Consider alternative designs
· Engage in participatory design
· Coordinate the total interface
· Check against heuristic guidelines
· Prototype
· Evaluate interface
· Design in iterations
· Follow up with studies of installed systems
Chapter 7
Usability and Evaluation

Methods of usability assessments
include:
– Thinking aloud (protocol analysis)
– Observation
– Interviews
– Focus groups
– Automatic logs
– Questionnaires
Table 7-4: Attitude questionnaire
1. Using a microcomputer could provide me with information that would lead to a better decisions.
2. I wouldn't use a microcomputer because programming it would take too much time.
3. I'd like to use a microcomputer because it is oriented to user needs.
4. I wouldn't use a microcomputer because it is too time consuming.
5. Using a microcomputer would take too much time away from my normal duties.
6. Using a microcomputer would involve too much time doing mechanical operations (e.g.,
programming, inputting data) to allow sufficient time for managerial analysis.
7. A microcomputer would be of no use to me because of its limited computing power.
8. I'd like to learn about ways that microcomputers can be used as aids in managerial tasks.
9. Using a microcomputer would result in a tendency to over design simple tasks.
10. I wouldn't want to have a microcomputer at work because it would distract me from my normal
job duties.
11. A microcomputer would give me more opportunities to obtain the information that I need
12. I wouldn't favor using a microcomputer because there would be a tendency to use it even when
it was more time consuming than manual methods.
13. I'd like to have a microcomputer because it is so easy to use.
14. I'd hesitate to acquire a microcomputer for my use at work because of the difficulty of
integrating it with existing information systems.
15. I'd discourage my company from acquiring microcomputers because most application packages
would need to be modified before they could be useful in our specific situation.
16. It is easy to access and store data in a microcomputer.
17. A microcomputer would be of no use to me because of the limited availability of application
program packages.
18. A microcomputer would be of no use to me because of its small storage capacity.
19. It is easy to retrieve or store information from/to a microcomputer.
20. Using a microcomputer would give me much greater control over important information.
Chapter 7
Usability and Evaluation
Step 7 in the engineering life cycle is check against heuristic
guidelines
Table 7-5: Heuristic guidelines
1. Create simple and natural dialog
2. Speak the user's language
3. Minimize the user's memory load
4. Be consistent
5. Provide feedback
6. Provide clearly marked exits
7. Provide shortcuts
8. Provide specific, corrective and positive error messages
9. Minimize propensity for error
Chapter 7
Usability and Evaluation

Evaluation is a broader term than usability – it
includes several goals:
1) assess the system's functionality against
the intended specifications,
2) assess the system's effect on the user's
behavior and attitude,
3) assess the system's impact on measures
of performance that are related to the user or
the objective of the system,
4) discover unintended problems and
perhaps opportunities.
Chapter 7
Usability and Evaluation

Evaluation techniques include:
1) Exploratory vs. model based.
2) Design or implementation.
3) Field study vs. laboratory testing.
4) Design vs. use.
5) Level of performance measures.
6) Degree of designed manipulation and
intrusion.
Chapter 7
Usability and Evaluation
Cognitive Walkthrough (Polson et al, 1992)
* Task description from the first-time user's viewpoint.
Include any special assumptions about the state of the
system assumed when the user begins work.
* Action sequence: Make a numbered list of the atomic
actions that the user should perform to accomplish the task.
* Anticipated users: Briefly describe the class of users
who will use this system. Note what experience they are
expected to have with similar or previous versions.
* User's initial goals: List the goals the user is likely to
form when starting the task. If there are other likely goals
list them, and estimate for each what percentage of user are
likely to have them.
Chapter 7
Usability and Evaluation
Table 7-6: Measures for comparing displays (Scott & Findlay,
1991)
· Typing mistakes made before correction
· Mistakes remaining
· Pauses of 3 seconds or more immediately before mode change
· Other pauses of 3 seconds or more
· Length of pause immediately before mode change
· Attempting to type-over whilst in insert mode
· Attempting to insert whilst in type-over mode
Chapter 7
Usability and Evaluation

Ben Shneiderman’s Questionnaire for
user satisfaction
Chapter 7
Usability and Evaluation
Other methods of evaluation
 Metrics (Rengger, 1991)

Chapter 7
Usability and Evaluation

Conclusion – Flow diagram from Olson
and Moran (1996)
Chapter 7
Usability and Evaluation
Table 7.8 ANSI/HFS 200 outline and status (1997)
Section Status
1 Introduction draft
2 accessibility draft
3 presentation of information ISO
document
4 user guidance ISO document
5 direct manipulation ISO document
6 color draft
7 forms fill-in ISO document
8 command languages ISO document
9 voice i/o
voice recognition draft
non-speech auditory output draft
interactive voice response draft
10 visually displayed menus redrafted ISO doc