Evaluating User Interfaces
Walkthrough Analysis
Joseph A. Konstan
konstan@cs.umn.edu
October 10



CSci 5115
Introduction to Evaluation
Cognitive Walkthrough
Other Evaluation Methods
2
Interface Development
Methodology

Prototype and Iterate
 keep
iterating until it is good
enough
 evaluate along the way to assess

What is Good? What is Good
Enough?
 set
usability goals
 should relate to tasks
CSci 5115
3
Casual Iteration

Find major usability problems
 missing
features
 user confusion
 poor interaction

Try interface with specific tasks
 first
use designers, then move
towards users
 observe overall usage
CSci 5115
4
Casual Iteration

Remember the goal
 don’t
defend the interface
 don’t bias the tests towards the
interface

If possible, allow user exploration
 may
even lead to capturing new
tasks

CSci 5115
Consider alternative ways to fix a
problem
5
Limits of Casual
Iteration



CSci 5115
Does not indicate when to stop
Financial trade-offs
Justification of delay
6
Usability Goals and
Measures

Concrete, quantitative measures of
usability
 learning
time
 use time for specific tasks and
users
 error rates
 measures of user satisfaction

Comparative usability goals
 compare
with prior versions or
competitors
CSci 5115
7
Things to Watch

Goals should be realistic
 100%

is never realistic
Many goals go beyond the
application UI
 training,

manuals
Testing goals should help improve
the UI
 detail--not
CSci 5115
just good/bad
8
Exercise:
Setting Usability Goals

In project groups, come up with 2
usability goals for your project
 discuss
the feasibility of testing
these goals
 what
CSci 5115
is needed for the test
 when in the process can they be
tested?
 how much effort, user
preparation/training, etc.?
 what would you learn from the
test?
9
Interface Evaluation

Goals of interface evaluation
 find
problems
 find opportunity for improvement
 determine if interface is “good
enough”
CSci 5115
10
With or Without Users

Users are expensive and
inconsistent
 usability
studies require several
users
 some users provide great
information, others little

Users are users
 cannot

CSci 5115
be simulated perfectly
Best choice--Both
11
Evaluation Without
Users

Quantitative Methods
 GOMS/keystroke
analysis
 back-of-the-envelope action
analysis

Qualitative Methods
 expert
evaluation
 cognitive walkthrough
 heuristic evaluation
CSci 5115
12
Walkthrough Analysis

Economical interface evaluation
 low-fidelity
prototype
 development team
 users

optional
Effective, if
 goal
is improvement, not defense
 some team members skilled
 proper motivation
CSci 5115
13
Cognitive Walkthrough

Goals
 imagine
user’s experience
 evaluate choice-points in the
interface
 detect confusing labels or options
 detect likely user navigation errors

Start with a complete TCUID
scenario
 never
try to “wing it” on a
walkthrough
CSci 5115
14
Tell a Believable Story



CSci 5115
How does the user accomplish the
task
Action-by-action
Based on user knowledge and
system interface
15
Best Approach

Work as a group
 don’t

partition the task
Be highly skeptical
 remember

CSci 5115
the goal!
Every gap is an interface problem
16
Who Should Do the
Walkthrough


Designers, as an early check
Team of designers & users
 remember:
goal is to find
problems
 avoid making it a show

CSci 5115
Skilled UI people may be valuable
team members
17
How Far Along

Basic requirements
 description
or prototype of interface
 know who users are (and their experience)
 a task description
 a list of actions to complete the task
(scenario)
 DO

CSci 5115
NOT try to create the action list on the fly!
Viable once the scenario and interface
sketch are completed
18
How to Proceed

For each action in the sequence
 tell
the story of why the user will do it
 ask critical questions
 will
the user be trying to produce the effect?
 will the user see the correct control?
 will the user see that the control produces the
desired effect?
 will the user select a different control instead?
 will the user understand the feedback to
proceed correctly?
CSci 5115
19
Walkthroughs are not
Perfect

They won’t find every problem
 limited
by nature
 new
users who know what task they
need to accomplish
 biased towards correct action sequence
 limited
 hard

CSci 5115
in implementation
to shed the expertise of evaluators
A useful tool in conjunction with others
20
Exercise: Cognitive
Walkthrough Analysis




In non-project groups of 3-5
Users and Task to be announced
Scenario developed jointly
Perform walkthrough
 identify
problems
 estimate error probabilities (25%
intervals)

CSci 5115
Remember who your users are!
21
GOMS/Keystroke
Analysis

Formal action analysis
 accurately
predict task completion
time for skilled users

Break task into tiny steps
 keystroke,
mouse movement,
refocus gaze
 retrieve item from long-term
memory

Look up average step times
 tables
CSci 5115
from large experiments
22
GOMS/Keystroke
Analysis

Primary utility: repetitive tasks
 e.g.,
telephone operators
 benefit: can be very accurate
(within 20%)
 may identify bottlenecks

Difficulties
 challenging
CSci 5115
to decompose
accurately
 long/laborious process
 not useful with non-experts
23
Back-of-the-Envelope
Action Analysis

Coarse-grain
 list
basic actions (select menu
item)
 each action is at least 2-3 seconds
 what must be
learned/remembered?
 what can be done easily?
 documentation/training?

CSci 5115
Goal is to find major problems
 Example:
1950’s 35mm camera
24
Expert Evaluation

Usability specialists are very
valuable
 double-specialists


CSci 5115
are even better
An inexpensive way to get a lot of
feedback
Be sure the expert is qualified in
your area
25
Looking Ahead


Next week: Heuristic Evaluation
Walkthroughs Due
 “raw”
notes
• notes from each step of walkthrough
• copy of prototype used, markups
• copy of scenarios used (note changes
or fixes)
 processed
results
• 1-2 pages of issues identified,
solutions not needed
CSci 5115
26