Heuristic Evaluation
Introduction to Human Computer Interaction & Design
Hao-Hua Chu
National Taiwan University
May 17, 2016
*** Adapt teaching materials from the Stanford HCI course (with permission & many thanks to Prof.
James Landay of Stanford) as well as Stanford D.School
9/22/2015
1
Outline
• Heuristic Evaluation Overview
• The Heuristics
• Exercise
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
2
Evaluation & Evaluation Techniques
• About figuring out how to improve design
• Issues with lo-fi tests?
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
3
Evaluation
• About figuring out how to improve design
• Issues with lo-fi tests?
Not realistic
– Lack visuals affordance not
realistic
– Performance
Not on actual interface
– can’t test alone
Need participants
– can be hard to find
repeatedly
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
4
Heuristic Evaluation
• Developed by Jakob Nielsen
• Helps find usability problems in a UI design
• Small set (3-5) of evaluators examine UI
– independently check for compliance with usability
principles (“heuristics”)
• In comparison to focus group
– evaluators only communicate afterwards
• findings are then aggregated
• Can perform on working UI or on sketches
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
5
Why Multiple Evaluators?
• Every evaluator
doesn’t find every
problem
• Good evaluators
find both easy &
hard ones
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
6
How many evauators? Decreasing Returns
problems found
benefits / cost
* Caveat: graphs for a specific example
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
7
Heuristic Evaluation Process
• Evaluations go through UI several (twice) times
– Inspect various dialogue elements
– Compare with list of usability principles
– Consider other principles that come to mind
• Usability principles
– Nielsen “Heuristics”
– Supplementary list of category-specific heuristics
• Competitive analysis & user testing of existing products
• Use violations to redesign/fix problems
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
8
Neilsen’s original Heuristics
•
•
•
•
•
•
•
•
•
•
H1-1: Simple & natural dialog
H1-2: Speak the users’ language
H1-3: Minimize users’ memory load
H1-4: Consistency
H1-5: Feedback
H1-6: Clearly marked exits
H1-7: Shortcuts
H1-8: Precise and constructive error message
H1-9: Prevent errors
H1-10: Help and documentation
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
9
Heuristics (revised set)
searching database for matches
What if we don’t have the status bar?
Why command line UI do not have status bar?
H2-1: Visibility of system status
Keep users informed about what’s going on
Pay attention to response time: 0.1 sec, 10 sec, longer delays
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
10
Heuristics
Bad: Mac desktop
dragging disk to trash
can to eject, should
delete it
H2-2: Match between
system & real world
The system should speak the users’
language, with words, phrases and
concepts familiar to the user, rather
than system-oriented terms.
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
11
Heuristics
• Wizards
– must respond to Q before going to next
– for infrequent tasks
• (e.g., modem config.)
– not for common tasks
– good for beginners
• have 2 versions (WinZip)
• H2-3: User control &
freedom (navigation)
– “exits” for mistaken choices,
undo, redo
– Users often choose system
functions by mistake and will
need a clearly marked
“emergency exit” to leave the
unwanted state without having
to go through an extended
dialogue. Supports undo and
redo and a clear way to
navigate.
Heuristics (H2-3 User control & freedom)
Search is easy to open, enter info, execute or cancel
Heuristics (H2-3 User control & freedom)
clearly marks where the person is and where they can go by showing selection in
each menu
Heuristics
• What is the usability problem?
• H2-4: Consistency & standards
– Users should not have to wonder whether different words, situations, or
actions mean the same thing. Follow platform conventions.
Heuristics (H2-4: Consistency & standards)
Gmail organizes
folders on the same
ones used in other
mail programs
MS Word, Excel, PowerPoint all use the same style
toolbar with the same primary menu options
Heuristics (cont.)
% rm –rf *
%
Bad
Good
H2-5 Error prevention
Even better than good error messages is a careful design, which prevents
a problem from occurring in the first place.
Heuristics
H2-6 Recognition rather than recall (memory)
Minimize the user’s memory load. Make objects, actions, and options visible. The
user should not have to remember information from one part of the dialogue to
another. Instructions for use of the system should be visible or easily retrievable
whenever appropriate.
Heuristics
H2-7 Flexibility and efficiency of use (EFFICIENCY)
Accelerators — unseen by the novice user — may often speed up the
interaction for the expert user such that the system can cater to both
inexperienced and experienced users. Allow users to tailor frequent
actions.
Heuristics
H2-8 Help users recognize, diagnose, & recover from errors
Dialogues should not contain information, which is irrelevant or rarely needed. Every extra unit of
information in a dialogue competes with the relevant units of information and diminishes their relative
visibility. Visual layout should respect the principles of contrast, repetition, alignment, and proximity.
Heuristics
H2-9 Help users recognize, diagnose, and recover from errors
Error messages should be expressed in plain language (no codes), precisely
indicate the problem, and constructively suggest a solution.
Heuristics
H2-10 Help and documentation
Ideally, all systems would run without the need for help but that’s not
realistic. When help and instructions are needed, they should be
accessible, understandable and accurate.
Heuristics
Phases of Heuristic Evaluation
1) Pre-evaluation training
– give evaluators needed domain knowledge and
information on the scenario
2) Evaluation
– individuals evaluate and then aggregate results
3) Severity rating
– determine how severe each problem is (priority)
• can do this first individually and then as a group
4) Debriefing
– discuss the outcome with design team
How to Perform Evaluation
• At least two passes for each evaluator
– first to get feel for flow and scope of system
– second to focus on specific elements
• If system is walk-up-and-use or evaluators are
domain experts, no assistance needed
– otherwise might supply evaluators with scenarios
• Each evaluator produces list of problems
– explain why with reference to heuristic or other
information
– be specific and list each problem separately
Heuristic Violation Examples
1. [H1-3 Minimize the users’ memory load]
Can’t copy info from one window to another
– fix: allow copying
2. [H2-4 Consistency and Standards]
Typography uses different fonts in 3 dialog
boxes
– slows users down
– fix: pick a single format for entire interface
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
26
Severity Ratings
0 - don’t agree that this is a usability problem
1 - cosmetic problem
2 - minor usability problem
3 - major usability problem; important to fix
4 - usability catastrophe; imperative to fix
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
27
Severity Ratings Example
1. [H1-4 Consistency] [Severity 3]
The interface used the string “Save” on the first screen for saving
the user’s settings, but used the string “Store” on the second
screen. Users may be confused by this different terminology for
the same function.
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
28
Review: Design Thinking
Contextual inquiry
Interview
Observation
Task analysis
How-Might-We
Brainstorm
Sketching
Storyboarding Experience Prototype
Video prototype
Concept video
Low-Fi Prototype
Empathize
Ideate
Define
Empathy map
Insights
Point-of-View
October 1, 2015
Prototype
Test
Heuristic evaluation
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
29
EXERCISE
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
30
EXERCISE
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
31
Problems Found
1. H2-4 Consistency
remove column, 4th item is different w/
checkboxes. [150]
2. H2-9 Error prevention
non-numeric data in the quantity. Do not allow.
[125]
3. H2-2 Match between system & real world
vehicle selection link not language I’d expect [100]
4. H2-1 Visibility of System Status
unclear which item to remove based on error
message (“red/bold”). [150]
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
32
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
33
October 28, 2015
dt+UX: Design Thinking for User Experience Design, Prototyping & Evaluation
34