Lecture 19
●
Questions about the group project?
●
Next Tuesday (Nov. 18) is a project work day
●
Groups must meet during class time
●
Reminder: Mid-project report (including peer
review forms) is due next Thursday, November
20
November 13, 2014
CS 350 - Computer/Human Interaction
1
Outline
●
Chapter 13 – Rapid evaluation methods
–
Design walkthoughs and reviews
–
UX inspection
–
Heuristic evaluation
–
RITE method
–
In-class exercise
November 13, 2014
CS 350 - Computer/Human Interaction
2
Introduction
November 13, 2014
CS 350 - Computer/Human Interaction
3
Rapid evaluation techniques
●
Aimed almost exclusively at collecting
qualitative data
–
●
●
Finding UX problems to fix
Seldom, if ever, includes quantitative
measurements
Heavy dependency on practical techniques
November 13, 2014
CS 350 - Computer/Human Interaction
4
Rapid evaluation techniques
●
Everything is less formal
–
●
●
Less protocol and fewer rules
Much more variability in process
–
Almost every evaluation “session” different
–
Tailored to prevailing conditions
This flexibility means more spontaneous
ingenuity
–
Something experienced practitioners do best
November 13, 2014
CS 350 - Computer/Human Interaction
5
Rapid evaluation methods
not used in “pure form”
●
Practitioners adapt and combine techniques to
suit their own:
–
Processes
–
Schedules
–
Resources
–
Type of product or system being developed
November 13, 2014
CS 350 - Computer/Human Interaction
6
Design walk-throughs and reviews
●
Early stages of a project
●
Have only
●
–
Your conceptual design
–
Scenarios, storyboards
–
Maybe some screen sketches or wireframes
Not enough for interacting with customers or
users
November 13, 2014
CS 350 - Computer/Human Interaction
7
Design walkthrough
●
Easy and quick evaluation method
●
Can be used at almost any stage
●
●
Especially effective early, before prototype
exists
Audience can include
–
Design team, UX analysts
–
Subject-matter experts, customer representatives
–
Potential users
November 13, 2014
CS 350 - Computer/Human Interaction
8
Design walkthrough
●
●
●
Goal is to explore design on behalf of users
No interaction, so you (evaluators on the design
team) do the driving
Leader tells stories about users and usage,
intentions and actions, and expected outcomes.
November 13, 2014
CS 350 - Computer/Human Interaction
9
Rapid evaluation beyond
early stages
●
Uses interactive prototype
–
●
Including paper prototypes
Most of rapid evaluation techniques are
variations of
–
Inspection techniques
–
Quasi-empirical testing
November 13, 2014
CS 350 - Computer/Human Interaction
10
UX inspection
●
●
●
Especially good for early stages and early
design iterations
Appropriate for existing system that has not
undergone previous evaluation
For when you cannot afford or cannot do labbased testing
November 13, 2014
CS 350 - Computer/Human Interaction
11
UX inspection
●
●
Also called “expert evaluation” or “expert
inspection” or “heuristic evaluation (HE)”
But heuristic evaluation is actually one specific
kind of inspection (Nielsen)
November 13, 2014
CS 350 - Computer/Human Interaction
12
UX inspection
●
Reminder: Cannot “inspect the user
experience”
●
But inspect design for user experience issues
●
An analytical evaluation method
●
The primary rapid evaluation technique
November 13, 2014
CS 350 - Computer/Human Interaction
13
UX inspection
●
●
●
Studies show inspection can be very effective
at finding UX problems
Caution: Not always a complete substitute for
user-based evaluation
There will always be some UX problems that
show up in real live user-based interaction that
you will not see in any inspection
November 13, 2014
CS 350 - Computer/Human Interaction
14
How many inspectors are needed?
●
●
Complex question with lots of theoretical
discussion (see textbook, end of Chapter 14)
Practical answer is “a few” (2 or 3)
November 13, 2014
CS 350 - Computer/Human Interaction
15
What kind of inspectors
are needed?
●
UX experts are good
●
“Dual experts” are even better
–
●
UX expertise plus subject-matter knowledge
Can also get this skill set by pairing up UX
expert with work domain expert
November 13, 2014
CS 350 - Computer/Human Interaction
16
Heuristic evaluation
●
●
●
Is one kind of UX inspection method
A heuristic is a simplified, abstracted design
guideline
Drive inspection with small number (about 10)
of heuristics
November 13, 2014
CS 350 - Computer/Human Interaction
17
Heuristic evaluation
●
Example heuristic: “Visibility of System Status”
The system should always keep users informed
about what is going on through appropriate
feedback within reasonable time.
November 13, 2014
CS 350 - Computer/Human Interaction
18
Heuristic evaluation
●
Another example heuristic: “Match Between
System and The Real World”
The system should speak the users’ language, with
words, phrases, and concepts familiar to the user
rather than system-oriented terms. Follow realworld conventions, making information appear in a
natural and logical order.
●
Full listing of heuristics in textbook, link on
course webpage
November 13, 2014
CS 350 - Computer/Human Interaction
19
Heuristic evaluation
●
●
●
Select small team (usually 2 or 3, could be up
to 5)
Select appropriate set of heuristics
Each inspector browses through each part of
interaction design
–
Asking (to self) the heuristic questions
–
Assessing compliance
–
Noting where heuristics are supported and where
violated, along with context (e.g., screenshots)
November 13, 2014
CS 350 - Computer/Human Interaction
20
Heuristic evaluation
●
Inspectors get together as a team
–
Discuss, compare, and merge problem lists
–
Brainstorm suggested solutions
–
Decide on recommendations
–
Write report (see textbook for details)
November 13, 2014
CS 350 - Computer/Human Interaction
21
Practical approach to UX inspection
●
Definitely for UX experts
●
Driven by experience
–
●
Not heuristics or guidelines
But in-depth knowledge of design guidelines is
essential
November 13, 2014
CS 350 - Computer/Human Interaction
22
Practical UX inspection
●
Driving perspective is usage
●
Focus on
●
–
Tasks
–
Work activities
–
Work context
Helps include an empirical flavor
November 13, 2014
CS 350 - Computer/Human Interaction
23
Practical UX inspection
●
Task-based expert UX inspection
●
Goal is to evaluate design on behalf of users
–
Simulate user’s view of usage
–
But see it with expert’s eye
November 13, 2014
CS 350 - Computer/Human Interaction
24
Use a co-discovery or team
approach in UX inspection
●
●
This is one activity where two heads almost
always better than one
Play off each other
–
Mutual sounding boards
–
Give-and-take interplay
–
Constant flow of think-aloud comments
November 13, 2014
CS 350 - Computer/Human Interaction
25
Co-discovery in UX inspection
●
●
Include customer representative or user for
global view and work flow
Explore design systematically to include
important features, tasks, activities
●
Follow up on clues; be a UX detective
●
Gather large numbers of problem notes
November 13, 2014
CS 350 - Computer/Human Interaction
26
Emotional impact inspection
●
●
●
Look for fun, aesthetics, innovation,
Include packaging and out-of-the-box
experience
Try to envision long-term experience
November 13, 2014
CS 350 - Computer/Human Interaction
27
The RITE UX Evaluation Method
●
Rapid Iterative Testing and Evaluation (Wixon
et al.)
●
A quasi-empirical method
●
A kind of abridged version of user-based testing
●
Fast collaborative test-and-fix cycle
–
Pick low-hanging fruit
–
Relatively low cost
November 13, 2014
CS 350 - Computer/Human Interaction
28
The RITE UX Evaluation Method
●
Whole team participates
●
Key feature is fast turnaround
●
UX problems analyzed right after evaluation
with a few users
November 13, 2014
CS 350 - Computer/Human Interaction
29
The RITE UX Evaluation Method
●
Whole project team decides on changes
●
Changes implemented immediately
●
Maybe use another iteration of testing and
fixing
November 13, 2014
CS 350 - Computer/Human Interaction
30
The RITE method: Preparation
●
Select UX practitioner to be facilitator
–
To direct session
●
Identify characteristics needed in participants
●
Decide tasks for participants to perform
–
Critical tasks, set of tasks every user must be able
to perform
November 13, 2014
CS 350 - Computer/Human Interaction
31
The RITE method: Preparation
●
●
●
Construct test script based on those tasks
Decide how to collect qualitative user behavior
data
Recruit participants (Chapter 14)
–
One to three participants
November 13, 2014
CS 350 - Computer/Human Interaction
32
The RITE method: Data collection
●
●
Gather entire project team and any other
relevant project stakeholders
–
Observation room of UX lab
–
Around table in conference room
Introduce everyone
November 13, 2014
CS 350 - Computer/Human Interaction
33
The RITE method: Data collection
●
●
Set stage
–
Explain process
–
Explain expected outcomes
Conduct evaluation session
–
One user participant at a time
November 13, 2014
CS 350 - Computer/Human Interaction
34
The RITE method: Data collection
●
●
Make sure everyone knows
–
Participant is helping evaluate system
–
Team is not evaluating participant
Have participant perform small number of
selected tasks
–
All project stakeholders observe silently
–
Have participants think aloud as they work
November 13, 2014
CS 350 - Computer/Human Interaction
35
The RITE method: Data collection
●
Work together with participants to find UX
problems
–
Identify critical incidents
–
Discover ways design should be improved
–
Take thorough notes on problem indicators
●
–
Example: task blocking and user errors
Note severity of each problem
November 13, 2014
CS 350 - Computer/Human Interaction
36
The RITE method: Data analysis
●
●
Give list of UX problems and their causes to
everyone on team
Whole development team
–
Identify obvious problems to be fixed first
–
Example: wording or labeling
November 13, 2014
CS 350 - Computer/Human Interaction
37
The RITE method: Data analysis
●
Whole development team determines
–
Other problems that reasonably can be fixed
–
Problems needing more discussion
–
Problems requiring more data (from more
participants)
November 13, 2014
CS 350 - Computer/Human Interaction
38
The RITE method: Data analysis
●
Whole development team
–
Set aside problems you cannot afford to fix right
now
–
Decide on feasible solutions for problems to be
addressed
–
Implement fixes for selected problems
November 13, 2014
CS 350 - Computer/Human Interaction
39
Immediate follow-up
evaluation session
●
Bring in new participants
●
Use modified design
●
●
Have them perform tasks associated with the
fixed problems
Can determine effectiveness of changes
–
Did fix work?
–
Did fix introduction new problems?
–
Did fix uncover any hidden problems?
November 13, 2014
CS 350 - Computer/Human Interaction
40
More about quasi-empirical UX
evaluation methods
●
●
Empirical because you take data using
participants
But get away from formal protocols and
procedures
November 13, 2014
CS 350 - Computer/Human Interaction
41
Quasi-empirical methods
●
No formal predefined “benchmark tasks”
●
For tasks, draw on
–
Usage scenarios
–
Essential use cases, step-by-step task interaction
models
November 13, 2014
CS 350 - Computer/Human Interaction
42
Quasi-empirical methods
●
●
Cut corners as much as possible
No quantitative data collected
–
●
Single paramount mission is to identify UX
problems that can be fixed efficiently
Can occur almost anywhere
–
–
–
–
–
UX lab space
Conference room
Office
Cafeteria
In the field
November 13, 2014
CS 350 - Computer/Human Interaction
43
Quasi-empirical methods
●
Forget controlled conditions
–
Interrupt and intervene at opportune moments
●
●
Elicit thinking aloud
Ask for explanations and specifics
November 13, 2014
CS 350 - Computer/Human Interaction
44
Quasi-empirical methods
●
●
●
Defined by freedom given to practitioners:
–
To innovate, to make it up as they go
–
To be flexible about goals and approaches
–
To make impromptu changes of pace, direction,
focus
Jump on issues as they arise
Milk them to get most information about
problems
November 13, 2014
CS 350 - Computer/Human Interaction
45
In-class Exercise
●
●
Get into the smart fridge design groups
Spend about 10-15 minutes making a list of key
tasks for evaluation (about 4 tasks). Give short
scenarios of what the tasks are to accomplish.
November 13, 2014
CS 350 - Computer/Human Interaction
46
In-class Exercise
●
●
●
Each team will evaluate the other team's design
Each team picks one person to represent the
design to the other team that will evaluate their
design
The design representative becomes part of the
evaluating team for this evaluation exercise
November 13, 2014
CS 350 - Computer/Human Interaction
47
In-class Exercise
●
●
●
The design representative will lead the
evaluation team in a design walkthrough using
the key task scenarios
The design representative is in charge of the
entire design (sketches, prototypes, etc.) for the
evaluating team
Choose your design representative now
November 13, 2014
CS 350 - Computer/Human Interaction
48
In-class Exercise
●
The design representative will inform the
evaluation team via the key taskscenarios
about:
–
The product or system goals
–
Target users
–
Any aspects of the design that need special
attention
November 13, 2014
CS 350 - Computer/Human Interaction
49
In-class Exercise
●
The design representative will serve as a
resource person during the evaluation, to
answers questions about the design and to
help keep the evaluation on track, but will not
influence the evaluation outcome.
November 13, 2014
CS 350 - Computer/Human Interaction
50
In-class Exercise
●
●
●
The evaluators can put themselves in the place
of users to appreciate the usage and design,
and anticipate problems.
Since there probably isn't enough design or
prototype to go on, make it up as you go, so
that this turns out to be a useful exercise.
Success will depend on how creative you can
be. If you really interpret this whole exercise as
a broad design endeavor, there will be a lot to
evaluate.
November 13, 2014
CS 350 - Computer/Human Interaction
51
In-class Exercise
●
●
You can evaluate the ecological aspects with
some walkthroughs emphasizing some key
workflows to get feedback on feasibility and
breakdowns.
You can evaluate the interaction design for
users to make settings via a display panel, etc.
November 13, 2014
CS 350 - Computer/Human Interaction
52
In-class Exercise
●
●
Bottom line
–
Design representative does walk-through
–
Evaluation team observes, ask questions, takes lots
of notes
–
Especially want notes on potential UX problems
Compile the task descriptions (from the design
representative) and the inspection notes (from
everyone else) into a PDF document and email
to the instructor.
November 13, 2014
CS 350 - Computer/Human Interaction
53