Rapid Prototype Co-Creation and Testing
Mattias Arvola
@mattiasarvola
Institutionen för datavetenskap
1
DESIGN
BRIEF
2
What is?
DESIGN
CRITERIA
What if?
NAPKIN
PITCH
What wows?
id
LEARNING
GUIDE
Le
ar
ni
ng
ea
ch
-c
r
un
co
la
er
s
tio
t
n
en
pm
tin
g
ot
ot
yp
in
g
pr
lo
te
s
ve
n
de
pt
io
to
m
us
ap
C
R
pt
m
ce
su
on
As
C
g
ng
ys
i
ng
al
pi
an
in
m
to
r
n
ap
pi
ai
m
m
ap
ns
d
ai
Br
in
M
y
ch
ne
ue
ur
Va
l
Jo
Visualization
What works?
T prototypes
• It’s a prototype – Not the Mona Lisa
• Set a deadline
• Prototype only what you need to test
your assumptions
3
Usability evaluations to test your
assumptions about customers
• Analytical thought experiments
• Empirical co-creation or learning
launches
• Different metrics and different
requirements on usability and ease of
use
4
Usability (ISO 9241-11)
• Extent to which a product can be used
by specified users to achieve specified
goals with effectiveness, efficiency and
satisfaction in a specified context of
use.
5
User experience
• "User experience" encompasses all
aspects of the end-user's interaction
with the company, its services, and its
products.
• According to the definition of usability, it
is a quality attribute of the UI, covering
whether the system is easy to learn,
efficient to use, pleasant, and so forth.
Again, this is very important, and again
total UX is an even broader concept.
https://www.nngroup.com/articles/definition-user-experience/
6
UX and Growth
• UX reduces friction
• UX creates natural virality
• UX research helps you understand your
target audience better
• Good UX is the key to great first
impressions
http://blog.usabilla.com/a-growth-hackers-perspective-on-ux/?utm_content=30039037&utm_medium=social&utm_source=linkedin
7
Analytical inspection
methods
Mattias Arvola
@mattiasarvola
Institutionen för datavetenskap
Linköpings universitet
8
Inspection methods
• Heuristic evaluation
• If there is time:
• Cognitive walkthrough
• Keystroke-level model
9
Heuristic evaluation
10
10 heuristics for usability
(Nielsen, 1993;)
1. Visibility of system status
8. Aesthetic and minimalist design
2. Match between system and the
real world
9. Help users recognize, diagnose,
and recover from errors
3. User control and freedom
10.Help and documentation
4. Consistency and standards
5. Error prevention
6. Recognition rather than recall
7. Flexibility and efficiency of use
11
1. Visibility of system
status
The system should always keep users
informed about what is going on,
through appropriate feedback within
reasonable time.
12
2. Match between system
and the real world
The system should speak the users'
language, with words, phrases and
concepts familiar to the user, rather
than system-oriented terms. Follow
real-world conventions, making
information appear in a natural and
logical order.
13
3. User control and
freedom
Users often choose system functions by
mistake and will need a clearly marked
"emergency exit" to leave the unwanted
state without having to go through an
extended dialogue. Support undo and
redo.
14
4. Consistency and
standards
Users should not have to wonder
whether different words, situations, or
actions mean the same thing. Follow
platform conventions.
15
5. Error prevention
Even better than good error messages
is a careful design which prevents a
problem from occurring in the first
place. Either eliminate error-prone
conditions or check for them and
present users with a confirmation option
before they commit to the action.
16
6. Recognition rather than
recall
Minimize the user's memory load by
making objects, actions, and options
visible. The user should not have to
remember information from one part of
the dialogue to another. Instructions for
use of the system should be visible or
easily retrievable whenever appropriate.
17
7. Flexibility and
efficiency of use
Accelerators – unseen by the novice
user – may often speed up the
interaction for the expert user such that
the system can cater to both
inexperienced and experienced users.
Allow users to tailor frequent actions.
18
8. Aesthetic and
minimalist design
Dialogues should not contain
information which is irrelevant or rarely
needed. Every extra unit of information
in a dialogue competes with the
relevant units of information and
diminishes their relative visibility.
19
9. Help users recognize,
diagnose, and recover
Error messages should be expressed in
plain language (no codes), precisely
indicate the problem, and constructively
suggest a solution.
20
10. Help and
documentation
Even though it is better if the system
can be used without documentation, it
may be necessary to provide help and
documentation. Any such information
should be easy to search, focused on
the user's task, list concrete steps to be
carried out, and not be too large.
21
Heuristic Evaluation (Nielsen, 1993)
• Take 3–5 experts and a validated set of
heuristics
• Evaluators familiarize with the heuristics
• Go through the system once, quickly
• Go through the system again, carefully
• Identify problems in relation to the heuristics
• Output: a list of identified problems related
to every heuristic
• Set the output from each evaluator together
22
• You get the best result if the evaluators
are both usability and domain experts
23
How many evaluators?
24
Severity ratings
Frequency: often or seldom
Impact: easy or difficult to recover
Persistency: problem every time or only
the first time of usage
25
Severity on a scale
0 = I don't agree that this is a usability problem
at all
1 = Cosmetic problem only: need not be fixed
unless extra time is available on project
2 = Minor usability problem: fixing this should
be given low priority
3 = Major usability problem: important to fix, so
should be given high priority
4 = Usability catastrophe: imperative to fix this
before product can be released
26
Co-creating and Testing
Paper Prototypes
Mattias Arvola
@mattiasarvola
Department of Computer and Information Science
Start with the test
scenarios
• What are typical use scenarios?
• Build the prototype to support
scenarios you want to test
• Don’t test too many tasks, and not too
few
• Make sure to have realistic tasks and
data
• If possible, balance the order
• Always make a pilot test!
28
29
Paper prototypes
Warfel, 2009
• ~80% of people in UX build paper
prototypes
• Versatile
• Cheap
• Fast
• Easy
• Manipulate on the fly
• Collaborative
• No hardware or software limitations
• Anywhere, anytime
30
Weaknesses of Paper Prototypes
Warfel, 2009
• Distributed work
• Imagination required
• Visual aesthetics
31
Materials
• Plain white paper for base
• Transparancies for rollovers and lightbox
effects
• Index cards for dialogues and widgets
• Post-its for changing states, highlighting
selections, or dialogues
• Colored pens or markers (fine and broad
point)
• Scotch tape, glue stick or blu-tack
• Dental tape for animations
32
Warfel, Todd Zaki. 2009. Prototyping: A Practitioner's Guide. New
York: Rosenfeld Media.http://www.rosenfeldmedia.com/books/
prototyping/
33
Warfel, Todd Zaki. 2009. Prototyping: A
Practitioner's Guide. New York: Rosenfeld
Media.http://www.rosenfeldmedia.com/books/
prototyping/
34
T-prototypes
• It’s a prototype – Not the Mona Lisa
• Set a deadline
• Prototype only what you need
35
36
Prepare test scenarios
Rettig, 1994
• Draw on your user research on typical
situations of use
• Construct the prototype to support a
few scenarios
• Reasonably small set of functions but
enough to allow meaningful tests
• Make sure you use realistic scenarios
and data: test them on someone
• Make a pilot run!
37
A good task scenario
Lewis & Reiman, 1993, 1994
• Change the speed limit on Canyon
Boulevard eastbound between Arapahoe
and 9th.
• Calculate projected traffic flows on
Arapahoe west of 6th assuming Canyon
speeds between 25 and 55 in increments
of 5 mph.
• Specific
• Describes a complete job
• Test how different features of the
interface work together
38
Select your users
Rettig, 1994
• Representative in terms
of for example:
•
•
•
•
Education and training
Computer knowledge
Domain familiarity
Typical tasks
• Try to get the whole range
of users
39
• Surrogate users
• Sometimes hard to get
• Politics involved
• Ethics
• Avoid friends and family
and other clearly biased
people
Test set-up
Rettig, 1994
40
Four roles that you
rotate in the team Rettig, 1994
• Greeter: Make
participants feel at ease
and introduce them. Hand
out test screener
• Facilitator: Gives
instructions, encourages
the user to express
thoughts during the test,
keeps track of time
41
• Computer: Simulates the
computer’s responses to
input, spoken input
simulates keyboard entry,
don’t explain
• Observers: take notes on
index cards. One
observation per card.
Ideas for solutions on the
same card
Three phases
Rettig, 1994
• A session typically lasts about an hour,
depending on the size of the prototype
• Getting ready
• Conducting the test
• Debriefing
42
Getting ready (and some ethics)
• Greetings, introductions,
refreshments, and icebreaking
• Set expectations and
explain:
• paper prototypes
• the equipment
• what you will do with the
data
• think aloud testing
• Assure anonymity
43
• Make sure that they
understand they can quit
at any time
• Clarify that you are
testing the design and not
testing the participants
• Setting up the camera to
record the interaction with
prototype but not people’s
faces
• Ask if there are any
questions
Uncovering the initial impression
and mental model Greenberg et al 2012
• What is this product/system/website for?
• What is your first impression?
• Then briefly introduce the design and ask
people to desribe their understanding, in
detail, of every design element on the
screen
• If their explanations doesn’t match your
intention you have a problem in people’s
initial mental model that you need to
address in a redesign
44
Conducting the test
• The facilitator hands
written tasks to the user.
Tasks should be clear and
focus on what the users
want to acheive but
describe not how.
45
• Facilitator encourages
people to think aloud if
they stops talking:
• What are you thinking right
now?
• What questions are on your
mind?
The observers
• May sometimes interject a question
• Never laugh, gape, say ”a-ha”, nudge
each other, or display any reaction
• Intimidation and humiliation can ruin
the relationship with the participant and
spoil the test
• Don’t ever explain or defend your
design to the users
46
Debriefing
• Discuss interesting behaviours that
you would like the participant to
explain without accusing the user
• Ask your final questions
• Gather impressions and participant’s
thoughts on the design
• Ask if the participant has any questions
• Express thanks
47
Informed consent (more ethics)
• You need to assure that
your participants
understand
• that they agree to
participate in the study
• that it is volontary to
participate
• that they agree to
immediatly raise conserns
or discomforts during the
session
48
• that their interaction is
recorded for research
purposes
• how you will use the results
and the recordings
• that they will be
anonymized
• Consider using a form
• Confirmed consent with
minors include inform
consent from their legal
guardians
Pre-test questions
Usability.gov
•
•
•
•
•
•
•
49
Male/female
Age
Income
Education
Work title
Location of workplace
Knowledge of competing
sites or products
• Computer habits
(platforms, activities,
browsers, websites and
applications often used)
• Language skills
• Domain knowledge
• Technical knowledge
• Contact info
Observation protocol
Usability.gov, Greenberg et al, 2012
• Pathways users take
• Degree of success on the
task scenario
0. not completed
1. completed with difficulty
2. easily completed
• Look for places where
participants have troubles
50
• Look for patterns of
behaviour that tells you if
the product was
understood correctly
• Review it afterwards on
the video
Debriefing questions
Usability.gov
Overall impression?
What did you like best?
What did you like least?
If you got to decide what
would be the first thing to
improve?
• Do you feel that there is
anything missing? Probe
for content and functions
•
•
•
•
51
• If you were to desbribe it
to a collegue in a
sentence or two, what
would you say?
• Final coments or
questions?
Analyzing the results
Rettig, 1994
• Arrange the paper prototype on a big
table
• Pile the notes near the relevant
interface component
• Divide the labour of going through the
piles to summarize and prioritize the
problems
• Go through the piles and agree on
suggested changes
52
Redesign
• Sketch down solutions and affix them
directly on the relevant part of the
paper prototype
• Use that as a basis for creating a
revised prototype, once more in paper,
or perhaps moving on to HiFi computer
prototypes
53
Paper Prototypes in
Summary
•
•
•
•
•
Set up test task scenarios
Prototype enough but not more
Make a pilot test
Introduce participants
Be clear on the roles when conducting
the test
• Debrief participants
• Analyze results and redesign
54
Summative Evaluation
and Getting some
Metrics
Mattias Arvola
@mattiasarvola
Department of Computer and Information Science
How to measure goal fulfilment
when doing user testing
• User tasks focus on important functions
• Desired qualities must be tested for.
• General usability qualities (ease of use,
ease of learning …)
• Specific qualities indicated by personas or
stakeholders
• inspiring
• relaxing
• engaging
• ...
56
Ex: H&M and the Shopping mom
• Effect goals: increase online
shopping by:
• reducing drop-out rate
• attract more customers to
the online shop
• sell more to each online
shopper.
• User Goals:
• Shop for herself and her
family online (problem:
product presentation,
filtering not good)
57
• Be inspired and get tips on
garments that fit together
• Be able to see other’s
ratings of children’s clothes
• Price sensitivity: relatively
high
• Quality importance:
relatively high
• Trendiness importance:
middling
• General usability goals
Metrics and Measurements
• Is it possible to shop (for self/family)?
• Does the subject feel inspired?
• Are combination items noted by the
subject?
• Are ratings by other customers found?
• Does the user feel that bargains can be
found?
• Is the rate of failed buying-tasks lower
than the previous drop-out rate?
• Is the SUS measure greater than 90?
58
Test tasks: ex shopping
• Task focus: finding clothing for self/
family
• Test task: Your son is outgrowing several
old sweaters, you need to order at least
one new. He is 140 cm tall, has several
brown pants that should go with the new
sweater (thus it should be brown or beige).
He likes Spiderman and Cars.
59
Measurements for formative eval
• Measure task completion
• Observe what is happening to find problems
• observe hesitations, suboptimal choices,
muttered questions, confusion...
• Try to understand why the user is
confused...
• Identify problems
• was the intended item found, why not? Did
it take a lot of effort? Was there information
missing? Functions not found?
• (why)
60
Measurements cont’d
• Questions afterwards for qualities:
• Intention to use: How probable is it that you
would buy clothes from this site? Why?
• How inspiring was the presentation of the
clothes? Why?
• To what extent do you feel you’ve made a
good bargain today? Why?
61
Measurements for summative
evaluation and learning launch
• Task completion
• at least 90% of the test
users should complete task
1 (find the spiderman
sweater).
• At least 90% of the users
should complete at least 5
out of 6 buying-tasks.
(compare to current dropout)
62
• We will not measure time to
completion, longer browsing
might induce more sales.
• Experience: Interview or
questionnaire
• How inspired did you feel
by the things you found?
not at all - a little - rather very
• At least 75% of the users
should feel ”very inspired”
Summative cont’d
• SUS - system usability scale. Research
shows that a system that gets at least
90 will be recommended to other users
(thus recruiting more users).
63
SystemUsabilityScale(SUS)
IthinkthatIwouldliketousethissystemfrequently.
Ifoundthesystemunnecessarilycomplex.
Ithoughtthesystemwaseasytouse.
IthinkthatIwouldneedthesupportofatechnical
persontobeabletousethissystem.
IfoundthevariousfuncBonsinthissystemwerewell
integrated.
Ithoughttherewastoomuchinconsistencyinthis
system.
Iwouldimaginethatmostpeoplewouldlearntouse
thissystemveryquickly.
Ifoundthesystemverycumbersometouse.
Ifeltveryconfidentusingthesystem.
IneededtolearnalotofthingsbeforeIcouldget
goingwiththissystem.
64
Strongly
Disagree
Strongly
Agree
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
1
2
3
4
5
Calculating the SUS value
• SUS gives a summary value. The
values of individual questions give no
information.
• Algorithm: Each question contributes
0-4 points. Question 1, 3, 5, 7, 9 are
calculated as mark-1. Question 2, 4, 6,
8, 10 calculated as 5-mark
• Multiply w 2,5 for total value: 0-100
65
Testing your personas
• Remember: they still contain many
assumptions
• When you do customer co-creation and
user testing, you have a customer/user
within reach...
• Compare assumptions to reality: ask
relevant questions of your participants
before and after testing
66
Two more analytical
inspection methods
Mattias Arvola
@mattiasarvola
Institutionen för datavetenskap
Linköpings universitet
67
Cognitive Walkthrough
68
• Focus on novices and ease of learning
• Construct a task: What are the goals of
the users and break it down to subtasks
• Act as the user would and document
problems in each subtask
69
GOAL
INTENTION
SEQUENCE OF
ACTIONS
EXECUTION!
70
4
EXPECTATION
Norman’s
action cycle
EVALUATION
INTERPRETATIO
N
PERCEPTION!
1. Input to the
walkthrough
• Who are the users?
• What are the tasks?
• What sequences of action do the tasks
consist of?
• What does the user interface consist of
and how is it laid out?
71
• Task: I want my calls to be
forwarded to my colleague
who has the phone
number 013 – 28 9998.
• Sequence of actions:
3. Write 490, press green
receiver. Phone: blip blip blip,
”Calls…”, connection
tone
1. Choose the Phone app
Phone: shows list of
latest calls
4. Press *21*99…
Phone: beep, beep, beep
2. Choose Keypad
Phone: Shows keypad
6. Press *21*9998#
Phone: bip-bip-bip-bip
5. (Go back to 3)
7. Press the red receiver
72
• User interface: The phone is an iPhone
with touch screen, on this professors
desk. Instructions for the phone
services are at the university site
• Anställd->Servicetjänster->Telefonväxeln>Telefonguide för MEX-telefon.
• http://www.liu.se/insidan/telefonvaxeln/mexguide?
l=sv
73
2. Gather the evaluators and
3. Go through the sequences of action for every task
• Create a credible story for what the users would do
1. Will the user be able to create the right effect?
2. While the user notice that the right action is
available?
3. Will the user connect the right action with
desirable effect?
4. If the right action is performed, will the user
recognize it as progress to solving the task?
74
4. Write down important
information and
5. Revise the interface
• What the users need to know
• Assumptions about the users
• Notes and design ideas
• A believable successful performence
75
Will the users know what to
do and how?
• Knowledge in the head
• A natural part of the task
• Experience
• Knowledge in the world
• The system tells the user what to do and
how
• Notices affordances
• Everything else looks wrong
76
KLM
77
The Keystroke-Level Model
• Estimate the time it takes to perform
a simple data input for experienced
users
• Focuses on efficiency in
performance of completely
predictable ways of usage
• Sequence of operations
• Operations is assigned a time
78
K, press key or button
Best Typist (135 wpm) --- 0.08 seconds
Good Typist (90 wpm) --- 0.12 seconds
Poor Typist (40 wpm) --- 0.28 seconds
Average Skilled Typist (55 wpm) --- 0.20 seconds
Average Non-secretary Typist (40 wpm) --- 0.28 seconds
Typing Random Letters --- 0.50 seconds
Typing Complex Codes --- 0.75 seconds
Worst Typist (unfamiliar with keyboard) --- 1.20 seconds
P, Point the mouse to an object on screen --- 1.10 seconds
H, Hand from keyboard to mouse or vice versa --- 0.40
seconds
M, Mental Preparation --- 1.35 seconds
R(t), User waiting for the system to respond
D(n0,l0), Draw n0 straight line segment with mouse of total
length l0 centimeters --- 0.9n0 + 0.16l0
79
• Set to boldface shortcut: Press CONTROL K --- 0.60
Type ‘b’ K --- 0.60
Release CONTROL K --- 0.60
Total --- 1.8 secs
• Set to boldface Format pulldown menu:
Point to Format menu with mouse P --1.10
Press and hold down mouse button K
--- 0.60 Move down to Bold P --- 1.10
Release mouse button K 0.60 Total --- 3.40 secs
80
Analytical inspection methods
• Heuristic evaluation for general usability
problems
• Cognitive walkthrough for novice users
• KLM for analysis of efficient
performance
81
www.liu.se