Qualitative User Research Methods
Mattias Arvola
2
The process has to begin with some kind of immersion
in context and exposure to people who are perhaps
experiencing that aspect of life right now. It starts by
finding out where we are gonna go, who we are going to
speak to, what is their world and really stepping into
their shoes. For us to start a design project without that
is a bit like cutting of our arms and legs and probably
our head as well, we don't know where to start in that
case.
– Anonymous User Experience Designer
3
4
• Visit your users.
• Speak with them.
• Write down what you have seen and heard.
5
Cost for Change Requests
Time
6
Cost for Change Requests
Impact of UX
Time
7
• Value is created in use by your users.
• Knowing them, their goals and activities is vital for
success.
8
Aims of User Research
• Gain insights on:
• What we should to design/build/develop
• Why?
• How to do it?
9
Research Questions
• Qualitative
• What?
• How?
• Why?
• Quantitative
• How much?
• How long?
• To what degree?
• What affect does X have on Y?
10
Qualitative Research
11
Interpretative research
• Research is interpretative when the understanding of
human sense making is gained through the study of
social constructions (e.g. language, shared meanings,
consciousness, artefacts etc.) as a situation emerges.
12
The Hermeneutical Circle
Whole
Parts
13
Characteristics of Qualitative Research
•
•
•
•
•
•
•
•
No sharp boundaries of study objects
Careful contextualization of the instance
Transferability rather than generalizability
Cooperation between researcher and participants
Purposeful sampling
Alternative interpretations
Suspicion of interpretations
Going beyond what is said and done to what is
intended and motivated
14
Validation Strategies
•
Prolonged engagement and
observation
•
Clarifying what subjective
things that have likely shaped
the the interpretation
•
Building trust
•
Learning the culture
•
past experiences,
•
Checking misinformation
and distortions
•
biases
•
prejudices
Finding the focus
•
orientations
•
•
•
Triangulation
•
Member checking
•
Researcher
•
Rich and thick description
•
Theory
•
•
Method
Revise working hypothesis
Negative cases with
disconfirming evidence
Peer review, debriefing and
external audits
15
Preparations
16
Initial Framing
• What is the project about?
• Finding the target group
• Not the most representative, but the most
important users
• Who’s needs are the most important for us to
succeed?
• Who might have something interesting to say?
• Who are the ones most affected by the project?
• Prioritise
17
Sensitizing Persona hypothesis
Imagine mainstream and extreme users
What do they do?
In what situations?
How do they do it?
Why do they do it?
What characterizes them in terms of motivations,
knowledge and skills.
• What do they lova an hate?
•
•
•
•
•
•
18
How many should we meet?
• Breadth and depth vs. cost
• Three participants from three places/contexts
• As different as possible but within the market
segment
• Recruiting takes time
19
Ethics and Informed Consent
• Knowing that you are observed
• Understanding what it means
• Potential risks
• Gains
• Anonymity or not
• Permission to record
• Provide the participants with the results
• Introduction in writing
20
Gathering Data
21
Setting a Focus
• Getting information and inspiration for design that
gives leverage
• Start with the brief and discuss it with clients
• Find things to handle in the project
22
The Persona and Scenario Sheet
•
•
•
•
•
A checklist with themes to cover
The scenario part: What people do
The persona part: How people are
Who does what, where, when, how, and why (5WH)
Download from:
http://www.ida.liu.se/~matar/tools.en.shtml
Foundation for personas
Goals and driving forces
Skills and knowledge
Experiences
Quote
Portrait
Family and contacts
Likes
Name:
Description:
Dislikes
Age:
Role:
Habits
Sex:
Income:
Hometown:
Background
Project:
Date:
Researcher:
Persona and scenario sheet. Mattias Arvola 2014.
An ordinary day for the persona
An ordinary day starts with…
Project:
Date:
Researcher:
Persona and scenario sheet. Mattias Arvola 2014.
Foundation for scenarios
OBSERVATIONS AND FACTS
WHAT is done?
INTERPRETATIONS
Weaknesses to address
WHY is it done?
(participants' motivation and researchers' interpretation)
Strengths to emphasize
HOW is it done?
Constraints to deal with
WHO does it and who are involved?
WHEN and WHERE is it done?
Project:
Date:
Researcher:
Persona and scenario sheet. Mattias Arvola 2014.
26
What to look for Observations and facts
Opportunities for change
Things to preserve and highlight
Constraints
What characterizes a good/bad situation?
Use a protocol for semi-structured interviews and
observations (the persona and scenario sheet?)
• Phenomena and patterns
•
•
•
•
•
•
27
Field notes
•
•
•
•
•
•
Separate observations and interpretations
Use pseudonyms
Quotes
Sketching with comments and detail studies
Note the background, steps and contexts of activities
Write the notes up as soon as possible
28
Recordings
•
•
•
•
•
•
Video, audio or neither
Quotes
Intrusiveness
Transcription and levels of it
Clips of episodes
No replacement for field notes
29
Interviews
• Open questions that make participants tell stories and
explain
• Tell me about the first time you…
• Tell me about a critical situation when matters
were brought to a head…
• Could you describe when you could not…
• Tell me about the last time you…
• Tell me about when it went really well or really
bad…
30
Keep them talking
• How do you mean when
you say…
• What are you doing
now?
• How do you think about
this situation?
• What are you taking
notice of in this
situation?
• Are there any rules of
thumb for how to
proceed now?
• What information du
you use here, how do
you get it, and what do
you do with it?
• Is there anything that
makes this particularly
difficult?
• Why is that important?
• Have I understood you
correctly if I were to
say…
31
WHY?
• Difficult question.
• Users may start to speculate, but sometimes they
know exactly why.
• Treat the answer with a certain degree of suspicion
and triangulate with observational data.
32
What to avoid
• Questions that participants cannot answer to
themselves
• Closed questions
• Do you like this web site?
• Do you think this web shop is slow?
• Leading questions
• Doesn’t this button look like a search button?
(leading)
• What does this button do? (less leading)
• What do you think about now? (inquiring)
33
Avoid asking users to speculate about the future
• Talk about past events and what they did do. They will
most likely keep along the same lines
• Only ask them to speculate on alternatives in relation
to a specific event
34
Observations
• Level of participation
• Fly on the wall
• Shadowing
• Secret agent
• Apprentice
35
Observation focus
• What order do people do things?
• What cognitive activites (problem solving, memory,
perception, decision making etc.) are involved?
• What info is used and how is it displayed?
• What tools are used and what do they contribute
with?
• Are there any workarounds people have developed to
compensate for constraints in tools and workplace?
• What kinds of errors can be made?
• What other kinds of support could be useful?
36
Contextual Inquiry
•
•
•
•
•
•
Combining interviews and observation
Being an apprentice
In context, in place
Show and tell
Figure out what is important for a participant
Focus points, but no strict protocol: semi-structured
Analysis
38
Affinitity diagramming
•
•
•
•
Categorisation
Finding themes
Sorting cards or sticky notes
Alone, in the design team, or with stakeholders
39
1. Read all notes and
highlight important
passages
5. Discuss what binds the
groups together: the
shared meaning
2. Every participant reads
the highlighted episodes
aloud and writes them
down on sticky notes.
Code the origin
6. Name the groups using
specific notes creating
categories. Categories that
belong together form
themes. There might be
sub-categories too.
3. Get them up on the wall
4. Group them by identifying
similarities or relations. In
silence. Keep moving
notes until nobody wants
to move any more
7. Draw lines between
themes, categories and
sub-categories to create a
hierarchy.
Theme 1
Theme 2
Category 1
Category 2
Category 3
Category 4
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
Idea
41
Reviewing the Affinity Diagram
•
•
•
•
•
What is surprising here?
What did we expect to find that we didn’t find?
What do data tell us that we didn’t know before?
What in the data make us look at things differently?
Why is a certain pattern, or a certain anomaly,
appearing?
42
Modelling the User Research
Think: Concept mapping
Feel: UX Analysis with IPA
Do: Task Analysis
Personas, Scenarios, Storyboards and Customer
Journey Maps
• Setting design objectives
•
•
•
•
43
Concept Mapping
hZp://www.dubberly.com/wp-­‐content/uploads/2008/06/ddo_brand.pdf hZp://cmap.ihmc.us/publica`ons/researchpapers/theorycmaps/theoryunderlyingconceptmaps.htm
46
Method: Concept Mapping
1. Focus question. How is thunder formed?
2. Identify the 15–25 most important concepts in the
domain and place them in ”the parking lot”
3. Write them down on sticky notes and move them
around to form a first map
4. Search for cross-links between concept in different
parts of the map
5. Revise the map, make sure concept-link-concept
create propositions
6. Work on the typography and the layout
47
UX Analysis
48
How do the users feel and experience the various aspects of the situation?
Aesthe9c
Impact!
form and material!
1!
norms!
character and innovation!
Ethics!
power!
experience of activity!
0!
habits!
symbolics!
Communicaton!
-1!
constraints!
cooperation and coordination!
Technical
Build quality!
possibilities!
usefullness!
business!
usability!
Functionality!
operations!
Organization!
Design proposal:__________________________________ !
!
IxDQI—Interaction Design Quality Indicator. Mattias Arvola. 2011-04-15!
!
!
!
!Reviewer:__________________________________ !
50
Method: Interpretative Phenomenological Analysis
• Decide what users (3–5)
to make a more in-depth
analysis of
• What do they mean for
the user and how are they
experienced?
• Read and re-read.
Carefully.
• Why are they important
and why are they
experienced in this way
(how do they talk about it
and what is the context)?
• Make first notes:
• What things are important
to the user (relations,
processes, places, events,
values and principles)?
• What underlying abstract
concepts can be used to
understand the meaning
of their statements?
51
Develop themes
• Form Statements about:
• how things are experienced
• what those experiences mean for the participant
• Order them according to their affinity
• What is similar across participants?
• What is different? Why?
52
Task Analysis
53
A task analysis describes
• Who does what, when, where, how and why
• How often?
• How long?
• How critical?
• What works well?
• What doesn’t work well?
54
Construct a task set in a flow chart
55
Peel the task set into layers
56
Remove redundant dependencies
57
Personas
• Precise descriptions of hypothetical individuals based
on facts about potential users.
• Portrayed like a real living person with a name and a
face to increase empathy
• Provides a shared understanding on whom to design
for
• A tool for communication in the team and with clients
• What would Bob the bank clark do?
• Shaped around goals
• Primary and secondary personas
Goals and tasks Matthew
Johnson
Program Staff
Director, USDA
51 years old Married Father of three children Grandfather of one child Has a Ph.D. in Agricultural Economics. He is focused, goaloriented within a
strong leadership role.
One of his concerns is
maintaining quality
across all output of
programs.
Spends his work time • Requesting and
reviewing research
reports,
• preparing memos
and briefs for agency
heads, and
He is comfortable
using a computer and
refers to himself as an
intermediate Internet
user. He is connected
via a T1 connection at
work and dial-up at
home. He uses email
extensively and uses
the web about 1.5
hours during his work
day.
Quote “Can you get me that
staff analysis by
Tuesday?”
• supervising staff
efforts in food safety
and inspection.
Environment http://www.usability.gov/how-­‐to-­‐and-­‐tools/methods/personas.html 60
Method: Personas
1. Identify roles to create personas for
2. Go through the data from users in the selected role
3. Identify variables on which the users differ and set up
bipolar scales
4. Place the users on the scales
5. Identify patterns by looking for users that go together
on several variables. They will form personas
6. Define goals for the interviewed users whom were
close to each other and put them down in the persona
61
7. Clarify distinctions between personas and add details
8. Repeat the procedure with the other personas if
needed
9. Group and prioritize the personas
10. Develop storytelling and other communication by
giving the persona a face and name. Make it come
alive.
62
Placing on scales and finding patterns
ANVÄNDER
Uses only ENBART EGEN
own computer
DATOR
Uses always ANVÄNDER
OFTA
MASKIN- machine ÖVERSÄTTNING
translation
Reads LÄSER SNABBT
quickly
24
3
1
5
3
1
2
5
ANVÄNDER
Uses only ENBART OFFENTLIG
public computers
DATOR
4
ANVÄNDER
ALDRIG
MASKINmachine
ÖVERSÄTTNING
Uses never translation
5 3
1
4
2
Reads slow LÄSER LÅNGSAMT
OCH METODISKT
and carefully
63
Triading the users
• Go trough data from three randomly selected users at
the time
• Find one characteristic that sets one apart from the
other two and write down as a bipolar scale
• Take a new randomly selected triad and do the same
• Go through ten triads
• Discuss and come to consensus on which variables to
include
64
Scenarios
Mr. and Mrs. Macomb are retired
schoolteachers who are now in their
70s. Their Social Security checks
are an important part of their
income. They've just sold their big
house and moved to a small
apartment. They know that one of
the many chores they need to do
now is tell the Social Security
Administration that they have
moved. They don't know where the
nearest Social Security office is and
it's getting harder for them to do a
lot of walking or driving. If it is easy
and safe enough, they would like to
use the computer to notify the
Social Security Administration of
their move. However, they are
somewhat nervous about doing a
task like this by computer. They
never used computers in their jobs.
However, their son, Steve, gave
them a computer last year, set it up
for them, and showed them how to
use email and go to websites. They
have never been to the Social
Security Administration's website,
so they don't know how it is
organized. Also, they are reluctant
to give out personal information
online, so they want to know how
safe it is to tell the agency about
their new address this way.
http://www.usability.gov/how-toand-tools/methods/scenarios.html
65
Storyboards
68
Customer Journey Mapping
Web
Mobile
r
e
s
u
Phone
Store
e
c
n
e
i
r
e
p
ex
hZp://www.adap`vepath.com/ideas/the-­‐anatomy-­‐of-­‐an-­‐experience-­‐map
http://adaptivepath.org/uploads/images/RailEurope_AdaptivePath_Touchpoints_V1.png
71
Setting Design Objectives
• Effect Goals
• The expected effects of a project
• Why a design work at all need to be made
• What it should contribute with to the business and
to the users
72
Why-­‐why-­‐why
• Why do people object to traffic congestion?: Because
it causes them irritation.
• Why does it cause irritation?: Because it delays one
unexpectedly.
• Why are unexpected delays irritating?: Because they
force one to alter one's plans.
• Why do road travellers have plans?: Because the car
journey is not an end in itself…or is it?
•
http://deseng.ryerson.ca/dokuwiki/design:why_why_why
73
UX Goals – How should it be?
• The UX Analysis gave a list of qualities that should be
emphasized or changed in the users’ experiences. E.g.:
• Respect for the child
• Empathy
• Equality
• Questioning of authorities
• Respect for culture
• Respect for nature
• Curiosity
• Breaking the norms
74
Objectives Tree
Desirable quality 1
Value
Desirable quality 2
Desirable quality 3
Quality criterium 1
Quality criterium 2
Quality criterium 3
Quality criterium 4
Quality criterium 5
Quality criterium 6
75
Business Impact Mapping
• What are the desired effects and values that the
design should contribute to?
• Who are the stakeholders who can create this value
and these effects?
• What are their goals?
• How can the design contribute to them?
Business goals
76
Target groups
Use goals
Scenarios and func9ons
Fritt baserat på en beskrivning av Antrop.
77
Fia, 37 år
VD Mekoväst
Tydliga svar
Vill veta vad som finns att söka för
just henne, i hennes situation
E1, E2, E3
E1, E2, E3
Mia, 43 år
Strateg för
företagarfrågor
(E1), E2, (E3)
Tillvaxtverket.se
Tillvaxtverket.se
(E1), E2, (E3)
Få svar på områdesspecifika frågor
Effektmål 1 (E1)
Vill bli guidad i hur hon ska göra
när hon ska söka bidrag
E1, E2, E3
E1, E2, E3
Få en långsiktig överblick på hur
myndighetens stödverksamhet ser ut
Effektmål
1 tydligt
(E1)
På webbplatsen
framgår
webbplatsen
vilken rollPå
Tillväxtverket
har.framgår tydligt
(E1), E2, (E3)
(E1), E2, (E3)
Webbplatsen
Tillvaxtverket.se
vilken roll Tillväxtverket har.
E1, E2, E3Effektmål 2 (E2)
Tillvaxtverket.se
Nätverka med andra som är aktiva
inom samma område
(E1), E2, (E3)
är
vilka
EffektmålPå1webbplatsen
(E1)
Effektmål
(E2)
Effektmål
1 det
(E1)2tydligt
stöd som
finns
för företagare
och tydligt vilka
På webbplatsen
framgår
tydligt
På
webbplatsen
är det
Effektmål
1 (E1)
På
webbplatsen
framgår
tydligt
regionala tillväxtansvariga.
stöd som På
finns
för företagare
och
vilken roll Tillväxtverket
webbplatsen
vilken rollhar.
myndigheten
har. framgår tydligt
regionala vilken
tillväxtansvariga.
roll Tillväxtverket har.
Effektmål 3 (E3)
EffektmålDet2 ska
(E2)
vara enkelt att göra en
Effektmål
2och
(E2)
3 (E3) 2 (E2)
På webbplatsen
är Effektmål
det
tydligt
vilka
Effektmål
ansökan
om stöd
delta i
Det
vara
enkelt
att göra
På
ärwebbplatsen
det tydligt
stöd som finns
förwebbplatsen
företagare
och
program
hos ska
Tillväxtverket.
På
ären
det tydligt vilka
stöd
och
delta
regionala tillväxtansvariga.
vilkaansökan
stöd somom
finns
för
stöd
som
finns
för iföretagare och
program
hos
Tillväxtverket.
Effektmål
(E4)
företagare4 och
folk på regionala
regionala
tillväxtansvariga.
mål E1-E3 bidrar
EffektmålGenom
3myndigheter.
(E3)
webbplatsen
till att en 4 (E4)
Det ska vara
enkeltEffektmål
att göra
Effektmål 3 (E3)
arbetsbelastningen på
Genom
mål
bidrar
ansökan omhandläggare
stöd och
delta
i E1-E3
ska vara
enkelt att göra en
och 3Det
Effektmål
(E3)
webbplatsen
till att
program hos
Tillväxtverket.
ansökan
om stöd och delta i
programansvariga
minskar.
arbetsbelastningen
på
Det ska
vara enkelt
att
göra
en
program hos Tillväxtverket.
handläggare
och
ansökan
och
delta
i
program
Effektmål 4 (E4)
programansvariga minskar.
hos myndigheten.
Genom mål E1-E3
bidrar Effektmål 4 (E4)
(E1), E2, (E3)
webbplatsen till att
Genom mål E1-E3 bidrar
E2, (E3)
arbetsbelastningen
på
webbplatsen
till
att
Effektmål 4 (E4)
handläggare och
arbetsbelastningen på
(E1), E2, (E3)
Genom mål E1-E3 bidrar
programansvariga minskar. handläggare och
E2, (E3)
webbplatsen
till
att
Calle, 56 år
programansvariga minskar. Martin, 43 år
Göra uppföljningar i det
projekt han leder
© 2013 ANTROP
E2, (E3)
Ta del av nyheter i regionen
© 2013
Få svarANTROP
på projektspecifika
frågor från finansiärer
© 2013 ANTROP
© 2013 ANTROP
Projektledare i
regional
planering
arbetsbelastningen på
handläggare minskar.
Chef på
utvecklings(E1), E2, (E3)
kontor
E2, (E3)
E2, (E3)
(E1), E2, (E3)
Ta del av fakta och statistik
Bevaka vad som händer
24"i regionen
(E1), E2, (E3)
Vara påläst om aktuella
utlysningar och
ansökningsomgångar
24"
24"
24"
78
Building an impact map – Adzic (2012)
•
•
•
•
As a <who?>
I want <what?>
So that I can <how?>
The <what?> can help me do that.
79
Design is a learning process
• Revise objectives
• Specify objectives
• Make them concrete as you learn more
@mattiasarvola
www.liu.se