PERVASIVE COMPUTING
USER STUDIES
A.J. Bernheim Brush
Who am I?




Ph.D. in Computer Science
Researcher at Microsoft Research
Technology for families, workgroups
(HCI/CSCW/Ubicomp)
“Love” studies
Why do a user study?

Bad Reasons
 You
don’t have anything else to do…
 You think it’s a requirement to get your paper accepted
 It might be fun to see how people use your stuff

Good Reasons
 You’re
designing stuff for people to use. Wouldn’t it be
nice to know how they might use it?
 There is a new domain or behavior you want to observe
The Reality….

User Studies are a lot of work
 Really,
more work than you ever expected
 No, really I’m not kidding

Understanding your
GOAL is critical
Studying current behavior: What are people doing now?
 Proof of concept: Does my novel technology work for
people?
 Experience using a prototype: How does using my prototype
change people’s behavior or allow them to do new things

WARNING: INTERACTIVE EVENT AT THE END
START THINKING ABOUT ONE OF YOUR OWN PROJECTS
Outline






Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
Goal
Types
Many types of studies
Ethnography
Survey
Interviews
Focus Groups
Logging
Learn about a domain
Design inspiration
“Discount Usability”
Heuristic Evaluations
Lab studies
Iterative Testing
Fix & understand your
prototype
Lab studies
Field Studies
Logging
Does it work?
Can people use it?
How does it compare to
other designs/prototypes?
Goal
Types
Many types of studies
Ethnography
Survey
Interviews
Focus Groups
Logging
Learn about a domain
Design inspiration
“Discount Usability”
Heuristic Evaluations
Lab studies
Iterative Testing
Fix & understand your
prototype
Lab studies
Field Studies
Logging
Does it work?
Can people use it?
How does it compare to
other designs/prototypes?
Surveys




Easy to get large number of people
Design guidance
Evaluation of deployed system
Surprisingly hard to do well….
 Phrasing
of questions
 Biased responses

Pilot your survey!
“Discount Usability” (Jakob Nielsen)
If you are building prototype very useful to get feedback from
users early and often
 Low-cost, Quick, Iterative, Small N, Identify big problems
 Lo-fi prototypes




Heuristic evaluation


Paper version can be very helpful
People feel ok telling you to change stuff
No feedback on responsiveness etc.
Experts review the interface based on list of heuristics
Cognitive walk-through

Determine tasks, review and ask questions for each task
WARNING: Not typically a research contribution
Lab Studies

Bring participants into a lab
 Minimize


variability
Hypothesis testing
Independent variables between
conditions
 Interface
A vs. Interface B
 Control condition?

Measure dependent variable
 Speed
of use, … (Quantitative)
 Preference, …. (Qualitative)
http://www2.sta.uwi.edu/usability/facilities.htm
Between vs. Within Subjects Designs

Between subjects

Participants each in different condition
E.g., everyone randomly assigned to a group
 ANOVA, t-test


Within subjects

I PREFER DUE TO PARTICIPANT VARIABILITY
Each participants experiences all conditions
E.g., expertise, practice or individual differences accounted for
 RM ANOVA, paired t-test


Have to worry about repetition and carryover effects

Counterbalancing, Latin square, etc.
Statistics

Descriptive Statistics (count, mean, …)
Nominal – categories: frequency only
 Ordinal – Ranked preference: frequency, median
 Interval – numbers: frequency, sum, median, mean..



Watch out for outliers
Inferential statistics
Are results statistically significant between groups?
 T-test, ANOVA, paired t-test, etc.
 Significance values


If p < 0.01, there is a 99% chance that the data collected
represents a real difference in the population rather than a
sampling error
Field study


In-situ (“not on your turf”)
Trading “control” for realism
 Think

carefully if this is important
All types
 current
behavior,
 proof-of-concept
 prototype
WARNING: Often good idea to do lab study before field study
Outline






Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
How do I choose?

You might not…
LINC: An Inkable Digital Calendar

What is your goal/research contribution?
Research Question/Goal
Your
Research Question is critical
Bad: How will families use
SPARCS?
Better: Do sharing suggestions
promote sharing?
There are very few right decisions, instead decisions you need to justify
Study Design


What type of study?
What will your participants do during the study?
 Give





them hardware? Give them tasks?
Why type of participants you should recruit?
What data will you collect?
How long will the study be?
Where can you skimp during the study….
What absolutely has to work (if it’s a prototype)
Human Subjects

Ethical treatment of people in the study
Respect—remember that they are doing YOU a favor
 Participants can stop at any time






Consent Forms
Privacy Statements
Compensation
Your organization should have some review process
THIS IS IMPORTANT!
What will they will be doing during the study
 How will you report on what they did

http://www.hhs.gov/ohrp/irb/irb_chapter3.htm
PILOT!!!!


Allow time to a pilot study,
Run through the entire methodology with volunteer
participants.
 Uncover
system problems
 Uncover
experimental design problems
 Uncover
problems with materials
IMPORTANT IMPORTANT IMPORTANT
Participant Profile

What types people do you want to participate?
 All
the same?
Participant Profile

What types people do you want to participate?
 All
the same?
 All different?
Participant Profile

What types people do you want to participate?
 All
the same?
 All different?
 People with extreme characteristics vs. “normal” people
Participant Profile

What types people do you want to participate?
 All
the same?
 All different?
 People with extreme characteristics vs. “normal” people

Consider
 Age
 Gender
 Technology
 ….
experience
How many participants?
 This
can be a difficult question
 Between or within subjects
 What claims you are making
 What is feasible
 Some will drop out!
Length


How long should the study be?
Another difficult question….
 Novelty
 What
are you asking participant to do?
 Long term use vs. feasibility
Data Collection



How will you understand if you answered your
research question?
Quantitative Data (logs, timing, # errors, …)
Qualitative Data (interviews, surveys, …)
TRIANGULATE between
multiple sources

Logging

You must have a plan going in about how you will
use the log data
 Risk
of forgetting to log something important
 Logging too much can create an analysis nightmare

Make a list of questions you expect to answer with
log data
 How
many times did they upload a photo?
 How many days did they use your prototype?
Logging
Question
How/Notes
1. How often was SPARCS running, when did it stop?
# of logfiles, LogfileStart(0)
LogfileClosed
2. Is automatic previewing setup? When
Settings:IsPreviewing
3. Is automatic publishing setup? When
Settings:IsAutoPublishing
4. How many people’s feeds were they subscribed to?
SPARCS_Feed ADD as well
as number of unique Create
View
SPARCS_Feed, add or
remove tags
Calendar_Type,
Settings:CalendarSource tag
5. When did they add or subtract feeds?
6. What type of calendar feed are they using
Qualitative Data

Surveys
Pre-survey
 Post condition
 Post-survey


Experience Sampling Methodology
Small set of question
 Event triggered, random….




Diaries
Interviews (semi-structured, structured)
Observation
Analyzing Qualitative Data



Affinity Diagramming
Coding of Comments
Inter-rater reliability
What if it doesn’t work?

There are many ways a study can fail
 Technical
problems
 They don’t like it


Nobody but you cares about usability problems
Brace yourself for this
 Figure
out what has to work and skimp other places
 Comparison between prototypes
 Pilot studies
Outline






Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
Example
CareNet
Consolvo, Roessler, Selton
Intel Research Seattle
CareNet Methodology





4 “care networks”
(3F, 1M)
13 people (4 elders and 9 care givers)
2-3 people had CareNet displays
3 weeks
Compensation
 With
CareNet - $150
 Elders $75- $300 (depend on data provided)

Sept. – Dec. 2003
CareNet Data Collection






Interviews, before and after (semi-structured,
recorded)
Questionnaires: half-way, end
Photos
3-6 times per day by phone to collect data for
display
Glowing display
People will take your technology apart, even when
you tell them not too.
Lots of other examples…
ButterflyNet
Yeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger, Paepcke
Stanford, Maryland
UbiFit
Consolvo, McDonald, Toscos,
Chen, Froehlich, Harrison, Klasnja,
LaMarca, LeGrand, Libby, Smith, Landay
Intel Research Seattle
SuperBreak
Morris, Brush, Meyers
Microsoft Research
Wearable Jersey Display
Page and Moere
University of Sydney
Outline






Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
#10 Not enough people involved



It takes a village
Huge time commitment, 24 hour support for field
studies
Do
 Include
multiple people
#9 Not being prepared


Don’t want to realize at the end of the study that
you forgot to do something important
Do
 Study
design document
 Research
Question
 Participant Profile
 Methodology (within/between etc)
 Timeline
 Data collection
 Pilot
studies
WARNING: If you do not plan, you plan to fail!
#8 Not enough time for logistics





Everything takes time…. (more than you think)
Recruiting
Installation (e.g. 16 people X 2 hours = )
Support
Do
 Allow
plenty of time for the study
 Have enough people
#7 Seeing what you want to see


We all want our prototypes to be popular
Do:
 Think
carefully about how you discuss the technology
with participants
 Avoid leading questions
 Stay close to the data and find multiple support for
conclusions
 Use neutral language, “Tell me more,” “ummm”
 Don’t get defensive
#6 Being judgmental



Users are always “right”
They may say things that are offensive,
objectionable, etc.
Do:
 Leave
your opinions at home
 Collect feedback
 Use neutral language
#5 Not monitoring usage


Don’t want to find out at the end of the study that
people were not using the prototype
Do:
 “phone
home” messages
 Server logs

Have a plan about when and how you might
intervene
#4 Not collecting a variety of data



Hard to understand logs without interviews
Hard to know if what someone says in an interview
matches their use without logs
Do
 Interviews
 Surveys
 Logs

Support for your findings in multiple ways gives you
(and reviewers) more confidence
#3 Prototype is not robust enough



Technical challenges are a distraction
You are making a “product”
Do
 Usability
studies
 Pilots with friendly folks
#2 Making inappropriate claims



One of the biggest and most common mistakes
Your study is one data point
Do:
 Being
careful with your language
 Include a “limitations” section
 Tell a clear story (don’t have to tell every finding)
#1 Not having a clear research question




Hard to explain the choices you made
Hard to explain your findings
May end up focusing on usability problems
Do
 Have
a clear research question
Your Turn….


Get into groups of 2-3
Choose an example prototype
…


Determine a research question
Come up with a study design
 Participants
 Length
 Data
to collect
 Timeline for study
Your thoughts
Acknowledgments
Ed Cutrell
MSR India
Sunny Consolvo
Intel Research Seattle
Beverly Harrison
Intel Research Seattle
User Studies Are Fun!



Always learn something
Often surprising!
Inspire you
References:




Brush, Ubiquitous Computing Field Studies in Ubiquitous Computing Fundamentals.
Green & Salkind Using SPSS for the Windows and Macintosh: Analyzing and
Understanding Data (nth Ed.)
APA Publication Manual 
HCI textbooks

Preece, J., Rogers, Y., and Sharp, H. Interaction design: beyond human-computer interaction. 2002.
John Wiley & Sons, Inc
A.J. Brush, ajbrush@microsoft.com