Gathering Usability Data
Observing users & subjective data
Directing Sessions
Issues:
Are you in same room or not?
Single person session or pairs of people
Objective data -- stay detached
Fall 2006
PSYCH / CS 6750
2
Collecting Data
Data gathering
Note-taking
Audio and video tape
Instrumented user interface
Post-experiment questions and interviews
Fall 2006
PSYCH / CS 6750
3
Collecting Data
Identifying errors can be difficult
Qualitative techniques
Think-aloud - can be very helpful
Post-hoc verbal protocol - review video
Critical incident logging - positive & negative
Structured interviews - good questions
“What did you like best/least?”
“How would you change..?”
Fall 2006
PSYCH / CS 6750
4
Observing Users
Not as easy as you think
One of the best ways to gather feedback
about your interface
Watch, listen and learn as a person interacts
with your system
Fall 2006
PSYCH / CS 6750
5
Observation
 Direct
 Indirect
 In same room
 Can be intrusive
 Users aware of your
presence
 Only see it one time
 May use 1-way mirror to
reduce intrusiveness
Fall 2006
PSYCH / CS 6750
 Video recording
 Reduces intrusiveness,
but doesn’t eliminate it
 Cameras focused on
screen, face & keyboard
 Gives archival record,
but can spend a lot of
time reviewing it
6
Location
Observations may be
In lab - Maybe a specially built usability lab
Easier to control
Can have user complete set of tasks
In field
Watch their everyday actions
More realistic
Harder to control other factors
Fall 2006
PSYCH / CS 6750
7
Challenge
In simple observation, you observe actions but
don’t know what’s going on in their head
Often utilize some form of verbal protocol
where users describe their thoughts
Fall 2006
PSYCH / CS 6750
8
Verbal Protocol
One technique: Think-aloud
User describes verbally what s/he is thinking and
doing
What they believe is happening
Why they take an action
What they are trying to do
Fall 2006
PSYCH / CS 6750
9
Think Aloud
Very widely used, useful technique
Allows you to understand user’s thought
processes better
Potential problems:
Can be awkward for participant
Thinking aloud can modify way user performs task
Fall 2006
PSYCH / CS 6750
10
Teams
Another technique: Co-discovery learning
(Constructive interation)
Join pairs of participants to work together
Use think aloud
Perhaps have one person be semi-expert (coach)
and one be novice
More natural (like conversation) so removes some
awkwardness of individual think aloud
Fall 2006
PSYCH / CS 6750
11
Alternative
What if thinking aloud during session will be
too disruptive?
Can use post-event protocol
User performs session, then watches video
afterwards and describes what s/he was thinking
Sometimes difficult to recall
Opens up door of interpretation
Fall 2006
PSYCH / CS 6750
12
Historical Record
In observing users, how do you capture events
in the session for later analysis?
?
Fall 2006
PSYCH / CS 6750
13
Capturing a Session
1. Paper & pencil
Can be slow
May miss things
Is definitely cheap and easy
Task 1
Time 10:00
10:03
10:08
10:22
Fall 2006
Task 2
S
e
PSYCH / CS 6750
Task 3
…
S
e
14
Capturing a Session
2. Recording (audio and/or video)
Good for talk-aloud
Hard to tie to interface
Multiple cameras probably needed
Good, rich record of session
Can be intrusive
Can be painful to transcribe and analyze
Fall 2006
PSYCH / CS 6750
15
Capturing a Session
3. Software logging
Modify software to log user actions
Can give time-stamped key press or mouse event
Two problems:
Too low-level, want higher level events
Massive amount of data, need analysis tools
Fall 2006
PSYCH / CS 6750
16
Issues
What if user gets stuck on a task?
You can ask
“What are you trying to do..?”
“What made you think..?”
“How would you like to perform..?”
“What would make this easier to accomplish..?”
Maybe offer hints
Can provide design ideas
Fall 2006
PSYCH / CS 6750
17
Subjective Data
Satisfaction is an important factor in
performance over time
Learning what people prefer is valuable data
to gather
Fall 2006
PSYCH / CS 6750
18
Methods
Ways of gathering subjective data
Questionnaires
Interviews
Booths (eg, trade show)
Call-in product hot-line
Field support workers
(Focus on first two)
Fall 2006
PSYCH / CS 6750
19
Questionnaires
Preparation is expensive, but administration is
cheap
Oral vs. written
Oral advs: Can ask follow-up questions
Oral disadvs: Costly, time-consuming
Forms can provide better quantitative data
Fall 2006
PSYCH / CS 6750
20
Questionnaires
Issues
Only as good as questions you ask
Establish purpose of questionnaire
Don’t ask things that you will not use
Who is your audience?
How do you deliver and collect questionnaire?
Fall 2006
PSYCH / CS 6750
21
Questionnaire Topic
Can gather demographic data and data about
the interface being studied
Demographic data:
Age, gender
Task expertise
Motivation
Frequency of use
Education/literacy
Fall 2006
PSYCH / CS 6750
22
Interface Data
Can gather data about
screen
graphic design
terminology
capabilities
learning
overall impression
...
Fall 2006
PSYCH / CS 6750
23
Question Format
Closed format
Answer restricted to a set of choices
Typically very quantifiable
Variety of styles
Fall 2006
PSYCH / CS 6750
24
Closed Format
Likert Scale
Typical scale uses 5, 7 or 9 choices
Above that is hard to discern
Doing an odd number gives the neutral choice in
the middle
Characters on screen
hard to read
1
2
Fall 2006
3
4
5
easy to read
6
7
PSYCH / CS 6750
25
Other Styles
Rank from
1 - Very helpful
2 - Ambivalent
3 - Not helpful
0 - Unused
Which word processing
systems do you use?
LaTeX
Word
___ Tutorial
___ On-line help
___ Documentation
FrameMaker
WordPerfect
Fall 2006
PSYCH / CS 6750
26
Closed Format
 Advantages
 Disadvantages
 Clarify alternatives
 Easily quantifiable
 Eliminate useless answer
Fall 2006
PSYCH / CS 6750
 Must cover whole range
 All should be equally
likely
 Don’t get interesting,
“different” reactions
27
Open Format
Asks for unprompted opinions
Good for general, subjective information, but
difficult to analyze rigorously
May help with design ideas
“Can you suggest improvements to this
interface?”
Fall 2006
PSYCH / CS 6750
28
Questionnaire Issues
Question specificity
“Do you have a computer?”
Language
Beware of terminology, jargon
Clarity
Leading questions
Can be phrased either positive or negative
Fall 2006
PSYCH / CS 6750
29
Questionnaire Issues
• Prestige bias
– People answer a certain way because they want
you to think that way about them
• Embarrassing questions
• Hypothetical questions
• “Halo effect”
– When estimate of one feature affects estimate of
another (eg, intelligence/looks)
Fall 2006
PSYCH / CS 6750
30
Deployment
Steps
Discuss questions among team
Administer verbally/written to a few people
(pilot). Verbally query about thoughts on
questions
Administer final test
Fall 2006
PSYCH / CS 6750
31
Interviews
Get user’s viewpoint directly, but certainly a
subjective view
Advantages:
Can vary level of detail as issue arises
Good for more exploratory type questions which
may lead to helpful, constructive suggestions
Fall 2006
PSYCH / CS 6750
32
Interviews
Disadvantages
Subjective view
Interviewer can bias the interview
User may not appropriately characterize usage
Time-consuming
Fall 2006
PSYCH / CS 6750
33
Interview Process
How to
Plan a set of questions (provides for some
consistency)
Don’t ask leading questions
“Did you think the use of an icon there was really
good?”
Can be done in groups
Get consensus, get lively discussion going
Fall 2006
PSYCH / CS 6750
34
Data Analysis
Simple analysis
Determine the means (time, # of errors, etc.) and
compare with goal values (coming up…)
Determine
Why did the problems occur?
What were their causes?
Fall 2006
PSYCH / CS 6750
35
Experimental Results
How does one know if an experiment’s results
mean anything or confirm any beliefs?
Example: 20 people participated, 11 preferred
interface A, 9 preferred interface B
What do you conclude?
Fall 2006
PSYCH / CS 6750
36
Hypothesis Testing
In experiment, we set up a “null hypothesis”
to check
Basically, it says that what occurred was simply
because of chance
For example, any participant has an equal
chance of preferring interface A over interface
B
Fall 2006
PSYCH / CS 6750
37
Hypothesis Testing
If probability result happened by chance is
low, then your results are said to be
“significant”
Statistical measures of significance levels
0.05 often used
Less than 5% possibility it occurred by chance
Fall 2006
PSYCH / CS 6750
38
Presentation Techniques
low
Middle 50%
Age
high
Mean
0
Fall 2006
20
PSYCH / CS 6750
Time in secs.
39
Upcoming
Audio
Web
Fall 2006
PSYCH / CS 6750
40
Using the Results
How do you use the results of your
evaluation?
How can you make your design better with
this knowledge?
Fall 2006
PSYCH / CS 6750
41