Lecture 5:
Evaluation Using
User Studies
Brad Myers
05-863 / 08-763 / 46-863: Introduction to
Human Computer Interaction for
Technology Executives
Fall, 2009, Mini 2
1
Why Evaluate with User Studies?


Following guidelines never sufficient for good
UIs
Heuristic analysis by experts not sufficient


Experts are not the same as users
Experts will generate long list of issues




Which are the important problems?
Experts miss issues
Need both good design
and user studies
(Similar to users with CI)
Good
designers
Average
designers
Quality, before and
after user tests
2
“Don’ts” of User Studies





Don’t test whether it works (quality assurance)
Don’t have experimenters evaluate it – get users
Don’t ask user questions. Not an “opinion survey.”
Instead, watch their behavior.
Don’t test with groups: see how well site works for
each person individually (not a “focus group”)
Don’t train users: want to see if they can figure it out
themselves.
3
Issue: Reliability

Do the results generalize to other people?



If comparing two systems



Individual differences
Up to a factor of 10 in performance
Statistics for confidence intervals, p<.01
But rarely are doing A vs. B studies
Also, small number of users cannot test an
entire site

Just a sample
4
Issue: Validity

Did the study measure what we wanted?


Wrong users
“Confounding” factors, etc,





Issues which were not controlled but not relevant to
study
Other usability problems, setting, etc.
Ordering effects
Learning effects
Too much help given to some users
5
Make a Test Plan

Goals:



Formative – help decide features and design  CIs
Summative – evaluate system  Now
Pilot tests



Preliminary tests to evaluate materials, look for bugs, etc.
Test the instructions, timing
Users do not have to be representative
6
Test Design

“Between subjects” vs. “within subjects”




For comparing different conditions
Within:
 Each user does all conditions
 Removes individual differences
 Add ordering effects
Between
 Each user does one condition
 Quicker for each user
 But need more users due to huge variation in people
Randomized assignment of conditions

To people, or order
7
Performance Measurements


Efficiency, learnability, user’s preference
Time, number of tasks completed, number of errors,
severity of errors, number of times help needed,
quality of results, emotions, etc.


Can instrument software to take measurements


Decide in advance what is relevant
Or try to log results “live” or from videotape
Emotions and preferences from questionnaires and
apparent frustration, happiness with system
8
Questionnaire Design

Collect general demographic information that
may be relevant



Age, sex, computer experience, etc.
Evaluate feelings towards your product and
other products
Important to design questionnaire carefully

Users may find questions confusing


May not answer the question you think you are asking
May not measure what you are interested in
9
Problematic Questionnaire
10
Questionnaire, 2

“Likert scale”

Propose something and let people agree or disagree:
The system was easy to use:

“Semantic differential scale”

Two opposite feelings:
Finding the right information was:

agree
disagree
1 .. 2 .. 3 .. 4 .. 5
difficult
easy
-2 .. -1 .. 0 .. 1 .. 2
If multiple choices, rank order them:
Rank the choices in order of preference (with 1 being most preferred and 4 being least):
Interface #1
Interface #2
Interface #3
Interface #4

(in a real survey, describe the interfaces)
11
Subject ID: ____________1. Please rank order your preference, using 1 for most preferred and 6 for least:
An Experimental Study on various input techniques
Survey example
Questionnaire
Mouse
Tapping on the SmartBoard
Using a Laser Pointer in one hand with the button in the other
Using a Laser Pointer built into a Palm (Symbol) device
Using a stylus on the PocketPC device
Remote control joystick provided with the projector
2. Please rate your proficiency in using computers.
Never used before
0
Please complete the following questionnaire.
questions.
Novice
1
Expert
2
3
4
5
6
7
Please feel free to ask any
3. Please rate your proficiency in using a Palm Pilot or other PDA.
Never used before
0
Novice
1
Expert
2
3
4
5
6
7
1. Please rate each input method.
4. Please rate your proficiency in using a laser pointer in general (not
necessarily tracked by the computer).
1. Mouse
Never used before
terrible
-3
very poor
poor
-2
OK
-1
good
0
very good
1
2
great
3
2. Tapping on the SmartBoard
0
1
-3
very poor
-2
poor
OK
-1
good
0
very good
1
2
great
-3
very poor
-2
poor
OK
-1
good
0
very good
1
2
0
great
-3
-2
poor
OK
-1
good
0
1
3
-3
very poor
-2
poor
-1
6
7
Novice
1
Expert
2
3
4
5
6
7
 No
7. Please supply some information about yourself:
very good
2
 Female
Age:
great
3
8. Please write down any other comments or suggestions for improvement:
__________________________________________________________________________
5. Using a stylus on the PocketPC device
terrible
5
 Yes, I have a:
 Male
very poor
4
6. Do you own a Palm Pilot or other Personal Digital Assistant (PDA)? If so,
which model?
4. Using a Laser Pointer built into a Palm (Symbol) device
terrible
3
3
3. Using a Laser Pointer in one hand with the button in the other
terrible
Expert
2
5. Please rate your proficiency in using a touch sensitive whiteboard like the
SmartBoard.
Never used before
terrible
Novice
OK
good
0
1
6. Remote control joystick provided with the projector
very good
2
great
3
__________________________________________________________________________
__________________________________________________________________________
12
__________________________________________________________________________
- End -
Videotaping

Often useful for measuring after the test


Useful for demonstrating problems to developers,
management


Compelling to see someone struggling
Facilitate Impact analysis



But very slow to analyze and transcribe
Which problems will be most important to fix?
How many users and how much time wasted on each problem
But careful notetaking will often suffice when usability
problems are noticed
13
“Think Aloud” Protocols






“Single most valuable usability engineering method”
Get user to continuously verbalize their thoughts
Find out why user does things
 What thought would happen, why stuck, frustrated, etc.
Encourage users to expand on whatever interesting
But interferes with timings
May need to “coach” user to keep talking
 Unnatural to describe what thinking
 Ask general questions: “What did you expect”, “What are you
thinking now”



Not: “What do you think that button is for”, “Why didn’t you click here”
Will “give away” the answer or bias the user
Alternative: have two test users and encourage discussion
14
Getting Users


Should be representative
If multiple groups of users


Issues:




Representatives of each group, if possible
Managers will pick most able people for testing
Getting users who are specialists
 E.g., doctors, dental assistants
 Maybe can get students, retirees
Paying users
Novices vs. experts

Very different behaviors, performance, etc.
15
Number of test users


About 10 for statistical tests
As few as 5 for evaluation

Can update after each user to correct problems
 But can be misled by “spurious behavior” of a single person


Accidents or just not representative
Five users cannot
test all of a system
16
Number of users, cont.

Jared Spool claims, for large and complete
web sites



Only found 35% of problems after 5 users
Needed about 25 users to get 85% of the
problems
Jared Spool and Will Schroeder, “Testing Web Sites: Five Users
is Nowhere Near Enough,” SIGCHI’2001 Extended Abstracts,
pp. 285-286.
17
Ethical Considerations


No harm to the users
Emotional distress


Emphasize system being tested, not user





Highly trained people especially concerned about looking
foolish
Don’t use terms like “subject”
Results of tests and users’ identities kept confidential
Stop test if user is too upset
At end, ask for comments, explain any deceptions,
thank the participants
At universities, have “Institutional Review Board” (IRB)
18
Milgram Psychology
Experiments

Stanley Milgram 1961-1962



Study created emotional distress


Subject (“teacher” T) told by
experimenter (E) to shock another
person ("Learner" L, an actor) if L gets
answers wrong
> 65% of subjects were willing to give
apparently harmful electric shocks –
up to 450 volts – to a pitifully
protesting victim
Some subjects needed significant
counseling afterward
http://www.stanleymilgram.com/
Image from Wikipedia
19
Prepare for the Test





Set up realistic situation
Write up task scenarios
Write detailed script of what you will say
PRACTICE
Recruit users
20
Who runs the experiment?

Trained usability engineers know how to run a valid
study



But useful for developers & designers to watch




Called “facilitators”
Good methodology is important
 2-3 vs. 5-6 of 8 usability problems found
Available if system crashes or user gets
completely stuck
But have to keep them from interfering
 Randy Pausch’s strategy
Having at least one observer (notetaker)
is useful
Common error: don’t help too early!
21
Where Test?

Usability Labs


Cameras, 2-way mirrors, specialists
Separate observation and control room



Should disclose who is watching
Having one may increase usability testing in an
organization
Can usually perform a test anywhere

Can use portable videotape recorder, etc.
22
Test Tasks and Test Script



(Covered in CI lecture)
Task design is difficult part of usability testing
Representative of “real” tasks






Sufficiently realistic and compelling so users are motivated to
finish
Can let users create their own tasks if relevant
Appropriate coverage of UI under test
Developed based on task analysis, scenarios
Short enough to be finished, but not trivial
Have an explicit script of what will say
23
Stages of a Test

Preparation


Introduction







Make sure test ready to go before user arrives
Say purpose is to test software
Consent form
Give instructions
Pre-test questionnaire
Write down outline to make sure consistent for all users
Running the test
Debriefing after the test

Post-test questionnaire, explain purpose, thanks
24
Introduce the Participants to
the Observation






Introduce yourself
Ask them if they are willing to hear your “pitch” for
participating in a study
Describe the purpose in general terms
Explain the terms of the study and get consent
Give them consent form & get signature
Ask them background questions
25
Conduct the Observation



Introduce the observation phase
Instruct them on how to do a think aloud
Final instructions (“Rules”)


You won’t be able to answer Qs during, but if
questions cross their mind, say them aloud
If you forget to think aloud, I’ll say “Please keep
talking”
26
Cleaning up After a Test

For desktop applications


Remove old files, recent file lists, etc.
Harder for tests of web sites:


In real tests of web sites, need to remove history
to avoid hints to next user
Browser history, “cookies”, etc.
27
Analyze Think-Aloud Data




NOT just a transcription of the tape.
Establish criteria for critical incidents
Record critical incidents and other
observations
(old: UAR Template):
http://www.cs.cmu.edu/~bam/uicourse/UARTemplate.doc

New: Form with rows:
http://www.cs.cmu.edu/~bam/uicourse/UserTestReport_template1.docx
28
Critical Incident Technique in
Human Factors

Definition
Flanagan, (1954), Psychological Bulletin, 51 (4), 327-358.
“By an incident is meant any observable human activity that is
sufficiently complete in itself to permit inferences and predictions to be
made about the person performing the act.
To be critical, an incident must occur in a situation where the purpose
or intent of the act seems fairly clear to the observer and where its
consequences are sufficiently definite to leave little doubt concerning its
effects.” (p. 327)
“Such incidents are defined as extreme behavior, either
outstandingly effective or ineffective with respect to attaining the
general aims of the activity.” (p. 338)

Origin: Aviation Psychology Program during WWII
29
Analyzing the data

Numeric data




Example: times, number of errors, etc.
Tables and plots using a spreadsheet
Look for trends and outliers
Organize problems by scope and severity


Scope: How widespread is the problem?
Severity: How critical is the problem?
30
Scope and Severity Separately
Proportion of users experiencing the
problem
Impact of the
problem on the
users who
experience it
Few
Many
Small
Low Severity
Medium Severity
Large
Medium Severity
High Severity
31
Composite Severity Ratings

Probably easier to use:
(From Nielsen: http://www.useit.com/papers/heuristic/severityrating.html

0 – not a real usability problem
1 – cosmetic problem only–need not be fixed
2 – minor usability problem–low priority
3 – major usability problem–important to fix
4 – usability catastrophe—imperative to fix
before releasing product




32
Write a Summarizing Report



“Executive” summary
Conceptual re-designs are most important
If just “tuning”, then a “top ten” list


Levels of severity help rank the problems
“Highlights” video is often a helpful
communications device
33
What to do with Results


Modify system to fix most important problems
Can modify after each user, if don’t need
statistical results


No need for other users to “suffer”
But remember: user is not a designer
34