i
Some Usability Engineering
Methods
Randolph Bias
Week 7
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
First . . .
• Let’s talk about your projects.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Remember . . .
Our approach, within usability
engineering of web sites and other user
interfaces, is:
• Empirical
• Iterative
• User-Centered Design
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
The Methods of Usability
Engineering . . .
• Are employed in order to enable you to
bring user data (empiricism) to bear on
the emerging product design.
• You (the usability engineer) become an
advocate for the user, in the product
development process.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
One big problem
• Cost
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Three Methods to address cost
• Remote End-user testing (lab testing)
• Heuristic Evaluations
• Usability Walkthroughs
Today let’s talk about WHY and WHEN we
employ one method or another, and
HOW to carry them out.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
End-user Testing
• Also called “lab testing”
• Can be done on paper-and-pencil
design, prototype, early code, existing
product
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
EUT - Benefits
• Gather performance and satisfaction
data
– Performance data: time on task, error rates,
# calls to the help desk, # references to the
documentation, . . . .
– Satisfaction data: End-of-test questionnaire
• Can be find-and-fix or benchmarking
• Ensure coverage of certain parts of the
UI -- good control
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
EUT - Limitations
• Artificial situation
• Successful test doesn’t “prove” the
product works
• Need representative users!
• Ease of learning vs. ease of use
• Hard to test longitudinally
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
EUT -- What to test?
• Can rarely cover all the UI.
• I like to test:
– critical tasks
– frequent tasks
– nettlesome tasks
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Rubin’s 4 Types
• Exploratory (working on the “skeleton”)
• Assessment test (working on the “meat and
flesh” of the UI)
• Validation test (does it meet the objectives?)
• Comparison test (compare two competing
designs)
– Rubin, J. Handbook of Usability Testing: How to
Plan, Design, and Conduct Effective Tests.
Wiley: New York, NY, 1994.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Set up
• Create environment (ambient setting, HW,
SW)
• Identify participants
• Establish test roles (test
monitor/administrator, data logger, timer,
video operator, product experts (SMEs), other
observers)
• Create test plan
• Prepare observers
• Prepare test materials
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
What materials?
•
•
•
•
•
Instructions
NDA
Permission to videotape
Test scenarios
Questionnaire(s)
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Conduct the Test
• Welcome the test participant
• Communicate that this is not a test of THEM,
and they can leave any time
• The scenarios should match the real world
setting
• Ask the test participants to “think aloud” to
better understand intent
• Offer post-test questionnaire, and debrief
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
After the Test
• Quick data to product team
• Assign severities and build
recommendations
• Build archival report
• Serve as change agents!
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Ah, but REMOTE
• Saves tons of travel money
• Allows you to get otherwise hard-to-get test
participants.
• Allows them to be in their own environments.
• Might allow product designers/developers to
watch from their own office.
• But . . . lose some fidelity of the test
environment (video?)
• Some added set-up cost (time)
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
What is a Heuristic Evaluation?
Evaluators systematically inspect the
application interface to check for compliance
with recognized usability guidelines
(heuristics). (Thus, an INSPECTION method.)
- Identifies major and minor usability problems
- Conducted by three to five experienced
usability engineers (or one!)
- Problems are reported along with the
heuristic it violates
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Problems Identified
The probability of one evaluator finding . . .
– A major usability problem - 42% *
– A minor usability problem - 32%
More evaluators = more problems identified
*from www.useit.com
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Strengths of an HE
-Done by people experienced in
usability not just “dumb” users
-Can identify both major and minor
usability problems
-Can be done relatively quickly and
inexpensively
-UNlike EUT, can sometimes cover
every corner of a UI or web site
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Weaknesses of an HE
- If done at end of design, designers may be
resistant to changes
- Some designers/developers may be unmoved
by “just opinions”
- Experienced usability evaluators may miss
content problems that actual users would find
-Can HELP address this issue by using SMEs
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Typical Methodology
-Interface is exercised
-1st pass to develop the big picture
-2nd pass to accomplish typical tasks
-Each problem is reported along with the heuristic
it violates
-Comments are consolidated
-Severity levels – Critical, Major, Moderate, Minor
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Heuristics Used
- Architecture
- Task Design
- Navigation and User Control
- Consistency and Standards
- Information is Meaningful
- Flexibility and Efficiency of Use
- User Support
- Recognition rather than Recall
- Aesthetic and Minimalist Design
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Severity Levels Used
Critical
Usability Issue:
- Loss of user
data
- System
shutdown
- Abandoned
task
Major Usability
Issue –
- Completed task
but considerable
frustration or extra
steps
Moderate Usability
Issue –
- Moderate work
around or multiple
attempts
* Usability Suggestion
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
i
Sample Summary of Results
TYPE
CRITICAL
MAJOR
MINOR
Installation
0
0
3
Project Creation
0
0
5
Visual Layout
0
3
16
Code Editor
0
1
1
Run Project
0
0
1
Debug
0
0
1
Main IDE
0
0
7
Icons
0
0
2
Help
0
0
2
0
4
38
Totals
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
Nielsen’s Usability Heuristics
•
Visibility of system status
–
•
Match between system and the real world
–
•
The system should always keep users informed about what is going on, through appropriate feedback
within reasonable time.
The system should speak the users' language, with words, phrases and concepts familiar to the user,
rather than system-oriented terms. Follow real-world conventions, making information appear in a
natural and logical order.
User control and freedom
–
Users often choose system functions by mistake and will need a clearly marked "emergency exit" to
leave the unwanted state without having to go through an extended dialogue. Support undo and redo.
•
Consistency and standards
–
•
Users should not have to wonder whether different words, situations, or actions mean the same thing.
Follow platform conventions.
Error prevention
–
Even better than good error messages is a careful design which prevents a problem from occurring in
the first place.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Nielsen’s Heuristics (cont’d.)
•
Recognition rather than recall
–
•
Flexibility and efficiency of use
–
•
Dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of
information in a dialogue competes with the relevant units of information and diminishes their relative
visibility.
Help users recognize, diagnose, and recover from errors
–
•
Accelerators -- unseen by the novice user -- may often speed up the interaction for the expert user such
that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent
actions.
Aesthetic and minimalist design
–
•
Make objects, actions, and options visible. The user should not have to remember information from one
part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable
whenever appropriate.
Error messages should be expressed in plain language (no codes), precisely indicate the problem, and
constructively suggest a solution.
Help and documentation
–
Even though it is better if the system can be used without documentation, it may be necessary to
provide help and documentation. Any such information should be easy to search, focused on the user's
task, list concrete steps to be carried out, and not be too large.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
How to . . .
• http://www.useit.com/papers/heuristic/he
uristic_evaluation.html
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Other Heuristics
• http://www.stcsig.org/usability/topics/artic
les/he-checklist.html
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Welcome . . .
• . . . to today’s Usability Walkthrough.
• You are here to evaluate the Zara Tours
web site.
• This is a test of the site, not of you.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Usability Walkthroughs
Purpose
• To collect user data -- from multiple
users at one time -- to help drive the
design of a user interface
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Context
• This is just one step in a series of “UserCentered Design” methods we’re
employing:
– Heuristic evaluation (professional judgment)
– Usability walkthrough (YOU ARE HERE)
– End-user testing in the lab
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Key Characteristics
• Three types of people in the room
• We’re going to go through some task flows, as a
group.
• You’ll have hard-copy packets, where you will write the
actions you would take, if you were online, carrying out
a certain, prescribed task.
• No discussion until all have written a response.
• We’ll announce the “correct” action (according to the
current design).
• We’ll discuss the page -- representative users first.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Please note . . .
Design
is
HARD
SW development teams budget time to debug
the code; we’re debugging the design!
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
This is a test of the DESIGN!
• Not a test of YOU.
• If you have trouble finding the “right”
answers, then WE have a problem with
the UI.
• The site design team is being very bold,
to expose their design to users early like
this -- but they’re doing it because they
realize the benefit, and they want to get
it right!
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
And so . . .
• Given the input we hear today, the site
design team (assisted by a usability
professional) will redesign the interface.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
The Flow of the Day
• We’ll hand you a piece of paper with a
scenario description on it.
• We’ll hand you a packet of screen shots,
in order. DO NOT LOOK AHEAD,
please.
• We’ll ask you to write down on the page
what action you’d take, to accomplish the
task in the scenario.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
The Flow (cont’d.)
• We’ll announce the “right” answer.
• I’ll ask you to indicate if you got it right.
When you did not, I’ll welcome
discussion.
• I’ll welcome the designers and
developers to jump in, as discussion
winds down.
• I’ll let SOME redesign go on, real-time.
• Write more comments under the screen.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Yet more flow . . .
• Then you’ll be asked to turn the page.
• Now, you’ll have to “reset,” and assume
you got to this new page (somehow).
• Then, what action would you take on
THIS page, while still trying to
accomplish the task?
• After a scenario, we’ll give you a
questionnaire to complete.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Benefits
• Lot of data early in the design cycle.
• Usability of individual screens,
terminology, SOME task flow.
• Collaborative redesign on the fly.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Limitations
• Can’t get some data (e.g., time on task).
• You can’t browse, as you might online -tendency to “lose your place.”
• Feel free to turn BACK in your packet,
but not ahead.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Any questions?
• We’ll try to finish up by 2:30 or so.
• Then have some more informal time to
discuss the interface.
• We may get more informal at points.
• THANK YOU very much for being here.
• Absolutely NO ONE should be
embarrassed -- not the users, not the
developers.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
OK, first task:
• Task 1: Find how many days it takes to
make the trek up Mt. Kilimanjaro via the
Machame Route.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Which, Why, When?
Method
HE
WT
EUT
Who?
Design?
When?
Usab. Expert
Complete Late
Hmmm? Full task Mid
Hmmm? ComponentMid
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i
Next Week
• Meet at BMC Software (map to follow,
online).
• Turn in test plan the week after Spring
Break.
R. G. Bias | School of Information | SZB 562B | Phone: 512 471 7046 | rbias@ischool.utexas.edu
i