Rocket Surgery Made Easy:
The Do-It-Yourself Usability
Testing Workshop
Bologna
20 September, 2012
Who is this guy, anyway?
 Steve Krug (steev kroog)
(noun) 1. Son, husband, father
2. Resident of Brookline, MA
3. Usability consultant
 Advanced Common Sense
 Me and a few well-placed mirrors
 Corporate motto: “It’s not rocket surgery™”
 Nice clients
 Lexus.com
 Bloomberg.com
 Technology Review
© 2001 Steve Krug
I get to work at home
© 2001 Steve Krug
I get to work at home
© 2001 Steve Krug
I get to work at home
© 2001 Steve Krug
My intention for today
 Give you some practice so you’re comfortable
testing
 Try to answer all your questions so you have
no reason left not to test
© 2001 Steve Krug
This morning
 Why do usability testing?
 Steve does a demo test
 Six Maxims
 Writing tasks and scenarios
© 2001 Steve Krug
This afternoon
 You do your first practice tests
 You do another practice test
 Assorted topics
 Lingering questions
 Farewell and feedback
© 2001 Steve Krug
Ground rules
 Tell the driver to speak up, if necessary
 Interrupt ANYTIME with questions
 I’ll answer questions about anything
 except that brief period during the late 70’s
© 2001 Steve Krug
Anybody here from out of town?
 Graphic designers
 Information architects
 Developers/programmers
 “Marketing”
 Usability ______
 Project managers
 Writers/editors
 Check signers
 Other?
 Left-handed?
© 2001 Steve Krug
Anybody here from out of town?
 Your experience with usability testing
 Have conducted tests (facilitator)?
 Have observed tests?
 Have read usability test reports?
 Your organization’s use of testing
 Never?
 Right before (or right after) product ships?
 Routine (several times during development)?
 Farmed out?
 Have your own lab (and white coats)?
© 2001 Steve Krug
One more question
 What are the biggest obstacles to your doing
testing?
© 2001 Steve Krug
Who’s read the books?
 DMMT?
 Rocket Surgery?
 Watched the video?
 Don’t worry, be happy, ask questions
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
So…
 Why usability testing?
 Ten years ago, I realized something…
© 2001 Steve Krug
© 2001 Steve
Krug
© 2001
Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
“My ideal home page,” as told by…
© 2001 Steve Krug
“My ideal home page,” as told by…
© 2001 Steve Krug
And now, a live demonstration
© 2001 Steve Krug
A brave volunteer?
 We’ll try an actual test
 It’s painless
 It’s brief
 You’ll get a round of applause when we’re done
 Qualifying criteria:
 Have used a Web browser
 English-speaking adult
 Doesn’t work on __________
During the test
 You are observers
 Jot down top 1 or 2 problems you observed
© 2001 Steve Krug
Debriefing
 What were the most serious problems?
 Observed problems
© 2001 Steve Krug
DIY usability testing (nutshell version)
 Three users
 You’ll find more than you can fix
 No lab or mirrors
 Set up a monitor in another room so the
development team can watch
 Record with Camtasia
 or Morae (Techsmith.com) or CamStudio or various
Mac products
 No stats, no exit questions, no faux validity
 No big honkin’ report
 Debrief over lunch
© 2001 Steve Krug
The maxims
 Six of them
 I could (and have) talk about them all day
 Questions highly encouraged
 What seems like it might not work for you?
© 2001 Steve Krug
A morning a month,
that’s all we ask.
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
When this happens:
You’re not absolutely sure you know what
the user is thinking (see below).
Something happens that seems to surprise
them. For instance, they click on a link and
go “Oh” when the new page appears.
Say this:
“What are you thinking?”
“What are you looking at?” (for variety)
“What are you doing now?” (e.g., if you
think they’re being silent because they’re
reading)
“Is that what you expected to happen?”
They’re trying to get you to give them a
clue. (“Should I use the ___?”)
“What would you do if you were at home?”
The participant makes a comment, and
you’re not sure what triggered it.
The participant suggests concern that he’s
not giving you what you need.
“Was there something in particular that
made you think that?”
“No, this is very helpful.”
The participant asks you to explain how
something is supposed to work. (“Do these
support requests get answered right
away?”)
“I can’t answer that right now, because we
need to know what you would do when you
don’t have somebody around to answer
questions for you. But if you still want to
know when we’re done, I’ll be glad to
answer it then.”
The participant seems to have wandered
away from the task.
“What are you trying to do now?”
“What would you do if I wasn't here?”
“This is exactly what we need.”
© 2001 Steve Krug
Start earlier than
you think makes
sense.
© 2001 Steve Krug
Incorrect thinking
© 2001 Steve Krug
Correct thinking
© 2001 Steve Krug
Recruit loosely and
grade on a curve.
© 2001 Steve Krug
Naturally, we need to
test people who are
just like our target
audience.
Representative
users!
… people who
actually use our
site.
Real
users!
… people who
are a lot like
our users.
© 2001 Steve Krug
© 2001 Steve Krug
Make it a
spectator sport.
© 2001 Steve Krug
© 2001 Steve Krug
Focus ruthlessly on
a small number of
the most important
problems.
© 2001 Steve Krug
What’s funny about this?
 Show of hands: Have you ever gone to a Web
site and run into a serious usability problem?
 Did you find yourself thinking “How can they
not have noticed this? And fixed it?”
 Did you go back months later and it was still
there?
© 2001 Steve Krug
The problem is, testing works too well
 If you’ve done any testing, you know it works
 Uncovers lots of problems quickly
 But I’ve finally realized this is part of the
problem
 You can find more problems in a day than you
can fix in a month
© 2001 Steve Krug
Problems you can
find with just a few
test participants
Problems you
have the
resources to fix
© 2001 Steve Krug
Things I have learned
 It’s easy to get seduced into fixing the easier
problems first
 As a result, the most serious usability problems
often remain for a long time
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
When fixing problems,
always do the
least you can do™.
© 2001 Steve Krug
Your motto
 When fixing usability problems, your motto
should be:
 What’s the smallest change we can make that we
think might solve the observed problem?
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
© 2001 Steve Krug
Nothing New Under the Sun Department
© 2001 Steve Krug
Choosing tasks to test
 What do you have to show?
 Try not to let this limit your thinking, though
 You can get a long way with
 A sketch or a few “comps”
 Linked wireframes to test navigation
 HTML of a few pages that let you complete a task
as long as you don’t stray (“garden path” testing)
 See Carolyn Snyder’s Paper Prototyping
© 2001 Steve Krug
Choosing tasks to test
 What do you wake up thinking about in the
middle of the night?
 What tasks are people likely to do?
 What tasks are crucial?
 …to the user and your business model
 Whenever possible, keep it real
 Free-range browsing tasks are a good thing
 Bad: “Buy a gift cupholder for under $35.”
 Better: “Order a book you’d like to have”
© 2001 Steve Krug
Tasks vs. Scenarios
 Task:
 “Apply for a doctoral program at Harvard Business
School”
 Scenario:
 “You’ve got an MBA, and after a lot of research you’ve
decided to enter the doctoral program at HBS in Science,
Technology & Management.
 Apply for admission to the program.”
 A scenario...
 Provides some context (“You are...”, “You need to...”)
 Just enough; DON’T get carried away
 Supplies specific information the user would actually
have
© 2001 Steve Krug
Writing the scenarios
 Don’t telegraph it
 Avoid using words that will appear on-screen
 Bad: “Customize your LAUNCHcast station.”
 Better: “Choose the kind of music you want to listen
to.”
 Can be the hardest part
 Make it unambiguous
 Misunderstandings waste time and don’t [usually]
produce useful insights
 Keep it short
 Trim any detail that doesn’t contribute
 Piloting the test will help
© 2001 Steve Krug
Exercise: Preparing tasks
 Jot down 3-7 of the most important tasks
people need to do on your site (4 min.)
 E.g., “Find directions to your nearest Bank of
America branch”, “Apply for a doctoral program”
 Choose one to work on
 Write a scenario for this task (10 min.)
 A short paragraph
 “Don’t use Search” is OK
 Write it out so you can read it verbatim
© 2001 Steve Krug
Now send me your scenario(s)
 Go to
tinyurl.com/krug-bologna
www.quicktopic.com/48/H/3MtSRYmk3ip
 Click the orange “Post a new message” button
 Use your name to sign in (so we can identify
them)
 Type the URL you’re testing
 Type your scenario
© 2001 Steve Krug
Exercise: Practice test
 Pair up with someone who has a laptop
 Take turns as facilitator/participant
 12 minutes each
 Start Camtasia if you have it
 Read the script
 Have them look at Home page (<2 min.)
 Give them the task
 Keep them thinking aloud
 Thank them
 Save your Camtasia file
 I can help if you have questions
© 2001 Steve Krug
How’d it go?
© 2001 Steve Krug
The debriefing
 Over lunch (or dinner, or breakfast)
 Right after the three test sessions
 Objective: Deciding what you’re going to
commit to fixing before the next round of
testing
© 2001 Steve Krug
The debriefing
 Go around the room
 Everyone contributes from their list of nine
problems
 Write on easel pad
 Leave some space for improvements/amendments
 People can say “Me too!”
 Treat all contributions with respect
 Not discussing yet
 Stick to observed problems!
© 2001 Steve Krug
The debriefing
 Decide which are most serious
 Some magic happens here
 Voting/Dictatorship
 Not usually as hard as it seems BECAUSE THEY ALL
SAW THE SAME BEHAVIOR
 Number them
 Copy the numbered list
 Ten is probably enough
 Leave space in between
© 2001 Steve Krug
The debriefing
 Start at the top
 Work down the list
 Come up with rough idea of how you’ll fix them
 who will do it
 the resources required
 When you’ve allocated the resources you can
commit in next month, STOP!
 Tear off the rest of the list
 Crumple it up
 Throw it away
Thanks to Susan Weinschenck
© 2001 Steve Krug
© 2001 Steve Krug
A great tip
© 2001 Steve Krug
And the companion volume…
© 2001 Steve Krug
© 2001 Steve Krug
Thanks for all the fish
 Send any questions, feedback, gripes to
 skrug@sensible.com
© 2001 Steve Krug
© 2012 Steve Krug