Preparing and Running
User Experiments
By Mei Li, Pearl Ho, and Deepika Gandhi
How to Prepare and Run Usability
Testing

How to gather participants and prepare a
proposal.
--Mei

Preparing the environment, test materials, and
test team.
--Pearl

How to measure usability and deal with
participants.
--Deepika
Preparing for your User
Requirements Activity
Introduction
We cover everything that happens or that you
should be aware of prior to collect your data.
Preparing for your User
Requirements Activity






Creating a proposal
Deciding the duration and timing of your session
Recruiting participants
Tracking participants
Creating a protocol
Piloting your activity
Creating a proposal


A usability activity proposal is a road map for the
activity you are about to undertake.
Why create a proposal?
 Surprises
 Assumptions
 Misconceptions
Tip:
Multiple activities, separate proposals
Creating a proposal




History
Objectives, measures,
scope of the study
Method
User profile




Recruitment
Incentives
Responsibilities
Proposed schedule
Creating a proposal
Recruitment
E.g. John Smith from Product team will recruit total 30
participants by an advertisement on the web and a recruitment
agency.
Creating a proposal
Incentives
E.g. Participants will receive $75 in AMEX gift checks for the
participation.
Creating a proposal
Incentives
E.g. Participants will receive $75,000 in AMEX gift checks for the
participation.
Creating a proposal
Responsibilities


Who is responsible for what task?
Key word – Specific
The product team is responsible for recruiting participants.
John Brown from the product team is responsible for recruiting
participants.
Creating a proposal
Proposed Schedule



Indicate time for each deliverable
Key word – Specific
People often underestimate the amount of time it
takes to prepare for an activity.
Tip:
Request deliverables a week before we absolutely need them.
Then, if deliverables are late (which they often are), it's OK.
Creating a proposal
Getting Commitment


Email the proposal?
Issues:
stakeholders will criticize:
 Skills/knowledge/objectivity of the person who
conducted the activity
– Be a member of the team and earn their respect
 Participants in the activity
 Tasks/activity conducted
– Getting everyone sign off on the proposal
Creating a proposal
Organize a meeting:





Objective of the activity
Data which will be collected
User profile
Responsibility
Schedule
Signed
Preparing for your User
Requirements Activity






Creating a proposal
Deciding the duration and timing of your session
Recruiting participants
Tracking participants
Creating a protocol
Piloting your activity
Deciding the duration and
timing of your session
Group Session



5-7pm or 6-8pm best
With some food, perfect
Break the session into small chunks if
possible
Preparing for your User
Requirements Activity






Creating a proposal
Deciding the duration and timing of your session
Recruiting participants
Tracking participants
Creating a protocol
Piloting your activity
Recruiting participants







Participants number
Participant incentives
Developing a recruiting screener
Creating a recruitment advertisement
Recruitment methods
Preventing no-shows
Recruiting international participants Recruit
special populations
Recruiting participants
Participants number

Facts to be concerned


Availability
Representative

Method – Convenience sampling

Use the available sample of the population, instead of
representatives from the population at large.
 e.g. Research done by college professors often uses college
students for participants.
Tip – Identify participants types

Where should you go to Find
Participants
For this type of Product
Look to this source
Travel reservation system
Travel agencies
Bank teller system
Banks
New version of existing operating
system
Customer lists
Medical office software
Phone lists of local doctors'
offices
Recruiting participants
Participant incentives

Mode






Cash, normally not. Store?
One of your product for free
Gift certificate (an electronics store, movie pass)
Gift checks
Charitable donations in the participant's name (for highly
paid individuals CEOs)
Amount


Make the incentive large enough to thank people for their
time and expertise, but nothing more.
Pay everyone in the same session the same amount in case
of amount changing.
Recruiting participants
Developing a recruiting screener







Via phone, don't email
Work with the product team, instill team-work sense.
Keep it short
Use test questions, you want honest participants
Collect demographic information, once decide a candidate
Eliminate competitors
Provide important details


Time, date, location, compensation, rules, signature
Prepare a response for people who do not match the profile
Recruiting participants
Creating a recruitment advertisement





Provide some details about your study
Include date, time, location of the study
Indicate key characteristics, not all
Don't stress the incentive
State how they should respond


e.g. generic email address
Be aware of types of bias
Recruiting participants
Recruitment Methods




Advertise on community bulletin board sites
Create an in-house database
Use a recruiting agency
Make use of customer contacts
Recruiting participants
Preventing no-shows




A 10% "no show " rate is common
Provide contact information
Remind
Over-recruit
Recruiting participants
Confirm the Appointment:








Thanks for agreeing to participate
The date and time you expect them to be there
Where to come, including a map and directions
How long to expect to be with you
The purpose of the test
Reminders about the video cameras
Incentive
A person's name and a phone number to call if they
have questions or need to reschedule
Recruiting participants
Recruiting International Participants





Agency
Cultural and behavioral taboos books
Translator
Punctuality
Holiday concern
Recruiting participants
Recruiting Special Populations

Children, elderly, disabilities



Transportation
Escorts
Facilities
Recruiting participants
Recruitment Methods




Advertise on community bulletin board sites
Create an in-house database
Use a recruiting agency
Make use of customer contacts
Preparing for your User
Requirements Activity






Creating a proposal
Deciding the duration and timing of your session
Recruiting participants
Tracking participants
Creating a protocol
Piloting your activity
Tracking Participants



Tax implications
Avoid professional participant
Create a watch list




Tax
Dishonesty
Poor attendance
No show-up
Creating a Protocol


All procedures and their order
Act as a checklist for all of the session steps
Piloting your Activity




Check audio-visual equipment working
Clarity of instructions and questions
Find bug or glitches
Attendee


Experienced
Product team member
Preparing the environment, test materials,
and test team.
--Pearl Ho