1
© IDC, IIT Bombay
BadDesigns.com
2
© IDC, IIT Bombay
3
© IDC, IIT Bombay
4
© IDC, IIT Bombay
5
© IDC, IIT Bombay
6
© IDC, IIT Bombay
7
© IDC, IIT Bombay
8
© IDC, IIT Bombay
9
© IDC, IIT Bombay
10
© IDC, IIT Bombay
11
© IDC, IIT Bombay
12
© IDC, IIT Bombay
13
© IDC, IIT Bombay
14
© IDC, IIT Bombay
15
© IDC, IIT Bombay
16
© IDC, IIT Bombay
17
© IDC, IIT Bombay
18
© IDC, IIT Bombay
World Is Full of Unusable Objects
 Home assignment
– Find 3 unique usability problems in the day-to-day objects that you have
observed or encountered
– Describe them as 3 slides in a presentation
19
Usability Evaluation
Objective
 To identify usability problems in an interface
– To improve the design – formative evaluation
– To compare competitive products or as acceptance test – summative
evaluation
21
© IDC, IIT Bombay
Many Methods
 Heuristic evaluation
 User Testing
 Thinking Aloud Protocol
 Card sort
 Feedback from real life usage
– Observational field studies
– Logging actual use
– User Feedback
– Questionnaires and Interviews
– Focus groups
22
Usability Evaluation Methods
 Heuristic evaluation
– Jakob Nielsen, Usability Engineering, Morgan Kaufman, 1990
23
Task
 For use by a shopkeeper in a kirana
shop to make a bill
– Find as many usability problems as you
can in 5 minutes in the Standard
Windows calculator
 For each problem
– State it succinctly (if you thought of a
design idea, tell us what problem do you
solve)
– Identify a (potential) user experience
goal that was not met
– Rate it for severity
– Identify layer of user experience
24
© IDC, IIT Bombay
Task
 Analyse the alarm app shown in this video for use by a patient
educated till class VII as reminder tool for his medication
– Find as many usability problems as you can in 5 minutes
25
© IDC, IIT Bombay
Analyse the app shown in this video
http://www.youtube.com/watch?v=s94JxnVwJtI
26
Task
 Analyse the alarm app shown in this video for use by a patient
educated till class VII as reminder tool for his medication
– Find as many usability problems as you can in 5 minutes
 For each problem
– State it succinctly (if you thought of a design idea, tell us what
problem do you solve)
– Identify a (potential) user experience goal that was not met
– Rate it for severity
– Identify layer of user experience where the problem occurs
27
© IDC, IIT Bombay
Heuristic Evaluation Technique
 5-10 evaluators examine the product and identify usability
problems
 It helps to have a mix of usability experts, domain experts and
users
– Single / double experts
– Same evaluators provide consistency across versions and save time
– New evaluators provide fresh eyes
28
© IDC, IIT Bombay
Heuristic Evaluation Technique
 Give assistance regarding the domain knowledge
– Brief about users, personas, scenarios, primary goals, and usability
goals
– Evaluators may be asked to focus on parts of the product / specific
personas
 Give a checklist of
– HCI heuristics and principles
– Platform standards, common practices
– Domain standards, common practices
29
© IDC, IIT Bombay
Heuristic Evaluation Technique …
30
 Each evaluator has the freedom to choose his/her method of
evaluation
– Each evaluator evaluates the product independently
– Evaluators should go through product 2-3 times
 First iteration
– Look for problems novice might face
– Analyze Surface and Skeleton issues
 Second iteration
– Use primary goals, usability goals, heuristics, domain standards,
platform standards to identify problems
– Look for conceptual problems, analyze Structure
 Third iteration
– Look for what is being missed out
– Question Scope and Strategy issues
© IDC, IIT Bombay
Heuristic Evaluation Technique …
 Sometimes, an observer notes the findings
– Each of us will note for ourselves here
 Observer may give hints about product use
– These are evaluators, not users
– Save time, capture the problem, and move on
– Help only when evaluators ask, after noting problems
31
© IDC, IIT Bombay
Heuristic Evaluation Technique …
 After the evaluations, have a debriefing meeting to compile
findings
 The debriefing meeting also produces design suggestions
– Make them sound like suggestions rather than decisions
– Do not express only a design idea, point to the problem first
 Typical debriefing sessions last 1-4 hours
– Produce preliminary report during the debriefing meeting
– Fine-tune later
32
© IDC, IIT Bombay
Heuristic Evaluation Output
 Output of heuristic evaluation is a list of usability problems
 Report formats
– Introduction, product evaluated, method used
– Scenario based problems
~ Enter the cubicle, swipe the card, enter PIN, check balance,
withdraw cash, collect cash, card and receipt, exit the cubicle
– Screen based problems in callouts
~ Idle screen, test keyboard screen, login screen, password screen,
main menu
 Common Industry Format (CIF)
~ ISO/IEC 25062:2006
33
© IDC, IIT Bombay
Heuristic Evaluation Some Numbers
 5 evaluators find about 75% of the problems
 10 evaluators find about 90% of the problems
– HE is prone to reporting false alarms
– Don’t negotiate for too long unless it is a very important issue
– If someone thinks it is a problem, capture it
34
© IDC, IIT Bombay
Usability Evaluation Methods
 Heuristic evaluation
 User Testing
35
User Testing
 A representative set of tasks
 Representative users try to perform these tasks
 An experimenter conducts the experiment
– Observes users
– Captures, interprets and prioritizes problems
36
© IDC, IIT Bombay
Planning a User Test
 Determine the key question that the UT should answer
~ Should we design an IVR? Can low-tech savvy users use IVRs?
~ Is the design OK? Should we start the development?
~ Can all voters use the new EVM? Can we use it in the next election?
– Subset of usability goals and persona goals
37
© IDC, IIT Bombay
Planning a User Test
 Determine the key question that the UT should answer
 Determine test tasks
– Determine training to be given prior to tasks
– Provide a scenario description
– Give users a goal to achieve, not instructions to execute
– Avoid leading instructions / hints / questions
– Set ‘error traps’ and planned disturbances
– Try to provide early success experiences
38
© IDC, IIT Bombay
Planning a User Test
 Determine the key question that the UT should answer
 Determine test tasks
>> Output of a user test
 Determine evaluation criteria
– Determine the output of the UT
– Identify performance measures
~ Quantitative
~ Success, time, errors, efficiency, satisfaction
~ Qualitative
~ Ability to handle new situations
~ Conceptual understandability of the product
~ Questions asked by users
~ Training procedure
39
© IDC, IIT Bombay
User Testing Output
 Qualitative output
~ What is the first reaction?
~ What is confusing? What do users ask? Why?
~ What do the users like / hate most?
~ What features are users likely to use?
~ What are user priorities?
– Good for formative evaluation
40
© IDC, IIT Bombay
User Testing Output
 Qualitative output
 Task success
~ Find the share price of Reliance
~ Find a good insurance policy
– Levels of success
~ Yes / no
~ Successful in first attempt / second attempt / with help
41
© IDC, IIT Bombay
User Testing Output
 Qualitative output
 Task success
 Task time
– Tell the participant to be as fast as possible
– But avoid declaring
~ “You took 6.3 seconds for that task. Great show.”
– Decide when to stop the timer
– To probe or not to probe?
– Include unsuccessful tasks in timing?
42
© IDC, IIT Bombay
User Testing Output




Qualitative output
Task success
Task time
Errors
– Error “an action of the user that does not accomplish the desired
goal”
– Severity of errors “the more difficult it is to recover from the error,
more sever it is”
Few users face the
Many users face the
problem / problem occurs problem / problem occurs
infrequently
frequently
Easy to recover
Difficult to recover
Low
Medium
Medium
High
43
© IDC, IIT Bombay
User Testing Output





Qualitative output
Task success
Task time
Errors
Efficiency
– Efficiency = success + time
– Number of mouse clicks
– “Lostness” – Smith
44
© IDC, IIT Bombay
User Testing Output






45
Qualitative output
Task success
Task time
Errors
Efficiency
Satisfaction scores
– What to measure?
~ Expectation, ease, satisfaction, time (no watch), smiles / frowns
– How to measure?
~ Likert scales, semantic differential scales
– When to measure?
~ Pre-test, post-test, in the field
– Try to make it look anonymous – online reduces bias
© IDC, IIT Bombay
Planning a User Test
 Determine the key question that the UT should answer
 Determine test tasks
 Determine evaluation criteria
– This will be the output of the UT
~ Qualitative
~ Quantitative: success, time errors, efficiency, satisfaction
– Correctness criteria for each task
– Things to measure
– Things to watch out for
– When to help users
46
© IDC, IIT Bombay
Planning a User Test




Determine the key question that the UT should answer
Determine test tasks
Determine evaluation criteria
Invite / recruit appropriate users
– Matching personas, target markets
– Covering independent variables
– Professional recruiters
– Screener
– Telephonic screening
47
© IDC, IIT Bombay
User Testing Screener
48
 Make it short
– Less than 20 questions that take about 5 minutes to run through.
 Be clear and specific
– Avoid questions that can lead to ambiguous answers
 Avoid unnecessary questions
– A screener is not an interview
– Each question should eliminate some users
 Order questions from general to specific
– Try to eliminate more people sooner
 Communicate the format of the UT and the expected time
commitments
– Clearly communicate if the user needs to be prepared in some way
 Plan on drop outs and no-shows
– Recruit 20% more users than you need
© IDC, IIT Bombay
Planning a User Test





Determine the key question that the UT should answer
Determine test tasks
Determine evaluation criteria
Invite / recruit appropriate users
Prepare the infrastructure
– Carry out all test tasks to ensure readiness
– Carry out pilot tests for practice
49
© IDC, IIT Bombay
User Testing Process
 Introduce the user to the topic and put him/her at ease
–
–
–
–
Explicitly say that it is the product that is being tested
Explain that the product is new and untested
Promise confidentiality, inform about any recordings
Avoid large crowds, conspicuous cameras, interruptions other than the
planned ones
 Give test tasks one at time
 Give hints to only if the user is stuck for a long time
– Or if they run into a known usability problem
 After the tests, debrief the user clarify any doubts
 Review test plan before repeating with another user
50
© IDC, IIT Bombay
Interview Structure
51
 Start
– Break ice – factual questions
– Ask 1-2 open ended questions
 Middle
– Give tasks, one at a time
– Contextualize findings – concrete, detailed data
– Keep track of test goals
 End
– Debrief, answer questions, discuss insights
– Ask for suggestions
 Let the user talk
– Avoid interruptions
– Try to build a partnership
© IDC, IIT Bombay
Questions for Non-directed Interviewing
 Don’t imply the answer
~ “We put in a lot of effort to design this product. Do you like it?”
52
© IDC, IIT Bombay
Questions for Non-directed Interviewing
 Don’t imply the answer
 Questions should be non-judgmental
~ “Don’t you think that calorie advice will be more useful as a
graph?”
~ “What do you learn from this graph?”
or
“With this tool select an appropriate dessert for the family.” then
observe and probe as appropriate
53
© IDC, IIT Bombay
Questions for Non-directed Interviewing
 Don’t imply the answer
 Questions should be non-judgmental
 Keep questions “open-ended”
~ “Which of these features is most important to you?”
implies “At least some features should be important to you.”
~ “Rate the importance of these features on a scale 1-5”
54
© IDC, IIT Bombay
Questions for Non-directed Interviewing




Don’t imply the answer
Questions should be non-judgmental
Keep questions open-ended
Focus on experience, not extrapolation
~ “Do you think car pooling is a useful feature?”
Implies “In the universe of all things, do you think that someone
somewhere will find carpooling useful?”
~ “Given your lifestyle and the amount of commuting you do, do you
think carpooling valuable to you right now?”
55
© IDC, IIT Bombay
Questions for Non-directed Interviewing





Don’t imply the answer
Questions should be non-judgmental
Keep questions open-ended
Focus on experience, not extrapolation
Further, focus on immediate experience
~ “Is a in-car GPS interesting to you?”
Implies “Will a in-car GPS facility be of interest to you at some
time in your life?”
~ “If your car is equipped with this GPS facility, would you use it?”
Then, if yes “How will you use it?”
56
© IDC, IIT Bombay
Questions for Non-directed Interviewing






Don’t imply the answer
Questions should be non-judgmental
Keep questions open-ended
Focus on experience, not extrapolation
Further, focus on immediate experience
Focus a question on a single topic
~ “How will you use the MP3 option at a party and while driving?”
57
© IDC, IIT Bombay
Running a Non-directed Interview
 Define ambiguous terms
 Restate answers to verify (= share interpretation)
– Listen to user’s questions carefully
 Use artifacts to ground discussions
 Don’t force opinions
– Never say that the user is wrong – you are here to find out what the
user thinks, not to convince him / her about your opinions
 Follow up with examples to probe
– But wait for an undirected answer first
 Review interview tapes
– You will find your own ways of improvement
58
© IDC, IIT Bombay
What can go wrong?
 Reliability
– “Can we get the result if we repeat the test”
– 10x difference in individual computer usage speeds
– Background of users
~ Access, age, familiarity with UTs
59
© IDC, IIT Bombay
What can go wrong?
 Reliability
 Validity
– “Does the test reflect the usability issues of interest”
– Choice of users
~ Are they a representative sample of real users?
– Choice of tasks
~ Are these tasks important? To whom?
– Conclusions and their relation with the experiment
~ User took time in first attempt, but this is a frequent task
~ We tested a prototype, then built the whole system, but the prototype
was too small, and the system is far too complex
~ We tested the new widget, but the real problem was finding it
~ Probing confusions and evaluating task time at the same time
60
© IDC, IIT Bombay
What can go wrong?
 Reliability
 Validity
 Experimental design (mostly more validity issues)
– Testing too many things
~ The UT took 4 hours, but the user was exhausted
~ You cannot text everything, you have to rely on the abilities and the
experience of the designers
– Confusing independent & dependent variables
~ Measuring tech-savvyness from task success
61
© IDC, IIT Bombay
What can go wrong?




Reliability
Validity
Experimental design
Experimenter bias
– Experimenter likes one design
– The way the task is given
– What the experimenter chooses to see, probe
62
© IDC, IIT Bombay
What can go wrong?





Reliability
Validity
Experimental design
Experimenter bias
Intervening variables
– Unforeseen circumstances that might interfere with the testing or
performance
~ Interruptions in task time
~ User took three attempts to enter amount (Rs. 500) – this gave him
extra practice with the PIN
63
© IDC, IIT Bombay
User Testing
 Gives ‘better’ results than heuristic evaluation
– More confidence about findings
 Needs more time and money
64
© IDC, IIT Bombay
Task: Find 2-3 UTs of Each Type






Qualitative output
Task success
Task time
Errors
Efficiency
Satisfaction scores
65
© IDC, IIT Bombay
Usability Evaluation Methods
 Heuristic evaluation
 User Testing
 Think Aloud Protocol
66
Think Aloud Protocol
 A set of users use the product while continuously thinking out
aloud
– An evaluator notes all comments
 Users need practice before real tasks
– Use a short, familiar but unrelated task
 Result is a summary of qualitative data
– Excellent for formative evaluation
– Can’t get performance data, particularly tasks times
 Videotaping sessions helps capture nuances
67
© IDC, IIT Bombay
Thinking Aloud Protocol Variants
 Constructive Interaction
– Two users use the system together
 Retrospective Testing
– Let users think aloud in a video-tape playback
 Coaching method
– Let an expert teach a novice
68
© IDC, IIT Bombay
Usability Evaluation Methods
 Heuristic evaluation
 User Testing
 Think Aloud Protocol
 Card sort
– Louis Rosenfield, Peter Morville, Information Architecture for the World
Wide Web
69
Categorise...
70
Brinjal
Food
Rice
Brush
Mango
Scale
Camera
Milk
Shampoo
Cauliflower
Mobile phone
Stationery
Chicken
Non-veg
Sweet potatoes
DVDs
Notebook
Tomatoes
Electronics
Pen
Toiletries
Easel
Pencil
Toothpaste
Eraser
Potatoes
Vegetables
Fruits
Raw Banana
Water colours
© IDC, IIT Bombay
What did you use as categories?
Did you make new categories?
71
Brinjal
Food
Rice
Brush
Mango
Scale
Camera
Milk
Shampoo
Cauliflower
Mobile phone
Stationery
Chicken
Non-veg
Sweet potatoes
DVDs
Notebook
Tomatoes
Electronics
Pen
Toiletries
Easel
Pencil
Toothpaste
Eraser
Potatoes
Vegetables
Fruits
Raw Banana
Water colours
© IDC, IIT Bombay
Where did you put these?
72
Brinjal
Food
Rice
Brush
Mango
Scale
Camera
Milk
Shampoo
Cauliflower
Mobile phone
Stationery
Chicken
Non-veg
Sweet potatoes
DVDs
Notebook
Tomatoes
Electronics
Pen
Toiletries
Easel
Pencil
Toothpaste
Eraser
Potatoes
Vegetables
Fruits
Raw Banana
Water colours
© IDC, IIT Bombay
Card Sort Technique
 Make index cards
– Categories
– Sub-categories
– Content
– Number the cards (for later analysis)
73
© IDC, IIT Bombay
Card Sort Technique
 Make index cards
 Give a practice task
– People we know
– Celebrities
– Fruits and vegetables
74
© IDC, IIT Bombay
Card Sort Technique
 Make index cards
 Give a practice task
 Ask users to
– Make piles of cards
– Label piles
– One pile could be ‘I don’t care’ pile
– Think aloud
 Take notes while users work
75
© IDC, IIT Bombay
Card Sort Variables
 Open / Closed
– Totally open – users write their own cards and categories
– In between – cards are given, users write category and sub-category
labels
– Totally closed – pre-labelled cards and categories
– Early on – use open (discovery)
– Later on – closed (validation)
76
© IDC, IIT Bombay
Card Sort Variables
 Open / Closed
 Phrasing
– Word, phrase, sentence
~ Equipment, Having fun, Honesty is the best policy
– Category with sample sub-categories
~ Equipment (Computers, Lathe machines, ACs)
– Picture, question, answer, topic, task
77
© IDC, IIT Bombay
Card Sort Variables
 Open / Closed
 Phrasing
 Granularity
– High-level or detailed
– Paragraphs, pages, main pages, sub-sites
– Focussed or generic
– Focussed or random
78
© IDC, IIT Bombay
Card Sort Variables




Open / Closed
Phrasing
Granularity
Heterogeneity
– Early on – mix it up
– Later on – high consistency
79
© IDC, IIT Bombay
Card Sort Variables





Open / Closed
Phrasing
Granularity
Heterogeneity
Cross-listing
– Early on – avoid copies (structure)
– Later on – allow copies (navigation)
80
© IDC, IIT Bombay
Card Sort Variables






Open / Closed
Phrasing
Granularity
Heterogeneity
Cross-listing
Qualitative / Quantitative
– Qualitative suitable for CI / mind-mapping / evaluating IA
– Quantitative suitable for summative evaluation, to do cluster analysis
81
© IDC, IIT Bombay
Card Sort Analysis
 Users’ terminologies
 Mind-maps
– Users’ conceptual models
 % times two cards go together
 % times a card goes in a category
82
© IDC, IIT Bombay
Questions?
 Heuristic evaluation
 User Testing
 Thinking Aloud Protocol
 Card sort
 Feedback from real life usage
– Observational field studies
– Logging actual use
– User Feedback
– Questionnaires and Interviews
– Focus groups
83
Assignment 4
 Pick a topic of your interest
– Accessibility, HCI and robotics, e-learning,
 Read 5 recent papers (2 years) on this topic
 Prepare a 10-minute presentation on this topic
– Submit a 2-page abstract with references
 Inform your topics to the Aakash and Abhishek by next class
– Presentations start from the class of 19-3
– The presentation slots will be drawn randomly and announced by 12-3
– This assignment will be evaluated by your peers
84