Tutorial Topic 2 Usability Evaluation Principles, Methods and Techniques
(Week 3)
Readings:
Text: (Preece, Rogers and Sharp, 2002): Chapters 10-15
Instructions
For this exercise, you are to consider what is involved in planning a usability evaluation.
Following on from the two techniques considered last week:
1. Heuristic evaluation and
2. User testing.
Remember that there were different types of results using the different techniques. Consider
these and other types of testing techniques to be added into an evaluation plan.
DECIDE framework
Determine the overall goals that the evaluation addresses
Explore the specific questions to be answered
Choose the evaluation paradigm
Identify the practical issues
Decide how to deal with the ethical issues
Evaluate, interpret and present the data
Activity 1: What to consider in an evaluation test
 In groups of 3-4, brainstorm what you need to consider before planning an evaluation test
for any type of system.
 Write up your ideas on paper.
Consider the Murdoch Website for the following activities
Activity 2: Define the purpose of the website
 In your groups, brainstorm the purpose of the site.
 Use paper to brainstorm and then write up your ideas on your white board.
 Prioritise if more than one purpose.
Activity 3: List the users (user groups)
 In your groups, the different user groups for the site.
 Use paper to brainstorm and then write up your ideas on your white board.
 Prioritise the users.
Activity 4: List the tasks for each user group
 In your groups, write up the tasks for the different user groups.
 Use paper to brainstorm and then write up your ideas on your white board.
 Prioritise the tasks.
Activity 5: Coming up with the dimensions
 Of these, what do you need to consider for testing this site.
ICT325 Semester 2, 2005
Topic 2


What dimensions do you need to test for? [See list of dimensions for ideas]
Use paper to brainstorm and then write up your ideas on your white board.
Activity 6: Coming up with the questions
 Choose 1-2 dimensions and list the questions to test that dimension.
 Write up your ideas on your white board.
Activity 7: Coming up with the measures
 In your groups, come up the measures. That is, what to measure for your chosen
dimensions.
 See list for ideas.
Activity 8: Coming up with the evaluation techniques
 In your groups, come up with possible evaluation techniques.
 Use paper to brainstorm and then write up your ideas on your white board.
Activity 9: Choose the best 2 evaluation methods
 Choose the best 2 techniques given the following constraints.
Budget: $2000
Time: 1 week
Resources: Access to users - Yes
Consider what is practical.
Activity 10: Write a scenario
 Write a short scenario that could be used for testing.
ICT325 Semester 2, 2005
Topic 2
Appendix: Additional material
Usability goals (criteria/dimensions):
Some options for usability goals for User Interfaces:
 Efficiency (initial performance; long-term performance) - speed; error rates; etc
 Effectiveness (initial performance; long-term performance) - for required range of tasks,
accomplished at better than some required level of performance, by some required
percentage of the specified target range of users;
 Advanced feature usage - for specific set of system features;
 Learnability (and retainability) - of specified user commands;
 Utility of Help and Documentation (if provided);
 Flexibility (adaptability) - can the system be used in different ways;
 Attitude (first impression; long-term user satisfaction) - in terms of: satisfaction;
tiredness; discomfort; frustration; personal effort; etc.
 Etc.
Some options for usability goals for Multimedia (Reeves):
 Ease of Use;
 Navigation;
 Cognitive Load;
 Mapping;
 Screen Design;
 Knowledge Space Compatibility
 Information Presentation;
 Media Integration;
 Aesthetics;
 Overall Functionality.
Dimensions of usability
Content:




Usefulness / relevance
Accuracy / truthfulness
Completeness
Currency
Hyperlinks:


Quantity
Quality, relevance, currency (dangling links?)
Clarity and Presentation:




Simplicity / elegance
Layout / structure
Use of graphics/diagrams
Readability
Adherence to Conventions or Standards
Navigation support:





Structured layout
Overview diagram/site map
Navigation buttons
Search / keywords / links panel / frames
Different design for different types of pages, e.g. Colour
ICT325 Semester 2, 2005
Topic 2
Search facilities:


Functionality
Utility
Attractiveness / Annoyance (attention getting/keeping):







Colour use
Text fonts - appropriate?
Use of graphics / images / videos
Music / sounds
Animations / blinking / wobbling - effective or annoying?
Banners - too distracting?
Creativity / uniqueness / innovations
Opportunities for Interaction:


Functionality
Utility
Error reduction and recovery
Help / documentation - availability and utility
Download Speed


Number /size / format of graphics
Thumbnails, progressive build up, etc
Hardware and Software requirements:



H/W restrictions
Reliance on downloads
Needing latest versions of s/w
Reliability - does server go down often
Customisability / user model/profile:

Cookies for user attributes
Ease of download of information:

Formats of text files
Integrity / ethics


Security (e.g. use of credit card no / email address)
Race / gender treatment
Appropriate measures for usability goal:
Some options for measures:
 Time to complete task;
 Number of keystrokes;
 Number or percentage of errors;
 Percentage of a task completed in a given time;
 Ratio of successes to failures;
 Ease of navigation through system functionality / data;
 Suitability of screen design / layout / terminology / feedback;
 Consistency of terminology, screen design, etc.;
 Time spent in errors or recovery;
 Number of commands/actions used to perform task(s);
 Frequency of help and documentation use;
 Number of repetitions of failed commands;
 Number of available commands not invoked;
 Time taken to learn a specific set of commands / interaction sequence;
 Number of commands remembered
ICT325 Semester 2, 2005
Topic 2


Number of times user expresses frustration or satisfaction;
Users' attitudes and opinions re specific aspects of system use.
plus others
Critical review of usability specification for specific system:
Some relevant questions:
 Is each usability attribute practically measurable?
 Are user groups (classes) specified clearly enough?
 Is the number of attributes to be measured reasonable?
 How well do the attributes capture usability for the design?
Evaluation techniques
Note: Consider those with users and those without Users
affinity diagramming
allocate tasks
attitude questionnaires
brainstorming
card sorting
competitor analysis
context of use analysis
contextual inquiry
cost-benefit analysis
critical incident analysis
design guidelines
diagnostic evaluation
evaluate prototype
evaluating an existing system
expert evaluation
field study
focus groups
paper prototyping
parallel design
participatory evaluation
patterns
performance testing
planning usability
pleasure
post release tesing
prototyping
questionnaires
rapid prototyping
remote testing
requirements meeting
scenarios of use
stakeholder meeting
standards: ISO 13407
storyboarding
style guides
subjective evaluation
surveys
getting started
task analysis
heuristic evaluation
interviews
ISO 13407 UCD standard
usability testing
use cases
user satisfaction
observation of users
wizard of oz
ICT325 Semester 2, 2005
Topic 2