Introduction to System
Evaluation
IS 588
Dania Bilal
Spring 2008
Design Philosophies
• System-centered design philosophy
– Software works well; why evaluate from users’
perspectives?
• User-centered design philosophy
– Software should be usable: based on users’
needs, tasks, and cognitive and affective
behaviors.
Why Evaluate
• Following design guidelines does not
guarantee usability
• Need to assess the user’s experience with
system, cognitively and affectively
– Allow identification of level of usability
• Saves money
– Problems can be fixed before system is
released
What Users Need from a System?
• A system that is
– Pleasing; satisfying
– Easy to learn; friendly
– Effective; challenging
– Efficient; saves user’s time and efforts
– Supported by positive, corrective feedback
mechanisms
– Predictable; consistent
Investing in User Testing
• Tognazzini:
– Problem can be fixed before product is
released
– Designers can identify “real” problems with
system
– Provide objective assessment for redesigning
system
– Save money and marketing time
– Robust design to sell
• High confidence in product
Why NOT Evaluate?
• Save time
• Save money
• Philosophy: users don’t know much about
system capabilities and some may not
articulate well their needs.
• Other???
• How would you convince system
developers to involve users in the design
and/or evaluation processes?
When to Evaluate?
• During product development
– After market research, mock-ups, low-fidelity prototypes, etc. are
designed and users’ reactions are elicited
– Formative evaluation
– Summative evaluation
• Before release of final product
– After prototypes and actual designs have been tested and
problems have been fixed
– Summative evaluation
• After release of final product
– To measure the usability and success of the product with users
in a variety of settings.
What to Evaluate?
• The user’s experience with the product
– User needs and requirements
– Success, ease of use, friendliness, and other
basic usability features
– Affective states
• Likes, dislikes, satisfaction, frustration, attitude
– Support of use patterns
– Other interaction behaviors based on purpose
of product
HutchWorld Case Study
• Virtual community chat system
• Developed by Microsoft Virtual Worlds
Research Group and Fred Hutchinson
Cancer Research Center
• Purpose
– To enable cancer patients, caregivers, family,
and friends to chat and gain emotional and
information support from one another.
HutchWorld Case Study
• Early design ideas
– Learn about the patients’ experiences.
• Informal on-site test
– Computers were set up in hospital and a
scaled-back prototype of V-chat was installed.
– A sample of patients and their families
interacted with prototype
HutchWorld Case Study: Usability
Issues Identified
• User preference for asynchronous
communication
• Addition of websites about cancer
information
• Addition of games and entertainment
features
HutchWorld Case Study: Redesign
Based on Issues Identified
• Software was redesigned based on user
needs
• HutchWorld became a portal containing a
variety of information, communication, and
entertainment areas
– Bulletin boards, email, text chat, web page
creation wizard, and other features
HutchWorld Case Study: Second
Usability Assessment
•
•
•
•
Seven participants (male & female)
Prior experience was assessed
Interaction with system
Performed at Microsoft usability labs
–
–
–
–
–
Structured tasks (see Text)
Verbalization (think aloud protocol)
Participants activities were captured online
Observer/evaluator interacted with participant in lab
Evaluator took observational notes
HutchWorld Case Study: Second
Usability Assessment
• Development team member interacted
with system as a participant
• Short questionnaire was completed by
each participant after completing tasks
• Participants rated difficulties in completing
each task (see Text).
Second Usability Assessment:
Lessons Learned
• Problems with software and their severity
were identified
– Rating of problems: low, medium, high
– Fixes were made
After the Second Usability
Assessment…
• Additional observations and testing were
done
• Six new participants
– Simultaneous interaction in labs
– More detailed and focused
– Structured similar to previous test
– Fewer problems were identified
– Fixes were made
Third Usability Assessment: Focus
Group Setting
• Focus group interacted with system in
Cancer Research Center
• Patients and caregivers observed
interaction
• Feedback on final version was obtained
Final Usability Assessment: Natural
Setting
• System was used in a residential building
with an Internet access
• Building housed patients and their families
• Informal observation of patterns of use
• Assess how HutchWorld integrated with
patients’ lives (medical care routines;
access to social support, etc.)
When to Stop System Testing?
Class Discussion
1. When do you think developers/designers
should stop testing and why?
2. Identify the various data collection
methods employed in HutchWorld Study?
Are these mixed methods useful and why?
Group Projects
• Work with your group on the upcoming
assignment/project due.