Chapter 11:
An Evaluation Framework
Group 4: Tony Masi, Sam Esswein,
Brian Rood, & Chris Troisi
Introduction



Evaluation helps ensure that product meets
the users’ needs
Recall HutchWorld & Olympic Messaging
System (OMS) – chapter 10
What to evaluate?


usability
user experience
Chapter Goals




Key concepts & terms to discuss
evaluation
Description of evaluation paradigms &
techniques
Conceptual, practical, and ethical issues
DECIDE framework for evaluation
Evaluation Paradigms


Key terms- evaluation paradigms, user
studies
4 core evaluation paradigms




“Quick and dirt” evaluation
Usability testing
Field Studies
Predictive Evaluation
Key Terms
User studies involve looking at how
people behave in their natural
environments, or in the laboratory, both
with old technologies and with new
ones.
Evaluation Paradigm is the set of
beliefs which guide any type of
evaluation
“Quick and Dirty”



Quick & Dirty evaluation describes the
common practice in which designers
informally get feedback from users or
consultants to confirm that their ideas are inline with users’ needs and are liked.
Quick & Dirty evaluations are done any time.
The emphasis is on fast input to the design
process rather than carefully documented
findings.
Usability testing



Usability testing involves recording typical
users’ performance on typical tasks in
controlled settings. As the users perform
these tasks they are watched & recorded on
video & their key presses are logged.
This data is used to calculate performance
times, identify errors & help explain why the
users did what they did.
User satisfaction questionnaires & interviews
are used to elicit users’ opinions.

Recall HutchWorld and OMS
Field Studies



Field studies are done in natural settings
The aim is to understand what users do
naturally and how technology impacts them.
In product design field studies can be used
to:
- identify opportunities for new technology
- determine design requirements
- decide how best to introduce new
technology
- evaluate technology in use.
Field Studies

Two Approaches


Outsider – observing and recording what
happens as an outsider looking in
Insider – participant in study that explores
the details of what happens in a particular
setting
Predictive Evaluation

Experts apply their knowledge of typical
users, often guided by heuristics, to predict
usability problems.




Note: heuristics- design principles used in practice
Another approach involves theoretically based
models.
A key feature of predictive evaluation is that
users need not be present
Relatively quick & inexpensive
Key Aspects of each
Evaluation Paradigm








Table 11.1 – page 344
role of users
who controls the process & relationship
during evaluation
location
when is it most useful to evaluate
type of data collected & how it is analyzed
how findings are fed back to the design
process
philosophy that underlies these paradigms
Evaluation Techniques





Observing users
Asking users their opinions
Asking experts their opinions
Testing users’ performance
Modeling users’ task performance to
predict the efficacy of a user interface
Observing Users

Techniques




notes
audio
video
interaction log
Asking users their opinions

Questions like:






what do you think about the product?
does it do what you want?
do you like it?
does the aesthetic design appeal to you?
did you encounter problems?
would you use it again?
Asking experts their opinions



Use heuristics to step through tasks
Typically use role-playing to identify
problems
It is inexpensive and quick to ask
experts rather than perform laboratory
and field evaluations
User Testing




Recall HutchWorld example
Usually conducted in a controlled
environment
Users perform well-defined tasks
Data can be collected and statistically
analyzed
Modeling users’ task performance



Model human-computer interaction to
predict the efficiency and problems in
the design
This is successful for systems with
limited functionality
Table 11.2 - page 347
DECIDE: framework






Determine the goals the evaluation
addresses.
Explore the specific questions to be
answered.
Choose the evaluation paradigm and
techniques to answer the questions.
Identify the practical issues.
Decide how to deal with the ethical issues.
Evaluate, interpret and present the data.
Determining the Goals




What are the goals of the evaluation?
Who wants it and why?
Goals influence the paradigm for the study.
Some examples of goals:




Check that evaluators have understood user needs
Check to ensure that the final interface is
consistent.
Investigate how technology affects working
practices.
Improve the usability of an existing product .
Explore the Questions


All evaluations need goals & questions to
guide them so time is not wasted on illdefined studies.
For example, the goal of finding out why
many customers prefer to purchase paper
airline tickets rather than e-tickets can be
broken down into sub-questions:
- What are customers’ attitudes to these new tickets?
- Are they concerned about security?
- Is the interface for obtaining them poor?
Choose Evaluation Paradigm
and Techniques



Evaluation Paradigms determine which
type of techniques will be used.
Trade-Offs
Combinations of Techniques
-HutchWorld
Identifying Practical Issues
For example, how to:
select users
 stay on budget
 staying on schedule
 evaluators
 select equipment

Decide on Ethical Issues




Consideration for peoples rights.
Develop an informed consent form
Participants have a right to:
- know the goals of the study
- what will happen to the findings
- privacy of personal information
- not to be quoted without their agreement
- leave when they wish
“do unto others only what you would
not mind being done to you”
Evaluate, Interpret, and
Present Data





Reliability
Validity
Biases
Scope
Ecological Validity
Pilot Studies



Pilot Study is a small trial run of the main
study.
Pilot studies are always useful for testing
plans for an evaluation, before launching the
main study
Often evaluators run several pilot studies.