CS160 Discussion Section
Evaluation Methodologies
April 24 2007
Evaluation Methodologies


Evaluation through expert analysis

Cognitive walkthrough

Heuristic evaluation

Model-based evaluation (GOMS)
Evaluation through user participation

Lab studies

Field studies
Choosing Evaluation Method


Comparative evaluation of Evaluation Methods is hard
because of the many uncontrolled variables.
Factors distinguishing evaluation techniques:

Stage/cycle at which the evaluation is carried out.

Style of evaluation

Level of subjectivity or objectivity of the technique

Type of measures provided

The information provided

The immediacy of the response

The level of interference implied

The resources required
Design vs. Implementation




Evaluation at Design

Need to be quick and cheap

Design experts only

Analytic
Evaluation of Implementation

A physical artifact exist, in the form of something ranging from
a paper mock-up to a full implementation

More comprehensive

Bring in users as participants
Participatory design involves users throughout the design process
Cognitive walkthrough are expert based and analytic but can be
used to evaluation implementation as well as design
Laboratory vs. Field Studies


Laboratory studies

Pro: allow controlled experimentation and
observation

Con: losing something of the naturalness of the
user's environment.

Dominate early states of design.
Field studies:

Pro: retains authenticity of the user's environment

Con: allows less control over user activity (i.e.,
Introducing confounding factors).

Conducted after a concrete implementation
Subjective vs. Objective

Subjective techniques

Rely heavily on the interpretation of the evaluator,
e.g., cognitive walkthrough or think aloud;
interviews/questionnaires inherently subjective.

Pro: powerful if used correctly, providing
information that may not be available from more
objective methods.

Con: Evaluator bias

Increasing the number of evaluators reduces the
risk of bias.
Subjective vs. Objective

Object techniques

Provide similar information for anyone correctly
carrying out the procedure (repeatable result),
eg., controlled experiments.

Pro: repeatable results, not dependent on the
persuasion of the particular evaluator; avoid bias
and provide comparable results.

Con: may not give detailed feedback on user
experience.
Qualitative vs. Quantitative
Measures


Qualitative

Non-numerical, more difficult to analyze

Can provide important detail that cannot be
quantified.

Subjective techniques tend to be qualitative.

Qualitative information can be mapped onto a scale:
questionnaires seek for qualitative information but
answers are represented quantitatively.
Quantitative

Numeric, easily analyzable using statistical techniques.

Objective techniques tend to be quantitative.
Information provided


Low-level

Allowing design decisions to be made (which font
or color to use)

Controlled experiment: an experiment to measure
a particular aspect of the interface
High-level

User's general impression of the system.

Answer questions such as “Is the system usable?”

Questionnaire, interview techniques.
Immediacy of response and
Intrusiveness

When is user's response recorded?

During interaction, eg., think aloud.




More intrusive: process of measurement can actually
alter the way user works.
Sensitivity on the part of the evaluator can help to
reduce this.
Automatic logging is an exception, but the amount of
useful information is limited.
Post interaction, e.g., interview, post task
walkthrough

Recollection is liable to suffer from bias in recall and
reconstruction
Resources





Resources to consider: equipment, time, money,
participants, expertise of evaluator and context.
Often have to pick the tactic that will produce the most
effective and useful information for the system under
consideration.
Some techniques are more reliant on evaluator expertise
than others, e.g., formal analytic techniques.
Heuristic evaluation requires less understanding of user
goal structures etc.
Sometimes it's not possible to gain access to the intended
users or it's not feasible to test the system in its intended
environment.
Summary




Evaluation is integral to the design process
Evaluation should take place throughout the
design life cycle.
Early evaluation helps to minimize the cost of
early design errors. Most techniques used at
this stage are analytic and involve experts.
Once a system has been developed (prototype
or full), experimental and observational
techniques can be used to get both
quantitative and qualitative information.
Q&A

Poster session

Implementation
Administrivia



Notify us by tomorrow if you intend to
participate in the competition.
Section will be canceled next week.
Make available your project implementation
online.