Evaluation Methods
April 20, 2005
Tara Matthews
CS 160
In 160 We’ve Covered…
•
•
•
•
Task Analysis & Contextual Inquiry
Cognitive Walkthrough
Heuristic Evaluation
WOZ usability study w/ paper prototypes
There are many more methods…
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
How to chose a method?
• Stage of study
– formative, iterative, summative
• Pros & cons
• Metrics
– depends on what you want to measure
• Qualitative vs. quantitative
• Research perspective
– CS vs. psychology vs. sociology
Pros & Cons
•
•
•
•
•
Realism
Precision
Generalizability
Time & cost
Researcher expertise
Methods
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Survey
• Online / paper questionnaires distributed
to target audience
• Can be used to
– tabulate quantitative data
– gather qualitative feedback (opinions,
feelings, etc.)
• Useful at any time in study
Survey
• Pros
– Easy to get a large number of responses.
– Quick and easy to conduct.
– Highly generalizable.
• Cons
– Self-selection.
– Participants often only offer enough information to
answer the question.
– Can miss details.
– Low in realism and precision.
Interview
• Evaluators formulate questions on the
issues of interest.
• Interview representative users, asking
them these questions in order to gather
information desired.
• Interviewer reads questions to user, who
replies verbally; interviewer records
responses.
Interview
• Pros
–
–
–
–
Quick and easy to conduct.
Gives designer quick feedback on a range of ideas.
Can get a person’s initial reaction to an idea.
Can get detailed information from a person.
• Cons
– Often takes place away from natural setting.
– Question wording or interviewer “body language” can bias
answers.
– High probability of false positives and missed problems (e.g.,
users may not have a clear idea of how an app will be used).
– Can miss details if interviewer doesn’t know what issues to
draw out.
Controlled Lab Experiment
• In lab, manipulate one feature of a system
to assess the causal effects of the
difference in that manipulated feature on
other behaviors of the system.
• Example:
– in lab, show users 4 versions of a website:
• blue, yellow, red, and black text
– measure time to find specific words
– compare
Controlled Lab Experiment
• Pros
–
–
–
–
Provides precise, quantifiable data.
Easier to draw inferences from data.
Relatively quick.
Can get a medium-sized number of participants.
• Cons
– Short duration of a lab experiment may not be enough
to allow users to become accustomed to an app.
– Not a natural setting – interaction may not be normal.
In-lab Observation
• Participants come to lab to "use" an interface
• Given sample tasks to complete with it
• Evaluators observe and possibly audio- or
videotape
• Participants may "think out loud"
• Can use lo-fi prototype (for a project in the
design stage) to an almost-complete interface
• Evaluators note participants’
– emotions, exclamations, facial expressions, and other
"qualitative" data
– take note of quantitative data such as time to
complete a task or number of errors
In-lab Observation
• Pros
– Relatively quick.
– Can get a medium-sized number of participants.
• Cons
– Observations are subjective and error prone.
– Short duration of lab observation is not enough time
for user to get accustomed to using the interface.
– Not a natural setting – interaction may not be normal.
Controlled Field Experiment
• In natural setting, manipulate one feature
of a system to assess the causal effects of
the difference in that manipulated feature
on other behaviors of the system.
• Example:
– Participants use 3 different input devices in
their own office: mouse with 1, 2, or 3 buttons
– Perform a set of tasks
– Measure differences
Controlled Field Experiment
• Pros
– Less intrusive than most other evaluation methods.
– Provides more precise data than field observation.
– Can observe natural behavior of user (though some
part of the system will be controlled/unnatural).
• Cons
– More intrusive than field observation.
– Less natural than field observation.
Field Observation Study
• Evaluator makes direct observations of
“natural” systems
• Takes care to not intrude on / disturb those
systems
• A.K.A. “ethnography”
Field Observation Study
• Pros
– Only way to observe natural behavior of user &
interaction between user & tools.
• Cons
–
–
–
–
Difficult and time consuming.
Hard to get permission to observe people.
Observations are subjective and error prone.
Cannot make strong interpretations from
observations.
– Not very generalizable.
Heuristic Evaluation
• Pros
– Quick and easy.
• Cons
– Nielson’s heuristics may not be as relevant to
non-GUIs.
– Results in false positives in missed problems,
especially when experts are not part of target
audience.
Cognitive Walkthrough
• Pros
– Quick and easy.
• Cons
– Results in false positives and missed
problems when evaluator is different from
target audience.
Automate Observation Study
• Techniques include
– video or audio recording of user
– pop-up screens
– screen shots
– time logging
– log users actions (collecting statistics about
detailed system use)
Automate Observation Study
• Pros
– Eases burden on observers for data collection
& analysis.
• Cons
– Setup is often more time-consuming to
complete.
– Harder to get approved if it involves analysis
of videotape or audiotape.
– May miss nuanced/interpretive details.
Experimental Simulation
• In-lab experiment that is as much like
some real situation as possible.
• Example:
– ground-based flight simulator
– behaves as closely as possible to a real flight
– still under researcher control
Experimental Simulation
• Pros
– Still fairly precise.
– More realistic than in-lab experiment.
• Cons (same as lab exp.)
– Short duration of a lab experiment may not be
enough to allow users to become accustomed
to an app.
– Not a natural setting – interaction may not be
normal.
Claims Analysis
• Claim = statement that a certain aspect (button,
scrollbar) of a design has psychological
implications reflected in how capable a user is in
using that design
• UI artifacts are listed along with their design
features & pros/cons
• Helps
– select among alternative designs
– clarify questions to be analyzed through user testing
by stating how the design should work (in claims)
GOMS
• A method to describe user tasks and how a user
performs those tasks with a specific interface design
• Views humans as information processors
– Small number of cognitive, perceptual, and motor operators
characterize user behavior
• To apply GOMS:
– Analyze task to identify user goals (hierarchical)
– Identify operators to achieve goals
– Sum operator times to predict performance
• GOMS =
– Goals: What a user wants to accomplish
– Operators: Cognitive or physical actions that change the state of
the user or the system
– Methods: Groups of goals and operators
– Selection rules: Determine which method to apply
GOMS
• Pros
– Predict human performance before committing to a
specific design in code or running user studies
– Many studies have validated the model (it works)
• Cons
– Assumes error-free, skilled user behavior
– No formal recipe for how to perform analysis
– Significant time investment
Computer Simulation
• Creating a complete & closed system that
models the operation of the concrete
system without users.
• Example:
– geophysical process going on in connection
with the eruption of Mount St. Helens
Computer Simulation
• Pros
– Supposedly high in realism (depends on
accuracy of data/system replication)
• Cons
– Low in precision & generalizability
Formal Theory
• Formulating general relations
(propositions, hypothesis, or postulates)
among a number of variables of interest.
• Pros
– Relatively generalizable
• Cons
– Not realistic or precise
How to chose a method?
• Stage of study
• Pros & cons
–
–
–
–
•
•
•
•
Realism
Precision
Generalizability
Time & cost
Researcher expertise
Metrics
Qualitative vs. quantitative
Research perspective
Methods
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Early Stage
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Early Stage
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Iterative & Summative Stages
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Iterative & Summative Stages
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Realism
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Realism
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Precision
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Precision
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Generalizability
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Generalizability
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Time & Cost
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Time & Cost
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Researcher Perspective
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Metrics: examples
• Traditional GUIs:
– efficiency (time to complete task)
– accuracy (# of errors)
– simplicity
• Peripheral Displays:
– awareness (recall)
– distraction (dual-task behavior)
– aesthetics
Peripheral Displays
• Survey
• Interview
• Controlled-lab
experiment
• In-lab observation
• Controlled field
experiment
• Field observation
study
• Heuristic Evaluation
• Cognitive Walkthrough
• Contextual Inquiry
• Automated
observation user study
• Experimental
simulation
• Claims analysis
• GOMS
• Computer simulation
• Formal theory
Questions?