GOMS Analysis &
Automated Usability Assessment
Melody Y. Ivory (UCB CS)
SIMS 213, UI Design & Development
March 8, 2001
GOMS Analysis Outline






2
GOMS at a glance
Model Human Processor
Original GOMS (CMN-GOMS)
Variants of GOMS
GOMS in practice
Summary
GOMS at a glance

Proposed by Card, Moran & Newell in 1983
–
apply psychology to CS

employ user model (MHP) to predict performance of tasks
in UI
–
–
applicable to


3
task completion time, short-term memory requirements
user interface design and evaluation
training and documentation
Model Human Processor (MHP)

Card, Moran & Newell (1983)
–
–
most influential model of user
interaction
3 interacting subsystems


cognitive, perceptual & motor
each with processor & memory
–

4
described by parameters
 e.g. capacity, cycle time
serial & parallel processing
Adapted from slide by Dan Glaser
MHP

Card, Moran & Newell (1983)
–
principles of operation

subsystem behavior under certain
conditions
–

5
e.g. Fitts’s Law, Power Law of
Practice
ten total
MHP Subsystems

Perceptual processor
–
–
–
6
sensory input (audio & visual)
code info symbolically
output into audio & visual image
storage (WM buffers)
MHP Subsystems

Cognitive processor
–
–
input from sensory buffers
access LTM to determine response

–
7
previously stored info
output response into WM
MHP Subsystems

Motor processor
–
–
8
input response from WM
carry out response
MHP Subsystem Interactions


Input/output
Processing
–
serial action

–
parallel perception

9
pressing key in response to light
driving, reading signs & hearing
MHP Parameters

Based on empirical data
–

Processors have
–

word processing in the ‘70s
cycle time ()
Memories have
–
–
–
storage capacity ()
decay time of an item ()
info code type ()

10
physical, acoustic, visual & semantic
Perceptual Subsystem Parameters

Processor
–

cycle time () = 100 msec
Visual Image Store
–
–
–
storage capacity () = 17
letters
decay time of an item () =
200 msec
info code type () = physical

physical properties of visual
stimulus
–
11
e.g. intensity, color,
curvature, length
One Principle of Operation

Power Law of Practice
–
task time on the nth trial follows a power law




12
Tn = T1 n-a, where a = .4
i.e., you get faster the more times you do it!
applies to skilled behavior (perceptual & motor)
does not apply to knowledge acquisition or quality
Original GOMS (CMN-GOMS)


Card, Moran & Newell (1983)
Engineering model of user interaction
–
task analysis (“how to” knowledge)

Goals - user’s intentions (tasks)
–

e.g. delete a file, edit text, assist a customer
Operators - actions to complete task
–
cognitive, perceptual & motor (MHP)
– low-level (e.g. move the mouse to menu)
13
CMN-GOMS

Engineering model of user interaction
–
task analysis (“how to” knowledge)

Methods - sequences of actions (operators)
–
–

Selections - rules for choosing appropriate method
–
–
method predicted based on context
explicit task structure

14
based on error-free expert
may be multiple methods for accomplishing same goal
 e.g. shortcut key or menu selection
hierarchy of goals & sub-goals
Text-Editing
Example
15
CMN-GOMS Analysis

Analysis of explicit task structure
–
add parameters for operators


–
predict user performance


–
16
approximations (MHP) or empirical data
single value or parameterized estimate
execution time (count statements in task structure)
short-term memory requirements (stacking depth of task
structure)
apply before user testing (reduce costs)
Limitations of CMN-GOMS

No directions for task analysis
–

Serial v.s. parallel perceptual processing
–


granularity (start & stop)
contrary to MHP
Only one active goal
Error-free expert performance
–
no problem solving or evaluation

17
Norman’s Action Cycle
Norman’s Action Cycle
Goals
Evaluation
Execution
Intention to act
Evaluation of interpretations
Sequence of actions
Interpreting the perception
Execution of sequence of actions
Perceiving the state of the world
GOMS
18
The World
Variants of GOMS

Keystroke-Level Model (KLM)
–
simpler than CMN-GOMS

six keystroke-level primitive operators
–
–
–
–
–
–


–
19
K - press a key or button
P - point with a mouse
H - home hands
D - draw a line segment
M - mentally prepare to do an action
R - system response time
no selections
five heuristic rules (mental operators)
still one goal activation
Text-Editing Example (KLM)
20
Variants of GOMS

Natural GOMS Language (NGOMSL)
–
more rigorous than CMN-GOMS

uses cognitive complexity theory (CCT)
–
user and system models
 mapping between user’s goals & system model
– user style rules (novice support)



task-analysis methodology
learning time predictions
flatten CMN-GOMS goal hierarchy
–
–
21
high-level notation (proceduralized actions) v.s. low-level operators
still one goal activation
Text-Editing Example (NGOMSL)
22
Variants of GOMS

Cognitive-Perceptual-Motor GOMS (CPMGOMS)
–
activation of several goals


–
23
uses schedule chart (PERT chart) to represent operators &
dependencies
critical path method for predictions
no selections
Text-Editing Ex. (CPM-GOMS)
24
GOMS in Practice





Mouse-driven text editor (KLM)
CAD system (KLM)
Television control system (NGOMSL)
Minimalist documentation (NGOMSL)
Telephone assistance operator workstation
(CMP-GOMS)
–
25
saved about $2 million a year
Summary


GOMS in general
–
The analysis of knowledge of how to do a task in terms of the components of
goals, operators, methods & selection rules. (John & Kieras 94)
–
CMN-GOMS, KLM, NGOMSL, CPM-GOMS
Analysis entails



task-analysis
parameterization of operators
predictions
–
26
execution time, learning time (NGOMSL), short-term memory
requirements
Automated Usability Assessment
Outline




27
Automated Usability Assessment?
Characterizing Automated Methods
Automated Assessment Methods
Summary
Automated Usability Assessment?



28
What does it mean to automate assessment?
How could this be done?
What does it require?
Characterizing Automated
Methods: Method Classes

Testing
–

Inspection
–

an evaluator uses a set of criteria or heuristics to identify
potential usability problems in an interface
Inquiry
–
29
an evaluator observes users interacting with an interface (i.e.,
completing tasks) to determine usability problems
users provide feedback on an interface via interviews, surveys,
etc.
Characterizing Automated
Methods: Method Classes

Analytical Modeling
–

Simulation
–
30
an evaluator employs user and interface models to
generate usability predictions
an evaluator employs user and interface models to
mimic a user interacting with an interface and report
the results of this interaction (e.g., simulated
activities, errors and other quantitative measures)
Characterizing Automated
Methods: Automation Types

None
–

Capture
–

software automatically identifies potential usability problems
Critique
–
31
software automatically records usability data (e.g., logging
interface usage)
Analysis
–

no level of automation supported (i.e.,evaluator performs all
aspects of the evaluation method)
software automates analysis and suggests improvements
Characterizing Automated
Methods: Effort Levels

Minimal Effort
–

Model Development
–

requires completion of freely chosen tasks (i.e., unconstrained
use by a user or evaluator)
Formal Use
–
32
requires the evaluator to develop a UI model and/or a user
model in order to employ the method
Informal Use
–

does not require interface usage or modeling
requires completion of specially selected tasks (i.e.,
constrained use by a user or evaluator)
Automated Assessment Methods
Method Class
Testing
Inspection
Inquiry
Analytical
Modeling
Simulation
33
Automation Type
Analysis
Log File Analysis
(FIM)
Capture
Critique
Performance
Measurement (F)
Remote Testing
(FI)
Cognitive
Guideline Review
Guideline
Walkthrough (F)
Review (M)
Questionnaires (FI)
GOMS Analysis (M)
Cognitive Task
Analysis (M)
Programmable User
Models (M)
Genetic Algorithm Information
Modeling
Processor Modeling
Information Scent (M)
Modeling (M)
Petri Net Modeling
(M)
Automated Assessment Methods:
Generating Usage Data

Simulation – Automated Capture
–

Mimic user and record activities for subsequent analysis
Genetic Algorithm Modeling
–
–
Script interacts with running interface (Motif-based UI)
Deviation points in script  behavior determined by genetic
algorithm

–
Inexpensively generate a large number of usage traces

–
34
Mimic novice user learning by exploration
Find weak spots, failures, usability problems, etc.
Requires manual evaluation of trace execution
Automated Assessment Methods:
Generating Usage Data

Information Scent Modeling
–
Mimic users navigating a Web site and record paths


Web site model – linking structure, usage data, and content
similarity
Considers information scent (common keywords between user
goals and link text) in choosing links
–


Does not consider impact of page elements, such as images,
reading complexity, etc.
Stopping criteria
–
–
Reach target pages or some threshold (e.g., maximum navigation
time)
Requires manual evaluation of navigation paths

35
Percentage of agents follow higher- and lower-scent links
Log file visualization tool (Dome-Tree Visualization)
Automated Assessment Methods:
Detecting Guideline Conformance

Inspection – Automated Analysis
–
–

Quantitative Screen Measures
–
–

Size of screen elements, alignment, balance, etc.
Possibly generate initial layouts (AIDE)
Interface Consistency (Sherlock)
–
–
36
Cannot automatically detect conformance for all guidelines
One study – 78% best case, 44% worst case
Same widget placement and terminology (Visual
Studies showed 10-25% speedup for consistent UIs
Automated Assessment Methods:
Detecting Guideline Conformance

Quantitative Web Measures
–
–
Words, links, graphics, page breadth & depth, etc.
(Rating Game, HyperAT, TANGO)
Most techniques not empirically-validated


HTML Analysis (WebSAT)
–
37
Web TANGO uses expert ratings to develop prediction
models
All images have alt tags, one outgoing link/page,
etc.
Automated Assessment Methods:
Detecting Guideline Conformance

Web Scanning Path
(Design Advisor)
–
Determine how users will
scan a page based on
attentional effects of
elements

38
motion, size, images, color,
text style, and position
Automated Assessment Methods:
Suggesting Improvements


Inspection – Automated Critique
Rule-based critique systems
–
Typically done within a user interface management system

–

–
Apply guidelines relevant to each graphical object
Widely applicable to Windows UIs
HTML Critique
–
39
X Window UIs (KRI/AG), control systems (SYNOP), space
systems (CHIMES)
Object-based critique systems (Ergoval)
–

Very limited application
–
Syntax, validation, accessibility (Bobby), and others
Although useful, not empirically validated
Automated Assessment Methods:
Modeling User Performance

Analytical Modeling – Automated Analysis
–
–

Predict user behavior, mainly execution time
No methods for Web interfaces
GOMS Analysis (previously discussed)
–
–
Generate predictions for GOMS models (CATHCI, QGOMS)
Generate model and predictions (USAGE, CRITIQUE)


Cognitive Task Analysis
–
Input interface parameters to an underlying theoretical model
(expert system)

–
40
UIs developed within user interface development environment
Do not construct new model for each task
Generate predictions based on parameters as well as
theoretical basis for predictions
Automated Assessment Methods:
Modeling User Performance

Programmable User Models
–
–
Cross between GOMS and CTA analyses
Program UI on a psychologically-constrained
architecture


41
Constraint violations suggest usability problems
Generate quantitative predictions
Automated Assessment Methods:
Simulating User Behavior


Simulation – Automated Analysis
Petri Net Modeling (AMME)
–
–
–

Construct petri net from logged interface usage
Simulates problem solving process (learning, decisions, and
task completion)
Outputs measure of behavior complexity
Information Processor Modeling (ACT-R, SOAR,
CCT,…)
–
Methods employ sophisticated cognitive architecture with
varying features

42
Modeled tasks and components, predictions, etc.
Automated Assessment Methods:
Simulating User Behavior

Web Site Navigation (WebCriteria)
–
Similar to GOMS Analysis

Constructs model of site and predicts navigation time for a
specified path
–
Based on idealized Web user (Max)
– Navigation time only for shortest path between endpoints
 Does not consider impact of page elements (e.g.,
colors,reading complexity, etc.)
– Reports on page freshness and composition of pages (text, image,
applets, etc.)
– Supports only a small fraction of analysis possible with guideline
review approaches
–
43
Used to compare sites (Industry Benchmarks)
Summary

Characterizing Automated Methods
–

Automated Methods
–
–
–
–

Mainly automated capture and analysis
Guideline review enables automated critique
Represented only 33% of 132 surveyed approaches
Most require formal or informal interface usage
More Information
–
–
–
44
Method Class, Automation Types, Effort Levels
webtango.berkeley.edu
Survey paper on automated methods
Papers on quantitative Web page analysis