Evaluation Types
GOMS and KLM
CS352
Announcements
• Mid-term Tue of Week 5
• Project presentations
– Your users (before mid-term)
• Next milestone of you project will be out later
today or tomorrow (Prototypes)
• Reading
– 12.3 (evaluation methods), 15.4 (GOMS)
2
Where we are in PRICPS:
• Predispositions: Did this in Project
Proposal.
• RI: Research was studying users.
Hopefully led to Insights.
• CP: Concept and initial (very low-fi)
Prototypes due some time next week
•  Evaluate throughout, repeat
iteratively!!
Evaluation
• Analytical – based on your head
• Empirical – based on data
– Formative
• inFORMs design
• what is (still) needed?
– Summative
• did it work?
Analytical methods
• You follow established
guidelines/procedures/models to decide
(in your head) how good your design is.
• Examples:
– GOMS/KLM – for skilled users, no errors.
• evaluating efficiency of regular use.
– Heuristic Evaluation
– Cognitive Walkthrough – for first-time users.
• evaluating ease of learning.
GOMS (and KLM)
• GOMS: a family of models.
• Predict user performance.
• Useful for predicting actual time a user will
take in UI.
• Useful for comparing different UIs.
GOMS Constructs
•
•
•
•
Goal, Operators, Methods, Selection rules
Goal: “what”.
Method: “how” steps (learned).
Operators: Cognitive processes + physical
actions to DO it.
• Selection rules: rules saying which method
to select.
GOMS example:
Delete a word
• Goal: delete a word in a sentence.
• Method #1: use the menu
– Recall that the word has to be highlighted.
– Recall that the command is “cut”.
– Recall that “cut” is in the Edit Menu.
– Accomplish goal of selecting and executing
“cut”.
– Return: goal accomplished.
GOMS example (cont.)
• Method #2: use the delete key
– Recall where to position cursor in relation to
word to be deleted.
– Recall which key is delete key.
– Press “delete” key to delete each letter.
– Return: goal accomplished.
• Operators used in these methods
– Click mouse, Drag cursor over text, Select
menu, Move cursor, Press KB key, Recall, ...
GOMS example (cont.)
• Selection rules:
– Use mouse/menu method (#1) if there’s a lot
of text to delete.
– Else use “delete” key (method #2).
Applications of GOMS
• Various application and fields
– Telephone operator workstation using CPMGOMS
– CAD system for ergonomic design using
NGOMSL
– Intelligent tutoring system using NGOMSL
– Mouse driven text editor using KLM
– Bank deposit reconciliation system using KLM
– Space operations database system using
KLM
Telephone operator workstation
using CPM-GOMS
• The task: a telephone company operator
responding to customer requests for assistance.
Limitations of GOMS [Card et al. (1980) ]
• applied to skilled users, not to beginners or
intermediates.
• doesn't account for either learning of the system
or its recall after a period of disuse.
• doesn't account for errors.
• does not address the amount and kind of fatigue
• individual differences among users is not
accounted for in the model.
KLM (a low-level variant of
GOMS)
• Keystroke Level Model.
• Simple, but accurate. Widely used.
• Scope:
– skilled users
– doing a task error-free.
– using a specific method in a UI.
• CogTool has this built-in.
KLM Operators
• User Operators:
– K (keystroke), P (point), H (homing), D
(drawing), M (mental: think).
– Times for each are provided to you
• based on extensive research/empirical data.
• System Operator:
– R (respond).
KLM/GOMS example 1:
Open a file
• 27 ways to
open a file
in Windows!
• Methods as
states +
transitions.
• (High op
granularity
combines
low-level ops.)
KLM/GOMS example 1 (cont.)
• Used to:
– Understand cost
of each.
– (break down
participants
actual costs).
– “RL”:recent lists.
– “WS/WX”:search.
• And to...
KLM/GOMS example 1 (cont.)
• Use the breakdowns to compare costs to
new UI alternative (new feature “FP”)
actual
calc’d
Hick’s Law
• GOMS’s “S” stands for “Selection”
• Hick’s Law predicts the time to make a choice
between multiple choices, n is the # of choices:
Fitts’ Law
• Predicts that the time required to rapidly move to
a target area is a function of the distance to and
the size of the target
In practice…
• GOMS not used often
• Fitts’ law often used for determining best
case for new kinds of input methods
CogTool Example
• CogTool examples.
– Calculating the cost of the task in this UI.
– Comparing the cost if do the task with
different widgets.
– Where are these cost differences coming
from?
• under the hood to ACT/R
What’s coming up next:
• Heuristic Evaluation
• Cognitive Walkthroughs
• Empirical Studies