2. Models for cognitive ergonomics
2.1.
the concept of models
2.2.
models in cognitive psychology
2.3.
cognitive ergonomics models
2.1.
the concept of models
“M is a model of A if M can be used
to answer questions about A” (Ross, 1983)
A model is a representation of relevant characteristics
of an object (Rohr & Tauber, 1984)
A model is a description that can be communicated, of a
certain aspect of part of the real world, viewed at a certain
level of abstraction (Oberquelle, 1984)
representation of relevant characteristics of an object
Mental model of
system
user
system
User
interface
aspect of part of the real world,
viewed at a certain level of abstraction
Aspect 1, graphics
Aspect 2,
dialogue
User interface
system
the concept of models
internal models & external models
External models are for communication,
should be represented explicitly
Internal models are for “execution”: there is an “agent” who
uses the model to make decisions based on the behavior of
the model, and to make predictions on the behavior of the
modeled reality
– if the agent is human: mental model
– if the agent is a machine: program, database
2.2.
models in psychology
2.2.1. models of human information processing
• modern theories, e.g., Barsalou
• mental models in Cognitive Psychology
2.2.2. mental models in Cognitive Ergonomics - Norman
2.2.1. Model of human information processing
long term MEMORY
input
PERCEPTION
TASK/
THOUGHT
STIMULI
working
MEMORY
output
MOTOR/BEHAVIORAL
ACTIVITITES OF THE COGNITIVE SYSTEM
PERCEPTION:
• complex sensorial processes
• primary images containing all information about the concrete features
• of objects and phenomena
• that act direct action upon the sensory systems
(visual, acoustic, kinesthetic, olfactory, gustatory).
MEMORY:
• ability to remember, recognize and recall
• information is encoded, stored and retrieved.
• active: structuring, constructive and creative psychic mechanism.
THOUGHT: the process of information processing in working memory.
VISUAL PERCEPTION - A COMPUTATIONAL THEORY
(pattern recognition)
PRIMARY PROCESSING
SECONDARY PROCESSING - RECOGNITION
Visual Stimuli
Descendant processing
Primary sketch
2.5 D sketch
texture movement color distance position
depth
Processing modules
form
gestalt
geons
principles
segmentation
3D Representation
VISUAL PERCEPTION - A COMPUTATIONAL THEORY
(pattern recognition)
Data-driven (bottom-up ) processing
1. mechanisms of edge detection, processing of texture, movement, color, distance
and depth, position and form from processing of shadow of stimulus
(2.5-D sketch,) - automated/ modular/preattentional/ unconsciousness processing
2. edge organization - gestalt principles of perception (applied to 2.5-D sketch) :
proximity, similarity, good continuation, closure
example:
IwOuLdLiKeToDrInKaBeErInArOmAnIaNbAr.
3. geons generation (geometrical ions)
segmentation of the 2.5-D sketch (intermediary image) in zones of maximal
concavity
4. Recognition
geons activate from memory the objects made-up by the respective geons, matching
the 2.5-D sketch with the representation stored in memory
GESTALT PRINCIPLES OF PERCEPTION
A. Proximity principle
B. Similarity principle
C. Good continuation principle
A
D
C
B
D. Closure principle
VISUAL PERCEPTION - A COMPUTATIONAL THEORY
(pattern recognition)
Conceptually driven (top-down) processing
• concepts and higher-level processes influence pattern recognition
• applied to 2.5-D sketch
• recognition in the case of:
verbal stimuli (word superiority effect; sentence superiority effect)
T EC T
THE WORK MUST GET DONE.
WORK
VISUAL PERCEPTION - A COMPUTATIONAL THEORY
(pattern recognition)
I’m zhizkizg tz enzoy zhiz wezk-ezd az thz sezsize.
- objects (object superiority effect)
- human faces
 OBS.: Importance of implicit/tacit knowledge about the (statistical) regularities
of the world in pattern recognition (physical support of things, reciprocal
occlusion, occurrence probability, relative size, position and rigidity of objects
etc.).
Violation of tacit assumptions causes visual illusions. This assumptions are not
specific (they are applied automatically to any object).
The importance of context
Illusions - at level of modular processing
AUTOMATIC PROCESSING
Learned automatic processing
Frequent association of a stimulus with a response produces a
production having a relatively autonomous status. Then, the
perception of the stimulus produces the response with no intention to
do so (ex. classical conditioning).
• in perception: orienting learned reflexes (shifting the attention
to important stimuli in the environment - ex.: a speaker who
utters your name)
(used by advertising industry)
• in memory: upon encoding a particular stimulus, people may
activate automatically information associated with it in memory
STRATEGIC PROCESSING
EXECUTIVE PRODUCTIONS
• cognitive mechanism that establish and execute acquired goals
• “if-then” productions
• current state of the environment and the cognitive system determine
which executive production is fired in order to pursue the high-level
goals of the cognitive system
• number of executive productions that can be fired at once is
extremely limited (one or possibly few) because of centralized
strategic resource
MECHANISMS OF STRATEGIC PROCESSING
• executive
productions operates on goals, scripts and reminded
episodes to select and coordinate information processing
subsystems during goal-directed behavior
• repeating particular patterns of strategic processing produces new
productions that automate repeated parts of relevant scripts, it
freeing executive productions to work on more subtle aspects of
task or to perform multiple tasks simultaneously
• skills develops as increasing amounts of the processing (that
executive production perform) become automated
PERFORMANCE CHARACTERISTICS OF INFORMATION PROCESSING
1. Limited capacity
information processing resource that limits strategic processing was
theorized as:
- limited processing energy (analogous with an electrical source)
(if one strategic task require all of the available processing
energy, no other task can be accomplish simultaneously)
- a single executive processor that applies and manages
executive productions - it can only execute one strategic task at a
time but can switch quickly from one strategic to another. Limits
on strategic processing arise from the limited ability of the
executive processor to switch between the tasks.
Elementary operations are: compare / choose / repeat /
compute / transform
PERFORMANCE CHARACTERISTICS OF INFORMATION PROCESSING
2. Selectivity
• to achieve a goal a cognitive system must be able to select and
coordinate information processing subsystems (ex. perceptual, motor,
memory), locations in perceptual fields and categories in memory
• corresponds to what theorists often mean by attention
• selection can be specific (ex.particular ear or eye, particular
information in memory)
DISCUSSIONS
Presence of a single executive does not imply that it
controls the entire cognitive system:
The executive may primarily schedule and monitor (it
may also direct processing when goals are new, difficult
and dangerous).
Many basic processes in perception and movement, many
acquired productions which control many automated
skills (ex. driving, typing), lie beyond its scope.
SENSORY MEMORY
•consist in persistence (prolongation) of the sensorial representation of
the stimulus after the stimulus is no longer acting on the receptors
• specific to a certain type of sensation
• format: neuro-physiological codes
• capacity: unlimited but the cognitive system will process further only
the relevant stimuli
• duration: - visual memory 100 ms - auditory memory 200 ms / 2 s
• automatic/ pre-attentional retention of the precategorical information
(information is in an unprocessed form)
LONG TERM MEMORY (LTM)
all the knowledge the cognitive system owns
• unlimited capacity
• duration: whole life of neural system
memory systems in LTM:
• explicit vs. implicit
• semantic vs. episodic (memory of general knowledge about our
environment vs.
memory of personal events)
format (encoding) of the information:
• verbal / analog (images) / semantic (propositional)
retrieval (activation) of knowledge: parallel search process
WORKING MEMORY (WM)
activated part of long-term memory
encoding: verbal, analogical (image), semantic (propositional)
capacity:
•depending on the level of expertise (by chunking); maximum
capacity of attention:
7 +/- 2 chunks
•limited capacity for a certain type of information (auditory, visual,
motor)
•limits for certain types of information are independent one from
another
(ex. if maximum of visual information is in WM this does
not decrease the maximum of auditory information that can be in
WM).
Relation between LTM, WM and attention
when there is a specific goal to be
reached
when there is no specific goal to be
reached
Attention
WM
WM =
Attention
LTM
LTM
ORGANIZATION OF KNOWLEDGE IN
LONG TERM MEMORY
EXPLICIT MEMORY: content is accessible to consciousness and can be tested by
recall and recognition tests
• low level structures
•propositional network (semantic memory)
•semantic network (semantic memory)
• high level structures (complex units of knowledge)
• schema/script/frame/plan (semantic and episodic memory)
• mental models (semantic and episodic memory)
IMPLICIT MEMORY: content is hardly accessible to consciousness and cannot be
tested by recall and recognition tests.
• production systems (cognitive and motor skills, priming, conditioned reflexes)
SEMANTIC NETWORK
- represents semantic contents from well-structured knowledge domains
- knowledge are represented by a network of nodes and relation between nodes
- nodes represents the concepts and the relations between nodes are labeled
- meaning of a concept (or node) is given by the pattern of its relations among which it
participates.
property
breath
animal
type
have skin
bird
property
type
fly
swim
type
have feather
fish
shark
type
predator
eagle
symbol of power
dangerous
chicken
not eatable
SCHEMAS (Rumelhart, 1980)
• represents generic concepts stored in memory underlying objects, situations, events,
sequence of events, actions and sequence of actions
• they are used for a class of stereotypical situation
• they vary the very simple to the very complex
• are organized in a hierarchical fashion:
• variables which have fixe value (the kernel)
• slots with optional values (pheripheral) which can be filled in with particular
instances of the concepts. If the instances are not specified then the slots will have
default values (prototypes)
• can be embedded one in other, e.g.,
• Human body (Head, Trunk, Limbs)
• Head (Face, Ears, Hair)
• relation between the elements are in spatial-temporal contiguity (ex. bread - butter)
• active processing devices (top-down processing) which produce an interpretation of
the world - they adapt reality to knowledge
• they are assumed to be shared across individuals (in a culture?)
SPECIAL TYPES OF SCHEMAS
SCRIPTS (Shank and Abelson,1977)
• schemas for frequently occurring sequences of events in a particular context
• 2 categories of variables: roles (filled by persons) and props (filled by objects)
• includes: - entries condition
- scenes
- results
• scripts are the result of social learning
• maintanance of the scripts is guaranted by a set of social contingencies
FRAMES (Minsky, 1975) - static representation
- schemas that do not possess active processors
- between the elements of the frame are enabling or causal relations
- framework that is adapted to fit reality
• generic frames (class):
ex.: car ( color, brand, driver, engine, transmission, wheels)
• specific frames (instantiated) - in a particular context
ex.: my friend’s car (black, Mercedes, John, 4 wheel drive)
FRAME FOR “CAR”
John
gasoline
type
type
buys
Driver
Fuel
type
Liz
operates
operates
flows
diesel
type
Engine
rotates
Transmission
Fixed values (kernel)
type
rotates
four-cylinder
type
six-cylinder
standard
steel
Specific instances
(default values)
Wheels
alloy
Task: Arrange a project meeting
Plan (meeting (project))
Consult (information source, information token, project meeting)
Identify (information source, information token, project meeting)
Search (information source, information token)
Retrieve (information token, information source)
Store (information token, project meeting, working memory)
Select (media message)
Identify (long-term memory,constrain, project meeting)
Choose (media, constrains )
Send message (meeting (project))
Consult (information token, information source)
Identify (information source, information token, letter)
Search (information source, information token)
Retrieve (information token, information source)
Store (information token, information source)
Represent (information token, message)
Write (information token, message, media)
Compare (message, information token)
Edit (information token, message)
Store (message copy, message file, media)
Execute (transaction requirements, message)
.
frame-based representation: how to create and send message to arrange a project meeting (Keane and Johnson, 1987)
Script: eating at a restaurant
Entry condition
hungry, had money, restaurant open
Roles
diner, waiter, cashier
Props
tables, money, chairs, menu, cutlery, food
Entry scene
Diner enters restaurant.
Waiter seats diner at table.
Waiter places menu on table.
Diner begins to read menu.
Ordering scene Diner selects food from menu.
Diner signals to waiter.
Waiter approaches the table.
Diner orders food.
Waiter leaves.
Eating scene
Waiter brings food to the table.
Waiter leaves.
Diner eats food with cutlery.
Diner finishes eating food.
Leaving scene
Diner signals to waiter.
Waiter approaches table.
Diner asks waiter for bill.
Diner checks bill.
Diner approaches cashier.
Diner gives cashier bill and money.
Cashier checks money.
Diner leaves restaurant.
MENTAL MODELS - dynamic representation
• frames in which the relations between (and attributes of) the elements are
analogous to a physical/organizational/procedural structure in the world
(component parts and relations between these) reflecting the actual state of
affairs in the world.
• parts of it become instantiated being triggered by an input (stimulus,
problem, event)
• can be run producing quasi-continuos simulation of the events (over
space and time) and can explain how events occurred (comprises
explanatory principles)
IMPLICIT MEMORY - TYPES OF KNOWLEDGE
• Cognitive and motor skills
they develop from a script-like representation of knowledge
• Conditioned reflexes
it develops by association of stimulus with response
• Priming
it develops by frequent exposure to a stimulus, modifying the judgement value of
the stimulus
IMPLICIT MEMORY - KNOWLEDGE REPRESENTATION
representation are in the form of production rules; a production is a
condition-action (if-then) pair
IF (condition for triggering) THEN (do these actions)
production rules are organized in production systems
production systems can be general or specific (defining expertise in a
certain domain)
they are triggered automatically by categorization of relevant stimuli by
matching current state of problem-solving (as a pattern in working
memory) or of a stimulus against the conditions of the productions rules
they are hardly accessible to consciousness
when needed, the script (on the basis on which production system has
been developed) can be reconstructed
A variant of the model, Card, Moran, Newell
Some principles
• perceptual processor cycle time varies inversely to
stimulus intensity
• cognitive processor cycle time:
– shorter with more task load
– shorter with more information
– shorter with practice in task domain
A variant of the model, Card, Moran, Newell
Some laws
• Fitts’ law:
time T to move hand to target of size S at distance D:
Tpos = 100[70~120] msec/bit log2 (D/S + .5)
• Power law of practice:
time Tn needed to complete a task at trial n:
Tn = T1 nª , where a = .4[.2~.6]
A variant of the model, Card, Moran, Newell
Example:
simple reaction time
A variant of the model, Card, Moran, Newell
Example:
symbol
detection
Calculation symbol detection
RT
= Tperceptual processor + 2 Tcentral processor + T motor processor
= 100[50~200] + 2*(70[25~170]) + 70[30~100]
= 310[130~640] msec
2.2.2. Mental models
from the point of view of Cognitive Ergonomics
•The functions of mental models in using complex systems
•What type of mental models are needed for using
complex systems
•How to “measure” mental models
The functions of mental models in using complex systems
•Planning
•execution of task delegation
•evaluation
•interpretation
What type of mental models are needed for using complex
systems
Characteristics of mental models
•
•
•
•
Incomplete (users are generaly aware of this)
parsimoneus, just sufficient basic knowledge
can only partly be "run"
unstable, change over time,
– using different systems
– new experiences (even experts)
• vague boudaries (e.g., application / operating system)
• superstition
How to “measure” mental models
Reconstructing semantic networks / frames:
•Pathfinder Algorithm - allows insight in “group
knowledge” and comparison between groups
Hermeneutic interpretation of individual instantiations
•Instantiated frames: Teach-back task “what is …?”
•Instantiated scripts: Teach-back task “how to …?”
DIGITAL INK PEN
Scenario 1
I discover a small garden yesterday, it
was warm and dry and there was
almost nobody around. And it was really
the most relaxing place I had been to in
months…
So I sat down, I did some writing and
drawing and thinking that it was months
that I communicated with my friends’
back home…
I decided that I should send them some
of my garden sketches by e-mail.
So, in order to send
the e-mails:
• I pressed the mode
button on the side
of the DI.
• That changes the
mode from record
my drawing and
hand writing...
• to command mode
which allows me to
tell DI what I wanted
to do next.
• In this case I wanted to send
e-mails to my friends so I
simply write:
the word send followed by
their e-mail addresses.
• The “->” terminates the
command and sends the
message
• Once the send command
has been written and read,
the screen displays the
progress of the command
Now my sketches are on
their computers.
Send to luisa@vu.nl->
Send to pepe@vu.nl->
Scenario 2
I was wandering around the city and I
bumped into Chris. We have only met
once or twice before but we ended up
talking for about one hour about my new
job and his new business in Piltsburg…
We had such a good
time and I really
wanted to get
together again with
him soon to talk
again.
So I took his phone
number and his email address and
stored them in my
pen just by writing
on the back of an
old receipt.
Chris@vu.nl
020- 4567890
Scenario 3
I am getting ready to go on vacation and
I have all this staff that I have to plan:
client meetings,… tons of work to do
and what better place to keep track of
important clients than on a napkin of a
cafeteria.
So I wrote down a
bunch of meetings,
whom I am going to
meet, and when and
what I will be talking
about and I was
done.
10:00 GUI meeting
12:30 lunch with Chris
14:00 meeting with
the design team
I left my list for the
waiter to take away
and…
• all I have to do is
take my DI home
tonight, put it into
the “DI well”
• and then I will
download my list of
things to do into my
computer and I will
be ready to go.
DI well
Scenario 4
With DI I can also
check my e-mail.
• Incoming e-mail is
indicated by a sound
from DI (“ding”).
Ding!
• When I want to read
e-mails I press the
mode button
• and write or speak
the word get
message to see the
e-mail or e-mails I
received

• then the senders of
my e-mails appear
numbered serially.
• To move through the
list I can simply write
or speak the words
up or down
• So if I want to read
the message of
Chris
• I write or speak the
word show message
And the message of
Chris will appear on
the screen.
Then I could write
back to him using
the DI e-mail or fax
command.
So those are things that DI can do. It’s a
new innovative product that works off
patterns that we are already familiar
with.
In fact, when DI is off I can use it as a
regular pen.
DIGITAL INK PEN
We would like to know how you imagine the “Digital Ink
Pen” after seeing the scenarios. Therefore, explain to your
friend “Lucas” what the “Digital Ink Pen” is. You can use
text, drawings, etc.
Your friend “Lucas” wants to send a fax to the administration
of the faculty. Explain to him how to do this using the
Digital Ink Pen.
“Lucas” has five messages in his “Digital Ink Pen” and he is
not interested in the third one. Try to explain to him how
to delete this e-mail from the list of e-mails he has.
How could Lucas dictate (speaking) a letter to Digital Ink and
download it later to his computer?
Teach-back questions:
• “what is”: We would like to know how you imagine the
“Digital Ink Pen” after seeing the scenarios. Therefore,
explain to your friend “Lucas” what the “Digital Ink Pen”
is. You can use text, drawings, etc.
• “how to” 1: Your friend “Lucas” wants to send a fax to
the administration of the faculty. Explain to him how to
do this using the Digital Ink Pen.
• “how to” 2: Y“Lucas” has five messages in his “Digital
Ink Pen” and he is not interested in the third one. Try to
explain to him how to delete this e-mail from the list of emails he has.
• “how to” 3: Y How could Lucas dictate (speaking) a
letter to Digital Ink and download it later to his computer?
Some examples
• of protocols
• and their interpretation
Well
understood,
but rather
conservative
regarding the
dialogue
possibilities
Extrapolation based on intuition and consistency
Reliability in
interpreting
visual
protocols ?
requires
training
User created verbal
command mode
User supposed restrictions in
dialogue - why ??
A wish list of
functionality
and with context
specifically
referred to
The pen “does understand”
2.3.
models in cognitive ergonomics
2.3.1. model of changeability
2.3.2. conceptual models of systems and of interaction
2.3.3. task models
2.3.1. model of changeability
of cognitive functions
Stable,
resistant to change
mainly influenced
by environment
personality
factors:
cognitive
styles:
strategies:
individual
knowledge:
-intelligence
(spatial ability)
-field(in)dependence
-verbalisers / imagers
-impulsivity
-serialists / holists
-schemas
-production rules
adapt to the user
the user may adapt
Individual knowledge
• Long term memory
– explicit
– implicit
• instruction / exploration
• meta-communication by system (user interface):
– implicit (dialogue, lexicon)
– explicit (error messages, help system, yellow stickers)
• content domain
– task knowledge
– system knowledge
strategies
• Domain dependent
• may be learned
• adapting to strategies is often useful: impulsivity - “undo”
an item of an impulsivity test:
“find the picture identical to the
target figure on the top”
Cognitive styles
• Domain independent / generic
• changeable through “education”
• imagers - symbolisers
/ field (in)dependence
two fragments from a field(in)dependency test:
“read the codes of the figures in the matrix as fast as possible”
Personality factors
Stable, needs adaptation to
e.g. spatial ability (intelligence factor)
• low s.a. is a handicap in working with complex system
objects that cannot be presented as a whole
• adaptation can overcome handicaps:
– navigation aids
– abstract overview
2.3.2. conceptual models of systems and of interaction
Norman
Norman (1983):
• target system
• mental model
• conceptual model
Conceptual model:
• perceptual interface & conceptual interface
• the user’s virtual machine
• the model for design & for meta-communication (help)
conceptual models of systems and of interaction
CLG
Moran (1981) Command Language Grammar (CLG)
views on conceptual model:
• linguistic view - modeling and analyzing the interaction
• the psychological view - modeling and analyzing
user knowledge and mental model
• the design view - modeling and analyzing
the interface (the user’s virtual machine)
conceptual models of systems and of interaction
CLG
Levels of analysis
• conceptual component
– task level
– semantic level
• communication component
– syntax level
– interaction level (key stroke level)
• material component
– spatial layout level
– apparatus level
conceptual component
task level
• tasks for the user / tasks delegated to the system
• task procedures, objects in task domain
semantic level
• tasks to delegate:
– system objects with attributes
– system operations
• user specifications, system states and state changes
communication component
syntax level
– dialogue style
– lexicographic structure
– user actions and system acts
– temporal structure
interaction level (key stroke level)
– user key-strokes, mouse handling, voice input, eye gaze
– system signs (beep, cursor indication, cursor
movements, tactile feedback)
material component
The domain of classical ergonomics
spatial layout level
• screen design, window layout
• spatial relation of input and output devices to work
environment (light conditions, sound conditions)
apparatus level
• characteristics of hardware (shape of buttons, key press
characteristics)
2.3.3. task models
Users of complex systems have a task model
related to the use of technology (internal model)
designers of complex systems need a task model
to make design decisions on how to support the user
(external model)
for details, see unit on (task) analysis