CSCTR – Session 8
Dana Retová



Rejects the standard view that amodal symbols
represent knowledge in semantic memory
Cognition shares the same mechanisms with
perception, action and introspection
Simulation
◦ A core form of computation in the brain
◦ Reenactment of perceptual, motor and introspective states
acquired during experience
◦ As experience occurs, the brain captures the states across
modalities and integrates them with a multimodal
representation stored in memory

Modal representation and imagery
representing knowledge
◦ Epicurus, Kant, Reid

Behaviorists
◦ Imagery not sufficiently scientific

Cognitivists
◦ Amodal representation (feature lists, semantic
networks, frames)
 Elegant and powerful formalisms for representing
knowledge
 Could be implemented in AI


No evidence supports the presence of amodal
symbols in cognition
Grounding problem
◦ Traditional theories fail to explain how cognition
interfaces with perception and action

Problem where the brain stores amodal
symbols
◦ How is it consistent with neural principles of
computation?



Simulation
Situated action
Bodily states
◦ Modal representations are central to knowledge

Cognitive Linguistics Theories
◦ Lakoff & Johnson (1980, 1999)
 Abstract concepts are grounded metaphorically in
embodied and situated knowledge

Theories of situated action
◦ Gibson (1979)
 Role of environment in shaping cognitive mechanisms
 Coupling of perception and action during goal achievement
 Social interaction
◦ Research in robotics
◦ Dynamic systems as preferred architecture
 Fixed representations do not exist

Memory theories
◦ Glenberg (1997)
 Memory is not just passive storage of information
 Perception of relevant objects triggers affordances for
action stored in memory
 Reasoning about future actions relies on remembering
affordances while suppressing perception of the
environment

Social simulation theories
◦ How we represent the mental states of other people
 We use simulations of our own minds
 To feel someone else’s pain we simulate our own pain
 Mirror neurons
 Empathy, imitation, social coordination

Perceptual Symbol Systems
◦ Synthetic approach
 Implements standard symbolic functions
 Type-token binding, inference, productivity, recursion, propositions
◦ A single multimodal representation system in the brain supports
diverse forms of simulation across different cognitive processes
 High-level perception
 Working memory
 long-term memory
 conceptual knowledge
 Convergence zone architecture (Damasio 1989, Simmons & Barsalou
2003)
 Single representation system controlled by multiple simulation mechanisms





ensemble of neurons within which many
feedforward/feedback loops make contact.
It 1) receives forward projections from
cortical regions located in the connectional
level immediately below
2) Sends reciprocal backward projections to
the originating cortices
3) Sends forward projections to cortical
regions in the next connectional level; and
4) Receives projections from heterarchically
placed cortices and from subcortical nuclei in
thalamus, basal forebrain, and brainstem.




Perceptual Inference
Perception-action coordination
Perception of space
Memory
◦ Implicit memory
◦ Explicit memory
◦ Working memory

Conceptual processing

Vision and motion
◦ Goldstone (1995)
 Association between shape and color
◦ Hansen et al. (2006)
 Object’s natural color distort achromatic perception of
the object toward the opponent color
◦ Motion (Freyd 1987, Shiffrar & Freyd 1990,1993)
 Subjects simulate the visual trajectory beyond its actual
trajectory
 Also during apparent motion, simulation of possible
action shapes the perception of motion
◦ Speech (Warren 1970) :
http://www.youtube.com/watch?v=UlJs24j3i8E
 Lexical knowledge produces simulation in speech
perception – missing phoneme simulation

Simulations of potential actions
◦ Viewing an object grasped with a precision or power grip (grape
vs. hammer) produces a simulation of the appropriate action
(Tucker & Ellis 1998)
 This is affected by
 object’s orientation (Symes et al. 2007)
 Size (Glover et al. 2004)
◦ Simulations of both grasping and functional actions (Bub et al
2007)
◦ Also name triggers simulation (Tucker & Ellis 2004)
◦ Hearing a word activates the articulatory action associated with
producing it (Pulvermuller 2006)
◦ Perceived effort affects visual perception (Proffitt 2006)
 Being tired from a run makes a hill look steeper
 Carrying a heavy pack makes a path look longer

Motor simulations
◦ Motor system constructs a feed-forward simulation
of the action to guide and correct it (Grush 2004,
Wolpert et al. 1999)
◦ Generating visual inferences about the anticipated
actions of perceived agents (Wilson & Knoblich
2005)

The perception of space is shaped by the
body and it’s relation to the environment
◦ Locating objects has various difficulty along
different axes
 Vertical
 easiest
 Front-back
 Left-right
 Most difficult – bodily cues are lacking
◦ Perception of near space extends with arm length
(Longo & Laurenco 2007)




Results form simulation of perceptual
memories
Repetition priming is strongest when the
modalities of the memory and stimulus match
(e.g. auditory) (Kirsner et al., 1989)
Repetition priming is strongest when
perceptual details of the memory and
stimulus match (e.g. orientation, size,…)
(Jacoby&Hayman, 1987)
Imagining produces repetition priming similar
to actual perception (Roediger&Blaxton,
1987)

Multimodal simulations of previous episodes
◦ Important for constructing future events

The retrieval of a word stimulates the modal
operations performed at encoding (Wheeler et
al. 2000)
◦ Visual areas become active during retrieval
following visual study while auditory areas become
active following auditory study

Greater activation in modal areas when
remembering something that really occurred
than false memories (Slotnick & Schacter
2004)

Absent stimulus is stored in working memory (Levy
& GoldmanRakic 2000)
◦ To maintain working memory, neurons in the frontal lobes
maintain a simulation of the absent stimulus in the modal
system that processed it originally.
 Some frontal regions maintain working memories of
objects, other spatial locations, motion, textures, etc.
 They are highly selective for the specific features

Visual imagery in working memory simulates visual
processing (Finke 1989, Kosslyn 1980,…)
◦ Analogously, motor imagery, auditory imagery, etc.
 Mental rotation of visual objects -> motor simulations of
making them turn (Richter et al. 2000)

Behavioral evidence
◦ When asked whether an property belongs to an objects
subjects simulate properties to verify them (Solomon &
Barsalou 2004)

Lesion evidence
◦ Lesions in one modality – losing categories that rely on it
for processing (Damasio 1994, …)
 E.g. damage to visual areas – losing of ability to
categorize animals (visual processing is dominant)
 Damage to motor areas – categorization of tools

Neuroimaging evidence (Martin 2001, 2007)
◦ When processing conceptual knowledge, brain areas
representing properties are active



Perceptual simulation
Motor simulation
Affective simulation

Situation models
◦ Evidence of modal representations in language
comprehension
 Spatial representation (Bower & Morrow 1990)
 People confused pictures with text (Intraub & Hoffman
1992)
 Replacing words with pictures did not disrupt sentence
processing (Potter 1986)

Subjects read a sentence and then processed
a picture that either matched or mismatched
something implied by the sentence
◦ “The ranger saw the eagle in the sky”
◦ Picture of an eagle – wings outstretched or folded

Visual irrelevant information interferes with
spatial inferences (Fincher-Keifer 2001)





Verbs for head, arm and leg actions produce
head, arm and leg simulation in the
respective areas of the motor system
(Pulvermuller 2005)
When action to make a response is consistent
with text meaning, the response is quicker
(Glenberg & Kaschak 2003)
Subjects simulate corresponding motion
through space (Richardson et al. 2003)
Positive/negative valence (Meier & Robinson)
High/low power (Schubert 2005)



Subjects’ faces configured according to
sentences with emotional content (Havas
2007)
When facial emotion matched the content
comprehension was better
Gesture
◦ Producing gestures helps speakers retrieve words
whose meaning are related to the gestures (Krauss
1998)
◦ Also help listeners comprehend what speaker says
◦ Children can gesture before speaking

Physical reasoning
◦ Gear, pulleys
◦ Driven by spatial simulation
◦ Sketchy, not holistic and detailed

Abstract reasoning
◦ Content effects
◦ Reasoning about time using space domain metaphors

Embodiment effects
◦ Activating elderly stereotype causes people to walk
slowly and to perform lexical decision slowly
(Dijksterhuis & Bargh 2001)
◦ Engaging the smiling musculature produces
positive affect (Strack et al. 1988)

Social mirroring
◦ Individual differences in the ability to simulate
other people’s mental states correlate with rated
empathy (Jackson et al. 2005)

Development
◦ Mirroring, object permanence…