Lecture Overview
•
•
•
•
Regier System: Limitations
Image Schemas: Recap
Force Dynamic Schemas: Recap
Sensory-Motor Schemas
– Evidence in Primates
– Evidence in Humans
•
•
•
•
Do motor schemas play a role in language?
A Computational Model of Motor Schemas
Learning Hand Action terms (Bailey)
Cultural Schemas and frames
Limitations
• Scale
• Uniqueness/Plausibility
• Grammar
• Abstract Concepts
• Inference
• Representation
on construction: Grammar and Inference
SCHEMA TR-LM
ROLES
tr: trajector
lm: landmark
SCHEMA Contact
ROLES
r1: bounded-region
r2: region
SCHEMA Support
ROLES
supporter: bounded-region
supportee: bounded-region
One sense of on
CONSTRUCTION On
FORM: Word
self.f.orth  “on”
MEANING
EVOKES TR-LM as trlm
EVOKES Contact as c
EVOKES Support as s
trlm.tr  c.r1
trlm.lm  c.r2
trlm.tr  s.supportee
trlm.lm  s.supporter
Image Schemas in Language
• You wake out of a deep sleep and peer out from
beneath the covers into your room. You gradually
emerge out of your stupor, pull yourself out from under
the covers, climb into your robe, stretch out your limbs,
and walk in a daze out of the bedroom and into the
bathroom. You look in the mirror and see your face
staring out at you. You reach into the medicine cabinet,
take out the toothpaste, squeeze out some toothpaste,
put the toothbrush into your mouth, brush your teeth in
a hurry, and rinse out your mouth. (Johnson 1987)
Image Schemas in Language
• You wake out of a deep sleep and peer out from
beneath the covers into your room. You gradually
emerge out of your stupor, pull yourself out from under
the covers, climb into your robe, stretch out your limbs,
and walk in a daze out of the bedroom and into the
bathroom. You look in the mirror and see your face
staring out at you. You reach into the medicine cabinet,
take out the toothpaste, squeeze out some toothpaste,
put the toothbrush into your mouth, brush your teeth in
a hurry, and rinse out your mouth. (Johnson 1987)
Fictive Motion and Image Schemas
(Talmy, Teenie Matlock)
• This fence goes from the plateau to the valley
• I looked out past the steeple
• The vacuum cleaner is down around behind the
clothes-hamper
• The scenery rushed past us as we drove along
Lecture Overview
•
•
•
•
Regier System: Limitations
Image Schemas: Recap
Force Dynamic Schemas: Recap
Sensory-Motor Schemas
– Evidence in Primates
– Evidence in Humans
•
•
•
•
Do motor schemas play a role in language?
A Computational Model of Motor Schemas
Learning Hand Action terms (Bailey)
Cultural Schemas and frames
Force Dynamics, modals and causatives
• A gust of wind made the pages of my
book turn.
• The appearance of the headmaster
made the pupils calm down.
• The breaking of the dam let the water
flow from the storage lake.
• The abating of the wind let the sailboat
slow down.
Schematic Representation
(Talmy)
FD Patterns
• A gust of wind made the pages of
my book turn.
• The appearance of the headmaster
made the pupils calm down.
• The breaking of the dam let the
water flow from the storage lake.
• The abating of the wind let the
sailboat slow down.
Semantic field
Force-dynamics
representation
Physical
The ball kept rolling
along the green
Physical/psychological
John can't go out of the
house
Intra-psychological
He refrained from
closing the door
Intra-psychological
(lexicalized)
She's civil to him
Socio-psychological
She gets to go to the
park
Closed Class vs. Open Class terms
• Image Schematic and Force Dynamic Patterns are
expressed by closed class terms in language
– Prepositions (in, on, into, out)
– Modals and causatives (make, let, might, prevent)
• How about open class terms?
– Verbs and Event descriptions –
• Motor Schemas - Embodied
– Is there evidence for motor schemas and if so are they
used in language?
• Frames – Composed from Image and motor schemas Cultural
Verb + Image Schemas in Language
• You wake out of a deep sleep and peer out from
beneath the covers into your room. You gradually
emerge out of your stupor, pull yourself out from under
the covers, climb into your robe, stretch out your limbs,
and walk in a daze out of the bedroom and into the
bathroom. You look in the mirror and see your face
staring out at you. You reach into the medicine cabinet,
take out the toothpaste, squeeze out some toothpaste,
put the toothbrush into your mouth, brush your teeth in
a hurry, and rinse out your mouth. (Johnson 1987)
Lecture Overview
•
•
•
•
Regier System: Limitations
Image Schemas: Recap
Force Dynamic Schemas: Recap
Sensory-Motor Schemas
– Evidence in Primates
– Evidence in Humans
•
•
•
•
Do motor schemas play a role in language?
A Computational Model of Motor Schemas
Learning Hand Action terms (Bailey)
Cultural Schemas and frames
Coordination of Pattern Generators
Coordination
• PATTERN GENERATORS, separate neural networks that
control each limb, can interact in different ways to
produce various gaits.
– In ambling (top) the animal must move the fore and hind
leg of one flank in parallel.
– Trotting (middle) requires movement of diagonal limbs
(front right and back left, or front left and back right) in
unison.
– Galloping (bottom) involves the forelegs, and then the
hind legs, acting together
Sensory-Motor Schemas
•A sensory (perceptual) schema determines whether a given
situation is present in the environment.
– Object Detection
– Spatial relation recognition
• Execution of current plans is made up of motor schemas which are
akin to control systems but distinguished by the fact that they can
be combined to form coordinated control programs
• Sensory and Motor Schemas are closely coupled circuits sensorymotor schemas.
Preshaping While Reaching to Grasp
Lecture Overview
•
•
•
•
Regier System: Limitations
Image Schemas: Recap
Force Dynamic Schemas: Recap
Sensory-Motor Schemas
– Evidence in Primates
– Evidence in Humans
•
•
•
•
Do motor schemas play a role in language?
A Computational Model of Motor Schemas
Learning Hand Action terms (Bailey)
Cultural Schemas and frames
The neural theory
Human concepts are embodied. Many concepts
make direct use of the sensory-motor
capacities of our body-brain system.
• Many of these capacities are also present in
non-human primates.
• Let us look at concepts that make use of our
sensory-motor capacities, ex. Grasp.
A New Picture
Rizzolatti et al. 1998
The fronto-parietal networks
3 circuits for grasp
Neural Tracers
F5-AIP
F4-VIP
F5-PF
Rizzolatti et al. 1998
Area F5
General Purpose Neurons:
General Grasping
General Holding
General Manipulating
General Purpose Neurons in Area F5
A Grasping with the mouth
B Grasping with the cl. hand
C Grasping with the ipsil. hand
(Rizzolatti et al. 1988)
General Purpose Neurons Achieve
Partial Universality: Their firing correlates with a
goal-oriented action of a general type, regardless of
effector or manner.
Area F5c
Convexity region of F5:
Mirror neurons
F5c-PF
Rizzolatti et al. 1998
Strictly congruent mirror neurons (~30%)
Observed Action
Executed Action
Executed Action
(Rizzolatti et al. Cog Brain Res 1996)
Category Loosening in Mirror Neurons (~60%)
(Gallese et al. Brain 1996)
A [C] is Observe (Execute) Precision Grip (Prototype)
B [D] is Observe (Execute) Whole Hand Pre-hension
The F5c-PF circuit
Links premotor area F5c and parietal area PF (or 7b).
Contains mirror neurons.
Mirror neurons discharge when:
Subject (a monkey) performs various types of goalrelated hand actions
and when:
Subject observes another
similar kinds of actions
individual
performing
Phases
Area F5 contains clusters of neurons that control distinct
phases of grasping: opening fingers, closing fingers.
Jeannerod, et al., 1995; Rizzolatti, et al., 2001.
Mirror Neurons Achieve
Partial Universality, since they code
an action regardless of agent, patient,
modality (action/observation/hearing),
manner, location.
Partial Role Structure, since they code
an agent role and a purpose role.
The Agent Role:
In acting, the Subject is an agent of that action.
In observing, the Subject identifies the agent of
the action as having the same role as he has
when he is acting – namely, the agent role.
The Purpose Role: Mirror neurons fire only for
purposeful actions.
Mirror Neurons Achieve
Category tightening and loosening
The F4-VIP circuit
The F4-VIP Circuit
Links premotor area F4 and parietal area VIP.
Transforms
the spatial position of objects in peri-personal space
into
motor programs for interacting with those objects.
Examples:
Reaching for the objects, or moving away from them
with various parts of your body such as the arm or head.
Area F4
Arm reaching
Head turning
Somato-Centered Bimodal RFs in area F4
(Fogassi et al. 1996)
(Graziano et al. 1999)
Somato-Centered Bimodal RFs in area VIP
(Colby and Goldberg 1999)
AIP and F5 (Grasping) in Monkey
F5 - grasp
commands in
premotor cortex
Giacomo Rizzolatti
AIP - grasp
affordances
in parietal cortex
Hideo Sakata
Size Specificity in a Single AIP Cell
•This cell is selective toward small objects, somewhat
independent of object type ( Hideo Sakata)
•Note: Some cells show size specificity; others do not.
Summary of Fronto-Parietal Circuits
Motor-Premotor/Parietal Circuits
PMv (F5ab) – AIP Circuit
“grasp” neurons – fire in relation to movements of hand
prehension necessary to grasp object
F4 (PMC) (behind arcuate) – VIP Circuit
transforming peri-personal space coordinates so can move toward
objects
PMv (F5c) – PF Circuit F5c
different mirror circuits for grasping, placing or manipulating
object
Together suggest cognitive representation of the
grasp, active in action imitation and action
recognition
Evidence in Humans for Mirror,
General Purpose, and Action-Location
Neurons
Mirror: Fadiga et al. 1995; Grafton et al. 1996;
Rizzolatti et al. 1996; Cochin et al. 1998;
Decety et al. 1997; Decety and Grèzes 1999;
Hari et al. 1999; Iacoboni et al. 1999;
Buccino et al. 2001, Graziano 2005
General Purpose: Perani et al. 1995; Martin et al.
1996; Grafton et al. 1996; Chao and Martin 2000.
Action-Location: Bremmer, et al., 2001.
Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
The Mirror System in Humans
BA6
MULTI-MODAL INTEGRATION
The premotor and parietal areas, rather than having
separate and independent functions, are neurally integrated
not only to control action, but also to serve the function of
constructing an integrated representation of:
(a) Actions, together with
(b) objects acted on, and
(c) locations toward which actions are directed.
In these circuits sensory inputs are transformed in order to
accomplish not only motor but also cognitive tasks, such as
space perception and action understanding.
FARS (Fagg-Arbib-Rizzolatti-Sakata)
Model
AIP extracts the set of
affordances for an attended
object.These affordances
highlight the features of the
object relevant to physical
interaction with it.
AIP
AIP
Dorsal
Stream:
dorsal/ventral
Affordances
streams
Ways to grab
this “thing”
Task Constraints
T(F6)
ask Constraints (F6)
Working Memory
Working M emory (46)
(46?)
Instruction Stimuli
Instruction Stimuli (F2)
(F2)
F5
Ventral
Stream:
Recognition
F5
“It’s a mug”
PFC
Itti: CS564 - Brain Theory and Artificial Intelligence. FARS Model
IT
Hypothetical coordinated control program for reaching and
grasping
recognition
criteria
Perceptual
Schemas
activation of
visual search
visual
input
Visual
Location
target
location
Size
Recognition
size
activation
of reaching
Motor
Schemas
visual
input
Orientation
Recognition
orientation
visual and
kinesthetic input
Fast Phase
Movement
Hand
Preshape
Slow Phase
Movement
Hand Reaching
visual
input
visual ,
kinesthetic, and
tactile input
Hand
Rotation
Actual
Grasp
Grasping
Dashed lines — activation signals; solid lines — transfer of data.
(Adapted from Arbib 2004)
Modeling Motor Schemas
• Relevant requirements (Stromberg, Latash, Kandel, Arbib,
Jeannerod, Rizzolatti)
– Should model coordinated, distributed, parameterized control
programs required for motor action and perception.
– Should be an active structure.
– Should be able to model concurrent actions and interrupts.
– Should model hierarchical control (higher level motor centers
to muscle extensor/flexors.
• Computational model called x-schemas
(http://www.icsi.berkeley.edu/NTL)