The Brain’s Concepts
The Role of the Sensory-Motor System
in Reason and Language
George Lakoff
University of California, Berkeley
(with Vittorio Gallese)
With Thanks to
The Neural Theory of Language Group
International Computer Science Institute
University of California, Berkeley
Especially Jerry Feldman, Srini Narayanan,
Lokendra Shastri, and Nancy Chang.
http://www.icsi.berkeley.edu/NTL
What Concepts Are: Basic Constraints
Concepts are the elements of reason, and
constitute the meanings of words and linguistic expressions.
The Traditional Theory
Reason and language are what distinguish human
beings from other animals.
Concepts therefore use only human-specific brain
mechanisms.
Reason is separate from perception and action, and
does not make direct use of the sensory-motor
system.
Concepts must be “disembodied” in this sense.
We Claim
Human concepts are embodied. Many concepts make
direct use of the sensory-motor capacities of our bodybrain system.
Many of these capacities are also present in nonhuman primates.
One example, the concept of grasping, will be
discussed in detail.
Amodality
The traditional theory implicitly claims that even
action concepts, like grasp, do not make use of
the sensory-motor system. As a concept, even
grasp must be disembodied.
Thus, it is claimed that the concept grasp is
amodal. Since it is a concept, it must be
modality-free, even if it designates an action in a
specific modality.
Concepts Are:
•Universal: they characterize all particular
instances; e.g., the concept of grasping is the
same no matter who the agent is or what the
patient is or how it is done.
•Stable.
•Internally structured.
•Compositional.
•Inferential. They interact to give rise to inferences.
•Relational. They may be related by hyponymy,
antonymy, etc.
•Meaningful.
•Independent of the words used to express them.
Concepts may be either
‘concrete’ (sensory-motor)
or
‘abstract’ (not sensory-motor).
Basic Ideas
•Multimodality — Permits universality
•Functional Clusters — High-level, function as
conceptual units
•Simulation — Necessary for meaningfulness
and contextual inference
•Parameters — Govern simulation, strict
inference, link to language
Multimodality
The action of grasping is not amodal,
but multi-modal in a way that makes
for universality.
Functional Clusters
Functional clusters form high-level units — with the
internal relational structure required by concepts.
There are two types: Local clusters and Network clusters.
Multi-modality is realized in the brain through network
clusters, that is, parallel parietal-premotor networks.
Network clusters are formed by interconnected local
clusters of neurons, like canonical and mirror neurons.
Simulation
To understand the meaning of the concept grasp,
one must at least be able to imagine oneself or
someone else grasping an object.
Imagination is mental simulation, carried out by
the same functional clusters used in acting and
perceiving.
The conceptualization of grasping via simulation
therefore requires the use of the same functional
clusters used in the action and perception of
grasping.
Simulation and Enactment
Visual imagination uses part of the same neural substrate
as vision.
Motor imagination uses part of the same neural substrate
is motor action.
Since you can understand a concrete concept like
grasping only if you can imagine doing it or observing it,
the capacity for mental simulation is taken as the basis for
meaningfulness.
Thus, action and observation provide the basis for
meaningfulness in NTL.
Parameters
All actions, perceptions, and simulations make use of
parameters and their values. Such neural
parameterization is pervasive.
E.g., the action of reaching for an object makes use of
the parameter of direction; the action of grasping an
object makes use of the parameter of force.
The same parameter values that characterize the
internal structure of actions and simulations of actions
also characterize the internal structure of action
concepts.
Structured Neural Computation in NTL
The theory we are outlining uses the computational
modeling mechanisms of the Neural Theory of Language
(NTL).
NTL makes use of structured connectionism (Not PDP
connectionism!).
NTL is ‘localist,’ with functional clusters as units.
Localism allows NTL to characterize precise
computations, as needed in actions and in inferences.
Because it uses functional clusters, NTL is not subject to
the “grandmother cell” objection.
Advantages of Structured Connectionism
Structured connectionism operates on structures of the
sort found in real brains.
From the structured connectionism perspective, the
inferential structure of concepts is a consequence of the
network structure of the brain and its organization in
terms of functional clusters.
Structured Connectionism comes with:
•A dynamic simulation mechanism
parameter values to situations.
that
adapts
•A neural binding theory.
•A
spreading-activation
probabilistic
inference
mechanism that applies to functional clusters.
These jointly allow for the modeling of both sensorymotor simulations and inference.
In NTL, there are fixed structures called schemas.
For example, a schema that structures an action has
an internal structure consisting of Roles, Parameters,
and Phases.
The ideas of Multimodality, Functional Clusters,
Simulation, and Parameters allow us to link
NTL, with structured connectionism, to neuroscience.
The Neuroscience Evidence Shows
In the sensory-motor system, it is possible to
characterize these aspects of concepts:
•Universality
•Semantic Role Structure
•Aspectual Structure (Phases)
•Parameter Structure
The Concept
Of
Grasping
Universality Is Achieved by MultiModality
Multimodal functional clusters for an action like grasping fire
when:
•Grasping is performed, observed, imagined, inferred, or heard;
•The grasping is of any type, done by any agent, on any object,
in any manner, and in any location.
In showing such multimodality for a functional cluster, we are
showing that the functional cluster plays the conceptual role of
universality.
Multi-Modal Integration
The premotor cortex is not a uniform field, but a mosaic
of functionally distinct areas (F1 to F7).
Each of these premotor areas is reciprocally connected
with distinct regions of the posterior parietal cortex.
The premotor cortex is part of a series of parallel
functional network clusters.
Multi-Modal Integration
Cortical premotor areas are endowed with sensory
properties.
They contain neurons that respond to visual,
somatosensory, and auditory stimuli.
Posterior parietal areas, traditionally considered to process
and associate purely sensory information, also play a major
role in motor control.
A New Picture
Rizzolatti et al. 1998
The fronto-parietal networks
Rizzolatti et al. 1998
Area F5
Three classes of neurons:
-Motor General Purpose neurons
-Visuo-Motor neurons:
-Canonical neurons
-Mirror neurons
Area F5
General Purpose Neurons:
General Grasping
General Holding
General Manipulating
General Purpose Neurons in Area F5
A Grasping with the mouth
B Grasping with the cl. hand
C Grasping with the ipsil. hand
(Rizzolatti et al. 1988)
General Purpose Neurons Achieve
Partial Universality: Their firing correlates with a
goal-oriented action of a general type, regardless of
effector or manner.
F5c-PF
Rizzolatti et al. 1998
The F5c-PF circuit
Links premotor area F5c and parietal area PF (or 7b).
Contains mirror neurons.
Mirror neurons discharge when:
Subject (a monkey) performs various types of goalrelated hand actions
and when:
Subject observes another
similar kinds of actions
individual
performing
Area F5c
Convexity region of F5:
Mirror neurons
F5 Mirror Neurons
Gallese and Goldman, TICS 1998
Strictly congruent mirror neurons (~30%)
(Rizzolatti et al. Cog Brain Res 1996)
Category Loosening in Mirror Neurons (~60%)
(Gallese et al. Brain 1996)
PF Mirror Neurons
(Gallese et al. 2002)
A (Full vision)
B (Hidden)
C (Mimicking)
D (HiddenMimicking)
Umiltà et al. Neuron 2001
Like humans, monkeys can also infer the goal of an
action, even when the visual information about it is
incomplete.
F5 Audio-Visual Mirror Neurons
Kohler et al. Science (2002)
Somatotopy of Action Observation
Foot Action
Hand Action
Mouth Action
Buccino et al. Eur J Neurosci 2001
The Mirror System in Humans
BA6
The Simulation Hypothesis
How do mirror neurons work?
By simulation.
When the subject observes another individual doing an action,
the subject is simulating the same action.
Since action and simulation use some of the same neural
substrate, that would explain why the same neurons are firing
during action-observation as during action-execution.
Mirror Neurons Achieve
Partial Universality, since they code
an action regardless of agent, patient,
modality (action/observation/hearing),
manner, location.
Partial Role Structure, since they code
an agent role and a purpose role.
The Agent Role:
In acting, the Subject is an agent of that action.
In observing, the Subject identifies the agent of
the action as having the same role as he has
when he is acting – namely, the agent role.
The Purpose Role: Mirror neurons fire only for
purposeful actions.
Mirror Neurons Achieve
Category tightening and loosening
Limited Prototype Structure
F5ab-AIP
The F5ab-AIP circuit
Links premotor area F5ab and parietal area AIP.
Transforms
intrinsic physical features of objects (e.g., shape, size)
into
hand motor programs required to act on them
Examples:
Manipulate objects, grasp them, hold them, tear them
apart.
Area F5ab
Bank region of F5:
Canonical neurons
F5 Canonical Neurons
Murata et al. J Neurophysiol. 78: 2226-2230, 1997
F5 Canonical Neurons
Murata et al. J Neurophysiol. 78: 2226-2230, 1997
The Simulation Hypothesis
How Do Canonical Neurons Work?
By Simulation.
The sight of a graspable object triggers the simulation of
grasping.
Since action and simulation use some of the same neural
substrate, that would explain why the same neurons are firing
during object-observation as during action-execution.
Canonical Neurons Achieve
Partial Universality, since they code
an action regardless of patient,
manner, and location.
Partial Role Structure, since they code
a patient role and a purpose role.
The Patient Role: Canonical neurons fire
in the presence of an appropriate patient
for a given action.
The Purpose Role: Canonical neurons fire
only for purposeful actions.
F4-VIP
The F4-VIP Network Custer
The F4-VIP Circuit
Links premotor area F4 and parietal area VIP.
Transforms
the spatial position of objects in peri-personal space
into
motor programs for interacting with those objects.
Examples:
Reaching for the objects, or moving away from them
with various parts of your body such as the arm or head.
Area F4
Arm reaching
Head turning
Somato-Centered Bimodal RFs in area F4
(Fogassi et al. 1996)
(Graziano et al. 1999)
Somato-Centered Bimodal RFs in area VIP
(Colby and Goldberg 1999)
Somato-Centered Bimodal RFs in area F4
(Fogassi et al. J Neurophysiol 1996)
The Simulation Hypothesis
How Do Action-Location Neurons Work?
By Simulation.
The sight or sound of a possible target location in
peri-personal space triggers the simulation of appropriate
actions toward that location.
Since action and simulation use some of the same neural
substrate, that would explain why the same neurons are
firing during location-perception as during actionexecution.
Action-Location Neurons Achieve
Partial Universality, since they code
an action regardless of patient.
Partial Role Structure, since they code
Location.
Evidence in Humans for Mirror,
Canonical, and Action-Location
Neurons
Mirror: Fadiga et al. 1995; Grafton et al. 1996;
Rizzolatti et al. 1996; Cochin et al. 1998;
Decety et al. 1997; Decety and Grèzes 1999;
Hari et al. 1999; Iacoboni et al. 1999;
Buccino et al. 2001.
Canonical: Perani et al. 1995; Martin et al.
1996; Grafton et al. 1996; Chao and Martin 2000.
Action-Location: Bremmer, et al., 2001.
MULTI-MODAL INTEGRATION
The premotor and parietal areas, rather than having
separate and independent functions, are neurally integrated
not only to control action, but also to serve the function of
constructing an integrated representation of:
(a) Actions, together with
(b) objects acted on, and
(c) locations toward which actions are directed.
In these circuits sensory inputs are transformed in order to
accomplish not only motor but also cognitive tasks, such as
space perception and action understanding.
Phases
Area F5 contains clusters of neurons that control distinct
phases of grasping: opening fingers, closing fingers.
Jeannerod, et al., 1995; Rizzolatti, et al., 2001.
Summary
Jointly, these functional clusters in the sensory-motor system
characterize the following conceptual properties of grasping:
•Stability
•Universality: Covers all particulars
•Internal Structure:
Semantic Roles
Phases (Aspectual Structure)
•Meaningfulness
•Independence of linguistic expression
Summary
In NTL, structured connectionist mechanisms apply to
units modeling functional clusters.
Compositionality is modeled via neural binding.
Inference is modeled via structured connectionist
mechanisms for: binding, spreading activation inference, and
simulation.
Conclusion 1
The Sensory-Motor System Is Sufficient
For at least one concept, grasp, functional clusters, as
characterized in the sensory-motor system and as modeled
using structured connectionist binding and inference
mechanisms, have all the necessary conceptual properties.
Conclusion 2
The Neural Version of Ockham’s Razor
Under the traditional theory, action concepts have to be
disembodied, that is, to be characterized neurally entirely
outside the sensory motor system.
If true, that would duplicate all the apparatus for
characterizing conceptual properties that we have
discussed. Unnecessary duplication of this sort is highly
unlikely in a brain that works by neural optimization.
How does NTL fit the Neuroscience?
Actions in NTL
For each type of action there is a Fixed Schema, consisting
of types of fixed parameters; for example:
•Role Parameters, like Agent and Patient
•Phase Parameters, like Initial and Final State
•Manner Parameters, like Degree of Force and Direction
Grasp Schema
Roles: Action, Agent, Patient, Location
Manners: Force, Type of Grip; Effector Used
Phases:
Initial State:: Object Location: Within Peri-personal Space
Starting Transition:: Reaching, with Direction: Toward Object
Location; Opening Effector
Central Transition:: Closing Effector, with Force: A function
of Fragility and Mass of Patient
Goal Condition:: Effector Encloses Object, with Manner: (a
grip determined by parameter values and situational
conditions)
Final State:: Agent In-Control-of Object
Fitting The Grasp Schema to the
Neuroscience of Grasping
A Fixed Schema Is a Network of Functional Clusters
Each Parameter Is a Functional Cluster of neurons
Each Parameter value Is either
A firing pattern over a functional cluster, or
A neural binding to another functional cluster,
as when the role Agent is bound to a particular
actor in context.
An Executing Schema (X-schema) Is a neural circuit
connecting the parameters of the fixed schema so that they
can dynamically coordinate firing over time and
adapt their values over time to inputs from context.
Note!
The same neurons that define the fixed schema are
the neurons subject to dynamic, contextually adjusted
activation by the executing schema during
performance, observation, and imagination.
Schemas are not like logical conditions. They run
bodies — as well as they can, in real time adjusting to
real conditions.
Other Differences From
Traditional Accounts of Concepts
•Not Necessary and Sufficient Conditions
•Not Representational
•Not Symbolic
Not Necessary and Sufficient Conditions
•The activation of functional clusters is not all-or none; there
are degrees.
•There are variations on schemas, as when certain phases
are optionally left out.
•There are extensions of schemas; for example, extensions
from the prototoype and metaphorical extensions.
Not Representational
We conceptualize the world on the basis of the way we
experience it; e.g., color is not in the world, nor is heat.
Since our experience is a function of our bodies, brains, and
our physical and social environment, so are our concepts.
Since our experience comes through our physical nature —
our bodies, brains, and physical functioning — so our concepts
are physical in nature.
They are physical brain structures that, when activated, result
in creative understandings shaped by the peculiar character of
our bodies, brains, and lived experiences.
Not Symbolic
.Note that we have written down symbols (e.g., Final State)
as our notation for functional clusters.
This does NOT mean that we take functional clusters
themselves to be symbolic. We only use symbols because we
have to write things down.
The symbols are only our names for functional clusters, which,
as we have seen, are made of neurons, though they function
— from a computational modeling point of view — as units.
Language is Multi-Modal, Not Modular
Concepts form the most interesting part of language, the
meaningful part.
Many concepts, which are part of language, are inherently
multi-modal, exploiting the pre-existing multi-modal
character of the sensory-motor system.
It follows that there is no single “module” for language —
and that human language makes use of mechanisms
present in nonhuman primates.
What About Abstract Concepts?
Abstract Concepts
Not all concepts are about physical things or what we do
with our bodies.
Some are about emotions, like love.
Others are even less concrete, like freedom.
Conceptual Metaphor Provides
Embodied Reasoning For Abstract
Concepts
Virtually all abstract concepts (if not all) have conventional
metaphorical conceptualizations — normal everyday ways
of using concrete concepts to reason systematically about
abstract concepts.
Most abstract reasoning makes use of embodied
reasoning via metaphorical mappings from concrete to
abstract domains
What Are Conceptual Metaphors?
In NTL, conceptual metaphors are structured
connectionist “maps” — circuits linking concrete source
domains to abstract target domains.
In the fit of NTL to Neuroscience, such metaphorical
maps would be neural circuits in the brain
linking sensory-motor regions to other regions.
We claim therefore that, in such cases, the sensorymotor system is directly engaged in abstract reasoning.
Metaphorical Grasping
There is a conceptual metaphor, Understanding Is Grasping,
according to which one can grasp ideas.
Reasoning patterns about physical grasping can be mapped by
conceptual metaphor onto abstract
reasoning patterns.
One can begin to grasp an idea, but not quite get a hold of it.
If you fail to grasp an idea, it can go right by you — or over your
head!
If you grasp it, you can turn it over in your mind.
You can’t hold onto an idea before having grasped it.
The Sensory-Motor System
in Abstract Reasoning
We have argued that the physical Grasping Schema is
realized in the sensory-motor system, and that its
inferences are carried out imaginatively in sensory-motor
simulation.
At least some of these inference patterns are used
metaphorically to do abstract reasoning about
understanding.
If our analysis is correct, then the sensory-motor system is
directly engaged in abstract reasoning.
Cogs
The exploitation of “general”
sensory-motor mechanisms
for
abstract reasoning
and
grammar
Premotor Versus Motor Cortex
Whenever we perform a complex motor movement, such as
picking up a glass and taking a drink, at least two distinct parts of
the brain are activated:
The motor cortex, where there are neural ensembles that control
“motor synergies” — relatively simple actions like opening or
closing the hand, flexing or extending the elbow, turning the wrist,
and so on.
Complex motor schemas, however, are carried out by neural
circuitry in the pre-motor cortex, circuitry connected via neural
bindings to the appropriate synergies in the motor cortex.
In picking up a glass and taking a drink, both pre-motor cortex and
motor cortex are activated, as are binding connections between
them.
The Controller X-Schema
In modeling complex premotor action schemas, Narayanan
made a remarkable discovery.
All complex premotor schemas are compositions of a single
type of structure.
He then showed that the same neural computational
structure, when disengaged from specific motor actions,
can characterize aspect (that is, event structure) in the
world’s languages.
When dynamically active, this structure can compute the
logic of aspect.
Narayanan called this structure the “Controller X-schema.”
The Structure of the Controller X-Schema
•Initial State
•Starting Phase Transition
•Precentral State
•Central Phase Transition (either instantaneous,
prolonged, or ongoing)
•Postcentral State*
•Ending Phase Transition
•Final State
Postcentral Options:
*A check to see if a goal state has been achieved
*An option to iterate or continue the main process
*An option to stop/resume
-Narayanan, 1997
The Controller X-Schema as a Computational
Model
The Controller X-Schema is implemented computationally
using Petri Nets that have been greatly revised and
extended to closely approximate neural systems.
Narayanan has developed his program to be a general
mechanism for imaginative simulation.
The computational model is intended to be mapped onto
neural structures so that we can speak of neural Controller
X-Schemas with the following properties.
The Properties of
A Neural Controller X-Schema
•It is a neural structure that is “general” in the sense that it
can be bound via connections to different specific sensorymotor structures elsewhere in the brain.
•When those connections are deactivated, it can be
connected to other regions of the brain and perform abstract
reasoning.
•In its reasoning mode, it characterizes the semantics of a
portion of grammar (e.g., aspect and its logic).
•The inference patterns it characterizes are “general,” in
that they can apply to a wide range of specific concepts.
I will call any neural structure with such properties a “Cog.”
Other Cogs
Other examples of Cogs include primitive image-schemas —
e.g., Containers, Source-Path-Goal, Contact, Rotation, FrontBack, Up-Down— as well as Talmy’s force dynamic schemas,
enumeration schemas (used in subitizing), and so on.
All of these can be bound to a wide range of specific sensorymotor details, can be used in reasoning, and can characterize
the meanings of grammatical constructions.
Other Uses of Cogs
Linking metaphors that join different mathematical
domains are Cog-to-Cog mappings.
e.g., Numbers Are Points on a Line
Cogs characterize form in art.
Dissociative learning is the inhibition of connections
between Cogs and specific details.
The Sensory-Motor Nature of Cogs
The primary function of Cogs is a sensory-motor function.
Both evolutionarily and developmentally, Cogs first function
to structure our embodied sensory-motor interactions in the
world.
That function is not lost. Cogs continue in their sensory-motor
function.
The sensory-motor characteristics of Cogs are exploited in
reason, language, mathematics, and art — the highest of
human cognitive functions.
All of these make direct use of the sensory-motor neural
substrate!