Understanding
• I Hear and I Forget
• I See and I Remember
• I Do and I Understand
Attributed to Confucius, ~500 BCE
H
E
G
F
D
A
B
C
An ambiguous image; A or G can be closer to you
How could a mass of chemical cells produce
language and thought?
Will computers think and speak?
How much can we know about our own experience?
How do we learn new concepts?
Does our language determine how we think?
Is language Innate?
How do children learn grammar?
How did languages evolve?
Why do we experience everything the way that we do?
Constrained Best Fit in Nature
inanimate
physics
chemistry
biology
vision
language
animate
lowest energy
state
molecular
minima
fitness, MEU
Neuroeconomics
threats,
friends
errors,
NTL
Embodiment
Of all of these fields, the learning of
languages would be the most impressive,
since it is the most human of these
activities. This field, however, seems to
depend rather too much on the sense
organs and locomotion to be feasible.
Alan Turing (Intelligent Machines,1948)
Artist’s Rendition of a Typical Cell Membrane
Pre-Natal Tuning: Internally
generated tuning signals
• But in the womb, what provides the feedback to establish which
neural circuits are the right ones to strengthen?
– Not a problem for motor circuits - the feedback and control networks for
basic physical actions can be refined as the infant moves its limbs and
indeed, this is what happens.
– But there is no vision in the womb. Recent research shows that
systematic moving patterns of activity are spontaneously generated prenatally in the retina. A predictable pattern, changing over time, provides
excellent training data for tuning the connections between visual maps.
• The pre-natal development of the auditory system is also interesting
and is directly relevant to our story.
– Research indicates that infants, immediately after birth, preferentially
recognize the sounds of their native language over others. The
assumption is that similar activity-dependent tuning mechanisms work
with speech signals perceived in the womb.
Post-natal environmental tuning
• The pre-natal tuning of neural connections using simulated
activity can work quite well –
– a newborn colt or calf is essentially functional at birth.
– This is necessary because the herd is always on the move.
– Many animals, including people, do much of their development after
birth and activity-dependent mechanisms can exploit experience in
the real world.
• In fact, such experience is absolutely necessary for normal
development.
• As we saw, early experiments with kittens showed that there
are fairly short critical periods during which animals deprived
of visual input could lose forever their ability to see motion,
vertical lines, etc.
– For a similar reason, if a human child has one weak eye, the doctor
will sometimes place a patch over the stronger one, forcing the
weaker eye to gain experience.
Tinbergen’s Four Questions
How does it work?
How does it improve fitness?
How does it develop and adapt?
How did it evolve?
5 levels of Neural Theory of
Language
Spatial
Relation
Motor
Control
Metaphor Grammar
Cognition and Language
abstraction
Computation
Structured Connectionism
Neural Net
Triangle Nodes
SHRUTI
Computational Neurobiology
Biology
Neural
Development
Quiz
Midterm
Finals
Brains ~ Computers
• 1000 operations/sec
• 100,000,000,000
units
• 10,000 connections/
• graded, stochastic
• embodied
• fault tolerant
• evolves, learns
• 1,000,000,000
ops/sec
• 1-100 processors
• ~ 4 connections
• binary, deterministic
• abstract
• crashes
• designed,
programmed
Freud’s Original Connectionist Model
Connectionist Model of
Word Recognition (Rumelhart
and McClelland)
Interactive Activation Reading Model
Modeling lexical access errors
•
•
•
•
Semantic error
Formal error (i.e. errors related by form)
Mixed error (semantic + formal)
Phonological access error
Phonological access error: Selection of
incorrect phonemes
Syl
FOG
f
r
d
Onsets
DOG
k
CAT
m
ae
RAT
MAT
o
t
Vowels
On Vo Co
g
Codas
Adapted from Gary Dell, “Producing words from pictures or from other words”
Representing concepts using
triangle nodes
Connectionist Circuit for Gender(RD5) = female
RD5
2
gender
female
Also shown as a triangle node
How does activity lead to structural
change?
• The brain (pre-natal, post-natal, and adult) exhibits a
surprising degree of activity dependent tuning and
plasticity.
• To understand the nature and limits of the tuning and
plasticity mechanisms we study
– How activity is converted to structural changes (say the ocular
dominance column formation)
• It is centrally important for us to understand these
mechanisms to arrive at biological accounts of
perceptual, motor, cognitive and language learning
– Biological Learning is concerned with this topic.
Long Term Potentiation (LTP)
• These changes make each of the winning synapses
more potent for an intermediate period, lasting from
hours to days (LTP).
• In addition, repetition of a pattern of successful firing
triggers additional chemical changes that lead, in time, to
an increase in the number of receptor channels
associated with successful synapses - the requisite
structural change for long term memory.
– There are also related processes for weakening synapses and
also for strengthening pairs of synapses that are active at about
the same time.
During normal
low-frequency
trans-mission,
glutamate
interacts with
NMDA and nonNMDA (AMPA)
and metabotropic
receptors.
With highfrequency
stimulation
The ICSI/Berkeley
Neural Theory of Language Project
Rods and Cones in the Retina
http://www.iit.edu/~npr/DrJennifer/visual/retina.html
Color Naming
Basic Color Terms (Berlin & Kay)
Criteria:
1. Single words -- not “light-blue” or “blue-green”
2. Frequently used -- not “mauve” or “cyan”
3. Refer primarily to colors -- not “lime” or “gold”
4. Apply to any object -- not “roan” or “blond”
© Stephen E. Palmer, 2002
The WCS Color Chips
• Basic color terms:
–
–
–
–
Single word (not blue-green)
Frequently used (not mauve)
Refers primarily to colors (not lime)
Applies to any object (not blonde)
FYI:
English has 11
basic color terms
Results of Kay’s Color Study
Stage I
II
IIIa / IIIb
IV
V
VI
VII
W or R or Y
W
W
W
W
W
W
Bk or G or Bu
R or Y
R or Y
R
R
R
R
Y
Y
Y
Y
G or Bu
G
G
G
Bk
Bu
Bu
Bu
Bk
Bk
Bk
Y+Bk (Brown)
Y+Bk (Brown)
Bk or G or Bu G or Bu
Bk
W
R
Y
R+W (Pink)
Bk or G or Bu
R + Bu (Purple)
R+Y (Orange)
B+W (Grey)
If you group languages into the number of basic
color terms they have, as the number of color
terms increases, additional terms specify focal
colors
Ideas from Cognitive Linguistics
• Embodied Semantics (Lakoff, Johnson, Sweetser, Talmy
• Radial categories
(Rosch 1973, 1978; Lakoff 1985)
– mother: birth / adoptive / surrogate / genetic, …
• Profiling (Langacker 1989, 1991; cf. Fillmore XX)
– hypotenuse, buy/sell (Commercial Event frame)
• Metaphor and metonymy
(Lakoff & Johnson 1980, …)
– ARGUMENT IS WAR, MORE IS UP
– The ham sandwich wants his check.
• Mental spaces (Fauconnier 1994)
– The girl with blue eyes in the painting really has green eyes.
• Conceptual blending (Fauconnier & Turner 2002, inter alia)
– workaholic, information highway, fake guns
– “Does the name Pavlov ring a bell?” (from a talk on ‘dognition’!)
Concepts are not categorical
Radial Structure of Mother
Genetic
mother
Stepmother
Unwed
mother
Surrogate
mother
Biological
mother
Adoptive
mother
Central
Case
Foster
mother
Birth
mother
Natural
mother
The radial structure of this category is
defined with respect to the different
models
Language, Learning and Neural Modeling
www.icsi.berkeley.edu/AI
• Scientific Goal
Understand how people learn and use language
• Practical Goal
Build systems that analyze and produce language
• Approach
Embodied linguistic theories with advanced
biologically-based computational methods
General and Domain Knowledge
• Conceptual Knowledge and Inference
–
–
–
–
Embodied
Language and Domain Independent
Powerful General Inferences
Ubiquitous in Language
• Domain Specific Frames and Ontologies
– Framenet (www.icsi.berkeley.edu/framenet)
• Metaphor links domain specific to general
– E.g., France slipped into recession.
Image schemas
• Trajector / Landmark (asymmetric)
– The bike is near the house
– ? The house is near the bike
TR
• Boundary / Bounded Region
– a bounded region has a closed boundary
LM
boundary
bounded region
• Topological Relations
– Separation, Contact, Overlap, Inclusion, Surround
• Orientation
– Vertical (up/down), Horizontal (left/right, front/back)
– Absolute (E, S, W, N)
Learning
System
dynamic relations
(e.g. into)
structured connectionist
network (based on
visual system)
We’ll look at the
details next lecture
Simulation-based language
understanding
“Harry walked to the cafe.”
Utterance
Constructions
Analysis Process
General
Knowledge
Belief State
Schema
walk
Trajector
Harry
Cafe
Goal
cafe
Simulation
Specification
Simulation
Simulation Semantics
• BASIC ASSUMPTION: SAME REPRESENTATION FOR
PLANNING AND SIMULATIVE INFERENCE
– Evidence for common mechanisms for recognition and
action (mirror neurons) in the F5 area (Rizzolatti et al (1996),
Gallese 96, Boccino 2002) and from motor imagery
(Jeannerod 1996)
• IMPLEMENTATION:
– x-schemas affect each other by enabling, disabling or
modifying execution trajectories. Whenever the
CONTROLLER schema makes a transition it may set, get,
or modify state leading to triggering or modification of other
x-schemas. State is completely distributed (a graph marking)
over the network.
• RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!
Active representations
• Many inferences about actions derive from what we
know about executing them
• Representation based on stochastic Petri nets
captures dynamic, parameterized nature of actions
walker at goal
energy
walker=Harry
goal=home
Walking:
bound to a specific walker with a
direction or goal
consumes resources (e.g., energy)
may have termination condition
(e.g., walker at goal)
ongoing, iterative action
Learning Verb Meanings
David Bailey
A model of children learning their first verbs.
Assumes parent labels child’s actions.
Child knows parameters of action, associates with word
Program learns well enough to:
1) Label novel actions correctly
2) Obey commands using new words (simulation)
System works across languages
Mechanisms are neurally plausible.
cow
apple
ball
juice
bead
girl
bottle
truck
baby
w oof
yum
go
up
this
no
m ore
m ore
spoon
ham m er
shoe
d ad d y
m oo
w hee
get
out
there
bye
banana
box
eye
m om y
uhoh
sit
in
here
hi
cookie
horse
d oor boy
choochoo
boom
oh
open
on
that
no
food
toys
yes
misc.
people
d ow n
sound emotion action
prep.
demon. social
Words learned by most 2-year olds in a play school (Bloom 1993)
System Overview
Learning Two Senses of PUSH
Model merging based on Bayesian MDL
Training Results
David Bailey
English
• 165 Training Examples, 18 verbs
• Learns optimal number of word senses (21)
• 32 Test examples : 78% recognition, 81% action
• All mistakes were close lift ~ yank, etc.
• Learned some particle CXN,e.g., pull up
Farsi
• With identical settings, learned senses not in
English
Task: Interpret simple discourse
fragments/ blurbs
France fell into recession. Pulled out by Germany
US Economy on the verge of falling back into recession
after moving forward on an anemic recovery.
Indian Government stumbling in implementing
Liberalization plan.
Moving forward on all fronts, we are going to be ongoing
and relentless as we tighten the net of justice.
The Government is taking bold new steps. We are
loosening the stranglehold on business, slashing
tariffs and removing obstacles to international trade.
Probabilistic inference
– Filtering
• P(X_t | o_1…t,X_1…t)
• Update the state based on the observation sequence and state
set
– MAP Estimation
• Argmaxh1…hnP(X_t | o_1…t, X_1…t)
• Return the best assignment of values to the hypothesis
variables given the observation and states
– Smoothing
• P(X_t-k | o_1…t, X_1…t)
• modify assumptions about previous states, given observation
sequence and state set
– Projection/Prediction/Reachability
• P(X_t+k | o_1..t, X_1..t)
Metaphor Maps
• Static Structures that project bindings from source
domain f- struct to target domain Bayes net nodes by
setting evidence on the target network.
• Different types of maps
– PMAPS project X- schema Parameters to abstract domains
– OMAPS connect roles between source and target domain
– SMAPS connect schemas from source to target domains.
• ASPECT is an invariant in projection.
Results
• Model was implemented and tested on discourse fragments
from a database of 50 newspaper stories in international
economics from standard sources such as WSJ, NYT, and the
Economist.
• Results show that motion terms are often the most effective
method to provide the following types of information about
abstract plans and actions.
– Information about uncertain events and dynamic changes in goals and
resources. (sluggish, fall, off-track, no steam)
– Information about evaluations of policies and economic actors and
communicative intent (strangle-hold, bleed).
– Communicating complex, context-sensitive and dynamic economic
scenarios (stumble, slide, slippery slope).
– Commincating complex event structure and aspectual information (on
the verge of, sidestep, giant leap, small steps, ready, set out, back on
track).
• ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATIC
INFERENCES PROVIDED BY X-SCHEMA BASED
INFERENCES.
Embodied Construction Grammar
ECG
1. Linguistic Analysis
2. Computational Implementation
a. Test Grammars
b. Applied Projects – Question Answering
3. Map to Connectionist Models, Brain
4. Models of Grammar Acquisition
Embodied Construction Grammar
• Embodied representations
– active perceptual and motor schemas
(image schemas, x-schemas, frames, etc.)
– situational and discourse context
• Construction Grammar
– Linguistic units relate form and
meaning/function.
– Both constituency and (lexical) dependencies
allowed.
• Constraint-based
– based on feature unification (as in LFG, HPSG)
– Diverse factors can flexibly interact.
“Harry walked into the cafe.”
Pragmatics
Semantics
Syntax
Morphology
Phonology
Phonetics
“Harry walked into the cafe.”
Pragmatics
Semantics
Syntax
Morphology
Phonology
Phonetics
U
T
T
E
R
A
N
C
E
ECG Structures
• Schemas
– image schemas, force-dynamic schemas, executing
schemas, frames…
• Constructions
– lexical, grammatical, morphological, gestural…
• Maps
– metaphor, metonymy, mental space maps…
• Spaces
– discourse, hypothetical, counterfactual…
Embodied schemas
schema name
schema Source-Path-Goal
roles
source
path
goal
trajector
role name
schema Container
roles
interior
exterior
portal
boundary
Boundary
Source
Trajector
Interior
Portal
Goal
Path
Exterior
These are abstractions over sensorimotor experiences.
Embodied constructions
Form
Meaning
construction HARRY
form : /hEriy/
meaning : Harry
Harry
cafe
ECG Notation
CAFE
construction CAFE
form : /khaefej/
meaning : Cafe
Constructions have form and meaning poles that are subject to type constraints.
Simulation-based language understanding
construction WALKED
form
selff.phon  [wakt]
meaning : Walk-Action
constraints
selfm.time before Context.speech-time
selfm..aspect  encapsulated
“Harry walked into the cafe.”
Utterance
Analysis Process
Constructions
General
Knowledge
Semantic
Specification
Belief State
CAFE
Simulation
Simulation specification
The analysis process produces a simulation specification that
•includes image-schematic, motor control and conceptual structures
•provides parameters for a mental simulation
Summary: ECG
• Linguistic constructions are tied to a model of
simulated action and perception
• Embedded in a theory of language processing
– Constrains theory to be usable
– Frees structures to be just structures, used in
processing
• Precise, computationally usable formalism
– Practical computational applications, like MT and NLU
– Testing of functionality, e.g. language learning
• A shared theory and formalism for different
cognitive mechanisms
– Constructions, metaphor, mental spaces, etc.
State of the Art
Natural Language Understanding
• Limited Commercial Speech Applications
transcription, simple response systems
• Statistical NLP for Restricted Tasks
tagging, parsing, information retrieval
• Template-based Understanding programs
expensive, brittle, inflexible, unnatural
• Essentially no NLU in QA, HCI systems
Key ideas for a NT of language
acquisition
Nancy Chang
• Opulence of the substrate
– Prelinguistic children already have rich sensorimotor
representations and sophisticated social knowledge
• Basic Scenes
– Simple clause constructions are associated directly with scenes
basic to human experience
(Goldberg 1995, Slobin 1985)
• Verb Island Hypothesis
– Children learn their earliest constructions
(arguments, syntactic marking) on a verb-specific basis
(Verb Island Hypothesis, Tomasello 1992)
Early talk about throwing
Transcript data, Naomi 1;11.9
Sample input prior to 1;11.9:
don’t throw the bear.
don’t throw them on the ground.
Nomi don’t throw the books down.
what do you throw it into?
Sample tokens prior to 1;11.9:
throw
throw off
I throw it.
I throw it ice. (= I throw the ice)
Par:
Par:
Child:
Child:
Par:
Par:
Child:
Child:
Child:
Par:
Child:
they’re throwing this in here.
throwing the thing.
throwing in.
throwing.
throwing the frisbee. …
do you throw the frisbee?
do you throw it?
throw it.
I throw it. …
throw frisbee.
she’s throwing the frisbee.
throwing ball.
Sachs corpus (CHILDES)
How do they make the leap?
0-9 months
18-24 months
• Smiles
• Responds differently to
intonation
• Responds to name and “no”
•
9-18 months
•
• First words
• Recognizes intentions
• Responds, requests, calls,
greets, protests
•
agent-object
– Daddy cookie
– Girl ball
•
agent-action
– Daddy eat
– Mommy throw
action-object
– Eat cookie
– Throw hat
entity-attribute
– Daddy cookie
•
entity-locative
– Doggie bed
Correlating forms and meanings
FORM (sound)
“you”
lexical constructions
you
MEANING (stuff)
Qu ic kTi me™ a nd a TIFF (U nc omp res se d) de co mpre ss or are n ee de d to se e thi s p i cture .
Human
“throw”
throw
Throw
thrower
throwee
“ball”
“block”
ball
block
Object
Learning-Analysis cycle
1. Learner passes input (Utt, Sitn)
Reorganize
and current CxnSet to Analyzer.
(Utterance, Situation)
2. Analyzer produces SemSpec
Constructions and Constructional Analysis.
Analyze
Semantic Specification,
Constructional Analysis
3. Learner updates CxnSet:
Hypothesize
a. Hypothesize new map.
b. Reorganize CxnSet
(merge or compose).
c. Reinforce
(based on usage).
How could a mass of chemical cells produce
language and thought?
Will computers think and speak?
How much can we know about our own experience?
How do we learn new concepts?
Does our language determine how we think?
Is language Innate?
How do children learn grammar?
How did languages evolve?
Why do we experience everything the way that we do?
Forthcoming Book
From Molecule to Metaphor:
The Neural Basis of Language and Thought
June 2006