Language Understanding
and Unified Cognitive Science
Jerome Feldman
International Computer Science Institute
U. California at Berkeley
Berkeley, CA
jfeldman@icsi.berkeley.edu
Unified Cognitive Science
Neurobiology
Psychology
Computer Science
Linguistics
Philosophy
Social Sciences
Experience
Take all the Findings and Constraints Seriously
Functionalism
In fact, the belief that neurophysiology is even relevant to the
functioning of the mind is just a hypothesis. Who knows if we’re
looking at the right aspects of the brain at all. Maybe there are other
aspects of the brain that nobody has even dreamt of looking at yet.
That’s often happened in the history of science. When people say that
the mental is just the neurophysiological at a higher level, they’re being
radically unscientific. We know a lot about the mental from a scientific
point of view. We have explanatory theories that account for a lot of
things. The belief that neurophysiology is implicated in these things
could be true, but we have very little evidence for it. So, it’s just a kind
of hope; look around and you see neurons: maybe they’re implicated.
Noam Chomsky 1993, p.85
Embodiment
Of all of these fields, the learning of
languages would be the most impressive,
since it is the most human of these
activities. This field, however, seems to
depend rather too much on the sense
organs and locomotion to be feasible.
Alan Turing (Intelligent Machines,1948)
Continuity Principle of the American Pragmatists
Lectures
I. Overview
2. Simulation Semantics
3. ECG and Best-fit Analysis
4. Compositionality
5. Simulation, Counterfactuals, and Inference
Utterance
Discourse & Situational
Context
Constructions
Analyzer:
incremental,
competition-based,
psychologically
plausible
Semantic Specification:
image schemas, bindings,
action schemas
Simulation
Psycholinguistic evidence
• Embodied language impairs action/perception
– Sentences with visual components to their meaning can
interfere with performance of visual tasks
(Richardson et al. 2003)
– Sentences describing motion can interfere with performance
of incompatible motor actions
(Glenberg and Kashak 2002)
– Sentences describing incompatible visual imagery impedes
decision task (Zwaan et al. 2002)
• Simulation effects from fictive motion sentences
– Fictive motion sentences describing paths that require
longer time, span a greater distance, or involve more
obstacles impede decision task (Matlock 2000, Matlock et al. 2003)
Neural evidence: Mirror neurons
• Gallese et al. (1996) found “mirror” neurons in
the monkey motor cortex, activated when
– an action was carried out
– the same action (or a similar one) was seen.
• Mirror neuron circuits found in humans (Porro et al.
1996)
• Mirror neurons activated when someone:
– imagines an action being carried out (Wheeler et al. 2000)
– watches an action being carried out (with or without
object) (Buccino et al. 2000)
The Mirror System
The mirror system, like the motor system, is
Buccino et al.,
somatotopically organized.
2001
humans watching
videos of actions
without objects
humans watching
same actions with
objects
Foot
foot actions
Hand
handactions
Mouth
actions
mouth
Fast Brain ~ Slow Neurons
Mental Connections are Active
Neural Connections
There is No Erasing in the Brain
Movement vs. Actions
Pulvermueller Lab
Brains ~ Computers
•
•
•
•
•
•
•
•
1000 operations/sec
100,000,000,000 units
10,000 connections/
graded, stochastic
embodied
fault tolerant
evolves
learns
•
•
•
•
•
•
•
•
1,000,000,000 ops/sec
1-100 processors
~ 4 connections
binary, deterministic
abstract, disembodied
crashes frequently
explicitly designed
is programmed
The ICSI/Berkeley
Neural Theory of Language Project
ECG
Learning early
constructions
(Chang, Mok)
Active representations
• Many inferences about actions derive from what
we know about executing them
• Representation based on stochastic Petri nets
captures dynamic, parameterized nature of actions
walker at goal
energy
walker=Harry
goal=home
Walking:
bound to a specific walker with a
direction or goal
consumes resources (e.g., energy)
may have termination condition
(e.g., walker at goal)
ongoing, iterative action
Learning Verb Meanings
David Bailey
A model of children learning their first verbs.
Assumes parent labels child’s actions.
Child knows parameters of action, associates with word
Program learns well enough to:
1) Label novel actions correctly
2) Obey commands using new words (simulation)
System works across languages
Mechanisms are neurally plausible.
System Overview
Learning Two Senses of PUSH
Model merging based on Bayesian MDL
The ICSI/Berkeley
Neural Theory of Language Project
ECG
Learning early
constructions
(Chang, Mok)
The Binding Problem
 Massively Parallel Brain
 Unitary Conscious Experience
 Many Variations and Proposals
 Our focus: The Variable Binding Problem
SHRUTI
• SHRUTI does inference
by connections between
simple computation
nodes
• Nodes are small groups
of neurons
• Nodes firing in sync
reference the same
object
Proposed Alternative Solution
• Indirect references
– Pass short signatures, “fluents”
• Functionally similar to SHRUTI's time slices
– Central “binder” maps fluents to objects
• In SHRUTI, the objects fired in that time slice
– Connections need to be more complicated than
in SHRUTI
• Fluents are passed through at least 3 bits
• But temporal synchrony is not required
Lectures
I. Overview
2. Simulation Semantics
3. ECG and Best-fit Analysis
4. Compositionality
5. Simulation, Counterfactuals, and Inference
Utterance
Discourse & Situational
Context
Constructions
Analyzer:
incremental,
competition-based,
psychologically
plausible
Semantic Specification:
image schemas, bindings,
action schemas
Simulation
Ideas from Cognitive Linguistics
• Embodied Semantics (Lakoff, Johnson, Sweetser, Talmy
• Radial categories
(Rosch 1973, 1978; Lakoff 1985)
– mother: birth / adoptive / surrogate / genetic, …
• Profiling (Langacker 1989, 1991; cf. Fillmore XX)
– hypotenuse, buy/sell (Commercial Event frame)
• Metaphor and metonymy
(Lakoff & Johnson 1980, …)
– ARGUMENT IS WAR, MORE IS UP
– The ham sandwich wants his check.
• Mental spaces (Fauconnier 1994)
– The girl with blue eyes in the painting really has green eyes.
• Conceptual blending (Fauconnier & Turner 2002, inter alia)
– workaholic, information highway, fake guns
– “Does the name Pavlov ring a bell?” (from a talk on ‘dognition’!)
Image schemas
• Trajector / Landmark (asymmetric)
TR
– The bike is near the house
– ? The house is near the bike
• Boundary / Bounded Region
LM
boundary
bounded region
– a bounded region has a closed boundary
• Topological Relations
– Separation, Contact, Overlap, Inclusion, Surround
• Orientation
– Vertical (up/down), Horizontal (left/right, front/back)
– Absolute (E, S, W, N)
Schema Formalism
SCHEMA <name>
SUBCASE OF <schema>
EVOKES <schema> AS <local name>
ROLES < self role name>: <role restriction>
< self role name> <-> <role name>
CONSTRAINTS <role name> <-
<value>
<role name> <-> <role name>
A Simple Example
SCHEMA hypotenuse
SUBCASE OF line-segment
EVOKES right-triangle AS rt
ROLES Comment inherited from line-segment
CONSTRAINTS
SELF <-> rt.long-side
Language understanding: analysis &
simulation
construction WALKED
form
selff.phon  [wakt]
meaning : Walk-Action
constraints
selfm.time before Context.speech-time
selfm..aspect  encapsulated
“Harry walked into the cafe.”
Utterance
Analysis Process
Constructions
General
Knowledge
Semantic
Specification
Belief State
CAFE
Simulation
Semantic specification
The analysis process produces a semantic specification that
•includes image-schematic, motor control and conceptual structures
•provides parameters for a mental simulation
Task: Interpret simple discourse
fragments/ blurbs
France fell into recession. Pulled out by Germany
US Economy on the verge of falling back into recession after
moving forward on an anemic recovery.
Indian Government stumbling in implementing Liberalization
plan.
Moving forward on all fronts, we are going to be ongoing and
relentless as we tighten the net of justice.
The Government is taking bold new steps. We are loosening
the stranglehold on business, slashing tariffs and removing
obstacles to international trade.
Results
• Model was implemented and tested on discourse fragments from a
database of 50 newspaper stories in international economics from
standard sources such as WSJ, NYT, and the Economist.
• Results show that motion terms are often the most effective method to
provide the following types of information about abstract plans and
actions.
– Information about uncertain events and dynamic changes in goals
and resources. (sluggish, fall, off-track, no steam)
– Information about evaluations of policies and economic actors and
communicative intent (strangle-hold, bleed).
– Communicating complex, context-sensitive and dynamic economic
scenarios (stumble, slide, slippery slope).
– Commincating complex event structure and aspectual information
(on the verge of, sidestep, giant leap, small steps, ready, set out,
back on track).
• ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATIC
INFERENCES PROVIDED BY X-SCHEMA BASED
INFERENCES.
Embodied Construction Grammar
• Embodied representations
– active perceptual and motor schemas
– situational and discourse context
• Construction Grammar
– Linguistic units relate form and meaning/function.
– Both constituency and (lexical) dependencies allowed.
• Constraint-based (Unification)
– based on feature structures (as in HPSG)
– Diverse factors can flexibly interact.
Embodied Construction Grammar
ECG
(Formalizing Cognitive Linguistics)
1. Community Grammar and Core Concepts
2. Deep Grammatical Analysis
3. Computational Implementation
a. Test Grammars
b. Applied Projects – Question Answering
4. Map to Connectionist Models, Brain
5. Models of Grammar Acquisition
Verb Constructions
Construction BITE1
subcase of Verb
form: bite
meaning: ForceApplication
constraints:
Effector ← teeth
Routine ← bite // close mouth
schema ForceApplication
subcase of MotorControl
evokes ForceTransfer as FT
roles
Actor ↔ FT.Supplier ↔ Protagonist
Acted Upon ↔ FT.Recipient
Effector
Routine
Effort ↔ FT.Force.amount
Semantic Specification
He bit the apple
EventDescriptor
eventtype
ProfiledProcess
ProfiledParticipant
RD27
category
CauseEffect
causer
affected
ForceApplication
actor
actedupon
routine  bite
effector  teeth
Person
Apple
RD55
category
Modeling context for
language understanding and learning
• Linguistic structure reflects experiential
structure
– Discourse participants and entities
– Embodied schemas:
• action, perception, emotion, attention, perspective
– Semantic and pragmatic relations:
• spatial, social, ontological, causal
• ‘Contextual bootstrapping’ for grammar learning
Constrained Best Fit in Nature
inanimate
physics
chemistry
biology
vision
language
animate
lowest energy
state
molecular
minima
fitness, MEU
Neuroeconomics
threats,
friends
errors,
NTL
Two perspectives on language
learning
Computational models
• Grammatical induction
– language identification
– context-free grammars,
unification-based
grammars
– statistical NLP
• Word learning models
– semantic representations
• logical forms
• discrete representations
• continuous representations
– statistical models
Developmental evidence
• Prior knowledge
–
–
–
–
concepts
event-based knowledge
social cognition
lexical items
• Data-driven learning
– basic scenes
– lexically specific patterns
– usage-based learning
Language Acquisition
• Opulence of the substrate
– Prelinguistic children already have rich
sensorimotor representations and sophisticated
social knowledge
– intention inference, reference resolution
– language-specific event conceptualizations
(Bloom 2000, Tomasello 1995,
Bowerman & Choi, Slobin, et al.)
• Children are sensitive to statistical information
– Phonological transitional probabilities
– Most frequent items in adult input learned earliest
(Saffran et al. 1998, Tomasello 2000)
Experiment: learning verb islands
• Question:
– Can the proposed construction learning model
acquire English item-based motion constructions?
(Tomasello 1992)
• Given: initial lexicon and
ontology
• Data: child-directed
language annotated with
contextual information
Form:
text : throw the ball
intonation : falling
Participants :
Mother, Naomi, Ball
Scene :
Throw
thrower : Naomi
throwee : Ball
Discourse :
speaker :Mother
addressee Naomi
speech act : imperative
activity : play
joint attention : Ball
The intuition behind learning a new
form-meaning pairing from context
construction Put
before
construction Coat
Put-Action
put-agent
put-theme
location
Coat
before
construction Here
Sofa
The learner learns a new lexically-specific
construction from the form-meaning pair
construction Put-Coat-Here
constituents
v: Put
o: Coat
p: Here
form
vf before of before pf
meaning: Caused-Motion-Scene
selfm.means  vm
selfm.mover  om
selfm.path  pm
Experiment: learning verb islands
Subset of the CHILDES database of parent-child
interactions (MacWhinney 1991; Slobin )
• coded by developmental psychologists for
– form: particles, deictics, pronouns, locative phrases, etc.
– meaning: temporality, person, pragmatic function,
type of motion (self-movement vs. caused movement;
animate being vs. inanimate object, etc.)
• crosslinguistic (English, French, Italian, Spanish)
– English motion utterances: 829 parent, 690 child utterances
– English all utterances: 3160 adult, 5408 child
– age span is 1;2 to 2;6
A quantitative measure: coverage
• Goal: incrementally improving comprehension
– At each stage in testing, use current grammar to analyze test set
• Coverage = % role bindings analyzed
• Example:
– Grammar: throw-ball, throw-block, you-throw
– Test sentence: throw the ball.
• Bindings: scene=Throw, thrower=Nomi, throwee=ball
• Parsed bindings: scene=Throw, throwee=ball
– Score test grammar on sentence: 2/3 = 66.7%
Learning to comprehend
Usage-based learning,
comprehension, and production
discourse & situational
context
world knowledge
utterance
comm. intent
constructicon
analyze
&
resolve
reinforcement
(usage)
hypothesize
constructions
& reorganize
analysis
simulation
reinforcement
(usage)
reinforcement
(correction)
generate
utterance
reinformcent
(correction)
response
Unified Cognitive Science
Neurobiology
Psychology
Computer Science
Linguistics
Philosophy
Social Sciences
Experience
Take all the Findings and Constraints Seriously
The ICSI/Berkeley
Neural Theory of Language Project
•
Principal investigators
 Jerome Feldman (UCB,ICSI)
 George Lakoff (UCB Ling)
 Srini Narayanan (UCB,ICSI)
 Lokendra Shastri (now India)
• Affiliated faculty
 Chuck Fillmore (ICSI)
 Eve Sweetser (UCB Ling)
 Rich Ivry (UCB Psych)
 Lisa Aziz-Zadeh (USC)
 Graduate Students




*Ellen Dodge (Ling)
Michael Ellsworth (Ling)
Joshua Marker (Ling)
Shweta Narayan (Ling)
 Alumni
 Terry Regier (UCB Ling,
CogSci)
 Johno Bryant (Ask)
 David Bailey (Google)
 Leon Barrett (Google)
 Nancy Chang (Sony Paris)
 Joe Makin (UCSF)
 Eva Mok (U. Chicago)
 Andreas Stolcke (ICSI, SRI)
 Dan Jurafsky (Stanford Ling)
 Olya Gurevich (Powerset)
 Benjamin Bergen (UCSD)
 Carter Wendelken (UCB)
Source-Path-Goal
SCHEMA: spg
ROLES:
source: Place
path: Directed Curve
goal: Place
trajector: Entity
Translational Motion
SCHEMA translational motion
SUBCASE OF motion
EVOKES spg AS s
ROLES
mover <-> s.trajector
source <-> s.source
goal
<-> s.goal
CONSTRAINTS
before:: mover.location <-> source
after::
mover.location <-> goal
Event Structure for semantic QA
Srini Narayanan
• Reasoning about dynamics
– Complex event structure
• Multiple stages, interruptions, resources, framing
– Evolving events
• Conditional events, presuppositions.
– Nested temporal and aspectual references
• Past, future event references
– Metaphoric references
• Use of motion domain to describe complex events.
• Reasoning with Uncertainty
– Combining Evidence from Multiple, unreliable sources
– Non-monotonic inference
• Retracting previous assertions
• Conditioning on partial evidence
Components of the System
• Object references
– Fluents
– Binder
• Short term storage
– Predicate state
• Long term storage
– Facts, mediators, what predicates exist
• Inference
– Mediators
• Types
– Ontology
Simulation-based language understanding
“Harry walked to the cafe.”
Utterance
Constructions
Analysis Process
General
Knowledge
Belief State
Schema
walk
Trajector
Harry
Cafe
Goal
cafe
Simulation
Specification
Simulation
Representing constructions: TO
construction TO
form
selff.phon  /thuw/
meaning
evokes
Trajector-Landmark as tl
Source-Path-Goal as spg
constraints:
tl.trajector  spg.trajector
tl.landmark  spg.goal
local alias
identification constraint
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
The INTO construction
TO vs. INTO:
INTO adds a
Container schema
and appropriate
bindings.
construction INTO
form
selff.phon  /Inthuw/
meaning
evokes
Trajector-Landmark as tl
Source-Path-Goal as spg
Container as cont
constraints:
tl.trajector  spg.trajector
tl.landmark  cont
cont.interior  spg.goal
cont.exterior  spg.source
An ECG analysis with THROW-TRANSITIVE