Embodied Construction Grammar
in language (acquisition and) use
Jerome Feldman
(jfeldman@icsi.berkeley.edu)
Computer Science Division, University of California, Berkeley, and
International Computer Science Institute
State of the Art
• Limited Commercial Speech Applications
transcription, simple response systems
• Statistical NLP for Restricted Tasks
tagging, parsing, information retrieval
• Template-based Understanding programs
expensive, brittle, inflexible, unnatural
• Essentially no NLU in HCI, QA Systems
What does language do?
A sentence can evoke an imagined scene and resulting inferences:
“Harry walked to the cafe.”
CAFE
– Goal of action = at cafe
– Source = away from cafe
– cafe = point-like location
“Harry walked into the cafe.”
CAFE
– Goal of action = inside cafe
– Source = outside cafe
– cafe = containing location
Language understanding
(Utterance, Situation)
Conceptual
knowledge
Linguistic
knowledge
Analysis
Interpretation
Language understanding: analysis & simulation
“Harry walked to the cafe.”
Utterance
Lexicon
Constructicon
General
Knowledge
Belief State
Analysis Process
Schema
walk
Trajector
Harry
Cafe
Goal
cafe
Semantic
Specification
Simulation
Interpretation: x-schema simulation
Constructions can
• specify which schemas
and entities are involved
in an event, and how
they are related
• profile particular stages
of an event
• set parameters of an
event
walker at goal
energy
goal=home
walker=Harry
Harry is walking home.
Traditional Levels of Analysis
Pragmatics
Semantics
Syntax
Morphology
Phonetics
“Harry walked into the cafe.”
Pragmatics
Semantics
Utterance
Syntax
Morphology
Phonetics
Construction Grammar
A construction is a form-meaning pair whose properties may not be
strictly predictable from other constructions.
(Construction Grammar, Goldberg 1995)
Form
Meaning
block
walk
to
Source
Trajector
Path
Goal
Form-meaning mappings for language
Linguistic knowledge consists of form-meaning mappings:
Form
Meaning
phonological cues
word order
intonation
inflection
event structure
sensorimotor control
attention/perspective
social goals...
Cafe
Constructions as maps between relations
Complex constructions are mappings between relations in form
and relations in meaning.
Form
Mover + Motion
before(Mover, Motion)
“is” + Action + “ing”
before(“is”, Action)
suffix(Action, “ing”)
Mover + Motion + Direction
before(Motion, Direction)
before(Mover, Motion)
Meaning
MotionEvent
mover(Motion, Mover)
ProgressiveAction
aspect(Action, ongoing)
DirectedMotionEvent
direction(Motion, Direction)
mover(Motion, Mover)
Embodied Construction Grammar
(Bergen and Chang 2002)
• Embodied representations
– active perceptual and motor schemas
– situational and discourse context
• Construction Grammar
– Linguistic units relate form and meaning/function.
– Both constituency and (lexical) dependencies allowed.
• Constraint-based (Unification)
– based on feature structures (as in HPSG)
– Diverse factors can flexibly interact.
Representing image schemas
schema name
schema Source-Path-Goal
roles
source
path
goal
trajector
role name
schema Container
roles
interior
exterior
portal
boundary
Boundary
Source
Trajector
Interior
Portal
Goal
Path
Exterior
These are abstractions over sensorimotor experiences.
Inference and Conceptual Schemas
•
Hypothesis:
– Linguistic input is converted into a mental simulation based on bodily-grounded structures.
•
Components:
– Semantic schemas
• image schemas and executing schemas are abstractions over neurally grounded perceptual and
motor representations
– Linguistic units
• lexical and phrasal construction representations invoke schemas, in part through metaphor
•
Inference links these structures and provides parameters for a simulation engine
Early Example
Understanding News Stories
France fell into recession. Pulled out by Germany
In1991, India set out on a path of liberalization.
The Government started to loosen its stranglehold on
business and removed obstacles to international
trade. Now the Government is stumbling in
implementing the liberalization plan.
Task
• Interpret simple discourse fragments/blurbs
– France fell into recession. Pulled out by Germany
– Economy moving at the pace of a Clinton jog.
– US Economy on the verge of falling back into recession after moving
forward on an anemic recovery.
– Indian Government stumbling in implementing Liberalization plan.
– Moving forward on all fronts, we are going to be ongoing and relentless as
we tighten the net of justice.
– The Government is taking bold new steps. We are loosening the
stranglehold on business, slashing tariffs and removing obstacles to
international trade.
I/O as Feature Structures
• Indian Government stumbling in implementing liberalization plan
Language understanding: analysis & simulation
construction WALKED
form
selff.phon  [wakt]
meaning : Walk-Action
constraints
selfm.time before Context.speech-time
selfm..aspect  encapsulated
“Harry walked into the cafe.”
Utterance
Analysis Process
Constructions
General
Knowledge
Semantic
Specification
Belief State
CAFE
Simulation
Embodied Construction Grammar provides
formal tools for linguistic description and analysis
motivated largely by cognitive/functional concerns.
• Allows precise specifications of structures/processes
involved in acquisition of early constructions
–Embodied constructions (structured maps between form and
meaning); lexically specific and more general
–Usage-based processes of learning new constructions to
account for co-occurring utterance-situation pairs
• Bridge to detailed psycholinguistic and neural imaging
experiments
Formal Cognitive Linguistics
• Schemas and frames
– Image schemas, force dynamics, executing schemas…
• Constructions
– Lexical, grammatical, morphological, gestural…
• Maps
– Metaphor, metonymy, mental space maps…
• Mental spaces
– Discourse, hypothetical, counterfactual…
Embodied constructions
Form
Meaning
construction HARRY
form : [hEriy]
meaning : Harry
Harry
cafe
Notation
CAFE
construction CAFE
form : [khaefej]
meaning : Cafe
Constructions have form and meaning poles that are subject to type constraints.
Schema Formalism
SCHEMA <name>
SUBCASE OF <schema>
EVOKES <schema> AS <local name>
ROLES < self role name>: <role restriction>
< self role name> <-> <role name>
CONSTRAINTS <role name> <-
<value>
<role name> <-> <role name>
<setting name> :: <role name> <-> <role name>
<setting name> :: <predicate> | <predicate>
A Simple Example
SCHEMA hypotenuse
SUBCASE OF line-segment
EVOKES right-triangle AS rt
ROLES Comment inherited from line-segment
CONSTRAINTS
SELF <-> rt.long-side
Source-Path-Goal
SCHEMA: spg
ROLES:
source: Place
path: Directed Curve
goal: Place
trajector: Entity
Translational Motion
SCHEMA translational motion
SUBCASE OF motion
EVOKES spg AS s
ROLES
mover <-> s.trajector
source <-> s.source
goal
<-> s.goal
CONSTRAINTS
before:: mover.location <-> source
after::
mover.location <-> goal
Construction Formalism
CONSTRUCTION<name>
SUBCASE OF <construction>
CONSTRUCTIONAL
EVOKES <construction> AS <local name>
CONSTITUENTS
< local name> : <construction>
CONSTRAINTS
// as in SCHEMAs
FORM
ELEMENTS
CONSTRAINTS
MEANING
// as in SCHEMAs
// as in SCHEMAs
Representing constructions: TO
construction TO
form
selff.phon  [thuw]
meaning
evokes
Trajector-Landmark as tl
Source-Path-Goal as spg
constraints:
tl.trajector  spg.trajector
tl.landmark  spg.goal
local alias
identification constraint
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
The INTO construction
construction INTO
form
selff.phon  [Inthuw]
meaning
evokes
Trajector-Landmark as tl
Source-Path-Goal as spg
Container as cont
constraints:
tl.trajector  spg.trajector
tl.landmark  cont
cont.interior  spg.goal
cont.exterior  spg.source
TO vs. INTO:
INTO adds a Container schema and appropriate bindings.
Grammatical Construction Example
CONSTRUCTION Spatial-PP
SUBCASE OF Phrase
CONSTRUCTIONAL CONSTITUENTS
rel: Spatial-Preposition
lm: Referring-Exp
CONSTRAINTS
rel.case <-> lm.case
FORM
rel < lm
MEANING CONSTRAINTS
rel.landmark <-> lm
The DIRECTED-MOTION construction
construction DIRECTED-MOTION
constructional
constituents
mover : Thing
motion : Motion-Process
direction : Source-Path-Goal
form
moverf before motionf
motionf before directionf
meaning
evokes Motion-Event as m
m.mover moverm
m.motion  motionm
m.path  directionm
directionm.trajector moverm
motionm.mover moverm
Semantic specification
The analysis process produces a semantic specification that
•includes image-schematic, motor control and conceptual structures
•provides parameters for a mental simulation
Language Understanding Process
Constructional analysis
Semantic Specification
Language understanding: analysis & simulation
construction WALKED
form
selff.phon  [wakt]
meaning : Walk-Action
constraints
selfm.time before Context.speech-time
selfm..aspect  encapsulated
“Harry walked into the cafe.”
Utterance
Analysis Process
Constructions
General
Knowledge
Semantic
Specification
Belief State
CAFE
Simulation
Simulation-based sense disambiguation
Ease of construing nominal as a CONTAINER determines what
sense of into is appropriate:
•
The scientist walked
into the laboratory.
LAB
•
The scientist walked
into the wall.
WALL
Bonk!!
CONTAINER sense
CONTACT sense
Simulation-based inference
Detailed inferences can result from simulation.
Image-schematic content of prepositions must fit with properties of
other elements of sentence.
•
The teacher drifted
into the house.
– Final location of Trajector
= inside cafe
– Portal = door
•
The smoke drifted
into the house.
– Final location of Trajector
= inside (possibly throughout) cafe
– Portal = door/window
World knowledge informs simulation
Physical knowledge of how people and gases interact with
houses determines:
– Relation between Trajector and Interior
The smoke drifted into the house and filled it.
?The teacher drifted into the house and filled it.
– Portal for motion across Boundary
The smoke drifted into the house
because the window had been left open.
?The teacher drifted into the house
because the window had been left open.
Getting From the Utterance to the
SemSpec
Johno Bryant
• Need a grammar formalism
– Embodied Construction Grammar (Bergen & Chang 2002)
• Need new models for language analysis
– Traditional methods too limited
– Traditional methods also don’t get enough leverage out of the
semantics.
Embodied Construction Grammar
• Semantic Freedom
– Designed to be symbiotic with cognitive approaches to
meaning
– More expressive semantic operators than traditional grammar
formalisms
• Form Freedom
– Free word order, over-lapping constituency
• Precise enough to be implemented
Traditional Parsing Methods Fall Short
• PSG parsers too strict
– Constructions not allowed to leave constituent order
unspecified
• Traditional way of dealing with incomplete
analyses is ad-hoc
– Making sense of incomplete analyses is important when
an application must deal with “ill-formed” input
(For example, modeling language learning)
• Traditional unification grammar can’t handle
ECG’s deep semantic operators.
Our Analyzer
• Replaces the FSMs used in traditional chunking (Abney
96) with much more powerful machines capable of
backtracking called construction recognizers
• Arranges these recognizers into levels just like in Abney’s
work
• But uses a chart to deal with ambiguity
Our Analyzer (cont’d)
• Uses specialized feature structures to deal with ECG’s
novel semantic operators
• Supports a heuristic evaluation metric for finding the
“right” analysis
• Puts partial analyses together when no complete
analyses are available
– The analyzer was designed under the assumption that the grammar
won’t cover every meaningful utterance encountered by the system.
System Architecture
Grammar/Utterance
Semantic
Chunker
Learner
Semantic
Integration
Ranked Analyses
The Levels
• The analyzer puts the recognizer on the level
assigned by the grammar writer.
– Assigned level should be greater than or equal to the levels
of the construction’s constituents.
• The analyzer runs all the recognizers on level 1,
then level 2, etc. until no more levels.
• Recognizers on the same level can be mutually
recursive.
Recognizers
• Each Construction is turned into a recognizer
• Recognizer = active representation
– seeks form elements/constituents when initiated
– Unites grammar and process - grammar isn’t just a static piece of knowledge in
this model.
• Checks both form and semantic constraints
– Contains an internal representation of both the semantics and the form
– A graph data structure used to represent the form and a feature structure
representation for the meaning.
Recognizer Example
Mary kicked the ball into the net.
This is the initial Constituent
Graph for caused-motion.
Agent
Patient
Action
Path
Recognizer Example
Construct:
Caused-Motion
Constituent:
Agent
Constituent:
Action
Constituent:
Patient
Constituent:
Path
The initial constructional tree for the instance of
Caused-Motion that we are trying to create.
Recognizer Example
caused  motion.m 
 patient.m

agent : 5
 agent.m
, 3category :


, 1category :
 

 scene : 4
 

 resolved  ref : resolved  ref :
action : 6

caused  motion.cm 
path.m

agent : 15

 source :

action.m






, 2tense :
4 patient : {3}{cm1} ,{7} path :






 x  schema :
action : 26

 goal :

 path : {7}

trajector : {cm1}
Recognizer Example
processed
Mary kicked the ball into the net.
A node filled with gray is removed.
Patient
Agent
Action
Path
Recognizer Example
Construct:
Caused-Motion
RefExp:
Mary
Constituent:
Action
Constituent:
Patient
Constituent:
Path
Mary kicked the ball into the net.
Recognizer Example
caused  motion.m 
 patient.m
agent : 5
 agent.m
, 3category :

, 1category : Person
 
 scene : 4
 

 resolved  ref : Mary  resolved  ref
action : 6

caused  motion.cm 
path.m

agent : 15

 source :

action.m






, 2tense :
4 patient : {3}{cm1} ,{7} path :






 x  schema :
action : 26

 goal :

 path : {7}

trajector : {cm1}



:
Recognizer Example
processed
Mary kicked the ball into the net.
Patient
Agent
Action
Path
Recognizer Example
Construct:
Caused-Motion
RefExp:
Mary
Verb:
kicked
Constituent:
Patient
Constituent:
Path
Mary kicked the ball into the net.
Recognizer Example
caused  motion.m 
 patient.m

agent : 5
 agent.m
, 3category :


, 1category : Person
 

 scene : 4
 

 resolved  ref : Mary  resolved  ref :
action : 6

caused  motion.cm 
path.m

agent : 15

 source :

action.m





, 2tense : simpPast 
4 patient : {3}{cm1} ,{7} path :






 x  schema : kick 
action : 26

 goal :

 path : {7}

trajector : {cm1}
Recognizer Example
processed
Mary kicked the ball into the net.
According to the Constituent Graph,
The next constituent can either be the
Patient or the Path.
Agent
Patient
Action
Path
Recognizer Example
processed
Mary kicked the ball into the net.
Patient
Agent
Action
Path
Recognizer Example
Construct:
Caused-Motion
RefExp:
Mary
Verb:
kicked
Det
RefExp:
Det Noun
Constituent:
Path
Noun
Mary kicked the ball into the net.
Recognizer Example
caused  motion.m 
 patient.m

agent : 5
 agent.m
, 3category : ball 

, 1category : Person
 

 scene : 4
 

 resolved  ref : Mary  resolved  ref :
action : 6

caused  motion.cm 
path.m

agent : 15

 source :

action.m





, 2tense : simpPast 
4 patient : {3}{cm1} ,{7} path :






 x  schema : kick 
action : 26

 goal :

 path : {7}

trajector : {cm1}
Recognizer Example
processed
Mary kicked the ball into the net.
Patient
Agent
Action
Path
Recognizer Example
Construct:
Caused-Motion
RefExp:
Mary
Verb:
kicked
RefExp:
Det Noun
Spatial-Pred:
Prep RefExp
RefExp
Det
Noun
Prep
Det
Noun
Mary kicked the ball into the net.
Recognizer Example
caused  motion.m 
 patient.m

agent : 5
 agent.m
, 3category : ball 

, 1category : Person
 

 scene : 4
 

 resolved  ref : Mary  resolved  ref :
action : 6

caused  motion.cm 
path.m

agent : 15

 source :

action.m





, 2tense : simpPast 
4 patient : {3}{cm1} ,{7} path :






 x  schema : kick 
action : 26

 goal : net

 path : {7}

trajector : {cm1}
Resulting SemSpec
After analyzing the sentence, the following identities
are asserted in the resulting SemSpec:
Scene = Caused-Motion
Agent = Mary
Action = Kick
Patient = Path.Trajector = The Ball
Path = Into the net
Path.Goal = The net
Chunking
L3 ________________________S_____________S
L2 ____NP _________PP
VP
NP ______VP
L1 ____NP P_______NP
VP
NP ______VP
L0 D
0
N
P D N
N V-tns Pron Aux V-ing
the woman in the lab coat thought you were sleeping
1
2 3
Cite/description
4
5
6
7
8
9
Construction Recognizers
Form
Meaning
Form
Meaning
D,N <-> [Cloth
“you”<->[Addressee]
num:sg]
NP
You
NP
want to put
NP
a cloth
Like Abney:
One recognizer per rule
Bottom up and level-based
NP
on
Form
Meaning
PP$,N <-> [Hand
num:sg
poss:addr]
NP
your hand
?
Unlike Abney:
Check form and semantics
More powerful/slower than FSMs
Chunk Chart
• Interface between chunking and structure merging
• Each edge is linked to its corresponding semantics.
You
want to put
a cloth
on
your hand
?
Combining Partial Parses
• Prefer an analysis that spans the input utterance with
the minimum number of chunks.
• When no spanning analysis exists, however, we still
have a chart full of semantic chunks.
• The system tries to build a coherent analysis out of
these semantics chunks.
• This is where structure merging comes in.
Structure Merging
• Closely related to abductive inferential mechanisms like
abduction (Hobbs)
• Unify compatible structures (find fillers for frame roles)
• Intuition: Unify structures that would have been coindexed had the missing construction been defined.
• There are many possible ways to merge structures.
• In fact, there are an exponential number of ways to
merge structures (NP Hard). But using heuristics cuts
down the search space.
Structure Merging Example
Utterance:You used to hate to have the bib put on .
Before Merging:
[Addressee < Animate]
Bib < Clothing
num:sg
givenness:def
Caused-Motion-Action
Agent: [Animate]
Patient: [Entity]
Path:On
After Merging:
Caused-Motion-Action
Agent: [Addressee]
Patient:
Bib < Clothing
num:sg
givenness:def
Path:On
Semantic Density
• Semantic density is a simple heuristic to choose
between competing analyses.
• Density of an analysis = (filled roles) / (total roles)
• The system prefers higher density analyses because a
higher density suggests that more frame roles are filled
in than in competing analyses.
• Extremely simple / useful? but it certainly can be
improved upon.
Summary: ECG
• Linguistic constructions are tied to a model of simulated
action and perception
• Embedded in a theory of language processing
– Constrains theory to be usable
– Frees structures to be just structures, used in processing
• Precise, computationally usable formalism
– Practical computational applications, like MT and NLU
– Testing of functionality, e.g. language learning
• A shared theory and formalism for different cognitive
mechanisms
– Constructions, metaphor, mental spaces, etc.
Issues in Scaling up to Language
• Knowledge
– Lexicon (FrameNet)
– Constructicon (ECG)
– Maps (Metaphors, Metonymies) (MetaNet)
– Conceptual Relations (Image Schemas, X-schemas)
• Computation
– Representation (ECG)
• expressiveness, modularity, compositionality
– Inference (Simulation Semantics)
• tractable, distributed, probabilistic concurrent, context-sensitive
The Buy schema
schema Buy
subcase of Action
evokes Commercial-Transaction as ct
roles
self  ct.nucleus
buyer actor ct.customer ct.agent
goods  undergoer  ct.goods
The Sell schema
schema Sell
subcase of Action
evokes Commercial-Transaction as ct
roles
self  ct.nucleus
seller actor ct.vendor ct.agent
goods  undergoer  ct.goods
Extending Inferential Capabilities
• Given the formalization of the conceptual schemas
– How to use them for inferencing?
• Earlier pilot systems
– Used metaphor and Bayesian belief networks
– Successfully construed certain inferences
– But don’t scale
• New approach
– Probabilistic relational models
– Support an open ontology
Semantic Web
• The World Wide Web (WWW) contains a large and expanding
information base.
• HTML is accessible to humans but does not formally describe
data in a machine interpretable form.
• XML remedies this by allowing for the use of tags to describe
data (ex. disambiguating crawl)
• Ontologies are useful to describe objects and their interrelationships.
• DAML+OIL (http://www.daml.org) is an markup language based
on XML and RDF that is grounded in description logic and is
designed to allow for ontology development, transfer, and use on
the web.
The ICSI/Berkeley
Neural Theory of Language Project
Acquisition of
early constructions
ECG
Probabilistic Relation Inference
• Scalable Representation of
– States, domain knowledge, ontologies
• (Avi Pfeffer 2000, Koller et al. 2001)
• Merges relational database technolgy with Probabilistic
reasoning based on Graphical Models.
– Domain entities and relational entities
– Inter-entity relations are probabilistic functions
– Can capture complex dependencies with both simple and composite slot
(chains).
• Inference exploits structure of the domain
Status of PRMs
• Summer Project
– Build the basic PRM codebase/infrastructure
• Fall Project
– Design Coordinated PRM (CPRM)
– Build Interface for testing
• Spring/Summer 03
– Implement CPRM to replace Pilot System DBN
– Test CPRM for QA
• Related Work
– Probabilistic OWL (PrOWL)
– Probabilistic FrameNet
Articulating Projects
• FrameNet – NSF (with Colorado, USD)
• SmartKom – International Consortium
• EDU – European Media Lab
• Acquaint – ARDA (with SIMS, Stanford)
Conclusion
• NLU is essential to large, open domain QA.
– Much of the web in unstructured data
• Substantial Progress in Enabling Technologies
– Knowledge Representation/Inference Techniques
• Active Knowledge – X-schemas, Simulation Semantics
• Dealing With Uncertainty – PRM’s
• Combining Statistics and Structure.
• Conceptual Relations – Schemas, Metaphor, ECG
– Scaling Up
• CYC, Wordnet, Term-bases
• FrameNet, Semantic Web, MetaNet
• Open Source
• The goal of NLU can be realized, perhaps!
– Anyway, it’s time to try again.