Bailey Model
• Showed how simple hand action verbs may be acquired
based on motor control schemas and parameterization.
• Used Model Merging which allowed for
– One-shot learning (Maps to recruitment learning)
• Could label and perform actions (given a command, world state
pair)
• Uses parameters over motor-control schemas as inductive bias
• Limitations
– Inference
• Connections between events
– Abstract uses
• Event Structure
• Frames
• Metaphor
Event Structure-1
Srini Narayanan
CS182/CogSci110/Ling109
Spring 2006
snarayan@icsi.berkeley.edu
Active representations
• Many inferences about actions derive from what we
know about executing them
• Representation based on stochastic Petri nets
captures dynamic, parameterized nature of actions
walker at goal
energy
walker=Harry
goal=home
Walking:
bound to a specific walker with a
direction or goal
consumes resources (e.g., energy)
may have termination condition
(e.g., walker at goal)
ongoing, iterative action
X-Schema Extensions to Petri
Nets
• Parameterization
– x-schemas take parameter values (speed, force)
• Walk(speed = slow, dest = store1)
• Dynamic Binding
– X-schemas allow run-time binding to different
objects/entities
• Grasp(cup1), push(cart1)
• Hierarchical control and durative transitions
– Walk is composed of steps which are composed of
stance and swing phases
• Stochasticity and Inhibition
– Uncertainties in world evolution and in action selection
Event Structure in Language
• Commonplace discourse fragments/blurbs
– Low inflation is starting to pull France out of recession.
– E3 continue to push Iran to uphold IAEA obligations.
– US Economy on the verge of falling back into recession
after moving forward on an anemic recovery.
– Indian Government stumbling in implementing
Liberalization plan.
– Moving forward on all fronts, we are going to be ongoing
and relentless as we tighten the net of justice.
– The Government is taking bold new steps. We are
loosening the stranglehold on business, slashing tariffs and
removing obstacles to international trade.
Event Structure in Language
• Fine-grained
• Rich Notion of Contingency Relationships.
– Phenomena: Aspect, Tense, Force-dynamics,
Modals, Counterfactuals
• Event Structure Metaphor:
– Phenomena: Abstract Actions are
conceptualized in Motion and Manipulation
terms.
– Schematic Inferences are preserved.
Aspect
• Aspect is the name given to the ways
languages describe the structure of events
using a variety of lexical and grammatical
devices.
– Viewpoints
• is walking, walk
– Phases of events
• Starting to walk, walking, finish walking
– Inherent Aspect
• run vs cough vs. rub
– Composition with
• Temporal modifiers, tense..
• Noun Phrases (count vs. mass) etc..
Grammatical Aspect
Languages have grammatical constructions
that indicate the type of situation described.
•
•
•
•
•
•
Progressive: She was running home.
Perfect: I’ve had a wonderful evening.
Inceptive: She started knitting.
Prospective: She’s about to leave.
Resumptive: Peace talks resume.
Iterative: They ran twice around the track.
Phases, Viewpoints, and
Aspects
•
•
•
•
•
•
•
•
•
•
•
•
John is walking to the store.
John is about to walk to the store.
John walked to the store.
John started walking to the store.
John is starting to walk to the store.
John has walked to the store.
John has started to walk to the store.
John is about to start walking to the store.
John resumed walking to the store.
John has been walking to the store.
John has finished walking to the store.
John almost walked to the store.
A Walk X-schema
A Climb X-schema
Common Patterns
START
FINISH
Posture = Up
Energy Available
Ready
Dest = top(obj)
Loop
BEGIN
Execute (subschema)
END
At Dest
Done
Posture = Up
Ground ok
Ready
Loop
BEGIN
Execute(subschema)
END
At Dest
Done
Pre-motor Versus Motor Cortex
Whenever we perform a complex motor movement, such as
picking up a glass and taking a drink, at least two distinct parts of
the brain are activated:
The motor cortex, where there are neural ensembles that control
“motor synergies” — relatively simple actions like opening or
closing the hand, flexing or extending the elbow, turning the wrist,
and so on.
Complex motor schemas, however, are carried out by neural
circuitry in the pre-motor cortex, circuitry connected via neural
bindings to the appropriate synergies in the motor cortex.
In picking up a glass and taking a drink, both pre-motor cortex and
motor cortex are activated, as are binding connections between
them.
The Controller X-Schema
In modeling complex premotor action schemas, we make
the following hypothesis
All complex premotor schemas are compositions of a single
type of structure.
The same neural computational structure, when
disengaged from the motor cortex, can characterize aspect
(that is, event structure) in the world’s languages.
When dynamically active, this structure can compute the
logic of aspect.
We call this structure the “Controller X-schema.”
The Structure of the Controller X-Schema
•Initial State
•Starting Phase Transition
•Precentral State
•Central Phase Transition (either instantaneous,
prolonged, or ongoing)
•Postcentral State*
•Ending Phase Transition
•Final State
Postcentral Options:
*A check to see if a goal state has been achieved
*An option to stop/resume
*An option to iterate or continue the main process
-Narayanan, 1997
A Schema Controller
iterate
Ready
Start
Cancel
Process
Finish
interrupt
resume
Done
Suspend
• An active controller that sends signals to the embedded schema
and transitions based on signals from the embedded schema.
• Useful for higher level monitoring and coordination of actions.
A Generic Process Schema
iterate
Ready
Start
Cancel
Process
Finish
interrupt
resume
Done
Suspend
• Part of Conceptual Structure.
• Generalizes over actions and events. Has internal state and
models evolution of processes.
Aspects of (Climb)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
About to + (Climb)
(Prospective)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Cancel + (Climb)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Start + (Climb)-ING
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Be + (Climb)-ING
(Progressive)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Suspend (Climb)-ING
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Resumed + (Climb)-ING
(Resumptive)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Finish (End) + (Climb)-ING
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Have + (Climb)-ed (Perfect)
Iterate
Ready
Start
Process
Finish
Done
resume
interrupt
Cancel
Energy
Ready
Standing
Suspend
BINDINGS
Hold
Find hold
Stabilize
Pull(self)
On top
Embedding: Has Started (to X)
Ready
Start
Process
Finish
Done
resume
interrupt
R
S
C
P
i
F
r
D
Suspend
S
X-Schema for X with bindings
Phasal Aspect Maps to the
Controller
Inceptive (start, begin)
Ready
Start
Iterative (repeat)
Iterate
Process
Finish
interrupt
resume
Cancel
Resumptive(resume)
Done
Suspend
Completive (finish, end)
Embedding: About to start (X)
Ready
Start
Process
Finish
resume
interrupt
R
S
C
P
i
F
r
D
S
X-Schema for X with bindings
Suspend
Done
Embedding: Has Started (to X)
Ready
Start
Process
Finish
Done
resume
interrupt
R
S
C
P
i
F
r
D
Suspend
S
X-Schema for X with bindings
Begins and Ends
• “This is not the end. It is not even the
beginning of the end. But it is, perhaps,
the end of the beginning."
– Speech given at the Lord Mayor's Luncheon,
Mansion House, London, November 10, 1942.
Winston Churchill
Embedding: It’s not this (the end)
Ongoing
Finish
Done
X-Schema for X with bindings
Embedding: It’s not this (beginning
of the end)
Ongoing
Finish
S
R
C
P
i
Done
F
r
D
S
X-Schema for X with bindings
Embedding: But this (The end of
the beginning)
Ready
Start
Process
Finish
Done
resume
interrupt
R
S
C
P
i
F
r
D
Suspend
S
X-Schema for X with bindings
Inherent Aspect (Aksionsart)
• Vendler-Dowty-Taylor (VDT) classification
– Events and States
– Events can be
• Punctual or Durative
• Atelic or Telic
– States satisfy the downward entailment
property
• If a state holds in some interval, it holds in all subintervals of that interval.
Inherent Aspect
• Much richer than traditional Linguistic
Characterizations (VDT (durative/atomic,
telic/atelic))
• Action patterns
– one-shot, repeated, periodic, punctual
– decomposition: concurrent, alternatives, sequential
• Goal based schema enabling/disabling
• Generic control features;
– interruption, suspension, resumption
• Resource usage
Basic Event X-schemas
•
•
•
•
•
•
State
Event Transition
Simple Event
Simple Action
Complex Event/Process
Complex State
Aspectual Types
Other Transitions in the Controller
may be coded
• Lexical items may code interrupts
– Stumble is an interrupt to an ongoing walk
• A combination of grammatical and aktionsart may code
of the controller phases
–
–
–
–
–
Ready to walk : Prospective
Resuming his run: Resumptive
Has been running: Embedded progressive
About to Finish the painting: Embedded Completive.
Canceling the meeting vs. Aborting the meeting.
Interaction of Aspect with Tense
• Reichenbach’s system uses three pointers
– Speech Time (S)
– Reference Time (R)
– Event Time (E)
• Tense is a partial ordering relation
between the pointers
– Simple Past E < R, E < S
– Perfect E < R < S
Viewpoint Aspect
(Perfective/Imperfective)
Perfective/Imperfective
Perfective
Imperfective
Simulation and Reference
Interval
Perfective
Imperfective
Levels of Granularity
• Events can be construed at different levels
of granularity based on various contextual
factors.
– In 1991, McEnroe injured his knee while
playing tennis.
– This morning, I injured my knee while playing
tennis.
– He is coughing (normal time scale vs. slowmotion film time scale).
Summary of Aspect Results
•
Controller mediates between linguistic markings and individual event/verbal xschemas (Cogsci99, Coling2002)
• Captures regular event structure; inspired by biological control theory
• Flexible: specific events may require only a subset of controller; interaction of underlying xschemas, linguistic markers and hierarchical abstraction/ decomposition of controller
accounts for wide range of aspectual phenomena.
•
Important aspectual distinctions, both traditional and novel, can be precisely
specified in terms of the interaction of x-schemas with the controller (CogSci97,
CogSci98, AAAI99, IJCAI99, CogSci04, CogLing2005):
•
•
•
•
•
•
•
stative/dynamic, durative/punctual: natural in x-schemas
telic processes: depletion of resources
continuous processes: consumption of resources
temporary/effortful states; habituals
dynamic interactions with tense, nominals, temporal modifiers
incorporation of world knowledge, pragmatics
Ongoing Work: Simulation Semantics and Tense-Aspect (with Laura Michaelis)
Simulation hypothesis
We understand utterances by mentally simulating
their content.
– Simulation exploits some of the
same neural structures activated during performance,
perception, imagining, memory…
– Linguistic structure parameterizes the simulation.
• Language gives us enough information to simulate
Simulation Semantics
• BASIC ASSUMPTION: SAME REPRESENTATION FOR
PLANNING AND SIMULATIVE INFERENCE
– Evidence for common mechanisms for recognition and
action (mirror neurons) in the F5 area (Rizzolatti et al (1996),
Gallese 96, Buccino 2002, Tettamanti 2004) and from motor
imagery (Jeannerod 1996)
• IMPLEMENTATION:
– x-schemas affect each other by enabling, disabling or
modifying execution trajectories. Whenever the
CONTROLLER schema makes a transition it may set, get,
or modify state leading to triggering or modification of other
x-schemas. State is completely distributed (a graph marking)
over the network.
• RESULT: INTERPRETATION IS IMAGINATIVE SIMULATION!
A Precise Notion of Contingency
Relations
Activation:
Executing one schema causes the enabling, start or continued execution
of another schema. Concurrent and sequential activation.
Inhibition:
Inhibitory links prevent execution of the inhibited x-schema by
activating an inhibitory arc. The model distinguishes between concurrent
and sequential inhibition, mutual inhibition and aperiodicity.
Modification:
The modifying x-schema results in control transition of the modified
xschema. The execution of the modifying x-schema could result in the
interruption, termination, resumption of the modified x-schema.
General and Domain Knowledge
• Conceptual Knowledge and Inference
–
–
–
–
Embodied
Language and Domain Independent
Powerful General Inferences
Ubiquitous in Language
• Domain Specific Frames and Ontologies
– FrameNet, OWL ontologies
• Metaphor links domain specific to general
– E.g., France slipped into recession.
Frames
• Frames are conceptual structures that may be culture
specific
• Words evoke frames
– The word “talk” evokes the Communication frame
– The word buy (sell, pay) evoke the Commercial
Transaction (CT) frame.
– The words journey, set out, schedule, reach etc. evoke the
Journey frame.
• Frames have roles and constraints like schemas.
– CT has roles vendor, goods, money, customer.
• Words bind to frames by specifying binding patterns
– Buyer binds to Customer, Vendor binds to Seller.
Event Frames
Event frames have temporal structure that
comprises of the controller event structure
and generally have constraints on what
precedes them, what happens during
them, and what state the world is in once
the event has been completed.
Sample Event Frame:
Commercial Transaction
Initial state:
Vendor has Goods, wants Money
Customer wants Goods, has Money
Transition:
Vendor transmits Goods to Customer
Customer transmits Money to Vendor
Final state:
Vendor has Money
Customer has Goods
Sample Event Frame:
Commercial Transaction
Initial state:
Vendor has Goods, wants Money
Customer wants Goods, has Money
Transition:
Vendor transmits Goods to Customer
Customer transmits Money to Vendor
Final state:
Vendor has Money
Customer has Goods
(It’s a bit more complicated than that.)
Partial Wordlist for Commercial
Transactions
Verbs:
pay, spend, cost, buy, sell,
charge
Nouns:
cost, price, payment
Adjectives: expensive, cheap
Meaning and Syntax
 The various words that evoke this frame
introduce the elements of the frame in
different ways.
 The identities of the buyer, seller, goods and
money
 Information expressed in sentences
containing these words occurs in different
places in the sentence depending on the
word.
Language understanding: analysis &
simulation
construction WALKED
form
selff.phon  [wakt]
meaning : Walk-Action
constraints
selfm.time before Context.speech-time
selfm..aspect  encapsulated
“Harry walked into the cafe.”
Utterance
Analysis Process
Constructions
Lexicon
General
Knowledge
Semantic
Specification
Belief State
CAFE
Simulation