What does language do? A sentence can evoke an imagined scene and resulting inferences: “Harry walked to the cafe.” CAFE – Goal of action = at cafe – Source = away from cafe – cafe = point-like location “Harry walked into the cafe.” CAFE – Goal of action = inside cafe – Source = outside cafe – cafe = containing location Language understanding (Utterance, Situation) Conceptual knowledge Linguistic knowledge Analysis Interpretation Language understanding: analysis & simulation “Harry walked to the cafe.” Utterance Lexicon Constructicon General Knowledge Belief State Analysis Process Schema walk Trajector Harry Cafe Goal cafe Semantic Specification Simulation Interpretation: x-schema simulation walker at goal Constructions can • specify which schemas and entities are involved in an event, and how they are related • profile particular stages of an event • set parameters of an event energy goal=home walker=Harry Harry is walking home. Traditional Levels of Analysis Pragmatics Semantics Syntax Morphology Phonetics “Harry walked into the cafe.” Pragmatics Semantics Utterance Syntax Morphology Phonetics Construction Grammar A construction is a form-meaning pair whose properties may not be strictly predictable from other constructions. (Construction Grammar, Goldberg 1995) Form Meaning block walk to Source Trajector Path Goal Form-meaning mappings for language Linguistic knowledge consists of form-meaning mappings: Meaning Form phonological cues word order intonation inflection event structure sensorimotor control attention/perspective social goals... Cafe Constructions as maps between relations Complex constructions are mappings between relations in form and relations in meaning. Form Mover + Motion before(Mover, Motion) “is” + Action + “ing” before(“is”, Action) suffix(Action, “ing”) Mover + Motion + Direction before(Motion, Direction) before(Mover, Motion) Meaning MotionEvent mover(Motion, Mover) ProgressiveAction aspect(Action, ongoing) DirectedMotionEvent direction(Motion, Direction) mover(Motion, Mover) Embodied Construction Grammar • Embodied representations – active perceptual and motor schemas – situational and discourse context • Construction Grammar – Linguistic units relate form and meaning/function. – Both constituency and (lexical) dependencies allowed. • Constraint-based (Unification) – based on feature structures (as in HPSG) – Diverse factors can flexibly interact. Representing image schemas schema name schema Source-Path-Goal roles source path goal trajector role name schema Container roles interior exterior portal boundary Boundary Source Trajector Interior Portal Goal Path Exterior These are abstractions over sensorimotor experiences. Inference and Conceptual Schemas • Hypothesis: – Linguistic input is converted into a mental simulation based on bodilygrounded structures. • Components: – Semantic schemas • image schemas and executing schemas are abstractions over neurally grounded perceptual and motor representations – Linguistic units • lexical and phrasal construction representations invoke schemas, in part through metaphor • Inference links these structures and provides parameters for a simulation engine Embodied Construction Grammar ECG (Formalizing Cognitive Linguisitcs) 1. Linguistic Analysis 2. Computational Implementation a. Test Grammars b. Applied Projects – Question Answering 3. Map to Connectionist Models, Brain 4. Models of Grammar Acquisition ECG Structures • Schemas – image schemas, force-dynamic schemas, executing schemas, frames… • Constructions – lexical, grammatical, morphological, gestural… • Maps – metaphor, metonymy, mental space maps… • Spaces – discourse, hypothetical, counterfactual… ECG Schemas schema <name> subcase of <schema> evokes <schema> as <local name> roles < local role >: <role restriction> constraints <role> ↔ <role> <role> <value> <predicate> schema Hypotenuse subcase of Line-Segment evokes Right-Tri as rt roles {lower-left: Point} {upper-right: Point} constraints self ↔ rt.long-side Source-Path-Goal; Container schema SPG subcase of TrajLandmark roles source: Place path: Directed–Curve goal: Place {trajector: Entity} {landmark: BoundedRegion} schema Container roles interior: Bounded-Region boundary: Curve portal: Bounded-Region Referent Descriptor Schemas schema RD roles category gender count specificty resolved Ref modifications schema RD5 // Eve roles HumanSchema Female one Known Eve Sweetser none ECG Constructions construction <name> subcase of <construction> constituents <name>:<construction> form constraints <name> before/meets <name> meaning: constraints // same as for schemas construction SpatialPP constituents prep: SpatialPreposition lm: NP form constraints prep meets lm meaning: TrajectorLandmark constraints selfm ↔ prep landmark ↔ lm.category Into and The CXNs construction Into subcase of SpatialPreposition form: WordForm constraints orth "into" meaning: SPG evokes Container as c constraints landmark ↔ c goal ↔ c.interior construction The subcase of Determiner form:WordForm constraints orth "the" meaning evokes RD as rd constraints rd.specificity “known” Two Grammatical CXNs construction DetNoun subcase of NP constituents d:Determiner n:Noun form constraints d before n meaning constraints selfm ↔ d.rd category ↔ n construction NPVP subcase of S constituents subj: NP vp: VP form constraints subj before vp meaning constraints profiled-participant ↔ subj construction ActiveSelfMotionPath subcase of ActiveMotionPath Construction WalkedVerb constituents subcase of {v: verb} PastPerfectiveVerb form constraints orth {pp:SpatialPP} "walked" form constraints meaning:WalkAction {v before pp} meaning:SelfMotionPathEvent constraints {spg ↔ pp} {profiled-participant ↔ mover} {profiled-process ↔ motion} {profiled-process ↔ v} Combined score determines best-fit • Syntactic Fit: – Constituency relations – Combine with preferences on non-local elements – Conditioned on syntactic context • Antecedent Fit: – Ability to find referents in the context – Conditioned on syntax match, feature agreement • Semantic Fit: – Semantic bindings for frame roles – Frame roles’ fillers are scored 0Eve1walked2into3the4house5 Constructs Schema Instances -------------------------------NPVP[0] (0,5) SelfMotionPathEvent Eve[3] (0,1) [1] ActiveSelfMotionPath HouseSchema[66] [2] (1,5) WalkAction[60] WalkedVerb[57] (1,2) Person[4] SpatialPP[56] (2,5) SPG[58] Into[174] (2,3) RD[177] ~ house DetNoun[173] (3,5) RD[5]~ Eve The[204] (3,4) House[205] (4,5) Unification chains and their fillers SelfMotionPathEvent[1].mover SPG[58].trajector WalkAction[60].walker RD[5].resolved-ref RD[5].category Filler: Person4 SelfMotionPathEvent[1] .landmark House[205].m RD[177].category SPG[58].landmark Filler:HouseSchema66 SpatialPP[56].m Into[174].m SelfMotionPathEvent[1].spg Filler: SPG58 WalkedVerb[57].m WalkAction[60].routine WalkAction[60].gait SelfMotionPathEvent[1] .motion Filler:WalkAction60 Summary: ECG • Linguistic constructions are tied to a model of simulated action and perception • Embedded in a theory of language processing – Constrains theory to be usable – Frees structures to be just structures, used in processing • Precise, computationally usable formalism – Practical computational applications, like MT and NLU – Testing of functionality, e.g. language learning • A shared theory and formalism for different cognitive mechanisms – Constructions, metaphor, mental spaces, etc. Embodied Compositional Semantics after Ellen Dodge edodge@berkeley.edu Questions • What is the nature of compositionality in the Neural Theory of Language? • How can it be best represented using Embodied Construction Grammar? Examples • • • • • • • He bit the apple He was bitten (by a toddler) He bit into the apple His white teeth bit into the apple. He shattered the window The window was shattered The window shattered Compositionality • Put the parts together to create the meaning of the whole. • Questions: – what is the nature of the parts? – How and why do they combine with one another? – What meaning is associated with this composition? Short answers • Parts = constructions, schemas • Combination = binding, unification • Meaning of the whole = simulation of unified parts Simulation parameters • Constructions unify to create semantic specification that supports a simulation • Two types of simulation parameters for event descriptions: – Event content – Event construal Summary • Parts = constructions, schemas • Combination = binding, unification • Meaning of the whole = simulation of the combined parts First example • He bit the apple. Schemas schema MotorControl subcase of Process roles Actor ↔ Protagonist Effector Effort Routine constraints Actor ← animate schema Contact subcase of SpatialRelation roles Entity1: entity Entity2: entity schema ForceTransfer evokes Conact as C roles Supplier ↔ C.entity1 Recipient ↔ C.entity2 Force schema MotorControl subcase of Process roles Actor ↔ Protagonist Effector Effort Routine constraints Actor ← animate schema ForceApplication subcase of MotorControl evokes ForceTransfer as FT roles Actor ↔ FT.Supplier ↔ Protagonist Acted Upon↔ FT.Recipient Effector Routine Effort ↔ FT.Force.amount Schema networks Contact MotorControl ForceTransfer Motion ForceApplication CauseEffect Effector Motion SelfMotion MotionPath Effector MotionPath Agentive Impact SPG SpatiallyDirectedAction SelfMotion Path Contact Verb Constructions Construction BITE1 subcase of Verb form: bite meaning: ForceApplication constraints: Effector ← teeth Routine ← bite // close mouth schema ForceApplication subcase of MotorControl evokes ForceTransfer as FT roles Actor ↔ FT.Supplier ↔ Protagonist Acted Upon ↔ FT.Recipient Effector Routine Effort ↔ FT.Force.amount Verb Constructions cxn BITE meaning: ForceApplication cxn GRASP meaning: ForceApplication cxn PUSH meaning: ForceApplication cxn SLAP meaning: AgentiveImpact cxn KICK meaning: AgentiveImpact cxn HIT meaning: AgentiveImpact schema MotorControl schema ForceApplication subcase of MotorControl schema Agentive Impact subcase of ForceApplication Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm CauseEffect schema schema CauseEffect subcase of ForceApplication; Process roles Causer ↔ Actor Affected ↔ ActedUpon ↔ Process.Protagonist Instrument ↔ Effector Schema Network Contact MotorControl Process ForceTransfer Motion ForceApplication CauseEffect Effector Motion SelfMotion MotionPath Effector MotionPath Agentive Impact SPG SpatiallyDirectedAction SelfMotion Path Contact Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes: EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm Schema Network Contact MotorControl Process ForceTransfer Motion ForceApplication CauseEffect Effector Motion SelfMotion MotionPath Effector MotionPath Agentive Impact SPG SpatiallyDirectedAction SelfMotion Path Contact Important points Compositionality does not require that each component contain different information. Shared semantic structure is not viewed as an undesirable redundancy Argument Structure Construction construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm Event Descriptor schema schema EventDescriptor roles EventType: Process ProfiledProcess: Process ProfiledParticipant: Entity ProfiledState(s): State SpatialSetting TemporalSetting Argument Structure Construction Construction ActiveTransitiveAction2 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Affected ↔ NPm Bindings with other cxns construction NPVP1 constituents: Subj: NP VP : VP form Constraints Subj f before VPf meaning: EventDescriptor ProfiledParticipant ↔ Subjm construction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant Affected ↔ NPm Bindings with other cxns Construction NPVP1 constituents: Subj: NP VP : VP form constraints Subj f before VPf meaning: EventDescriptor ProfiledParticipant ↔ Subjm schema EventDescriptor roles EventType ProfiledProcess ProfiledParticipant ProfiledState(s) SpatialSetting TemporalSetting construction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant Affected ↔ NPm Bindings with other cxns construction NPVP1 constituents: Subj: NP VP : VP form Constraints Subj f before VPf meaning: EventDescriptor ProfiledParticipant ↔ Subjm schema EventDescriptor roles EventType ProfiledProcess ProfiledParticipant ProfiledState(s) SpatialSetting TemporalSetting construction ActiveTransitiveAction2 subcase of VP constituents: V ; NP form: VF before NPF meaning: CauseEffect evokes; EventDescriptor as ED constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant Affected ↔ NPm Unification Meaning EventDescriptor EventType ProfiledProcess ProfiledParticipant Constructions NPVP1 CauseEffect causer affected TransitiveAction2 ForceApplication actor actedupon BITE NP2 ReferentDescriptor THE ReferentDescriptor NP1 HE APPLE Unification Meaning EventDescriptor EventType ProfiledProcess ProfiledParticipant Constructions NPVP1 CauseEffect causer affected TransitiveAction2 ForceApplication actor actedupon BITE NP2 ReferentDescriptor THE ReferentDescriptor resolved referent NP1 HE APPLE Unification Meaning EventDescriptor eventtype ProfiledProcess ProfiledParticipant Constructions NPVP1 CauseEffect causer affected TransitiveAction2 Verb ForceApplication actor actedupon BITE NP2 ReferentDescriptor THE ReferentDescriptor resolved referent NP1 HE APPLE Unification Meaning EventDescriptor eventtype ProfiledProcess ProfiledParticipant Constructions NPVP1 subj CauseEffect causer affected TransitiveAction2 ForceApplication actor actedupon BITE NP2 ReferentDescriptor THE ReferentDescriptor NP1 HE APPLE Unification Meaning EventDescriptor eventtype ProfiledProcess ProfiledParticipant Constructions NPVP1 CauseEffect causer affected TransitiveAction2 NP ForceApplication actor actedupon BITE NP2 ReferentDescriptor THE ReferentDescriptor NP1 HE APPLE Semantic Specification He bit the apple EventDescriptor eventtype ProfiledProcess ProfiledParticipant RD27 category CauseEffect causer affected ForceApplication actor actedupon routine bite effector teeth Person Apple RD55 category Simulation He bit the apple CauseEffect ForceApplication Protagonist = Affected ↔ ActedUpon Process Protagonist = Causer ↔ Actor Simulation He bit the apple CauseEffect ForceApplication Protagonist = Affected ↔ ActedUpon Process Protagonist = Causer ↔ Actor Passive voice He was bitten (by a toddler) Argument Structure Construction He was bitten (by a toddler) construction PassiveTransitiveAction2 subcase of VP constituents: V : PassiveVerb (PP: agentivePP) form constraints: VF before PPF meaning: CauseEffectAction evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Affected ↔ ED.ProfiledParticipant FA ↔ Vm Causer ↔ FA.Actor Affected ↔ FA.ActedUpon Causer ↔ PP.NPm Semantic Specification He was bitten (by a toddler) EventDescriptor eventtype ProfiledProcess ProfiledParticipant RD27 category CauseEffect causer affected ForceApplication actor actedupon routine bite effector teeth Person Person RD48 category Simulation He was bitten (by a toddler) CauseEffect Action = Bite Protagonist = Affected ↔ ActedUpon Effect = Process Protagonist = Causer ↔ Actor Variations on a theme • He shattered the window • The window was shattered • The window shattered Verb Construction -- shatter Construction SHATTER1 subcase of Verb form: shatter meaning: StateChange constraints: Initial :: Undergoer.state ← whole Final :: Undergoer.state ← shards schema StateChange subcase of Process roles Undergoer ↔ Protagonist Argument Structure Construction He shattered the window construction ActiveTransitiveAction3 subcase of VP constituents: V : verb NP: NP form constraints: VF before NPF meaning: CauseEffect evokes: EventDescriptor as ED; StateChange as SC constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Causer ↔ ED.ProfiledParticipant SC ↔ Vm Affected ↔ SC.Undergoer Affected ↔ NPm Semantic Specification He shattered the window EventDescriptor eventtype ProfiledProcess ProfiledParticipant RD27 category CauseEffect causer affected StateChange Undergoer state “wholeness” Person Window RD189 category Simulation He shattered the window CauseEffect Action Protagonist = Affected ↔ Undergoer Protagonist = Causer Process Argument Structure Construction The window was shattered construction PassiveTransitiveAction3 subcase of VP constituents: V : PassiveVerb (PP: agentivePP) form constraints: VF before NPF meaning: CauseEffect evokes: EventDescriptor as ED; StateChange as SC constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Affected ↔ ED.ProfiledParticipant SC ↔ Vm Affected ↔ SC.Undergoer Causer ↔ PP.NPm Semantic Specification The window was shattered EventDescriptor eventtype ProfiledProcess ProfiledParticipant CauseEffect causer affected Window StateChange Undergoer state “wholeness” RD175 category Simulation The window was shattered CauseEffect Action Protagonist = Affected ↔ Undergoer Protagonist = Causer Process Argument Structure Construction The window shattered construction ActiveIntransitiveAction1 subcase of VP constituents: V : verb form meaning: Process evokes: EventDescriptor as ED; StateChange as SC constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Protagonist ↔ ED.ProfiledParticipant SC ↔ Vm Protagonist ↔ SC.Undergoer Semantic Specification The window shattered EventDescriptor eventtype ProfiledProcess ProfiledParticipant Process protagonist Window StateChange Undergoer state “wholeness” RD177 category Simulation The window shattered Process Protagonist = Undergoer Process Some more variations on a theme • He bit the apple • He bit into the apple • His white teeth bit into the apple. Argument Structure Construction He bit into the apple construction ActiveEffectorMotionPath2 subcase of VP constituents: V : verb PP: Spatial-PP form constraints: VF before PPF meaning: EffectorMotionPath evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Actor ↔ ED.ProfiledParticipant FA ↔ Vm Actor ↔ FA.Actor Effector ↔ FA.Effector // INI Target ↔ FA.ActedUpon SPG ↔ PPm Target ↔ PPm .Prep.LM Schema schema EffectorMotionPath subcase of EffectorMotion subcase of SPG // or evokes SPG roles Actor ↔ MotorControl.protagonist Effector ↔ SPG.Tr ↔ M.Mover ↔ Motion.protagonist Target ↔ SPG.Lm Schema Network Contact MotorControl Process ForceTransfer Motion ForceApplication CauseEffect Effector Motion SelfMotion MotionPath Effector MotionPath Agentive Impact SPG SpatiallyDirectedAction SelfMotion Path Contact Argument Structure Construction He bit into the apple construction ActiveEffectorMotionPath2 subcase of VP constituents: V : verb PP: Spatial-PP form constraints: VF before PPF meaning: EffectorMotionPath evokes: EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Actor ↔ ED.ProfiledParticipant FA ↔ Vm Actor ↔ FA.Actor Effector ↔ FA.Effector // INI Target ↔ FA.ActedUpon SPG ↔ PPm Target ↔ PPm .Prep.LM EffectorMotionPath Action Protagonist = Actor Effector Motion Source Protagonist = Effector Path Goal Argument Structure Construction He bit into the apple construction ActiveEffectorMotionPath2 subcase of VP constituents: V : verb PP: Spatial-PP form constraints: VF before PPF meaning: EffectorMotionPath evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Actor ↔ ED.ProfiledParticipant FA ↔ Vm Actor ↔ FA.Actor Effector ↔ FA.Effector // INI Target ↔ FA.ActedUpon SPG ↔ PPm Target ↔ PPm .Prep.LM Simulation: He bit into the apple Action Protagonist = Actor Effector Motion Source Protagonist = Effector Path Goal Argument Structure Construction His white teeth bit into the apple construction ActiveEffectorMotionPath3 subcase of VP constituents: V : verb PP: Spatial-PP form constraints: VF before PPF meaning: EffectorMotionPath evokes; EventDescriptor as ED; ForceApplication as FA constraints: {Selfm ↔ ED.EventType} {Vm ↔ ED.ProfiledProcess} Effector ↔ ED.ProfiledParticipant FA ↔ Vm Actor ↔ FA.Actor // INI Effector ↔ FA.Effector Target ↔ FA.ActedUpon SPG ↔ PPm Target ↔ PPm .Prep.LM Simulation: His white teeth bit into the apple Action Protagonist = Actor Effector Motion Source Protagonist = Effector Path Goal Non-agentive biting • He landed on his feet, hitting the narrow pavement outside the yard with such jarring impact that his teeth bit into the edge of his tongue. [BNC] • The studs bit into Trent's hand. [BNC] • His chest burned savagely as the ropes bit into his skin. [BNC] Schema Network Contact MotorControl Process ForceTransfer Motion ForceApplication CauseEffect Effector Motion SelfMotion MotionPath Effector MotionPath Agentive Impact SPG SpatiallyDirectedAction SelfMotion Path Contact Simulation: His teeth bit his tongue Motion Protagonist = Mover Source Path Goal Summary • Small set of constructions and schemas • Composed in different ways • Unification produces specification of parameters of simulation • Sentence understanding is simulation • Different meanings = different simulations Concluding Remarks • Complexity • Simulation Concluding Remarks • • • • Complexity Simulation Language understanding is simulation Simulation involves activation of conceptual structures • Simulation specifications should include: – which conceptual structures to activate – how these structures should be activated Extra slides follow: Prototypes and extensions? CauseMotion Path: • He threw the ball across the room • He kicked the ball over the table • He sneezed the napkin off the table • [He coughed the water out of his lungs] Key points • In prototypical verb-argument structure construction combinations, verb meaning is very similar to argument structure meaning. • Verbs whose meaning partially overlaps that of a given argument structure constructions may also co-occur with that argument structure construction • These less prototypical combinations may motivate extensions to the central argument structure constructions