Using Propositional Graphs for
Soft Information Fusion
Michael Prentice and Stuart C. Shapiro
Department of Computer Science and Engineering,
Center for Cognitive Science,
Center for Multisource Information Fusion
University at Buffalo, Buffalo, NY 14260
{mjp44,shapiro}@buffalo.edu
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
1 / 26
Outline
1
Overview
2
Syntax
3
Frame Semantics
4
Propositional Graphs
5
Propositionalizer
6
Fusing Propositional Graphs
7
Conclusions, Future Work, Acknowledgments
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
2 / 26
Overview
Soft Information Fusion
Combine information from multiple natural language messages.
Counterinsurgency domain.
Message 12: 01/13/07 — Cell phone call from unidentified male in
Adhamiya to unidentified male in Ramadi lasted just five
seconds with the words “my brother sends greetings” spoken
by originator of call.
Message 14 extract: 01/14/07 — Originator of 1/13/07 cell phone call
to Ramadi from Adhamiya has now been identified as Sufian
Mashhadan. The recipient has been identified as Ziyad
al-Obeidi.
Some problems:
Embedded in many messages.
Same cell phone calls?
from vs. originator
to vs. recipient
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
3 / 26
Overview
Tractor
1
Syntactic Processing
Produce dependency graph
2
Propositionalizer
Map dependency graph to propositional graph
3
Contextual Enhancement
Add relevant ontological and other background information
4
Fuse contextually enhanced propositional graphs
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
4 / 26
Syntax
Dependency Structure of Sentences
Clauses
Head: Verb
Dependents:
Adverbs
Noun phrases
Prepositional phrases
Prepositional Phrases
Head: Preposition
Dependents: Noun phrase
Noun Phrases
Head: Noun
Dependents:
Determiner
Adjectives
Nouns
Prepositional phrases
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
5 / 26
Syntax
Dependency Parse for Message 12
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
6 / 26
Frame Semantics
Caseframes for Semantics
Based on “The Case for Case” [Fillmore, 1968]
and The Berkeley FrameNet Project [Baker, Fillmore, & Lowe, 1998]
Frame
schematic representation of a situation
with a set of participants
and conceptual roles.
Eliminates syntactic differences.
E.g.
Sufian called Ziyad.
Ziyad was called by Sufian.
a call from Sufian to Ziyad
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
7 / 26
Frame Semantics
Structure of Frames
Frame
set of frame elements
semantic roles filled by entities of certain types.
Filler can be another frame.
Core Frame Elements:
“conceptually necessary component”
Non-Core (Peripheral) Frame Elements:
Does not “introduce additional, independent or distinct events”
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
8 / 26
Frame Semantics
Contacting Frame
Core Elements
Communicator, type: Sentient
Addressee, type: Sentient
Communication
...
Non-Core Elements
Medium
Time, type: Time
...
E.g.
Time: 2007-01-13
Medium: cell phone
Communicator: Sufian Mashhadan
Addressee: Ziyad al-Obeidi
Communication: “My brother sends greetings.”
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
9 / 26
Frame Semantics
Frame for Message 12
Time: 2007-01-13
Communicator: Some Sentient
Message:
Duration: 5 seconds
Entity:
Medium: Cell phone
Communicator: Some male in Adhamiya
Addressee: Some male in Ramadi
Communication: “My brother sends greetings.”
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
10 / 26
Propositional Graphs
Propositional Graphs
A Knowledge Representation (KR)
labeled directed acyclic graph
with formal syntax
and formal semantics
Atomic Node
Corresponds to individual constant
No outgoing arcs
Denotes entity in domain
Molecular Node
Corresponds to functional term
Outgoing arcs labeled with argument position (role)
Compositional semantics
Denotes
Entity in domain
Possibly a proposition
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
11 / 26
Propositional Graphs
Basic Principles
Comprehensiveness
Every entity, person, category, property, value, etc.
Every proposition, belief, fact, etc.
is represented by a node.
Uniqueness Principle
No two nodes with same ID.
No two molecular nodes with same labeled arcs to same nodes.
No two nodes representing the (obviously) same domain entity.
Base case for fusing propositional graphs
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
12 / 26
Propositional Graphs
SNePS KR System
Logic-based
Frame-based
Propositional graph-based
Each view supports a different style of inference
(not covered in this talk)
Uses Arbitrary and Indefinite terms [Shapiro, 2004]
E.g.,
(some ind1 () (Instance ind1 Male)
(Location ind1 Ramadi))
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
13 / 26
Propositional Graphs
SNePS Graph::Frame::Logic
Atomic node :: atomic symbol :: individual constant
Node ID = atomic symbol
Molecular node :: frame :: functional term
Node ID = wfti [!]
Arc label :: frame slot :: argument position
Uniqueness Principle
No two nodes with same ID
No two molecular nodes with same labeled arcs to same nodes
Base case for fusing SNePS propositional graphs
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
14 / 26
Propositional Graphs
Lowest Parts of Propositional Graph for Message 12
some Male in Ramadi; some Male in Adhamiya; some Sentient; 5 seconds
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
15 / 26
Propositional Graphs
Contacting in Propositional Graph for Message 12
Some Male in Ramadi communicated “my brother sends greetings”
to some Male in Adhamiya.
Note 3-ary relation.
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
16 / 26
Propositional Graphs
Cell Phone in Propositional Graph for Message 12
The contacting was via cell phone and lasted 5 seconds.
Note nesting of functional terms.
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
17 / 26
Propositional Graphs
Propositional Graph for Sender of Message 12
Some Sentient said that it lasted 5 seconds.
Note pedigree information.
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
18 / 26
Propositional Graphs
Final Propositional Graph for Message 12
Message 12 was sent on 2007-01-13
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
19 / 26
Propositionalizer
Propositionalizer
FrameNet DB
Dependencies &
Named Entities
FrameNet Tagger
Syntax-Semantics
Mapper
Syntax-Semantics
Translation Rules
Propositional
Graph
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
20 / 26
Propositionalizer
Example Syntax-Semantics Mappings
Sample mappings from dependency graph to propositional graph.
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
21 / 26
Fusing Propositional Graphs
Propositional Graph for Message 14
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
22 / 26
Fusing Propositional Graphs
Fused Propositional Graph
Given common Medium, Time, Locations,
Data Association should fuse calls, communicators, and addressees.
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
23 / 26
Conclusions, Future Work, Acknowledgments
Conclusions
Tractor
analyzes an English message
builds a dependency graph
and then a propositional graph.
Propositional Graphs
Are a Knowledge Representation
Based on FrameNet frames
Can represent n-ary relations
Can represent meta-information
Can represent pedigree
Support Data Association by graph matching
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
24 / 26
Conclusions, Future Work, Acknowledgments
Future Work
Automating tagging of words with FrameNet frames
Automating Syntax-Semantics Mapper
Designing test and evaluation measures
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
25 / 26
Conclusions, Future Work, Acknowledgments
Acknowledgements
Multidisciplinary University Research Initiative (MURI)
Grant (Number W911NF-09-1-0392) for “Unified Research on
Network-based Hard/Soft Information Fusion”
Issued by the US Army Research Office (ARO) under the program
management of Dr. John Lavery.
Center for Multisource Information Fusion
Dr. James Llinas, Executive Director
Contact information
Michael Prentice (email: mjp44@buffalo.edu)
Dr. Stuart C. Shapiro (email: shapiro@buffalo.edu)
M. Prentice & S. C. Shapiro (UB)
Fusion 2011
26 / 26