University of Rochester
 Activities
 Abductive Inference of Multi-Agent Interaction
 Capture the Flag Data Collection
 Representing Beliefs & Goals of Multiple Agents
 Modal Markov Logic
 Recognizing Indoor Activities using Multi-Modal
Data
 Fusing RFID and Machine Vision
DEPARTMENT of COMPUTER SCIENCE
Multi-Agent Interaction
 Many agent behaviors can only be
understood in the context of the
actions of other agents
 Exercising?
 Being chased?
 Chasing someone?
 Location alone provides a surprisingly rich source of
information about behavior
 GPS data can be used to learn a probabilistic model of a
individual’s common activities (Liao, Fox, & Kautz 2007)
 Goal: learn models of groups and interactive activities
 Relational learning problem: ideal for ML
 Needed: dataset of competitive & cooperative interaction
DEPARTMENT of COMPUTER SCIENCE
Capture the Flag
 Capture the Flag Data Collection
 UR campus
 Up to 150 x 300 m area
 Complex topology
 14 players, 8 games
 GPS loggers
 Accuracy varies 1-9 m
 Average game 12 m
DEPARTMENT of COMPUTER SCIENCE
Start of Game
DEPARTMENT of COMPUTER SCIENCE
End of Game
1.
2.
3.
4.
red & orange
guarded by
green
green leaves
prisoners
violet releases
red & orange
red captures
flag
DEPARTMENT of COMPUTER SCIENCE
Ground Truth
 For supervised learning
methods, need to create a
labeled training set
 First attempt: record voice
annotations from players
 Failed: players too involved
to accurately comment on
their actions
 Second attempt: post-hoc
annotation
 Created general annotation
tool for relations over GPS
streams
DEPARTMENT of COMPUTER SCIENCE
Supervised Weight Learning
 Discrete features calculated from GPS streams
 Supervised learning applied to simple 2-slice model
 Precision: 46% (second by second)
 Recall 64%
 12 hours to label 1 hour of training data
DEPARTMENT of COMPUTER SCIENCE
Observations
 Humans can accurately perceive interactive
behaviors
 High agreement between annotators
 GPS noise often obscures geometric details
 Reasoning about intention over extended
temporal context disambiguates action
DEPARTMENT of COMPUTER SCIENCE
Year 2 Goals
 Improve quality of data (features) using
physical constraints
 Hard constraints: walls
 Soft constraints: paths
 CRF “snapping” tool
 Model long temporal dependencies
 Unsupervised learning: discover behaviors,
tactics, strategies
DEPARTMENT of COMPUTER SCIENCE
Representing Beliefs & Goals
of Multiple Agents
 Abduction often requires reasoning about the
establishment of “propositional attitudes”
 Belief, desire, intention, commitment, …
 Example: principles of communication:
 If A tells B that P, then A believes P.
 If A tells B that P, then B will believe that A wants B
to believe P.
 If A is cooperative with B, and B wants P, then A
will want P.
 Such principles are defeasible
DEPARTMENT of COMPUTER SCIENCE
Modal Operators
 In logic
 Predicates relate one object to another
 Modal operators relate objects (agents) to
propositions (sentences)
 Different modalities can be axiomatically
characterized
 Deductive closure:
 Transitivity:
B(a, P)  B(a, P  Q)  B(a,Q)
B(a, P)  B(a, B(a, P))
DEPARTMENT of COMPUTER SCIENCE
Modal Operators in Markov Logic
 ML defines a probability distribution over
propositional truth assignments
 Idea: define probability distribution over
assignments that are modally consistent
Non-modal atoms
Modal atoms
Modal consistency check
DEPARTMENT of COMPUTER SCIENCE
Inference
 Complexity of consistency check
 Depends on target modal logic
 Belief (KD45):
 Unbounded nesting: PSPACE-complete
 Bounded nesting: NP-complete
 Modal Markov Logic Inference
 Rejection sampling
 Optimizations
 Cache g(M)
 Compute g(M) incrementally
DEPARTMENT of COMPUTER SCIENCE
Year 2 Goals
 Implement MML in Alchemy
 Applications
 Understanding indirect speech acts
 Capture the flag
 Establishing knowledge by perception
 Representing degrees of belief
 Functional modal operators
B(a, B(b, P)  0.9)  0.75
DEPARTMENT of COMPUTER SCIENCE