Recognizing Activities of
Daily Living
from Sensor Data
Henry Kautz
Department of Computer Science
University of Rochester
Activity Recognition

Much recent interest in recognizing human
activity from heterogeneous sensor data





Motion sensors
GPS
RFID
Video
Compelling applications


Military / security operations (e.g. ASSIST)
Smart homes & offices
Gathering data on indoor
activities
Interpreting RFID Data
(using Switching HMM)
Gathering Multi-view Video
Interpreting Video
Computing scene statistics
Computing object statistics
Ai = activity
Oi = object
Si = scene statistic
Di = object statistics
Ri = RFID label (for training)
Gathering data on outdoor
activities

Raw GPS
Discovering significant places
Conditional Random Field
Predicting transportation
goals
Dynamic Bayesian Network
Issue

Previous work on activity recognition has
used a wide variety of probabilistic models for
different tasks and kinds of data


Background knowledge is implicitly encoded
in the structure of the models


HMMs, DBNs, CRFs, …
E.g.: Relation between transportation goals,
plans, actions
Increasingly difficult to scale & integrate
Markov Logic

Markov logic will provide common modeling
language & inference tools, enabling


Easier integration of multiple sensors
Easier generalization



From one activity at a time to multiple ongoing
activities
From one individual to multiple individuals
Easier modification of background knowledge

Add / modify library of plans and goals
Example Scenario





John goes into his kitchen (video)
He takes out a jug from the refrigerator, and a
bowl from the cabinet (RFID)
He leaves his apartment, and walks to a
convenience store (GPS)
He returns carrying a box (video)
He pours the box into the bowl (accelerometer)
and the contents of the jug (accelerometer &
RFID)

Why did John leave the apartment? What did he do?
UR Contributions to MURI: Scenario
Development & Data Collection

Develop set of activity recognition scenarios of
increasing complexity



Enact and gather sensor data



Activities in the home
Outdoor activities
Heterogeneous: GPS, RFID, video, motion, …
Intermittent and noisy
Make dataset available to team



Including feature sequences extracted from video and
acceleration data
Ground truth
1st data set mid-Year One, then ongoing
UR Contributions to MURI: Unified ML
Model of Daily Activities



Recast our previous work on recognition using
HMMs, DBNs, CRFs in Markov Logic
Integrate and generalize earlier results
Year One:

HMM  ML



Generalize to multiple ongoing activities
Handle novel observations using similarity
Representing actions, intentions, and goals



Extend ML to include “modal operators”
Distinguish beliefs of observer from beliefs of subject
Ability to model imperfect agents, whose plans are flawed
From HMMs to ML


Hidden Markov models describe the world as
probabilistic state machine
ML encoding of HMM can be relaxed to
allow subject to be in multiple states (multiple
activities) by making “unique state” constraint
soft
w : a  a, i . Activity (a, i )  Activity (a, i)
From HMMs to ML
Novel observations can be handled by
applying background knowledge about
similarity
w : a, obj , obj  . Uses (a, obj )  Similar (obj , obj )
 Uses( a, obj )

Modal Operators

Most previous work on probabilistic activity
recognition does not distinguish





What system infers is true about the world
What the subject believes is true about the world
What the system predicts will happen
What the subject intends to happen
Modal operators relate agents to attitudes

Bel( John, contains(jug, gasoline) )


But system may know jug is empty
Goal( John, ignite(jug) )

Knowledge of subject’s goal can drive cooperative system to
help subject, or antagonistic system to block user
Semantic Inference

Modal operators do not work like ordinary predicates
or logical connectives


Modal proof theory is hard to automate
However:

Modal operators have well-understood “possible world”
semantics



Agent believes P in possible world W iff P is true in all worlds
W’ such that reachable(W,W’)
ML’s inference engine works at the semantic level (direct
search over possible worlds)
Promising approach: semantic inference for modal
constructs in ML

Explicitly model reachability relationships for each attitude
and agent
Idea

Alchemy searches over models (truth
assignments)


Structure = set of models and accessibility
relationships over the models


Modal formulas are evaluated over structures
Structures are too big to explicitly search
Modify Alchemy to search over samples drawn
from structures
Holds( Bel (agent , formula), w) 
 w.R(agent , w, w)  Holds( formula, w) 