Understanding Human
Behavior from Sensor Data
Henry Kautz
University of Washington
A Dream of AI
 Systems that can understand ordinary
human experience
 Work in KR, NLP, vision, IUI, planning…
o Plan recognition
o Behavior recognition
o Activity tracking
 Goals
o Intelligent user interfaces
o Step toward true AI
Plan Recognition, circa 1985
Behavior Recognition, circa 2005
Punch Line
 Resurgence of work in behavior
understanding, fueled by
o Advances in probabilistic inference
 Graphical models
 Scalable inference
 KR U Bayes
o Ubiquitous sensing devices
 RFID, GPS, motes, …
 Ground recognition in sensor data
o Healthcare applications
 Aging boomers – fastest growing demographic
This Talk
 Activity tracking from RFID tag data
o ADL Monitoring
 Learning patterns of transportation use from
GPS data
o Activity Compass
 Learning to label activities and places
o Life Capture
 Kodak Intelligent Systems Research Center
o Projects & Plans
This Talk
 Activity tracking from RFID tag data
o ADL Monitoring
 Learning patterns of transportation use from
GPS data
o Activity Compass
 Learning to label activities and places
o Life Capture
 Kodak Intelligent Systems Research Center
o Projects & Plans
Object-Based Activity Recognition
 Activities of daily living involve the
manipulation of many physical objects
o Kitchen: stove, pans, dishes, …
o Bathroom: toothbrush, shampoo, towel, …
o Bedroom: linen, dresser, clock, clothing, …
 We can recognize activities from a timesequence of object touches
Application
 ADL (Activity of Daily
Living) monitoring for
the disabled and/or
elderly
o Changes in routine
often precursor to
illness, accidents
o Human monitoring
intrusive & inaccurate
Image Courtesy Intel Research
Sensing Object Manipulation
 RFID: Radiofrequency
identification tags
o
o
o
o
Small
No batteries
Durable
Cheap
 Easy to tag objects
o
Near future: use
products’ own tags
Wearable RFID Readers
 13.56MHz reader, radio, power supply, antenna
o 12 inch range, 12-150 hour lifetime
 Objects tagged on grasping surfaces
Experiment: ADL Form Filling
•Tagged real home with 108 tags
•14 subjects each performed 12 of
65 activities (14 ADLs) in arbitrary
order
•Used glove-based reader
•Given trace, recreate activities
Results: Detecting ADLs
Activity
Prior
Work
SHARP
Legend
General solution
Personal Appearance
92/92
Oral Hygiene
70/78
Toileting
73/73
Washing up
100/33
Appliance Use
100/75
Use of Heating
84/78
Care of clothes and linen
100/73
Making a snack
100/78
Making a drink
75/60
Use of phone
64/64
Leisure Activity
100/79
Infant Care
100/58
Medication Taking
100/93
Housework
100/82
Point solution
Inferring ADLs from
Interactions with
Objects
Philipose, Fishkin,
Perkowitz, Patterson,
Hähnel, Fox, and
Kautz
IEEE Pervasive
Computing, 4(3), 2004
Experiment: Morning Activities
 10 days of data from the morning routine in
an experimenter’s home
o 61 tagged objects
 11 activities
o Often interleaved and interrupted
o Many shared objects
Use bathroom Make coffee
Set table
Make oatmeal Make tea
Eat breakfast
Make eggs
Use telephone
Clear table
Prepare OJ
Take out trash
Baseline: Individual Hidden
Markov Models
68% accuracy
11.8 errors per episode
Baseline: Single Hidden Markov
Model
83% accuracy
9.4 errors per episode
Cause of Errors
 Observations were types of objects
o Spoon, plate, fork …
 Typical errors: confusion between
activities
o Using one object repeatedly
o Using different objects of same type
 Critical distinction in many ADL’s
o Eating versus setting table
o Dressing versus putting away laundry
Aggregate Features
 HMM with individual object observations
fails
o No generalization!
 Solution: add aggregation features
o Number of objects of each type used
o Requires history of current activity
performance
o DBN encoding avoids explosion of HMM
Dynamic Bayes Net with
Aggregate Features
88% accuracy
6.5 errors per episode
Improving Robustness
 DBN fails if novel objects are used
 Solution: smooth parameters over
abstraction hierarchy of object types
Abstraction Smoothing
 Methodology:
o Train on 10 days data
o Test where one activity substitutes one
object
 Change in error rate:
o Without smoothing: 26% increase
o With smoothing: 1% increase
Summary
 Activities of daily living can be robustly
tracked using RFID data
o Simple, direct sensors can often replace
(or augment) general machine vision
o Accurate probabilistic inference requires
sequencing, aggregation, and abstraction
o Works for essentially all ADLs defined in
healthcare literature
D. Patterson, H. Kautz, & D. Fox, ISWC 2005
best paper award
This Talk
 Activity tracking from RFID tag data
o ADL Monitoring
 Learning patterns of transportation use from
GPS data
o Activity Compass
 Learning to label activities and places
o Life Capture
 Kodak Intelligent Systems Research Center
o Projects & Plans
Motivation: Community Access
for the Cognitively Disabled
The Problem
 Using public transit cognitively challenging
o Learning bus routes and numbers
o Transfers
o Recovering from mistakes
 Point to point shuttle service impractical
o Slow
o Expensive
 Current GPS units hard to use
o Require extensive user input
o Error-prone near buildings, inside buses
o No help with transfers, timing
Solution: Activity Compass
 User carries smart cell phone
 System infers transportation mode
o GPS – position, velocity
o Mass transit information
 Over time system learns user
model
o Important places
o Common transportation plans
 Mismatches = possible mistakes
o System provides proactive help
Technical Challenge
 Given a data stream from a GPS unit...
o Infer the user’s mode of transportation, and
places where the mode changes

Foot, car, bus, bike, …

Bus stops, parking lots, enter buildings, …
o Learn the user’s daily pattern of movement
o Predict the user’s future actions
o Detect user errors
Technical Approach
 Map: Directed graph
 Location: (Edge, Distance)
o Geographic features, e.g. bus stops
 Movement: (Mode, Velocity)
o Probability of turning at each intersection
o Probability of changing mode at each location
 Tracking (filtering): Given some prior estimate,
o Update position & mode according to motion model
o Correct according to next GPS reading
Dynamic Bayesian Network I
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
Time k-1
Time k
GPS reading
Mode & Location Tracking
Measurements
Projections
Green
Bus mode
Red
Car mode
Blue
Foot mode
Learning
 Prior knowledge – general constraints on
transportation use
o Vehicle speed range
o Bus stops
 Learning – specialize model to particular
user
o 30 days GPS readings
o Unlabeled data
o Learn edge transition parameters using
expectation-maximization (EM)
Probability of correctly
predicting the future
Predictive Accuracy
How to improve
predictive
power?
City Blocks
33
Transportation Routines
Home
A
B
Workplace
 Goal: intended destination
o Workplace, home, friends, restaurants, …
 Trip segments: <start, end, mode>
o Home to Bus stop A on Foot
o Bus stop A to Bus stop B on Bus
o Bus stop B to workplace on Foot
Dynamic Bayesian Net II
gk-1
gk
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
Time k-1
Time k
GPS reading
Predict Goal and Path
Predicted goal
Predicted path
Improvement in Predictive
Accuracy
Detecting User Errors
 Learned model represents typical
correct behavior
o Model is a poor fit to user errors
 We can use this fact to detect errors!
 Cognitive Mode
o Normal: model functions as before
o Error: switch in prior (untrained)
parameters for mode and edge transition
Dynamic Bayesian Net III
ck-1
ck
gk-1
gk
Cognitive mode
{ normal, error }
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
Time k-1
Time k
GPS reading
Detecting User Errors
Status
 Major funding by NIDRR
o National Institute of Disability & Rehabilitation
Research
o Partnership with UW Dept of Rehabilitation
Medicine & Center for Technology and Disability
Studies
 Extension to indoor navigation
o Hospitals, nursing homes,
assisted care communities
o Wi-Fi localization
 Multi-modal interface studies
o Speech, graphics, text
o Adaptive guidance strategies
Papers
Liu et al, ASSETS 2006
Indoor Wayfinding: Developing a Functional
Interface for Individuals with Cognitive
Impairments
Liao, Fox, & Kautz, AAAI 2004 (Best Paper)
Learning and Inferring Transportation Routines
Patterson et al, UBICOMP 2004
Opportunity Knocks: a System to Provide Cognitive
Assistance with Transportation Services
This Talk
 Activity tracking from RFID tag data
o ADL Monitoring
 Learning patterns of transportation use from
GPS data
o Activity Compass
 Learning to label activities and places
o Life Capture
 Kodak Intelligent Systems Research Center
o Projects & Plans
Task
 Learn to label a person’s
o Daily activities
 working, visiting friends, traveling, …
o Significant places
 work place, friend’s house, usual bus stop, …
 Given
o Training set of labeled examples
o Wearable sensor data stream
 GPS, acceleration, ambient noise level, …
Learning to Label Places &
Activities
FRIEND’S HOME
RESTAURANT
OFFICE
PARKING LOT
HOME
STORE
 Learned general rules for labeling “meaningful places”
based on time of day, type of neighborhood, speed of
movement, places visited before and after, etc.
Relational Markov Network
 First-order version of conditional random field
 Features defined by feature templates
o All instances of a template have same weight
 Examples:
o
o
o
o
o
Time of day an activity occurs
Place an activity occurs
Number of places labeled “Home”
Distance between adjacent user locations
Distance between GPS reading & nearest street
RMN Model
Global soft constraints
s1
p1
a1
g1
g2
a2
g3
p2
a3
g4
a4
g5
g6
time 
Significant places
a5
g7
g8
Activities
g9
{GPS, location}
Significant Places
 Previous work decoupled identifying
significant places from rest of inference
o Simple temporal threshold
o Misses places with brief activities
 RMN model integrates
o Identifying significant place
o Labeling significant places
o Labeling activities
Results: Finding Significant
Places
Results: Efficiency
Summary
 We can learn to label a user’s activities and
meaningful locations using sensor data &
state of the art relational statistical models
(Liao, Fox, & Kautz, IJCAI 2005, NIPS 2006)
 Many avenues to explore:
o
o
o
o
Transfer learning
Finer grained activities
Structured activities
Social groups
This Talk
 Activity tracking from RFID tag data
o ADL Monitoring
 Learning patterns of transportation use from
GPS data
o Activity Compass
 Learning to label activities and places
o Life Capture
 Kodak Intelligent Systems Research Center
o Projects & Plans
Intelligent Systems Research
Henry Kautz
o Kodak Intelligent Systems Research Center
ISRC
Intelligent Systems Research Center

New AI / CS center at Kodak Research Labs
o Directed by Henry Kautz, 2006-2007
o Initial group of 4 researchers, expertise in machine vision
o Hiring 6 new PhD level researchers in 2007

Mission: providing KRL with access to expertise in
o Machine learning
o Automated reasoning & planning
o Computer vision & computational photography
o Pervasive computing
14 March 2007
ISRC
University Partnerships

Goal: access to external expertise & technology
o
o
o
o
o
o
o

> $3M @ year across KRL
Unrestricted grants: $20 - $50K
Sponsored research grants: $75 - $150K
CEIS (NYSTAR) eligible
KEA Fellowships (3 year)
Summer internships
Individual consulting agreements
Master IP Agreement Partners
o University of Rochester
o University of Massachusetts at Amherst
o Cornell

Many other partners
o UIUC, Columbia, U Washington, MIT, …
14 March 2007
ISRC
New ISRC Project


Extract meaning from combination of image content and
sensor data
First step:
o GPS location data
o 10 hand-tuned visual concept detectors
 People and faces
 Environments, e.g. forest, water, …
o 100-way classifier built by machine learning
 High-level activities, e.g. swimming
 Specific objects, e.g. boats
o Learned probabilistic model integrates location
and image features
 E.g.: Locations near water raise
probability of “image contains boats”
14 March 2007
ISRC
Multimedia Gallery/Blog (with automatic annotation)
Flying
Sailing
Driving
Walking
Swimming in the Caymans
14 March 2007
ISRC
Enhancing Video

Synthesizing 3-D video from ordinary video
o Surprisingly simple and fast techniques use motion information to
create compelling 3-D experience

Practical resolution enhancement
o New project for 2007
o Goal: convert sub-VGA quality video (cellphone video) to highquality 640x480 resolution video
o Idea: sequence of adjacent frames contain implicit information
about “missing” pixels
14 March 2007
ISRC
Workflow Planning Research Opportunities
Problem: Assigning workflow(s)
Problem: Sets of jobs often not
to particular jobs requires Problem: Users find
scheduled optimally, wasting
extensive expertise
it hard to define new
time & raising costs
Solution: System learns general
workflows correctly
Solution: Improved scheduling
job planning preferences from
(even with good
algorithms that consider
expert users
tools)
interactions between jobs (e.g.
Solution: Create
ganging)
and verify new
workflows by
automated planning
14 March 2007
Conclusion: Why Now?
 An early goal of AI was to create
programs that could understand
ordinary human experience
 This goal proved elusive
o Missing probabilistic tools
o Systems not grounded in real world
o Lacked compelling purpose
 Today we have the mathematical tools,
the sensors, and the motivation
Credits
 Graduate students:
o Don Patterson, Lin Liao, Ashish
Sabharwal, Yongshao Ruan, Tian Sang,
Harlan Hile, Alan Liu, Bill Pentney, Brian
Ferris
 Colleagues:
o Dieter Fox, Gaetano Borriello, Dan Weld,
Matthai Philipose
 Funders:
o NIDRR, Intel, NSF, DARPA