Making Sense of Sensors
Henry Kautz
Department of Computer Science & Engineering
University of Washington, Seattle, WA
Funding for this research is provided in part by IISI and AFRL/IF
Making Sense of Sensors
or … Climbing the Data Interpretation Food-Chain
The Ubiquitous Future
Rapidly declining size and cost of sensing and
networking technology makes it practical to
rapidly deploy systems that monitor large
environments in great detail
– factories, airports, hospitals, homes
– oceanic regions, cities, countryside
Problem: it is easier to collect data than make to
sense of it!
Data Fusion
Traditional work in data-fusion attacks problem
of recovering specific physical phenomena from
the readings of homogeneous networks of noisy
sensors
E.g.: given readings from underwater
microphone array, determine the position of a
submarine
Current Trends
Heterogeneous sensors
– Instrumented environment: motion detectors, weight
detectors, video, audio, …
– Instrumented personnel: smart badges, GPS phones,
metabolic sensors. …
Goal: high-level understanding
– What actions are being performed?
– What are the goals of the subjects?
– Do we need to intervene?
Example: Security
System monitors activity in a post office
Tracks common tasks performed by individuals
– Mailing packages
– Getting mail from PO boxes
– Buying stamps
Alerts operator when abnormalities noted
– Person leaves package on floor and exits
– Loitering (but not waiting in line!)
Example: Guiding
Activity Compass: GPS system that
– Learns daily patterns of travel
– Understands walking, car, bus, bike
– Integrates external information
• Real-time bus data
Predicts problems
– Will user miss appointment?
– Is user on the wrong bus?
Offer proactive help
– E.g., suggest alternative travel plan
Triple-Use Technology
Commercial
Software
Military
surveillance
augmented cognition
Plan-Aware
Computing
Patient
Care
intelligent user
interfaces
aging in place
assisted cognition
Key Issue
How to go from noisy and incomplete sensor
measurements to
A meaningful description of what a person is doing
• “Waiting to mail package”
• “Trying to get home”
A decision by the system about whether or not to
intervene
… in a principled and scalable manner!
Data Interpretation Food Chain
Interventions
Intentions
Behavior
Movement
Model-Based Interpretation
General approach: build a probabilistic model of
– Common user goals
– Plans (complex behaviors) that achieve those goals
• Feasibility constraints
• Temporal constraints
• Failure (abnormality) modes
– How simple behaviors are sensed
Run model “backwards” to interpret sensed data
Million-Mile View
In principal we know how to estimate the state of
the system under observation:
Bel( xt )  Pr( zt | xt )  Pr( xt | xt 1 ) Bel( xt 1 )dxt 1
state at
time t
observation
at time t
system
dynamics
To make this practical, we must take advantage
of the regular structure of the domain
Technical Foundations
Hidden Markov models
– Mathematical framework for describing processes
with hidden state that must be inferred from
observations
Hierarchical plan networks
– Represents how a task can be broken down into
subtasks
Hierarchical hidden Markov models*
– Key to climbing food-chain!
* Precisely speaking: factorial hierarchical hidden semi-Markov models
Example
Mail Package
Wait in
line
Enter PO
Let go
package
Pay
cashier
Location
Video
Door Sensor
Motion
Exit PO
Example
Retrieve Mail
Enter PO
Go to PO
boxes
Open PO
box
Pick up
mail
Location
Video
Door Sensor
Motion
Exit PO
Example
PO Patron
Outside PO
Mail Package
Retrieve Mail
Location
Video
Door Sensor
Motion
Inexplicable Observations
Mail Package
Enter PO
Wait in
line
Let go
package
Pay
cashier
Exit PO
Go to PO
boxes
Open PO
box
Pick up
mail
Exit PO
Retrieve Mail
Enter PO
Enter PO
Let go
package
Exit PO
Absolute Timing Constraints
Mail Package active [9 am – 4 pm]
Enter PO
Retrieve Mail active [6 am – 8 pm]
Enter PO
Relative Timing Constraints
Retrieve Mail
Go to PO
boxes
Open PO
box
seconds
Forgot combo?
Safecracking?
seconds
Timeout
Summary
Commonsense knowledge base of “significant”
behaviors
–
–
–
–
–
Hierarchically organized
Probabilistic at all levels
Many parallel ongoing activities possible
Absolute and relative timing constraints
Probabilities “tuned” by machine learning techniques for
individual users
– Inexplicable observations and failure modes – points of
possible intervention
Interventions
Framework allows system to predict when an
anomalous situation is likely
Different anomalies have different costs
– Confused patron
– Deliberate loitering
– Planting bomb
Must avoid:
Deciding When to Intervene
G = prediction that help is needed
(Horvitz 98)
Common Architecture
Activity Compass
Palm-based wireless GPS
– No explicit programming – learns pattern of
transportation plans
– Accesses user’s calendar, real-time bus information
– Constantly tries to predict where user will go next, and
whether problems will arise
– Proactive help:
• “Walk faster or you’ll miss the 9:15 bus!”
• “Green St bus is late, suggest you take Elm St bus instead”
Substeps
Cleaning up GPS data
– 3 meter accuracy
– frequent signal loss
– determine most likely path
walk
bus
indoors
car
Infer mode of transportation
Predict when and where transitions in
mode of travel will occur
Predict points of possible failure
bike
Gathering Data
On Foot: Across Campus
By Bus: Across Seattle
Transition Prediction
Training Data:
–
20,000 GPS readings gathered over 3 weeks
Inferring current mode
–
–
Input: current location, time, velocity
98% accuracy (10 FCV)
Predicting next transition
–
–
Input: current mode, location, time, velocity
97% accuracy (10 FCV)*
* Don is a very organized guy. Your accuracy may vary.
Predicting Transition Location
User Interface
Assisted Cognition
“Plan aware” systems to help people with cognitive
disabilities
New project based at University of Washington
– Computer Science & Engineering
– UW Medical Center, ADRC
– Collaborators: Intel, OGI, Elite Care
http://assistcog.cs.washington.edu/
Summary
Potential of widespread sensor networks just
beginning to be tapped
Key issue: interpreting data in terms of human
behavior, plans, and goals
Researchers in data fusion, AI, and “ubicomp”
coming together around a core set of
representations and algorithms