Assisted Cognition
Henry Kautz, Oren Etzioni, & Dieter Fox
University of Washington
Department of Computer Science & Engineering
1
An Epidemic of Alzheimer’s
4.6 million people in the US with
Alzheimer’s
16 million people by 2050
Today costs $100 billion @ year for care
Additional $61 billion in lost productivity from
family members
$ ½ Trillion total cost by 2050!
Projections even worse for Japan and Europe
2
3
Cognition in Context
Can often compensate for physical
disabilities by change in environment
Wheelchairs
Redesigned appliances
Cognitive competence also depends on
environment
Can you cook dinner, given a dead animal,
a stone knife, and set of flints?
4
Problem
Caregiver burnout
½ of all family caregivers suffer depression
“The 36 Hour Day”
Far too few professional caregivers to
provide constant 1-on-1 help in
institutional settings
Already a nationwide shortage of good
staff
5
Vision
Computer systems that improve the independence
and safety of people suffering from cognitive
limitations by…
Understanding human behavior
from low-level sensory data
Using commonsense knowledge
Learning individual user models
Actively offering prompts and other forms of
help as needed
Alerting human caregivers when necessary
6
Data Interpretation Food Chain
Interventions
Intentions
Behavior
Movement
7
Model-Based Interpretation
General approach: build a probabilistic model of
– Common user goals
– Plans (complex behaviors) that achieve those goals
• Feasibility constraints
• Temporal constraints
• Failure (abnormality) modes
– How simple behaviors are sensed
Run model “backwards” to interpret sensed data
8
Activity Compass

Zero-configuration personal
guidance system



Learns model of user’s
travel on foot, by public
transit, by bike, by car
Predicts user’s next
destination, offers proactive
help if lost or late
Integrates user data with
external constraints


Maps, bus schedules,
calendars, …
EM approach to clustering
& segmenting data
The Activity Compass Don Patterson, Oren Etzioni, and Henry Kautz (2003)
9
Scenario I
Joe gets off the bus on the way to the
community center.
He can’t remember which way to walk.
He consults his Activity Compass. It
has predicted his destination based on
past behavior, and guides him to the
community center.
10
Scenario II
Joe has a regular physical therapist’s
appointment at 2 pm.
Compass alerts Joe that he will need to
leave home in 10 minutes to catch the usual
bus there.
Joe fails to leave in time. Compass asks
whether goal of going to physical therapist is
valid.
If answer is affirmative, Compass creates
alternative plan to get Joe there as soon as
possible.
11
Minimalist User Interface
12
Data Source: Elite Care Residences
Elite Care is the realization of our dream
to provide a fundamentally different
approach to assisted living. Extended
Family Residences promote a family
lifestyle of close staff-resident
relationships, meaningful communitybuilding activities, and physical and
mental achievement while providing
assistance when needed… Technology
helps us maintain our family environment
by allowing us to the focus on residents…
- Lydia Lundberg & Bill Reed
13
Activity of daily living monitor
& prompter
Foundations of Assisted Cognition Systems. Kautz, Etzioni, Fox, Weld, and 14
Shastri, 2003
Recognizing unexpected events
using online model selection

fill kettle
put kettle
on stove

User errors, abnormal
behavior
Select model that
maximizes likelihood of
data:


fill kettle
put kettle
on stove
put kettle
in closet


Neurologically-plausible
corruptions



Fox, Kautz, & Shastri (forthcoming)
Generic model
User-specific model
Corrupt (impaired) user
model
Repetition
Substitution
Stalling
15
Technical Foundations
Hidden Markov models
– Mathematical framework for describing processes
with hidden state that must be inferred from
observations
Hierarchical plan networks
– Represents how a task can be broken down into
subtasks
Hierarchical hidden Markov models*
– Key to climbing food-chain!
* Precisely speaking: factorial hierarchical hidden semi-Markov models
16
General Architecture
Ubiquitous computing infrastructure
Sensors – position, motion, sound, vision
Output – speech, graphics, robots
Portable wireless computing devices
17