The Assisted Cognition
Project
Henry Kautz, Dieter Fox, Gaetano Boriello
Lin Liao, Brian Ferris, Evan Welborne
(UW CSE)
Don Patterson
(UW / UC Irvine)
Kurt Johnson, Pat Brown, Mark Harniss
(UW Rehabilitation Medicine)
Matthai Philipose
(Intel Research Seattle)
Trend 1: Sensing Infrastructure
 Robust direct-sensing technology
o
o
o
GPS-enabled phones
RFID tagged products
Wearable multi-modal sensors
 Rapid commercial deployment
Trend 2: Healthcare Crisis

Demand for community integration of the cognitively
disabled
o
o
o


100,000 @ year disabled by traumatic brain injury
7.5 million in US with mental retardation
4 million in US with Alzheimer’s
Family burnout
Nationwide shortage of professionals
Assisted Cognition
 Technology to support independent living by
people with cognitive disabilities
o at home
o at work
o throughout the community
by
o Understanding human behavior from sensor data
o Actively prompting and advising
o Alerting human caregivers when necessary
Building Partnerships
 UW Assisted Cognition seminar
o CSE, medicine, nursing, Intel
 ACCESS
o
o
o
o
UW CSE & Rehabilitation Medicine
Grant from NIDDR (Dept. of Education)
Help cognitively disabled use public transportation
Prototype: Opportunity Knocks
 Intel Proactive Health effort
o Computing for wellness & caregiving
o Promote partnerships with government, universities,
healthcare organizations
o Intel Seattle: sensors for activity tracking
Example
 Way-finding Assistant
o Help user travel throughout community
 On foot
 Using public transportation
o Detect user errors
 Proactively help user recover
 “You missed your stop, so get off at the next stop
and then wait for the #16 bus...”
o Potential users
 TBI, MR, mild memory impairment
Example
 ADL Assistant
o Activities of daily living
 Eating, bathing, dressing, ...
 Cooking, cleaning, emailing, ...
o Monitoring
 Changes in ADLs signal changes in health
o Reminding / prompting
 “Time to take your blue meds”
o Step-by-step guidance
 “Turn on the tap ... now pick up the brush ...”
o Potential users
 Disabled, ordinary aging
General Model
geospatial
DB
user
model
intervention
decision
making
commonsense
KB
user
interface
wearables
sensors
environmental
sensors
caregiver
alerts
General Model
cognitive state
geospatial
DB
goals
intervention
decision
making
activity
commonsense
KB
physical motion
& position
user
interface
wearables
sensors
environmental
sensors
caregiver
alerts
Deciding to Intervene
A = system intervenes
G = user actually needs help
ACCESS
Way-finding Assistant
supported by
National Institute on Disability &
Rehabilitation Research
DARPA IPTO
The Need: Community Access
for the Cognitively Disabled
Problems in Using Public
Transportation
•Learning bus routes and numbers
Problems in Using Public
Transportation
•Learning bus routes and numbers
•Transfers, complex plans
Problems in Using Public
Transportation
•Learning bus routes and numbers
•Transfers, complex plans
•Recovering from mistakes
Result
•Need for extensive life-coaching
•Need for point-to-bus service
Result
•Need for extensive life-coaching
•Need point-to-bus service
•Isolation
Current GPS Navigation
Devices
 Designed for drivers, not bus riders!
o
o
o
Should I get on this bus?
Is my stop next?
What do I do if I miss my stop?
 Requires extensive user input
o
Keying in street addresses no fun!
 Device decides which route is “best”
o
Familiar route better than shorter one
 “Catastrophic failure” when signal is lost
New Approach
 User carries GPS cell phone
 System infers transportation mode
o
Position, velocity, geographic information
 Over time, system learns about user
o
o
Important places
Common transportation plans
 Breaks from routine = possible user errors
o
Ask user if help is needed
User Model
ck-1
ck
gk-1
gk
Cognitive mode
{ routine, novel, error }
Goal
tk-1
tk
Trip segment
mk-1
mk
Transportation mode
xk-1
xk
Edge, velocity, position
qk-1
qk
Data (edge) association
zk-1
zk
Time k-1
Time k
GPS reading
Error
Detection:
Missed
Bus Stop
Prototype: Opportunity Knocks
 GPS camera-phone
 “Knocks” when there is
an opportunity to help
o Can I guide you to a
likely destination?
o I think you made a
mistake!
o This place seems
important – would you
photograph it?
Status
 User needs study
 Algorithms for learning and predicting
transportation behavior
o Best paper award at AAAI-2004
 Proof of concept prototype
 Now: user interface studies
o Modality: Audio, Graphics, Tactile, ...
o Guidance strategies: Landmarks, User
frame of reference, Maps, ...
ADL Monitoring from
RFID Tag Data
UW CSE
Intel Research Seattle
demo at Intel this afternoon
Object-Based Activity Recognition
 Activities of daily living involve the
manipulation of many physical objects
o Kitchen: stove, pans, dishes, …
o Bathroom: toothbrush, shampoo, towel, …
o Bedroom: linen, dresser, clock, clothing, …
 We can recognize activities from a timesequence of object touches
Sensing Object Manipulation
 RFID: Radio-frequency
identification tags
o Small
o Long-lived – no batteries
o Durable
 Easy to deploy
 Bracelet touch sensor
 Wall-mount movement
sensor
Example Data Stream
Example Activity Model
Creating Models of ADLs




Hand-built
Learn from sensor data
Mine from natural-language texts
All of the above...
Experiment: Morning Activities
 10 days of data from the morning routine in
an experimenter’s home
o 61 tagged objects
 11 activities
o Often interleaved and interrupted
o Many shared objects
Use bathroom Make coffee
Set table
Make oatmeal Make tea
Eat breakfast
Make eggs
Use telephone
Clear table
Prepare OJ
Take out trash
DBN with Aggregate Features
88% accuracy
6.5 errors per episode
Improving Robustness
 Tracking fails if novel objects are used
 Solution: smooth parameters over
abstraction hierarchy of object types
Status
 Accurate tracking of wide variety ADLs
 Active collaboration with Intel
 Current work
o Detecting user errors in ADL performance
o Learning more complex ADLs
 Preconditions/effects
 Multi-tasking
 Temporal constraints
o Reminding & prompting
Concluding Remarks
 Research on Assisted Cognition going
great guns at UW and (a few) other
universities
o CMU / Pitt / U Michigan (Nursebot,
Autominder – M. Pollack)
o Georgia Tech (Aware Home, G. Abowd)
o MIT (House N, Stephen Intille)
Some Thoughts on Funding
 Getting funding for work in this area is
currently challenging
o We were fortunate once with NIDRR, but
less than 1% of their budget is for research
o NIH & NIA spend relatively little on
caregiving research
 New NIH “Roadmap” for interdisciplinary
exploratory research completely leaves out
caregiving!
o NIN has good people, but no real money
Some Thoughts on Funding
 Getting funding for work in this area is
currently challenging
o NSF supports some of the underlying, multiuse technology, but not medically-oriented
applications
 Exception: helping disabled use computers
o Industry support is vital, but more for
collaboration than actual dollars
 Good industry grant = 1 grad student
o There’s a gap waiting to be filled...