Technologies to Improve Quality of
Life for People with Cognitive
Disabilities
Henry Kautz
Department of Computer Science
University of Rochester
Rochester, NY
Assisted Cognition Systems
Henry Kautz
Department of Computer Science
University of Rochester
Rochester, NY
Vision
Computer systems that improve the independence
and well-being of people suffering from cognitive
disabilities by…
•
•
•
Understanding human behavior
from sensor data
Actively prompting, warning,
and advising
Alerting caregivers as necessary
Foundations
• Pervasive sensing
• Movement, location, object manipulation, physiology
• On user or embedded in the environment
• AI / machine learning
• Model everyday life
• Adapt to individual's behavior and goals
• Human computer interaction
• Keep focus on user
• Ease of use
Outline
• Designing an adaptive wayfinding system for
cognitively impaired users
• Creating a context-aware task prompting
system
• Challenges in research on assisted cognition
systems
Outline
• Designing an adaptive wayfinding system for
cognitively impaired users
– U Washington Computer Science
• Alan Liu, Gaetano Borriello
– U Rochester Computer Science
• Henry Kautz
– U Washington Rehabilitation Medicine
• Kurt Johnson, Mark Harniss, Pat Brown
• Creating a context-aware task prompting system
• Challenges in research on assisted cognition
systems
Wayfinding Difficulties
• Alice doesn’t go to new places by herself because even
with practice, she still has trouble remembering
directions.
• Bob gets lost easily, and cannot use a map to orient
himself and figure out where he is or where to go.
• Carol takes a regular route between home and work,
but one day a road along the way is closed and she
doesn’t know how to adapt.
7
GPS Systems
• Why aren't regular GPS systems adequate?
• "One size fits all" route planning
– Minimize distance or time when driving
• "One size fits all" presentation of directions
– Map and turn-by-turn directions
• What makes a route safe and easy to follow by
a cognitively impaired user?
Research Challenge
• Individuals with cognitive impairment can
have vastly different abilities and disabilities
– Designing for a "universe of one" (Kintsch & DePaula)
• However:
– Can we uncover common features that lie behind
successful solutions?
– Can we incorporate these features into a general
system that adapts to the individual?
9
Research Path
• Formative usability studies
– Studied direction-following capabilities and priorities
– Potential users interacting with prototypes in situ
• Landmark-based directions
– Incorporated landmarks into wayfinding
– Balancing cognitive effort and attention to environment
• Modeling individual variation
– Implemented framework to generate customized and
adaptable directions
– Measure impact of adaptation on users
Formative Studies
• Indoor and outdoor wayfinding
using a "Wizard of Oz" system
• Wide variation in user preference
and expectation of directions
– Modality (image, audio, text)
– Complexity
– Timing
• Attention demand for turn-by-turn
directions problematic
• Alternative: landmark-based
directions
Liu et al. ASSETS 2006, PervasiveHealth 2009
`
Landmark-Based Directions
• Landmark-based directions
– Encourage attention to world
– Orient user
– More natural
• What are features of good
landmarks?
• 9 study participants
– TBI, stroke, MS, Dev Dis.
A.L. Liu, H. Hile, H. Kautz, G. Borriello, P.A. Brown, M. Harniss, K. Johnson, PervasiveHealth 2009
Landmark Selection
• Leverage collections of geo-tagged photos
– Heuristic: more popular photos more useful
• Can augment / warp photos to make them
more closely match user's perspective
Results
• 180 total directions given, 150 correctly followed
– Much variation between participants
• Features of most successful landmark-based
directions:
– In perspective
– Close
– Path-aligned
• Making turns in relation to landmark required
more cognitive effort
– Turn-based directions should be user-centric
Modeling Individual Variation
• Studies showed how to create a wayfinding
system that was usually better for most users
• However: system was still far from optimal for
many users
• Can we design a system that adapts to the
individual user?
Adaptation as Optimization
• Approach: for each user, learn a function:
D irection Features  L ocation Features
 Probability Success in Follow ing D irection
• Generating directions = finding optimal
solution to a sequential decision problem
• Issue: scarcity of user-specific training data
– Solution: use data from all users, but weigh
particular user's data more heavily
Adaptation Study
• 5 participants
• Compared performance on
– Route generated with generic
model
– Route generated with
personalized model
• Qualitative result:
personalized routes
considered easier
– Need larger, longer study to
quantify significance
17
Summary
• Studied wayfinding with real potential users
– Learned range of abilities and preferences
– Informed technological requirements
• Incorporated landmarks as wayfinding option
– Explored landmark dimensions that affect usability
• Proof-of-concept of automatic adaptation
– Creating a personal model using data from both
the individual and the larger user base
18
Outline
• Designing an adaptive wayfinding system for
cognitively impaired users
• Creating a context-aware task prompting
system
– Attention Control Systems: Richard Levinson
– University of Rochester: Henry Kautz
– VA Palo Alto: Dr. Harriet Zeiner
– UC Santa Cruz: Beth Ann Hockey
• Challenges in research on assisted cognition
systems
PEAT
• PEAT: handheld-based scheduling and cueing
system for persons with executive function
impairment (Attention Control Systems)
– Multimedia prompts to start and end tasks
– Assists with task (re)scheduling, by reasoning about
deadlines, durations, and priorities
Problems with Prompting Systems
• Require input from user whenever a task
begins or ends
– Cost of use may discourage users
• Prompts may be out of sync
– Self-initiation confuses system
– Encourages dependency
• No support for contingent prompts
– Remind the user to take her walker when she goes
outside
Integrating Sensing and Prompting
• Goal: use information from sensors to improve
prompting
• Determine user's current activity from
– Location: Home automation motion sensors, GPS
– Objects user is handling: Intel Research Seattle RFID
reader bracelet
– Schedule of expected activities
Context-Aware PEAT
• R&D funded by grants from DARPA,
Office of Defense, and NIH
• Platform: Android cell phone
– Internal sensors: GPS, acceleration
– Home base station (laptop) for setup
and relaying data from Insteon
home automation sensors and Intel
Research RFID reader
– State estimator: employs
probabilistic algorithms to infer user
activity and learn user model
Home sensors
Cellphone
RF
Wireless connections
GPS
WiFi
WiFi
Relay
State Estimator
ZigBee
PEAT
State DB
Daily Activity
Planning & Cueing
PEAT User
Interface
User
RFID
reader
bracelet
Reduced Prompting
• Prompting cancelled if user self-initiates (or selfterminates) task
– Reduces confusion / out of sync prompts
– Encourages, rather than punishes, self-initiation
• Can system do even more to encourage selfinitiation?
– Perhaps: prompt later than scheduled time
– If prompt is too late and user does
not self-initiate, task will fail
– What is the optimal time
for system to prompt?
Optimal Prompting
• System can be configured to use a fixed
prompting schedule (e.g. 5 minutes before
scheduled time) or an adaptive schedule
• Caregiver specifies rewards associated with
– Successful (timely) task completion
– User self-initiation of task
• System
– Learns model of user self-initiation
– Computes prompting time that maximizes expected
reward
Learning the User Model
• User model includes probability distribution
over how long it takes user to self-initiate
– Technical challenge: how to estimate this
parameter quickly and accurately?
– Can use statistical techniques developed for
reliability analysis
System Trial
• In-home trials by Palo
Alto VA start next month
• Help regularize patients'
daily schedule for
–
–
–
–
Sleep
Meals
Exercise
Behavioral therapy
"homework"
Outline
• Designing an adaptive wayfinding system for
cognitively impaired users
• Creating a context-aware task prompting
system
• Challenges in research on assisted cognition
systems
Point versus General Solutions
• “[A] tension exists between satisfying the
need for customized solutions that address
the immediate needs of users, and identifying
generalizable results that can be used to
achieve long-term research goals.”
[Generalizability in Research with Cognitively Impaired Individuals,
Moffatt, Findlater, Allen, 2006]
Evaluating with a Universe of One
• “Twenty-eight papers presenting 25 studies
were reviewed. The total number of
participants was 423. Most identified papers
described case reports or non-randomized
clinical trials. Only one randomized controlled
trial was identified.”
[Efficacy and usability of assistive technology for patients with cognitive deficits:
a systematic review,
Joode, Heugten, Verhey, & Boxtel, 2010]
The Training Hurdle
• “The [studies] that do describe the exact time
spent on training show a substantial amount
of variation (from 30 minutes up to 9 hours).”
[Efficacy and usability of assistive technology for patients with cognitive deficits:
a systematic review,
Joode, Heugten, Verhey, & Boxtel, 2010]
Promise of Context-Aware
Systems
Medication Prompting:
• “The context-based prompting resulted in significantly
better adherence (92.3%) as compared to time-based
(73.5%) or no prompting (68.1%) conditions (p < 0.0002,
χ2 = 17.0). In addition, subjects had better adherence in
the morning than in the evening. We have shown in this
study that a system that generates reminders at an
opportune time to take the medication significantly
improves adherence.”
[A Study of Medication-Taking and Unobtrusive, Intelligent Reminding,
Hayes, Cobbinah, Dishongh, Kaye, Kimel, Labhard, Leen, Lundell,
Ozertem, Pavel, Philipose, Rhodes, & Vurgun, 2009]
Pitfalls of Context-Aware
Systems
• “The participant was building a mental model of how
the system behaved, and his model did not map well
onto the behavior of the system. The problem was
especially pronounced when an activity was not
detected properly. ... The participant further admitted
to changing his behavior in order to ‘fool‘ the system,
but since he didn’t understand how it worked, this
behavior was counter-productive.
[User-adaptive Reminders for Home-based Medical Tasks,
Kaushik, Intille, Larson, 2008]