Informing the Design of an
Automated Wayfinding System for
Individuals with Cognitive Impairments
Alan L. Liu, Harlan Hile, Gaetano Borriello
Computer Science & Engineering
University of Washington, USA
Pat A. Brown, Mark Harniss, Kurt Johnson
Rehabilitation Medicine
University of Washington, USA
Henry Kautz
Computer Science
University of Rochester, USA
Wayfinding Concerns Limit Independence
of Persons with Cognitive Disabilities
•Learning new routes
–training from job coaches / caregivers
–burden on caregivers and Access transit
•Adapting to changes in routes
–missing a bus
–encountering an unexpected barrier
•Stress of being lost lowers meta-cognition
–can become overwhelmed and “stuck”
2
Convergence of Technologies
Geographic
information
Location-sensing
technologies
Mobile
computing
Online photo
collections
Intelligent
wayfinding
system
User
modeling
3
Our Earlier Results on Wayfinding
• Wide variation in user preference and expectation of
directions
–modality (image, audio, text)
–complexity (e.g., direction length)
–visual distinctiveness
–timing
`
• Direction ambiguity problematic
–nonstandard turns, especially outdoors
• Attention demand problematic
–being distracted or becoming unaware of
surroundings
Liu et al. ASSETS 2006, PervasiveHealth 2009
4
Needs for More Intelligent Wayfinding
Assistance
•Map and/or turn representation not suitable for all
–Using good landmarks can beneficially augment
directions
•Current systems employ static navigation
strategies
–Ignore health conditions, error behavior, desired detail,
safety concerns, place familiarity, and other individual
differences
•Need to support customization and adaptation
5
Contributions
• Automatic landmark selection system
• Markov-Decision Process based system
for optimizing routes and cues
• User study to establish further
customization / adaptation needs
Landmark selection system
•Leverage collections of geo-tagged photos
•Select image of landmark based on proximity to
user location and near user orientation
•Heuristic: landmarks useful for wayfinding tend to
be more popular
•Further enhancements and alternative
views are ongoing
7
Modeling Wayfinding with Markov
Decision Processes
•MDPs have been used in robot navigation planning
•MDP defined by states, actions, transitions, costs
–state: user position and history of recent events
–action: direction cue
–transition: next states and their likelihood given action
–cost: notion of effort or preference for route
•A solution to a MDP is a policy that minimizes expected
costs
•The MDP framework can support adaptation using
reinforcement learning
8
Modeling Wayfinding using Markov Decision
Processes
Observation
Next state transitions
State
State
(location,
orientation,
recent history)
System Action
(direction cue)
State
Transition
probabilities
Pr(St+1|St,A)
State
Cost / Reward
9
Modeling customization
• Alter costs based on individual preferences
–Preference of direction types
–Route concerns (distance, accessibility, complexity)
• Alter transition probabilities based on user ability
–Does user move in correct direction in response to cue?
•Study: Does system customization affect user
experience?
–Model 1: Naïve model: Use any direction that guides user along
shortest path
–Model 2: Landmark cues more costly (require more cognitive
resources) than turn-based directions, but can take user farther
with each cue
Policy will use landmarks for long segments,
turns for short segments
10
Location Wizard-of-Oz interface
11
Wayfinding application user interface
•Image, audio (TTS), text, vibration output
•Repeat audio and Help input
–Help button requests alternative direction (and possibly
route)
12
Participants
•7 participants (6 male) with cognitive impairments
•Age ranging from 21-49 (mean 34)
•Due to limited size of study, focus on notable user
reaction to wayfinding experience
13
Results
•Participants split on usefulness of landmark
directions
–Participants 1, 2, 3*, 5 expressed affinity for landmarks
–Participants 4, 6, 7 preferred turn-based directions
–Difference in location between photo and participant
problematic
P1: When you have an actual picture of what's in front of the
person, that's excellent.
P2: The towers [of the Campanile] were behind trees, but I
trusted myself to go forward.
14
Results cont'd
• Some participants indifferent to unfamiliar landmark names,
others distracted by them
–Suggestions included only referring to landmark name when it was
destination, and in other important situations
• Compound directions, meant to give user knowledge of
upcoming turns, caused confusion
–Participants often thought follow-up turn direction was separate
–Participant 6 sometimes reversed the directions
15
Results cont'd
•Attentiveness
–Most participants stated that they paid attention to
surroundings, while others commented that familiarity
would change their interaction mode
I was thinking for a while, maybe I should stick this in my
pocket and react when the thing goes off. I didn't do it
because [I thought], “Well, by the time I take it out, will I
miss the instruction?” but obviously not, the instruction
stayed on which was good, so I should have.
16
Current steps
•Learning user-specific cost function
•Use of help button
•Time required to understand cue
•Experience sampling
•Learning transition probabilities
•Predict success or failure of cue based on cue
type and features of the state
•Better landmark-based images.
17
Learning Transition Probabilities by Linear
Regression
0.8
0.7
Difficulty incidence rate
0.6
0.5
0.4
0.3
0.2
0.1
0
-0.1
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
Model's predicted difficulty
18
Better Landmark-Based Images
19
Conclusion
• Wayfinding support for individuals with cognitive disabilities
should be customized – no one strategy is optimal for all
users
• The Markov Decision Process framework provides a way to
generate custom strategies that take into account user's
needs (abilities) and preferences
• Effectiveness of a strategy is inseparable from details of
presentation
• Provided initial results on learning custom models (success
probabilities)
20
Thank You
Alan L. Liu, Harlan Hile, Gaetano Borriello
Computer Science & Engineering
University of Washington, USA
Pat A. Brown, Mark Harniss, Kurt Johnson
Rehabilitation Medicine
University of Washington, USA
Henry Kautz
Computer Science
University of Rochester, USA
This material is based upon work supported by the National Institute on Disability and
Rehabilitation Research (NIDRR) Grant #H133A031739, Microsoft Research Intelligent
Systems for Assisted Cognition Award, and Nokia Research.