Artificial Intelligence in the
Design of Assistive Technology
Martha E. Pollack
Computer Science & Engineering
University of Michigan
An Example of “Use-Inspired
Research”


The research is motivated by an important societal
problem
But is not overly constrained by the problem
The research should have fundamental interest
 And thus be potentially useful for multiple
applications

The Population is Aging
> Age 60
World
2000
2050
China
6.9%
22.7%
10.0%
21.4%
India
7.5%
20.1%
Japan
23.3%
42.4%
Myanmar
6.8%
20.5%
Belarus
19.3%
37.6%
Germany
23.2%
34.5%
Italy
24.1%
40.6%
Australia
16.4%
29.9%
Netherlands
18.2%
30.7%
Fiji
5.7%
22.7%
Slovenia
19.2%
41.5%
Egypt
6.8%
18.7%
United States
16.1%
25.5%
Iran
6.4%
24.8%
Mexico
6.9%
26.2%
Jordan
4.6%
19.0%
Brazil
7.8%
25.9%
Columbia
6.9%
22.7%
Botswana
4.2%
6.0%
Ethiopia
4.6%
7.7%
Mali
3.9%
5.3%
Source: United Nations Population Division
http://esa.un.org/unpp/
The “Oldest Old”
> Age 80
2000
2050
World
1.1%
4.2%
United
States
3.2%
7.2%
China
0.9%
7.2%
Source: United Nations Population Division
http://esa.un.org/unpp/
Intelligent Technology
Can support people with
 Mobility impairment
Obstacle-avoiding wheelchairs

Declines in sensory function
Courtesy, R. Simpson, Univ.
Pittsburgh & AT Sciences
Courtesy Oticon
Adaptive digital hearing aids

Social and emotional challenges
Elder-friendly Internet and email systems

Cognitive decline
Courtesy
Generations on Line
The Challenge

Cognitive impairment can impact performance of daily
activities necessary for health and wellbeing



ADLs: eating, dressing, bathing, toileting, . . .
IADLs: managing medicines, housekeeping, arranging
transportation, preparing meals, . .
Can become difficult to follow a daily plan
Autominder: An Adaptive Reminder
System

Uses Artificial Intelligence techniques to

Model, update, and maintain the client’s plan


Monitor the client’s performance


Including complex temporal and causal constraints
Updating the plan as execution proceeds
Reason about what reminders to issue, and when

Ensuring compliance, without sacrificing client
independence
Autominder Interaction
Req/Opt
R
R
O
R
Activity
check
blood
glucose
lunch
TV
check
blood
glucose
Allowed
11:5012:10
12:151:00
17:0017:30
16:5517:15
Expected Observed
11:55
REMIND
12:42
12:40
REMIND
16:55
Autominder
WhatArchitecture
should the client do?
Activity Info
Plan
Updates
Plan
Manager
Client
Modeler
Sensor
Data
Inferred Activity
Client
Plan
Client
Model
smart
home
Activity
Info
Preferences
Client
Model
Info
Intelligent
Reminder
Generator
Reminders
Plan Manager


Maintains up-to-date record of client’s planned
activities
Techniques:
AI Planning
 Temporal Constraint Satisfaction

Plan Representation

Plans are structured sets of activities
causal connections
 temporal constraints




qualitative, quantitative, disjunctive
conditional constraints
Temporal constraints modeled as Disjunctive
Temporal Problems (DTPs)
Disjunctive Temporal Problems

A set of time points (variables) V and a set of
constraints C of the form:
lbji  Xi – Xj  ubji  …  lbmk  Xk – Xm  ubmk


Solution: assignment of times to all variables, so that
all constraints in C are satisfied
Generalization of the Simple Temporal Problems

Disjunctions critical to model non-overlap requirements
Example Temporal Constraints
Blood glucose should be checked between 11:50 and 12:10:
C1:
11:50  BS – TR  12:10
Lunch takes between 15 and 30 minutes:
C2:
:15  LE – LS  :30
The TV show takes 30 minutes:
C3:
:30  TVE – TVS  :30
Watching TV can begin at 17:00 or at 18:00:
C4:
17:00  TVS – TR  17:00 
18:00  TVS – TR  18:00
Checking glucose and watching TV should not overlap:
C5:
0  TVS – BE  0  BS – TVE
Dishes should be taken within 1 hour of finishing lunch.
C6:
0  DS – LE  60
DTP Solving as Meta-Level
Propagation
C1 : {c11 : y – x  5}
C2 : {c21 : w – y  5}  {c22 : x – y  -10} 
{c23 : z – y  5}
C3 : {c31 : y – w  -10}
NOT: c21, c31
NOT: c11, c22
Component STP: C1  c11, C2  c23 , C3  c31
One exact solution: {x = 0, y = 1, z = 2, w = 12}
Temporal Constraint Satisfaction




Efficiently solving DTPs
Add temporal uncertainty
[Tsamardinos & Pollack, AIJ 2003]
[Morris, Muscettola, & Vidal, IJCAI
2001, Venable & Yorke-Smith, IJCAI 2005, Rossi, Venable, & Smith, CP 2004]
causal uncertainty [Tsamardinos & Pollack, Constraints, 2003]
preference functions [Khatib et al., IJCAI 2001, 2003; Peintner &
Pollack, AAAI 2004, 2205, Morris, Morris, Khatib & Yorke-Smith, ICAPS
Wkshp. 2005, Sheini, Peintner, Sakallah & Pollack, CP 2005, Moffitt & Pollack,
AAAI 2006]

partial satisfaction

dynamic structure
hybrid constraints

[Moffitt & Pollack, FLAIRS 2005, Peintner,
Moffitt, & Pollack, ICAPS 2005, Moffitt & Pollack, IJCAI 2005, Liffiton,
Moffitt, Pollack, & Sakallah, IJCAI 2005]
[Schwartz & Pollack, ICAPS Wkshp. 2005]
[Moffitt, Peintner, & Pollack, AAAI 2005]
Efficiently Solving DTPs
Example: Removal of Subsumed Variables
If this assignment to
Ci is implied by the
partial assignment
above it, prune the
other values for Ci
Ci  cij
Ci  cik
Ci  cil
DTPs with Preferences


Plan includes exercising and a visit from a friend
Should finish exercise before visit or start after visit:
VS – EE  [5,] v ES – VE  [0,]
3
2
1
not allowed

V
0 5 10 15 … 
VS – EE
3
2
1
not allowed
0 5 10 15 … 
ES – VE
Now an optimization problem
Solution Technique

[AAAI 2006]
Convert each constraint into a set of valued constraints,
one per preference level
3
2
1
not allowed
V
0 5 10 15 … 
VS – EE
3
2
1
not allowed
V=1
0 5 10 15 … 
ES – VE
V=2
V=3
V
0 5 10 15 … 
VS – EE


0 5 10 15 … 
VS – EE
0 5 10 15 … 
ES – VE
0 5 10 15 … 
VS – EE
Apply branch-and-bound with partial satisfaction
Performs very well in comparison with previous methods
Same Techniques Apply Elsewhere


Rectangle Packing
Rectangle i must be contained within the
enclosing space of dimensions W x H:
xi  0, yi  0,

xi + wi  W,
yi + hi  H
Rectangle i and j must not overlap:
xi + wi  xj  xj + wj  xi  yi + hi  yj  yj + hj  yi

Orientation of rectangles adds hybrid
constraints:
xi + B(oi)  xj  xj + B(oj)  xi  …
Comparison of Approaches [ICAPS 2006]
For the case of optimal packings fixed orientations
Number of squares
12
16
19
State-of-the-art in…
VLSI / Floorplanning1
7:45:49
TIMES
OUT
TIMES OUT
Operations Research2
0:00:13
0:10:05
3:00:53:18
Artificial Intelligence
0:00:00
0:00:03
0:00:04:09
10 day time-out limit
1
2
(Chan and Markov, 2004)
(Clautiaux, Carlier, & Moukrim, 2004)
Autominder
Architecture
What
has the client done?
Activity Info
Plan
Updates
Plan
Manager
Client
Modeler
Sensor
Data
Inferred Activity
Client
Plan
Client
Model
smart
home
Activity
Info
Preferences
Client
Model
Info
Intelligent
Reminder
Generator
Reminders
Client Modeler
 Given what can be observed (sensor input, clock
time, stored plan, reminder information, etc.), infer
probabilities that various actions were performed
 Techniques:
 Wireless sensor networks
 Reasoning under uncertainty
 Ensuring privacy and security of data collected is
paramount!
Probabilistic Reasoning for Activity
Recognition


Use Hidden Markov Models (HMMs), Dynamic
Bayes Nets (DBNs), etc.
Example: Train one HMM per activity type;
observed variables = sensors firings
For discrete activities, compute probability of each
type
 Key question: segmentation

Make Tea:
Pr(faucet) = .6 Pr(faucet) = .5
Pr(cup) = .4
Pr(tea) = .45
...
...
Courtesy M. Philipose et al.,
Intel Research
Same Techniques Apply Elsewhere

Recent work at Intel Seattle on using these
techniques to train anesthesiologists
Autominder Architecture
Activity Info
Plan
Updates
Plan
Manager
Client
Modeler
Sensor
Data
Inferred Activity
What action should the Client
system
Plan
take?
Client
Model
Activity
Info
Preferences
smart
home
Client
Model
Info
Intelligent
Reminder
Generator
Reminders
Intelligent Reminder Generation

Given a client’s plan and its execution status:



Easy to generate reminders at earliest possible time of each
action
Harder to “remind well”
 Maximize likelihood of appropriate performance of
activities
 Allow flexibility
 Facilitate efficient performance
 Avoid annoying client and/or making the client overly
reliant
Techniques:


iterative refinement (local search)
machine learning
One Approach: Iterative Refinement
[AIPS 2002]
8:00
12:00
B
L
16:00
D
12:00
Midnight
TV
8:30
12:32
8:00
12:00
B
L
16:00
D
12:00
Midnight
8:00
12:00
B
L
12:00
Midnight
TV
TV
16:00
D
Alternative Approach: Learning
[ICML 2004]



Use reinforcement learning to induce the best
interaction strategy
Decide whether, when, and how to remind, given
information about the client’s “state”
Add a dynamic action proposer—the plan
manager—to a standard RL architecture
Augmented RL Architecture
Environment
Client
Sensors
Sensors
Actuators
Reminder
Production
Payoff
Agent
Adaptive
Reminder
Generator
Selected
Actions
Percepts
State Estimator
Client
Modeler
Estimated
State
Plan
Action Proposer
Plan Manager
Sample Experiment (with a client
simulator)
Autominder Platforms
Key Research Challenges





Integrated activity recognition—inside and outside the
home
Enhanced privacy assurance
Elaboration of machine learning techniques for
interaction strategies
Platform and interface design
Use of adaptive systems for additional purposes




Social integration
Cueing for behavioral adaptation
Incoporation of other assistance mechanisms (e.g., face
recognition)
Field testing and iterative design!