ADL Assistant: An Aware Remote
Caregiving System
Tom Keating
Eugene Research Institute
Ray Keating
Pontimax Technologies, Inc.
Remote Caregiving Support
• Integrating home sensor networks, smart software and webbased tools to provide data for consumers and caregivers
• Applications:
- Providing long distance behavioral support
- Helping avoid victimization
- Monitoring physical health and communicating information to
health care providers
- Monitoring environmental changes related to health and safety
Test Bed Example of Remote
Monitoring in Practice
Main Components
 Picture Planner™ activity planning and prompting
application
 X10-based home sensor network
 Home automation software
 Consumer and caregiver monitoring computers
 Intelligent activity recognition software
 Webcams or IP Cameras
Apartment Layout
7 PIR motion sensors
1 magnetic reed sensor
CM15 powerline
interface
Remote Desktop View
Web Portal for Caregiver
Bath
Shower
Bdr
LR
DR
Porch
Fr. dr.
Kitchen
Summaries
______________________M_____T____W____Th____F____S____Su
Showers
1
1
0
1
0
1
0
Sleep hours
6.5
12
7
8
8.5
8.5
6
Nighttime door
0
0
0
0
0
0
0
Nighttime bath use
2
1
1
1
2
3
1
Benefits
• Improved self-management for activity and task
completion
• Feasible universally cost effective remote support
technology
• Caregiver effectiveness increased
• Caregiver peace of mind increased
• Some functionality increasingly available “off the
shelf”
Smart Prompting Prototype:
Planned Activity vs. Actual Behavior
Conceptual Model for Behavioral
Inferencing in Residential Settings
Behavioral Inferencing for
Activities of Daily Living
The “Sensible”
Spatio-Temporal Way
Objective 1
Provide near real time oversight of consumer in
their activities of daily living by:
•
Infer simple-to-complex ADL of interest from motion
sensor data
•
Selectively generate multi modal, immediate (email,
SMS texting) advisories to caregiver
Objective 2
• Provide positive consumer ADL direction though:
•
Use of inferred events, activities & behaviors to drive
generation of multi-media audio & video prompts
•
Communicate prompts to consumer at residence
domain area
Objective 3
• Use least obtrusive, least expensive (!)
means possible to achieve Objectives 1 & 2
Working Back From Objective 3
•
Passive infrared (PIR) sensors can detect motion
•
Are inconspicuous
•
Are inexpensive ($15 - $30)
•
Can be easily installed at the consumer’s residence
What’s a PIR Sensor?
• Are area sensors
• Fire on motion in their field of view (FOV)
• Commonly used to turn on a light when the
room is entered
Can they meet the challenges posed by objectives 1 & 2?
The Challenge is one of BOTH
Sense & Sensibility
• “Sensing”
– Can the ADL be captured by motion sensors?
• “Sensibility”
– Do ADL have characteristic patterns in the form of
spatio-temporal motion occurrences?
– What would these ADL characteristics be based on?
Can the ADL be Captured by
Motion Sensors?
• To answer the question…
– Let’s take a look at how the ADL motion data
generation and collection occurs using PIR
type motion sensors
Capturing the ADL Motion Data
• Coverage
– Consumer residence “domain” divided into
“Domain Areas”:
- Shower, bath, bedroom, living room,
dining area, kitchen, front door, porch
• One PIR sensor per domain area
• Sensing Capabilities
– Installed PIR sensors fire once every ten
seconds for motions in FOV
– Motions continuously detected for all
domain areas
Capturing the ADL Motion Data,
Con’t.
• Real time data collection
– Consumer site unit receives transmitted sensor motion data
– Time stamps location tagged motion data
– Continuous uploading to database server
OK, it seems that movements at locations (spatial) can be
detected by motion sensors and chronologically (temporal)
recorded in a database
- How are these spatio-temporal motion patterns
recognized as a particular ADL?
Characterizing ADL with
Spatio-Temporal Fact Patterns
• Empirical studies indicate that many ADL can be characterized
as:
– A Certain number of motions
– Within a time period
– At a particular location
Clearly, location (& maybe time) context is paramount!
Example: Taking a daily shower
• If more than X motions within Y duration at the SHOWER domain
area, then consumer has taken their daily shower!
How to Define the SpatioTemporal Fact Patterns?
• Might…
– Caregiver experience…
– Mathematical analysis…
– Behavioral research…
…Be
used?
How to Define the Spatio-Temporal
Fact Patterns, con’t-2
• Example:
– The Caregiver knows from experience (“A Posteriori”)
that the Consumer generally takes five to ten minute
showers.
– The motion sensor detects movement, at most,
once every ten seconds so in a five to ten minute
period there should be a pattern of motion detections
in the range of 50 to 100 occurrences, assuming
constant movement in the shower.
Let’s use some data analysis to add some
Assurance to our caregiver’s experience…
How to Define the SpatioTemporal Fact Patterns, con’t-3
• Use Time-Location Based Clustering Analysis:
- Do the data collection over some sufficient period -- say a
week
- Aggregate the sensor data occurrences for each
domain area by time period, using the caregiver’s
A Posteriori observation.
• Results:
The basic spatio-temporal “shower taken” pattern
confirmed, albeit, at a lower occurrence rate of 30 to 60
occurrences over a five to ten minute duration.
How do we Apply our ADL Fact
Patterns to the Data?
• Three requirements:
– Describe the spatio-temporal fact patterns
– Store them for access from the database
– Use them to inference ADL occurrences
Use Spatio-Temporal Predicate
Expressions to Specify the ADL Patterns
• General form:
– @Pattern Meta Function [tag-value argument set]:
• Spatio-temporal context (tag-value context reference:value
set)
– Predicate Operator (=,>=,<=, !=)
– ! Pattern Meta Criterion [criterion argument(s)]:
• Spatio-temporal context (tag-value context reference:value
set)
• Showering_Activity:
@OCCURRENCES[SCANFREQ=’10’,DURATION=’10’]:
CONTEXT(LOCATION=‘SHOWER‘, TIMESCOPE=‘2000-2200’ )
>= !OCCURRENCES[CRITERION_OCCURRENCES,CRITERION_DURATION]:
CONTEXT(LOCATION=‘SHOWER', TIMEFRAME=‘0000-2400’)
CAUTION:
• Many ADL patterns can’t be reliably inferred by just
a single motion pattern
• Can the Level of “Sensibility” be raised?
– Sure, just specify additional inferencing patterns
to form a predicate inferencing chain:
• If fact pattern-1, then if fact pattern-N…
ADL of interest occurred
Spatio-Temporal Fact Patterns
• For Shower taken, for example, how can it be assured
that the consumer didn’t just step in and out of the
shower?
Simple – define a spatio-temporal fact pattern to
establish that the consumer was continuously in the
shower…
• CONTINUOUS_PRESENCE-SHOWER:
@DURATION[SPAN='CONTINUOUS']:CONTEXT(LOCATION=
‘SHOWER',TENSE='CURRENT')
>=
!DURATION[ACTUAL_DURATION]:CONTEXT
(LOCATION=‘SHOWER',TEMPORAL='CURRENT')
Inference Chaining of ADL
Predicate Patterns
• A “shower taken” is now defined as:
Shower_Taken
Showering -activity
Continuous-Presence-Shower
• Chained Inferencing Predicate Description
– If the sensor data pattern described by Showering_activity occurred AND
if at that same time CONTINUOUS_PRESENCE-SHOWER were true,
then it is very likely that a shower was taken by the consumer!
Behavioral Plans
• Inferencing hierarchies of ADL concepts can be defined to for a
“Plan”
• The ADL concepts provided are:
Plans
Behaviors
Activity patterns
Activities
Events
- Made up of behaviors
- Made up of activity patterns
- Made up of activities
- Made up of events
- Made up of inference fact patterns
• Currently goal times can be specified for each concept entity.
• Soon, goal expressions consisting of inference
fact patterns will be specifiable for all
behavior plan concepts.
General and Predicate Events
• Can be defined as stand-alone entities
• Consist of inference fact patterns
• Can be “scored” or not - scored have goal times and
have non-occurrence as well as occurrence inferencing
performed
• SHOWER_TAKEN is a general event
• Location context maintenance accomplished by a set of
general events
• Predicate events can be defined to be inferenced upon
occurrence of a specified triggering event.
Consequent Actions
• Can be specified for any “inferencing entity”
– Behavior plan concepts: inferred events, activities, etc.
– General and predicate events
• Performed on the occurrence of the inferred entity
• Can be specified to be performed on positive occurrence,
negative occurrence of ALL
• Currently email event occurrence reports, cell phone (SMS)
texting and text-to-speech prompt generation
• Can be specified as “external action” consequent actions.
The Inferencing Agent uses the Pattern
Meta Specification to do the Inferencing
• Retrieves the stored behavior plans, general & predicate
events, and inference fact pattern meta specifications
• Generates the necessary data base queries and inferencing
rules and fact patterns
• Performs the Inferencing per the fact pattern’s datescope and
timescope, at the scan frequency specified
• Accomplishes any specified consequent actions
Questions?
• tkeating@eugeneresearch.org
• rayk@pontimax.com