Adaptive Social Assistants:
Using Mobile Computing
Devices to Assist Individuals
with Cognitive Disabilities
Project Update
Cathy Bodine
University of Colorado
June 14, 2001
Project Objectives
• Develop power-aware mobile computing devices
that adapt to their users based upon observed and
predicted behavior.
• Configure these devices as adaptive social
assistants that simplify daily living and job-related
tasks for persons with cognitive disabilities.
• The role of the social assistant is to replace many of
the functions of a personal attendant, while
enhancing the independence and functioning of the
user as he or she engages in daily life activities as a
contributing member of society.
Project Team
•
•
•
•
•
•
•
•
•
•
•
•
John Bennett - CU Dept. of Comp. Science
Dirk Grunwald - CU Dept. of Comp. Science
Clayton Lewis - CU Dept. of Comp. Science
Michael Mozer - CU Dept. of Comp. Science
Tamara Sumner - CU Dept. of Comp. Science
Timothy Brown - CU Dept. of ECE and ITP
Cathy Bodine - UCHSC Depts. Of Peds. & Rehab. Med.;
Director, Assistive Technology Partners
Linda Crnic - UCHSC Depts. of Pediatrics and Psychiatry
Deborah Fidler - CSU Dept. of Human Development and
Family Studies
Lori Ramig - CU Dept. of Speech, Lang., and Hearing Sci.
Sally Rogers - UCHSC Dept. of Psychiatry
David Patterson - UCHSC Depts. of Biochemistry and
Molecular Genetics and Medicine
Project Expertise by Area
•
•
•
•
•
•
•
•
•
•
•
•
John Bennett (distributed, parallel, and mobile computing)
Dirk Grunwald (computer systems and networking)
Clayton Lewis (human-computer interaction)
Michael Mozer (machine learning)
Tamara Sumner (design and cognition, HCI)
Timothy Brown (wireless computing)
Cathy Bodine (assistive technology)
Linda Crnic (mental retardation)
Deborah Fidler (developmental disabilities, esp. outcomes)
Lori Ramig (speech pathology and remediation)
Sally Rogers (developmental disabilities, esp. autism)
David Patterson (genetics of developmental disorders)
Project Status
• Preproposal ($5M budget) submitted Fall ’00
(1 of 661 submitted)
• Favorable action on preproposal by NSF
(1 of 258 (39%) approved for full proposal)
• Full proposal submitted April ’01
(Sept. ’01 start date if funded)
(about $90M allocated => ~ 50 proposals will be
funded)
• $1M matching support pledged from Coleman
Institute
Core Technology
• Machine Learning
(the means by which the social assistants learn
from and adapt to user behavior)
• Mobility and Data Management
(power management, how we support both user
and device mobility, and how we represent, access,
update, and protect information)
• Human-Computer Interaction
(how the user interacts with the assistant)
Target Tasks
• Route Navigation
• Assistant provides auditory / verbal instructions and
observes trajectories to build up expectations. Significant
violations of these expectations results in warning or other
remedial action.
• Communication
• Assistant provides augmentative / alternative
communication. “Keypad” adapts to user (not the other way
around). We intend to build on work of Enkidu and Saltillo.
• Memory Prosthesis
• Assistant provides memory cues to help user stay on and
complete tasks, as well improving ability to complete tasks
more successfully. We intend to build upon work of
AbleLink, integrating adaptive behavior into “Pocket Coach”
like devices.
Design Issues
• Safety, Welfare & Privacy
• Reliability, Durability, Ease of Use & Wearability
• High Abandonment Rate of Assistive Devices
• Training
• The Role of Adaptation
Participant Selection
• Working or job-seeking adult aged ~16 - ~40
• Diagnosed with developmental disability such
as one of Down, Williams, fragile X, or PraderWilli syndromes, or autism
• Ambulatory and within gross normal visual and
hearing acuity limits
• Able to follow single step instructions
• Must have 24/7 emergency support
• For now, exclude those with clinical range
aberrant behaviors, serious communicable
diseases, severe motor impairments
Syndrome prevalence
• Down syndrome (the most common genetic
mental retardation syndrome)
• 1 in 700-1000 live births
• Williams syndrome
• 1 in 20,000 live births
• Prader-Willi syndrome
• 1 in 10,000 live births
• fragile X syndrome
• 1 in 2,000- 4,000 live births
Targeting Syndrome Differences
• Persons with Down Syndrome
• Tend to require visual support of auditory information
• Show strength in sequential over simultaneous processing
=> social assistant should provide step-by-step
instructions
• Persons with Williams Syndrome
• Tend to require auditory support of visual information
• Persons with Prader-Willi or fragile X syndrome
• Show strength in simultaneous over sequential processing
=> social assistant should provide instructions in larger,
integrated context
Broader Impact of Project
• Increasing independence and improving quality of
life of people with cognitive disabilities
• Extension to general population, including:
•
•
•
•
•
the elderly
persons who cannot read
persons who are blind
persons with traumatic brain injury (impaired memory)
anyone who needs help with daily tasks, including busy
executives and college professors
• Education and outreach
• Improving our understanding of individuals with
cognitive disabilities
• Technology transfer
Five Year Research Plan
• Year 1 – [Fact Gathering, Exploration, and
Participatory Design]
• Year 2 – [Deliver First (Basic) Prototypes]
• Year 3 – [Evaluation; Additional Deployment
and Testing]
• Year 4 – [Design Refinement]
• Year 5 - [Evaluation, Analysis, and Technology
Transfer]
Year 1 – [Fact Gathering, Exploration, and Participatory Design]
• Establish dialogue with users, family members, caregivers, clinicians and
manufacturers of assistive technology devices in order to develop an
understanding of users needs and capabilities related to the target tasks.
Refine target tasks as necessary.
• Design and implement data encryption, transmission and transaction
mechanisms.
• Establish project web site.
• Begin development of laboratory prototypes of social assistants.
• Begin exploration of different user interface mechanisms and sensor
technologies.
• Develop appropriate machine learning objective functions for device
adaptation.
• Experiment with cellular, GPS, and other location sensing technologies.
• Complete Institutional Review Board approval process prior to device
deployment in Year 2.
• Develop and offer joint course in assistive technology for engineering and
health science graduate students.
• Establish project Advisory Board
Year 2 – [Deliver First (Basic) Prototypes]
• Complete high-level design of system architecture.
• Establish initial server infrastructure for both development and
field use.
• Develop basic data and mobility management support
mechanisms.
• Develop initial task modeling interface for use by clinicians and
caregivers.
• Perform initial testing of machine learning objective functions
for device adaptation.
• Develop initial user interface design.
• Deploy 40 field units (20 at midyear, 20 more by year end.)
Release new software to field at 6-month intervals.
• Begin to collect and analyze user data.
Year 3 – [Evaluation; Additional Deployment and Testing]
• Expand user population and task repertoire.
• Expand server infrastructure as needed.
• Refine and extend task modeling interface for use by clinicians
and caregivers.
• Evaluate device adaptation.
• Extend basic data and mobility management support
mechanisms.
• Complete initial user interface design.
• Deploy an additional 40 field units (20 every 6 months).
Release new software to field at 6-month intervals.
Year 4 – [Design Refinement]
• Expand user population and task repertoire.
• Expand server infrastructure as needed.
• Develop final task modeling interface for use by clinicians and
caregivers.
• Continue to evaluate device adaptation and refine ML objective
functions based on user data.
• Evaluate and refine user interface design based upon field
data.
• Deploy an additional 40 field units (20 every 6 months).
Release new software to field at 6-month intervals.
• Initiate technology transfer.
Year 5 - [Evaluation, Analysis, and Technology Transfer]
• Expand user population to full size and complete task
repertoire.
• Expand server infrastructure as needed.
• Finalize task modeling interface for use by clinicians and
caregivers.
• Finalize user interface design.
• Evaluate device adaptation based on analysis of user data.
• Deploy 20 additional units and final software release at mid
year.
• Perform comprehensive evaluation of system architecture and
design.
• Complete transfer of project technology to commercial
developer.