From: AAAI Technical Report WS-02-02. Compilation copyright © 2002, AAAI (www.aaai.org). All rights reserved. An Overview of the Assisted Cognition Project Henry Kautz, Dieter Fox, OrenEtzioni, GaetanoBorriello, and Larry Arnstein Dept. Computer Science & Engineering University of Washington Seattle, WA98195 Abstract The rise of Alzheimer’s disease is one of the greatest health crises facing the industrialized world. Today, approximatelyfour million Americanssuffer from Aizheimer’sdisease; by 2050, the numberis expectedto rise to 15 million people. As a result of the increasing longevity of the elderly, manysufferers are nowaware that their capacities to remember, to learn, and to carry out the tasks of everydaylife are slowlybeinglost. The Assisted Cognition Project is a newjoint effort between the University of Washington’sDepartmentof ComputerScience, MedicalCenter, and Alzheimer’sDisease ResearchCenterthat is exploringthe use of AI systems to support and enhancethe independenceand quality of life of Alzheimer’spatients. Thegoal of the Assisted Cognitionproject is to developnovelcomputersystemsthat will enhancethe quality of life of peoplesuffering from Alzheimer’sDiseaseand similar cognitive disorders. Assisted Cognition systems use ubiquitous computing and artificial intelligence technologyto replace someof the memoryand problem-solvingabilities that have been lost by an Aizheimer’spatient. Twoconcrete examplesof the Assisted Cognition systems we are developing are an ACTIVITY COMPASS that helps reduce spatial disorientation both inside and outside the home, and an ADAPTIVE PROMPTER that helps patients carry out multi-step everydaytasks. Introduction In the last 50 years Alzheimer’s disease has changed from a relatively rare ailment to one of the defining characteristics of industrialized societies. In 1950 at most 200,000 people in the UShad Alzheimer’s disease; this number increased to 500,000 by 1975, and stands at 4 million today (National Institute 2000). By 2050 the numberof Alzheimer’s patients in the US is expected to be 15 million, out of a world total of 80 million (Shenk 2001). The economic toll of Alzheimer’s is staggering: caring for a single person with Alzheimer’s disease costs more than $47,000 a year whether a person is at homeor in a nursing home(Rice, Fox, & Max 1993). Alzheimer’s disease is a specific medical condition distinct from the "normal" kinds of forgetfulness that Copyright(~) 2002, AmericanAssociationfor Artificial Intelligence (www.aaai.org).All rights reserved. 6O come with aging (Hooyman& Kiyak 2001). Reisberg’s seven stage theory describes the typical progress of the disease. In stage 1, the disease has begun but is asymptomatic. In early Alzheimer’s, stages 2-3, patients exhibit forgetfulness, easily becomelost, have difficulty with word and name recognition, misplace objects, etc. In the middle stages, 4-5, the ability to perform multistep tasks without assistance and frequent prompts is often impaired. Late Alzheimer’s, stages 6-7, include loss of awarenessof surroundings, loss of speech, and finally loss of basic psychomotorskills. For centuries technology has been created to help people with physical limitations: consider eyeglasses, hearing aids, wheelchairs, and artificial limbs. Wehave coined the name "Assisted Cognition" for new synthesis of AI and ubiquitious computing technology designed to help people with the cognitive limitations associated with Alzheimer’s disease and similar conditions. With physical limitations it is obvious that an individual’s level of function is determined by the total system of the individual and his or her environment. For example, a wheelchair bound person in a city without wheelchair ramps and wheelchair accessible buses has limited mobility, while that same individual in a wheelchair-friendly city can have high mobility. In judging cognitive function it is likewise important to take account of the environmental context. For example, in a familiar, static environment, a person in early stages of Alzheimer’s mayshowa high level of function in performing everyday tasks as dressing and housework. The familiar sights and sounds of the home provide implicit prompts to initiate appropriate behavior. For example, when the person wakes in the morning the sight of the light slanting in the window,the smell of coffee coming up the stairs, and the sounds of the person’s spouse movingabout in the kitchen below triggers the start of the person’s morning routine: e.g., getting out of bed and walking toward the bathroom. Place the same person in a novel environment and he or she have difficulty in performing such basic tasks (Lawton 1983). Just as important as the physical environment is the social context of other people. For example, an Alzheimer’s patient may cometo rely on the memory,remindings, guidance, and prompts of a caregiver, such as a spouse or other family member(Reisberg et al. 1982; Shenk2001). Thusthe social environmentcan provide active interventions that extendthe patient’s ability to handle the challenges of everydaylife. But there is a limit to any person’s capability to provide such help: physical and emotional ’burnout" of caregivers, often with serious health consequences,is a common phenomena (Hooyman& Kiyak 2001). The goal of Assisted Cognition is to develop computer systems that can provide such active assistance to an Alzheimer’spatient. In brief, Assisted Cognition systems(i) sense aspects of an individual’s location and environment,both outdoors and at home,relying on a wide range of sensors such as Global Positioning Systems(GPS),active badges, motiondetectors, and other ubiquitouscomputinginfrastructure; (ii) learn to interpret pattemsof everydaybehavior, and recognize signs of distress, disorientation, or confusion,using techniques fromstate estimation, plan recognition, and machine learning; and (iii) offer help to patients through various kinds of interventions, and alert humancaregivers in case of danger. Theultimate aim is for Assisted Cognitionsystemsto becomean integral part of a patient’s physical and social contexts. After providingan overviewof organization of the Assisted Cognitionproject at the University of Washington,wewill briefly describe two specific Assisted Cognitionsystemswe are developing, the ACTIVITY COMPASS and the ADAPTIVE PROMPTER. The Assisted Cognition Project at UW The Assisted CognitionProject is an interdisciplinary effort between manyorganizations with expertise in computerscience and/or care of Alzheimer’spatients. The first category includes the University of Washington’s Dept. of ComputerScience and Engineering, the Dept. of Electrical Engineering,and Intel Research.The secondincludes the Alzheimer’sDisease ResearchCenter (ADRC),the UWschools of Nursing, Health and Public Policy, and Medicine, and the UWMedicalCenter. UWis a national center for research on Alzheimer’s disease. The ADRC is one of 32 centers fundedby the National Institute on Aging, and has a unique emphasis on the care of Alzheimer’spatients in the home.The UWMedical Center runs the Alzheimer’s Disease Patient Registry, whichtracks a populationof 3,500people, both normallyaging and those with Alzheimer’s,and the school of Medicinehosts the National AlzheimerCoordinating Center. An important partner in the Assisted Cognition Project is EliteCare, a private companywhosework in building and running retirement homesis pushing the state of the art in ubiquitiouscomputing applicationsfor the elderly (ComputerWorld). Oatfield Estates is a living prototype homefor 60 residents in Portland, OR.In additionto providinghigh-speedinternet access and customizedapplications for its residents, the homesenses and records nearly all activity going on in or around it’s campus.This includes the movements of all resi61 dents and staff, operation of all lights and appliances, movement of all doors, etc. This enormousand detailed real-world source provides the primaryraw data for our project. In addition, weare also gathering other realworld data sources. For example, a major part of the ACTIVITY COMPASS system involves learning patterns of behavior from GPSdata. Wehave built small GPS recorders fromoff-the-shelf hardwareand are gathering initial data sets fromthe movements of students involved in the project. Ourrich set of collaborationsallow us to take a principled approachto Assisted Cognition, wherein(i) research is basedon real, not simulateddata, (ii) applications are designedin collaboration with experts on care of the elderly, and (iii) developedsystemsare tested real patient populations,rather than simplyshownoff as a one-time demo. Examples of Assisted Cognition Systems Webegin by describing two concrete examplesof Assisted Cognition systems we are developing, and then briefly describing someof the common underlying technical research. The Activity Compass The ACTIVITY COMPASS is an Assisted Cognition tool that helps direct a disorientedpersontowardstheir destination. Thecompassis basedon our client/server architecture (see Figure 1) wherethe client handles interaction withthe user, and the server stores sensor readings, constructed models, and backgroundinformation. The ACTIVITY COMPASS client is a hand held device. We refer to it as a compassbecausethe central user interface elementis a large arrowthat directs a personbased on their current location and the system’sassessmentof their desireddestination. For example, whencoupled with a Global Positioning System(GPS), the compassmaydetect that a patient is wanderingaround and appears to be lost. In this case, it maysuggest that the patient headhomeby pointing the arrowin an appropriate direction, and displaying the graphic for "home".Inside the home,the compasscan rely on indoorsensors to direct a persontowardsthe kitchen whenmeal time is overdue,or towards the medicinecabinet (with appropriate graphics, again) whenappropriate. Theseexamplesmayappear straight forward, but several capabilities haveto be in place before the ACTIVITY COMPASS Canfunction effectively. First, the compass needsto construct a high level modelof a person’sactivities basedon sensors readings and correspondingbackgroundinformation. Outdoors,for example,the sensors are simply GPScoordinates and the backgroundinformation is a detailed mapannotated with key locations and addresses(e.g., "home","doctor’s office"). In this case, the compassneeds to record the person’s typical activities such as favorite walks, destinations, etc.and construct modelsof typical behavior. In addition, GPS readings can be augmentedby additional sensors to com- pute the person’sorientation; outdoors, attaching a tiny "real" compassto the client can provide orientation information, and triangulation on IR signals can achieve the sameeffect indoors. Next, plan recognition is essential for the compassto makesense of the stream of sensor readings about the patient current actions. Finally, the compassneeds to assess whethera deviation froma normalpattern constitutes a causefor interventionor not. For example,if the patient is walkingto a newlocation in a commercial area she maybe going to a newly openedcafe; the compass should not attempted to direct her home.However,if the patient has beenwandering in circles for hours, then intervention is clearly appropriate. Intermediatecases require appropriate reasoningunderuncertainty. The ACTIVITY COMPASS is designed to learn from its ownexperience of interacting with the patient. Which interventions are well received? Whichare ignored? Is the patient’s behaviorchangingover time? Aremoreinterventions warranted?Frequently, the compasswill be able to detect the answersto these questions automatically and updateits behavioraccordingly. The Adaptive Prompter A commonproblem in the early to middle stages of Alzheimer’sis a difficulty in carryingout complex tasks, while the ability to performsimple actions is relatively unimpaired(Reisberg et al. 1982). Somecomplextasks -- such as driving -- should certainly not be attempted by anyonewith Alzheimer’s,while manyother activities of daily living -- such as personal grooming,hobbies, and housework--should be supported as long as possible. For example,an Alzheimer’spatient maybe able to performthe individual steps in dressing him or herself, but be unable to recall the proper sequenceof actions, e.g. that socksgo on before shoes. The adaptiveprompteris a systemthat helps guide an impairedindividual throughthe steps of such an activity. Input to the system comesfrom a sensor network embeddedin the homeenvironment.Data from the network,together with other informationsuch as the time of day, is integratedto create a modelthat predicts whatthe patient is trying to do. Thesensors mayinclude ones for sound, motion,position of objects, movement of doors, operation of appliances, and so on. For example, the systemmightnote that it is morning,and that the patient entered the bathroomand turned on the sink. Sometime passes, and the patient remains motionless. The system predicts that the "morningtoothbrushing and bathing" activity has begunbut becomestalled. Finally, the systemdecidesto interveneby promptingthe patient to pick up the toothbrush. Apromptcould be verbal or visual (e.g., using a spotlight that can highlightany object in the room),or a combination. Note that a promptis not given unless it is deemednecessary, and promptsare not pre-programmed for certain times of the time. AnAssistedCognitionsystem must show"emotional intelligence" (Klein, Moon, &Picard 2(X)2),becauseunnecessarypromptingis likely to be moreharmfulthan helpful. 62 Technical Components Both the ACTIVITY COMPASSand ADAPTIVE PROMPTER are based on a layered architecture (see Fig. 1) linking sensor data to behaviors, behaviorsto plans, and plans to potential interventions. Eachlayer takes in noisy and uncertain information,abstracts and fuses the data to reduce (but not alwayseliminate) uncertainty, and passes this higher-level view of the world to the next layer. Feedbackfromthe effects of invention feed back downthrough the layers, in order to improvethe accuracy and effectiveness of the underlying models. Belowwe briefly describe someof the ongoingresearch issues for eachlayer. Sensor Layer A fundamental and commonsubproblem to manyAssisted Cognitionapplications is the issue of localization (Brumitt & Sharer 2001) -- namely, knowingthe position of objects and/or people with a certain degree of accuracyand precision. In this project, wewill install and investigate a variety of sensorsfor locationtracking, including infrared based active badges, GPS,Berkeley motes(attached to householdobjects), and accelerometers (Hightower&Borriello 2001). These sensors have very different capabilities, ranging fromsub-centimeter distance accuracy(infrared ID tags) to purely incremental motionsensing (accelerometers).Otherkinds of sensors we are employinginclude weight sensors (on beds and furniture) and monitorson all lights and appliances. Our Intel Researchpartners are nowdesigning specialized sensors for use in a bathroom,including water and waste flow sensors and an instrumented medicine cabinet. Data Fusion Layer The ability to extract information from a continuous streamof data collected by a large numberof sensors is one of the fundamentalproblemsfor Assisted Cognition systems. In order to deal with noisy sensor information weapply Bayesfilters, whichestimate posterior probability densities overthe completestate of a dynamicsystem. In the AssistedCognitionproject, the state vector can contain a variety of information such as the location of a person, the current weathercondition, or the fact whetherthe person wears a rain coat. The sensor modeldescribes the uncertainty of the sensor in connection with a modelof the environment.Posterior densities over the state are updatedusing a version of Baye’srule. All variables can be high-dimensionalvectors yielding a potentially exponential growthin complexity.Fortunately, independences betweendifferent parts of the state space can be used to reducethe complexityof the reasoning process (Koller & Lerner 2001). Figure 2 shows exampleof the track of a user of an early prototypeof the activity compassas he walks around the UWcampus. A major preconditionfor successful location tracking is the availability of a mapof the environmentdescribing the sensors andtheir locations. Asdiscussedin (Brumitt &Shafer 2001), geometricmodelscan significantly Client Server Sensors Decision Making (graphics,speech) Plan Recognition Data Fusion DataCollection Figure 1: Architecture of Assisted Cognition systems. The server module consists of layers of increasing levels of abstraction. Probabilistic models of sensors and the environment are applied to process noisy sensor data. User activities are tracked using hierarchical Bayesian models. Probabilistic descriptions of these activities help to determine the user’s plans and goals, which enable the system to choose which interactions to trigger. The models are continuously updated and refined based on information collected by the sensors. Figure 2: An prototype activity compass monitoring the movementof a user as he walks around the UWcampus. The x’d line is based on the raw GPSreadings. The smoothline is the actual trail of the user. increase the performanceof ubiquitous computingenvironments. Unfortunately, such models do not exist for most homeenvironments,and a manualconstruction can be extremelydifficult. Weproposeto build geometric mapsusing a mobile robot as an active sensing device (Thrun, Burgard, & Fox 2000). A map of a 50m x 50mlarge environmentand the mobilerobot used to learn this mapis shownin Figure1. Behavior Recognition Layer Theoutput of the data fusion layer are probability distributions over locations of people and objects at each point in time. These state sequences form the basis for the estimation of user activities. Overthe last years, there has been tremendousprogress in the developmentof state-based Bayesian approachesfor user modeling(Horvitz et al. 1998) and humanaction recognition (Pentland 1995). Theseapproachesconnect seamlessly with sample-based Bayesfilters and they are well- suited to bridge the gap betweenlowlevel state estimation and higher level behaviorrecognition. For example, a sequenceof locations leading to the grocerystore followedby a loss of GPSsignal can be used to extract a highprobability that the person is currently buyinggroceries. Thetechniques discussed so far do not only apply to location tracking but extend directly to morecomplex scenarios like the ADAPTIVE PROMPTER. For example, the state can additionally contain information whether the faucet is running or whether the person took the toothbrush from the bathroomcabinet. Plan and Intention Recognition Layer Plan recognition is the task of identifying an actor’s plans andgoals givena partial viewof that actor’s behavior (Schmidt, Sridharan, &Goodson1978; Kautz1991). Solutions to this problemare essential, becauseif the computeris to provide appropriate assistance to a per- son it needsat least a roughunderstandingof the goals the person is trying to achieve, and prototypical methods (i.e. plans) for achievingthose goals. Thecomputer needsto assess whetherthe plan is proceedingsmoothly (no interventionis required)or, alternatively, whetherthe personis stymiedand intervention is appropriate. Acentral idea in plan recognitionis that plans are hierarchical in nature, with a lattice of moreand moreabstract plans connectingobservableactions to high-level goals. Weare currently developinga probabilistic version of the version-spaceapproachto plan recognition (Lesh Etzioni 1995;Lau, Domingos,&Weld2001), in order to handle uncertain observationsand plans that maycontain errors on the part of the performer.This is an openand challengingarea of research. Intervention Layer Oneof the key decisions for any intelligent user interface, and for AssistedCognitionsystemsin particular, is under whatcircumstancesshould the systeminitiate an interactionwiththe user. If the systemis overlyreluctant to makesuggestions,then it risks being ineffective; but if it is overlyintrusive, thenit is likely to be turnedoff by the user. Akey role for the plan recognitionlayer is to discriminate betweenthe followingcases: the patient is executing a plan successfullyor merelypottering about and oughtto be left in peace;the patient is executinga flawed plan but a specific suggestionor a gentle reminderwill get her on track; the patient is disoriented or confused and needsto be re-directed; or the patient is in danger and an alert is appropriate. Making such discriminations reliably is one of the keychallengesof this project. One sourceof informationthat will help is endowing our system with a modicum of "emotionalintelligence" (Klein, Moon,&Picard 2002)in an attemptto track not only the patient’s actions but also her emotionalstate. Emotions as measuredby voice timber and intonation, skin conductivity, and pulse rate can provideimportantcuesas to whetherthe patient needs help, and howshe is responding to specific suggestions. Weformulate the discrimination problem in a decision-theoretic framework(see (Etzioni 1991; Horvitz et al. 1998)) that weighs the following factors: (i) The benefit to the patient from the system’s suggestion; (ii) The probability the system’s model the situation is accurate; and (iii) The cost of making a mistake, in terms of negative impact on both the patient’s health and perceivedusefulness of the system. Wetrack each intervention (and each "close call" wherea decision was madeto not intervene) along with the factors that led to that decision,and subsequentreadings of the patient’s and the environment’sstate. This data formsthe basis for negative and positive examples that help the Assisted Cognitionsystemevolve its policies regarding whento intervene and whento hold back. Related Work There are a numberof recent projects on AI systemsfor care of the elderly, but none(to our knowledge) targeted at Alzheimer’spatients, or basedon large datasets from a potential user population. TheAwareHomeat Georgia Techprototypes technologies to create a homethat can perceive and assist its occupants(Sanders 2000). The Nursebotproject at CMU, U. Pittsburgh, and U. Michigan (Baltus et al. 2000) aims at developing personal robots to help elderly peopleduringtheir everydaylives. Research in Learning Humansproject at the MITMedia Lab (Pentland 1995)has developedprobabilistic algorihtms for behaviorrecognition that are used in our work,althoughwe are focusingon a distinct set of input modalities (GPSand motionrather than vision). Finally, general workon behavior recognition with the goal of creatingintelligent user interfacesis the focus of several projects at Microsoft Research, including Easy Living (Brumitt &Shafer 2001)and Attentional UserInterfaces (Horvitz &Paek 2001). Project Status OrganizationAssisted CognitionProject at the University of Washingtontook place in the Fall of 2001, and an initial teamof faculty, students, and staff fromIntel Research and EliteCare are nowengagedin laying the technical groundwork:building the sensor infrastructure, collecting data, and implementing initial versions of the data fusion layer. In parallel we are workingon a principled approachto the design and linkage of the behavior and plan recognition layers. In the Spring of 2002 we began an ongoing seminar on Assisted Cognition, bringing together medicaland engineeringfaculty and students. Webelieve that research on AI for cognitive aids has boundlessopportunityfor bothpractical applications and fundamentalinsights into perception, reasoning, and human-computer interaction. References Baltus,G.; Fox,D.; Gemperle, E; Goetz,J.; Hirsh,T.; Magaritis, D.; Montemerlo, M.;Pineau,J.; Roy,N.; Schulte,J.; andThrun,S. 2000.Towards personalservicerobots for the elderly. In Proe.of the Workshop on InteractiveRoboticsand Entertainment(WIRE-2000). Brumitt,B., and Shafer,S. 2001.Better living throughgeometry.Personalarut UbiquitousComputing 5(1). ComputerWorld. Oatfield estates elite care assisted living units. A Search for NewHeroes: The 2001 ComputerWorld21st CenturyAchievement Awards,online at http’//www.cwheroes.org/. Etzioni, O. 1991.Embedding decision-analyticcontrol in a learningarchitecture.Artificial Intelligence49(1-3):129160. Hightower, J., and Borriello, G. 2001.Locationsystemsfor ubiquitouscomputing.Computer 34(8). Hooyman, N. R., and Kiyak,H. A. 2001. Social Gerontology:a Multidisciplinary Perspective(6th Edition).Allynand Bacon. 64 Horvitz, E., and Pack, T. 2001. Harnessingmodelsof users’ goals to mediateclarification dialog in spokenlanguagesystems. In Proceedingsof the Eighth International Conference on User Modeling. Horvitz, E.; Breese, J.; Heckerman, D.; Hovel, D.; and Romincise, K. 1998. The iumiere project: Bayesian user modeling for inferring the goals and needs of softwareusers. In Proceedingsof the FourteenthConferenceon Uncertaintyfir Artificial Intelligence. Kautz, H. 1991. A formal theory of plan recognition and its implementation.In Allen, J.; Kautz,H.; Pelavin, R.; and Tennenberg, J., eels., ReasoningAbout Plans. MorganKaufmann Publishers. 69-126. Klein, J.; Moon,Y.; and Picard, R. W. 2002. This computer respondsto user frustration: Theory,design,results, and implications. Interacting with Computers- Special Issue on Cognitionand Enwtion(to appear). Koller, D., and Lemer,U. 2001. Samplingin factored dynamic systems. In Doucet, A.; de Freitas, N.; and Gordon, N., eds., SequentialMonteCarloin Practice. NewYork: Springer-Vedag. Lau, T.; Domingos, E; and Weld, D.S. 2001. Programmingby demonstrationusing version space algebra. Machine Learning. In submission. Lawton,M. E 1983. Environmentand other determinants of well-being in older people. Gerontologist23(4):349-57. Lesh, N., and Etzioni, O. 1995. A soundand fast goal recognizer. In Proceedingsof the FourteenthInternational Joint Conferenceon Artificial Intelligence, 1704-1710.San Francisco, CA: MorganKaufmann. 2000. Progress report on alzheimer’sdisease 2000. online at http://www.alzheimers.org. Pentland, A. 1995. Machineunderstanding of humanaction. In Proceedings7th International Forumon Frontier of Telecorn Technology. Reisberg, B.; Ferris, S. H.; Leon,J. J.; and Crook,T. 1982. Theglobal deterioration scale for the assessmentof primary degenerative dementia. AmericanJournal of Psychiatry. Rice, D.; Fox, E J.; andMax,W.1993. The economicburden of alzheimer’sdisease care. HealthAffairs 12(2): 164-176. Sanders,J. M.2000. Sensingthe subtleties of everydaylife. Research Horizons. Schmidt, C. E; Sridharan, N. S.; and Goodson,J.L. 1978. Theplan recognition problem:an intersection of psychology andartificial intelligence. Artificial bztelligence11:45-83. Shenk,D. 2001. TheForgetting: AIzherimer’s:Portrait of an Epidemic. NewYork, NY: Doubleday. Thrun, S.; Burgard, W.; and Fox, D. 2000. A real-time algorithmfor mobilerobot mappingwith applications to multirobot and 3Dmapping.In Proc. of the IEEEInternational Conferenceon Robotics & Automation(ICRA). 65