Building an Aware Home: Understanding the symbiosis between computing and everyday activities
advertisement
Building an Aware Home: Understanding the symbiosis between computing and everyday activities Irfan Essa, Gregory Abowd Future Computing Environments College of Computing, Georgia Institute of Technology www.cc.gatech.edu/fce Who are we? Faculty (Future Computing Env.): – Gregory Abowd, Chris Atkeson, Aaron Bobick, Irfan Essa, Blair MacIntyre, Beth Mynatt, Thad Starner ~20 PhD students Affiliations: CoC, GVU, BTC, ECE Collaborations – ECE (Wireless, DSP), Architecture, Rehab Technologies, Psychology, etc. © Irfan Essa and Georgia Institute of Technology, 1999 Outline Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in place” Conclusion / Discussion © Irfan Essa and Georgia Institute of Technology, 1999 Research Objectives Build a living laboratory in an everyday setting that is aware of its occupants’ activities .. supports the continuous interactions and activities of a small community. .. understand the usage of such a laboratory as it applies to symbiosis of computing and everyday living. © Irfan Essa and Georgia Institute of Technology, 1999 Living Laboratory for Human-Home Symbiosis How can we obtain ubiquitous and continuous connection? – testbed for technologies tech-centric How does this change life / living? – – – – information at fingertips support human communication build community user-centric unite over distance © Irfan Essa and Georgia Institute of Technology, 1999 An aware home A home that is aware of its inhabitants and their activities … can provide support for day-to-day activities … do so without increasing the load on the inhabitants … can augment daily functions … provide connectivity In and around the home © Irfan Essa and Georgia Institute of Technology, 1999 Specific Applications awareness and connectedness w/ others augmentation (cognitive, memory) education monitoring security / surveillance Care Facility (Elder, Child, Health, …) © Irfan Essa and Georgia Institute of Technology, 1999 Where are we? Georgia Research Alliance (GRA) – ~$600,000 Broadband Telecommunications Center (BTC) Ground breaking May 1999. Occupancy by Jan. 2000 © Irfan Essa and Georgia Institute of Technology, 1999 Outside South East © Irfan Essa and Georgia Institute of Technology, 1999 Basement © Irfan Essa and Georgia Institute of Technology, 1999 Living floors (2 floors) © Irfan Essa and Georgia Institute of Technology, 1999 Other “smart homes” Home automation – X10, – hobby Many others www.cc.gatech.edu/fce/seminar/fa98 -info/smart_homes.html (Brad Stenger) MSFT Research (Barry Brummit) © Irfan Essa and Georgia Institute of Technology, 1999 Outline Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in Place” Conclusions / Discussion © Irfan Essa and Georgia Institute of Technology, 1999 Research themes Aware Home Technology-centered Human-centered Human-Home Symbiosis © Irfan Essa and Georgia Institute of Technology, 1999 Technological Challenges - I Perception Technologies – make the environment aware of the users and their activities – what is happening? – ubiquitous sensing – interpret (rich) multi-modal streams – long-term vs. short term © Irfan Essa and Georgia Institute of Technology, 1999 Technological Challenges - II Ubiquitous Interfaces / Displays – – – – – – – “Off-the-desktop” Context-aware applications capture / integrate / access anytime, anywhere, ease of use, … diverse resources/media software infrastructure multimedia-based collaboration/interaction © Irfan Essa and Georgia Institute of Technology, 1999 Technological Challenges - III Systems & Networking – – – – – – fast distributed secure adaptive storage easy to deploy / configure • wireless/wireline – inside, around, and to the home © Irfan Essa and Georgia Institute of Technology, 1999 User-centric Challenges - I Understand the needs of the domain – physical house vs. home (familial connections) • awareness/connectedness with others – privacy / security – decrease cognitive load – What home activities are • desirable • can be improved through technology – Care facility (elderly, young, health) © Irfan Essa and Georgia Institute of Technology, 1999 User-centric Challenges - II Elderly home care / assistive healthcare – prolong independence in familiar surroundings – understand rhythms, patterns, deviations – provide contact – memory augmentation © Irfan Essa and Georgia Institute of Technology, 1999 Outline Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in Place” Conclusions / Discussion © Irfan Essa and Georgia Institute of Technology, 1999 Aware Spaces Aware environments that know their inhabitants, their preferences, their activities – – – – Who is there? Where? What is happening? How it can best be supported? © Irfan Essa and Georgia Institute of Technology, 1999 Computational Perception Signal Interpretation/Coding – – – – – – Computer Vision, Audio/Speech, Tactile / Contact, RF/IR emitters, Sonar, Usage Sensor, …… Instrument a Space with Sensors © Irfan Essa and Georgia Institute of Technology, 1999 Perceptual Analysis Signal Interpretation to determine – – – – geometry, calibration, context is anyone there?, who? locate users/people recognize their actions, activity, gestures, expressions – speech, non-verbal, communicative streams Dynamic / Long-term / Interactive © Irfan Essa and Georgia Institute of Technology, 1999 Sensors (Optical / Cameras) High-end vs. low-end Task / Resource specific NEX V25 microprocessor, powerline modem, ... Analog / Digital Cameras to commercial PCs Specific hardware solutions © Irfan Essa and Georgia Institute of Technology, 1999 Experiences Reconstruction of a Scene Pose Estimation Multiple Camera-Multiple Person Tracking Context-based Activity / Object Recognition © Irfan Essa and Georgia Institute of Technology, 1999 3D models of rooms (Brostow & Essa, ICCV 1999) Use motion information to model 3D scenes (models from movement). © Irfan Essa and Georgia Institute of Technology, 1999 Video © Irfan Essa and Georgia Institute of Technology, 1999 Multiple Cameras (Stillman, Essa, et al., AVBPA 1999) Track multiple people with multiple cameras Develop an architecture to support communication between multiple processors/cameras Combine fixed and PTZ cameras to track and identify people © Irfan Essa and Georgia Institute of Technology, 1999 Video © Irfan Essa and Georgia Institute of Technology, 1999 System Architecture Camera 1 (Fixed) Color Tracking Camera 3 (PTZ) Video Motion Tracking Locations Calibrated PTZ locations Video Face Tracking Face Recog. Server Camera 4 (PTZ) Color Tracking Motion Tracking Color Tracking Expression Gesture Video Video Camera 2 (Fixed) More Cameras PTZ locations More Cameras © Irfan Essa and Georgia Institute of Technology, 1999 Color Tracking Face Tracking Pose tracking (Schödl, Essa, PUI 1998, PDPTA 1999) Use a 3D model of head Extract texture Match texture on model to moving head (with non-linear optimization) Repeat for every frame Develop distributed/parallelized implementation © Irfan Essa and Georgia Institute of Technology, 1999 Video © Irfan Essa and Georgia Institute of Technology, 1999 Distributed Tracking xt xt+1 Camera Generate Test Parameters Console Node 1 Compute Image Pyramid Node n Compute Image Pyramid 4X Estimate New Minimum Render Head Model Calculate Matching Error & Gradient Render Head Model Calculate Matching Error & Gradient Time t Console + Parallel Nodes, n=7 in our tests © Irfan Essa and Georgia Institute of Technology, 1999 Speed-up Curves Relative Frame Size 9 Frames/Second 8 7 1x 6 2x 5 4 3x 3 10x 2 40x 1 0 1 2 3 4 Nodes 5 6 7 Frame rate as a function of Cluster nodes and the size of image (1x = real frame size, 14.4KB). © Irfan Essa and Georgia Institute of Technology, 1999 Example: Recognizing Activity (Moore, Essa, Hayes, AVBPA 1999 and ICCV 1999) Develop an framework (architecture) for relating actions and objects Track the relations between actions and objects for recognition Use HMMs for temporal recognition © Irfan Essa and Georgia Institute of Technology, 1999 Video © Irfan Essa and Georgia Institute of Technology, 1999 Recognition Results Domain Actions Objects Kitchen Stirring, cutting, scrapping, open, close, cleaning, adjusting controls, shaking, washing, drying, etc. Office Picking up/down, returning, flip forward/backward, open, close, drink, type, point, etc. RRate Bowl, cabinet, 72-100% cutting board, can opener, appliances, pots/pans, sink, etc. Bookcase, book, 80-100% notepad, keyboard, mouse, phone, printer, etc. Car Change gears, hand-brake, Gearbox, parking 80-100% adjustments, turn left/right, break, radio, roll up/down, drink, etc. steering wheel, window, controls, etc. Overall Total actions: 597 ~92% © Irfan Essa and Georgia Institute of Technology, 1999 Other Projects (not there yet!) Smart Carpet – Recognizes people based on their footsteps Audio-visual tracking – analysis of audio & visual-kinetic data – audio-visual tracking Auto calibration (inside/outside) Other sensors (wearable etc.) © Irfan Essa and Georgia Institute of Technology, 1999 Outline Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-Aware Applications “Aging in Place” Discussion / Future © Irfan Essa and Georgia Institute of Technology, 1999 What is context? Characterizing a situation Sensed information Identity, location, activity of people, places, things Who? Where? When? What? Why? © Irfan Essa and Georgia Institute of Technology, 1999 Context-aware applications Present context information to users – Example: fridge informs the user of what is running out Tailor the interaction according to context changes – Examples: Activity in kitchen and near dinner time provide recipe help based on available food and preferences © Irfan Essa and Georgia Institute of Technology, 1999 Easier said than done! Designing and implementing such context-aware applications is difficult! Goal: Provide software infrastructure to support rapid development © Irfan Essa and Georgia Institute of Technology, 1999 The Context Toolkit (Salber, Dey, & Abowd CHI 1999) Separation of concerns – context sensing from reaction – insulate sensors and applications from each other An analogy to GUI development – separation of presentation and functionality We want glue between perception and interaction © Irfan Essa and Georgia Institute of Technology, 1999 Beyond the GUI analogy Context widgets are distributed – They can be shared by applications Context widgets are persistent – They store a history of context information Context widgets may be unreliable – They must provide confidence factors © Irfan Essa and Georgia Institute of Technology, 1999 Components Context widgets – abstraction of a sensor – taxonomy of context types Interpreters – translation between context values Entity servers – persistence and aggregation of context © Irfan Essa and Georgia Institute of Technology, 1999 Experience Electronic In/Out Board Informal capturing whiteboard Mobile Conference Assistance Home Monitoring system More empirical experience needed © Irfan Essa and Georgia Institute of Technology, 1999 Outline Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context-aware Applications “Aging in Place” Conclusions & Discussion © Irfan Essa and Georgia Institute of Technology, 1999 “Aging in Place” Design aware homes that support elderly – allow them to be independent, yet connected – supported, cared for – stay home (as opposed to move to an elder care facility) – health monitoring © Irfan Essa and Georgia Institute of Technology, 1999 Connected Family Is Mom doing well? Eating well? (peace of mind) interface that leads to connectivity see snapshots of “activities”, “day’s events” active connection (in the periphery) continuously updating “portrait” of Mom displaying how she is doing. © Irfan Essa and Georgia Institute of Technology, 1999 Cognitive Support Assist in daily routines to offset cognitive impairments Aid memory – take medication – locate lost items – out of site / out of mind (connected) Avoid institutionalization effects © Irfan Essa and Georgia Institute of Technology, 1999 Requirements Analysis What “matters” in free choice envs. What is productivity? Quantify ??!! Why do people move to assistive care facilities? Why don’t they want to leave their homes Ethnographic Interviews …… © Irfan Essa and Georgia Institute of Technology, 1999 Outline Motivation “Living Laboratory” Focus Areas / Research Questions Awareness Context & Domains “Aging in Place” Conclusions & Discussions © Irfan Essa and Georgia Institute of Technology, 1999 Test-beds Future Computing Lab (5/1998) Computational Perception Lab (1/98) New Labs for “off-the-desktop” computing (7/1999) “Aware Home” (1/2000) – Kitchen, Living Room, Entertainment Room, Home-office. © Irfan Essa and Georgia Institute of Technology, 1999 Future Pursue both technology-centric and human-centric approaches, understand the domain and build it – better sensing and perceptual analysis mechanisms – software, systems, networking infrastructure – evaluate the human-home symbiosis © Irfan Essa and Georgia Institute of Technology, 1999 Summary Described the Aware Home Project Making it aware – sensing – context-enabled Challenging Application – Care Facility. © Irfan Essa and Georgia Institute of Technology, 1999 The end © Irfan Essa and Georgia Institute of Technology, 1999 Why a living laboratory? It is not sufficient to achieve technological breakthroughs. Greater contribution lies in the understanding of impact on everyday life. Domain specific. © Irfan Essa and Georgia Institute of Technology, 1999 Living laboratory experience Classroom 2000 (education) – a classroom used daily for 3 years – captured experiences in a classroom Smart Spaces – Rooms, Offices, … A large-scale test-bed for research in … © Irfan Essa and Georgia Institute of Technology, 1999
