Gracing Aging and Happy Healthy Living at the Aware Home
Ellen Yi-Luen Do
Aware Home Research Institute, College of Architecture and School of Interactive Computing,
GVU Center, Health Systems Institute, and Center for Music Technology
Georgia Institute of Technology, Atlanta, GA 30332, USA
ellendo@gatech.edu
1. Introduction – Toward a Smart Living Environment
It’s a spring day in the year 2050. As you finish your breakfast your table displays a picture of the
medicine and vitamin to take after the meal. When you go to the kitchen to get yourself a glass of
water, you notice the handle of the kettle is red to remind you that the water is hot and ready for
tea. Glancing out the window, you see fresh snow accumulated on the ground over night and feel
thankful that the heat was automatically turned up while you were asleep. Meanwhile the living
room starts playing the music for your exercise routine. You may be sixty, eighty or a hundred
years old now. You are happy and healthy. You are aging gracefully and living alone with your
life style partner, the Aware Home.
The age of ubiquitous/pervasive/ambient computing is upon us. What does this mean when we
have computing without computers, where information processing has diffused into everyday life,
and become invisible? Can our homes help us stay active, alive and vital?
This vision is already becoming a reality. Nowadays it is easy install a network of sensors and
computers throughout one’s home. A computer is no longer a desktop machine, instead, it is
becoming part of the room, part of the building, and constantly present. Can we imagine the
world with things that think, spaces that sense, and places that play? Can we employ computing
creatively to enhance our lives? Can we use technological innovation to unlock and augment
human potential? Design and Human-Computer Interaction are crucial components of
information technologies in daily life and they color our experience of computation and
communication. Computing that is aware of what people are doing and what they want would
significantly impact our life.
The Aware Home Research Initiative (AHRI) at Georgia Tech is devoted to the multidisciplinary
exploration of emerging technologies and services based in the home. Since 1998, faculty and
students involved in the initiative have focused their efforts on solving problems of significant
social and economic impact, particularly in the areas of wellness and health. Core to our research
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
and teaching is an understanding of individual needs; how individuals perceive and interact with
different devices; and how differently individuals accept these technologies and devices in their
homes and everyday lives.
2. Be Well – Happy Healthy Living at Home
The idea of wellness is related to human potential. What is wellness? Wellness is the presence of
wellbeing. Wellness is about being active, alive and vital. It concerns individuals, communities
and our environments. Wellness has multiple dimensions. A popular notion of the six dimensions
of wellness is consisted of physical, emotional, occupational, social, intellectual and spiritual
(Hettler 1976). The physical dimension concerns diet, nutrition and physical activities. The
emotional dimension recognizes awareness and acceptance of feelings and behaviors. The
occupational dimension realizes achievement and enrichment through work. The social
dimension encourages contribution to the environment, community and the world. The
intellectual dimension seeks creative and stimulating activities. The spiritual dimension
recognizes the search for values, meaning and purpose in life.
Achieving wellness is a Grand Challenge. We are concerned about the quality of life for
ourselves and for our society. As human beings we want to develop and cultivate our untapped
potential for a happy, healthy, creative and fulfilling life. Technological innovation may be just
the key to unlock human potential for the Holy Grail of wellness.
The Georgia Institute of Technology aspires to the commitment to improving the human
condition through advanced science and technology. As a top technological university with
genuine concerns about human conditions, Georgia Tech has created a culture of possibilities
fostering the growth of interdisciplinary research centers such as the Aware Home Research
Initiative, GVU Center, the Health Systems Institute and the Center for Music Technology. As a
faculty member with a joint appointment in the College of Architecture and the School of
Interactive Computing, and housed in the Health Systems Institute, I have enjoyed being an active
member in these research centers, witnessed and participated in various research initiatives.
The research and innovating projects produced by these centers are too numerous to list them all.
To briefly demonstrate the spirit and the scope of the types of the research efforts, the rest of this
paper will focus on three areas of interest – health, awareness, and entertainment, each illustrated
by a couple projects, to form part of the picture of the theme of “happy, healthy living at home
and graceful aging with ambient intelligence.”
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
2. Health: for the old and the young, with capturing, recording and notification
Besides supporting healthcare services and capabilities ranging from surgery planning to
diagnosing and treating chronic disease, many opportunities exist to help people live a more
carefree and independent life.
The Digital Family Portrait (Mynatt 2001) helps family members at a distance to “keep an eye”
out for their love ones in a casual, lightweight manner. Figure 1 shows a picture of Grandma
displayed together with other family portraits. The picture is surrounded by the images of the
butterflies that change daily, reflecting some portion of Grandma’s activities recorded by the nonobtrusive sensors installed in the house that is an “Aware Home,” a living laboratory for
ubiquitous computing research that is a two-story single-family house built with sensing
infrastructure that is capable of knowing information about itself and the whereabouts and
activities of its inhabitants (Cory 1999).
Figure 1. Digital Family Portrait of Grandmaʼs activities as butterfly icons
Although for privacy reasons, one may appreciate not being constantly under the surveillance
camera, there are times that a recording and monitoring service may come in handy. Take Cook’s
Collage (Tran 2007) for example, a capture system installed on the kitchen countertop that
provides visual summary of recent cooking activity serves as a memory aid. Imagine you are in
the middle of making a cake and you stop to answer the phone. When you get back, you wonder
if you have already put in either three or four cups of flour in your mixing bowl. The display
shows visual snapshots arranged as a series of comic strip panels for you to touch and review past
events. Figure 2 shows a person touching Cook's Collage display to check to see how many cups
of flour he already put into the bowl.
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Figure 2. Cookʼs Collage captures cooking activities for later review
Living well at home is a concern for people of all ages. Children with asthma enjoy playing
outdoors as much as others. Parents are concerned about their children’s wellbeing even when
they are out of sight. With the Pervasive Remote Asthma Monitoring (Chhabra 2007) added to a
cell phone or a cute animal pendant worn on their necks, their coughing and wheezing can be
recorded no matter where they are. This monitoring could alert caregivers remotely and enable
them to quickly supply medical attention in the event of an emergency. Figure 3 shows that, with
embedded electronics and software, the voice can be processed through digital signal processing,
sent through the network and incorporated into patients’ electronic medical records.
Figure 3. Pervasive Asthma Monitoring helps detect patientsʼ coughing and wheezing patterns
If you are disabled or confined to a wheelchair at home, Personal Robotic Assistant (Kemp 2008)
can turn your world into a clickable interface with a laser pointer. If you want your robotic
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
assistant to pick up a toy or a remote control on the floor and deliver it to your friend on the other
side of the room, all you need to do is to point the green laser at the object first, and then at the
person. Your robotic helping hand will then follow your command and deliver the object. Figure
4 illustrates how the robot would perceive objects and a person, instructed through green laser
pointer.
Figure 4. Instructing the Personal Robotic Assistant by clicking on highlighted objects with a laser pointer
The appearance of the in-home personal assistant does not have to take the form of a robot. It
could be an invisible fairy or a guardian angel that watches over your shoulder and guides you
through unfamiliar or difficult tasks. For example, the Technology Coach (Roger 2007) provides
feedback to assist older adults in using medical devices (such as a blood glucose meter) for the
first time, or the hundredth time, or if they made an error when using the device. Figure 5 shows
that the Technology Coach using a computer vision system to recognize user actions and to
recognize potential errors and provide appropriate guidance.
Figure 5. Technology Coach “watches” user actions to provide feedback on home medical device operation
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
3. Awareness: of self & the environment, for physical, emotional, social & spiritual wellness
Information technology can serve as a medium to help us connect with ourselves, with other
people, and with our environment. Applications in a hospital setting may be designed to help
relieve anxiety by the patients and their family members. Applications at a household that inform
people about energy consumption patterns may help residents monitor their energy use and
modify their behavior patterns. Application in a prayer practice may even help people connect
with their spiritual needs. Meanwhile, custom home sensor networks using multiple cameras or
existing residential power line could provide location and event detections to support a variety of
applications in our daily activities. Let’s look at some examples here.
Being sick is no fun. It is uncomfortable and stressful. While confined in a hospital bed, a patient
may feel overwhelmed and alone. Being able to learn about one’s own illness and medical
procedures, or to communicate with clinicians, friends and families may help alleviate the stress.
The Patient Interactive Communication and Learning System (PILS) simplifies the patient's
hospital communication experience by combining communication tools into one easy-to-use
system mounted on the patient bed. As shown in Figure 6, PILS has a one-touch call button that
connects to nurse’s video phone. Family video conferencing, educational video, entertainment
media, and vital signs information are all just “one-touch away.” The entertainment system also
doubles as display for medical records so that the clinicians could use it to explain medical
conditions and treatment options to the patients and concerned family members.
Figure 6. Patient Interactive Communication & Learning System mounted on the bedside connects patients,
care givers & family members at home, monitors vital signs and provides learning opportunities
Or consider energy awareness – we are all concerned about climate change and our impact on the
environment and the world. Would you like to be aware of the energy consumption at your home
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
and change your usage pattern? Would you adopt an electronic pet for this purpose? The Energy
Puppet (Abdelmohsen 2008) is an ambient display device that provides peripheral awareness of
energy consumption for individual home appliances. The display produces different “petlike”
behavioral reactions according to energy use patterns of the appliances to give homeowners an
indication of their energy consumption. The puppet would raise its “arms” in victory to display
normal consumption rate (see Figure 7), or its “eyes” would change color to red and “roar” to
warn the homeowners when the specific appliance reaches high consumption rates.
Figure 7. Energy Puppet raises its arm to celebrate low energy consumption in the home.
Besides energy monitoring, many home automation, entertainment and healthcare applications
with activity sensing require large numbers of sensors or extensive installation procedures.
Wouldn’t it be nice if we could have low-cost and easy-to-deploy location and sensing
technologies at home by simply leveraging the existing infrastructure? Power Line Positioning
(Patel 2006) is an inexpensive technique that uses fingerprinting of multiple tones transmitted
along the power line for location detections. The system is based on a popular wire-finding
technique used by electricians to locate or trace hidden wires behind a wall or underground.
Figure 8 shows that by plugging in two signal generator modules into electrical outlets at extreme
ends of a house the custom tag would detect these signals and determine its location. A sister
project called Power Line Event Detection uses a single plug-in sensor to detect a variety of
electrical events throughout the home using machine learning techniques to recognize electrically
noisy events such as turning on or off a particular light switch, a television set, or an electric stove
and classify various electrical events with accuracies ranging from 85-90% (Patel 2007).
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Figure 8. Power Line Positioning system: two signal-generating modules (top right) at extreme ends of a
house (left), the location tag with a receiver and antenna connected to a handtop computer (bottom right)
Another approach to understand human activity in natural environments over long periods of time
is to use camera and visualization tools to facilitate longitudinal in-situ behavioral analysis. Vis-aViz is a visualization tool (see Figure 9) that interprets data from the overhead cameras, computes
motions and blob tracking, aggregates over space and time and provides interactive navigation
(Romero 2008). By having interpretation and visualization in a spatial or geographical
information system, where the floor plan of the inhabited space serves as the geography and time
is stacked as layers on top of the plane, the tool helps contextualize video analysis in the space
and time to facilitate rapid overview, filtering and zooming, and details on demand of large
volumes of video data, that helps identify potentially sparse target behaviors.
Figure 9. Vis-a-Viz system processes data from overhead camera and turns it into blobs, activity data and
activity map for further analysis.
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Sun Dial (Wyche 2008) is an application that supports Muslims’ prayer practices by showing the
natural environmental cues of the movement of the sun through the sky pictorially in the screen of
a mobile phone. The ritual involves several cycles and prayer takes place during a “window of
opportunity,” instead of an exact time. Therefore, an alarm clock function is not appropriate. With
a simulated sun movement through the sky on a cell phone, even people in an interior space with
no view to the sky can perceive the different phases of the day for prayer opportunities. This
project provides an example of how prayer as an activity can be supported with technology.
Figure 10 shows the sketch used in the study with the Muslim community and the resulting screen
shot for Sun Dial showing a mosque in profile and the sun locations in the blue sky background.
Figure 10. Sun Dial provides environmental cues on a cell phone for prayer opportunities.
4. Entertainment – video, music and technology
Watching TV or listening to stereo is a favorite pass time for many people. Finding the remote
control can be a chore. You can’t quite remember where you put it. You want to change the
channel or turn the volume up but you just can’t find the remote. Some of the remotes may be too
complicated and difficult to use. Or you may be listening to music when the phone rang and you
just wanted to wave at it to turn it off. The Gesture Pendant project (Starner 2000) allows
controlling of ordinary household devices, literally, with the wave of a hand (see Figure 11). The
magic is the wireless camera embedded in the small pendant you wear. By making gestures in
front of the pendant you could control anything from a home theater system, to lighting, to the
kitchen sink. The system can also analyze user movements and detect loss of motor skill or
tremors in the hand that might indicate an illness or problems with medication. Regular eating or
moving patterns can also be detected for health condition monitoring or to help elderly and
disabled people achieve more independence in their homes.
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Figure 11. Waving a finger in front of the Gesture Pendant could control a device at home.
Interested in editing and annotating video for personal diary or to send a commentary along with
a video program to your grand children? With Videotater you would be able to play, cut segments
of digital video and tag them by using a pen on a Tablet PC (Diakopoulos 2006). The tool
provides efficient and intuitive navigation, selection, segmentation, and tagging of digital video.
It automatically segments the video into meaningful categories (i.e., by characters or events) and
signals to the user where appropriate segment boundaries should be placed. It also allows rapid
review and refinement of manually or automatically generated segments, all with simple pen
gestures.
Figure 12. Videotater supports video segmentation and tagging with a pen-based interface
Music is an important thing in life. We hear it everywhere: in elevators, vehicles, concert halls
and shopping malls. Music is an art form that helps us express ourselves, and our feelings. Music
is fun. An upbeat happy music can cheer you up. A soft melody can soothe you. Appreciation of
music is more than just entertainment. It also has social, cultural, and emotional implications.
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Music helps promote wellness, manage stress, alleviate pain, express feelings and improve
communication (Campbell 2000, AMTA 2008). Hand drum has been used to help Alzheimer's
patients improve their short-term memory and increase social interaction, help autistic children
increase their attention spans, and to aid Parkinson's patients and stroke victims to regain their
movement control of or increase their gaits. (Friedman 2000, Chiang 2008)
Imagine finding a musician that who can jam with a group of players and produce inspiring
rhythms in various varying speed and intensity that go beyond human player’s ability? Haile is an
interactive robotic percussionist that can listen to live players, analyze their music in real-time,
and use the results of the analysis to play back in an improvisational manner. Besides being able
to imitate human musicians, Haile can sense, analyze, and react to perform with acoustic diversity
and dynamic range, and with varying velocities between the two arms that are difficult for human
players. The experience of playing with an interactive robotic percussionist may facilitate a
musical experience that is inspiring and encourages novel expressions and interactions. Figure 13
shows a drumming session with the interactive robotic percussionist.
Figure 13. Drumming with an interactive robotic percussionist
Too busy and can’t find time to practice piano? How about wearing a glove that can help you
learn to play music while you’re at your desk or on the move. The Piano Touch (Huang 2007)
project provides a new way for people to learn to play the piano. Wirelessly synchronized with an
iPod, cell phone or other music playing device, the Piano Touch is a light-weight glove outfitted
with little vibration motors to cue the musicians about which finger they need use to play the next
note. Figure 14 shows a converted golf glove with vibration electronics playing on a lighted
keyboard. A pilot study shows that students learned the songs that they were practicing with the
Piano Touch glove much better than the songs that they were practicing without the glove. We
are currently working on making Piano Touch an engaging, pervasive hand rehabilitation aid for
patients who suffer loss of functionality of their hands.
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Figure 14. Piano Touch is a lightweight glove that cues which fingers to play.
5. May the Force Be with You!
In this paper we have covered topics of human potential and the dimensions of wellness. We have
seen interesting technological innovation projects addressing different aspects of wellbeing in our
everyday lives - about health, awareness and entertainment. These projects would not have been
possible without the culture of interdisciplinary collaboration, the synthesis of research across
many domains, and the aspiration to support fundamental human needs. Let’s move forward
together towards a bright and exciting future!
All good things must come to an end. Let me quote from an old Irish Blessing here: “May the
road rise up to meet you, may the wind be always at your back. May the sun shine warm upon
your face and the rain fall softly on your fields.” And until we meet again, my friend, be well.
6. References
Abdelmohsen, Sherif Morad and Ellen Yi-Luen Do (2008) “Energy Puppet: An Ambient
Awareness Interface for Home Energy Consumption,” in SID 08, 7th International Workshop on
Social Intelligence Design, Designing socially aware interactions, Dec 3-5, Universidad de Puerto
Rico, San Juan, PR http://cdr.uprrp.edu/SID2008/default.htm
AMTA (2008) American Music Therapy Association, http://www.musictherapy.org/
Campbell, Don (2000) The Mozart Effect for Children: Awakening Your Child's Mind, Health
and Creativity With Music, William Morrow Publishing
Chhabra, J., Asmi, A., Ragavan, L, (2007). Pervasive Asthma Monitoring, in Pediatric Center of
the Future class, taught by Do, E. YL., Zimring, C., Cowan, D., Lamb, G. Lindgren, J., Jones, J.,
http://www.hsi.gatech.edu/pedcenter
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Chiang, M. M. (2008) Research on Music and Healing in Ethnomusicology and Music Therapy,
Master Thesis, University of Maryland, College Park
Diakopoulos, N. and I. Essa. Videotater, (2006) An Approach for Pen-Based Digital Video
Segmentation and Tagging. Symposium on User Interface Software and Technology (UIST) pp.
221-224
Fischer, T., Davis, T, Vargas, C. (2006) Patient Interactive Communication and Learning System
http://cool.coa.gatech.edu/patientroom, the Patient Room of the Future class at Georgia Tech,
taught by Do, E. YL, Zimring, C. Cowan, D., Lamb, G., Kahn, S., Mullick, A., Winegardend, C.
Friedman, R. L. (2000) "The Healing Power of the Drum", White Cliffs Media, Inc.
Hettler, Bill, (1976) Six Dimensions of Wellness, National Wellness Institute (www.nwi.org),
and http://www.hettler.com/sixdimen.htm
Huang, Kevin, Ellen Yi-Luen Do, Thad Starner, (2008) “PianoTouch: A Wearable Haptic Piano
Instruction System For Passive Learning of Piano Skills,” in ISWC 2008, 12th IEEE International
Symposium on Wearable Computers, http://www.iswc.net/, pp 41-44, Sep 28 - Oct 1, Pittsburgh,
Pennsylavania
Kemp, Charles C., Cressel Anderson, Hai Nguyen, Alex Trevor, and Zhe Xu, (2008) A Pointand-Click Interface for the Real World: Laser Designation of Objects for Mobile Manipulation,
3rd ACM/IEEE International Conference on Human-Robot Interaction (HRI), 241-248
Kidd, Cory D., Robert J. Orr, Gregory D. Abowd, Christopher G. Atkeson, Irfan A. Essa, Blair
MacIntyre, Elizabeth Mynatt, Thad E. Starner and Wendy Newstetter. (1999) "The Aware Home:
A Living Laboratory for Ubiquitous Computing Research," in the Proceedings of the Second
International Workshop on Cooperative Buildings - CoBuild'99. Position paper, October 1999.
http://awarehome.imtc.gatech.edu/publications
Mynatt, E.D., Rowan, J., Craighill, S. and Jacobs, A. (2001). "Digital family portraits: Providing
peace of mind for extended family members." Proceedings of the ACM Conference on Human
Factors in Computing Systems (CHI 2001) Seattle, Washington: ACM Press, pp. 333-340
Patel, S.N., K.N. Truong, and G.D. Abowd. (2006) PowerLine Positioning: A Practical SubRoom-Level Indoor Location System for Domestic Use. Proceedings of Ubicomp 2006. Pp. 441458
Patel, S.N., Robertson, T., Kientz, J.A., Reynolds, M.S., Abowd, G.D. (2007) At the Flick of a
Switch: Detecting and Classifying Unique Electrical Events on the Residential Power Line.
Proceedings of Ubicomp 2007, pp. 271-288
Rogers, W. A., Essa, I., & Fisk, A. D. (2007). “Designing a technology coach”. Ergonomics in
Design. A Publication of the Human Factors and Ergonomics Society, pp 17-23
Romero, Mario, Jay Summet, John Stasko, Gregory Abowd, (2008) “Viz-A-Vis: Toward
Visualizing Video through Computer Vision,” IEEE Transactions on Visualization and Computer
Graphics, vol. 14, no. 6, pp. 1261-1268, Nov./Dec. 2008, doi:10.1109/TVCG.2008.185
Starner, Thad, Jake Auxier, Daniel Ashbrook, Maribeth Gandy (2000) The Gesture Pendant: A
Self-illuminating, Wearable, Infrared Computer Vision System for Home Automation Control
and Medical Monitoring. ISWC 2000: 87-94
Tran, Q., Calcaterra, G., Mynatt, E. (2007) "Using Memory Aid to Build Memory Independence."
Proceedings of HCII: Human Computer Interaction International, 959-965
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86
Weinberg, Gil and Scott Driscoll (2007) The interactive robotic percussionist: new developments
in form, mechanics, perception and interaction design, Proceedings of the ACM/IEEE
international conference on Human-robot interaction, Arlington, Virginia, USA, Pages: 97 - 104
Wyche, Susan, Kelly E. Caine, Benjamin Davison, Micheal Arteaga, Rebecca E. Grinter, (2008)
Sun Dial: Exploring Techno-Spiritual Design through a Mobile Islamic Call to Prayer
Application, CHI extended abstract: 3411-3416
4th. International Symposium on Construction Technology Information- Aging Society and Ambient Intelligence”, July 21, Tokyo, Japan
hosted by I2CTI – International Institute of Construction Technology Information, pp. 69-86