Building an Aware Home: Understanding the symbiosis between computing and everyday activities

advertisement
Building an Aware Home:
Understanding the symbiosis between
computing and everyday activities
Irfan Essa, Gregory Abowd
Future Computing Environments
College of Computing,
Georgia Institute of Technology
www.cc.gatech.edu/fce
Who are we?
Faculty (Future Computing Env.):
– Gregory Abowd, Chris Atkeson, Aaron
Bobick, Irfan Essa, Blair MacIntyre,
Beth Mynatt, Thad Starner
~20 PhD students
Affiliations: CoC, GVU, BTC, ECE
Collaborations
– ECE (Wireless, DSP), Architecture,
Rehab Technologies, Psychology, etc.
© Irfan Essa and Georgia Institute of Technology, 1999
Outline
Motivation “Living Laboratory”
Focus Areas / Research Questions
Awareness
Context-Aware Applications
“Aging in place”
Conclusion / Discussion
© Irfan Essa and Georgia Institute of Technology, 1999
Research Objectives
Build a living laboratory in an
everyday setting that is aware of its
occupants’ activities
.. supports the continuous
interactions and activities of a small
community.
.. understand the usage of such a
laboratory as it applies to symbiosis
of computing and everyday living.
© Irfan Essa and Georgia Institute of Technology, 1999
Living Laboratory for
Human-Home Symbiosis
How can we obtain ubiquitous and
continuous connection?
– testbed for technologies
tech-centric
How does this change life / living?
–
–
–
–
information at fingertips
support human communication
build community
user-centric
unite over distance
© Irfan Essa and Georgia Institute of Technology, 1999
An aware home
A home that is aware of its
inhabitants and their activities
… can provide support for day-to-day
activities
… do so without increasing the load on
the inhabitants
… can augment daily functions
… provide connectivity
In and around the home
© Irfan Essa and Georgia Institute of Technology, 1999
Specific Applications
awareness and connectedness w/
others
augmentation (cognitive, memory)
education
monitoring
security / surveillance
Care Facility (Elder, Child, Health, …)
© Irfan Essa and Georgia Institute of Technology, 1999
Where are we?
Georgia Research Alliance (GRA)
– ~$600,000
Broadband Telecommunications
Center (BTC)
Ground breaking May 1999.
Occupancy by Jan. 2000
© Irfan Essa and Georgia Institute of Technology, 1999
Outside
South
East
© Irfan Essa and Georgia Institute of Technology, 1999
Basement
© Irfan Essa and Georgia Institute of Technology, 1999
Living floors (2 floors)
© Irfan Essa and Georgia Institute of Technology, 1999
Other “smart homes”
Home automation
– X10,
– hobby
Many others
www.cc.gatech.edu/fce/seminar/fa98
-info/smart_homes.html (Brad
Stenger)
MSFT Research (Barry Brummit)
© Irfan Essa and Georgia Institute of Technology, 1999
Outline
Motivation “Living Laboratory”
Focus Areas / Research Questions
Awareness
Context-Aware Applications
“Aging in Place”
Conclusions / Discussion
© Irfan Essa and Georgia Institute of Technology, 1999
Research themes
Aware
Home
Technology-centered
Human-centered
Human-Home Symbiosis
© Irfan Essa and Georgia Institute of Technology, 1999
Technological Challenges - I
Perception Technologies
– make the environment aware of the
users and their activities
– what is happening?
– ubiquitous sensing
– interpret (rich) multi-modal streams
– long-term vs. short term
© Irfan Essa and Georgia Institute of Technology, 1999
Technological Challenges - II
Ubiquitous Interfaces / Displays
–
–
–
–
–
–
–
“Off-the-desktop”
Context-aware applications
capture / integrate / access
anytime, anywhere, ease of use, …
diverse resources/media
software infrastructure
multimedia-based
collaboration/interaction
© Irfan Essa and Georgia Institute of Technology, 1999
Technological Challenges - III
Systems & Networking
–
–
–
–
–
–
fast
distributed
secure
adaptive
storage
easy to deploy / configure
• wireless/wireline
– inside, around, and to the home
© Irfan Essa and Georgia Institute of Technology, 1999
User-centric Challenges - I
 Understand the needs of the domain
– physical house vs. home (familial
connections)
• awareness/connectedness with others
– privacy / security
– decrease cognitive load
– What home activities are
• desirable
• can be improved through technology
– Care facility (elderly, young, health)
© Irfan Essa and Georgia Institute of Technology, 1999
User-centric Challenges - II
Elderly home care / assistive
healthcare
– prolong independence in familiar
surroundings
– understand rhythms, patterns,
deviations
– provide contact
– memory augmentation
© Irfan Essa and Georgia Institute of Technology, 1999
Outline
Motivation “Living Laboratory”
Focus Areas / Research Questions
Awareness
Context-Aware Applications
“Aging in Place”
Conclusions / Discussion
© Irfan Essa and Georgia Institute of Technology, 1999
Aware Spaces
Aware environments that know their
inhabitants, their preferences, their
activities
–
–
–
–
Who is there?
Where?
What is happening?
How it can best be supported?
© Irfan Essa and Georgia Institute of Technology, 1999
Computational Perception
Signal Interpretation/Coding
–
–
–
–
–
–
Computer Vision,
Audio/Speech,
Tactile / Contact,
RF/IR emitters,
Sonar,
Usage Sensor, ……
Instrument a Space with Sensors
© Irfan Essa and Georgia Institute of Technology, 1999
Perceptual Analysis
Signal Interpretation to determine
–
–
–
–
geometry, calibration, context
is anyone there?, who?
locate users/people
recognize their actions, activity,
gestures, expressions
– speech, non-verbal, communicative
streams
Dynamic / Long-term / Interactive
© Irfan Essa and Georgia Institute of Technology, 1999
Sensors (Optical / Cameras)
High-end vs. low-end
Task / Resource specific
NEX V25 microprocessor, powerline
modem, ...
Analog / Digital Cameras to
commercial PCs
Specific hardware solutions
© Irfan Essa and Georgia Institute of Technology, 1999
Experiences
Reconstruction of a Scene
Pose Estimation
Multiple Camera-Multiple Person
Tracking
Context-based Activity / Object
Recognition
© Irfan Essa and Georgia Institute of Technology, 1999
3D models of rooms
(Brostow & Essa, ICCV 1999)
Use motion information to model 3D
scenes (models from movement).
© Irfan Essa and Georgia Institute of Technology, 1999
Video
© Irfan Essa and Georgia Institute of Technology, 1999
Multiple Cameras
(Stillman, Essa, et al., AVBPA 1999)
Track multiple people with multiple
cameras
Develop an architecture to support
communication between multiple
processors/cameras
Combine fixed and PTZ cameras to
track and identify people
© Irfan Essa and Georgia Institute of Technology, 1999
Video
© Irfan Essa and Georgia Institute of Technology, 1999
System Architecture
Camera 1
(Fixed)
Color
Tracking
Camera 3
(PTZ)
Video
Motion
Tracking
Locations
Calibrated
PTZ locations
Video
Face
Tracking
Face
Recog.
Server
Camera 4
(PTZ)
Color
Tracking
Motion
Tracking
Color
Tracking
Expression
Gesture
Video
Video
Camera 2
(Fixed)
More Cameras
PTZ locations
More Cameras
© Irfan Essa and Georgia Institute of Technology, 1999
Color
Tracking
Face
Tracking
Pose tracking
(Schödl, Essa, PUI 1998, PDPTA 1999)
Use a 3D model of head
Extract texture
Match texture on model to moving
head (with non-linear optimization)
Repeat for every frame
Develop distributed/parallelized
implementation
© Irfan Essa and Georgia Institute of Technology, 1999
Video
© Irfan Essa and Georgia Institute of Technology, 1999
Distributed Tracking
xt
xt+1
Camera
Generate
Test
Parameters
Console
Node 1
Compute
Image Pyramid
Node n
Compute
Image Pyramid
4X
Estimate
New
Minimum
Render
Head Model
Calculate
Matching Error
& Gradient
Render
Head Model
Calculate
Matching Error
& Gradient
Time t
Console + Parallel Nodes, n=7 in our tests
© Irfan Essa and Georgia Institute of Technology, 1999
Speed-up Curves
Relative
Frame Size
9
Frames/Second
8
7
1x
6
2x
5
4
3x
3
10x
2
40x
1
0
1
2
3
4
Nodes
5
6
7
Frame rate as a function of Cluster nodes and the size of image
(1x = real frame size, 14.4KB).
© Irfan Essa and Georgia Institute of Technology, 1999
Example: Recognizing Activity
(Moore, Essa, Hayes, AVBPA 1999 and ICCV 1999)
Develop an framework (architecture)
for relating actions and objects
Track the relations between actions
and objects for recognition
Use HMMs for temporal recognition
© Irfan Essa and Georgia Institute of Technology, 1999
Video
© Irfan Essa and Georgia Institute of Technology, 1999
Recognition Results
Domain
Actions
Objects
Kitchen Stirring, cutting, scrapping,
open, close, cleaning,
adjusting controls, shaking,
washing, drying, etc.
Office Picking up/down, returning,
flip forward/backward, open,
close, drink, type, point, etc.
RRate
Bowl, cabinet,
72-100%
cutting board, can
opener, appliances,
pots/pans, sink, etc.
Bookcase, book,
80-100%
notepad, keyboard,
mouse, phone,
printer, etc.
Car
Change gears, hand-brake,
Gearbox, parking
80-100%
adjustments, turn left/right, break, radio,
roll up/down, drink, etc.
steering wheel,
window, controls,
etc.
Overall Total actions: 597
~92%
© Irfan Essa and Georgia Institute of Technology, 1999
Other Projects (not there yet!)
Smart Carpet
– Recognizes people based on their
footsteps
Audio-visual tracking
– analysis of audio & visual-kinetic data
– audio-visual tracking
Auto calibration (inside/outside)
Other sensors (wearable etc.)
© Irfan Essa and Georgia Institute of Technology, 1999
Outline
Motivation “Living Laboratory”
Focus Areas / Research Questions
Awareness
Context-Aware Applications
“Aging in Place”
Discussion / Future
© Irfan Essa and Georgia Institute of Technology, 1999
What is context?
Characterizing a situation
Sensed information
Identity, location, activity
of people, places, things
Who? Where? When? What? Why?
© Irfan Essa and Georgia Institute of Technology, 1999
Context-aware applications
Present context information to users
– Example: fridge informs the user of
what is running out
Tailor the interaction according to
context changes
– Examples: Activity in kitchen and near
dinner time provide recipe help based
on available food and preferences
© Irfan Essa and Georgia Institute of Technology, 1999
Easier said than done!
Designing and implementing such
context-aware applications is
difficult!
Goal: Provide software infrastructure
to support rapid development
© Irfan Essa and Georgia Institute of Technology, 1999
The Context Toolkit
(Salber, Dey, & Abowd CHI 1999)
Separation of concerns
– context sensing from reaction
– insulate sensors and applications from
each other
An analogy to GUI development
– separation of presentation and
functionality
We want glue between perception and
interaction
© Irfan Essa and Georgia Institute of Technology, 1999
Beyond the GUI analogy
Context widgets are distributed
– They can be shared by applications
Context widgets are persistent
– They store a history of context
information
Context widgets may be unreliable
– They must provide confidence factors
© Irfan Essa and Georgia Institute of Technology, 1999
Components
Context widgets
– abstraction of a sensor
– taxonomy of context types
Interpreters
– translation between context values
Entity servers
– persistence and aggregation of context
© Irfan Essa and Georgia Institute of Technology, 1999
Experience
Electronic In/Out Board
Informal capturing whiteboard
Mobile Conference Assistance
Home Monitoring system
More empirical experience needed
© Irfan Essa and Georgia Institute of Technology, 1999
Outline
Motivation “Living Laboratory”
Focus Areas / Research Questions
Awareness
Context-aware Applications
“Aging in Place”
Conclusions & Discussion
© Irfan Essa and Georgia Institute of Technology, 1999
“Aging in Place”
Design aware homes that support
elderly
– allow them to be independent, yet
connected
– supported, cared for
– stay home (as opposed to move to an
elder care facility)
– health monitoring
© Irfan Essa and Georgia Institute of Technology, 1999
Connected Family
Is Mom doing well? Eating well?
(peace of mind)
interface that leads to connectivity
see snapshots of “activities”, “day’s
events”
active connection (in the periphery)
continuously updating “portrait” of
Mom displaying how she is doing.
© Irfan Essa and Georgia Institute of Technology, 1999
Cognitive Support
Assist in daily routines to offset
cognitive impairments
Aid memory
– take medication
– locate lost items
– out of site / out of mind (connected)
Avoid institutionalization effects
© Irfan Essa and Georgia Institute of Technology, 1999
Requirements Analysis
What “matters” in free choice envs.
What is productivity? Quantify ??!!
Why do people move to assistive care
facilities?
Why don’t they want to leave their
homes
Ethnographic Interviews ……
© Irfan Essa and Georgia Institute of Technology, 1999
Outline
Motivation “Living Laboratory”
Focus Areas / Research Questions
Awareness
Context & Domains
“Aging in Place”
Conclusions & Discussions
© Irfan Essa and Georgia Institute of Technology, 1999
Test-beds
Future Computing Lab (5/1998)
Computational Perception Lab (1/98)
New Labs for “off-the-desktop”
computing (7/1999)
“Aware Home” (1/2000)
– Kitchen, Living Room, Entertainment
Room, Home-office.
© Irfan Essa and Georgia Institute of Technology, 1999
Future
Pursue both technology-centric and
human-centric approaches,
understand the domain and build it
– better sensing and perceptual analysis
mechanisms
– software, systems, networking
infrastructure
– evaluate the human-home symbiosis
© Irfan Essa and Georgia Institute of Technology, 1999
Summary
Described the Aware Home Project
Making it aware
– sensing
– context-enabled
Challenging Application
– Care Facility.
© Irfan Essa and Georgia Institute of Technology, 1999
The end
© Irfan Essa and Georgia Institute of Technology, 1999
Why a living laboratory?
It is not sufficient to achieve
technological breakthroughs.
Greater contribution lies in the
understanding of impact on everyday
life.
Domain specific.
© Irfan Essa and Georgia Institute of Technology, 1999
Living laboratory experience
Classroom 2000 (education)
– a classroom used daily for 3 years
– captured experiences in a classroom
Smart Spaces
– Rooms, Offices, …
A large-scale test-bed for research
in …
© Irfan Essa and Georgia Institute of Technology, 1999
Download