UC BERKELEY
UC DAVIS
UC MERCED
UC SANTA CRUZ
Summary and New Research Directions
Ruzena Bajcsy, director of CITRIS
http://www.citris-uc.org
1
Outline
 CITRIS Overview
 New Research Directions
2
Initial Applications (1)
 Energy Efficiency
 Disaster Response and
Homeland Defense
 Education
3
Initial Applications (2)
 Monitoring Health Care
 Environmental Monitoring
 Transportation Planning
4
February 2003
February 2001
February 2000
August 2001
February 2002
5
Mobile Sensors
6
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
7
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
 Predicting Ground Failure
8
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
 Predicting Ground Failure (2)
9
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
 Predicting Ground Failure (2)
 Fire and Disaster Response
10
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
 Predicting Ground Failure (2)
 Fire and Disaster Response
 Habitat Monitoring on
Great Duck Island
11
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
 Predicting Ground Failure (2)
 Fire and Disaster Response
 Habitat Monitoring on
Great Duck Island
 Monterey Bay Monitoring
 Groundwater Monitoring
 Taking Sensors to Market
12
Sensor Nets – A few sample projects
 Earthquake Response of Retrofitted
Buildings
 Predicting Ground Failure (2)
 Fire and Disaster Response
 Habitat Monitoring on
Great Duck Island
 Monterey Bay Monitoring
 Groundwater Monitoring
 Taking Sensors to Market
 Teleactors
13
SOCIETAL-SCALE INFORMATION SYSTEMS—SIS
Scalable, Reliable,
Secure Services
Building & Using
Sensor Nets
MEMS for
Sensor Nets
CITRIS
University of California
14
PicoRadio – low power sensor nodes
 Jan Rabaey
 Original technology goals
» Limit power to 100 microwatt for energy scavenging
» Enable energy scavenging from building vibrations
Light Level
Low Indoor Light
Fluorescent Indoor Light
Partly Cloudy Outdoor Light
Bright Indoor Lamp
High Light Conditions
Vibration Level
2.2m/s2
5.7m/s2
Duty Cycle
0.36%
0.53%
5.6%
11%
100%
Duty Cycle
1.6%
2.6%
15
PicoRadio –future goals
 Technology: smaller (mm3) and lower power (< 10 mwatt)
 Applications: “Smart surfaces”: plane wings, smart
construction materials, intelligent walls
 How to get there? Go non-traditional!
» Use non-tuned mostly passive radio’s – center carrier
frequency randomly distributed
» Use statistical distribution for reliable data propagation
200 mwatt Receiver
50 mwatt RF Oscillator
Both in Fab
16
Energy Scavenging
 Paul Wright, Jan Rabaey, David Culler, …
 Goals:
» Technology: build sensors motes that can run from solar
and vibrational (piezoelectric) sources
» Applications: Meet California Energy Commission goal of
“energy aware houses” that can automatically measure
and control energy use, based on real-time pricing
 Feb 10 2004 demo to CEC of temperature sensors
powered by vibrations on wooden steps
 Jan 30 2004 demo to Assemblymember Loni
Hancock
17
Common Sources of Vibrations
Frequency
of Peak
(Hz)
Peak
Acceleration
(m/s2)
Kitchen Blender Casing
121
6.4
Clothes Dryer
121
3.5
Door Frame (just after door closes)
125
3
Small Microwave Oven
121
2.25
HVAC Vents in Office Building
60
0.2-1.5
Wooden Deck with People Walking
385
1.3
Bread Maker
121
1.03
External Windows (size 2ftx3ft) next to a
Busy Street
100
0.7
Notebook Computer while CD is Being Read
75
0.6
Washing Machine
109
0.5
Second Story of Wood Frame Office Building
100
0.2
Refrigerator
240
0.1
Vibration Source
18
Tiny Temp
Storage
Capacitor
Power
Circuit
Mote
•3 people ran on the stairs
for 40 minutes, charging
capacitor
•3.28 V for 816 ms
• 2 temperature readings
transmitted
Piezoelectric
Power
Generator
Thermistor
19
Redwoods go Wireless
 Todd Dawson
 Goal: understand role of trees in the hydrological cycle
» Forests recycle ~66% of all fresh water
» What is relative important of tree biology and climate in
water loss?
 Local Context
» Understand interaction of maritime fog and coastal
redwoods
» Maritime fog major source of water; some communities
in Chile, W. Africa depend on it
20
Redwood Data
Humidity vs. Time
101
104
109
110
111
Rel Humidity (%)
95
85
75
65
55
45
35
Temperature vs. Time
Temperature (C)
33
28
23
18
13
8
7/7/03 7/7/03 7/7/03 7/7/03 8/7/03 8/7/03 8/7/03 8/7/03 8/7/03 8/7/03 9/7/03 9/7/03 9/7/03
9:40
13:41 17:43 21:45 1:47
5:49
9:51
13:53 17:55 21:57 1:59
6:01
10:03
Date
21
Optimizing Building Operations
 Ed Arens
 Goal: use sensor nets. to
» Save energy ( up to 1/3 of US energy wasted in misheated/cooled/lit buildings)
» Increase occupant comfort and productivity
 Obstacle:
» >90% of cost of conventional sensors is wiring for power and
communication
 Approach:
» Make wireless, battery-less sensors cheap and reliable
 CEC goal
» $20/home to put sensors in every California household
• Thermostat, appliances, networked to utility for price info
» Enable “demand-response pricing”
22
Seismic and Wildfire Monitoring
 Steve Glaser
 Goals
» Create a “downhole” seismic array to
monitor ground motion
» Monitoring of wildfire to guide firefighters
23
Wild Fire Monitoring
• Drop sensors with weather
boards, GPS and radio
•Helicopter based real-time
topographic and ground cover
mapping
•Measured data coupled to
real-time burn model
•Visualization of actual and
modeled observations returned
to firefighters via GIS
•Status:
•HW, SW partially complete
•grad students certified for “controlled burns”
24
Golden Gate Bridge Monitoring
 Greg Fenves, David Culler, Jim Demmel
 Monitor structural “health”
» Vibrations, unusual behavior, from wind, earthquake,
or local damage
 Challenge: high accuracy, low jitter, lots of data
 Two Accelerometers on prototype board:
» $10, 200 mG, ADXL and $150, 30 mG, Silicon Designs
25
At the Golden Gate Bridge
26
New Research Directions
 Eric Brewer
» Information Technology for the 3rd World
 Alice Agogino
» Lighting and Medical Personalization
 Pravin Varaiya
» Traffic Monitoring
 AnnaLee Saxenian
» RFID and Privacy
 George Akerlof et al
» Experimental Social Sciences Lab (XLAB)
 Paul Wright, Kathy Yelick et al
» On-line education
27
Information Technology
For the 3rd World
 Eric Brewer et al
 Who lives in the “3rd World”?
» 3 - 4 B people with incomes < $2K / year
» Little education
» Local and disorganized markets
 But they spend on technology
» 85% have TV, 21% telephone (Dharavi, India)
 How can IT help these people profitably?
28
3rd World Success Stories
 Eradication of Dengue Fever in West Africa
» Sensors, models to time spraying of larvicide
» Protects 30M people, freed 100K sq miles of farm land
 Sri Lankan Rice Market
» Wireless devices to tell farmers real price of rice
» Eliminated cheating by middlemen, doubled incomes
 Grameen Bank in Bangladesh
» Microlender (to women) for businesses
» Has loaned $4B, 99% recovery rate
» Funded national cell phone network, profitably
 Can we systematically help create more of these?
» Health, education, Government, commerce – many examples
29
Research Challenges
 Low cost, low power devices
» Must be cheap, not depend on wired power
 Rural network coverage
» 802.xx variations
 Low-power intermittent networking
» Just when daily bus goes through village
 Literacy issues and User Interfaces
» Assume users illiterate
 Shared devices and infrastructure
» Data center for persistent storage
30
Partners





NSF
Intel, HP, HP Labs India
Grameen Bank, UNDP, Markle
IIT Delhi and Kanpur
UC Social Scientists
 First deployment in India 2005
 Looking for second deployment
31
Traffic Monitoring (1)
 Pravin Varaiya
 Goal
» Monitor traffic to better control street lights, onramps lights, identify accidents…
» Replace current technology with sensors that are
• Cheaper to install (don’t dig up road for loop
detectors)
• Cheap to maintain
• Reliable, low delay (1/10 sec at intersections)
32
Sample Traffic Data
Midlane Microphone
MidlaneMagnetometer
Wind disturbance
Current work:
•Comparing network protocols
•Better signal processing, especially microphone data
•Collecting data on Hearst near Euclid, and Richmond FS
33
RFID and Privacy
 AnnaLee Saxenian, Ross Stapleton-Gray
 The “Sorting Door” project
» Equip “doors” with RFID readers
» Network them to Identification Engine (IE)
» IE uses Object Name Service (ONS), other databases, to
identify tags
» Distribute doors in experimental environment (eg campus)
 Research questions
» How effective are RFID readers?
» How aware are people of RFID?
• Ex: “Door” could announce what people are wearing or
reading, record reaction
» What behavioral patterns can be extracted?
» What behavioral deductions can be made using IE?
» Is there a “tipping point”, where RFID dramatically effects
privacy?
34
What is Tele-Immersion?
 Tele-immersion is an emerging technology that enables
cooperative interaction between geographically distributed
sites.
 This is achieved through realistic reconstruction of the
scenes in virtual space in real-time.
35
What is Tele-Immersion?
Tele-immersion connects remote places through a
shared virtual environment.
Internet 2
3D scene
Interactive 3D display
at remote location
UPENN
Internet 2
3D scene
Interactive 3D display
at remote location
Internet 2
3D scene
Interactive 3D display
at remote location
UC Berkeley
UIUC
Bidirectional 3D data transmission
36
Applications
Tele-Immersion can be used in
 surveillance
 coordinated global decision making through integrated
view of the scene
 remote learning and training
 coordination of activities between distant places
(physical therapy, exercise, distributed ballet, … etc)
 unobtrusive monitoring of people
 group behavior study
37
Overview of Tele-Immersion
Image processing
Synchronized
data transmission
Reconstruction
These should be done in real-time with minimal delay both
in computation and data transmission.
38
Tele-immersion Setup
The system consists of camera clusters and display system.
A camera cluster of 3 BW
and 1 color cameras
Teleimmersion Lab
39
How Tele-Immersion works?
Image Processing
Real-time depth computation
 fast stereo algorithm
Multiple cameras to fully acquire the scene
 >12 camera clusters to cover 360 degrees
Challenges in computer vision
 Texture-less regions
 occlusion
 highlights
computation
3D model
3D model acquisition
from multiple images
40
How Teleimmersion works?
Data transmission
Gigabit internet connection between multiple sites
 currently implemented on wired network
Issues in communication
 bandwidth requirements (>500Mbps between two sites)
(320x240x5)(byte/data) x7(stream)x10(fps) x 2 (sites)
 real-time constraint
 minimal latency and delay
 data synchronization
• synchronization of multimedia data (video + sound)
• synchronization between multiple sites
41
How Teleimmersion works?
Reconstruction

Reconstruction of dynamic activities in virtual space
42
Teleimmersion and Social Science
 Understanding group function is critically important to
society and organizations.
ex) group formation, diffusion of information, consensus building
 Technology have changed the way in which groups and
teams operate.
mediated meeting
Face to face
distance

through telephone, teleconferencing,
email and internet messaging
43
Teleimmersion and Social Science
 Mediated group research has been relatively less explored.
• limited to comparison study
• no systematic / integrated research established due to lack of
infrastructure!
• technology advances >> mediated group research
Tele-immersion can bridge this gap by providing
collaborating environment and infrastructure.
44
Teleimmersion and Social Science
 Tele-immersion can serve as
• a new testbed for studying mediated group
interactions
• a tool for analyzing systematically human
behavior
• a system for constructing a large collection of
database of group interactions.
45
Group meeting in Tele-immersion
People meet in a
shared virtual space
Display
Display
Data
Acquisition
UI
Data
Acquisition
Integrate
UCB
Display
Display
UPENN
Display
Data
Acquisition
Display
46
Group interactions in Tele-immersion
Meeting in the virtual space
site 1
site 3
site 2
analysis based on
verbal communication
(text, speech)
non-verbal and interactions with
technology
47
Systematic analysis of human behavior
using 4D Video Stream
 Analysis of dynamics of
human behavior
• quantitative measurements of
human motion and facial
expression
• automatic retrieval of
elementary behaviors
estimation of body parts
from 3d point clouds
48
Database for group study
 Tele-immersion can construct database of a large
collection of group interaction samples.
• 4D dataset of audio-visual data
• online repository
• data resources for other analysis
49
Summary
 Tele-immersion is the state of art technology that combines
computer vision, graphics and network communications.
 Tele-immersion can provide the infrastructure for studying
mediated groups by establishing collaborative environment.
 Systematic analysis of human behavior is possible in the
new environment.
 Tele-immersion can build database of group interactions
that enables further extensive studies.
50