Advanced Media-oriented
Systems Research at Georgia
Tech: A Retrospective
September 1999 – August 2005
Umakishore Ramachandran, Karsten Schwan,
Phillip Hutto, Matthew Wolf
College of Computing, Georgia Tech
A presentation for the NSF CISE/CNS
Pervasive Computing Infrastructure Experience Workshop
Urbana, Illinois  July 27, 2005
Outline







Research vision
Evolution of goals
Infrastructure: capture, access, interpretation
Theme: experimental or production facility?
Lessons learned: the devil in the details
Outcomes: integrating big and small
Conclusions
Advanced Media-oriented Systems Research:
Ubiquitous Capture, Access, and Interpretation

Faculty involved with RI-related projects




Kishore Ramachandran, Mustaque Ahamad, Karsten
Schwan, Richard Fujimoto, Ken Mackenzie, Sudha
Yalamanchili, Irfan Essa, Jim Rehg, Gregory Abowd,
Yannis Smaragdakis, Santosh Pande, Ramesh Jain,
Calton Pu, Ling Liu, Thad Starner...
Federal funding

NSF RI, NSF ITR, DARPA, DOE

Yamacraw, GT Broadband Institute

HP, Intel, Microsoft, IBM
State funding
Industry funding (equipment and personnel)
Comprehensive,
application-driven
systems research
Overview
Ubiquitous
Access/capture points
Applications
Interpretation
QoS/Resource mgt
DB/Storage
Middleware/
Runtime
Standard/custom kernels/networks
compute
cluster
storage/
network
sensor
array
network infrastructure
Goal
• an adaptive systems infrastructure that
• seamlessly spans hardware continuum
• supports multi-stream coordination
• features rich QoS and high-availability
• includes context-aware security/privacy
• expresses and exploits parallelism
• provides time-dependent processing
• offers powerful stream interpretation
www.cc.gatech.edu/~rama/nsf-ri
Vision
• pervasive systems
• hardware continuum
• “sensors to servers”
• computational utility
• ambient, perceptual
• sensor networks
• stream-oriented
• time-dependent
• distributed, parallel
• real-time
• novel security needs
Original Research Statement
“Using two large-scale research applications—a distributed
education repository, and perceptual computational spaces for
multimedia-based collaboration—as drivers, we propose to
carry out extensive systems research and integration to
support ubiquitous access, capture, and interpretation of a
variety of multimedia systems.”
-- Grant proposal executive summary ‘99
perceptual
computational
spaces
distributed
education
repository
What We Anticipated…
 Demand for rich access of complex data/media streams
 Ubiquitous computational resources (sensors to servers)
 Ubiquitous connectivity
 high-speed Internet, wireless, Bluetooth
 Emerging application class:
 multiple media streams and data sources
 real-time coordination
 fusion, correlation, sampling, feature extraction
 Aim: end-to-end analysis of required infrastructure …
 … driven by experience with real-world applications
Confirmation of Vision




Recent terrorist attacks
Emerging sensor network applications
Rich home media networks
Increasingly sophisticated cell phone apps
Confirmation of Vision
Evolution of Research Goals

Change in driver applications


Change in personnel


e.g. Increasing importance of sensor networks
Changes based on research experiences


Atkeson, Chervenak -> many participants
Changes in research interests


Education repository -> Aware Home, Event Web, TV
Watcher, Surveillance, …
e.g. Increasing importance of middleware, “plumbing”
Changes due to emerging technologies

Grid, sensors, virtualization …
Remaining True to Original Intent

Many infrastructure components


involving many different driver applications


DFuse, MediaBroker, Energy Aware Traffic Shaping and
Transcoding, Stream Scheduling, Agile Store,
Differential Data Protection …
Aware Home, Smart Spaces, High Performance
Computing Program Steering, Event Web, TV Watcher,
Vehicle-to-Vehicle Networks …
and many application domain collaborators

Abowd, Essa, Fujimoto, Jain, Rehg, Starner …
Infrastructure







Compute servers
Clusters
Networking enhancements
Storage
Display facilities
Conferencing facilities
Sensor technologies
Infrastructure: Overview
Compute Servers
Video Wall
Immersadesk
Systems
Studio
Crestron
Video Kiosk
Storage
Aware Home
Sensor Lab
Conference
Room
Clusters
Networking
Enhancements
Infrastructure: Systems Studio










Multi-use experiential smart space
Video wall with touchpad control
Immersadesk with SGI display engine
Sophisticated control center/interconnect
Projectors, whiteboard
Various cameras
Various microphones speakers
Conference table
Various workstations, servers
Sensor lab elements (location tracking, motes, temp
sensors, robot, marquee sign, mobile devices, RFID)
Infrastructure: Experiential
Conference Room



Smaller, more up-to-date version of
Systems Studio
Dual-use for production meetings as well as
research targeted to augmented
conferencing
“Experiential meeting room” - Jain
Theme: Research/Commodity Tension



We believe a natural tension arises
over the proper use of facilities
funded by long-lived, equipment-based
grants seeking to involve diverse
personnel (such as RI grants),
particularly for smart spaces and
pervasive computing infrastructure
Various forces conspire to treat
experimental research facilities as
“commodity” or “production” facilities
The resulting tension has far-reaching
consequences
Theme: Research/Commodity Tension

Research facilities






must be flexible
undergo frequent transformation,
reconfiguration
include new equipment that must be explored
host experimental software
in short, research facilities are often “broken”
Production facilities


must be accessible, friendly, easy to use,
available, reliable
in short, production facilities must “just work”
Theme: Research/Commodity Tension




To encourage wide use by various
researchers, research facilities must
be friendly, easy to use, available; that
is, they must have qualities of
production facilities
In addition, smart research spaces
need “users” to evaluate effectiveness
Finally, smart spaces, when successful
implemented, encourage “production”
use (by their very usefulness!)
But “production use” of research
facilities hinders research usage;
catch-22
Lessons Learned
1.
2.
3.
4.
5.
6.
Plan ahead, get “buy in”, put it in writing
Experimental wireless research is difficult
Use professional services where appropriate
Knowledge transfer is difficult
Avoid the urge to buy “cool stuff”
Plan on upgrading long-lived facilities
Plan Ahead …

Plan ahead, get “buy in”, get it in writing



System Studio originally imagined as dedicated
research facility
Almost immediate pressure for public use (e.g.
general use workstations, regular meetings,
etc.)
Some hazy memories about original agreements
as administration personnel changed over the
years
Wireless Research is Difficult

Research wireless infrastructure increasingly
conflicts with production wireless




Research nets viewed as “rogues” interfering with
production wireless
Constraints on what we could do with
campus wireless
E.g. detection of “local” machines by
wireless clients for “cyber foraging”
application
Solutions?
 Wireless isolation of research facility?
 Negotiate more flexibility with campus wireless?
 Special off-sight wireless lab?
Use Professional Services

Use professional services where
appropriate, even if researchers/students
can do the work



Case in point: audio infrastructure in
conferencing facilities
E.g. Echo cancellation algorithms
Sometimes “doing it yourself” has
benefits; often just a distraction
Knowledge Transfer is Difficult

Knowledge transfer of detailed
equipment usage information over time
is difficult




We often worked on certain equipment and
then moved on
Later, other researchers wanted to use the
same equipment and had to go through a
time consuming “spin up” on how to use
equipment
In the worst case, students that previously
worked with equipment were gone, with much
knowledge lost
Solutions:
 Documentation, build simplifying utilities
 Research scientists effectively maintained
“institutional knowledge” about equipment
Avoid the Urge to Buy “Cool Stuff”

Avoid the urge to buy interesting new
equipment without a clearly defined use;
avoid the “We’ll figure it out later”
syndrome




Carefully weigh cost/benefit of all proposed
equipment acquisitions
Equipment should either be easy to use out of
the box or have a clearly defined, high-priority
role in ongoing research (or be made
serviceable by professionals)
Without, equipment will be under-utilized
We experienced this with some switching
infrastructure in the Systems Studio and with
the Immersadesk to some extent
Plan on Upgrading

Plan on upgrading facilities that will be
in use for more than three years





We staged equipment purchases over the
lifetime of the grant which reduced the need
for upgrade
Still we needed to “refresh” some of the
earliest purchases for continued use
We upgraded (and enhanced) Systems Studio
infrastructure during the 5th grant year
We also upgraded early cluster purchases
SGI display engine reached its end-of-life
during the grant and was not replaced
Synergistic Research Outcomes:
Integrating “Big and Small”

Application-driven research has significantly
enhanced interaction among systems and applicationdomain researchers



Collaborations most vigorous when funded by related grants
Collaboration often facilitated by “sharing” grad students
Synergy of equipment/researchers helped clarify
integration of “big and small” in service of pervasive
applications with computationally intensive demands
(e.g. video analysis)
Research Nuggets

MediaBroker


DFuse


Clearing house for sensors and actuators in a
pervasive computing environment
Data fusion architecture for futuristic sensor
networks
Streamline

Scheduling heuristic for stream-based
applications on the grid
Research Nuggets (contd.)

Dynamic energy aware transcoding of
media streams



Changing application needs
Changing resource availability
Dynamic differential data protection

Streaming in public networks to user devices
Summary






Reviewed six year history (9/99-8/05) of
advanced media-oriented systems research at
Georgia Tech
Discussed original goals, evolution and adaptation
Summarized facilities
Highlighted fundamental tension between
research and production use of such facilities
Offered lessons learned
Characterized synergistic research outcomes
Questions?
Applause!!!

Systems Studio
Systems Lab
DFuse


[ACM SenSys 2003]
An architecture for Infrastructure Adaptation
A sample scenario for an aware environment




Field trip for a class!
Deployed power-constrained sensors
Dynamic wireless network consisting of the students’
PDAs
In-network stream filtering and aggregation
DFuse Fundamentals

Sensors
Fusion Module: Deploys
task graph on sensor
network

Collage

Sink (Display)
Filter
Cameras

Placement Module:
Employs a self-stabilizing
algorithm to place fusion
points in the network

Sample surveillance application task graph:
filter and collage are the fusion functions.


Deployed on iPAQ farm!
Tested 4 cost-functions
Comprehensive API for
fusion and migration
Low-overhead

Energy and application
aware cost functions
Localized decisions
DFuse Evaluation
Application Timeline showing Network Traffic
Streamline



Ubiquitous and dynamic nature of streaming applications
require distributed HPC resources, possibly spanning
administrative domains
Goal: Develop middleware-infrastructure that provides
access to ambient HPC resources for performing
compute-intensive tasks of streaming applications
Grid Computing




Provide access to ambient HPC resources
Focused on scientific and engineering applications
Some recent efforts for Interactive and Streaming
applications (Interactive Grid – HP , GATES – OSU)
Existing infrastructure primarily for batch-oriented
applications
Streaming Grid
Applications
Uniform Access Through Service Framework
Domain Specific Contents that Needs to be Shared
data, software, processes, knowledge
Uniform Access Through Service Framework
Domain Independent Software and Services “connecting
big and small”
registries, scheduling services, QoS monitoring services, security services,
data management services Streaming Grid (under development)
“Big” - Globus Toolkit (registries,
scheduling services, security services,
data management services)
“Specialized” - Streamline, Monitoring
and Load-balancing, Programming
Abstractions
Physical Resources
Network, Storage, Computers, Sensors, Specialized Hardwares
Streamline Scheduling
Streaming
Appication
Data
Sources
Resource
Information
Service
Authentication
Service
Grid
Boundary
Application
Policies
QoS
Monitoring
Service
Scheduling
Service
Application
Information
Service
Streaming Application
GRAM
HPC
Resource
GRAM
GRAM
HPC
Resource
HPC
Resource
Streamline Scheduling Heuristic
S0
S2
S3
Stage Prioritization
S1
{S2 S0 S1 S3}
R0
R3
R1
Resource Filtering
Application
Policies
{S2 {R0 R2 R3}}
Resource Selection
Resource
Policies
{S2 R0}
R2
Streamline Results




Platform
 Scheduling heuristic, Streamline, using Globus Toolkit
 Baseline grid scheduler for streaming apps using Condor
 Approximation algorithm using Simulated Annealing for
comparison as an Optimal (although infeasible in practice)
Results
 Streamline outperforms baseline by an order of magnitude
for both compute and communication bound kernels,
particularly under non-uniform load conditions
 Streamline performs close to Simulated Annealing with very
low scheduling overhead (by a factor of 1000)
"Streaming Grid: A Proposal for Grid Middleware Services
Supporting Streaming Applications“ – Bikash Agarwalla - 2nd place
winner of 2005 IBM North America Grid Scholars Challenge
“Streamline: A Scheduling Heuristic for Streaming Applications on
the Grid” - Bikash Agarwalla, Nova Ahmed, David Hilley,
Umakishore Ramachandran - Under Review
Energy-Aware Transcoding

Wireless multimedia:



faster processors, high-speed wireless links
energy is the constraining resource!
But energy is “different”:



can be finite (battery), non-replenishable, out of energy, app
over
can be associated with costs (server cluster, heat dissipation,
cooling)
tightly coupled with other resources (utility-cost model):
 add CPU, network, disk, memory resources: increased utility
 add energy: increased cost (reduced mission time, increased
expenses, increased heat dissemination, cooling)
Utility
CPU
Network
Cost
Network
Memory
Resources
Memory
CPU
Resources
Application Level Media Transcoding
Camera
raw image
Transcoder 1
(Resizing)
Transcoder 4
(Gray Conversion)
Transcoder 2
(Compression)
Transcoder 3
(Decompression)
Display
Mobile Device 2
Mobile Device 1
wireless link


Transcoders bring data from one form into another: reduce
size, reduce color, compress, ...
Some are mandatory, some are optional
Power
ECPU1 + ENET1
ENET1
ECPU1
CPU
Power
Network
Time
Transcoder
ECPU2
CPU
ECPU2 + ENET2
ENET2
Network
Time
Transcoder Characteristics
'Crop'
600
Compression
6000
Input Data Size 230 kBytes
Input Data Size 360 kBytes
Input Data Size 518 kBytes
5000
Energy Consumption (mJ)
500
400
300
Huffman
LZ77
4000
3000
200
2000
100
1000
0
0
0
10
20
30
40
50
60
70
80
90
0
Size of Cropped Data (in % of the Original Data)



400
600
800
1000
Original Data Size (kBytes)
Transcoder characteristics:


200
Relationship input data size – output data size (rd)
Relationship input data size – transcoder run-time (rr)
Use rd,rr, Kd(n), and Kr(n) to predict potential
energy savings
Compare transcoders for a given frame
Transcoder Evaluation
size(d)
rr
*
runtime
Eloss
*
Kr(n)
compare
size(d)
rr
*
size(d')
Egain
*
ENET1
*
ENET2
Kd(n)
size(d)
-
Energy Aware Transcoding: Discussion



Transcoder characteristics can be obtained offline and online
Power model obtained online, future devices may have capabilities
for determining power model online or will have it part of the
architecture
Experiments:




Overhead O(t*p):



run-time predictions vary about 5-7% from measured numbers
data size predictions vary less than 1%
energy predictions vary about 5-8.5%
number of transcoders t (low, e.g., 1-10)
number of parameters p (typically set to Uij(min) to minimize energy)
Online transcoder evaluation:



add new transcoders
data content changes
update transcoder characteristics with measured results
Dynamic Differential Data Protection: D3P
Device/network diversity
Remote devices
Limited bandwidth
Dynamic Differentiation: Performance
- Expose custom data streams
- Adapt to heterogeneity, changing
environment
Example:
Public Access to Data
Differential Data Access: Protection
- Expose exactly the data a user
wishes to expose to exactly
those who need it
- Least privilege
Per-user customization
Face/scene recognition
Dynamic Differentiation: Functionality
- Create exactly the desired new operations
- Control who can introduce them
and which new operations are admissible
PBIO/D3P
Implementation
publisher (type “640x480 image”)
High-performance binary wire formats
 Reflection, component-wise access
 PBIO objects  capabilities
 crypto, object-specific rights, payloads
derived EChannel
Morpher
IOContext
IOConversion
type A  type B
subscriber
(type “greyscale”)
PBIO types
•
•
•
•
•
Adorner
type A  type A
Inspector
“Reader makes right”
dynamic code generation for marshaling
type information cached locally
IOContext  local type namespace
IOConversion  marshaling information
(multiple conversions possible)
type A  boolean
derived EChannel
subscriber (type “640x480 image”)
IOConversion
derived EChannel
subscriber (type “640x480 image”)
IOConversion
Active Video Streams
Wireless
TCP/IP
webcam
driver
image display &
control GUI
f()
Installation of image filters
(greyscale, edge detection, etc)
D3P: Performance Win
time (s)
D3P vs. Non-D3P
4
3.5
3
2.5
2
1.5
1
0.5
0
D3P 640x480c
D3P 320x240c
D3P 160x120c
Non-D3P 640x480c
10
500
number of frames
5000