Media Processing in
the Network
Wei Tsang Ooi
Research Area
How
to build multimedia network
applications ?
Outline
Dalí
Multimedia Library
Problems and related work
Research proposal
Question
How
to build multimedia network
applications ?
Dalí
Current Solutions
black
box library code
easy to write but too high level
 need finer optimization, resource
control

write

from scratch
too time consuming
What is Dalí
A
small set of simple, fast, reusable
primitives and abstractions for
image, audio and video processing
Example: Black Box Code
gd
GIF programming library 1.3
gdImagePtr im;
FILE *in;
in = fopen("mygif.gif", "rb");
im = gdImageCreateFromGif(in);
fclose(in);
gdImageRectangle(im,0,0,8,8,red);
Example: Dalí code
f = fopen(argv[1], "r");
BitStreamFileRead(bs,f,0);
BitParserWrap(bp, bs);
GifSeqHdrParse(bp, sh);
rmap = ImageMapNew();
gmap = ImageMapNew();
bmap = ImageMapNew();
n = GifSeqHdrGetCtSize(sh);
GifCtParse(bp,n,rmap,gmap,
bmap);
ih = GifImgHdrNew();
ph = PnmHdrNew();
status = GifImgFind(bp);
GifImgHdrParse(bp, ih);
h = GifImgHdrGetHeight(ih);
w = GifImgHdrGetWidth(ih);
r = ByteNew(w, h);
g = ByteNew(w, h);
b = ByteNew(w, h);
i = ByteNew(w, h);
GifImgParse(bp,sh,ih,i);
ImageMapApply(rmap,i,r);
ImageMapApply(gmap,i,g);
ImageMapApply(bmap,i,b);
Dalí : Abstraction
 Byte
Image
 Audio Buffer
 Bitstream
 MPEG Headers
 JPEG Headers
 Lookup Table
 Bit
Image
 DCT Image
 Vector Image
 Convolution
Kernel
Dalí : Abstraction
 Byte
Image
 Audio Buffer
 Bitstream
 MPEG Headers
 JPEG Headers
 Lookup Table
 Bit
Image
 DCT Image
 Vector Image
 Convolution
Kernel
Byte Image
Two
dimensional array of bytes
Can be either physical or virtual
physical
image
virtual
image
Composable Abstraction
Byte
images can be used to
represent:
Gray scale images
 RGB images
 YUV images
 Alpha channels

Example Code : Fractal
image
newh
smaller
neww
smaller = byte_new (neww, newh);
byte_shrink_2x2(image,smaller);
target
image
dx
2dy
newh
smaller
neww
target = byte_clip(image,0,0,dx,2*dy);
byte_set(target,0);
target = byte_clip(image,3*dx,0,dx,2*dy);
byte_set(target,0);
target
image
dx
2dy
newh
smaller
neww
target = byte_clip(image,dx,0,neww,newh);
byte_copy(smaller,target);
image
newh
smaller
neww
target = byte_clip(image,2*dx,2*dy,neww,newh);
byte_copy(smaller,target);
target = byte_clip(image,0,2*dy,neww,newh);
byte_copy(smaller,target);
Dalí strategies
specialized

byte_shrink_2x2
memory

sharing
byte_clip
explicit

primitives
memory allocation
byte_new
Dalí : MPEG headers
seq gop
hdr hdr
pic
hdr
gop
pic
gop
hdr
pic
hdr
...
gop
pic
...
seq
end
Dalí : MPEG headers
seq gop
hdr hdr
pic
hdr
gop
pic
gop
hdr
pic
hdr
...
gop
pic
...
seq
end
Basic Header Primitives
find
parse
skip
dump
encode
Example
Skip
to the n-th frame
for (i = 0; i < n; i++) {
mpeg_pic_hdr_find(bs);
mpeg_pic_hdr_skip(bs);
}
Dalí: Bitstream
chunk of memory
1011001
parser
Mode of Operations
File
i/o
static memory
parser
fread/fwrite
Mode of Operations
Network
static memory
parser
socket
Mode of Operations
Memory
map
entire file
parser
Strategies
expose

MPEG headers
explicit

structure
I/O
Bitstream abstraction
Novel Features of Dalí
explicit
I/O
explicit memory management
predictable performance
programs are highly reconfigurable
push semantics to programmer
composable primitives, abstractions
Other Features of Dalí
fast
easily
extensible
support wide ranges of formats
(GIF, JPEG, MPEG-1, WAV, AVI, ..)
Dalí : Status
Currently
in alpha release
C, Tcl and Java binding
Better performance than other
tools (e.g. Rivl, mpeg_play)
Back to the question ..
How
to build a multimedia network
applications ?
Breaking it up ..
on one
machine
capture
process
compress
source
on two
machines
capture
process
store
receiver
compress
decompress
display
Optimization
on one
machine
8 bit colors,
10 frame/s
capture
process
compress
source
on two
machines
capture
process
store
receiver
compress
decompress
display
Optimization
on one
machine
8 bit colors,
10 frame/s
capture
process
compress
store
8 bit colors,
10 frame/s
on two
machines
capture
process
compress
decompress
display
Optimization
What
if there are more than one
receiver, and they have conflicting
requirements ?
Problem and
Current Solutions
Problem
Network
heterogeneity
100Mbps
S
36kbps
128kbps
R3
R1
R2
Suggested Solution
Layered
Multicast
streams are hierarchically encoded
 each layer is multicast into a group
 incrementally combine the layers to
refine the quality of the streams

Example
Layer 1
Layer 2
Layer 3
Layered Multicast
100Mbps
S
36kbps
128kbps
layer 1,2 R3
R1
layer 1,2,3,4,5
R2
layer 1
Layered Multicast
Receivers
find out if they can join
the next higher layer by
experiments
Join the layer, if packet drops/low
throughput detected, leave it
Problems
Needs
layered encoding
More groups means more states to
maintained
Uses a lot of multicast addresses
Alternative Approach
Move
computation into the network
where to run ?
 when to run ?
 how to specify the computation ?

Related Work
QOS
Filters (Yeadon 96)
where to run ?
 when to run ?
 how to specify the computation ?

Filters
transform
one or more input
streams into an output stream
Examples :
drop frames
 decode frames
 mix input audio
 transcode to lower quality

Architecture
R1
S
filters
filters
flow spec
session mgr
R2
Architecture
ack
R1
S
filters
filters
session mgr
R2
Architecture
data
R1
S
filters
filters
session mgr
R2
Filter Propagation
Filter Propagation
Answers
QOS

Filters
where to run ?
filter

servers
when to run ?
when

resource(out) < resource(in)
how to specify the computation ?
pick
a pre-determine filter
Related Work
Active
Service (Amir et.al 98)
where to run ?
 when to run ?
 how to specify the computation ?

Active Services
MeGa
Clients
Vic
need MeGa
Servlets
Active
Servers
Servlets : MeGa
An
application level video gateway
Transcodes MJPEG to H.261
Written using Mash
next generation multimedia network
toolkit
 built on tcl

AS Location
S1
bottleneck
R1
S2
R2
S3
R3
MeGa
Active Service
Where

active server at bottleneck
When

to run ?
when receiver requested it
How

to run ?
to specify a computation ?
request for pre-determine servent
Problems
How
to specify computation ?
Not easily extensible
 Significant extension requires manual
installation of servlets/filter

Where

to run ?
AS : statically located
Proposal
The answers should be
How

to specify a computation ?
receivers can inject customized
programs into network
Where
to run ?
can be specified by users or
 dynamically place itself

Thesis Proposal
active

media network (Medan)
allow customizable media processing
code to be injected into the network
application:

self-adaptive transcoder (SAT)
Medan
Medan node
source
receiver
prog
Medan
Medan node
source
receiver
prog
Medan Components
media
processing language
active network kernel
user interface
Language Requirements
Safe
& portable
Support common format/ops
Fine grain operations

distributed processing possible
Explicit

resource usage
resource management and admission
control
Dalí : How does it fit ?
Features

of Java Dalí:
safe and portable
Features
of Dalí :
thin primitives
 predictable
 explicit memory allocation
 explicit I/O

Claim:
Dalí
is a suitable programming
language for Medan
Medan Components
media

Dalí
active

network kernel
???
user

processing language
???
interface
Medan Components
media

Dalí
active

network kernel
JKernel, SANE, etc.
user

processing language
???
interface
Medan Components
media

Dalí
active

network kernel
JKernel, SANE, etc.
user

processing language
interface
Mash
Medan: Todo
Engineering

issues
make Mash, Dalí, and active network
environment talk to each other
Mash
encoder,
Mash decoder,
objects etc.
source
sink
video capture,
network
video display,
network
Mash + Dalí
Dalí
Dalí
Mash + Dalí + AN
Dalí
Safe environment
Medan:
Problems to be solved
Resource
usage determination
Load balancing
Deployment
Security*
Resource management*
Medan Application:
Self-adaptive
Transcoder (SAT)
Injected into the network when a
session starts
Perform bandwidth adaptation by
transcoding session’s packets
SAT Scenario
SAT Scenario
congested
SAT Scenario
congested
SAT Features
transparent
to receivers
one group per session
lower latency
Basic algorithm
each
host who joined a group will
receive at an initial rate
each node will report it’s state
(congested, under-utilized) to it’s
parent
parent will adapt sending rate
accordingly
Problems to be solved
Detail
algorithm
Scalability
Reflect receiver’s interest
Interaction between programs
Deployment
More Problems
Flexible
transcoding algorithm
Congestion detection
Other Medan Applications
Remote
Sensing
Multimedia Database
Video Conference Auto-Editing
Receiver Heterogeneity
Contributions
Dalí
: a media processsing library
Medan : a framework for active
media processing in the network
SAT : apply Medan to solve the
network heterogeneity problems
Summaries
How
to build multimedia apps
Adding the word “network”
Current solutions and problems
Proposal
Where Medan is going..
Computation
can organized itself
(divided, pipelined, factored) into
sub-computations
Packets can carry computation
Capsules
 Mobile Code

The End