A Multimedia Software Library
Wei Tsang Ooi
Brian Smith, Hsi Haye Chan, Matthew Chiu,
Sugata Mukhopadhyay, Dan Rubinovitz,
Jiesang Song, Steve Weiss.

Current Software Solution
Black
Box
C Source

MPEG to RealVideo
Black
Box
OpenFile(f)
GetNextFrame()
RGB
Real
Video
Encoder
MPEG Real

MPEG to RealVideo
YUV to RGB
OpenFile(f)
GetNextFrame()
RGB
YUV
Decoder
MPEG compressed data
RGB to YUV
YUV
Encoder
Real

MPEG to RealVideo
YUV to RGB
YUV
Decoder compressed data
MPEG
RGB to YUV
Encoder
Real

Decode I-Frame only
Black
Box
OpenFile(f)
GetNextFrame()
?
RGB
MPEG

Decode I-Frame only
YUV to RGB
OpenFile(f)
GetNextFrame()
?
RGB
YUV
Decoder
MPEG compressed data

Decode I-Frame only
YUV to RGB
Decoder
I-frames
I Frame filter
RGB compressed data
MPEG
?

Others Motivations..
Black
Box



Read from network ?
Tune I/O ?
Run on handheld ?

Current Software Solution
Black
Box
C Source

Dalí is in between..
Black
Box
Dalí
C Source

Example :

Process a subset of the gray scale version of each I-frame

MPEG structures sequence hdr GOP hdr picture picture pic hdr pic data

What we want to do parse sequence header

What we want to do parse sequence header
while not eof do
- find & parse picture header

What we want to do parse sequence header
while not eof do
- find & parse picture header
- if it is an I-frame then
- parse the picture into DCT domain
- IDCT into spatial domain

What we want to do parse sequence header
while not eof do
- find & parse picture header
- if it is an I-frame then
- parse the picture into DCT domain
- IDCT into spatial domain
- take the gray scale version of the frame and process it

Abstractions




For input/output
For representing elements in MPEG
For storing images in DCT domain
For storing images in spatial domain

BitStream & BitParser
BitParser
BitParser
BitStream
File

MPEG Headers
MpegSeqHdr MpegGopHdr MpegPicHdr Picture Data



Find
Skip
Dump


Parse
Encode

Find
MpegSeqHdr MpegGopHdr MpegPicHdr Picture Data



Find
Skip
Dump


Parse
Encode

Skip
MpegSeqHdr MpegGopHdr MpegPicHdr Picture Data



Find
Skip
Dump


Parse
Encode

Dump
MpegSeqHdr MpegGopHdr MpegPicHdr Picture Data
MpegGopHdr



Find
Skip
Dump


Parse
Encode

Parse
MpegSeqHdr MpegGopHdr MpegPicHdr Picture Data



Find
Skip
Dump width: 320 height: 240 pic rate: 30 fps etc


Parse
Encode

Encode
MpegSeqHdr



Find
Skip
Dump width: 320 height: 240 pic rate: 30 fps etc


Parse
Encode

ScImage
ScBlock :-
{ short dc; char index[63]; int ac[63];
} represents 8x8 pixels

ByteImage unsigned char
0..255
represents a pixel

Example
MpegSeqHdr *seqHdr = MpegSeqHdrNew ();
MpegPicHdr *picHdr = MpegPicHdrNew ();
BitStream *bs = BitStreamNew (65536);
BitStream

Example
BitParser *bp = BitParserNew ();
BitParserAttach (bp, bs);
BitParser
BitStream

Example
BitStreamReadFromFile (bs, f, 0);
MpegSeq
Hdr
BitStream
MpegPic
Hdr
BitParser File

Example
MpegSeqHdrFind (bp);
MpegSeq
Hdr
MpegGop
Hdr
MpegPic
Hdr
BitParser

Example
MpegSeqHdrParse (bp, seqHdr); w = (seqHdr->width + 7)/8; h = (seqHdr->height + 7)/8;
MpegSeq
Hdr
MpegGop
Hdr
MpegPic
Hdr
BitParser

Example
ScImage *y = ScNew (w, h);
ScImage *u = ScNew (w/2, h/2);
ScImage *v = ScNew (w/2, h/2); y u v

Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) {
MpegPicHdrParse (bp, picHdr);
} status = MpegPicHdrFind (bp);

Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) {
MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) {
}
} status = MpegPicHdrFind (bp);

Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) {
MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) {
MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv);
:
}
} status = MpegPicHdrFind (bp);

Example status = MpegPicHdrFind (bp); while (status != NOT_FOUND) {
MpegPicHdrParse (bp, picHdr); if (picHdr->type == I_FRAME) {
MpegPicIParse ( bp,seqHdr,picHdr, scy,scu,scv);
:
}
BitStreamFill(bp, f); status = MpegPicHdrFind (bp);
}

Refill the BitStream
BitParser

Refill the BitStream
BitParser

Refill the BitStream
BitParser File

Memory Mapped BitStream bs = BitStreamMMapReadNew(“tennis.mpg”);
File

Operate on a subset

Physical ByteImage width: 8 height: 6 x: 0 y: 0 parentWidth: 8 isVirtual: 0 firstByte *

Virtual ByteImage width: 5 height: 4 x: 1 y: 1 parentWidth: 8 isVirtual: 1 firstByte *

Example
ScImage *virt = ScClip (y,6,2,20,20);
6 20 blocks

Example
ScImage *virt = ScClip (y,6,2,20,20);
ByteImage *byte = ByteNew (160, 160);
ScToByte (virt, byte);
6 20 blocks

Implementation


C library
Tcl bindings available

Organization




Code are organized into packages
Basic
– ByteImage, ScImage, etc.
MPEG
– MPEG-1 Video/Audio/System
Vision
– Useful vision routines

Other Packages




AVI
WAVE
JPEG
GIF

Availability




 http://www.cs.cornell.edu/dali open source binaries for linux/win32/sunos examples documentations