DOA 2001
Supporting Distributed Processing of
Time-based Media Streams
Viktor S. Wold Eide, Frank Eliassen and Olav Lysne
Department of Informatics
University of Oslo
Norway
http://www.ifi.uio.no/˜dmj/
20. September 2001
1 of 20
DOA 2001
Outline
• Overall project goal.
• Environment.
• Introduction of case for illustration purposes.
• Challenges.
• Middleware goals, focus of this work.
• Architecture.
• Design issues - time, synchronization and interaction model.
• Deployment of processing.
• Prototype.
• Conclusion.
2 of 20
DOA 2001
Overall project goal
Address and devise solutions for an extensible framework for on-line
content analysis of time-based media streams, where media streams are
transported over a network.
• The analysis is performed on-line, having real time requirements.
• The purpose of the content analysis is to index and annotate the
media streams.
3 of 20
DOA 2001
Environment
Media source/
Sensor
Network
Processing
Control, Management
Processing
4 of 20
DOA 2001
Case: Traffic Surveillance
Media source/
Sensor
Network
Processing
Control, Management
Processing
5 of 20
DOA 2001
Challenges
• Real time requirements:
— limit the time available for processing each media sample - 25
frames/second video ⇒ 40 ms/frame.
• Massive amounts of data:
— Even a single compressed media stream has significant data rate,
audio 10kbps - 100kbps, video 1Mbps - 10Mbps.
• A number of different concurrent media streams increase resource
requirements even further.
6 of 20
DOA 2001
Challenges
• Feature extraction:
— Computationally demanding extraction of quantitative features
(e.g. color histogram calculations, edge detection, texture
calculations).
Edge detection
Color histogram calculation
— OCR (Optical Character Recognition), extracting registration plate
number.
7 of 20
DOA 2001
Challenges
• Interpretation of extracted features:
— Very hard problem.
— Must limit interpretation to a specific domain to make it feasible
and to improve reliability.
— Interpretation of extracted quantitative features in traffic
surveillance case - deduce that a car is in a video frame based on
extracted features such as color histograms, then extract
registration plate number.
Interpretation/Classification
AND
Car! (in video frame)
— Not focus of this paper/presentation.
8 of 20
DOA 2001
Middleware goals
• Development of content analysis applications is difficult.
— Allow development by third parties, raise the abstraction level and
allow reuse.
• Always some new technology.
— Simplify integration. New sub-technologies should be pluggable
into middleware as they become available (new video analysis
algorithms, etc.)
• Content analysis requires large amounts of computational resources.
Some application domains have an inherent distributed nature.
— Support distributed processing to improve scalability and
reliability.
• Best effort environments give no guarantees with respect to
resources.
— Support adaptation, reconfiguration, migration and replication.
9 of 20
DOA 2001
Architectural overview
• Component based approach.
C
C
C
E
E
E
E
F
F
F
F
S
S
C : Classification
E : feature Extraction
F : Filtering
S : media streaming Source
: Event
: Filtered media stream
: Media stream
10 of 20
DOA 2001
Architectural overview example
Average speed too high?
C
Regnr.
C
Car?
C
OCR E
E
E
E
F
F
F
F
S
S
C : Classification
E : feature Extraction
F : Filtering
S : media streaming Source
: Event
: Filtered media stream
: Media stream
11 of 20
DOA 2001
Time and synchronization
Communication between feature extraction and classification (F and E)
components designed as exchange of Event Descriptors.
• Associate time interval to a Event Descriptor.
— event: (eStart, eStop, eType, eInstance)
• Association of time intervals to Event Descriptors allows specification
of temporal relations between events:
— e1 before e2: e1.Stop < e2.Start
— e1 while e2: e1.Start > e2.Start and e1.Stop < e2.Stop
e1 before e2
e2
e1
Time
e1 while e2
e2
e1
Time
12 of 20
DOA 2001
Time and synchronization
• Synchronization of hosts in a distributed environment:
e2
e1
Host 1
Host 2
Host 3
Global Time
• In our design we assume that all hosts are synchronized to global
time with some level of accuracy.
— By using e.g. the Network Time Protocol, IETF.
13 of 20
DOA 2001
Interaction model
• The interaction model should:
— allow bindings of different causalities, in general many to many
communication is required.
— support both push and pull style interaction.
— provide a level of indirection to simplify adaptation,
reconfiguration, migration and replication.
⇓
• These requirements fit the publish/subscribe interaction paradigm
very well, leading to an event based model.
• An event notification service should:
— be realized as a distributed and scalable service.
— balance requirements for real time communication and low event
propagation delay against ordering and reliability guarantees
associated with event delivery.
14 of 20
DOA 2001
Interaction model
F
F
E
E
C
EC
Event Broker
C
E
F
E
F
EC : End Consumer
C : Classification
E : feature Extraction
F : Filtering
: Filtered media stream
: Media stream
15 of 20
DOA 2001
Deployment of components to hosts
C
C
C
E
E
E
E
F
F
F
F
S
S
C : Classification
E : feature Extraction
F : Filtering
S : media streaming Source
: Event
: Filtered media stream
: Media stream
: Host clock synchronization
: Host boundary
16 of 20
DOA 2001
Case: Traffic Surveillance
C
Regnr.
F
S
E
F
e1 e2
C
OCR E
OCR E
E
F
F
e1
e2
S
Network
C
Regnr.
Media source/
Sensor
17 of 20
DOA 2001
Prototype
• A prototype of the middleware has been implemented:
— Proof of concept implementation and for gaining experience.
— Event notification service based on Mbus, work in progress by IETF.
— A component repository of reusable components has been
developed. Filtering and feature extraction components are based
on Java Media Framework (SUN Microsystems Inc.)
• A test application for segmenting video has been implemented by
using this middleware.
— Experiments performed by using standard PC’s connected by LAN.
— The protocol stack for media streaming was MJPEG/RTP/UDP/IP.
— Deployment of components onto hosts is performed manually.
— The flexibility of the Mbus based event broker was confirmed.
— Distribution improved scalability.
— Some successful migration experiments have been conducted.
18 of 20
DOA 2001
Conclusion
• The middleware is targeted at, but not limited to, time-based media
streams.
• Development of content analysis applications and integration of new
sub-technologies is simplified by our component based approach.
• The real time requirements, the massive amounts of data, and the
computational complexity requires a distributed approach.
Additionally, some target application domains have an inherent
distributed nature.
• Much attention has been paid to mechanisms necessary for resource
management - adaptation, reconfiguration and migration.
19 of 20
DOA 2001
Questions?
20 of 20