NIK 2000
Distributed Journaling of
Distributed Media
1
Viktor S. Wold Eide, Frank Eliassen
Ole Christoffer Granmo, Olav Lysne
Department of Informatics, University of Oslo
22. November 2000
1 of 18
NIK 2000
Media journaling, motivation and goal
Media journaling: On-line
— capture of multimedia content and related discrete information of
distributed multimedia sessions
— creation of indexed and annotated record of user defined events
occuring in the session
Motivation: New generation of multimedia applications
— built around support for media journaling
— simplify use of continuous media - persistence and manipulation
— add value to established application areas - distributed meetings, games,
tele medicine, surveillance systems, control systems etc.
Goal: Address and devise solutions for an extensible framework for
on-line content analysis
2 of 18
Media journaling
Database
Network
Sensor
Event
Content analysis
ALARM
Retrieval
Control
Content Analysis - overview
0HGLDVWUHDP
'LIIHUHQFH
+LVWRJUDP
/RZOHYHOPHGLDIHDWXUH
H[WUDFWRU
H[WUDFWRU
H[WUDFWLRQDOJRULWKPV
3UHFHGLQJUHFRJQLWLRQV
%
/RZOHYHOPHGLD
IHDWXUHV
C olorbins
Knowledge base
C olorbins
3HUVRQ
+LJKOHYHOFRQFHSW
UHFRJQL]HU
UHFRJQLWLRQ
1RSHUVRQ
%
a
3HUVRQ
%
a
C olorbins
'LIIHUHQFH
+LVWRJUDP
&RQFHSW
3HUVRQ
+LJKOHYHOFRQFHSWV
NIK 2000
Challenges
The uncertain relationship between high-level concepts and
low-level media features
The difficulty of specifying content analysis tasks
The computational complexity of feature extraction algorithms
The massive amounts of data to be analyzed in real time
The intricate relationship between reliability, latency and resource
usage
5 of 18
NIK 2000
Our approach
Use probabilistic interpretation models to deal with uncertainty
Simplify specification by using object-orientation to encapsulate
— the feature extraction algorithms, and
— the interpretation models
within combinable high-level concepts
Cope with computational complexity and massive amounts of data
through a distributed processing environment (DPE)
Integrate probabilistic interpretation models and DPE
— enables trading off reliability against latency and resource usage
The uniqueness of our approach is highlighted in bold font
6 of 18
NIK 2000
Dynamic Bayesian networks (DBNs) I
Within_pre
Not Person1 15.0
Person1 and
...80.0
Person1 and
...5.00
Within
Not Person1 15.7
Person1 and
...78.6
Person1 and
...5.70
Person1
True 84.3
False 15.7
I1
I2
I3
I4
I5
Difference1
15.2
15.8
26.1
25.8
17.2
11.9 ± 9
I1
I2
I3
I4
I5
Person2
True 13.6
False 86.4
Histogram1
1.57
13.6
51.4
5.00
28.4
15.5 ± 9.2
I1
I2
I3
I4
I5
Difference2
61.2
19.3
8.39
6.66
4.44
4.8 ± 7.2
I1
I2
I3
I4
I5
Histogram2
8.64
52.5
12.5
5.00
21.4
13.4 ± 9.1
A recursive graphical probabilistic model
Periodic activation to track the state of variables across time
The probability of each state in the current time slice is inferred from
— the probability of each variable state in the previous time slice
— observations made in the current time slice
7 of 18
NIK 2000
Dynamic Bayesian networks (DBNs) II
Within_pre
Not Person1 0.99
Person1 and
...22.7
Person1 and
...76.3
Within
Not Person1 1.03
Person1 and
...10.2
Person1 and
...88.8
Person1
True 99.0
False 1.03
I1
I2
I3
I4
I5
Difference1
0
0
0
100
0
10
I1
I2
I3
I4
I5
Person2
True 89.8
False 10.2
Histogram1
0
0
100
0
0
10
I1
I2
I3
I4
I5
Difference2
0
0
0
0
100
30
I1
I2
I3
I4
I5
Histogram2
0
0
100
0
0
10
Minimum expected indexing error rate is achieved by indexing each
time slice with the most probable state of each variable
Can be used to automatically recognize high-level concepts from
low-level media features across time
8 of 18
Specification of content analysis tasks in DMJ I
Person Object Class
Person1—
Person1
...
Histogram1?
Person1—
Difference1?
Input variable
Causal relationship
Person1
Output variable
Based on the framework of object-oriented DBNs
DBN parameters and low-level media features are encapsulated
within reusable DBN object classes representing high-level concepts
NIK 2000
Specification of content analysis tasks in DMJ II
Within Object
Within-
Event 1
Within
Event 2
Person Object
Person1-
Event 1
Person2-
Person1
Histogram1?
Input variable
Difference1?
Within-
Output to input
reference
Person1
Output variable
Person Object
Histogram2?
Variable from
preceding time slice
Person2
Difference2?
Histogram2?
Information
variable
Causal relationship
Within
Hypothesis variable
A content analysis task is specified at the end-user level in terms of
high-level concepts by
— combining appropriate instances of predefined DBN object classes
10 of 18
Media processing strategies
Result from calculation
of an algorithm.
Path, chosen based
upon result.
Classification node.
Result from
picture difference
algorithm
Within=
no
Within=
no
Timeslice n-1
Result from
color histogram
algorithm
Within=
no
Within=
yes
Result from
color histogram
algorithm
Within=
no
Timeslice n
Within=
yes
Within=
yes
Within=
yes
Timeslice n+1
NIK 2000
Media processing strategies
State of the art content analysis systems normally execute feature
extraction algorithms indiscriminantly in every time slice
— Fixed resource usage, latency and reliability
The DMJ system can adapt to a greater range of processing
environments by trading off reliability against resource usage and
latency
A trade off (media processing) strategy is implicitly represented as a
classification tree
— Results from already executed feature extraction algorithms are
used to determine which algorithms to execute next
Appropriate media processing strategies are generated real time
based on our novel hypothesis driven inference approach
12 of 18
NIK 2000
Distributed processing I
Units of distribution
Primitive Detector, PD
operates directly on media streams
executes a low-level quantitative media processing algorithm (color
histogram/picture difference), producing media features
Composite Detector, CD
contains a DOOBN which implicitly builds classification trees
composes results (color histogram/picture difference values) from
other PDs
CDs and PDs may execute on any host computer as a collection of
interacting components
13 of 18
NIK 2000
Distributed processing II
Interaction model
Possible styles of communication:
push - process continuously and transmit
eager pull - process continuously, no transmission
lazy pull - demand driven processing and transmission
The interaction model should:
simplify adaptation - due to changes in resource availability
simplify reconfiguration - due to large changes in resource availability
Fits the publish/subscribe interaction paradigm
be flexible - loose coupling by indirect communication between components
Event based model
14 of 18
Distributed processing III
Computational architecture
Mediastream
PD
PD
P
CD
P
P
C
C
Event Broker
C
P
P
PD
Filter
Filter
C
PD
C
Mediastream
Filter
Metadata Database
PD: Primitive Detector
Filter
(color histogram/picture difference)
CD: Composite Detector
(person)
P: Producer
C: Consumer
NIK 2000
Distributed processing IV
Resource management
Effective utilization of available resources in different environments
requires different solutions.
In a “best effort” environment, monitor and detect situations where
resources are insufficient/available and adapt/reconfigure - change
to strategy with lower/higher cost
Environments supporting resource reservation gives some
guarantees, but is not commonly available
The DMJ system should be deployable in different kinds of
environments, but take advantage of eventual resource reservation
capabilities
16 of 18
NIK 2000
Experiments and empirical results
Currently a prototype is in development
Separate parts have been implemented
Integrating these parts is current work
Preliminary results for trade off between resource usage and reliability:
0,25
0,2
(UURUUDWH
0,15
0,1
0,05
0
0
0,1
0,2
0,3
0,4
0,5
0,6
0,7
0,8
0,9
1
5HODWLYHFRVW
17 of 18
NIK 2000
Summary
We have presented a framework for on-line content analysis of
networked multimedia sessions, including
An approach for specification of content analysis tasks in terms of
high-level concepts
A methodology for real-time generation and execution of media
processing strategies
— trades of content analysis reliability against resource usage
A flexible interaction model supporting
— dynamic adaption and reconfiguration of content analysis tasks
A computational architecture for distributed processing
Preliminary empirical and experimental results indicate the suitability
of our approach
18 of 18