http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Multimedia-Systems:
Synchronization
Prof. Dr.-Ing. Ralf Steinmetz
Dr. L.Wolf, Dr. S.Fischer
TU Darmstadt - Darmstadt University of Technology,
Dept. of Electrical Engineering and Information Technology, Dept. of Computer Science
KOM - Industrial Process and System Communications, Tel.+49 6151 166151,
Merckstr. 25, D-64283 Darmstadt, Germany, Ralf.Steinmetz@KOM.tu-darmstadt.de Fax. +49 6151 166152
GMD - German National Research Center for Information Technology
IPSI - Integrated Publication and Information Systems Institute, Tel.+49 6151 869869
Dolivostr. 15, D-64293 Darmstadt, Germany, Ralf.Steinmetz@darmstadt.gmd.de Fax. +49 6151 869870
14-sync.fm 1 22.October.99
Usage
Services
Systems
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Scope
Applications
Learning & Teaching
Content
Processing
Documents
Design
Security
Opt. Memories
Group
SynchroCommuninization
cations
...
Databases
Media-Server
User Interfaces
Programming
Operating Systems
Communications
Quality of Service
Networks
Basics
Compression
14-sync.fm 2 22.October.99
Computer
Architectures
Image &
Graphics
Animation
Video
Audio
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
14-sync.fm 3 22.October.99
Contents
1. Motivation - What is (Multimedia) Synchronization?
2. Synchronization and Multimedia
3. Reference Model for Multimedia - Synchronization
4. Synchronization in a Distributed Environment
5. Synchronization Techniques
6. Case Studies
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
1. Motivation - What is (Multimedia) Synchronization?
Application 1
Content
Space
Application 2
Application 4
Application 3
Time
We concentrate on:
• Temporal relationship
• New flavor in the context of multimedia
• Essential for communications
14-sync.fm 4 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Content Relations
Dependence of media objects on data
Examples:
• A graphic that visualizes data from a spreadsheet
• Two graphics showing different views of the same data
Explicit definition of dependences for automated update
• Only the data are edited
• All views of the data are
• generated automatically
• cannot be edited directly
• An update of the data triggers an update of the related views.
Implementation of content relations is based on the use of
• common data structures or
• common object interfaces
14-sync.fm 5 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Spatial Relations
Usually known as layout relationships
• Define the space which is used for the presentation of a media object
• on an output device
• at a certain point of time in a multimedia presentation
Typically expressed in layout frames:
• A layout frame is placed and a content is assigned to this frame
• The positioning of a layout frame in a document may be
• fixed to a position in a document
• fixed to a position on a page
• relative to the positioning of other frames
Spatial relations for time-dependent media objects:
• A frame or a group of frames may be presented in a window
• An audio presentation can be positioned on a stereo output device
14-sync.fm 6 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Temporal Relations
Importance for time-dependent media objects
• new flavor in context of Multimedia
• essential for communications
Example for temporal relations:
• Temporal relation between
• video object
• audio object
recorded during a concert
At presentation time:
• temporal relation of the two media objects must correspond to the temporal
relation at the recording moment
Relations:
• at the same time
• independently
• after another
14-sync.fm 7 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Intra-object Synchronization
Intra-object Synchronization
Intraobject synchronization defines the time relation between
various presentation units of one time-dependent media object
Example:
• Time relation between the single frames of a video sequence.
For a video with a rate of 25 frames per second each of the frames has to
be displayed for 40 ms
40ms
t
Frames of a video sequence that shows a jumping ball
14-sync.fm 8 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Inter-object Synchronization
Inter-object Synchronization
Inter-object synchronization defines the synchronization between
media objects
Example:
• Time relations of a multimedia synchronization
that starts with an audio/video sequence,
followed by several pictures
and an animation that is commented by an audio sequence
Audio1
Audio2
P1
Video
P2
P3
Animation
t
14-sync.fm 9 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Logical Data Units
Logical Data Unit
An information unit - usually part of a sequence of information
units - of a time-dependent media object
Granularity levels of LDUs
• Application dependent
• Imply a hierarchical decomposition of media objects
• Example:
Samples
Notes
Movements
Symphony
The LDU hierarchy of a Symphony
14-sync.fm 10 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Classification of LDUs
Closed LDUs
• have a predictable duration
Example:
• LDUs that are parts of stored media objects of continuous media like
• audio and video or
• stored media objects with a fixed duration.
Open LDUs
• duration is not predictable before the presentation execution
Example:
• Open LDUs typically represent
• input from a life source, for example a camera or a microphone, or
• media objects that include an user interaction
14-sync.fm 11 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Classification of LDUs (cont.)
Physical units as LDUs:
Pic. 1 Pic. 2 Pic. 3 Pic. 4
Pic.n
1/30 s
t
t
Physical units of a video sequence as LDUs
Blocks of samples as LDUs:
physical frame duration = 1/sample frequency (e.g., 1/8000 s)
Audio 1 Audio 2 Audio 3 Audio 4
Audio n
t
duration of a Logical Data Unit of 512 Bytes (e.g., = 0.064 s)
Blocked physical units of an audio sequence as LDUs
14-sync.fm 12 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Classification of LDUs (cont.)
User selected LDU duration
• typically in the case of computer generated media objects.
Frame1 Frame 2 Frame 3 Frame 4
Frame n
t
duration of Logical Data Units of selected size (= e.g.1/15 s)
LDUs of an animation sequence of user selected size
Varying durations of LDUs
• typically in the case of recorded user interactions.
Action 1
Action 2
A.3
Action 4
t
LDUs of a recorded user interaction
LDUs of an user interaction in a window system with varying durations
14-sync.fm 13 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Classification of LDUs (cont.)
Open LDUs with unpredictable duration
• Typically a user interaction with in advance unknown duration
User selection
LDU duration not known before the actual processing
An open LDU representing an user interaction
Timers represented as stream of empty LDUs
LDUs of 10 ms duration
LDUs of a timer
14-sync.fm 14 22.October.99
t
t
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Classification of LDUs - Overview
14-sync.fm 15 22.October.99
Duration defined by
Fixed Duration of LDU
Variable or unknown
Duration of LDU
Capturing
User
Audio, Video
Animation,
Timer
Recorded
interaction
User
interaction
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Examples
Lip synchronization
• demands for a tight coupling of audio and video streams with
• a limited skew between the two media streams
Frame 1 Frame 2 Frame 3 Frame 4
Audio 1
Frame n
Audio 2
Audio max. 80 ms advance
Audio n
Video max. 80 ms advance
LDU view of lip synchronization
Slide show with audio comment:
Slide 1
Slide 2
Slide 3
Audio sequence
LDU view of a slide show
14-sync.fm 16 22.October.99
Slide 4
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Application Example
Example used further in the course:
A lip synchronized audio video sequence (Audio1 and Video) is followed by a replay of a recorded user interaction (RI), a slide
sequence (P1 - P3) and an animation (Animation) which is partially
commented using an audio sequence (Audio2). Starting the animation presentation, a multiple choice question is presented to the
user (Interaction). If the user has made a selection, a final picture
(P4) is shown.
RI
Audio1
Video
Interaction
P1 P2 P3
Audio2
P4
Animation
t
Synchronization example
14-sync.fm 17 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
2. Synchronization and Multimedia
MM User Interface
MM
Application
MM
Programming
Environment
Sync. QoS
Requirements
MM
Operating
System
Environment
Typical Requirements:
• hard vs. soft
Strong Interaction with QoS
• acceptable, too bad ...
Variety of Mechanisms
• Live Synchronisation
• Live Synchronisation with Storage
• Synthetical Synchronisation
14-sync.fm 18 22.October.99
Sync. QoS
Provision
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Why Multimedia-Synchronization?
MM User Interface
MM
Application
MM
Programming
Environment
Sync.
QoS
Requirements
MM
Operating
System
Environment
Media
Video
Audio
14-sync.fm 19 22.October.99
Sync.
Qos
Provision
Animation
Audio
Image
Text
Animation
Audio
Image
Text
Pointer
Influenced by
Skew (QoS)
?
?
?
?
?
?
?
?
?
?
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Multimedia Synchronization - Experiments
Example:
• at IBM ENC Heidelberg to quantify synchronization requirements for:
• Audio/video synchronization
• Audio/pointer synchronization
Selection of material:
• Duration
• 30s in experiments
• 5s would have been sufficient
• Reuse of same material for all tests
Introduction of artificial skew:
• By media composition with professional video equipment
• With frame based granularity
Test conditions:
• Huge set of test candidates
• Professional: cutter at TV studios
• Casual: every day “user”
• Awareness of the synchronization issues
• Set of tests with different skews lasted 45 min
14-sync.fm 20 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Feedback - Questionnaire
1
While watching this video clip, did you detect any artifact or
strange effect?
If so, please try to describe it in a few words. (-)
....
2
Are you able to identify if audio was ahead of or behind the
moving pictures? (.)
a)
Yes, I identify audio to be played ahead
of video
b)
Yes, I identify audio to be played behind
video
c)
No, I notice that audio is out of sync with
respect to video but, I am not sure if
audio is played ahead of or behind video.
3
14-sync.fm 21 22.October.99
You noticed a synchronization error.
How would you qualify this error if you have to watch all your
TV programs with such an error? (.)
a)
I would not mind, the error is acceptable
b)
I dislike it, the error is annoying
c)
I am not sure if I would accept such an
error or if I would really dislike it
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Lip Synchronization: Major Influencing Factors
Video:
• Content
• Continuous (talking head) vs. discrete events (hammer and nails)
• Background (no distraction)
• Resolution and quality
• View mode (head view, shoulder view, body view)
Audio:
• Content
• Background noise or music
• Language and articulation
14-sync.fm 22 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Lip Synchronization: Level of Detection
Detected Errors [%]
100
80
60
40
20
0
-320 -280 -240 -200 -160 -120 -80 -40
Head View
Shoulder View
audio behind video
Areas:
• In sync QoS: +/- 80 ms
• Transient
• Out of sync
14-sync.fm 23 22.October.99
0
40 80
Body View
120 160 200 240 280 320
Skew [msec]
audio ahead of video
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Lip Synch.: Level of Accuracy/Annoyance
Level Of Annoyance [%]
100
80
60
40
20
0
-320 -280 -240 -200 -160 -120 -80 -40
annoying
indifferent
audio behind video
Some observations:
• Asymmetry
• Additional tests with long movie
• +/- 80 ms: no distraction
• -240 ms, +160 ms: disturbing
14-sync.fm 24 22.October.99
0
40
acceptable
80 120 160 200 240 280 320
Skew [msec]
View: Shoulder
audio ahead of video
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Pointer Synchronization
Fundamental CSCW shared workspace issue
Analysis of CSCW scenarios:
• Discrete pointer movement (e.g. “technical sketch”)
• Continuous pointer movements (e.g. “route on map”)
Most challenging probes:
• Short audio
• Fast pointer movement
14-sync.fm 25 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Pointer Synchronization: Level of Detection
Detected Errors [%]
100
80
60
40
20
0
-3000 -2000 -1500 -1250 -1000 -750 -500 -250
Map
250
Technical Sketch
audio behind pointer
Observations:
• Difficult to detect “out of sync”:
• i.e., other magnitude than lip sync
• Asymmetry:
• According to every day experience
14-sync.fm 26 22.October.99
0
500 750 1000 1250 1500 2000 3000
Skew [msec]
audio ahead of pointer
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Pointer Synchronization: Level of Annoyance
Level Of Annoyance [%]
100
80
60
40
20
0
-3000 -2000 -1500 -1250 -1000 -750 -500 -250
Map
audio behind pointer
Areas:
• In sync: QoS -500 ms, +750 ms
• Transient
• Out of sync
14-sync.fm 27 22.October.99
0
250
Technical Sketch
500
750 1000 1250 1500 2000 3000
Skew [msec]
audio ahead of pointer
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization QoS: Skew Values
Media
Mode, Application
Video Animation correlated
+/- 120 ms
Audio
lip synchronization
+/- 80 ms
Image
overlay
+/- 240 ms
non-overlay
+/-500 ms
overlay
+/- 240 ms
non-overlay
+/-500 ms
Text
Audio Animation event correlation (e.g., dancing)
+/- 80 ms
tightly coupled (stereo)
+/- 11 µs
loosely coupled (dialogue mode with various
participants)
+/- 120 ms
loosely coupled (e.g., background music)
+/- 500 ms
tightly coupled (e.g., music with notes)
+/- 5 ms
loosely coupled (e.g., slide show)
+/- 500 ms
Text
Anmerkungen zu Text
+/- 240 ms
Pointer
Audio Related to the Itlem
-500ms
+750 ms
Audio
Image
14-sync.fm 28 22.October.99
QoS
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
3. Reference Model for Multimedia - Synchronization
A reference model is needed
• to understand the various requirements for
multimedia synchronization
• to identify and structure runtime mechanisms
• to identify interfaces between runtime mechanisms
• to compare system solutions for multimedia synchronization systems
14-sync.fm 29 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Existing Synchronization Models
Classification methods:
• Physical, system, and human level
• Intra-stream (fine-grain) synchronization and
inter-stream (coarse-grain) synchronization
• Live and synthetical synchronization
• Mapping
Multimedia
Object
t
Time
composition
Derived
Media Objects
Derivation
Media Objects
Interpretation
Uninterpreted
Bytestream
14-sync.fm 30 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Classification of Synchronization Techniques
Inter-object synchronization techniques:
• used to control jitter between media streams
• classification according to the type and location
of the synchronization control
Distributed synchronization control:
• distributed control based on protocols
• distribution based on servers
• distribution on nodes without server structure
Local synchronization control:
• control on several layers
• use of local servers
14-sync.fm 31 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Four Layer Model
Multimedia application
Specification layer
Stream layer
Media layer
14-sync.fm 32 22.October.99
Abstraction
Object layer
high
low
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Specification
Synchronization specification
• describes all temporal dependencies of the included objects
in a multimedia object
• is part of the description of a multimedia object
• should comprise:
• Intra-object synchronization specifications
for the media objects of the presentation
• Quality of service descriptions for the intra-object synchronizations
• Inter-object synchronization specifications
between the media objects of the presentation
• Quality of service descriptions for the inter-object synchronizations
• may comprise:
• alternative presentation forms
• must be explicitly described in a synthetical synchronization
• is implicitly defined in a live synchronization
14-sync.fm 33 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Layered Model - Summary
Layer
Interface Abstraction
Tasks
Specification
The tools performing the tasks of this
layer have interfaces, the layer itself has
no upper interface
to edit
to format
to map the quality of service values to
the object layer
Object
Objects that hide types of enclosed
media
Plan and coordinate presentation
execution
Synchronization Specification
Initiate presentation of time-dependent
media objects
Media-oriented quality of service (In
terms like allowed skew, allowed jitter)
Stream
Streams and groups of streams
Guarantees for intra-stream
synchronization
Initiate presentation of time-independent media objects
Resource Reservation and scheduling
of LDU processing
Guarantees for inter-stream synchronization of streams in a group
Media
Device independent access to LDUs.
Guarantees for single LDU processing
14-sync.fm 34 22.October.99
File and device access
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
4. Synchronization in a Distributed Environment
More complex than in a local environment:
• Distributed storage of synchronization information
• Distributed storage of media objects
• Communication delays and jitter
• Multi-party communication patterns
14-sync.fm 35 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Transport of Synchronization Specification
Delivery of the complete synchronization information
before the start of the presentation
Use of an additional synchronization channel:
Video
Audio
Source
Sink
Synchronization
Use of a multiplexed channel:
Video
Source
multiplexed
Audio
Synchronization
14-sync.fm 36 22.October.99
Sink
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Location of Synchronization Operations
Synchronization at the sink node
Source
Bandwidth
demand
Sink
Synchronization at the source node
Source
14-sync.fm 37 22.October.99
Bandwidth
demand
Sink
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Clock Synchronization
Problem: Skew between the clocks
Source A
Ta, Sa
Audio
Nla
Sink AV
Tav
Tv, Sv
Source V
Video
Nlv
• Clock skew has to be considered in the planning of the presentation:
• Tav: Start of presentation
• Nla: Known net delay
• Sa,Sv as Skews
• Start time of audio transmission of Source A: Ta=Tav - Nla - Sa
• Start time of video transmission of Source V: Tv=Tav - Nlb - Sv.
• Restricted buffer size demands for restricted skew
14-sync.fm 38 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Multiple Communication Relations
Sources : Sinks
1:1
n:1
1:n
n:m
Sources
14-sync.fm 39 22.October.99
Sinks
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Multi Step Synchronization
• Synchronization during object acquirement
• Synchronization of retrieval
• Synchronization during delivery of the LDUs to the network
• Synchronization during the transport
• Synchronization at the sink
• Synchronization within the output device
14-sync.fm 40 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
5. Synchronization Techniques
Specification of great importance
• Simple specification method:
A before B
A overlaps B
A starts B
A equals B
A
B
A
B
A
B
A meets B
A during B
A finishes B
A
B
Types of temporal relations between two objects
14-sync.fm 41 22.October.99
A
B
A
B
A
B
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Specification Methods - Requirements
Complex specification comprise:
• multiple object synchronization
• user interactions
Requirements for a specification method:
• Support of object consistency and for maintenance of specifications
• Media objects should be kept as one logical unit in the specification
• All types of synchronization relations should be easily describable
• Support for the integration
of time-dependent and time-independent media objects
• Quality of service requirements must be describable
• Support of hierarchical levels of a synchronization
14-sync.fm 42 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Basic Hierarchical Specification
Two main synchronization operations:
• Serial synchronization of actions
• Parallel synchronization of actions
Action:
• an atomic action handles the presentation of either a single-media object
or a user input or a delay.
• a compound action is a combination of synchronization operators
and atomic actions.
Pic. 1
Pic. 2 Pic. 3
Slide sequence
Serial and parallel presentations
14-sync.fm 43 22.October.99
Audio
Video
Lip synchronous audio/video
presentation
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Basic Hierarchical Specification
Delay as a possible action allows for
• delays in serial presentations and
• delayed presentations of objects in a parallel synchronization.
Application example:
RI
Audio1
Pic.1
Pic.2
Pic.3
Video
Ani.1
Ani.2 Aud.2
Ani.3
UI
Pic.4
Hierarchical specification of the application example (RI = Recorded Interaction, Pic.
= Picture, Aud. = Audio, Ani. = Animation, UI = User Interaction)
14-sync.fm 44 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Basic Hierarchical Specification
Not all relations can be described.
A1
A2
B1
A3
B2
C1
B3
C2
C3
Example of a synchronization that can not be described
Advantages
Disadvantages
Easy to understand
Additional description of skew
quality of service necessary
Natural support of
For the presentation of time-independent media objects presentation durations must be added
hierarchies
Integration of
interactive objects is
easy
Need for the splitting of media objects for
synchronization purposes
No adequate abstractions for media object contents
Some synchronization scenarios can not be described
14-sync.fm 45 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Based on a Global Timer
All single-media objects are attached to a time axis
Application example
Record. UI
Audio1
Video
Interaction
P1 P2 P3
Animation
P4
Audio2
t
?
Time axis synchronization specification of the application example
14-sync.fm 46 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Based on a Global Timer
Advantages
Easy to understand
Objects of unknown duration cannot be integrated, extensions to the model are required
Support of hierarchies easy to realize
Skew quality of service has to be specified
indirectly by using the common time axis or
additional quality of service specifications
have to be given
Easy to maintain because of the mutual
independence of objects
In a distributed environment the notion of a
global unique time has to be kept
Good abstraction for media contents
Integration of time-independent objects is
easy
14-sync.fm 47 22.October.99
Disadvantages
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Based on Virtual Axes
Specification using
• coordinate systems with user defined measurement units
• several virtual axes to create a virtual coordinate space
Example:
Virtual axes with measurement unit pitch
Virtual axis with measurement unit beat
Music notes as example for virtual axes
At runtime the virtual axes are mapped to real axes
14-sync.fm 48 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Based on Virtual Axes
Application example:.
P4
Time axis
t
Interaction
Interaction axis
P1 P2 P3
Audio1
Animation
Record. UI
Video
Audio2
Time axis
Virtual time axis specification of the application example
14-sync.fm 49 22.October.99
t
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization Based on Virtual Axes
Advantages
Easy to understand
Skew quality of service only indirectly or by
additional specifications definable
Often specification according to the problem
space possible
Specification may become complex by
many axes
Good possibility for building hierarchies
Mapping of axes at runtime may be complex and time consuming
Easy maintainable, because objects are
kept as units and mutual independent
objects
Good abstraction for media content
Easy integration of time independent media
objects
Interactive objects can be included using
specialized axes
14-sync.fm 50 22.October.99
Disadvantages
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization based on Reference Points
Synchronization via reference points:
• Time-dependent single-media objects are regarded as sequences of
LDUs
• Reference points:
• the start time and the stop time of the presentation of a media object
• the start-times of the subunits of time-dependent media objects
• Synchronization point:
• A set of connected reference points
• Defines the synchronization between objects
• LDUs have to be started resp. stopped when it is reached
• Description of temporal relations without explicit reference to time.
14-sync.fm 51 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization based on Reference Points
Lip synchronization example
Video
Audio
Lip synchronization in the reference point synchronization model
Slide show example.
Slide 1
Slide 2
Slide 3
Slide 4
Audio
Example of a slide show with audio sequence in the reference point model
14-sync.fm 52 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization based on Reference Points
Application example
Record. UI
Audio
Video
Interaction
P1 P2 P3
Audio
P4
Animation
Example of the integration of time-dependent and time-independent media objects
as well as closed and open LDUs in a reference point synchronization specification
14-sync.fm 53 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization based on Reference Points
.
Advantages
Easy to understand
More difficult to change because of the
direct description of dependencies
Natural description of
temporal relations
Sometimes complex hierarchies
Easy integration of open LDUs
Integrated description of skew quality of service
Easy integration of time-independent
objects
Time axis are special case of the reference
point synchronization model
14-sync.fm 54 22.October.99
Disadvantages
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Event-based Synchronization
Event-based synchronization presentation actions
are initiated by the synchronization events
Typical presentation actions are:
• to start a presentation
• to stop a presentation
• to prepare a presentation
Events
• may be external,
e.g. generated by a timer, or
• internal to the presentation
generated by a time-dependent media object that reaches a specific LDU
14-sync.fm 55 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Event-based Synchronization
Application example:
Event
Action
Start
Audio1
start
Video
start
Pic.1
start
Timer1
start(3)
Pic.2
...
14-sync.fm 56 22.October.99
Audio.stop
Timer1.
ready
stop
start
...
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Event-based Synchronization
Advantages
Disadvantages
Easy integration of interactive objects
Not easy to handle
Easy extensible by new events
Complex specification
Flexible because any event
can be specified
Hard to maintain
Integration of time-independent objects by
using additional timers
Separate descriptions of quality of
service necessary
Difficult use of hierarchies
14-sync.fm 57 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Petri net-based synchronization
The rules for a timed petri net are:
• A transition fires if all input places contain a non blocking token.
• If a transition fires, on each input place a token is removed and to each
output place a token is added
• After firing, a token is blocked for the duration that is assigned to its new
place
Lip synchronization example.
F1
F2
F3
F4
F5
11 ms
11 ms
11 ms
11 ms
11 ms
A1
A1
33ms
33ms
Petri net based synchronization of a lip synchronization
14-sync.fm 58 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Petri-Net based synchronization
Example for a hierarchy:
D
0 ms
F1
F2
11 ms
11 ms
A1
33ms
S
33 ms
14-sync.fm 59 22.October.99
F3
11 ms
D
0 ms
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Petri-Net based synchronization
Application example:
Audio-Video
subnet
Record.UI
Picture
subnet
S
R
P
60 s
20 s
15 s
User interaction
PI
A1
Animationsubnet 1
Closing picture
P4
A2
Au
Animation subnet 2
and Audio subnet
Petri net specification of the application example
14-sync.fm 60 22.October.99
A3
Animation
subnet 3
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Petri-Net based synchronization
.
14-sync.fm 61 22.October.99
Advantages
Disadvantages
Hierarchies can be created
Not easy to handle
Easy integration of time-independent
objects
Complex specification
Easy integration of interactive objects
Splitting of media objects
Integrated skew quality of service
Insufficient abstraction for the content of the
media objects
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Scripts
Script
• Textual description of a synchronization scenario
• Elements of scripts are activities and subscripts
• Three main operations in a script are
• serial presentation
• parallel presentation
• repeated presentation of a media object
• Often full programming languages extended by the timing operations
14-sync.fm 62 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Scripts
activity
activity
activity
activity
activity
activity
activity
activity
activity
activity
DigAudioAudio(“video.au”);
SMPVideo(“video.smp”);
XRecorderRecorder(“window.rec”);
PicturePicture1(“bild1.jpeg”);
PicturePicture2(“bild2.jpeg”);
PicturePicture3(“bild3.jpeg”);
PicturePicture4(“bild4.jpeg”);
StartIntercationSelection;
DigAudioAniAudio(“animation.au”);
RTAnimaAnimation(“animation.ani”);
script Picture_sequence 3Pictures=
Picture1.Duration(5) >>
Picture2.Duration(5) >>
Picture3.Duration(5);
script Lipsynch AV = Audio & Video;
script AniComment AA = Animation &
AniAudio.Translate(2);
script Multimedia Application_example {(
AV >> Record. UI >> 3Pictures >>
( (Selection >> Picture4) & AA )}
14-sync.fm 63 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Scripts
Advantages
Disadvantages
Easy hierarchies
Not easy to handle
Logical structure of objects
can be kept
Complex specification
Easy integration of time-independent
objects
Implicit usage of common
timers necessary
Easy integration of interactive objects
Special constructs for skew quality of service necessary
Skew quality of service can
be specified
Easy extensible by new synchronization
constructs
Flexible because programmable
14-sync.fm 64 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
6. Case Studies
MHEG
Coded Representation of Multimedia and Hypermedia Objects
• representation and encoding of
• final form multimedia and hypermedia information objects to be
interchanged within or across applications and services
MHEG objects
• Content objects encapsulate media objects that are part of the
presentation
• Composite objects act as a ‘container’ to group a set of MHEG objects
• Action objects are used to exchange sets of actions to be performed
between objects
• Link objects are used to specify spatial, temporal and conditional
relations between MHEG objects
14-sync.fm 65 22.October.99
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Synchronization in MHEG
Generic space:
• provides a virtual coordinate system
• used to specify the layout and relation of content objects
• one time axis of infinite length measured in generic units of time
• three spatial axes (X=latitude, Y=longtitude, Z=altitude) of finite length
in an interval of [-32768,+32767]
• MHEG maps from the virtual to the real coordinate space
Presentation of content objects:
• Based on the exchange of action objects that are sent to an object
• Examples for actions are:
• prepare for setting the object in a state that enables the object
to present itself
• run to start the presentation
• stop to stop the presentation
14-sync.fm 66 22.October.99
Action list:
• Combination of actions to action lists
• Parallel action lists are executed in parallel
• Each list is composed of a delay followed by delayed sequential actions.
delay
actions
sequential
Parallel lists of actions
Delayed
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
MHEG action lists
action
delay
action
Links:
• used to create event-based synchronizations
• can trigger events
14-sync.fm 67 22.October.99
...
Application
MHEG Engine
Object Manager
Link Processor
Interpreter
Operating System
14-sync.fm 68 22.October.99
Generic
Presentation Services
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
MHEG Engine
AV-Subsystem
User
Interface
Services
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
MHEG Relation to Reference Model
Specification layer
Object layer interface
MHEG Engine
Object layer
Stream layer interface
Stream layer
Media layer interface
Media layer
14-sync.fm 69 22.October.99
MHEG coded multimedia object
Interpreter
Object Manager
Link Processor
User Interface
Service
AV Subsystem
http://www.kom.e-technik.tu-darmstadt.de
http://www.ipsi.gmd.de
© Ralf Steinmetz
Case Studies
Other systems to be mentioned:
• HyTime
• FireFly
• MODE (Multimedia Objects in a Distributed Environment)
• Multimedia Tele-Orchestra
• ACME (Abstractions for Continuous MEdia)
• Apple QuickTime
• ...
14-sync.fm 70 22.October.99