The MPEG-7 Standard
- A Brief Tutorial -
Ali Tabatabai
Sony US Research Laboratories
February 27, 2001
Outline
 Objectives of the MPEG-7 Standard
 Main Elements of MPEG-7
 Scope of MPEG-7
 MPEG-7 Application Areas
 MPEG 7’s relation with other standards
2
Why do we need MPEG-7 ?
Support for Advanced Query
• Fast & Accurate Access
Need
• Personalized Content
Production and
Consumption
• Content Management
+
•Visual
•Audio
•Sketch
• Automation
3
MPEG: A Brief History (1)
 MPEG: Moving Picture Experts Group
Established in 1988
 ISO / IEC/JTC1/SC29/WG11
A Working Group of ISO/IEC in charge of the
Development of Standards for
Coded Representation of Digital Audio and Video
4
MPEG: A Brief History (2)
 MPEG-1: Interactive CD and MP3
11 / 1992
 MPEG-2: DTV, STB, DVD
11 / 1994
 MPEG-4: Web and Mobility
ver1: 09 /1998
ver2: 11 /1999
 MPEG-7:
???
 MPEG-21: Multimedia Framework
08 / 2001
11 / 2001
5
MPEG-7: What Is It ?
THE MPEG 7
STANDARD
IS NOT a COMPRESSION Standard
similar to MPEG-1/2/4 or their
Extension
IS NOT a STANDARD for
FEATURE
EXTRACTION/MATCHING
Content Description of
Various Audio Visual
Information
Types of Audio Visual Information
• Audio, Speech
• Moving video, still pictures, graphics
• Information on how objects are combined
in scenes
6
MPEG-7: Application Areas
 Storage and retrieval of audiovisual databases (image, film, radio
archives)
 Broadcast media selection (radio, TV programs)
 Surveillance (traffic control, surface transportation, production chains)
 E-commerce and Tele-shopping (searching for clothes / patterns)
 Remote sensing (cartography, ecology, natural resources management)
 Entertainment (searching for a game, for a karaoke)
 Cultural services (museums, art galleries)
 Journalism (searching for events, persons)
 Personalized news service on Internet (push media filtering)
 Intelligent multimedia presentations
 Educational applications
 Bio-medical applications
MPEG-7
Description Scope for AV Content
 Description Granularity
 Low-level
 High-level
 Form
 Access
 Classification
 Link
 Context
8
MPEG-7: Main Elements
Descriptors (D)
 syntax and semantics of each feature representation
Description Schemes (DS)
 structure and semantics of the relationships between
components
Description Definition Language (DDL)
 creation of new DS’s
 modification/extension of existing DS’s
9
MPEG-7: Major Functionalities
 Systems
(ISO / IEC 15938 - 1)
 Description Definition Language (ISO / IEC 15938 - 2)
 Visual
(ISO / IEC 15938 - 3)
 Audio
(ISO / IEC 15938 - 4)
 Multimedia Description Schemes (ISO / IEC 15938 - 5)
 Reference Software
(ISO / IEC 15938 - 6)
10
MPEG-7: Main Elements (2)
DDL
Instantiation
DS1
DS2
<Object>
<Label/>
<Definition/>
.
.
</Object>
DS2
Systems
DS3
D1
D2
D3
0001100
11
MPEG-7: Systems
It defines tools to:
 provide for efficient storage and transport
 synchronize between content and description
 manage and protect intellectual property
12
MPEG-7: DDL and its Components
 Description Definition Language:
 Creation of the Ds and DS’s: XML Schema & MPEG-7 Extensions
 Instantiation of XML
 XML Schema:
 Data types
 Simple and Complex types
 Elements, attributes
 Inheritance, Abstract types
 MPEG-7 extensions:
 Array and Matrix data type
13
MPEG-7: Audio
 Sound Effects
 Music Instrument Timbre
 Spoken Content
 Melody Contour
14
MPEG-7: Visual (1)
Color
 quantization, dominant, scalable, color-structure, layout,
GoF/GoP
Texture
Shape
 region-based, contour-based, 3D
Motion
 camera motion, motion trajectory, parametric motion, motion
activity
15
MPEG-7: Visual (2)
Localization
 spatio temporal
Others
 face recognition
16
MPEG-7: Basic Visual Structures
 Grid Layout
 2D-3D Multiple View
 Time Series
 Spatial 2D Coordinates
 Temporal Interpolation
17
Low level Audio Visual descriptors
Video segments
Still regions
•
•
•
•
•
•
•
•
Color
Camera motion
Motion activity
Mosaic
Moving regions
Color
Shape
Position
Texture
Audio segments
•
•
•
•
Color
Motion trajectory
Parametric motion
Spatio-temporal
shape
• Spoken content
• Spectral
characterization
• Music: timbre,
melody
18
MPEG-7: MMDS Basic Elements
Root, Top-level
elements,
Packages
Time, Duration,
Medialocators
Language
Annotation,Person, Place
Basic elements
Datatype &
structures
Schema
tools
Link & media
localization
Basic DSs
19
MPEG 7: Content Management and
Description
Format, Coding, Instances,
Identification, Transcoding
Hint, etc.
(Several instances)
Title, Creator, Creation
location & date, Purpose,
Classification, Genre,
Review, Parental guidance,
etc. (Author generated)
Rights holder, Access rights,
Usage Record, Financial aspects,
etc. (Evolution)
Creation &
production
Media
Content
Usage
Content management
Content description
Structural
aspects
Conceptual
aspects
Viewpoint of the structure: Segments
Basic
elements
• Spatial
/ temporal structure
• Audio,&video low-level
Ds
Schema
Datatype
• Elementary
tools
structures semantic information.
Viewpoint of conceptual notions
• Events, objects, abstract concepts, and
Link & media
their
relation
Basic
DSs
localization
20
Example of Segment trees
SR1:
 Creation, Usage meta
information
 Media description
 Textual annotation
 Color histogram, Texture
SR6:
 Color Histogram
 Textual annotation
Background
SR3:
 Shape
 Color Histogram
 Textual annotation
Foreground
SR2:
 Shape
 Color Histogram
 Textual annotation
SR5:
 Shape
 Textual annotation
SR4:
 Shape
 Color Histogram
 Textual annotation
21
Time
Axis
Segment Tree
Shot1
Segment 1
Sub-segment 1
Shot2
Semantic DS (Events)
Shot3
• Introduction
• Summary
Sub-segment 2
Sub-segment 3
• Program logo
• Studio
• Overview
Sub-segment 4
• News Presenter
segment 2
• News Items
Segment 3
• International
• Clinton Case
• Pope in Cuba
Segment 4
• National
Segment 5
Segment 6
• Twins
• Sports
• Closing
Segment 7
22
MPEG 7: Navigation and Access
Creation &
Efficient
support of: discovery,
production
browsing,
navigation, visualization
Media
Content
Usage
Content management
Navigation &
Access
Summary
Content description
Structural
aspects
Basic elements
Datatype &
structures
Conceptual
aspects
Substitution of the original content
Adaptation to terminal, network, or
Schema user preferences
Link & media
tools
localization
Variation
Basic DSs
23
MPEG 7: Hierarchical summary
Hierarchical
Summary
HighlightLevel
Highlight
Segment
Highlight
Segment
Highlight
Segment
HighlightLevel
Highlight
Segment
Highlight
Segment
Highlight
Segment
Highlight
Segment
A-V
Data
24
MPEG 7: Sequential summary
Frame
Property
Sequential
Summary
Text
Property
Sound
Property
Frame
Property
Frame
Property
Text
Property
Sound
Property
Sound
Property
A-V
Data
25
MPEG 7: Variation
Universal Multimedia Access
Adapt delivery to network and terminal characteristics (QoS)
26
MPEG-7: Content Organization
Collection &
Classification
Content organization
Description and organization of
Creation &
collection of documents
Analytic
Model
Navigation &
Access
production
Media
Content
Usage
Content management
Summary
Content description
Structural
aspects
Conceptual
aspects
Variation
Basic elements
Datatype &
structures
Schema
tools
Link & media
localization
Basic DSs
27
MPEG-7: Collection
28
MPEG 7: User Interaction
Collection &
Classification
Content organization
Analytic
Model
Navigation &
Access
Creation &
production
Media
Content
Usage
Content management
Summary
User
preferences
Content description
Structural
aspects
Conceptual
aspects
Variation
User identification and preferences:
Filtering, search and browsing
Basic elements
Datatype &
structures
User
Interaction
Schema
tools
Link & media
localization
Basic DSs
29
MPEG-7
Its Relation with other standards
 AHG on “Metadata harmonization”:
 SMPTE: Metadata dictionary, KLV encoding
 Dublin Core Metadata Initiative
 European Broadcast Union
 AHG on TV AnyTime Application
 Large number of Liaisons:
 SMPTE
 Dublin Core
 W3C (XML Schema)
 etc.
30
MPEG-7: TimeLine - The Work Plan
1996
Competition:
Divergence
• Individual work
• Definition
scope and
r
1998
Call for proposals
Convergence
1999
Working draft
2000
Committee draft
Final committee
draft
2001
International
standard
Draft international
standard
Conclusions on AV Content Description
and MPEG-7
MPEG-7:
 AV content description for interoperable application
Description Definition Language:
 XML Schema (flexibility) + Binary version (efficiency)
Description Schemes:
 Library of description tools
 Covers a wide range of generic needs
32