The MPEG-7 Visual Standard for
Content Description-An Overview
Thomas Sikora, Senior Member, IEEE
A presentation by Modupe Omueti
For
CMPT 820:Multimedia Systems
Spring 2005
1
Contents
Introduction
 Scope
 Methodology
 Visual Descriptors
 Conclusion

2
Introduction

Moving Pictures Expert Group
MPEG-1 for interactive video (1992)
 MPEG-2 for digital television (1994)
 MPEG-4 for multimedia with emphasis on
visual objects (1998 v1, 1999 v2)
 MPEG-7 for multimedia content description
(2001)

3
Trends
Initially few sources of audio, image and
video
 Increase in volume of digitized audio,
images and video
 Still images
digital video

4
MPEG-7




Formally named Multimedia Content Description
Interface
Supports some degree of interpretation of the
information’s meaning
Interpretation can be passed on to or accessed
by a device or computer code
Not aimed at one application in particular
5
Scope

Goals



Elements





Standardized descriptions
Meaningful descriptions
Description tools: visual decriptors and
description schemes
Description Definition Language
System tools
Figure 1: Scope of MPEG-7
Figure 2: MPEG-7 main elements
6
Normative
part of
MPEG-7
standard
Figure 1: Scope of MPEG-7
7
Figure 2: MPEG-7 main elements
8
Applications
Digital libraries (image catalogue, film)
 Broadcast media selection (TV channels)
 Investigation services (human
characteristics recognition, forensics)
 Multimedia editing (personalized
electronic news service)
 Figure 3: Abstract Representation

9
Figure 3: Abstract representation of
possible applications using MPEG-7
10
Methodology
Standard Development
Specification for Technology Requirements
Technology Request
Proposal Evaluation
Experimentation Model Definition
Core Experiments
11
Visual Descriptors

General visual descriptors


Color, texture, shape, and motion features
Domain specific

Identification of human faces and face
recognition
12
Visual Color Descriptors

Color Spaces (HSV, HMMD)



Supports above for normative purposes
Also supports RGB, YCbCr color spaces
Scalable color descriptor Figure 4





Global color Distribution of Images in color
histograms
HSV space, uniformly quantized into 255 bins
Haar Transform used to encode histogram
Histogram bin non-uniformly quantized
color coefficients or histogram bin values for
matching
13
Visual Color Descriptors

Dominant color descriptor



Color layout descriptor


Spatial distribution of color in an arbitrarily shaped region
Color structure descriptor



Global + local spatial color distribution
Colors clustered into a small no of representative colors
representative color, %age, spatial coherency, variance
HMMD, local color feature, sliding window
Histogram on color appearance count
Group of Frames/Group of Pictures


SCD for a collection of similar images (frames) or video
frames
Average, median, intersection histograms of GoF or GoP
14
Figure 4: Three color images and their MPEG-7 histogram
color distribution, depicted using a simplified color histogram.
Based on the color distribution, the two left images would be
recognized as more similar compared to the one on the right.
15
Visual Texture Descriptors

Texture Features
Visual patterns (homogenous or nonhomogenous)
 Multiple colors in images
 Multiple intensities in images
 Surface structural information Figure 5

16
Figure 5: Examples of grayscale images with different textures.
Using the MPEG-7 Visual texture descriptors, the two images on
the bottom would be rated of similar texture, while less similar
in texture compared to the two images on the top.
17
Visual Texture Descriptors

Homogenous texture descriptor Figure 6
Scale and orientation sensitive filters
 Mean and SD of frequency coefficients (RT-FT)
 Scale and rotation-invariant description and matching
 2D Gabor functions for filtering feature channels
 Non homogenous texture descriptor (Edge Histogram)
 Spatial distribution of edges
 Division of image into 16 non overlapping blocks of equal
size
 Five edge categories: vertical, horizontal, 45 , 135 , and non
directional edge.
 Rotation-sensitive and rotation-invariant
 Non uniform quantization using 3 bits, descriptor size of 240
bits (16x5x3)

18
Figure 6: Frequency layout for MPEG-7 Homogenous
Texture Descriptor frequency extraction. Energy and
energy deviation values are extracted from this frequency
division into 30 channels.
19
Visual Shape Descriptors



Provides a powerful visual clue
Invariant to scaling, rotation, and translation
2-D or 3-D in nature
For 2-D there are two categories
 Contour based which uses only boundary
information of objects
 Region-based which the entire shape region
20
Visual Shape Descriptors

3-D Shape Descriptor—Shape Spectrum





Based on a shape spectrum concept
Histogram of a shape index
Measures local convexity of each local 3-D surface
Histograms with 100 bins are used—each quantized
by 12 bits.
Region Based Shape Descriptor (Art) Figure 7



Uses all pixels constituting a shape within a frame
Region-based moments invariant to transformations
Coefficients of ART basis functions quantized
21
Figure 7: Examples of various shapes that can be indexed using MPEG-7
Region-Based Shape Descriptor. Images contained in either of the sets (a)–(d)
would be rated similar and dissimilar to the ones in the remaining sets. For
example, images in set (a) would be identified being similar and dissimilar to
the ones in set (b), (c), or (d).
22
Types of Visual Shape Descriptors

Contour based shape descriptor Figure 9
Curvature scale-space (CCS)
 Eccentricity and circularity values
 Robust to non-rigid motion partial occlusion of
the shape and perspective transformations


2-D/3-D shape descriptor

Representation of 3-D objects using multiple
2-D snapshots
23
Figure 8: Examples of shapes that can be indexed using MPEG-7 ContourBased Shape Descriptor.
24
Motion Descriptors for Video

Motion Activity Descriptors





Activity level and pace of motion in a scene
Motion activity intensity descriptor
SD of motion vector magnitude
SDs quantized into five activity levels
Optional Features




motion direction
spatial distribution of motion activity
Temporal distribution of motion activity
Camera Motion Descriptor Figure 9

Global motion parameters in time



zoom activity
translatory motion
Motion similarity matching in particular time periods
25
Figure 9: Camera model for MPEG-7 Camera Motion Descriptor. Perspective
projection to image plane p and camera motion parameters. The (virtual)
camera is located in O.
26
Motion Descriptors for Video

Warping Parameters


Parametric motion descriptor
Object description using 2-D parametric models






translations, rotations, scaling and combination of them
planar perspective models
quadratic models
Arbitrary objects, defined as regions (group of pixels)
in the image over a specified time interval
Global sprite or mosaic
Motion Trajectory


Description for independently moving objects
Object displacement over time
27
Conclusion
Identify, filter and browse images using
visual content
 Specification to allow interoperability and
flexibility
 Other MPEG-7 standards
 Storage, access and transmission of
descriptors and descriptors schemes in
system specification

28
29
Thank you
30