MPEG for the
past, present and future
of television
Leonardo Chiariglione
Keynote Speech at ATSC Annual Meeting
Washington, DC – 2012/05/08
MPEG for the past of
television
 System
 Transport Stream
 DSM-CC
 Timed metadata
 Video
 MPEG-2 High Profile
 Audio
 MPEG-1 Audio Layer II
 MPEG-2 AAC
MPEG for the past, present and future of TV
2012/05/02
2
An assessment
 MPEG has provided the means for the television
business on air, cable and satellite to migrate from the
analogue to the digital age
 It has been a collective efforts
 Representatives from the entire spectrum of the
broadcasting world provided their requirements
 Packages of patent licences were created (outside of
MPEG)
 Today there is virtually no broadcasting system that is
not based, at least partially, on MPEG standards
MPEG for the past, present and future of TV
2012/05/08
3
MPEG for the present of
television
 Systems
 Carriage of MPEG-4 on MPEG-2 TS
 Dynamic Adaptive Streaming over HTTP (DASH)
 Video
 MPEG-4 Advanced Video Coding (AVC)
 3D extensions
 Audio
 Various forms of Advanced Audio Coding (AAC)
 Universal Speech and Audio (USAC)
MPEG for the past, present and future of TV
2012/05/08
4
An assessment
 The deployment of 2nd generation digital broadcasting
systems continues successfully
 Use of the AVC on MPEG-2 TS allows packing twice as
many programs as MPEG-2 Video in an analogue
channel
 With IPTV the notion of broadcasting is expanding
 MPEG is helping the broadcasting industry
 To enrich the broadcasted user experience
 Through standard technologies that serve multiple fields
MPEG for the past, present and future of TV
2012/05/08
5
MPEG for the future of
television
 Well, that’s what I am supposed to talk about…
MPEG for the past, present and future of TV
2012/05/08
6
Broadcasting is a great
business to be in…
 Broadcasting is a proven business with a constant
revenue flow
 Many technologies appear that
 Can help new competing businesses but
 Can also be used to extend the broadcasting business
 MPEG is a neutral body devoted to the development of
standard technologies for user communities to exploit
MPEG for the past, present and future of TV
2012/05/08
7
MPEG-H: a video standard for
future broadcasting
 ISO/IEC 23008 High Efficiency Coding and Media
Delivery in Heterogeneous Environments
 A suite of standards for media coding and delivery
 Currently scheduled to appear in 3 parts
 MPEG Media Transport
 High Efficiency Video Coding
 3D Audio
 Timeline
Part
CD
DIS
FDIS
MMT
12/07
13/01
13/07
HEVC
12/02
12/07
13/01
3D Audio
14/04
14/10
15/01
MPEG for the past, present and future of TV
2012/05/08
8
Service composition in MMT
MMT Package
composition
information
friendly
with html5
<subtitle>
Bird is flying…
UD contents
3D Earth
Bird is flying…
legacy broadcasting
MPEG for the past, present and future of TV
future broadcasting
2012/05/02
9
Smart delivery in Clouds
video
file
Web
Clouds
pull
pull
MMT Encapsulator
-
Service Component List
Component location
Composition relationship
QoE requirement
How to deliver
How to consume
MMT Frame
- Configuration
- Adaptive delivery
- Sync, Delay, Loss control
2012/05/08
10
Mash-up service
Audio
Video1
Video2
widget
Web page
MMT
assets
Service 1 (MMT Package1)
MPEG for the past, present and future of TV
Service 2 (MMT Package2)
2012/05/08
11
Second screen management
MMT Package
composition
information
<device 1>
…
<device2>
…
Video content
on 2nd device
Additional Information with
Live Content (HTML)
MPEG for the past, present and future of TV
2012/05/08
12
MPEG-H part 2
High Efficiency Video Coding
 Primary goal: ≥ 50% better compression than AVC
 For large displays (e.g. 4kx2k)
 For mobile (e.g. WVGA)
 Developed jointly with ITU-T by Joint Collaborative
Team (JCT) on Video Coding
 Technically in the same stream of past MPEG Video
Coding standards (AVC and prior standards)
 Currently at Committee Draft level
MPEG for the past, present and future of TV
2012/05/08
13
HEVC vs AVC Subjective
Performance Measurements
 Preliminary subjective tests carried out for
HEVC Committee Draft
 Reasonably small and non-overlapping
confidence intervals
 Average rate savings at same (interpolated)
MOS points



HEVC with 67% lower rate in class B (Full HD)
HEVC with 49% lower rate in class C (WVGA)
HEVC with 58% lower rate overall
MPEG for the past, present and future of TV
2012/05/08
14
A performance sample
MPEG for the past, present and future of TV
2012/05/08
15
For those technology aware…
 Block-based/Variable block sizes
 Block motion compensation
 Fractional-pel motion vectors
 Spatial intra prediction
 Spatial transform of residual difference
 Integer-based transform designs
 Arithmetic or VLC-based entropy coding
 In-loop filtering to form final decoded picture
MPEG for the past, present and future of TV
2012/05/08
16
Where we are (May 2012)
 Activity: level of 1,000 documents per meeting crossed
 Very diverse participation from industry/academia
 Significant technical advance over prior standards
 Challenge: computational/implementation complexity
 Deliverables:
 Spec
 Reference software
 Conformance
 Currently only one profile
MPEG for the past, present and future of TV
2012/05/08
17
HEVC Scalability Extensions/1
 Serve the needs of heterogeneous environment of
future digital TV distribution
 Multiple devices: SHDTVs, set-top boxes, tablets, smart
phones, PCs etc.
 Wide range of processing powers, display sizes and power
consumption needs
 Multiple networks: Cable, Satellite, Terrestrial, CDN,
Cable Modem, xDSL, WiFi, 4G
 Wide range of channel bandwidths and QoS
 Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDP
 Multiple services: Broadcasting, On demand, Streaming,
Over The Top
MPEG for the past, present and future of TV
2012/05/08
18
HEVC Scalability Extensions/2
 Options for backward compatible migration paths
 From 720p60 to 1080p60
 From1080p60 to 4kx2kx60
 From AVC to HEVC
 Time Line
 May 2012 Preliminary Call for Proposals (public)
 July 2012 Final Call for Proposals (public)
 October 2012
 Evaluation of proposals & start of collaborative design phase
 Development of the time line associated with standardisation
phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD)
MPEG for the past, present and future of TV
2012/05/08
19
Service Transitions
 1950s: Color TV
 Analog, backward compatible
 1990s: Digital TV
 New infrastructure required
 Transitions from SD to HD
 2010s: 3D
 Mixed results from services introduction is
 Not a single format across all services
MPEG for the past, present and future of TV
2012/05/08
20
Current Status of 3D Video
 Increasing investments in 3D production and services
 Increasingly capable 3D displays in the market
(many competing and emerging technologies)
 Market is primarily stereo
(mix of different formats being deployed right now)
 Focus of current MPEG work: development of a new
3D format and associated compression techniques that
could facilitate generation of multiview output
MPEG for the past, present and future of TV
2012/05/08
21
Existing 3DV Coding Formats





Left

Right
Pack pixels from left and right views
into a single frame (loss of
resolution)

Leverage existing infrastructure and
equipment, only require additional
signaling to de-interleave frame

Embraced by broadcasters for first
phase of 3D broadcast services
Full-resolution coding of stereo and
MV video as extension of AVC
High coding efficiency achieved via
inter-view prediction techniques
2D compatibility supported
Adopted as format for 3D Blu-ray
Disc
Being considered for second phase
of broadcast standards
Left View
Right View
Effective prediction between views for high compression performance
MPEG for the past, present and future of TV
2012/05/08
22
Targets of Future 3DV Format
Stereoscopic displays
• Variable stereo baseline
• Adjust depth perception
Left
Right
Limited
Camera
Inputs
Data
Format
Data
Format
Constrained Rate
(based on distribution)
Auto-stereoscopic
N-view displays
MPEG for the past, present and future of TV
• Wide viewing angle
• Large number of
output views
2012/05/08
23
3DV Reference Framework
Limited # Video Inputs
(e.g., 2 or 3 views)
Depth
Estimation
Video/Depth
Encoder
Decoder and
View Synthesis
Larger #
Output Views
1010001010001
+
Binary Representation
& Reconstruction Process
MPEG for the past, present and future of TV
2012/05/08
24
Overview of Call For
Proposals
 Test Material & Conditions




8 test sequences (1920x1080 & 1024x768)
4 target bit rates per sequence
2-view and 3-view test scenarios
AVC and HEVC compatible test categories
 23 responses (12 AVC + 11 HEVC)
 Evaluation in Nov 2011
 Objective quality (PSNR of synthesized views)
 Subjective assessment (stereo and auto-stereo)
MPEG for the past, present and future of TV
2012/05/08
25
3DV Testing Scenarios
2-view Configuration
Left
Depth
Estimation
3D Video
Codec
Multiview
Rendering
virtual
view
Output to
Stereo
Display
Right
3-view Configuration
Left
Output to
Auto-Stereo
N-view
Display
3D Video
Codec
Center
Right
Depth
Estimation
MPEG for the past, present and future of TV
Multiview
Rendering
virtual
views
2012/05/08
26
Tool Categories
 Texture coding
 Independent of depth
 E.g., inter-view prediction of color view, inter-view prediction
of motion parameters and residual data
 Using depth data
 E.g., view synthesis prediction, motion prediction
 Depth coding
 Independent of texture
 E.g., depth modeling modes, weighted prediction, reduced
resolution coding
 Using texture data
 E.g., motion parameter inheritance, intra prediction
MPEG for the past, present and future of TV
2012/05/08
27
3DV Current Status
 Extensive subjective evaluation of
20+ proposals to the 3DV CfP
 Considered both AVC and HEVC
compatible
 Many new coding tools proposed
for MV texture and depth coding
 Substantial rate savings compared
to capabilities of existing standards
 AVC and HEVC extensions
standardisation under wayd
 Add support for depth
 New AVC-based coding tools
 MV extensions of HEVC
 Hybrid architectures
MPEG for the past, present and future of TV
2012/05/08
28
Hybrid Architectures
 From a pure compression efficiency point of view, it is
always best to use the most advanced codec
 However, when introducing new services, providers
must also consider capabilities of existing receivers
and transition plan
 Use Case:
 Many terrestrial broadcast systems based on MPEG-2
 US cable systems based on mix of MPEG-2 and AVC
 May not be easy to simply switch codecs in near term
MPEG for the past, present and future of TV
2012/05/08
29
Multiview Extensions of
HEVC
 HEVC Stereo/MV extensions being considered
 Extensions expected to be completed approximately 12
months after the base specification, i.e., Jan 2014
 Reportedly simple extensions of HEVC can achieve 3040% bit rate reduction compared to HEVC simulcast
 Further gains expected if block-level changes to codec
are considered
MPEG for the past, present and future of TV
2012/05/08
30
MPEG-H part 3: 3D Audio
 Well, there is no Systems and Video without Audio…
MPEG for the past, present and future of TV
2012/05/08
31
3D Audio requirements/1
 High quality: quality of decoded sound perceptually
transparent
 Localization and Envelopment: Accurate sound
localization and very high sense of sound envelopment
within a targeted listening area (“sweet spot”)
 Flexible Loudspeaker Placement: the transmitted
audio program to be mapped to a rendering setup in
which loudspeakers are in alternate locations
 Interactivity: Interactive modification of the sound
scene rendered from the coded representation
MPEG for the past, present and future of TV
2012/05/08
32
3D Audio requirements/2
 Rendering on setups with fewer loudspeakers
(including headphones): ability to derive a signal from
the transmitted program material for reproduction with
fewer loudspeakers
 Audio/visual alignment & consistency: ability to
adapt the rendered acoustic scene to be consistent
with the visual
 Efficiency for decoding on different setups: ability to
be rendered on all reproduction loudspeaker setups or
headphone
MPEG for the past, present and future of TV
2012/05/08
33
3D Audio requirements/3
 Appropriate computational complexity and system
latency for the target application (Broadcasting, Spatial
two-(n-)way communication / telepresence
 Transcoding for low bandwidth devices: A lower
bandwidth signal can be extracted from the original
program material
 Backward compatibility: (e.g. to 5.1 channel
programs and decoders or transmission of HRTF
encoded signals)
MPEG for the past, present and future of TV
2012/05/08
34
Envisioned Architecture
22.2 Loudspeakers
Home
Theatre
Spatialized on Headphones
22.2 Channel
Audio Program
Audio Objects
3D Audio
Encoder
500 kb/s
bitstream
3D Audio
Encoder
In-Frame Speakers
Tablet
TV
Spatialized on Headphones
Audio Scene
Transcoder
 Inputs

Audio Channels



Channel
–
“normal” content
Audio Objects

•
64 kb/s
bitstream
–
“helicopter overflight”
500 kb/s primary
channel
64 kb/s cellular channel
Smart
Phone
TV
Spatialized on
Headphones
•
Outputs
–
–
Loudspeakers
Headphones
•
–
Spatialized sound
In-frame loudspeakers
•
“Sound Bar”
Audio Scene

Ambisonics
MPEG for the past, present and future of TV
2012/05/08
35
Audio, part of
an Audio-visual Experience
 What is new in video?
 Larger, high-resolution displays
 Greater sense of envelopment
 Possibly closer viewing distance
 More efficient compression
 Wireless transmission to portable devices
 Immersive and enveloping audio experience
MPEG for the past, present and future of TV
2012/05/08
36
Use Case 1
Home Theatre
 Many loudspeakers
 10.1
 22.2
 “3D” Experience
 Height loudspeakers for greater envelopment
 Sense that audio objects are near or distant
 When near, they are consistent with video image
MPEG for the past, present and future of TV
2012/05/08
37
22.2 multichannel sound
system (NHK)
TpFC
TpFL
TpFR
Top layer
9 channels
TpC
TpSiL
TpSiR
TpBC
TpBR
TpBL
FLc
FRc
FC
FR
FL
Middle layer
10 channels
TV Screen
SiL
SiR
BtFC
BtFL
BtFR
LFE1
LFE2
BR
BL
BC
MPEG for the past, present and future of TV
Bottom layer
3 channels
LFE
2 channels
2012/05/08
38
Home Theater – Issues
 Maintain compatibility with legacy systems: 5.1, stereo
 Will content providers really step up to 22.2?
 Will consumers adopt 22.2?: 2 front high
 How to compress 22.2 for transmission?
 MPEG AAC 64 kb/s/channel: -> 1.5 mb/s
 How to render for legacy setups?
 Relatively slow-growth market
MPEG for the past, present and future of TV
2012/05/08
39
Use Case 2
Personal TV
 Small but high-resolution display
 “super-tablet”
 Local wireless communications
 To cable or fiber home hub
 Possible audio presentation
 Headphones
 Loudspeakers around perimeter of display
 Issue: To what extent can the user get an enveloping
experience from only “front” loudspeakers?
MPEG for the past, present and future of TV
2012/05/08
40
Use Case 3
Mobile TV
 Hand-held display: Smartphone
 Headphone listening:
 Stereo, perhaps with binauralisation
 Fastest growing market: Quick rollout and adoption of
standard
 What is new?
 USAC for stereo; MPEG Surround for binauralisation
 What is the “Wow” factor?
 To spur adoption of new technology
MPEG for the past, present and future of TV
2012/05/08
41
User Experience
Spatialized on headphones
Home Theatre
Tablet TV
MPEG for the past, present and future of TV
2012/05/08
42
Flexible Rendering
 Content providers create 22.2 program
 Consumed on stereo, 5.1, 10.1, 22.2 layouts
 Consumed on “wrong” layouts
 Mis-positioned surrounds
 Missing surrounds
 Non-standard layouts
 2 front high, 5.1 mid
MPEG for the past, present and future of TV
2012/05/08
43
3D Audio Test Platform
 NHK “Loudspeaker
Array Frame” (LAF)
 Suggestions for
 Alternative platform
for assessing “Home
Theatre” use case
 Model for assessing
“Personal TV” use
case
MPEG for the past, present and future of TV
2012/05/02
44
Summary
 Provide compelling immersive audio experience for
audio/visual programs
 Create content once, present on many different
loudspeaker layouts or on headphones
 Provide high compression and high-quality audio
presentation
MPEG for the past, present and future of TV
2012/05/08
45
MPEG-V: Humans have
more than two senses
MPEG for the past, present and future of TV
2012/05/08
46
Augmented Reality
Application Format (ARAF)
Remote
Real World
Environment
Remote
Sensors &
Actuators
Local
Real World
Environment
Local
Sensors &
Actuators
MARBle
User
ARAF file
or stream
Media
Servers
MPEG for the past, present and future of TV
Service
Servers
2012/05/08
47
Augmentation Region in
ARAF
AR service
provider A
User A
Broadcaster
User B
Augmentation
Region
MPEG for the past, present and future of TV
AR service
provider B
2012/05/02
48
The past, the present
and the future, again
 25 years ago MPEG selected the digital media way
bringing innovation and interoperability
…and thrived
 This year the broadcasting industry has started a move
in future broadcast services with interoperability at its
core
…and it is bound to thrive
 MPEG looks forward to continuing a cooperation with
the broadcasting industry providing the necessary
standard technologies
MPEG for the past, present and future of TV
2012/05/08
49
http://mpeg.chiariglione.org/
MPEG for the past, present and future of TV
2012/05/08
50