International Telecommunication Union
Video and Image Coding
in Mobile Networks
Dr. Stephan Wenger
Principal Scientist, Nokia Research Center
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Structure of the Presentation
o
ITU-T
o
o
o
o
o
o
o
dates
Mobile phones – important application for video and
images
3GPP and 3GPP2 as the predominant standardization
organizations in the field
Multimedia-enabled Services over 3GPP/3GPP2
Technical requirements for image/video codecs in
mobile phones
Current and near-term deployment of image/video
codecs
•
In products (including recently announced products)
•
In standards (ratified, close to ratification)
•
Under investigation
Ongoing activities in 3GPP/3GPP2 related to
Image/video codecs
Our View of the ongoing work in
VCEG/JVT/MPEG/JPEG
Wish list to core image/video standardization groups
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
2
Image and Video Coding in
Mobile Phones
ITU-T
o
o
In 2004, mobile phones were an EUR 50 Billion+ equipment market
38% of all phones sold in 2004 included a camera
• up from 16% in 2003
• Mainstream/Commodity in 2005.
• Non-camera phones will be a niche application from 2006 on
• Only for users which are not allowed to use camera phones (security,
privacy)
• Perhaps Premium Pricing
o
Color displays standard today, QVGA resolution becomes
mainstream
• (close to) VGA resolutions in premium products already available
o
A wealth of applications now available – new trends started
•
•
•
•
•
dates
Multimedia Messaging (MMS)
Blogs
Video-Blogs
Streaming (point-to-point and multicast/broadcast)
Mobile video telephony (with some initial success)
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
3
UMTS Phones w/ Video Telephony
ITU-T
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
4
3rd Generation Partnership Project
o Established December 1998 as global successor to ETSI’s GSM
ITU-T
o Current “Organizational Partners”: ARIB, CCSA,ETSI, ATIS, TTA,
o
o
o
o
and TTC
Individual Members (264) as primary contributors
Other forms of “Partners” with limited right to contribute
Mission: Standardization of 3rd Generation Mobile System based
on evolved GSM core (and GSM maintenance; GPRS, EDGE)
Structure
Project Coordination Group (PCG)
Technical Specification Groups (GERAN, RAN, SA, CT)
TS Working Groups SA WG 4 (Codecs)
Ad-Hoc (e.g. Video)
o Individual members are encouraged to contribute to ITU-T’s
IMT 2000 process as well
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
5
3rd Generation Partnership Project 2
o Established early 1999 covering IMT2000 work in the
ITU-T
o
o
o
o
o
ANSI/TIA/EIA-41 world (North America, large parts of Asia)
Current “Organizational Partners”: ARIB, CCSA,TIA, TTA, and
TTC
Individual Members (80) as primary contributors
Other forms of “Partners” with limited right to contribute
Mission: Global specifications for the radio transmission
technologies (RTTs) supported by ANSI/TIA/EIA-41
Structure
Steering Committee (SC)
Technical Specification Groups (TSG-A/C/S/X)
TSG C (CDMA 2000) WG 1 (Application Services)
SWG 1.2 (Multimedia Services)
o Individual members are encouraged to contribute to ITU-T’s
IMT 2000 process as well
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
6
Multimedia-enabled Services over
3GPP/3GPP2
o
Circuit-Switched Video Telephony (3G.324M)
•
•
•
•
ITU-T
o
Based on ITU-T Rec. H.324 with mobile extensions / Annex C
Currently being rolled out by major operators
Almost (?) all UMTS phone models currently available support this service
No standards based solution in the 3GPP2 world
Video Codecs in 3GPP for packet based services (except MBMS):
• H.263 (mandatory), MPEG-4 SP (optional), H.264 baseline (optional)
o
o
Image Codecs in 3GPP: JPEG
Multimedia Messaging (MMS, MMD)
• Send Images and Video Clips to other users, SMS extension
• Widely deployed, supports a number of carefully selected codecs
• H.264 proposed for 3GPP2 Rel. A
o
Point-to-point Streaming (PSS, MSS) (RTSP-based)
• H.263 and either one of H.264 and MPEG-4 in 3GPP2 (operator selected)
o
Packet-Switched Conversational (Video Telephony, RTP based, forthcoming)
• No decision of video codecs yet in 3GPP2
o
Broadcast (MBMS, BCMCS, forthcoming)
• Multicast File Transfer (FLUTE based) and Multicast Streaming modes
• Packet based Forward Error Correction reduces application layer packet loss
rate
• Streaming: H.264 (and H.263) in 3GPP, no decision yet in 3GPP2
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
7
Requirements
o
Price: low BOM and high volume allows for (and to some extent requires)
•
•
•
•
ITU-T
Specialized HW accelerators (often tailored towards a single standard)
Highly optimized DSP software
Not necessarily extremely low complexity, but DSP/HW friendliness
Licensing cost – adding more (optional) codecs and tools becomes
increasingly difficult
• RFB was a good approach – consider a FREE codec (H.261+)
o
Consumer Electronic – SW updates undesirable or impossible
• Software has to be “bug free”, no deployments of experimental code
• Need for high level QA implies slow software deployment cycles
• Trend towards OS/Application architectures targeted for whole classes of mobile
phones
o
Complexity constraints
• BOM
• Battery Power, Size, Weight
• Not always is the more expensive device also the more powerful one
o
Network constraints
• Bandwidth, power vs. bandwidth when sending
• Link errors (packet erasures)
• Balance between source and channel coding
• no standardization group feels sufficiently responsible
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
8
Deployment in Products
ITU-T
o
o
Almost all phones sold today contain JPEG
Premium phones and Smartphones contain H.263 baseline (QCIF,
increasingly CIF) and MPEG-4 Simple Profile (increasingly w/ levels
above 1)
• Local playback, Local recording, Streaming, Video call in UMTS
phones
o
Some Premium phones contain H.264
• Nokia N6111 and N6265,
• Local playback, Streaming
In Korea: DMB phones include H.264 (non-3GPP broadcasting
service)
o Transport in files (ISO-style container formats), H.324/M, and
IP/UDP/RTP
o In case of RTP, Packetization according to Internet RTP Payload
RFCs (2429, 3016, 3984)
o
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
9
Supported in Standards
o
With 3GPP Rel. 6 (ratified earlier this year)
•
•
•
•
ITU-T
H.263 baseline is mandatory for all services except MBMS
H.263 profile 3 optional
MPEG-4 SP optional
H.264 baseline optional for all services except MBMS
• (some H.264 baseline tools excluded for services except PSC)
• MBMS: H.264 recommended, H.263 baseline optional, no MPEG-4
• JPEG in MMS
o
Not included in 3GPP/3GPP2 standards
• JPEG 2000
• MPEG-4 system layer mechanisms, profiles above SP
• Any form of scalable video
• H.264 baseline does contain forms of temporal scalability that could be
used by encoders
• H.263++ Annex V
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
10
Under Investigation
ITU-T
o Under Investigation as of mid 2005
• Graphics format, scene descriptions
• Some kind of streamable interactivity framework
• Discussions on Requirements
• Video Codec Performance Requirements
•
•
•
•
Idea: guaranteed Quality of Experience
Also complexity vs. quality tradeoff
Some call it Test Model work
Could become a Compliance issue (Encoder(!) Performance)
o Not under Investigation
• JPEG2000 and JPEG2000 mobile extensions (not even on the radar
screen)
• Scalability, SVC
• H.264 low complexity Profile
• (currently, the gain is too low to warrant changes in HW designs)
• H.265
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
11
Ongoing activities in 3GPP / 3GPP2
o 3GPP WI: Dynamic and Interactive Multimedia Scenes
ITU-T
• Discussion on requirements – no agreement yet
o 3GPP WI: Performance Requirements for Video Codecs
• Goal: Encoder/Decoder MUST meet a certain quality under
certain (error prone) conditions
• Is 3GPP SA4 Video ad-hoc the right forum for this work?
• Number of delegates, expertise, connection to academia?
• Why is there so little guidance from VCEG/MPEG/JVT?
• Up until ~2000, a lot of very good test model work has been
performed. Where is it now?
o 3GPP2 WI Packet Switched Video Telephony and WI on
BCMCS
• Codec selection (including work on quality metrics), codec
sub-profiling, packetization
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
12
Our View to ongoing work
NOTE: not all 3GPP/3GPP2 members would agree
o
ITU-T
SVC (Scalable Video Coding) – doubts in technical solutions, not really
needed very soon
• Spatial scalability underdeveloped. This tool perhaps of relevance for 3GPP
MBMS.
• Danger of the Layered Coding concept failing once more, because of
technical deficits
• Suggest pushing the delivery times in MPEG (perhaps by at least one year?)
o
H.264 Low Complexity Profile – confuses the market and adds little value
• Hardware accelerators for H.264 are well on their way, covering specifically
pixel based operations (loop filter, motion compensation)
• couldn’t use those accelerators with the new profile
• Motivation for the whole project is unclear.
• Very significant complexity reduction (50%) necessary,
• or close the project completely.
o
H.265: Time to start is not before 2007.
• When starting, start from scratch.
• Exclude legacy tools to support interlace, as the industry trend goes
towards progressive scan
o
JPEG 2000: No evidence of need for JPWL and JPIP
• Both would have to be “marketed” to the wireless industry first
o
VQEG & friends: Hurry! And get us a good quality metric for Multimedia
• Needed for codec shootouts and Performance Requirements WI
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
13
Wish List as of July 2005
NOTE: not all 3GPP/3GPP2 members would agree
o Difficult to absorb another image/video compression
ITU-T
standard/profile in the next few years
o VCEG/JVT/MPEG/JPEG have time to do good work, there is
no need to rush
o Accelerate work towards perceptual quality metrics for
the multimedia field
o More Test Model work needed targeting our applications
• Use of emerging perceptual quality metrics
• Error resilience work should be given higher priority
o Re-consider and Improve the Efficiency of the
Requirements process in the various groups
dates
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
14
International Telecommunication Union
Thank you
Questions? Comments?
ITU-T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva