International Telecommunication Union
Towards a Standardised
Perceptual Quality Metric
for Multimedia
David Hands
Perceptual Engineering Group
BT
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Overview
ITU-T
o Emerging video services
o Need for objective quality metrics
o VQEG Multimedia ad-hoc group
o Timeline
o Sample metric
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Emerging video services
ITU-T
o New services in Asia, Europe and N. America
o Infrastructure ready for video/multimedia (codecs,
networks, devices)
o Services
• Streaming video, VoD, Video telephony, TVoDSL/IPTV, mobile TV
o Service Providers
• HomeChoice, Neuf, Fastweb, SBC, Verizon, NTT, Vodafone, …
o Networks
• NGNs (e.g. BT), increasing b/w, fiber to the home, wireless networks
(802.11a/b/g/n, Wi-Max), 3G mobile
• high quality services offered on cost-effective networks providing
multi-service access (any service to any device from anywhere)
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Need for objective quality metrics
ITU-T
o Measurement of user experience is essential to the
industry
• defining quality thresholds of media prior to launch
• selection of equipment and packaging of content
• attracting and maintaining customer base
o Traditional quality measures are inadequate
• PSNR
• Network QoS
o Requirement is for objective perceptual metrics
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Objective perceptual quality metrics
ITU-T
o Perceptual quality is
• content-dependent
• codec dependent (e.g. codec settings, decoder error handling)
o Performance testing
• codec evaluation
• quality profiling
(e.g. content x bit-rate x frame rate)
• network dimensioning
o In-service monitoring
• E2E quality monitoring
• SLA negotiation and policing
• Fault identification
QoS Control
and optimisation
Client 1
Client 2
Video
Server
Network probe
Client n
Fault diagnosis
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
QoS feedback
ITU-T
QoS Control
and optimisation
Client 1
Client 2
Video
Server
Network probe
Client n
Fault diagnosis
QoS feedback
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
VQEG Multimedia ad-hoc group
ITU-T
o VQEG FR standards
• ITU-T Rec. J.144 and ITU-R BT. 1683
• specific to digital TV (MPEG-2)
• for video containing compression errors
o VQEG MM
•
•
•
•
Goal is to evaluate objective metrics
FR, RR, NR models
3 stages: video only, audio only, audio-video
set ambitious testing requirements
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Scope of VQEG MM Stage 1
ITU-T
o
o
o
o
o
o
Video test
Image sizes: CIF, QCIF, VGA
Display: high performance LCD
Viewing distance: variable within 6-10H / 6-8H / 6H
Codecs: WM9, RV, MPEG-4, H.263, H.264
Range of frame rates (30fps - 2.5 fps)
o Error conditions:
•
•
•
•
compression errors
compression x transmission errors
post-processing
live network conditions
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Test Process
ITU-T
Content
Test Labs
VideoConf
Movie
Sports
Music
Ads
Cartoon
News
Home Video
Prop Lab A
Image size
HRC
...
Prop Lab N
Ind Lab A
Single-stimulus
ACR-hrr
Excellent
Good
Fair
Poor
Bad
...
Ind Lab N
Objective
Model
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Results
capture
and model
evaluation
Timeline
ITU-T
Sep ‘05
Agree
Testplan
Mar ‘06
Prepare
Content
Submit
Models
June ‘06
Run
Tests
Obtain
Subj / Obj
Ratings
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Sep ‘06
Final
Report
Multimedia Perceptual Models
ITU-T
o Industry
•
•
•
•
•
•
•
•
o Research Centres
BT
Genista
KDDI
NTT
Opticom
Psytechnics
SwissQual
TDF
• NTIA
• Toyama University
• Yonsei University
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
A FR Model for Multimedia
ITU-T
o BT have developed a suite of perceptual
quality solutions (FR, RR, NR)
o FR model part of new standards
o Model tested for
• codecs (MPEG-2, MPEG-4, H.263, H.264)
• image sizes (D1 - QCIF)
• frame rates (30, 25, 15, 10, 5)
• variable content
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Model Performance
ITU-T
MOS
5
4
3
2
1
1
2
3
4
5
MOSp
Correlation between subjective and objective MOS for 412 test sequences. R
= 0.92
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Model Performance
o Various content
ITU-T o SIF, QCIF
o 25 - 5 fps
o MPEG-4, H.263
o 320kbit/s - 16kbit/s
o With and without transmission errors
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva
Demonstration
ITU-T
Link to demo
ITU- T VICA Workshop
22-23 July 2005, ITU Headquarter, Geneva