ITU Workshop on
“Benchmarking QoS Evaluation of Multimedia
Networks”
(Ouagadougou, Burkina Faso, 18 July 2013)
Overview of
ITU-T Study Group 12 Activities
Joachim Pomy, SG 12 Rapporteur
Consultant, Opticom GmbH
Consultant@joachimpomy.de
Ouagadougou, Burkina Faso, 18 July 2013
1
Where it All Begins:
Real Communication Situation
Ouagadougou, Burkina Faso, 18 July 2013
2
... and where Technology comes to Play:
Employing a Telecommunication System
Ouagadougou, Burkina Faso, 18 July 2013
3
SG 12 Key History Events (1)
Established in 1957
Separate SG 16 for Transmission Planning
Old SG 12 just did subjective testing & telephone standards
At CCITT Plenary Assembly in 1984, these SGs were merged
to form the present-day SG 12
During same time SG 12 had a Laboratory WP
Subjective & objective testing for loudness of telephones
In the CCITT Laboratory in Geneva - closed in 1988
Since 1986 Speech Quality Experts Group (SQEG)
To better provide the quality requirements and subjective
testing for speech coding algorithms standardized by SG 18
(later SG 15, now SG 16)
Key role in the standardization of all of the G.711-G.729
speech and audio codecs, and still exists as Q7/12.
Ouagadougou, Burkina Faso, 18 July 2013
4
SG 12 Key History Events (2)
Since 1997 Video Quality Experts Group (VQEG)
With a nucleus of subjective and objective video quality expertise from
SG12
SG12 still maintains close contact with VQEG
Since the early 1990s work in SG 12 has expanded
Into many new technology areas
ATM
VoIP
Multimedia
Digital wireless
IP QoS classes
Ethernet
IPTV, etc.
Network Performance
Resource Management.
Since 2008 SG12 was given responsibility for
Operational aspects of telecommunication network service quality
Responsibility for the QoS Development Group (QSDG).
Ouagadougou, Burkina Faso, 18 July 2013
5
SG 12 Mandate
Performance, QoS and QoE
Responsible for Recommendations on performance, quality
of service (QoS) and quality of experience (QoE) for
full spectrum of terminals, networks and services
ranging from speech over fixed circuit-based networks
to multimedia applications over networks that are mobile and
packet based
Included are
operational aspects of performance, QoS and QoE
end-to-end quality aspects of interoperability
development of multimedia quality assessment methodologies,
both subjective and objective.
SG 12 is the Lead SG on
quality of service and quality of experience
driver distraction and voice aspects of car communications
http://www.itu.int/en/ITU-T/studygroups/20132016/12/Pages/default.aspx
Ouagadougou, Burkina Faso, 18 July 2013
6
SG 12 Recommendations
E.420-E.479, E.800-E.859
G.100-series, except G.160-, G.180- and
G.190-series
G.1000-series
I.350-series (including
Y.1501/G.820/I.351), I.371, I.378, I.381
P-series, except P.900-series
Y.1220-, Y.1530-, Y.1540-, Y.1560-series
http://www.itu.int/ITU-T/recommendations/index_sg.aspx?sg=12
Ouagadougou, Burkina Faso, 18 July 2013
7
SG 12 Leading Team
Chairman
Kwame Baah-Acheamfuor (Ghana)
Vice Chairmen
Paul Barrett (United States)
Vinvent Barriac (France)
Gamal Amin Elsayed (Sudan)
Hyung-Soo Kim (Republic of Korea)
Al Morton (United States)
Qi Feng (China)
José Guadalupe Rojas Ramírez (Mexico)
Akira Takahashi (Japan)
Hassan Talib (Morocco)
TSB Support
Hiroshi Ota, Counsellor
Sarah Scott, Assistant
Ouagadougou, Burkina Faso, 18 July 2013
8
Working Parties
WP 1 Terminals and multimedia subjective
assessment
CHAIR: Lars Birger Nielsen (Denmark)
VICE CHAIR: Gunilla Berndtsson (Sweden)
WP 2 Objective models and tools for
multimedia quality
CHAIR: Paul Barrett (United States)
VICE CHAIR: Vincent Barriac (France)
WP 3 Multimedia QoS and QoE
CHAIR: Paul Coverdale (China)
VICE CHAIR: Akira Takahashi (Japan)
Ouagadougou, Burkina Faso, 18 July 2013
9
Groups overseen by SG12
CarCom Focus Group
(closed in 2013)
Hans Gierlich, Chair
Driver Distraction Focus
Group (closed in 2013)
Scott Pennock, Chair
QoS Development Group
Yvonne Umutoni, Chair
Regional Group for Africa
Gamal Amin Elsayed, Chair
Ouagadougou, Burkina Faso, 18 July 2013
10
CarCom Focus Group
In car communication
Quality parameters & testing methods
Interaction of car hands free systems with radio
Requirements on sub-system level
Requirements and testing procedures for superwideband and fullband systems
Interaction with other audio components and
systems in the car
Special requirements/testing procedures for speech
recognition systems in cars
Quality models: what and how can be applied for
the car environment?
Ouagadougou, Burkina Faso, 18 July 2013
11
QoS Development Group
Encourages wider participation in international QoS activities
Identifies and develop performance monitoring activities for
QoS
Identifies procedures & practices for inclusion in the ITU-T
Rec.s
Disseminates information relating to QoS techniques &
procedures
Development of coordinated approach to study of QoS
Pursues other activities which may improve international QoS
Ouagadougou, Burkina Faso, 18 July 2013
12
Work Programme (1/8)
Coordination of QoS/performance studies
Communications Quality of Service: A framework and definitions
Vocabulary for performance and quality of service
Mean Opinion Score (MOS) terminology
Transmission characteristics for wideband digital handset and
headset telephones - Inclusion of extended wideband terminals
Transmission characteristics for cordless and moblie digital
terminals
Transmission characteristics for wideband digital loudspeaking and
hands-free telephony terminals - Inclusion of extended wideband
terminals
Technical requirements and test methods for the universal wired
headset or headphone interface of digital mobile terminals
Narrowband hands-free communication in motor vehicles
Wideband hands-free communication in motor vehicles
Ouagadougou, Burkina Faso, 18 July 2013
13
Work Programme (2/8)
User interface requirements for automotive applications
Subsystem requirements for automotive speech services
Artificial ears
Calculation of loudness ratings for telephone sets
Use of head and torso simulator (HATS) for hands-free and
handset terminal testing
Bone conduction testing setup
Multiple test positions setup
Setups and testing techniques for terminal performance
measurements with background noise
Artificial voices
Speech processing devices for acoustic enhancement
Transmission characteristics and speech quality parameters of
hands-free terminals
Ouagadougou, Burkina Faso, 18 July 2013
14
Work Programme (3/8)
Test signals for use in telephonometry
Objective test methods for speech communication systems using
complex test signals
Methods for determining the ‘Objective Quality Number’ as overall
quality value for terminals
Applicability of psycho-acoustic models to HATS based setups for
the benefit of terminal testing
Setup signals for terminal with background noise
Subjective quality evaluation of telephone services based on
spoken dialogue systems
Subjective performance of active signal processing devices
Cultural/language/nationality dependence of subjective quality
Subjective evaluation of generic sound activity detectors
Multidimensional scaling quality assessment
Subjective assessment methods for music quality of narrowband
and wideband telephony
Ouagadougou, Burkina Faso, 18 July 2013
15
Work Programme (4/8)
Test methodology for web browsing and browser based
applications
Subjective usability evaluation of speech-based or
multimodal interactive services
Q.7/12 defines test plans for tests of speech coders, and
reports the test results and analysis
E-model extension for non-handset user interfaces
E-model extension for speech processing devices
E-model update for conversational quality features
Mixed-band and beyond-wideband E-model
E-model for monitoring
Operational quality estimator
Perceptual approaches for multi-dimensional analysis
Perceptual objective noise reduction
Ouagadougou, Burkina Faso, 18 July 2013
16
Work Programme (5/8)
Perceptual Objective Prediction of Speech Intelligibility
No-reference models for quality prediction
Subjective method for simulated conversation tests
addressing audio and audiovisual call quality
Effect of delays on the telemeeting quality
Use of auditory and visual cues for high-quality
telemeetings
Overall telemeeting quality value (long term goal)
Quality implications and requirements for telemeeting and
conferencing services
Spatial audio meetings quality evaluation
Frequent updates of Appendices
Quality of Experience aspects of Multi Connection
End-to-end QoE aspects of tandemmed speech processing
devices
Ouagadougou, Burkina Faso, 18 July 2013
17
Work Programme (6/8)
End-to-end performance for managed voice over LTE networks
This Recommendation describes the key aspects impacting endto-end performance of managed voice applications over LTE
networks.
Definitions and associated measurement methods for user-centric
parameters for call handling in cellular mobile voice service
Mobile QoS
Supplement XX to ITU-T E.800-series Recommendations
(Guidelines on Regulatory Aspects of QoS)
End-user multimedia QoS categories
Network model for evaluating multimedia transmission
performance over Internet Protocol
Opinion model for video and audio streaming applications
QoE of Web-browsing
QoE requirements for multicast video streaming services
QoE requirements for telepresence services
Guidance for the use of P.1201 and P.1202 in operational contexts
Ouagadougou, Burkina Faso, 18 July 2013
18
Work Programme (7/8)
QoE-diagnostic information from P.120X.Y type models on technical causes
of QoE impairments
Integral index of quality for general service monitoring (per user-session);
KQI definitions
Media-session QoE model (temporal pooling, long sequences; UDP- and
TCP-based streaming)
Parametric non-intrusive assessment of audiovisual progressive download,
considering stalling/re-buffering & quality integration
Parametric non-intrusive assessment of TCP-based multimedia streaming
quality, considering adaptive streaming
In-service non-intrusive measurement device – Voice service
measurements
Analysis and interpretation of INMD voice-service measurements
Conformance testing for voice over IP transmission quality assessment
models
Conversational model
Framework for invoking diagnostic functions
Voice Service Diagnosis Framework
Ouagadougou, Burkina Faso, 18 July 2013
19
Work Programme (8/8)
Technical cause analysis
Buffer models for development of client performance
metrics
Hand-over performance among multiple access networks
Internet protocol data communication service – IP packet
transfer and availability performance parameters
Network performance objectives for IP-based services
Measurements in IP networks for inter-domain performance
assessment
Ethernet frame transfer and availability performance
Ethernet service activation test methodology
Home network performance parameters
Proposed Appendices for Y.1566 : QoS mapping and
interworking between Ethernet, IP and MPLS
Ouagadougou, Burkina Faso, 18 July 2013
20
Cannot find your Topic ?
Work in ITU-T is contribution driven !
Next Meeting of Study Group 12
3 – 12 December 2013 in Geneva
Deadline for Contributions
20 November 2013
http://www.itu.int/net/ITU-T/ddp/Default.aspx?groupid=T13-SG12
Ouagadougou, Burkina Faso, 18 July 2013
21
Any questions ?
Joachim Pomy
Telecommunications & Int'l Standards
Germany
Tel.: +49 177 78 71958
Email: consultant@joachimpomy.de
Ouagadougou, Burkina Faso, 18 July 2013
22