ITU Workshop on “Monitoring and Benchmarking
of QoS and QoE of Multimedia Services in
Mobile Networks”
(Buenos Aires, Argentina, 24-25 July 2014)
Overview of ITU-T Study Group 12
Recommendations on QoS and QoE
Joachim Pomy
Consultant@joachimpomy.de
OPTICOM, Germany
SG 12 Recommendations
E.420-E.479, E.800-E.859
G.100-series, except G.160-, G.180- and
G.190-series
G.1000-series
I.350-series (including
Y.1501/G.820/I.351), I.371, I.378, I.381
P-series, except P.900-series
Y.1220-, Y.1530-, Y.1540-, Y.1560-series
http://www.itu.int/ITU-T/recommendations/index_sg.aspx?sg=12
Buenos Aires – Argentina – 24 – 25 July 2014
2
Work Programme (1/8)
Coordination of QoS/performance studies
Communications Quality of Service: A framework and definitions
Vocabulary for performance and quality of service
Mean Opinion Score (MOS) terminology
Transmission characteristics for wideband digital handset and
headset telephones - Inclusion of extended wideband terminals
Transmission characteristics for cordless and moblie digital
terminals
Transmission characteristics for wideband digital loudspeaking and
hands-free telephony terminals - Inclusion of extended wideband
terminals
Technical requirements and test methods for the universal wired
headset or headphone interface of digital mobile terminals
Narrowband hands-free communication in motor vehicles
Wideband hands-free communication in motor vehicles
Buenos Aires – Argentina – 24 – 25 July 2014
3
P.800.1 Relation between some MOS qualifiers
Buenos Aires – Argentina – 24 – 25 July 2014
4
Work Programme (2/8)
User interface requirements for automotive applications
Subsystem requirements for automotive speech services
Artificial ears
Calculation of loudness ratings for telephone sets
Use of head and torso simulator (HATS) for hands-free and
handset terminal testing
Bone conduction testing setup
Multiple test positions setup
Setups and testing techniques for terminal performance
measurements with background noise
Artificial voices
Speech processing devices for acoustic enhancement
Transmission characteristics and speech quality parameters of
hands-free terminals
Buenos Aires – Argentina – 24 – 25 July 2014
5
Typical development cycle for a car speakerphone hands-free
system and applicability of clauses of Rec. P.1100
Typically HFT development for a new car
Research end Pre-development end
Dev. B sample end
A sample
Dev. C sample end
B sample
Dev. D sample end
C sample
PPAP
SOP
D sample
Application of the different clauses of this Recommendation
Part 8 digital interface
Part 8 digital interface
Parts 11 and 12 HFT and SRWTEP parameters
Part 13 subjective car-to-car test
Part 10 microphone parameters
What is covered by the various parts of the test Recommendation?
Part 7: Covers test arrangements used within this Recommendation.
Part 8: Covers non-mandatory objective measurements using a digital interface.
Part 9: Covers test signals used within this Recommendation.
Part 10: Covers non-mandatory objective measurements of the microphone or/and microphone system.
Part 11: Covers mandatory objective measurements of the hands-free terminal (HFT).
Part 12: Covers non-mandatory objective measurements of short-range wireless transmission enabled phone (SRWTEP).
Part 13: Covers non-mandatory subjective evaluation of the hands-free terminal (HFT).
P.1110(09)_F6-1
Buenos Aires – Argentina – 24 – 25 July 2014
6
Work Programme (3/8)
Test signals for use in telephonometry
Objective test methods for speech communication systems using
complex test signals
Methods for determining the ‘Objective Quality Number’ as overall
quality value for terminals
Applicability of psycho-acoustic models to HATS based setups for
the benefit of terminal testing
Setup signals for terminal with background noise
Subjective quality evaluation of telephone services based on
spoken dialogue systems
Subjective performance of active signal processing devices
Cultural/language/nationality dependence of subjective quality
Subjective evaluation of generic sound activity detectors
Multidimensional scaling quality assessment
Subjective assessment methods for music quality of narrowband
and wideband telephony
Buenos Aires – Argentina – 24 – 25 July 2014
7
Rating scales used in ITU-T P.806 subjective testing methodology
P.MULTI Perceptual Quality Scales
PQ Scales
S-FLT
S-RUF
S-LFC
S-HFC
B-LVL
B-VAR
Description
Scale Descriptors
Slow-varying Degradation in the
fluttering, babbling,
Speech Signal
discontinuous
Fast-varying Degradation in the
rough, raspy, harsh
Speech Signal
Degradation of Low-Frequency
dull, muffled, smothered
Coloration in the Speech Signal
Degradation of High-frequency
small, distant, thin
Coloration in the Speech Signal
Degradation due to the Level of
hissing, rushing, roaring
Background Noise
Degradation due to the Variability bubbling, intermittent,
of the Background Noise
variable
Overall Scales
Description
LOUD
Overall Loudness in the Speech Signal + Background Noise
OVRL
Overall Quality in the Speech Signal + Background Noise
Buenos Aires – Argentina – 24 – 25 July 2014
8
Work Programme (4/8)
Test methodology for web browsing and browser based
applications
Subjective usability evaluation of speech-based or
multimodal interactive services
Q.7/12 defines test plans for tests of speech coders, and
reports the test results and analysis
E-model extension for non-handset user interfaces
E-model extension for speech processing devices
E-model update for conversational quality features
Mixed-band and beyond-wideband E-model
E-model for monitoring
Operational quality estimator
Perceptual approaches for multi-dimensional analysis
Perceptual objective noise reduction
Buenos Aires – Argentina – 24 – 25 July 2014
9
Test setup for Rec. P.1501 Quality of web browsing
Buenos Aires – Argentina – 24 – 25 July 2014
10
Work Programme (5/8)
Perceptual Objective Prediction of Speech Intelligibility
No-reference models for quality prediction
Subjective method for simulated conversation tests
addressing audio and audiovisual call quality
Effect of delays on the telemeeting quality
Use of auditory and visual cues for high-quality
telemeetings
Overall telemeeting quality value (long term goal)
Quality implications and requirements for telemeeting and
conferencing services
Spatial audio meetings quality evaluation
Frequent updates of Appendices
Quality of Experience aspects of Multi Connection
End-to-end QoE aspects of tandemmed speech processing
devices
Buenos Aires – Argentina – 24 – 25 July 2014
11
Rec. P.1301 Decision tree for Subjective quality evaluation
of audio and audiovisual multiparty telemeetings
Start
Communication
mode including text / graphics
(e. g., web
conferencing) ?
Yes
ITU-T P.1301 Annex B
No
Audio
Communication
mode?
Video
ITU-T P.1301 Annex C
Audiovisual
Test mode
= audio
Test mode
= video
Audio
Video
Test
mode?
Audio-visual
Type of
quality?
Conversational
Continue with
Flowchart B
Type of
quality?
Noninteractive
Continue with
Flowchart C
Conversational
Continue with
Flowchart D
Type of
quality?
Noninteractive
Continue with
Flowchart E
Conversational
Special test
method required
Noninteractive
Continue with
Flowchart F
P.1301(12)_FA
Buenos Aires – Argentina – 24 – 25 July 2014
12
Work Programme (6/8)
End-to-end performance for managed voice over LTE networks
This Recommendation describes the key aspects impacting endto-end performance of managed voice applications over LTE
networks.
Definitions and associated measurement methods for user-centric
parameters for call handling in cellular mobile voice service
Mobile QoS
Supplement XX to ITU-T E.800-series Recommendations
(Guidelines on Regulatory Aspects of QoS)
End-user multimedia QoS categories
Network model for evaluating multimedia transmission
performance over Internet Protocol
Opinion model for video and audio streaming applications
QoE of Web-browsing
QoE requirements for multicast video streaming services
QoE requirements for telepresence services
Guidance for the use of P.1201 and P.1202 in operational contexts
Buenos Aires – Argentina – 24 – 25 July 2014
13
Rec. G.1010 – Model for user-centric QoS categories
Error
tolerant
Conversational
voice and video
Voice/video
messaging
Streaming audio
and video
Fax
Error
intolerant
Command/control
(e.g. Telnet,
interactive games)
Transactions
(e.g. E-commerce,
WWW browsing,
Email access)
Messaging,
Downloads
(e.g. FTP, still image)
Background
(e.g. Usenet)
Interactive
(delay <<1 s)
Responsive
(delay ~2 s)
Timely
(delay ~10 s)
Non-critical
(delay >>10 s)
T1213060-02
Buenos Aires – Argentina – 24 – 25 July 2014
14
Work Programme (7/8)
QoE-diagnostic information from P.120X.Y type models on technical causes
of QoE impairments
Integral index of quality for general service monitoring (per user-session);
KQI definitions
Media-session QoE model (temporal pooling, long sequences; UDP- and
TCP-based streaming)
Parametric non-intrusive assessment of audiovisual progressive download,
considering stalling/re-buffering & quality integration
Parametric non-intrusive assessment of TCP-based multimedia streaming
quality, considering adaptive streaming
In-service non-intrusive measurement device – Voice service
measurements
Analysis and interpretation of INMD voice-service measurements
Conformance testing for voice over IP transmission quality assessment
models
Conversational model
Framework for invoking diagnostic functions
Voice Service Diagnosis Framework
Buenos Aires – Argentina – 24 – 25 July 2014
15
Rec. G.1029 - Voice Service Diagnosis Framework
Terminals
Framework for invoking diagnostic functions
(Work Item 1)
QoE/QoS
Decision
parameters
Decision
process
Invocation
network
QoE/QoS
Diagnostic
parameters
Diagnostic
parameter to
root-cause
mapping
QoE/QoS
Root-cause
analysis
parameters
Framework for mapping diagnostic parameters to
root-cause parameters (Work Item 2)
Remedial
action
QoE/QoS
Root-cause
analaysis
application
Buenos Aires – Argentina – 24 – 25 July 2014
16
Work Programme (8/8)
Technical cause analysis
Buffer models for development of client performance
metrics
Hand-over performance among multiple access networks
Internet protocol data communication service – IP packet
transfer and availability performance parameters
Network performance objectives for IP-based services
Measurements in IP networks for inter-domain performance
assessment
Ethernet frame transfer and availability performance
Ethernet service activation test methodology
Home network performance parameters
Proposed Appendices for Y.1566 : QoS mapping and
interworking between Ethernet, IP and MPLS
Buenos Aires – Argentina – 24 – 25 July 2014
17
Cannot find your Topic ?
Work in ITU-T is contribution driven !
Next Meeting of Study Group 12
2 – 11 September 2014 in
Cartagena de Indias, Colombia (tbc)
Deadline for Contributions
20 August 2014
http://www.itu.int/net/ITU-T/ddp/Default.aspx?groupid=T13-SG12
Buenos Aires – Argentina – 24 – 25 July 2014
18
Any questions ?
Joachim Pomy
consultant@joachimpomy.de
Buenos Aires – Argentina – 24 – 25 July 2014
19