ITU Workshop on
“Quality of Service of Regulatory and Operational Issues”
(Dubai, UAE 2-3 November 2014)
Quality of Service (QoS), Quality of Experience (QoE) and Performance
Joachim Pomy
Consultant@joachimpomy.de
OPTICOM, Germany
Dubai - UAE - 2-3 November 2014
1
Where it All Begins:
Real Communication Situation
Dubai - UAE - 2-3 November 2014
2
... and where End-to-End Quality comes to Play:
Employing a Telecommunication System
... can you
hear me ?
Dubai - UAE - 2-3 November 2014
... I want to
speak now !
3
Definitions start here: ITU-T Rec. E.800
 Network Performance (NP)
 Pre-requisite to Quality of Service (QoS)
 Not directly visible to the user
 Quality of Service (QoS)
 Performance of the Service offered to the User
 Some QoS Aspects directly perceivable, some indirectly
Network Performance





Charging Performance
Provisioning Performance
Administration Performance
Availability Performance
Transmission Performance
Dubai - UAE - 2-3 November 2014
Quality of Service




Service Support Performance
Service Operability Performance
Serveability
Service Security Performance
4
Four Viewpoints of QoS
 Consistent Approach to QoS
 Well-defined and Relevant (e.g. Customer-affecting)
 Used to Plan and Deploy Networks
 Includes Monitoring Service Quality
 ITU-T Rec. G.1000 defines four Viewpoints of QoS
 Customer's QoS Rrequirements
 Service provider's offerings of QoS (or targeted QoS)
 QoS achieved or delivered
 Customer survey ratings of QoS
 Ideally there would be 1:1 Correspondence between
Delivered QoS and Perceived QoS
Dubai - UAE - 2-3 November 2014
5
4 Viewpoints of QoS
CUSTOMER
SERVICE
PROVIDER
Customer’s
QoS
Requirements
QoS Offered
By
Provider
QoS
Perceived
By Customer
QoS
Achieved by
Provider
Dubai - UAE - 2-3 November 2014
T1213040-02
6
ITU-T Rec. G.101
 The Transmission Plan
 Fundamental principles of transmission planning
 A good transmission plan is set up in order to deliver to users
signals that are at a desirable level and free from
objectionable amounts of delay, echo and distortion
 Has to take into account transmission parameters and
impairments, different network configurations and elements
 Specific transmission plans have to be set up in order to take
care of specific transmission impairments and conditions for




traditional narrow-band telephone networks
mobile networks
packet switched networks
multimedia applications
Dubai - UAE - 2-3 November 2014
7
Traditional Transmission Planning
International Switching
Centres (ISCs)
b
a
t
t
a
National system
b
International chain
National system
T1204G.101_FI.1
Exchange
ISC that carries international transit traffic
a, b
Virtual International Connecting Points
Dubai - UAE - 2-3 November 2014
8
Transmission Planning Today
 ITU-T Rec. G.108: Transmission Planning with
the E-Model
 Traditional transmission planning
methodologies no longer flexible enough to
account for new factors
Dubai - UAE - 2-3 November 2014
9
Transmission Planning Challenges - 1
Multinational networks require planning which
takes into account regional differences in loss plan
requirements and inter-network transmission plans
Due to liberalization of the telecommunication
markets (e.g. in Europe) there are no longer laid
down ranges of values for transmission parameters
by regulation
The changing scenario in the public network
operator domain is impacting transmission
performance
Dubai - UAE - 2-3 November 2014
10
Transmission Planning Challenges - 2
G.108 is applicable to the use of new technology
within the networks, including wireless (cordless or
mobile), IP transmission etc.
G.108 provides planning methods and contains
necessary information and tools which will enable
the planner to design the network transmission plan
Guidelines and planning examples are based on the
use of the E-Model
Dubai - UAE - 2-3 November 2014
11
E-Model - ITU-T Rec. G.107
 Computational model for use in transmission planning
 Assessing the combined effects of variations in several
transmission parameters that affect conversational
quality of 3.1 kHz handset telephony
 Covers also packet loss
 For many combinations of high importance to
transmission planners, the E-model can be used with
confidence
 Caution must be exercised when using the E-model for
some conditions
Dubai - UAE - 2-3 November 2014
12
Reference connection of the E-model
Receive side
Send side
OLR
SLR
RLR
0 dBr point
Weighted Echo
Path Loss WEPL
Ds-Factor
Round-Trip
Delay Tr
Room
Noise Ps
Coding / Decoding
Circuit Noise Nc
referred to 0 dBr
Equipment Impairment Factor Ie
Packet-Loss Robustness Factor Bpl
Packet-Loss Probability Ppl
Dr-Factor
Room
Noise Pr
Sidetone Masking
Rating STMR
Listener Sidetone
Rating LSTR
(LSTR =
STMR + Dr)
Mean one-way Delay T
Absolute Delay Ta
Quantizing Distortion qdu
Expectation Factor A
Dubai - UAE - 2-3 November 2014
Talker Echo
Loudness Rating
TELR
G/107_F01
13
Effects of Talker Echo
in the Presence of Delay
100
E-Model Rating R
90
no Talker Echo
TELR=65 dB
TELR=55 dB
TELR=45 dB
TELR=35 dB
TELR=25 dB
80
70
60
50
0
50
100
150
200
250
300
350
400
450
500
Mouth-to-Ear-Delay / ms
Dubai - UAE - 2-3 November 2014
14
Voice Quality Continuum
Categories of Communication Quality
in Terms of Users' Satisfaction Classes
Dubai - UAE - 2-3 November 2014
15
Example with Delay as Impairment
Dubai - UAE - 2-3 November 2014
16
QoE Definition
 ITU-T Rec. G.100 / P.10 defines
Quality of Experience (QoE): The overall
acceptability of an application or service, as
perceived subjectively by the end-user.
NOTE 1 – Quality of experience includes the
complete end-to-end system effects (client,
terminal, network, services infrastructure, etc.).
NOTE 2 – Overall acceptability may be influenced by
user expectations and context.
Dubai - UAE - 2-3 November 2014
17
QoE Implications
 QoE includes „everything“
Many aspects out of control of Operators
Includes Terminal Aspects
Conext and Environment of the User
 Proper QoS and NP
Technical pre-requisites
For achieving desired QoE
Dubai - UAE - 2-3 November 2014
18
Users‘ Perception of Speech Quality
Sound Quality &
Naturalness
Intellegibility
Speech
Charakteristic
Individual
Perception
Speech
(Transmission)
Quality
Listening & Talking
Efforts
Environmental
Conditions
Doubletalk
Capability
...
Dubai - UAE - 2-3 November 2014
Conversational
Efforts
Network
Conditions
Expectation
Backgroundnoise
Transmission
...
19
Motivation for Multimedia Quality - 1
 Quality as perceived by the User
A Promotional Factor for the Market
 User compares Quality of New
Telecommunication Services
With Quality experienced in the Past
With other Telecommunication Service offers
With Quality experienced for Entertainment
Services
Dubai - UAE - 2-3 November 2014
20
Motivation for Multimedia Quality
 Individual Quality Threshold
Users try new Service only few times ( ~ 3x … 5x )
If Quality below Indivdual Threshold Users give up
e.g. Download of a Website takes too long
 User remembers this experience
 Will try a few times and conclude this as Static Effect:
"This website is not useable - let's try the Offer of the
Competitor…"
Dubai - UAE - 2-3 November 2014
21
Diffusion, Transmission Quality
and Expectation for an Innovation
 Diffusion Theory generally accepted
for describing Consumer Behaviour
on the Introduction of an Innovation
or New Service
 Number of Users develops in
S-shaped Curve
 5 Classes of Users:





(1) Innovators
(2) Early Adaptors
(3) Early Majority
(4) Late Majority
(5) Laggards
 Trade-off between Transmission
Quality and New Functionality
Dubai - UAE - 2-3 November 2014
22
Changes in Users' Behaviour - 1
 Users tend to be much more reluctant to accept lower
quality
 This is quoted frequently
 True for some sorts of social calls
 Definitively NOT true for sensible business calls
 Does it help network operators when defining QoS for their
network ?
 High quality has to be provided when demanded by business
customers or other sensible clients
 But the distribution of quality acceptance over time and areas cannot
be matched with the occurrence of impairments in the network
 Not really useful for designing networks
Dubai - UAE - 2-3 November 2014
23
23
Changes in Users' Behaviour - 2
 Users switch between different communication
devices
Wireline, wireless, PC, PDA etc
Depending on place, task, purpose
 And depending on QUALITY
Dubai - UAE - 2-3 November 2014
24
24
Key Parameters affecting MM Quality






Media Distortion
End-to-End Delay
Echo Effects
Information Loss
Background Noise Distortion
Loss of Synchronization between Media
Streams
Dubai - UAE - 2-3 November 2014
25
Example: Lip Sync
Dubai - UAE - 2-3 November 2014
26
Impairments in packet networks
 Distinction between Effects
that occur in the Network and
Mechanisms in the Terminals that are affected
 Terminals can be used to correct for the Effects
in the Network
 Remaining Issues are:
End-to-End Delay is increased when compensating
for other Effects
Loss of Information can be Concealed but Not
Recovered
Dubai - UAE - 2-3 November 2014
27
Impairments in packet networks
Dubai - UAE - 2-3 November 2014
28
QoS Layers in Mobile
 QoS model for mobile has four layers.
 First layer is the Network Availability
 defines QoS rather from the viewpoint of the service provider than
the service user
 Second layer is the Network Access
 from user's point of view basic requirement for all the other QoS
aspects and parameters
 Third layer contains other QoS aspects
 Service Access, Service Integrity & Service Retainability
 Different services are located in the fourth layer
 Their outcome are the QoS parameters as perceived by the user
Dubai - UAE - 2-3 November 2014
29
QoS aspects of Mobile
Network
Availability
Layer 1
Network
Accessibility
Layer 2
circuit
switched
Service
Accessibility
packet
switched
Service
Integrity
Service
Retainability
E-Mail
File
Transfer
MMS
Mobile
Broadcast
Ping
PoC
SMS
Streaming
Telephony
Video
Telephony
Layer 3
Layer 4
Web
Browsing
Dubai - UAE - 2-3 November 2014
30
POLQA™ - Rec. P.863
 The limitations of existing standards that are now addressed by POLQA
 CDMA
 Chinese 3G TD-SCDMA
 POLQA offers immediate, strong support for testing of new wideband
4G/LTE networks delivering HD-quality voice services
 Tests carried out during the POLQA evaluation included future
technologies such as




Unified Communications
Next Gen Networks
4G/LTE
HD Voice, i.e. "wide-band" and "super-wide-band"
 See POLQA: The Next Generation in Voice Quality Testing
http://www.polqa.info
Dubai - UAE - 2-3 November 2014
31
Dubai - UAE - 2-3 November 2014
32
Performance Validation
•
•
The ITU has validated POLQA on:
•
47000 file pairs across
•
64 subjective experiments
Languages included in the POLQA validation:
American English and British
English
•
German
•
Swiss German
•
Chinese (Mandarin),
•
Italian,
•
Czech,
•
Japanese,
•
Dutch,
•
Swedish
•
French,
•
Dubai - UAE - 2-3 November 2014
33
POLQA Introduction - (c) OPTICOM GmbH 2010
33
Confidence Intervalls for Different Sample Sizes (1)
 Effect of different sample sizes in a measurement campaign
 based on the Pearson-Clopper formulas for calculation of confidence
intervals
 valid in a generic way and even for small sample sizes
 for higher sample numbers, the calculation of confidence intervals based
on the approximation of a normal distribution can be applied
 Three different graphs are depicted: Sample sizes in the range:
 between 100 and 1 100 samples;
 between 1 100 and 2 100 samples; and
 between 1 000 and 11 000 samples.
Dubai - UAE - 2-3 November 2014
34
Confidence Intervalls for Different Sample Sizes (2)
30
Width of confidence interval for fixed sample size (Pearson-Clopper)
10
15
20
100 Samples
300 Samples
500 Samples
700 Samples
900 Samples
1100 Samples
0
x xx
x xx
x xxx xx
x xxx xxx xxx
xxx
xx
xx
xx
+
+++
+
++
++
+
+
+
+
xx
+
+
+
+
+
+
+
+
x
x
+ + + o o o o o oo o o o oo o o o o o + + + + +
+
x
+
x
o
o
x + + + o o o o o ox x x x x x x x x x x x x x x x x x x x ox o o o o o + + + x
x
+++
x
++
++
+
o
x
+
+
+
+
x
o
+
+
+
x + o o x x x + +++ +
+ +++ + x x x o o + x
++ ++ x x o + x
x ++ ++
x+
x + + o ox x +
x xo +
++ +
o
+
x
x ox o + x
++
x ++
x + ox +
++
xo
o
+
x
x+
+
+
ox +
ox
+
+x
x
+
+
o
o
x
x
+
+
xx
5
Width of confidence interval in percent
25
o
x
+
o
x
+
xx
0
20
40
60
80
100
Estimated rate in percent
Dubai - UAE - 2-3 November 2014
35
Confidence Intervalls for Different Sample Sizes (3)
2
3
4
5
xx xxx xxx x
xx x
xx
xx
x
+ +++ +++ + x x x
+
x
+
+
++ + x x
x x + ++
x + + o o o o o o ooo o o o o o o + + x
+
x
o
o + x
xxx
x ++o o o x x x x x x x + x x x x x x x o o o ++ x
+
+
+
+
+
+
x + o x x ++ +
+ +++ x x o + x
++ x x o o + x
x ++o o x x +++ +
+ + x o+ x
x
+
o
x+ x +
++ x o +
x + o ox x + +
+ x xo + x
+
+ xo
x +o x +
+ +x
x
+
+ xo+
o
x+
+ xo x
x+
o
+
+x+
+x
o
1100
Samples
xo +
+ ox x
x
++
x 1300 Samples
++
o
ox
xx
1500 Samples
x
+
++
++
ox
o 1700 Samples
ox
+x
x 1900 Samples
x+
+
+
ox
ox
2100 Samples
+
+
+
0
1
Width of confidence interval in percent
6
7
Width of confidence interval for fixed sample size (Pearson-Clopper)
x
+
o
x
+
0
x
+
o
x
+
20
40
60
80
100
Estimated rate in percent
Dubai - UAE - 2-3 November 2014
36
Confidence Intervalls for Different Sample Sizes (4)
10
Width of confidence interval for fixed sample size (Pearson-Clopper)
4
6
1000 Samples
3000 Samples
5000 Samples
7000 Samples
9000 Samples
11000 Samples
xxx x
xx xxx xxx xxx xxx xx
x xx
xx
xx
++
+++
++
+
+
+
+++ ++
+
+
xx
+
+
+ ++
x
++ +
+
+
x
000
00
00
+
0
0
0
0
+
0
0
x +++ 0 0 0 0 0 0 x x x x x x x x x x x 0 0 0 0 0 0 +++ x x
x x x x x x x0 00 + x
x +
x xx
0 0x x x x x +
x x x 0 0 0 ++ x
x + + 0 0 0x 0x x +
+ + + ++ ++ + +++ + ++ ++ + + + + +
++
x x x0 + x
x +++ ++
+++
x + + 0 0x +
x+
+ ++
x 0x 0 +
x+
x+
x 0x + x
0x 0
++
x+
+
0
+
+ 0x +
x
++
+ 0x
0
x
+
+
2
x
xx
0
Width of confidence interval in percent
8
o
x
+
o
x
+
xx
xx
x
0x
+
0
x
0x
+
20
40
60
80
100
Estimated rate in percent
Dubai - UAE - 2-3 November 2014
37
KPIs based on Network Counters
 Vendor specific = network internal KPIs
 different strategies
 how to count network events
 which events are included in which counter(s)
 Requires knowledge of specific system
 specialists with detailed system knowledge
 testing the counters
 documentation may be faulty
 approach to counter change with system update
 Mobile operators struggling with this
 most operator live in a multi vendor environment
 counters from different vendors cannot be directly compared
 requires continous attention and a strategy
Dubai - UAE - 2-3 November 2014
38
KPIs from Users' Perspective = KQIs
 Key Quality Indicators (KQIs) = external
indicators
can be assessed in the Field
 For Monitoring, Regulation etc.
a subset can be selected
 applicable across all vendors & operators
 not limited to mobile, but also good for broadband
Dubai - UAE - 2-3 November 2014
39
KPIs versus KQIs
 Sometimes confused
 KPIs = internal indicators





part of network performance
based on network counters
essential for operation, maintenance, business model
could be reported, audited etc.
however, meaningless when out of context
 KQIs = external indicators






basis for QoS assessment as perceived by the user
vendor independant
operator independant
ideal to compare different operators on a statistical basis
cannot be reported from the system itself
requires some kind of field testing, drive, walk etc.
Dubai - UAE - 2-3 November 2014
40
Any questions
?
Contact:
Consultant@joachimpomy.de
Dubai - UAE - 2-3 November 2014
41