ITU Workshop on
“Benchmarking QoS Evaluation of Multimedia
Networks”
(Ouagadougou, Burkina Faso, 18 July 2013)
Content and presentation of
Recommendation E.MQoS
Joachim Pomy, SG 12 Rapporteur
Consultant, Opticom GmbH
Consultant@joachimpomy.de
Ouagadougou, Burkina Faso, 18 July 2013
1
Draft New Rec. E.MQoS
E-series:
Overall network operation, telephone service,
service operation and human factors
E.800-series:
Quality of telecommunication services:
concepts, models, objectives and dependability
planning
Title:
QoS Aspects for Popular Services in Mobile
Networks
Ouagadougou, Burkina Faso, 18 July 2013
2
Scope of E.MQoS (1/2)
Assessment of Quality of Service
Summary of basics of QoS, always seen from the user's perspective
Differences to Quality of Experience (QoE)
Guidance to assure QoS assessments conducted in a meaningful way
Definition of Quality of Service parameters and their computation
QoS parameters and their computation for popular services
abstract definition = generic description of parameters
abstract equation
user and technical trigger points
Harmonized definitions are prerequisites for the comparison of QoS
measurements and measurement results
Typical procedures for Quality of Service measurement equipment
Measurement procedures needed to perform the measurements of
QoS parameters
Ouagadougou, Burkina Faso, 18 July 2013
3
Scope of E.MQoS (2/2)
Requirements for QoS measurement equipment
Defines minimum requirements of QoS measurement equipment
values and trigger points needed to compute the QoS parameter
allow performing measurements in a reliable and reproducible way
Definition of typical measurement profiles
Specifies typical measurement profiles
required to enable benchmarking of different mobile networks
both within and outside national boundaries.
Post processing and statistical methods
Describes procedures for statistical calculations
for QoS measurement of mobile networks using probing systems.
Network based Quality of Service measurements
E2E QoS measurements inside the network
without direct access to the end point terminal.
Ouagadougou, Burkina Faso, 18 July 2013
4
Popular Services
Services are not standardized
Focus on popular services
Widely used in the market
Test results are meaningfull to customers
Examples
Web radio
E-mail clients
Web portals
Ouagadougou, Burkina Faso, 18 July 2013
5
QoS aspects of Mobile
Network
Availability
Layer 1
Network
Accessibility
Layer 2
circuit
switched
Service
Accessibility
packet
switched
Service
Integrity
Service
Retainability
E-Mail
File
Transfer
MMS
Mobile
Broadcast
Ping
PoC
SMS
Streaming
Telephony
Video
Telephony
Layer 3
Layer 4
Web
Browsing
Ouagadougou, Burkina Faso, 18 July 2013
6
Main structure of E.MQoS
6
7
Assessment of Quality of Service
Definition of Quality of Service parameters
and their computation
8 Typical procedures for Quality of Service
measurement equipment
9 Requirements for Quality of Service
measurement equipment
10 Definition of typical measurement profiles
11 Post processing and statistical methods
12 Network based Quality of Service
measurements
Appendices (informative)
Ouagadougou, Burkina Faso, 18 July 2013
7
6
Assessment of Quality of Service
6.1
6.2
6.3
6.4
6.5
QoS Background
QoS Assessment Process
Basic Settings for QoS
Assessments
Service Independent QoS
Criteria
Service dependent QoS
Criteria
Ouagadougou, Burkina Faso, 18 July 2013
8
7
Definition of Quality of Service
parameters and their computation
7.1
7.2
7.3
7.4
QoS Parameter Basics
Service independent QoS
Parameters
Direct Services QoS
Parameters
Store-and-forward (S&F)
Services QoS Parameters
Ouagadougou, Burkina Faso, 18 July 2013
9
7.2 Service independent QoS
Parameters
7.2.1
7.2.2
7.2.3
7.2.4
7.2.5
7.2.6
7.2.7
7.2.8
7.2.9
7.2.10
7.2.11
Radio Network Unavailability [%]
Network Non Accessibility [%]
Attach Failure Ratio [%]
Attach Setup Time [s]
PDP Context Activation Failure Ratio [%]
PDP Context Activation Time [s]
PDP Context Cut-off Ratio [%]
Data Call Access Failure Ratio [%]
Data Call Access Time [s]
DNS Host Name Resolution Failure Ratio [%]
DNS Host Name Resolution Time [s]
Ouagadougou, Burkina Faso, 18 July 2013
10
7.3 Direct Services QoS Parameters
7.3.1
7.3.2
7.3.3
7.3.4
7.3.5
7.3.6
7.3.7
7.3.8
7.3.9
7.3.10
7.3.11
7.3.12
7.3.14
7.3.13
File Transfer (FTP)
Mobile Broadcast
Ping
Push to Talk over Cellular (PoC)
Streaming Video
Telephony
Video Telephony
Web Browsing (HTTP)
Web Radio
WLAN service provisioning with HTTP based
authentication
Wireless Application Protocol (WAP)
IMS Multimedia Telephony
Group Call
E-mail
Ouagadougou, Burkina Faso, 18 July 2013
11
7.4 Store-and-forward (S&F) Services
QoS Parameters
7.4.1
7.4.2
7.4.3
7.4.4
Generic Store-and-forward
Parameters
E-mail
Multimedia Messaging Service
(MMS)
Short Message Service (SMS)
Short Data Service (SDS)
Ouagadougou, Burkina Faso, 18 July 2013
12
8
Typical procedures for Quality of
Service measurement equipment
8.1
8.2
8.3
8.4
8.5
8.6
Aim of measurement
Classification of services
General aspects for all types
of services
Telephony measurements
Store-and-forward services
measurement
Data measurements
Ouagadougou, Burkina Faso, 18 July 2013
13
8.4 Telephony measurements
8.4.1
General aspects
8.4.2
Speech telephony
8.4.4
Group Call
8.4.3
Video telephony
Ouagadougou, Burkina Faso, 18 July 2013
14
8.5 Store-and-forward services
measurement
8.5.1
General aspects
8.5.2
SMS measurements
8.5.3
MMS
8.5.4
E-Mail
8.5.5
SDS
Ouagadougou, Burkina Faso, 18 July 2013
15
8.6 Data measurements
8.6.1
8.6.2
8.6.3
8.6.4
8.6.5
8.6.6
8.6.7
Common aspects
FTP
HTTP
E-mail
WAP
Streaming Video
Media Download
Ouagadougou, Burkina Faso, 18 July 2013
16
9
Requirements for Quality of
Service measurement equipment
9.1
9.2
9.3
9.5
9.4
Overview
General requirements
Fixed QoS Test-equipment
(FQT)
Mobile based measurement
equipment
Mobile QoS Test-equipment
(MQT)
Ouagadougou, Burkina Faso, 18 July 2013
17
10 Definition of typical
measurement profiles
10.1
Measurement profiles
10.1.3
10.1.1
10.1.2
Usage Profiles for Data
Sessions
Classification of
measurement environments
Service profiles
Ouagadougou, Burkina Faso, 18 July 2013
18
11 Post processing and
statistical methods
11.1
11.2
11.3
11.4
11.5
11.6
Important measurement data
types in mobile
communications
Distributions and moments
Visualization techniques
Time series modelling
Data aggregation
Assessment of performance
indices
Ouagadougou, Burkina Faso, 18 July 2013
19
12 Network based QoS
measurements
12.1
Network Measurement Basics
12.2
Measuring QoS Parameters in
the Network
12.3
Comparing Network and Endpoint Test Measurements
Ouagadougou, Burkina Faso, 18 July 2013
20
Appendices (informative)
I
II
III
IV
V
Examples for measuring trigger points
Streaming explanations
Push to Talk over Cellular Information
QoS parameter export
Which parameters have an impact on the effect of
blocking ?
VI Reference {SMS | SDS}
VII Content integrity checking
VIII Transfer times versus used data rate and content size
IX Examples of statistical calculations
X
The concept of QoE reporting
XI Examples of Network Based QoS Measurements
XII 3GPP SA5 "UE Management"
Ouagadougou, Burkina Faso, 18 July 2013
21
Confidence Intervalls for Different Sample Sizes (1)
The following examples show the effect of
different sample sizes in a measurement
campaign. It is also based on the PearsonClopper formulas for the calculation of confidence
intervals. Therefore, the examples are valid in a
generic way and even for small sample sizes. For
higher sample numbers, the calculation of
confidence intervals based on the approximation
of a normal distribution can be applied.
Three different graphs are depicted: Sample sizes in the
range:
between 100 and 1 100 samples;
between 1 100 and 2 100 samples; and
between 1 000 and 11 000 samples.
Ouagadougou, Burkina Faso, 18 July 2013
22
Confidence Intervalls for Different Sample Sizes (2)
30
Width of confidence interval for fixed sample size (Pearson-Clopper)
10
15
20
100 Samples
300 Samples
500 Samples
700 Samples
900 Samples
1100 Samples
0
x xx
x xx
x xxx xx
x xxx xxx xxx
xxx
xx
xx
xx
+
+++
+
++
++
+
+
+
+
xx
+
+
+
+
+
+
+
+
x
x
+ + + o o o o o oo o o o oo o o o o o + + + + +
+
x
+
x
o
o
x + + + o o o o o ox x x x x x x x x x x x x x x x x x x x ox o o o o o + + + x
x
+++
x
++
++
+
o
x
+
+
+
+
x
o
+
+
+
x + o o x x x + +++ +
+ +++ + x x x o o + x
++ ++ x x o + x
x ++ ++
x+
x + + o ox x +
x xo +
++ +
o
+
x
x ox o + x
++
x ++
x + ox +
++
xo
o
+
x
x+
+
+
ox +
ox
+
+x
x
+
+
o
o
x
x
+
+
xx
5
Width of confidence interval in percent
25
o
x
+
o
x
+
0
xx
20
40
60
80
100
Estimated rate in percent
Ouagadougou, Burkina Faso, 18 July 2013
23
Confidence Intervalls for Different Sample Sizes (3)
2
3
4
5
xx xxx xxx x
xx x
xx
xx
x
+ +++ +++ + x x x
+
x
+
+
++ + x x
x x + ++
x + + o o o o o o ooo o o o o o o + + x
+
x
o
o + x
xxx
x ++o o o x x x x x x x + x x x x x x x o o o ++ x
+
+
+
+
+
+
x + o x x ++ +
+ +++ x x o + x
++ x x o o + x
x ++o o x x +++ +
+ + x o+ x
x
+
o
x+ x +
++ x o +
x + o ox x + +
+ x xo + x
+
+ xo
x +o x +
+ +x
x
+
+ xo+
o
x+
+ xo x
x+
o
+
+x+
+x
o
1100
Samples
xo +
+ ox x
x
++
x 1300 Samples
++
o
ox
xx
1500 Samples
x
+
++
++
ox
o 1700 Samples
ox
+x
x 1900 Samples
x+
+
+
ox
ox
2100 Samples
+
+
+
0
1
Width of confidence interval in percent
6
7
Width of confidence interval for fixed sample size (Pearson-Clopper)
x
+
o
x
+
0
x
+
o
x
+
20
40
60
80
100
Estimated rate in percent
Ouagadougou, Burkina Faso, 18 July 2013
24
Confidence Intervalls for Different Sample Sizes (4)
10
Width of confidence interval for fixed sample size (Pearson-Clopper)
4
6
1000 Samples
3000 Samples
5000 Samples
7000 Samples
9000 Samples
11000 Samples
xxx x
xx xxx xxx xxx xxx xx
x xx
xx
xx
++
+++
++
+
+
+
+++ ++
+
+
xx
+
+
+ ++
x
++ +
+
+
x
000
00
00
+
0
0
0
0
+
0
0
x +++ 0 0 0 0 0 0 x x x x x x x x x x x 0 0 0 0 0 0 +++ x x
x x x x x x x0 00 + x
x +
x xx
0 0x x x x x +
x x x 0 0 0 ++ x
x + + 0 0 0x 0x x +
+ + + ++ ++ + +++ + ++ ++ + + + + +
++
x x x0 + x
x +++ ++
+++
x + + 0 0x +
x+
+ ++
x 0x 0 +
x+
x+
x 0x + x
0x 0
++
x+
+
0
+
+ 0x +
x
++
+ 0x
0
x
+
+
2
x
xx
0
Width of confidence interval in percent
8
o
x
+
o
x
+
xx
xx
x
0x
+
0
x
0x
+
20
40
60
80
100
Estimated rate in percent
Ouagadougou, Burkina Faso, 18 July 2013
25
Schedule for draft new Rec. E.MQoS
31 August 2013
Final timeline for submitting comments directly to the editors
7 November 2013
Submission as Contribution to SG 12
20 November 2013
Deadline for Contributions submitted to TSB
12 December 2013
Planned Consent of SG12 Plenary
January 2014
Start of Alternate Approval Procedure (AAP)
March 2014
Pre-Publication
??? 2014
Publication
Ouagadougou, Burkina Faso, 18 July 2013
26
Any questions ?
Joachim Pomy
Telecommunications & Int'l Standards
Germany
Tel.: +49 177 78 71958
Email: consultant@joachimpomy.de
Ouagadougou, Burkina Faso, 18 July 2013
27