ITU Workshop on
“Workshop on Practical measurement of QoS/QoE
Parameters for Regulatory Compliance”
(Cotonou, Benin, 16-17 July 2012)
Equipment for Testing Mobile QoS
Joachim Pomy, SG 12 Rapporteur
Consultant, Opticom GmbH
Consultant@joachimpomy.de
Cotonou, Benin, 16-17 July 2012
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Measurement of QoS in Mobile
Measured from the perspective of an
expert
Assessing the network to see if it delivers
the quality that the network planner
targeted (cf. Rec. ITU-T G.1000)
Tools and Methods for Operation and
Maintenance
Protocol analysers are connected to BTSs,
BSCs, and MSCs for a period of time to check
for problems in the cellular network
Drive test (also walk tests) are used for
monitoring operationally achieved QoS
Measurement of QoS in Mobile
Drive tests allow the mobile network to be
tested through the use of a team of people
who take the role of users
Rate the QoS of the network
Not applicable to the entire network
Always a statistical sample
In Operation and Maintenance Centres (OMCs),
counters are used for various events
Providing the network operator with vendor specific
information on the state and quality of the network
Also, customer complaints are a vital source of
feedback on the QoS, and must not be ignored
End-to-end QoS Testing in Mobile
Minimum requirement for QoS
measurement equipment to ensure that
for all QoS parameters
trigger points are well defined
values are determined following wqell defined
procedures
Test-equipment fulfilling the specified
minimum requirements
Will allow performing the proposed
measurements
In a reliable and reproducible way
Parameters are based on Field
Measurements
Measurements from customers point of view
Full End-to-End perspective, taking into account needs
of testing
Assumed end customer can handle his equipment and
services to use
Operability is not evaluated at this time
For the purpose of measurement it is assumed that
service is not barred for any reason
routing is defined correctly without errors
target subscriber equipment is ready to process service
request
speech and video quality values measured
Only to be applied to calls ended successfully
In order to receive proper statistical analysis
Not yet any ITU Standards in Place
ETSI TS 102 250 part 4
Speech and multimedia Transmission
Quality (STQ); QoS aspects for popular
services in GSM and 3G networks; Part 4:
Requirements for Quality of Service
measurement equipment
globally recognized as "the
requirement" in the area
extremely compehensive
detailed instructions
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Test Scenarios
Test scenarios need to distinguish
the following principal user cases
User-to-user services (typically
telephony)
Store-and-forward services (e.g. SMS)
Information services (e.g. accessing the
internet or FTP download)
Push services (sender initiated transfer)
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Considerations on trigger points
It can be assumed that
any feasible test equipment will contain some kind of
communication terminal (UE) which may be a special
type (e.g. a Trace Phone)
or a standard UE
Also, it can be assumed that each such device
will provide information from different
communication layers
from Application Layer (close to the user interface)
down to lower layers, e.g. operating-system events,
TCP/IP layer, or Layer 3 signalling information, which is
used as trigger points for QOS PARAMETERS processing
Typically AT commands can be used in majority of cases
Action is typically triggered
by some emulated user action
causing some action on the air interface
Considerations on trigger points
Process of event propagation is deterministic
allowing for kind of mapping between layers
associated with communication and processing delay in each stage.
Choice of the layer for trigger point information retrieval
determines the view expressed in a QOS PARAMETER
choosing lower-level events ( Layer 3) gives more network-centric
view
events on higher levels to produce views more user-related
for same QoS PARAMETER source layer for events used as trigger
points should be the same
In benchmarking, for all all networks under test
use same type of UE, and QOS PARAMETERS
use trigger points from the same layer.
Changing the source layer for a given trigger point
changes in QOS PARAMETERS should be expected
calibration measurements required
to assess influence on QOS PARAMETERS both quantitatively and
qualitatively.
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Mobile QoS Test-equipment (MQT)
Geographical data collection during a
mobile measurement
Position, speed and heading
Taken from GPS whenever possible.
If no GPS available use other tools
e.g. navigation on a geo-referenced bitmap
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Telephony Voice
Means to generate necessary calls or to
answer automatically incoming calls
Received speech sample is measured
via the analogue output of the User Equipment
hardware setup to include necessary electrical
adaptation
Hardware setup of such a system is
essential for the correctness of the
measured QoS Parameters
Includes a UE specific part for the electrical
adaptation
audio output of handset to input of the soundcard
Video Telephony
Means to generate necessary calls or to
answer automatically incoming calls
Capability to provide the following
measurement methods.
Video Telephony using the User Equipments
application; or
Video Telephony using a Video Telephony
Stack on the PC.
MQT for FTP
FTP client to support the following
functions:
Active / Passive Mode
Common Firewall support
Downloading from subdirectories.
Uploading to subdirectories
Mobile based measurement equipment
Functionalities of the MQT can also
be realised on a single mobile phone
Different types of QoS tests can run
Test results can be logged for post
processing
Mobile based measurement equipment
may be controlled
by an application, remotely or
may be operated by a human
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Practical Hints
Which Mobile Phone to use ?
should
should
should
should
be
be
be
be
one
one
one
one
offered in the shop ?
used by new customers ?
hipe to young people ?
with good QoS ?
BUT
it MUST be a so-called Trace Mobile
ASK YOUR PHONE PROVIDER !!!
they are RARE !!!!
[Content]
ITRs: Setting the stage for
a connected world
Equipment for Testing Mobile QoS
Measurements
Test Scenarios
Trigger Points
Test Equipment
Testing for various Services
Practical Hints
POLQA - Voice Quality - Hot Topic
Hot Topic:
POLQA™ - Rec. P.863
PESQ
Acoustic measurements

Not easy
POLQA

Correct scoring with high background
noise


AMR vs EVRC codec comparison


Representative scoring of reference
signals


Effects of speech level in samples


Narrowband (300Hz -3400Hz)



Use SWB
Superwideband, SWB
(50Hz – 14000Hz)


Linear Frequency distortion sensitivity


Wideband (100Hz-7000Hz)
POLQA Introduction - (c) OPTICOM GmbH 2010
30
What
we
Perceive
…
In a subjective ACR experiment POLQA, PESQ and human beings perceive the
following distortions:
Factor
Human POLQA
PESQ
Level too high or too low
X
X
no
Strong linear filtering
X
X
no
Noise in the reference signal
X
X
no
High timbre in reference
X
X
no
Level variation
X
X
poor
SWB noise on NB/WB signal
X
X
no
POLQA Introduction - (c) OPTICOM GmbH 2010
31
Performance Validation
•
•
The ITU has validated POLQA on:
•
47000 file pairs across
•
64 subjective experiments
Languages included in the POLQA validation:
American English and British
English
•
German
•
Swiss German
•
Chinese (Mandarin),
•
Italian,
•
Czech,
•
Japanese,
•
Dutch,
•
Swedish
•
French,
•
POLQA Introduction - (c) OPTICOM GmbH 2010
32
Any questions
?
Contact:
Consultant@joachimpomy.de