TOWARDS QUALITY AND QOS ASSESSMENT FOR
VOLTE IN ETSI
TC INT chairman Giulio Maggiore - IMS World Forum 30th April 2014
© ETSI 2013. All rights reserved
Outline
ETSI Technical Committee TC INT for
Core Network and Interoperability
Testing
ETSI INT technical work specifications
Quality and QOS Assessment for
VoLTE in ETSI
What’s next?
2
© ETSI 2013. All rights reserved
CORE NETWORK AND INTEROPERABILITY
TESTING (INT)
3
© ETSI 2013. All rights reserved
TC INT (Core Network and Interoperability
Testing)
Member of EP E2NA
• In 2012 TC INT participated in the creation of the ETSI E2NA project
contributing with its expertise in network testing as described in the
E2NA ToRs approved by the Board#89.
TC INT change of ToRs and name
• According to E2NA ToRs approved by Board#89
• The modification of the ToRs in order to align its activities and
contributions with EP E2NA.
• Change of name with Core Network and Interoperability Testing (INT)
to better reflect the areas of activity.
• No change of acronym that will remain INT
4
ETSI/BOARD(13)94_041
ETSI INT SPECIFICATIONS
Interoperability , Performance , Security and Conformance
5
© ETSI 2013. All rights reserved
Production of test specifications
Production of test specifications:
• (11) Interoperability
• (4) IMS/NGN Performance Benchmark
• (1) Security
Production of IMS Conformance:
• (32) Test Suite Structure and Test Purposes
(TSS&TP)
• (24) Abstract Test Suite (ATS)
• (24) Protocol Information Conformance
Statement (PICS)
• (24) Protocol Implementation eXtra Information
for Testing (PIXIT)
INT full work programme
© ETSI 2013. All rights reserved
6
QUALITY AND QOS ASSESSMENT ETSI
APPROACH
7
© ETSI 2013. All rights reserved
Quality: the new approach
►
Quality is the most important factor in client choice and churn.
►
In order to offer the best user experience it’s necessary to shift in the quality monitoring: from a
“Network Centric” point of view to “Customer Centric”
Network Centric
Customer Centric
APN
Core
APN
Access
Core
APN
Access
8
Customer Centric monitoring requires new KPIs
►
Moving from the network to the user centric point of view a new set of KPIs are required able
to analyze the experience of the single user
►
In order to monitor the e2e network chain the KPIs have to be technology/access
Technical KPI
independent Service KPI
•
Immediate overview of the service
•
bricks of service KPI
monitor specific network elements
Critical Indicators
Localized
Access layer
Operative System
Device
Network section
Core
9
9
2013 – VOLTE QOS ASSESSMENT
TECHNOLOGY EVALUATION EVENT
10
© ETSI 2013. All rights reserved
VoLTE QoS Assessment event overview
Scope: Technology evaluation Plugtests aiming at
validating the different methodologies for end to end QoS
Assessment for VoLTE described in ETSI TS 103 189 over a
deployed LTE/IMS network.
Dates: 18-20 November 2013
Venue: ETSI, Sophia Antipolis, FR
Web: http://www.etsi.org/news-events/events/698-volte-qos-assessment-2013
Participants:
Supporter:
11
© ETSI 2013. All rights reserved
VoLTE QoS Assessment event scope
End to end tests performed with several different UEs over
a deployed LTE / IMS test network.
4 Test Setups
• Acoustic to acoustic
• Acoustic to electrical
• Electrical to acoustic
• Electrical to electrical
Voice quality tests based on instrumental assessment of
speech samples performed on different combinations of
the acoustic and electrical interfaces which allows
evaluating the different methodologies and setups.
Parallel network performance evaluation
12
© ETSI 2013. All rights reserved
2013 – VoLTE QoS Assessment Technology
Evaluation event – test description TS 103 189
Sub-parameters delivered by the particular tests:
End-to-end Tests based on Instrumental Assessment
of Speech Samples (POLQA model deployed):
•
•
•
•
Mean Opinion Score (MOS)-Listening Quality Objective
Delay variation over time
Difference in loudness level
MOS stability
End-to-end Tests based on Instrumental Assessment
of Video Samples (PEVQ model deployed):
•
•
•
•
•
•
DMOS (Difference Mean Opinion Score)
Delay and Quality variation over time
Jerkiness and Blockiness
Blur, Chrominance and Luminance
Temporal Distortion
Frozen and Skipped Frames
2013 – VoLTE QoS Assessment Technology
Evaluation event – test description TS 103 189
Sub-parameters delivered by the particular tests:
Other end-to-end tests in the voice channel:
•
•
•
•
•
End-to-end frequency response
Overall loudness rating
End-to-end delay
Quality of echo cancellation
Double talk performance
End-to-end Tests based on functional parameters:
• Telephony Call Setup Time
• Streaming Service Access Time
Network Performance Parameters
• UNI-to-UNI delay over time
• UNI-to-UNI jitter
• UNI-to-UNI packet loss
2013 – VoLTE QoS Assessment Technology
Evaluation event – test description
General description of tests defined in ETSI TS 103 189:
QoS tests based on the ETSI INT specifications as
described in the latest versions of:
•
•
•
•
ETSI TS 186 011-1 (Test purposes for IMS NNI Interoperability)
ETSI TS 186 011-2 (Test descriptions for IMS NNI Interoperability)
ETSI TS 102 901 (IMS NNI interoperability test descriptions for RCS)
ETSI TS 103 029 (IMS & EPC Interoperability).
QoS tests divided into two parts:
• First one focusing on the instrumental assessment of speech
and video quality by using real speech and video samples
• Second one dealing with functional parameters of voice and
video over LTE connections and other parameters in
the end-to-end voice channel
• On the top of it, selected network performance parameters
listed in the TS 103 189
15
© ETSI 2013. All rights reserved
16
VoLTE QoS Assessment – Setup 1
Acoustic-to-Acoustic
© ETSI 2013. All rights reserved
ITU-T Specifications for Quality
measurements
Recommendation ITU-T J.247: "Objective perceptual multimedia video quality measurement
in the presence of a full reference".
Recommendation ITU-T P.50: "Artificial voices".
Recommendation ITU-T P.57: "Artificial ears".
Recommendation ITU-T P.58: "Head and torso simulator for telephonometry".
Recommendation ITU-T P.64: "Determination of sensitivity/frequency characteristics of local
telephone systems".
Recommendation ITU-T P.79: "Calculation of loudness ratings for telephone sets".
Recommendation ITU-T P.340: "Transmission characteristics and speech quality parameters of
hands-free terminals".
Recommendation ITU-T P.380: "Electro-acoustic measurements on headsets".
Recommendation ITU-T P.501: "Test signals for use in telephonometry".
Recommendation ITU-T P.505: "One-view visualization of speech quality measurement
results".
Recommendation ITU-T P.502: "Objective test methods for speech communication systems
using complex test signals".
Recommendation ITU-T P.581: "Use of head and torso simulator (HATS) for hands-free
terminal testing".
Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology".
VoLTE QoS Assessment – Setup 2
Acoustic to Electrical
18
© ETSI 2013. All rights reserved
VoLTE QoS Assessment – Setup 3
Electrical to Acoustic
19
© ETSI 2013. All rights reserved
VoLTE QoS Assessment – Setup 4
Electrical to Electrical
20
© ETSI 2013. All rights reserved
MAIN OUTCOME OF THE VOLTE QOS
ASSESSMENT AND NEXT STEPS…
21
© ETSI 2013. All rights reserved
Main event outcome
Validation of 76% of the test cases in TS
103 189
A number of gaps and potential
improvements in TS 103 189 have been
identified
A new work item has been open for the
update of TS 103 189 with the new setup,
acoustic-to-electrical identified during the
event
TS 103 189 has been published and is
available on the ETSI portal
22
© ETSI 2013. All rights reserved
Further steps with regard to ETSI TS 103 189
Verification of remaining voice quality test cases
• tbd
Verification of video quality test cases (RTP-based
video streaming):
• Test houses willing to do so are needed
Extension of video quality test cases
towards MPEG-DASH
• ITU-T SG12 Question 14 currently running a WI called
P.NATS
23
Maputo - Mozambique - 14 - 16 April 2014
2013 – VoLTE QoS Assessment Technology
Evaluation event – test results
24
© ETSI 2013. All rights reserved
2013 – VoLTE QoS Assessment Technology
Evaluation event – test results
25
© ETSI 2013. All rights reserved
Verification of video quality test cases (RTP-based
video streaming)
What we are looking for?
What are the parameter we want to measure?
What are the tools expected?
Parameters
Covered by
TS 183.189
26
© ETSI 2012. All rights reserved
test tools
focused on
video quality
Prototipization
and validation
Customers perception KPIs
VoLTE
Browsing
Non Accessibility
Session Setup Failure
Setup time
Cut off Ratio
Packet loss rate
Latency
Jitter
MOS Audio/Video Quality
Number of HTTP
Sessions
Average Duration of
HTTP Sessions
Average HTTP
Throughput
Peak HTTP
Throughput
HTTP Setup Time
HTTP Dropped
Connections
HTTP Sessions
Unavailability
Retransmission Rate
Packet loss rate
Latency
Jitter
Loading Page
27
© ETSI 2012. All rights reserved
Gaming
Number of Gaming
Sessions
Gaming Sessions
Average Duration
Gaming Average
Throughput (DL & UL)
Gaming Overall
responsiviness [ms]
Retransmission Rate
Packet loss rate
Latency
Jitter
Number of streaming sessions
Average duration of streaming
sessions
Average Streaming Throughput
Peak Streaming Throughput
Setup Streaming Time
Streaming Dropped Sessions
Streaming
(HTTP YouTube sessions included) Streaming Unavailability
Re-Buffering Time during Streaming
Streaming Synchronization
Packet loss rate
Latency
Jitter
MOS Audio/Video Quality
Networks KPIs/KQIs
4/9/2015
© Zhilabs 2014, Confidential Information
28
Next steps
Get comments on TS 103 189
Extend TS 103 189 addiming
more Customer Perception
KPIs
Get proposal from Video Test
Houses
Next event will be launched
when VoLTE Native Handset
and PEVQ tools will be
available for testing
29
Contact Details:
Giulio Maggiore
Telecom Italia
ETSI TC INT Chairman
Giulio.Maggiore@telecomitalia.it
Thank you!
30
© ETSI 2013. All rights reserved
TC INT TERMS OF REFERENCE
31
© ETSI 2013. All rights reserved
TC INT (Core Network and Interoperability Testing)
Terms of Reference
The scope of INT includes:
Development of Core Network test
specifications (interoperability,
conformance, performance, security)
based on 3GPP and E2NA specifications.
To initiate and supervise interoperability
events (such as Plugtests) related to Core
Networks as well as other events
(workshops and seminars)
To coordinates interoperability efforts with
other organisations GSMA, OMA, IMS/NGN
Forum, IMTC, etc.
Endorsement of test specifications
from/to other SDOs e.g. ITU-T.
32
© ETSI 2013. All rights reserved
ETSI INT SPECIFICATIONS
Interoperability , Performance , Security and Conformance
33
© ETSI 2013. All rights reserved
TC INT Interoperability Specification Work
R6
TS 186 011
V 1.1.1
2007
R1
34
ETSI/BOARD(13)94_041
R7
TS 186 011
V 2.1.1
Late R7
TS 186 011
V 2.2.1
2008
2009
R2
Late R2
R8
TS 186 011
V 3.1.1
TS 102 901
V 1.1.1
2010
R9
R10
TS 186 011
V 4.1.1
TS 102 901
V 2.1.1
TS 186 011
V 5.0.2
TS 102 901
V 5.0.6
TS 103 129
TS 103 129
V2.0.4
2011
2012 2013
E2NA
Conformance Test Specifications
IMS Basic Call Specifications and Supplementary Services;
Diameter Conformance testing for Rx interface TS 101 580;
Diameter Conformance testing for Gx interface TS 101 606;
Diameter Conformance testing for S6a and S9 interface just
started;
35
ETSI/BOARD(13)94_041
Interoperability Test Specifications
IMS NNI Interworking Test
Specifications TS 186 011
RCS TS 102 901
IMS&EPC interoperability
TS 103 129
36
ETSI/BOARD(13)94_041
IMS Interworking Test
Specifications
IMS Interconnection Test
Specifications
INTEROPERABILITY EVENTS
37
© ETSI 2013. All rights reserved
Interoperability events
2007 – The first event examined interconnection aspects of
3GPP IMS Release 6, including basic call on the Mw interface.
2008 - The second event had a wider scope that included the
testing of 3GPP IMS Release 7 interworking, roaming, border
control, and integration of application servers executing
selected Multimedia Telephony supplementary services.
2009 - The third ETSI IMS event assessed the interoperability
as well as conformance of IMS core networks (composed of
P/I/S-CSCF, IBCF, AS, DNS and HSS) which are implemented on
the basis of ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0)
(3GPP late release 7).
2012 – RCS&VoLTE interoperability event in cooperation with
GSMA and MSF held in Kranj (Slovenia) and Beijng (China)
38
ETSI/BOARD(13)94_041
Press Coverage: RCS VoLTE Test Event
39
ETSI/BOARD(13)94_041
2007 and 2008 IMS interoperability events
• The first event examined interconnection
aspects of 3GPP IMS Release 6, including basic
call on the Mw interface.
• The second event had a wider scope that
included the testing of 3GPP IMS Release 7
interworking, roaming, border control, and
integration of application servers executing
selected Multimedia Telephony supplementary
services.
40
© ETSI 2013. All rights reserved
2009 - IMS interoperability events
The third ETSI IMS event assessed the
interoperability as well as conformance of IMS core
networks (composed of P/I/S-CSCF, IBCF, AS, DNS
and HSS) which are implemented on the basis of
ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0)
(3GPP late release 7).
41
© ETSI 2013. All rights reserved
2012 – RCS & VoLTE interoperability event
The forth event endorsed and tested a number of
GSMA PRDs and validated the Testing Scenarios
developed by MSF and ETSI to facilitate multivendor deployment of LTE./EPC/IMS technology for
RCS and VoLTE services.
http://www.etsi.org/news-events/pastevents/642-2012-rcs-volte-interop-event
42
© ETSI 2013. All rights reserved