TOWARDS QUALITY AND QOS ASSESSMENT FOR VOLTE IN ETSI TC INT chairman Giulio Maggiore - IMS World Forum 30th April 2014 © ETSI 2013. All rights reserved Outline ETSI Technical Committee TC INT for Core Network and Interoperability Testing ETSI INT technical work specifications Quality and QOS Assessment for VoLTE in ETSI What’s next? 2 © ETSI 2013. All rights reserved CORE NETWORK AND INTEROPERABILITY TESTING (INT) 3 © ETSI 2013. All rights reserved TC INT (Core Network and Interoperability Testing) Member of EP E2NA • In 2012 TC INT participated in the creation of the ETSI E2NA project contributing with its expertise in network testing as described in the E2NA ToRs approved by the Board#89. TC INT change of ToRs and name • According to E2NA ToRs approved by Board#89 • The modification of the ToRs in order to align its activities and contributions with EP E2NA. • Change of name with Core Network and Interoperability Testing (INT) to better reflect the areas of activity. • No change of acronym that will remain INT 4 ETSI/BOARD(13)94_041 ETSI INT SPECIFICATIONS Interoperability , Performance , Security and Conformance 5 © ETSI 2013. All rights reserved Production of test specifications Production of test specifications: • (11) Interoperability • (4) IMS/NGN Performance Benchmark • (1) Security Production of IMS Conformance: • (32) Test Suite Structure and Test Purposes (TSS&TP) • (24) Abstract Test Suite (ATS) • (24) Protocol Information Conformance Statement (PICS) • (24) Protocol Implementation eXtra Information for Testing (PIXIT) INT full work programme © ETSI 2013. All rights reserved 6 QUALITY AND QOS ASSESSMENT ETSI APPROACH 7 © ETSI 2013. All rights reserved Quality: the new approach ► Quality is the most important factor in client choice and churn. ► In order to offer the best user experience it’s necessary to shift in the quality monitoring: from a “Network Centric” point of view to “Customer Centric” Network Centric Customer Centric APN Core APN Access Core APN Access 8 Customer Centric monitoring requires new KPIs ► Moving from the network to the user centric point of view a new set of KPIs are required able to analyze the experience of the single user ► In order to monitor the e2e network chain the KPIs have to be technology/access Technical KPI independent Service KPI • Immediate overview of the service • bricks of service KPI monitor specific network elements Critical Indicators Localized Access layer Operative System Device Network section Core 9 9 2013 – VOLTE QOS ASSESSMENT TECHNOLOGY EVALUATION EVENT 10 © ETSI 2013. All rights reserved VoLTE QoS Assessment event overview Scope: Technology evaluation Plugtests aiming at validating the different methodologies for end to end QoS Assessment for VoLTE described in ETSI TS 103 189 over a deployed LTE/IMS network. Dates: 18-20 November 2013 Venue: ETSI, Sophia Antipolis, FR Web: http://www.etsi.org/news-events/events/698-volte-qos-assessment-2013 Participants: Supporter: 11 © ETSI 2013. All rights reserved VoLTE QoS Assessment event scope End to end tests performed with several different UEs over a deployed LTE / IMS test network. 4 Test Setups • Acoustic to acoustic • Acoustic to electrical • Electrical to acoustic • Electrical to electrical Voice quality tests based on instrumental assessment of speech samples performed on different combinations of the acoustic and electrical interfaces which allows evaluating the different methodologies and setups. Parallel network performance evaluation 12 © ETSI 2013. All rights reserved 2013 – VoLTE QoS Assessment Technology Evaluation event – test description TS 103 189 Sub-parameters delivered by the particular tests: End-to-end Tests based on Instrumental Assessment of Speech Samples (POLQA model deployed): • • • • Mean Opinion Score (MOS)-Listening Quality Objective Delay variation over time Difference in loudness level MOS stability End-to-end Tests based on Instrumental Assessment of Video Samples (PEVQ model deployed): • • • • • • DMOS (Difference Mean Opinion Score) Delay and Quality variation over time Jerkiness and Blockiness Blur, Chrominance and Luminance Temporal Distortion Frozen and Skipped Frames 2013 – VoLTE QoS Assessment Technology Evaluation event – test description TS 103 189 Sub-parameters delivered by the particular tests: Other end-to-end tests in the voice channel: • • • • • End-to-end frequency response Overall loudness rating End-to-end delay Quality of echo cancellation Double talk performance End-to-end Tests based on functional parameters: • Telephony Call Setup Time • Streaming Service Access Time Network Performance Parameters • UNI-to-UNI delay over time • UNI-to-UNI jitter • UNI-to-UNI packet loss 2013 – VoLTE QoS Assessment Technology Evaluation event – test description General description of tests defined in ETSI TS 103 189: QoS tests based on the ETSI INT specifications as described in the latest versions of: • • • • ETSI TS 186 011-1 (Test purposes for IMS NNI Interoperability) ETSI TS 186 011-2 (Test descriptions for IMS NNI Interoperability) ETSI TS 102 901 (IMS NNI interoperability test descriptions for RCS) ETSI TS 103 029 (IMS & EPC Interoperability). QoS tests divided into two parts: • First one focusing on the instrumental assessment of speech and video quality by using real speech and video samples • Second one dealing with functional parameters of voice and video over LTE connections and other parameters in the end-to-end voice channel • On the top of it, selected network performance parameters listed in the TS 103 189 15 © ETSI 2013. All rights reserved 16 VoLTE QoS Assessment – Setup 1 Acoustic-to-Acoustic © ETSI 2013. All rights reserved ITU-T Specifications for Quality measurements Recommendation ITU-T J.247: "Objective perceptual multimedia video quality measurement in the presence of a full reference". Recommendation ITU-T P.50: "Artificial voices". Recommendation ITU-T P.57: "Artificial ears". Recommendation ITU-T P.58: "Head and torso simulator for telephonometry". Recommendation ITU-T P.64: "Determination of sensitivity/frequency characteristics of local telephone systems". Recommendation ITU-T P.79: "Calculation of loudness ratings for telephone sets". Recommendation ITU-T P.340: "Transmission characteristics and speech quality parameters of hands-free terminals". Recommendation ITU-T P.380: "Electro-acoustic measurements on headsets". Recommendation ITU-T P.501: "Test signals for use in telephonometry". Recommendation ITU-T P.505: "One-view visualization of speech quality measurement results". Recommendation ITU-T P.502: "Objective test methods for speech communication systems using complex test signals". Recommendation ITU-T P.581: "Use of head and torso simulator (HATS) for hands-free terminal testing". Recommendation ITU-T P.800.1: "Mean Opinion Score (MOS) terminology". VoLTE QoS Assessment – Setup 2 Acoustic to Electrical 18 © ETSI 2013. All rights reserved VoLTE QoS Assessment – Setup 3 Electrical to Acoustic 19 © ETSI 2013. All rights reserved VoLTE QoS Assessment – Setup 4 Electrical to Electrical 20 © ETSI 2013. All rights reserved MAIN OUTCOME OF THE VOLTE QOS ASSESSMENT AND NEXT STEPS… 21 © ETSI 2013. All rights reserved Main event outcome Validation of 76% of the test cases in TS 103 189 A number of gaps and potential improvements in TS 103 189 have been identified A new work item has been open for the update of TS 103 189 with the new setup, acoustic-to-electrical identified during the event TS 103 189 has been published and is available on the ETSI portal 22 © ETSI 2013. All rights reserved Further steps with regard to ETSI TS 103 189 Verification of remaining voice quality test cases • tbd Verification of video quality test cases (RTP-based video streaming): • Test houses willing to do so are needed Extension of video quality test cases towards MPEG-DASH • ITU-T SG12 Question 14 currently running a WI called P.NATS 23 Maputo - Mozambique - 14 - 16 April 2014 2013 – VoLTE QoS Assessment Technology Evaluation event – test results 24 © ETSI 2013. All rights reserved 2013 – VoLTE QoS Assessment Technology Evaluation event – test results 25 © ETSI 2013. All rights reserved Verification of video quality test cases (RTP-based video streaming) What we are looking for? What are the parameter we want to measure? What are the tools expected? Parameters Covered by TS 183.189 26 © ETSI 2012. All rights reserved test tools focused on video quality Prototipization and validation Customers perception KPIs VoLTE Browsing Non Accessibility Session Setup Failure Setup time Cut off Ratio Packet loss rate Latency Jitter MOS Audio/Video Quality Number of HTTP Sessions Average Duration of HTTP Sessions Average HTTP Throughput Peak HTTP Throughput HTTP Setup Time HTTP Dropped Connections HTTP Sessions Unavailability Retransmission Rate Packet loss rate Latency Jitter Loading Page 27 © ETSI 2012. All rights reserved Gaming Number of Gaming Sessions Gaming Sessions Average Duration Gaming Average Throughput (DL & UL) Gaming Overall responsiviness [ms] Retransmission Rate Packet loss rate Latency Jitter Number of streaming sessions Average duration of streaming sessions Average Streaming Throughput Peak Streaming Throughput Setup Streaming Time Streaming Dropped Sessions Streaming (HTTP YouTube sessions included) Streaming Unavailability Re-Buffering Time during Streaming Streaming Synchronization Packet loss rate Latency Jitter MOS Audio/Video Quality Networks KPIs/KQIs 4/9/2015 © Zhilabs 2014, Confidential Information 28 Next steps Get comments on TS 103 189 Extend TS 103 189 addiming more Customer Perception KPIs Get proposal from Video Test Houses Next event will be launched when VoLTE Native Handset and PEVQ tools will be available for testing 29 Contact Details: Giulio Maggiore Telecom Italia ETSI TC INT Chairman Giulio.Maggiore@telecomitalia.it Thank you! 30 © ETSI 2013. All rights reserved TC INT TERMS OF REFERENCE 31 © ETSI 2013. All rights reserved TC INT (Core Network and Interoperability Testing) Terms of Reference The scope of INT includes: Development of Core Network test specifications (interoperability, conformance, performance, security) based on 3GPP and E2NA specifications. To initiate and supervise interoperability events (such as Plugtests) related to Core Networks as well as other events (workshops and seminars) To coordinates interoperability efforts with other organisations GSMA, OMA, IMS/NGN Forum, IMTC, etc. Endorsement of test specifications from/to other SDOs e.g. ITU-T. 32 © ETSI 2013. All rights reserved ETSI INT SPECIFICATIONS Interoperability , Performance , Security and Conformance 33 © ETSI 2013. All rights reserved TC INT Interoperability Specification Work R6 TS 186 011 V 1.1.1 2007 R1 34 ETSI/BOARD(13)94_041 R7 TS 186 011 V 2.1.1 Late R7 TS 186 011 V 2.2.1 2008 2009 R2 Late R2 R8 TS 186 011 V 3.1.1 TS 102 901 V 1.1.1 2010 R9 R10 TS 186 011 V 4.1.1 TS 102 901 V 2.1.1 TS 186 011 V 5.0.2 TS 102 901 V 5.0.6 TS 103 129 TS 103 129 V2.0.4 2011 2012 2013 E2NA Conformance Test Specifications IMS Basic Call Specifications and Supplementary Services; Diameter Conformance testing for Rx interface TS 101 580; Diameter Conformance testing for Gx interface TS 101 606; Diameter Conformance testing for S6a and S9 interface just started; 35 ETSI/BOARD(13)94_041 Interoperability Test Specifications IMS NNI Interworking Test Specifications TS 186 011 RCS TS 102 901 IMS&EPC interoperability TS 103 129 36 ETSI/BOARD(13)94_041 IMS Interworking Test Specifications IMS Interconnection Test Specifications INTEROPERABILITY EVENTS 37 © ETSI 2013. All rights reserved Interoperability events 2007 – The first event examined interconnection aspects of 3GPP IMS Release 6, including basic call on the Mw interface. 2008 - The second event had a wider scope that included the testing of 3GPP IMS Release 7 interworking, roaming, border control, and integration of application servers executing selected Multimedia Telephony supplementary services. 2009 - The third ETSI IMS event assessed the interoperability as well as conformance of IMS core networks (composed of P/I/S-CSCF, IBCF, AS, DNS and HSS) which are implemented on the basis of ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0) (3GPP late release 7). 2012 – RCS&VoLTE interoperability event in cooperation with GSMA and MSF held in Kranj (Slovenia) and Beijng (China) 38 ETSI/BOARD(13)94_041 Press Coverage: RCS VoLTE Test Event 39 ETSI/BOARD(13)94_041 2007 and 2008 IMS interoperability events • The first event examined interconnection aspects of 3GPP IMS Release 6, including basic call on the Mw interface. • The second event had a wider scope that included the testing of 3GPP IMS Release 7 interworking, roaming, border control, and integration of application servers executing selected Multimedia Telephony supplementary services. 40 © ETSI 2013. All rights reserved 2009 - IMS interoperability events The third ETSI IMS event assessed the interoperability as well as conformance of IMS core networks (composed of P/I/S-CSCF, IBCF, AS, DNS and HSS) which are implemented on the basis of ETSI TS 124 229/3GPP TS 24.229 (Version 7.14.0) (3GPP late release 7). 41 © ETSI 2013. All rights reserved 2012 – RCS & VoLTE interoperability event The forth event endorsed and tested a number of GSMA PRDs and validated the Testing Scenarios developed by MSF and ETSI to facilitate multivendor deployment of LTE./EPC/IMS technology for RCS and VoLTE services. http://www.etsi.org/news-events/pastevents/642-2012-rcs-volte-interop-event 42 © ETSI 2013. All rights reserved