ITU Regional Standardization Forum for Africa Livingstone, Zambia 16-18 March 2016 Session 1 : Methods and Tools for the evaluation of the QoS/QoE of the Fixed and Mobile Internet - Study Case MoroccoHassan TALIB, Vice-Chair ITU-T SG 12, Head DCT ANRT Talib@anrt.ma // Htalib@ties.itu.int Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 2 Background for the Follow-up of the QoS/QoE at ANRT Adoption of a Device QoE Portal: Q oE and downloaded applications An international regulatory framework (standards): - A national regulatory framework: Remits QoS Obligations for operators Feedback on uses Complaints, Press, Associations of consumers, … - Itu-T: Series E, G, P, Y, QoS Handbook, ... Regional: ETSI (series EG), IEEE,… The ANRT exerts a regular monitoring of the QoS/QoE The field measures (campaigns) carried out by ANRT and QoE measures communication strategy (Publication) A benchmark of best practices and a technological watch Analysis of the data received from operators in terms of performance, KPIS 3 Background for the follow-up of the QoS Establishment of a global device for receipt of KPIS of operators: all networks and all services Elaboration of agreed reference documents by all the actors managing the field measures of the QoS Complete model of operational KPI data according to the frequencies of fixed sending Mastered outsourcing of measures and use of results by the operators 4 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 5 The fixed and mobile Internet Services 3G and 4G in Morocco • Three global operators : Itissalat Al-Maghrib (Morocco Telecom), Medi Telecom (Méditel) and Wana Corporate (INWI): – Fixed (national and international); – Mobile: 2G (GSM and CDMA), 3G (UMTS and CDMA 2000) and 4G LTE, • Fixed Internet: ADSL and FTTX (coupled offers: Tripleplay). • Mobile Internet: 3G (marketed since March 2007) and 4G (marketed since June 2015). • Internet Marketing: telecom operators (licenses) or service providers (declarations). Livingstone, Zambia, 15-18 March 2016 6 The Internet service at Morocco Mobile telephony: park and traffic Evolution du trafic voix sortant du Mobile Evolution du Parc et du Taux de pénétration du Mobile 50000 45000 130% 39 016 36 554 31 982 40000 35000 30000 25000 20000 15000 10000 5000 52 874 150% 44 115 42 424 43 080 48 259 50 000 127,27% 110% 40 072 40 000 32 520 90% 25 311 22 816 20 029 16 005 12 393 9 337 7 333 6 198 4 776 2 852 364 0 30 000 70% 50% 20 000 11 528 9 659 10 813 30% 10 000 10% -10% 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Parc Mobile (en Milliers) 23 315 Taux de pénétration Mobile The park of the mobile subscribers has experienced a very strong evolution since 1999. The penetration rate has reached more than 127% at the end of 2015. 4 267 14 076 5 857 0 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Trafic voix sortant du Mobile (en millions de minutes) The traffic for outgoing voices of the Mobile has exceeded 52 billion minutes in 2015, registering a growth of more than 9% compared to 2014. Livingstone, Zambia, 15-18 March 2016 The Internet service at Morocco Average usage and mobile ARPM Evolution of the average revenue per minute Mobile (ARPM) Evolution of the average use mobile outgoing 120 1.8 101 92 83 1.4 75 80 57 60 43 40 1.64 1.6 In HT DH/month Minutes/month/subscriber 100 41 40 1.6 1.27 1 0.92 1.27 1.12 0.9 0.84 0.8 0.6 0.81 0.74 0.62 0.4 0.56 0.53 0.47 0.44 0.41 0.4 0.2 20 0.36 0.32 0.3 0.29 0.27 0.26 0 2008 2009 2010 2011 ARPM Global Mobile 0 2008 1.24 1.2 2009 2010 2011 2012 2013 2014 The average monthly outgoing use per mobile client is appreciated from 43 to 101 minutes/client/month, or a growth of almost 135% between 2008 and 2015. 2015 2012 2013 2014 2015 ARPM Mobile Postpaid ARPM Prepaid Mobile The ARPM "Average Revenue per Minute" Mobile has evolved from 1.27 DHHT/min at the end of 2008 to 0.27 DHHT/min at the end of 2015 recording a decrease of 78.7%. For the prepaid Mobile, this indicator is about 0.26 DHHT/min while it is 0.29 DHHT/min for the Postpaid. Livingstone, Zambia, 15-18 March 2016 The Internet service at Morocco The Internet: Park and penetration 16000000 14 471 10645,00% 14000000 40,00% Distribution of Internet Park by type To access (December 2015) 0.02% 35,00% 12000000 7.82% 9 969 023 30,00% 10000000 25,00% 8000000 5 776 173 3 957 068 6000000 4000000 3 182 116 1 866 963 1 186 923 757 453 2000000 399 720 526 080 262 324 20,00% 15,00% 10,00% 5,00% 0,00% 0 92.16% 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 Parc des autres offres Internet Parc ADSL Parc Internet mobile Taux de pénétration Internet ADSL Mobile Internet Other Internet offerings The Internet Park has experienced a sustained growth during the last five years. This growth is mainly due to the development of mobile Internet offers. On nearly 14.5 million Internet subscribers, more than 92% use the mobile connections versus 7.8% for the fixed ADSL access. Livingstone, Zambia, 15-18 March 2016 The Internet service At the Morocco Invoice monthly Internet by client Evolution of the invoice of Internet on average 180 167 160 154 157 In HT DH/month/subscriber 139 140 121 120 116 110 100 83 111 98 82 93 94 23 18 24 17 80 58 60 53 42 37 40 36 27 21 20 0 2008 2009 ADSL 2010 2011 The Global Internet 2012 2013 2014 2015 Mobile Internet On the period 2008-2015: the monthly invoice on average per Internet subscriber has registered a decline of 84%. At the end of 2015, the monthly invoice on average per Internet subscriber has reached 24 DHHT/month/customer. For the mobile Internet, this invoice is about 17 DHHT/month/client while it is 94 DHHT/month/client for the ADSL. Livingstone, Zambia, 15-18 March 2016 The Internet service At the Morocco International bandwidth and domain names 60000 Evolution de la bande passante Internet 500 000 450 000 450 000 412 000 450 000 50000 59,138 54,450 Evolution of the number of domain names ".ma "50,945 46,806 42,187 37,969 34,008 400 000 40000 350 000 300 000 Mb ps 266 000 29,450 25,890 22,583 19,025 30000 250 000 200 000 150 000 100 000 50 000 20000 124 400 10000 75 000 51 200 25 130 25 130 11 500 200 310 1 2407 100 9,476 5,298 3,586 674 1,998 0 0 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 The international Internet Bandwidth has stabilized at 450 000 Mbps in 2015. The park of the domain names in " .ma" continues to grow in reaching 59138 accounts at the end of 2015, or a growth of almost 8.6% compared to the end of 2014. The domain names created directly under the extension " .ma « prevail over (89.6%). The total number of providers".ma" is 33 at the end of December 2015, of whom 04 were refisteredin the course of the 4Th Quarter 2015. Livingstone, Zambia, 15-18 March 2016 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 12 QoS of The Internet on The xDSL Measurement device for the qualification and eligibility of the xDSL line: Length, diameter of the line (caliber ), crosstalk (telecrosstalk), LOSSES, … Interface of the test for speed ADSL on the site Menara : The heart of the network Livingstone, Zambia, 15-18 March 2016 QoS of the Internet on the xDSL Through the device of qualification of the line we measure the qualities of electrical transmission of the ADSL line between the equipment of the client and the connection point of the fixed network (DSLAM/MSAN): Upward speed /Downward speed, ( Current /contractual rising and upward speeds the test device and the DSLAM/MSAN) Maximum Upward speed /Maximum downward speed (Maximal speed upward and downward that can withstand the ADSL line) Capacity (% use) of The line upwardly and downwardly Estimated length of the line. Livingstone, Zambia, 15-18 March 2016 QoS of the Internet on the xDSL By connecting to the Web site Menara (measures the quality of the ADSL Internet access between the customer equipment and the MenaraTest Server ): Actual upward speed/actual downward speed Latency ( deadlines) and Jitter (jitter), Recorded debit/theoretical and marketed debit. Livingstone, Zambia, 15-18 March 2016 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 16 QoS of The Internet Mobile • Methodology of the Evaluation of the Quality of Services QoS Data for the 3G networks (UMTS or CDMA 2000 on PC or SP ) is perfectly valid and adapted the mobile and broadband networks: 4G ( ), Wifi Outdoor (offloading ),… Livingstone, Zambia, 15-18 March 2016 17 QoS of The Internet Mobile • Types and conditions of the QoS Measurements : – Mobile Internet on PC: USB dongles (Prepaid Or PostPaid) on Computers for UMTS, CDMA-2000 or LTE. – Mobile Internet on Smartphones: SIM/3G or 4G for Subscribers on SP/tablets only for the UMTS or LTE. – The measurements in FTP or HTTP modes: to carry out on the files tailored for measurements (Up/Dw): • For the networks 3G uplink: =U=1Mb and downlink=D=5MB. • For the networks 4G Uplink: =U=250MB And Downlink=D=1GB. 18 Livingstone, Zambia, 15-18 March 2016 QoS Of The Internet Mobile • Types and conditions of the QoS measurements : – Evaluation of the QoS ≠ Evaluation of the Coverage. – Measurement should be performed only on areas declared covered by all the relevant operators together : Exercise of the geographic sample and the mapping for Coverage. – The exercise of Mapping for Coverage should be conducted in accordance with quarter details instead of being conducted per town. Livingstone, Zambia, 15-18 March 2016 19 QoS Of The Mobile Internet -Measurements Tools • Measurement Tools (Precautions) – Terminal Equipement (PC, SP and dongles/USB) , The criteria sf Choices : • Choose among the most used manufactured models on the National market • Providing measurement performances that do not restrain the higher measured speed rates in any case. For PCs(OS, firewalls, antivirus,…) and for terminals (Twindowsize, MTU, CPU , RAM,…). Livingstone, Zambia, 15-18 March 2016 20 QoS Of The Internet Mobile - Tools Of The measures • Measurement Tools (precautions) – Profiles of subscriptions used: • Post-paid or prepaid. • Beware of speed Downgrading if the downloaded volume reaches the thresholds. • The tests on the SP must be done in forced mode 3G or 3G/4G and not in dual (avoid confusing GPRS and EDGE). • For each operator (a server and a link) only one measurement is to performed at the same time. Livingstone, Zambia, 15-18 March 2016 21 QoS of The Mobile Internet Mobile - MeasurementTools • Measurement Tools – Application of Softwares (Agents installed in the terminals : PC and SP): • At each data connection , the application provides information enabling the automatic calculation of all the indicators of QoS of the connection (level of fields, SC, failures, success,…) as well as the GPS position. • The application allows you to store automatically and directly all results (indicators) in a server that centralizes the preparation of reports required in the framework of dedicated portal. Livingstone, Zambia, 15-18 March 2016 22 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 23 QoS Of the mobile Internet • For each type of measurement, what indicators are relevant? – Jitter, time limit, flow, losses of data,…? – Pragmatic goals (by survey of satisfaction) pertaining to the users experience : Measure the accessibility (rate and time of connection), the reliability and the speed (flow of transmission and reception). – Transformation of these elements in ten (10) Indicators: Livingstone, Zambia, 15-18 March 2016 24 QoS Of The Mobile Internet • List Of the ten (10) Measured Indicators : – The rate of successful connections: A connection is successful if it is established in a period of less than 1 minute. The rate of successful connections is calculated on the basis of the set of measures carried out. – The rate of successful connections in a period of less than 10 seconds: The rate of successful connections in a period of less than 10 seconds is calculated on the basis of the set of measures carried out. – The rate of files of U Mo sent in a period of less than 2 minutes: A file is considered sent if the file is sent in full in a DMAX deadline and if its content is correct. The rate is calculated on the basis of the total number of files sent. – The rate of files of Mo received within a period of less than 5 minutes: A file is considered to be received if the file is received in full and if its content is correct. The rate is calculated on the basis of the total number of files downloaded. Livingstone, Zambia, 15-18 March 2016 25 QoS Of The Internet Mobile • List Of the ten Measured Indicators : – The speed of data reached for 90% of the U Mo files sent: This indicator corresponds to the percentile to 90% of the files sent. – The speed of data reached for 50% of U Mo files sent: This indicator corresponds to the percentile to 50% of the files sent. – The flow of data reached for 10% of U Mo files sent: This indicator corresponds to the percentile to 10% of the files sent. Livingstone, Zambia, 15-18 March 2016 26 QoS Of The Mobile Internet • . List Of the ten Measured Indicators : -The speed of data reached for 90% of the U Mo files received: This indicator corresponds to the percentile to 90% of the files to be downloaded. -The speed of data reached for 50% of U Mo files received: This indicator corresponds to the percentile to 50% of the files to be downloaded. -The flow of data reached for 10% of U Mo files received: This indicator corresponds to the percentile to 10% of the files to be downloaded. Livingstone, Zambia, 15-18 March 2016 27 QoS of the Mobile Internet • Measured Indicators: – Important Details: The speed for 3G or 4G networks is the one shared between the users. The actual speed rate (operated) is always less than the theoretical speed rate (marketed). It is technological dimension. But, this speed is measurable for all operators through the same device. Livingstone, Zambia, 15-18 March 2016 28 QoS Of The Mobile Internet • Additional indicator: TUDC The Rates of use of the contractual speed (TUDC) corresponds to the ratio between the speed used (observed) to the contractual speed(marketed) by the relevant operator . This report is expressed in percentage. Débits observés/ Débits commercialisés (en %) • An actual Example for a Mobile Network: (Débit atteint par 50% des fichiers reçus) 100.00% 90.00% 80.00% 70.00% 60.00% 50.00% 54.10% 48.83% 41.80% 36.23% 40.00% 39.53% 31.64% 30.00% 24.61% 20.00% 10.00% 0.00% Casablanca Rabat Agadir Marrakech Livingstone, Zambia, 15-18 March 2016 Fès Tanger Moyenne nationale 29 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 30 Platform Measurement Servers • Platform Measurement Servers for the Internet QoS for fixed or mobile networks? – Measuring the performance of an Internet network , is measuring the QoS of a connection through this network between a terminal and a data server. – Several external factors can affect one of the segments of the path of test. Hence the controversy the results of measures by the relevant operator . – Options to set aside: • Measures with a server installed abroad; • Install the server in the operator’s premises. Livingstone, Zambia, 15-18 March 2016 31 Platform Measurement Servers • Platform Measurement Servers: – Problem: Lack of server (in the host premises), at national level, connected to all networks at the same time by a capacity greater than the supply of the highest speed!!! – Solution: The design of a platform of measurement servers in the premises of the regulator. The capacities of the links are greater than the measured speed (example 10Mb/s for measuring 7,2MB/s and 20Mb/s for measuring 14.4Mb/s and 120MB/s for measuring the 4G). Livingstone, Zambia, 15-18 March 2016 32 Platform Measurement Servers • Platform Servers: All files tests (U and D) are installed in each server in the platform connected by FO to the networks of operators. Livingstone, Zambia, 15-18 March 2016 33 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 34 Evaluation of the QoE Of the mobile Internet The Solution enables users (Smartphones/tablet more than 40 million) to participate in the evaluation of the QoE and download a mobile application that will be used to forward, in a transparent way, information and data on the QoE to the ANRT. This device should enable us to achieve : The collection and the calculation of indicators of the QoE of various services on a continuous basis; The active tests of end-to-end connections; The supervision of QoE parameters from the premises of ANRT ; The Provision of the reports on QoE in a dynamic way; The extension for supplementing with new data or new reports. Compliance with the three basic principles: Consent of users; Observance of the regulation of personal data; Transparent Application and no impact on the user ( neither on the account nor on the Device: battery or performance). Livingstone, Zambia, 15-18 March 2016 Evaluation of the QoE Of the mobile Internet The solution of the evaluation of the QoE Is composed of the following three elements: Mobile Application : It is downloadable free of charge from a portal on the Web site of the ANRT and on the different sites of applications at least on App-Stores and Play-Stores and enabling to perform automatic and continuous measurements of the QoE. Transparent Application and no impact on the user (neither on the account nor on the device). System for the collection and processing of data (ANRT): This System is installed in the premise of ANRT (the Data Server) and it enables the collection and automatic processing of the results of measures as well as the calculation of the QoE indicators . System of Reporting (ANRT): It must include a user interface allowing the visualization of the QoE indicators as well as a powerful tool for producing reports. The Results must be, after processing, in the form of reports Including the results of Performance per operator, the geographical coordinates and the network parameters Including coverage (2G, 3G and 4G…), dashboards as well as graphics. Livingstone, Zambia, 15-18 March 2016 Evaluation of the QoE Of the mobile Internet The Solution enables to evaluate the QoE of Mobile networks of Telecommunications in accordance with the following services: The QoE of the services of the 2Th Generation networks (2G) : The Voice 2G; The service of the SMS messaging; The QoE of the services of the 3Th Generation networks (3G: UMTS/HSPA/HSPA+): The Voice 3G; The data 3G on Smartphones/tablets. The QoE of the services of the 4Th Generation networks (4g/LTE): The Voice 4G (Future utilization); The data 4G on Smartphones/tablets. Livingstone, Zambia, 15-18 March 2016 Evaluation of the QoE Of the mobile Internet Voice Service 2G, 3G and 4G: Service Data 3G and 4G: Time of establishment of call (second); Failure Rate (%); Cut-off rate (%); Call Duration per cut off (minute); Success Rate (%); SMS Service: Unavailability of the network; Time to access the service per operator and per (type of service); Availability of the service per operator and per (type of service); Identification of areas with no data coverage per operator. Rates of not received SMS; Rate of received SMS; Duration of SMS receipt ; Rate of successful connections; Average speed in Up-Link; Average speed in Down-Link; Peak speed in Up-Link; Peak speed in Down-Link; Loading time of the Web pages. Coverage Livingstone, Zambia, 15-18 March 2016 Plan of the presentation • • • • Introduction: background of QoS/QoE… The Internet service at Morocco Evaluation of the QoS of the fixed Internet Evaluation of the QoS of the Mobile Internet 3G and 4G – Mobile Internet on PC, – Mobile Internet on Smartphones – FTP or HTTP measures • • • • Measured indicators (Definitions) Platform of Measurement Servers Evaluation of the QoE of the Internet Conclusions and Recommendations Livingstone, Zambia, 15-18 March 2016 39 Conclusions and Recommendations • Recommendation N°1 – Permanent consultation with the operators on the methodology at the upstream level. Adoption of a concerted and a reference approach for the follow-up. – Adoption of a framework (protocol) of measures for the evaluation of the QoS/QoE. – None of the operators is informed on the sites of the QoS measurements QoS; nor on the period. Livingstone, Zambia, 15-18 March 2016 40 Conclusions and Recommendations • Recommendation N°2 – Achieve a significant number of blank measurements QoS/QoE , with the same tools as those used in the actual campaigns of measures and preferably in the presence of the representatives of the operators concerned by the measures (marginalize or eliminate any potential controversy of operators). Livingstone, Zambia, 15-18 March 2016 41 Conclusions and Recommendations Recommendation N°3 – The measurements for operators: positive use of the results by the operators (use of the platform servers for tests//financing campaigns by the operators). Recommendation N°4 – Release the results (comparative), adopt a communication strategy and envisage sanctions for defects as the last resort. Livingstone, Zambia, 15-18 March 2016 42 Conclusions and Recommendations Recommendation No. 4 (continued) – Publication of the results of measures: The first Voice campaign on a large sample (30 Cities and crowded facilities airports, tourist centers…). 1Er Quarter of the year Communicating Results to the operators Timeline of 5 to 6 months for The Correction of defects Identified by the ANRT Livingstone, Zambia, 15-18 March 2016 The second similar campaign to the first one and based on the same sample broadly. 4Th Quarter of The Year (Publication of the Results and communication) 43 Thank you for your attention Questions/Answers Talib@anrt.ma // Htalib@ties.itu.int Livingstone, Zambia, 15-18 March 2016 44