ITU Regional Standardization Forum for Africa
(Kampala, Uganda, 23-25 June 2014)
Monitoring QoS/QoE of mobile
networks-novel approach
Tahitii Obioha
Network Planning and Optimisation Engineer
Planet Network International, France.
Kampala, Uganda, 25 June 2014
Why Monitor QoS/ QoE of mobile networks
Monitoring = observing, checking and keeping a continuous
record of the progress or quality of something.
It is the use of any available technical tool to assess permanently
or for a given period of time a particular QoS parameter, e.g. call
set up time. (ETSI EG 202 009-2)
Reasons
For end-users : Satisfaction, get the best value for their money
amidst others
For operators : end-user Satisfaction, Increase revenue,
achieve customer loyalty, maintain competitive
edge amidst others. Surveys have shown that
about 90% of customers will not complain before defecting.
For Regulators: protect consumers, ensure end-user satisfaction,
create a level playing field for operators to compete amidst others.
They all share a common Objective CUSTOMER SATISFACTION
Kampala, Uganda, 25 June 2014
2
QoE Measurement
Quality of Experience(QoE) has become an important
concept since the growth of mobile services.
QoE is very SUBJECTIVE in nature as it involves human
dimensions.
Measuring and improving QoE though challenging,
should be undertaken in order to assess the most
accurate and complete vision of the value offered by
the provider to end-users.
If the QoE is high, the user is happy, satisfied and loyal
while poor QoE will result in dissatisfied customers.
Kampala, Uganda, 25 June 2014
3
QoE Measurement
QoE KPIs
USER EXPECTATIONS CAN BE GROUPED UNDER TWO MAIN CATEGORIES
1. RELIABILITY (Service quality of Accessibility and Retainability KPIs)
2. COMFORT (Service quality of Integrity KPIs)
Reliability QoE KPIs
QoE KPI
Most important measures
Service
availability(anywhere)
Local and global coverage included
How seamless it is for the user?
Service
accessibility(anytime)
The success rate of user connections for any
service
Service access time
The delays in setting up any service connection
Continuity of service
connection-interruption
ratio
The retainability of the service connection and its
performance overtime
Kampala, Uganda, 25 June 2014
4
QoE Measurement
Comfort QoE KPIs
QoE KPI
Most important measures
Quality of Session
Jitter (delay variation %)
Average throughput(kbits/s) towards mobile
Bearer stability
Application layer packet loss ratio
Average end-to-end delay (ms)
Ease of use
How easy is it to use service offered by the network
Level of support
How quick and easy is it to get customer support
Kampala, Uganda, 25 June 2014
5
QoE Measurement
Two pratical approaches to measuring QoE
Service level approach using Statistical Samples
•Approach
relies on a statistical sample of the overall network
the QoE for all the users in the network. (Mean Opinion Score (MOS))
users
Network Management system approach using QoS parameters
•Approach
relies on using NMS to
and comparing them with predefined target
collect
QoS
KPIs
from
netwo
Knowing full well that the holy grail of subjective measurements is to predict it from
the objective measurements i.e. predict QoE from a given set of QoS parameters.
NMS approach provides high accuracy than Service level because of its deterministic
and objective nature.
Using both methods nevertheless provides a better and more accurate picture of the
users’ experience
Kampala, Uganda, 25 June 2014
6
Relationship between QoS and QoE
QoE = f(AQoS, NQoS) + other subjective factors (expectations, mood, etc.)
Strong Correlation between QoS and QoE
Relationship Between QoE and QoS may be near-linear for a voice call
but non-linear for a video call
Kampala, Uganda, 25 June 2014
7
Relationship between QoS and QoE
There is an exponential relationship between QoS and QoE
End-to-end QoS is an important enabler for QoE in otherwords
a better network QoS in many cases will result in better QoE
QoE
vice
can
be predicted
versa
from a
(ETSI
given
set
TS
of
QoS param
102
QoE may be improved with the right QoS mechanisms.
Kampala, Uganda, 25 June 2014
8
QoS Measurement
Quality of Service(QoS) arrived with 2nd Generation Networks
Ability of a Network to provide a service with an assured service
level.
It is intrisincally a technical concept, measured and understood in
terms of networks and network elements.
An objective metric that refers to the ability of the network to
achieve a more deterministic behaviour.
Service Qualification is based on six 6 primary components as defined
in ITU-T Rec E-800:
Operability
Accessibilty
Retainability
Integrity
Security
Support
Performance
Assurance
85%
15%
Key Performance
Indicators(KPIs)
Key Quality (Assurance)
Indicators(KQIs)
QoS can be measured using KPIs and KQIs
Kampala, Uganda, 25 June 2014
9
Relationship between NP and QoS
Strong Correlation between NP and
QoS
In otherwords Poor Network
Performance ultimately results to Poor
QoS
Network related QoS
QoS aspects of service usage(ETSI TS 102 250-1)
Kampala, Uganda, 25 June 2014
10
Relationship between NP and QoS
Qos Criteria
Network related QoS criteria
85%
Non network related QoS
criteria
15%
Mapping
Network Performance
Indicators/Parameters
Target - range or limit
CALL DROP RATE
CALL SETUP SUCCESS RATE
……
PARAMETER N
< 2%
< 90%
……
<= x%
Kampala, Uganda, 25 June 2014
11
QoS Measurement
NP
Active
Objectiv
e
QoS
Non-NP
Subjective
Passive
Intrusive
NonIntrusive
Surveys, churnrate, etc…
Walk/Drive
Test Around
test
OMC-R
Counters
using NMS
Ex.NMS: RPM System, Net Act,
imanager M2000 …
Ex. DT Tools: Nemo, XCAL, TEMS …
Relationship between NP,QoS and QoE
NP
QoS & QoS
QoE
NP is a subset of QoS which in turn, is a subset of the overall QoE scope
NP parameters ultimately determine the QoS
QoE can
vice
be
predicted
versa
Kampala, Uganda, 25 June 2014
from
a
(ETSI
given
set
of
TS
QoS
param
102
13
QoS/QoE Assesement
TWO DIFFERENT AND COMPLEMENTARY APPROACHES TO MOBILE
QoS (ETSI EG 202 057-3)
Walk/Drive- round tests
Measurements based on network element counters using NMS
QoS Assesment Target
Best Suitable QoS
Approach(es)
One-time snap shot
DT
Acceptance Procedure
DT/ NMS
Continuous Monitoring
NMS
Optimisation Cycle
DT + NMS
Monitoring :use of any available technical tool to assess
permanently or for a given period of time a particular QoS
parameter. (ETSI EG 202 009-2)
Kampala, Uganda, 25 June 2014
14
Difference Between NMS and DT
Network Management System
Drive Test (DT)
QoS experienced by users are taken into account as
the calls are made with the real terminals by the real
users.
The QoS experienced is not taken into account by
users depends to some extent on the design of their
terminals which may differ somewhat from those used
for formal tests.
Representativity is assured as real traffic provides
measurements from the entire network 24h/7
Drive test does not provide a representative of the
performance of the whole network because of
variation in user density and the complexity of
providing service in different areas. (60% of calls
made in a network are made indoor).
Track performance down to the cell level
Performance stops at Base station level
Provides comparabilty of Congestion and network
failures
Misses out on planned Site outages and network
failures.
Follow up on Operators’ network expansion, type of
equipment deployed and mobile evolution
Impossible to track Network Expansion using DT
NMS reports are actionable as it tracks the availability
of network elements (Ex. BSC/RNC)
DT reports are simply informative
Indepedent of enviromental factors
Depedent on environmental factors such as weather
conditions, seasons, location, timing etc
Traffic carried over the network and associated grade
of service (GoS) are obtained using this approach
DT has no information on Traffic carried over a given
network.
KPI Build flexibility from numerous counters of NEs
Limited KPI observation as it measures the network
from an external point.
Relatively no OPEX involved
High OPEX as well as time and Labor intensive
Passive and non- intrusive, best approach in
continious QoS monitoring.
Active and intrusive, best approach for QoS
(coverage) benchmarking purposes.
Kampala, Uganda, 25 June 2014
15
Draw backs of DT illustrated
Kampala, Uganda, 25 June 2014
16
Draw backs of DT illustrated
Kampala, Uganda, 25 June 2014
17
Draw backs of DT illustrated
Kampala, Uganda, 25 June 2014
18
Quest for a Monitoring Tool
Monitor simultaneously 24H/7 the state of all network
elements/acess technologies in an operator network passively
without compromising their daily operations
Put a check on service degradation and sometimes outages to the
benefit of the government and the masses.
Quest for a system that can monitor the performance of mobile
operators independent of the vendor and technology
Flexible Benchmark Audit; weekly, monthly or quarterly.
Track the evolution and expansion of the mobile operators network.
Reduce churn rate of end-users directly or indirectly
A robust system that is easily upgraded to meet the rapid evolution
of mobile technology.
A platform that will facilitate actionable decision making to the
benefits of all parties involved with little or no OPEX.
Create a level playing field for all operators and protect the interest
of all consumers not just those in the urban areas.
Kampala, Uganda, 25 June 2014
19
Network Management Systems
EXAMPLES OF NMS solutions in the market today
For REGULATORS (Vendor-Independent)
Regulators (QoS)Performance Management (RPM)system- compatible
with all major network vendors (Huawei, ZTE, Ericcsson, Alcatel and NSN)
FOR OPERATORS (Vendor –dependent)
imanager M2000 for Huawei vendor
NetNumen U31 for ZTE vendor
NEtAct for Nokia Siemens Vendor
NetOp EMS for Ericsson and Omnivista 2500 for Alcatel vendor
Kampala, Uganda, 25 June 2014
20
RPM SYSTEM OVERVIEW
Regulators’ (QoS) Performance Management System is the novel NMS solution that
handles the task of interfacing all the operators’/service providers’ network
monitoring systems, collect perfomance data records and create KPI reports that
renders a given network performance against published benchmarks.
Main Features
• QoS monitoring and reporting 24/7 all year round
• Unlimited number of Users/ Single License
• Multi vendor Multi access technology
• Automatic Reporting and 3D GIS
display(compatible with google earth
• Friendly and intuitive web user interface
• No running costs and upgrades easily as mobile
access technology evolves
• Built in compliance with ETSI TS 102 250-4&7/
ETSI EG 202 057-3 and ETSI EG 202 009-2
Kampala, Uganda, 25 June 2014
21
Conclusion
The assessment of the QoS/QoE is expected to be evaluated in
checking criteria against reference values. These criteria are
measured either objectively via technical means or subjectively via
surveys amongst the users. Experts agree that a mix of
objective(intrusive and non-intrusive) and subjective
measurements remains the best means to get the whole QoS
picture.
Today, four out of five Regulators only employ Drive Test(intrusive)
both for one-time snap shot, benchmarking and continuous QoS
monitoring(reactive sense). Given DTs drawbacks and inaccuracy
with respect to continuous QoS monitoring, its high-time
Regulators, gaining from operators’ perspective, adopted the novel,
practical and more efficient approach – the use of Network
Management Systems (NMS) for proactive continuous QoS
monitoring.
Kampala, Uganda, 25 June 2014
22
Recommendations
Regulators, as recommended in ETSI EG 202 009-1
clause 7, in the quest for effective continuous QoS
monitoring and assessment should perform the
measurements themselves and make the results
available publicly rather than entrust in mobile
operators to provide the QoS information
Thus Regulators should procure RPM system or any
other NMS that facilitates the complex task of
enforcing mobile operators to adhere to agreed
KPIs, put a check on service degradations /outright
outages and even penalities for repeated failure to
achieve minimum QoS/ QoE targets.
Kampala, Uganda, 25 June 2014
23
THANK YOU FOR YOUR
ATTENTION
For more information on RPM
system or any other NMS
Feel free to contact me
or email
cblanchard@planetworkint.com
Kampala, Uganda, 25 June 2014
24