ITU Workshop on “Performance, Quality of Service and Quality of
Experience of Emerging Networks and Services”
(Athens, Greece, 7-8 September 2015)
Network Performance vs. Application
Testing; findings and considerations
Henry Kwong
Senior Consultant, Omnitele
henry.kwong@omnitele.com
agenda
© Omnitele Ltd. 2015
2
1.
Introduction
− About Omnitele
2.
Beyond bitrates
Drive Testing vs. App Testing
3.
OTT delivery chain
Understanding the QoS results
4.
Key take-aways
Summary
Maximised Experience, Minimised Cost
International And Independent
We provide consulting and expert services for telecom
operators and regulators in network strategy, design
and quality assurance. Our mission is to maximise
mobile subscriber quality of experience and minimise
operator network expenditures.
Our headquarters is located in Helsinki, Finland. We
have local presence in the Netherlands. Our company
is owned by Finnish telecom investors and we are
independent of operator groups and network vendors.
Delivering Omnitele Experience
The Omnitele Way
The company was founded in 1988 to set up world’s
first GSM network. Since then we have completed over
1000 projects in over 80 countries around the globe.
Always delivering Omnitele Experience – a fact proven
by our long lasting client relationships.
Our unique way of working sets us apart from the
competition and gives us a strong identity in the world
of telecommunications. We call this the Omnitele Way,
which means being Straightforward, Trusted and
Intelligent.
www.omnitele.com
© Omnitele Ltd. 2015
3
our services
Technology
Strategy
Design and
Optimisation
Audit and
Benchmark
Performance
Management
maximised customer experience
minimised network cost
© Omnitele Ltd. 2015
4
agenda
© Omnitele Ltd. 2015
5
1.
Introduction
− About Omnitele
2.
Beyond bitrates
Drive Testing vs. App Testing
3.
OTT delivery chain
Understanding the QoS results
4.
Key take-aways
Summary
Monitoring End-User QoS in…
…pre-HSPA networks
General user experience
depends on two basic services
Voice Services
Data Services
MOS, Success Rates, Drop
Rates, Setup Time
Throughput
© Omnitele Ltd. 2015
6
Generally available bitrates are not sufficient for data
services. As throughput is the bottleneck, it becomes the
measure to define the QoS
Customer Experience Challenge
Bitrate is no longer the ultimate mobile
network quality indicator
QoS: WWW page waiting time [s]
1000KB WWW Page Waiting Time vs. Bitrate
HSPA
10
LTE
8
6
4
2
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20
NW Performance: Bitrate [Mbit/s]
© Omnitele Ltd. 2015
7
Customers mind service quality, not
network performance
Subscribers using (%)
Mobile data use cases and data volume
80%
72%
Share of data volume (%)
70.5 %
 Customers desire smooth
experience, not average bitrate
60%
40%
 WWW & Video dominate usage
20.0 %
20%
20%
7%
3.3 %
5%
1.1 %
0%
General Browsing
© Omnitele Ltd. 2015
8
Video Streaming
P2P
Gaming
5%
0.3 %
VoIP
4.7 %
M2M
Competitive end-user QoS benchmark
OBJECTIVE: gain total visibility to mobile
service experience & quality positioning
METHODOLOGY: “be the customer” approach to capture end-user experience
RESULT: a high quality benchmark
report suitable for public distribution
© Omnitele Ltd. 2015
9
Capturing the End-user Experience
Independently selected dense
population areas & landmarks
WHERE
WHEN
Test focus on peak hours, no
empty network testing
HOW
Terminals & test cases as per
real subscriber behaviour
© Omnitele Ltd. 2015
10
Customer Experience Challenge
Case study: Omnitele benchmark in Netherlands
big difference in NW performance…
…thin margins in Quality of Service
bitrate [Mbit/s]
buffering time [s]
40
2
30
1.5
20
1
10
0.5
0
0
T-Mobile
Vodafone
KPN
T-Mobile
Vodafone higher bitrate,
but KPN faster YouTube
© Omnitele Ltd. 2015
11
Vodafone
KPN
Drive Testing vs. App Testing
Case study – Omnitele benchmark in Estonia
Traditional Drive Testing
App Testing
Insta
Estonia
FTP Downlink
Operator 1
Operator 2
Operator 3
© Omnitele Ltd. 2015
Average (Mbps)
43.8
24.3
20.3
12
Estonia
Dropbox DL
Operator 2
Operator 3
Operator 1
Average (Mbps)
13.5
12.8
11.1
Drive Testing vs. App Testing
Case study – Omnitele benchmark in Estonia
Traditional Drive Testing
App Testing
Insta
Estonia
LTE RB Utilisation
Operator 1
Operator 2
Operator 3
Average (%)
91.2
80.0
72.8
Estonia
LTE RB Utilisation
Operator 2
Operator 3
Operator 1
Average (%)
21.9
26.6
19.9
Results from Drive Testing and App Testing do not match at
all. Network usage patterns are completely different
© Omnitele Ltd. 2015
13
Customer Experience Challenge
all-IP PS network
OFDMA access scheme
spectral efficiency 15 bit/s/Hz
SC-FDMA uplink
adaptative modulation
low latency
300 Mbps
20 MHz bandwidth
Carrier Aggregation
improved cell-edge performance
8x8 MIMO
© Omnitele Ltd. 2015
14
Customer Experience Challenge
all-IP PS network
news refreshing time?
OFDMA access scheme
app download
spectral efficiency 15 bit/s/Hz
time?
SC-FDMA uplink
it takes too long to
adaptative
upload
a pic in modulation
Instagram…
low latency
300 Mbps
© Omnitele Ltd. 2015
15
20 MHz bandwidth
facebook
Carrier Aggregation
refreshing
improved cell-edge performance
time?
8x8 MIMO
Customer Experience Challenge
Where to focus your monitoring efforts?
Insta
Depends on the objectives that
you want to achieve
Network
Performance
© Omnitele Ltd. 2015
16
End-User
QoS
agenda
© Omnitele Ltd. 2015
17
1.
Introduction
− About Omnitele
2.
Beyond bitrates
Drive Testing vs. App Testing
3.
OTT delivery chain
Understanding the QoS results
4.
Key take-aways
Summary
OTT Delivery Chain in Mobile Network environment
OTT Content
Delivery Platforms
Environment NOT
controlled by service
provider
Insta
Internet
Request
Response
GGSN/PGW
Core
3G
© Omnitele Ltd. 2015
18
Radio
Access
4G
Environment
CONTROLLED by
service provider
OTT Delivery Chain in Mobile Network environment
OTT Content
Delivery Platforms
•
Environment is not fully
controlled by network
operator
•
OTT measurement results
will not depend only on
operator performance
•
Special attention should be
paid when analysing OTT
performance results (Facebook
Insta
Internet
Request
Response
waiting times, YouTube buffering time, etc.)
GGSN/PGW
•
Core
3G
© Omnitele Ltd. 2015
19
Radio
Access
4G
Multiple points of
measurement within
controlled environment
would facilitate problem
isolation
agenda
© Omnitele Ltd. 2015
20
1.
Introduction
− About Omnitele
2.
Beyond bitrates
Drive Testing vs. App Testing
3.
OTT delivery chain
Understanding the QoS results
4.
Key take-aways
Summary
Summary
1. Data throughput is no longer the unique KPI to take into
consideration when monitoring Mobile Data QoS
2. Network Performance vs. App Testing – depending on your
own objectives you may setup your measurements one way or
another
− Use of network resources differs and thus, results obtained
3. OTT delivery chain is not fully controlled by service provider,
thus results may be influenced by external factors
− Setting up different measurement points within controlled environment will
assist in troubleshooting potential issues
© Omnitele Ltd. 2015
21
thank you for your attention!
Questions?
© Omnitele Ltd. 2015
22
We answer
Contact details
Omnitele Ltd
Name
Henry Kwong
Phone
+358 9 695 991
Title
Senior Consultant
Email
contact@omnitele.com
Phone
+358 440 432 900
Website
www.omnitele.com
Email
henry.kwong@omnitele.com
Address
Omnitele Ltd. Mä kitorpantie 3B P.O. Box 969 00101 Helsinki, Finland
© Omnitele Ltd. 2015
23
maximised customer experience
minimised network cost
© Omnitele Ltd. 2015
24
www.omnitele.com