ITU Workshop on “Monitoring and Benchmarking
of QoS and QoE of Multimedia Services in
Mobile Networks”
(Dubai, UAE 2-3 November 2014)
Application and segment driven QoS
measurements and benchmarking
Esa Vesterinen
Vice President
Omnitele
esa.vesterinen@omnitele.com
Topics discussed
• Application QoS – Be the customer
assessment
• Case studies:
•
•
Commuting traffic in Finland
Dutch football experience
©Omnitele Ltd. 2014
2
Quality perception is application and segment dependent
Location

Different segments have different QoS needs: service – location – time

Average QoS hides the most important - user segments’ customer experience
There is no average customer, nor average quality
©Omnitele Ltd. 2014
3
Be-the-Customer QoS Assessment
TARGETED
one segment at a time
©Omnitele Ltd. 2014
4
REAL USAGE
Segment dependent
MEANIGNFUL
service level KPIs
Our recent case examples...
WWW browsing in
commuting trains
Marketing Benchmark
Telia-Telenor challenge
WWW Browsing for
Dutch football fans
©Omnitele Ltd. 2014
5
International Roaming
Business travellers
CASE PASSENGER:
WWW Browsing Experience in Train
©Omnitele Ltd. 2014
6
Segment example: The “Passenger”
HOME (HYVINKÄÄ)
1 HOUR IN TRAIN
WORK (HELSINKI)
WiFi
Cellular Data
WiFi
“Passenger” relies on mobile data only while commuting to work 2 hours/day:
Hyvinkää-Helsinki-Hyvinkää. On WiFi at home & office.
For “Passenger”, macro-cellular average QoS is irrelevant
©Omnitele Ltd. 2014
7
The Passenger is not alone
#Travels annually
< 0.1 M
0.1 – 0.5M
0.5 – 1.0M
1.0 – 2.0M
1.0 – 3.0M
3.0 – 5.0M
>5.0 M
©Omnitele Ltd. 2014
8
Location and time dependent usage – Passenger
QoE lost in noise
The passengers visible in
0.3% sites
~6000 sites example network in total
~20 sites on “Passenger” rail route
The passengers counts for
0.14% time
24 h: typical aggregation in statistics
2 min: one passing train in typical cell area
But daily ~10 000 Passengers only see the in-train QoS
©Omnitele Ltd. 2014
9
Cluster average could be OK even if the in-train QoS was horrible!
©Omnitele Ltd. 2014
10
Be-the-Customer Measurements
3 Operators benchmarked:

Location: Train 100km Hyvinkää-Helsinki-Hyvinkää

Time: to/from work morning 7:30-9:00 / evening 16:00-18:00

Service: WWW Browsing, commercial SIM cards & iPhone 5s
©Omnitele Ltd. 2014
11
Results: Poor WWW Success
Only 77% of all WWW page retrievals are successful (N=1090)
WWW Page Success Ratio (%)
100%
99%
80%
99%
98%
87%
77%
60%
67%
40%
20%
0%
Operator X
Operator Y
Reference Cluster
©Omnitele Ltd. 2014
12
In-train
Operator Z
Results: Slow WWW Surfing
In train experience is 2-3 times slower than in network reference
clusters
WWW Page Waiting Time (s)
10
9.2
8
7.9
6
6.6
4.9
4
2
3.8
3.7
0
Operator X
Operator Y
Reference Cluster
©Omnitele Ltd. 2014
13
Operator Z
In-train
Poor in-train WWW success not location specific
Hyvinkää
WWW success, Operator Y
Successful WWW retrieval
Drop / Time-out
Helsinki
©Omnitele Ltd. 2014
14
Case take-out
 Different user segments have very
different needs and usage patterns
 Average QoS easily hides the
segment’s experience
 Need to address targeted segments
with meaningful application KPIs
©Omnitele Ltd. 2014
15
CASE FOOTBALL FAN:
WWW Browsing Experience
– WC2014 in Amsterdam
©Omnitele Ltd. 2014
16
Web Browsing Customer Experience in
Amsterdam - Football WC2014 experience
CEM
sol ut i on
CHALLENGE | best mobile WWW browsing experience?
HOW | WC2014 related tests with Anite’s CEM Observer
WHERE | locations where football fans watch the games
©Omnitele Ltd. 2014
17
Benchmark Overview
38 test locations in
Amsterdam where fans
watch the WC2014
games
Measurement profile built on football
experience
browsing of WC2014 related pages:
results, stories, statistics, interviews
http://www.fifa.com/worldcup/
http://www.nusport.nl/wk-nieuws
http://www.telegraaf.nl/telesport/wk-voetbal-2014/
http://nos.nl/wk2014/schema
http://www.speelschemawk2014.com/
http://www.liveuitslagen.nl/voetbal/wereld/wk/
http://www.vi.nl/dossier/dossier-alles-over-het-wk-2014in-brazilie.htm
©Omnitele Ltd. 2014
18
Benchmark Overview
38 test locations in Amsterdam
where fans watch WC2014 games
©Omnitele Ltd. 2014
19
CEM
sol ut i on
Measurement profile built on WC2014
results, stories, statistics, interviews
http://www.fifa.com/worldcup/
http://www.nusport.nl/wk-nieuws
http://www.telegraaf.nl/telesport/wk-voetbal-2014/
http://nos.nl/wk2014/schema
http://www.speelschemawk2014.com/
http://www.liveuitslagen.nl/voetbal/wereld/wk/
http://www.vi.nl/dossier/dossier-alles-over-het-wk-2014-inbrazilie.htm
Results: the delusion of bitrate
importance…
end-user bitrate [Mbps]
15
10
12.8
10.8
8.8
5
If bitrates were benchmarked,
T-Mobile has the best service
0
WWW page waiting time [s]
6
4
5.6
5.5
2
0
©Omnitele Ltd. 2014
20
5.7
…but in customer experience
operators are neck and neck
who offers the best customer
experience?
14
Best in most
locations
Fastest browsing
on average
Highest throughput
on average
6
12
10
4
8
26%
6
39%
2
4
2
0
10.8
©Omnitele Ltd. 2014
12.8
21
8.8
5.5
0
5.6
5.7
34%
KEY MESSAGES
1
Bitrate not self-sufficient to
represent Customer Experience
2
Different result aggregations
may yield different conclusions
3
Customer Experience measurements give
means to focus network optimisation
©Omnitele Ltd. 2014
22
Optimising Mobile Web Customer Experience
©Omnitele Ltd. 2014
23
Average throughput tells nothing - domain
performance varies greatly
WWW page waiting time [s] - domain performance
10
8
Operator average 5.5 – 5.7 s
6
4
2
0
always poor?
©Omnitele Ltd. 2014
24
always good?
websites differ but expectations don’t
#content elements
data volume [kB]
25
193
©Omnitele Ltd. 2014
25
477
2400
customer
expectations
are same
Test sites: http://www.fifa.com/worldcup; http://www.telegraaf.nl/telesport/wk-voetbal-2014/
Frustration level of web
browsing
5
“very fast”
>5s  often frustration
expectations vary among
markets, segments, time…
“acceptable”
“a bit too slow”
4
“too slow, annoying”
“out of my tolerances”
3
2
1
0
2
4
6
8
10
WWW page waiting time [s]
[QoE results from Omnitele case study 2013]
©Omnitele Ltd. 2014
26
12
14
Focus your optimisation effort to right
areas
CEM
sol ut i on
WWW page waiting time [s] - domain performance
10
8
6
“frustration threshold”
4
2
0
Focus: internet connectivity
©Omnitele Ltd. 2014
27
Focus: Local optimisation
www waiting time [s]
Operator X: Identifying the locations
for RAN optimisation…
4, 22, 24, 30, 32: telegraaf.nl
OK  probably little or no
gain with radio optimisation
20
CEM
sol ut i on
36, 38: fifa.com & telegraaf.nl
correlates, potential gain from
radio optimisation
15
10
5
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38
location ID
fifa.com
©Omnitele Ltd. 2014
28
telegraaf.nl
Example: Long web browsing loading time for operator X
1: Due to poor LTE coverage, the
mobile is camping on UMTS2100.
2: When trying to visit a website,

the mobile attempts to access
the RACH, but fails.

Only after 5 seconds, an
attempt is successful.

The mobile sends
measurement event 4a and

is immediately sent from
carrier 10836 to carrier 10811,
where the data transfer starts.
3: Even though radio condition are good and download of
the website is not yet complete, a physical channel
reconfiguration reduces the throughput from 7M Mbps to
700 kbps.
This reconfiguration combined with the RACH access
problems create an extremely long web page loading
time.
©Omnitele Ltd. 2014
29
Solution:
Investigate the capacity of the UMTS cell
1. on carrier 10836 (RACH failure) and
2. 10811 (dropping throughput).
Case take-out
1
Dutch football fans have certain
expectations where and how they
can access their interest sites
2
www waiting time was measured and
variations were analysed
3
network performance was measured
simultaneously and root cause
analysed and solved
©Omnitele Ltd. 2014
30
Questions?
For any further info, kindly refer
esa.vesterinen@omnitele.com
+358 44 324 7668
www.omnitele.com
©Omnitele Ltd. 2014
31
Omnitele QoE management services
ASSESS
customer experience benchmark
IMPROVE
customer experience optimisation
CONTROL
QoE driven performance management
©Omnitele Ltd. 2014
32
max imis ed c u s t omer ex p erien ce
min imised n et w ork cost
©Omnitele Ltd. 2014
33
www.omnitele.com
why we exist
Omnitele is an international telecommunications
consulting and engineering company. We provide
services for telecom operators and regulators in
domains of network strategy, design and quality
assurance. Our mission is to maximise mobile
subscriber quality of experience and minimise
operator network cost.
presence
Our headquarters is located in Helsinki, Finland.
We have local presence in the Netherlands and
the Hashemite Kingdom of Jordan. Our company
is owned by Finnish telecom investors and we are
independent of all international operators and
telecom network infrastructure equipment
vendors.
track record
We have solid track record and over 25 years of
experience in telecoms industry. The company
was founded in 1988 to set up world’s first GSM
operator and network. Since then we have
completed 1000+ projects in 80+ countries.
omnitele way
Our unique way of working sets us apart from the
competition. We call this the Omnitele Way: being
Straightforward, Trusted and Intelligent. We
deliver tangible results and ensure excellent
Omnitele Experience.
©Omnitele Ltd. 2014
34
www.omnitele.com
our services
Technology
Strategy
Design and
Optimisation
Audit and
Benchmark
Performance
Management
maximised customer experience
minimised network cost
©Omnitele Ltd. 2014
35