Quality of Service of Over-The-Top Services
Cyril Lau
ITU Athens QoS and QoE of OTT 2015 © Ascom
1
Presentation Objective and Agenda
Presentation Objective
 To discuss the Quality of Experience (QoE) aspects of over-the-top (OTT) and
social media services
Agenda
 Rise of OTT and mobile social media apps
 Factors affecting subscriber experience and quality
 Case study – OTT QoE measurement
 Whatsapp, Facebook, youtube
 Q&A
ITU Athens QoS and QoE of OTT 2015 © Ascom
2
[ ]
OTT SERVICES OVERVIEW
ITU Athens QoS and QoE of OTT 2015 © Ascom
3
Rise of OTT services
- OTT driven by:
- faster, more reliable data connectivity on cellular mobile networks
- Smartphone devices with better performance, interfaces and larger screens
- Better codec development for voice and video transmission
- Method of accessing a new market of users formerly controlled solely by
operators
340m
users
ITU Athens QoS and QoE of OTT 2015 © Ascom
1bn users
4
Types of OTT Services
- Social Media
- Twitter, Facebook, Instagram
- Communications
- Viber, Skype, WhatsApp
- Content
- Netflix, YouTube
- Trend is that social media and communications functions are consolidating
(Facebook purchase of WhatsApp etc)
ITU Athens QoS and QoE of OTT 2015 © Ascom
5
[ ]
FACTORS AFFECTING QUALITY OF EXPERIENCE
ITU Athens QoS and QoE of OTT 2015 © Ascom
6
The Carrier Challenge
 Service quality is critical to maintaining subscriber loyalty, even
though some areas are outside operator’s control
 Customer Satisfaction impacted by:
Performance
Quality
Availability
ITU Athens QoS and QoE of OTT 2015 © Ascom
7
LTE QoE- QoS - KPIs MAPPING
Network
Performance
Customer
Experience
MME / S-GW
MME / S-GW
S1
S1
S1
S1
X2
E-UTRAN
eNB
eNB
X2
X2
eNB
ITU Athens QoS and QoE of OTT 2015 © Ascom
8
Factors Affecting Availability
- Radio environment/coverage
- Mobility issues with handovers, missing neighbours etc. giving interruptions
to data sessions
- Core Network
- Including DNS, routing, IP, transport issues
- Content Provider Servers
- Capacity, resilience, redundancy issues
ITU Athens QoS and QoE of OTT 2015 © Ascom
9
Factors Affecting Performance
- Throughput
- Low throughput creates buffering on video streaming, poor download
speeds for files
- Latency
- Jitter and delay in packets causing interruptions with real-time UDP streams
ITU Athens QoS and QoE of OTT 2015 © Ascom
10
QoE expectations for each OTT is different……
 Video Streaming – requires good bandwidth. Not as much in latency as long
as the buffer can handle.
 Online Gaming – requires good latency. Not as much in Bandwidth.
 Social media – high availability is essential to keep messages/posts up-todate
 Online Shopping – Security and availability is essential. Lower requirement
on bandwidth and latency
ITU Athens QoS and QoE of OTT 2015 © Ascom
11
Other Non Network Related Factors Affecting Quality
Quality of Experience is a function of the performance and availability of
the underlying network and services, but also other factors such as:
- Encoding and delivery mechanism (video/voice)
- Adaptive content systems or lack of
- Poorly encoded content
- Devices
- Poorly designed radios, CPU and memory restrictions, operating systems
- Service Design
- Interfaces, protocols
ITU Athens QoS and QoE of OTT 2015 © Ascom
12
[ ]
CASE STUDY – OTT QOE MEASUREMENT
ITU Athens QoS and QoE of OTT 2015 © Ascom
13
What to consider when measuring OTT services?
From the consumer side:
 Can I access the service?
 Are my messages sent and received in a timely fashion?
 Do I receive notifications correctly?
 Is the audio quality good?
From the operator side:
 What is the uptime of the service?
 How fast are messages delivered?
 What average MOS/POLQA do customers get in a voice call?
ITU Athens QoS and QoE of OTT 2015 © Ascom
14
ITU Athens QoS and QoE of OTT 2015 © Ascom
15
Whatsapp case study:
Whatsapp signon
https://r.whatsapp.net
Ping
r.whatsapp.net
DNS lookup
r.whatsapp.net
We can test the QoE of whatapp services with real devices:
1. DNS lookup
2. Ping
3. WhatsApp signon
But is this enough for an operator?
ITU Athens QoS and QoE of OTT 2015 © Ascom
16
Whatsapp case study
User Device
Mobile network operator
Whatsapp
Clients
RAN
Core
network
Whatsapp
Whatsapp
Server
The whatsapp OTT value chain
 How can an operator ensure whatsapp service delivery over the mobile
network? Remember:
 RAN and Core Network are controlled by the operator
 User device and whatsapp servers are outside operator’s control
ITU Athens QoS and QoE of OTT 2015 © Ascom
17
Whatsapp case study: 24 by 7 monitoring and
location of errors
Cell colour indicates average error
DNS
Ping
Whatsapp
1
2
3
Locations
4
1
Clients
2
3
Different locations or
access methods
ITU Athens QoS and QoE of OTT 2015 © Ascom
4
Vertical red
means
general failure
Horizontal red
means error at a
specific location
Different resolution actions in
each case
18
Whatsapp case study: Results
OTT “value
chain”
Core
network
WhatsappS
ervice
By understanding expected
behavior Monitor Master can
identify and alert on failures
DNS time is the time to find
the IP address of the server
Ping time is the time to locate
the server
WhatsApp time is the time to
connect and login to the
server
Sign on time to WhatsApp
application ~0.2s
ITU Athens QoS and QoE of OTT 2015 © Ascom
19
Whatsapp case study: Pinpointing problems
Correlation between Ping
and Logon times shows
when an issue is outside the
control of the operator
ITU Athens QoS and QoE of OTT 2015 © Ascom
20
Facebook
“Be a subscriber” using
test equipment to
measure real
experience
This example
demonstrating a test to
perform a FB update
and measure time
taken and outcome
ITU Athens QoS and QoE of OTT 2015 © Ascom
21
Facebook benchmarking between carriers
Facebook
Internet
Internet
Complex
multi party
interactions
Carrier3
Carrier1
Carrier
RANs
Carrier2
Clients
TEMS Monitor Master is used to actively test Facebook interactions for different carriers
Different aspects of the service interaction must be supported by the carrier network
Facebook
“value chain”
Interactions include logon, messaging, content upload, homepage display
Results used to improve core network and interworking connection
Results also important for marketing
ITU Athens QoS and QoE of OTT 2015 © Ascom
22
SOLUTION OVERVIEW
TEMS Monitor Master probes set up with scripts to test Facebook interactions
Active testing access using multiple different carriers
Tests simulate different clients such including PC and Smartphone and use different bearers
Tests exercise all the important functionality of Facebook from the user’s perspective
Engineering detail gives radio and IP trace to help pinpoint where issues occur
Multiple reports configured specifically to meet the customer’s requirements and add value
ITU Athens QoS and QoE of OTT 2015 © Ascom
23
Detailed results and Value
Facebook
Carrier
RANs
Reports shows average performance
of key Facebook interactions for
different carriers
Facebook
“value chain”
Shows at a glance how different carriers
are performing
Shows how different aspects of the
Facebook service perform
Results used to support marketing
campaigns
Detailed results used to troubleshoot
errors and improve MTTR for the owning
carrier
Differences between bearer also used to
troubleshoot
Different clients also used to give further
insight into performance
ITU Athens QoS and QoE of OTT 2015 © Ascom
24
ITU Athens QoS and QoE of OTT 2015 © Ascom
25
Case Study - Video test per geographic location
Non compressed video
for regular users during
off peak hours
Compressed video for
regular users during
peak hours
APN 1
GGSN 1
APN 2
GGSN 2
…
…
APN X
GGSN Y
3rd party video
streaming
service
3rd party live TV
3rd party
subscription TV
Compressed video for
“all you can eat data”
users at all times
TEMS Monitor Master is used to actively test video streaming services
Content delivery network must deliver video with different encoding schemes correctly
Access point names (APNs) must deliver different service correctly
Mobile network must deliver from geographical locations 24x7
Video Compression applied in different scenarios must be within certain QoE
ITU Athens QoS and QoE of OTT 2015 © Ascom
26
Case Study - Video compression per subscriber profile
3rd party video
streaming
service
Video CDN
Non compressed video
for regular users during
off peak hours
Video
optimisation
and load
balancing
Compressed video for
regular users during
peak hours
3rd party live TV
3rd party
subscription TV
Compressed video for
“all you can eat data”
users at all times
TEMS Monitor Master is used to actively test video streaming services
Content delivery network must deliver video with different encoding schemes correctly
Access point names (APNs) must deliver different service correctly
Mobile network must deliver from geographical locations 24x7
Video Compression applied in different scenarios must be within certain QoE
ITU Athens QoS and QoE of OTT 2015 © Ascom
27
Video Quality Comparison: different compressions
Reports show key video quality KPIs for different compression methods
Video size is reduced for compressed videos
Content
Delivery
Network
Video
content
Clients
MOS is lower for compressed video as expected, but still within acceptable
ranges
Stalling/re-buffering % is similar for both compressed/non-compressed videos
Video CDN
value chain
ITU Athens QoS and QoE of OTT 2015 © Ascom
28
[ ]
CONCLUSION
ITU Athens QoS and QoE of OTT 2015 © Ascom
29
Conclusion
 Think about how users interact with OTT services, and what factors affect
their experience
 Identify methods of measuring those factors in the OTT delivery chain,
especially the areas within operator’s control
 Quantify and measure those factors through active testing
 Act to improve those factors and measure impact
ITU Athens QoS and QoE of OTT 2015 © Ascom
30
[ ]
Q&A
ITU Athens QoS and QoE of OTT 2015 © Ascom
31
[ ]
THANK YOU!
ITU Athens QoS and QoE of OTT 2015 © Ascom
25 YEARS OF INNOVATION
32