[ ]
ITU-T ATHENS
FUNCTIONS AND MONITORING OF THE QOE AND QOS IN 4G
NETWORKS/LTE.
Cyril Lau
ITU-T, Athens, September 2015
Who we are
Independent Auditor
+30
patents
worldwide
1
Innovative
More than
Leader
30 patents
worldwide
2
Cyril Lau
ITU-T, Athens, September 2015
Next-generation
technology
ITU-T
ETSI
VQEG
Wireless
network
performance
3
Formative Player
Agenda
• LTE QoE- QoS- KPIs MAPPING RELATIONSHIPS
• SOME CHALLENGES EMERGED FROM THE LTE ECOSYSTEM
• ASCOM LTE QoS/QoE TESTING APPROACHES
• SHARING EXPERIENCE: VoLTE Testing
• CONCLUSIONS
Cyril Lau
ITU-T, Athens, September 2015
[ ]
LTE QOE- QOS- KPIS MAPPING RELATIONSHIPS
Cyril Lau
ITU-T, Athens, September 2015
ASCOM APPROACH WITHIN THE ITU-T PERSPECTIVE ON QoS/QoE CYCLE
(ITU-T G.1000)
Network/service provider
Subscriber/user
QoS Requirements of Subscriber/User
OR
QoE Requirements
Execution gap
Value gap
QoS Perceived by Subscriber/User
OR
QoE of the User
QoS Targeted by Provider
(Target SLA)
Alignment gap
QoE/QoS
Cycle
Perception gap
KPI, QoS
QoS Delivered/Achieved by Provider
(Achieved SLA)
QoE
Subscriber-centric
Quality (QoE)
Cyril Lau
ITU-T, Athens, September 2015
Network-centric
Quality (QoS)
LTE QoE- QoS - KPIs MAPPING
Network
Performance
Customer
Experience
MME / S-GW
MME / S-GW
S1
S1
S1
S1
X2
E-UTRAN
eNB
eNB
X2
X2
eNB
Cyril Lau
ITU-T, Athens, September 2015
COST EFFICIENT TOP DOWN APPROACH ENABLED BY PREDEFINED LTE QoE-QoS-KPIs MAPPINGS
Increased
operational efficiency
Top-down customer
experience centric
approach
Voice Service: ‘97 Auryst,
‘03 PESQ, ‘12 POLQA
Video-Audio Services: ‘08
VSQI/MTQI, ’12 PEVQ, ‘12
VQmon
Application Layer
(integrity, accessibility, retainability)
QoE
Video Streaming QoE Behavior
QoS / SLAs
QoS/KPI
Upper Layers (IP/User Data Protocol , IP/Transport
Control Protocol): Throughput, Delay, Packet Loss
Layer 2& Layer 3 messaging reports related to QoE
dimensions e.g. Packet Data Protocol context, HO info,
codec usage),
4
3
2
1
0
MOS - QoE
packet Loss
0
50
Time (sec)
100
Re-buffering
Network performance (KPIs)
Physical layer
(e.g. coverage, interference)
Reduce troubleshooting size and time by using QoE centric mapping to QoS and KPIs per service type
Cyril Lau
ITU-T, Athens, September 2015
7
[ ]
SOME CHALLENGES EMERGED FROM THE LTE ECOSYSTEM
Cyril Lau
ITU-T, Athens, September 2015
THE LTE ECOSYSTEM CHALLENGE – Why QoE?
MOBILE VOIP
(0.3% OF DATA
CONSUMPTION)
MOBILE VIDEO
(70.5% OF DATA
CONSUMPTION)
Cyril Lau
ITU-T, Athens, September 2015
ACCOUNT FOR
~70% OF TRAFFIC
ACCOUNT FOR
~?% OF REVENUE
VoLTE-ViLTE-RCS EXPERIENCE
Call quality as perceived by users
 MOS / ITU-T P.863; J.247 & P.120x.x based
(video)
Codecs
(KPI)
 On device VoLTE client (MTSI client): rebuffering/time scaling for voice; error
concealment type for video
 Voice path delay (“mouth to ear”), echo, video –
voice lip sync
Devices
Clients
(KPI)
 RTP Packet loss, latency, jitter
Call
Experience
(QoE)
 HOIT (LTE HO, eSRVCC)
 Throughput
Network
(QoS/KPI)
 Voice /video codec type and bit rates
Call control performance
Session set-up: SIP signaling statistics / IMS Registration, RTT
Session Accessibility, Retainability
QCI allocation verification
LTE RRC connection and HO statistics
Cyril Lau
ITU-T, Athens, September 2015
VoLTE QoE AND ITS ROOT CAUSES (QoS, KPIs SOURCES)
Perceived
frequency
Spectrum (QoE)
Interruptions
(QoE)
(incl. time
clipping)
Network (QoS):
Limited
Bandwidth
Network
(QoS/KPI)
Device (KPIs):
Spectral
shaping
Reverberations
Device based
signal processing (KPIs) (e.g.
(IP/IMS loss, jitter,
RAN erroneous bits,
RAN HO)
NR, EC)
Codec/client
(KPIs):
PL concealment
schemes
Aggressive
VAD schemes
Cyril Lau
ITU-T, Athens, September 2015
“Mouth to ear
delay” (QoE)
Network (QoS)
(IMS path can be
key contributor)
Codec/device
signal
processing
(KPIs)
Noisiness (QoE)
(incl. musical
noise)
Network (QoS):
Limited BWD with
noisy speech
(speech
contamination)
Device (KPIs):
Imperfect NR
(musical noise)
Loudness
during silent
periods
Non-optimal
loudness levels
Codec/client
(KPIs)
PLC interpolation
based (“additive
artifacts”)
Perceived call
session
performance (QoE)
IMS network
(QoS/KPIs):
SIP statistics,
IMS registration
ITU-T on going work:
G.VoLTE,
P.TCA (Technical
Cause Analysis)
MOBILE VIDEO STREAMING EXPERIENCE
Codec types:
Terminals/devices/clients
High
• Different form factors
Low
• Different 2D/3D displays
Adaptive bit rates
• Adaptive error concealment schemes
Network centric:
Wide variety of content & bit rates
Packet loss
Jitter
Allocated GBR and QCI
HTTP/TCP vs. RTP/RTSP
Cyril Lau
Service-Centric
ITU-T, Athens, September 2015
MOBILE VIDEO STREAMING QOE AND ITS ROOT CAUSES (QoS,
KPIs SOURCES)
QoS
Low throughput (limited bandwidth)
MME / S-GW
Video resolutions/rescaling
Encoding/transcoding
rates
Compression schemes
Packet loss , discard (RTP), late arrival (TCP)
IP delays / jitter
S1
S1
S1
X2
E-UTRAN
Limited coverage, interference
eNB
eNB
X2
Bandwidth estimation
algorithms impacting the
bit rate selection
KPIs
S1
Possible additional transcoding
X2
KPIs
MME / S-GW
eNB
Content server load,
(competing video streams)
Content complexity (e.g.
low to high movement)
KPIs
Client under/over flow;
improper buffer lengths/adaptation
length
Initial buffering settings
Display resolutions, form factors
Cyril Lau
ITU-T, Athens, September 2015
Visual impariments
Bockiness,
Blueriness,
Jerkiness,
Freezing with and/or without skipping
Perceived service accessibility
/ access time
QoE
[ ]
ASCOM LTE QOE/QOE TESTING APPROACHES
Cyril Lau
ITU-T, Athens, September 2015
Measuring QoE:
TEST THE NETWORKS LIKE A REAL CUSTOMER IS USING IT
A CUSTOMER CENTRIC TESTING SOLUTION
 Enables testing of end user terminal
QoS settings and IP stack
characteristics
 Provides framework for future services
to be added easily and controlled by a
single client (e.g. Blixt implementation)
Cyril Lau
ITU-T, Athens, September 2015
Service M
IP Logging
 Repeatable “control scripts” supported
across multiple devices (repeatable
testing)
Service B
Call Control
First to test VoLTE-ViLTE as a user via
on-device VoLTE client , test everything
ON Device
 Server resides inside the terminal,
managing the supported services
(VoLTE, IP Logging, Call Control etc)
Service A
VoLTE
 Laptop-based drive test solution
(TEMS Investigation) maintains
control and coordination through an
On-Device Server
QoS setting per service
TCP/IP (SIP, RTP, etc)
On-Device
Server
UNDERSTAND AND USE ITU-T BASED QoE METRICS
BEST FITTED TO THE TESTED SERVICE;
LTE VOICE AND MOBILE VIDEO SERVICES
LTE Broadcast/
Streaming server
Video stream
(YouTube, eMBMS)
IP recordings (RTP KPIs/QoS)
synchronized with RAN KPIs
Encoding
Cellular Voice &
Data Network
Decoding
Conversational
Video
Solutions
Voice: MOS P.OLQA (ITU P.863)
Video: MOS PEVQ (ITU J.246)
Voice
Video
stream
Voice/Conversational Video
IP transport/payload
parameters
Full Reference listening media quality evaluation
(intrusive, perceptual) measurement
Direct RTP KPIs/QoS!
MOS, media based KPIs
Solutions
Voice: MOS - ITU P.564 based
Video-audio: MOS – P.120x.x based (VQmon)
Non Reference listening media quality evaluation
(non-intrusive, parametric)
MOS, network based KPIs
Cyril Lau
ITU-T, Athens, September 2015
[ ]
SHARING EXPERIENCE: VOLTE TESTING
Cyril Lau
ITU-T, Athens, September 2015
VoLTE MOS SCORE and VoLTE KPI’s/QoS
MOS
Speech
GSMA IR 92, IR 94
QoE Evaluation &
Troubleshooting
Speech & Client
centric reasons
Client
[TEMS Discovery]
[TEMS Investigation]
[TEMS Pocket]
POLQA; speech path delay, volume, echo
Client Information (re-buffering, codec)
ESM configuration (QCI, RoHC)
Real time IP trace & L3 logging
IP recording (RTP stats), SIP stats
RAN (HOIT, Scheduling, RSRP, CINR,
CQI, PMI/RI, UE category, MTU Size,
Protocol stack configuration)
Cyril Lau
ITU-T, Athens, September 2015
Network
Network centric
reasons
VoLTE CALL SESSION
Accessibility Statistics
Session Setup Delay (s)
[Count]
[All]
[MO]
[MT]
100
90
80
70
60
50
40
30
20
10
0
1
2
3
4
5
[sec]
Registration Delay (ms)
All
Power Up
Re-registration
% of Total
% of Total
100.0
90.0
80.0
70.0
60.0
50.0
40.0
30.0
20.0
10.0
0.0
50.0
40.0
45.0
35.0
40.0
30.0
35.0
25.0
30.0
20.0
25.0
20.0
15.0
15.0
10.0
10.0
5.0
1
2
3
4
5
6
5.0
0.0
0.0
1
Cyril Lau
IMS Latency (ms)
SIP Handshake Time (ms)
ITU-T, Athens, September 2015
2
3
4
5
6
1
2
3
4
5
6
VoLTE CALL SESSION
Retainability Statistics
Session Completion Success
Rate (%)
Session Duration Time
Session Disconnect Delay (ms)
%
40
300
35
250
2
30
200
[Count]
[Count]
3
150
100
50
1
2
3
4
[Duration]
90.0%
92.0%
94.0%
96.0%
98.0%
20
15
10
0
1
25
5
6
7
5
0
1
2
3
4
5
[ms]
100.0%
RRC Connection Setup Time (ms)
RRC Connection Setup
500
[Count]
400
300
1
200
2
100
0
1
2
[ms]
Cyril Lau
ITU-T, Athens, September 2015
3
VoLTE CALL SESSION
Retainability Analysis
Accessibility Analysis
Session Setup Failure Causes
RRC Connection Drop Causes
1. 1.
2.
1.
1
2
40.
3
4
Session Completion Failure Cause
5
6
8.33
25.
1
58.33
8.33
2
3
4
Cyril Lau
ITU-T, Athens, September 2015
VoLTE CALL SESSION
Integrity Evaluation
Conversation Start Delay (ms)
Speech Path Delay (ms)
All
MO
Acceptable Call Quality
MT
MOS Threshold=2.7
[Distribution]
70.0
60.0
250
5.
50.0
1
40.0
150
2
30.0
100
20.0
50
10.0
43.
0.0
0
1
2
3
1
2
3
4
5
6
7
[ms]
Interarrival Jitter
300000
250000
[Count]
[Count]
200
200000
150000
100000
50000
0
1
2
3
4
[ms]
Cyril Lau
ITU-T, Athens, September 2015
5
6
VoLTE CALL SESSION
Integrity Analysis
Handover Interruption Time Inter/IntraeNB - Control Plane (ms)
Low POLQA Category - Distribution
To be correlated to user perceived
interruptions (speech interruptions - to
be soon released by TEMS
100
90
80
[Count]
70
60
50
40
30
20
10
0
1
2
3
4
[ms]
Cyril Lau
ITU-T, Athens, September 2015
5
6
3G CS to VOLTE voice call
Now we know what QoE
metrics we can measure, how
do we locate the problem
area when there is an issue?
UMTS
PSTN
UE A
UE B
Sv
I2
Cx
Sh
Mr
Iu-CS
C
S6a
Mw
ISC
Rx
S5
S1-MME
Iu-PS
ISUP
Gn
Gm
SGi
Gx
S11
S1-U
Dh
Dx
Uu
Gr
RTP
LTE-Uu
More than 28 interfaces
1
VoLTE Service Monitoring Solution, © Ascom
Extract them from all around the NW
Check Compliance individually
Correlate them all together
24
Only one party VOLTE voice call
monitoring network architecture
TRANSPORT layer and traffic management
ISBC1
HSS2
HSS2
HSS1
MME1
PCRF1
ISBC2
ISBC1
PCRF2
PCRF1
P-CSCF1
I/SCSCF1
TAS1
MRF1
P-CSCF2
I/SCSCF2
TAS2
MRF2
P-CSCF3
I/SCSCF3
TAS3
MRF3
P-CSCF4
I/SCSCF4
TAS4
MRF4
P-CSCF5
I/SCSCF5
TAS5
MRF5
P-CSCF6
I/SCSCF6
TAS6
MRF6
SEG1
S-GW1
eNodeB1
P-GW1
SEG2
MME2
S-GW2
P-GW2
SEG3
MME2
eNodeB2
SEG4
S-GW2
P-GW3
EPC
2
IMS
More than 10 network elements involved
Consider a high availability Network (Network Element Failure ≠ Service Unavailability)
VoLTE Service Monitoring Solution, © Ascom
25
Ascom approach: Combined Core/RAN active testing
Test point 1:
E2E QoE
Test point 3:
IMS QoS
Test point 2:
E2E QoS
HSS
Rx
S6a
PCRF
MME
Rx
P-CSCF
S-CSCF
S1-U
S5/S8
SGW
Mr’
MRF
PGW
RTP
EnodeB
LTE
TAS
Gx
Gm
Uu
ISC
EPC
IMS
Emulated Interface
Internal Interface
Media Path
VoLTE Solution 2015 © Ascom
26
Measuring QoE: E2E testing
• E2E testing using a commercial UE.
• The SUT effectively includes the whole of oprators’s core and radio network, plus the UE itself!
• QoE is affected by UE firmware, SW app under test….etc
• This is useful for problem identification, but wide coverage makes it difficult to locate the point of failure.
TOOL
SUT (System Under Test)
HSS
Rx
S6a
PCRF
MME
Test UE
Rx
P-CSCF
S-CSCF
S1-U
S5/S8
SGW
Mr’
MRF
PGW
RTP
EnodeB
LTE
TAS
Gx
Gm
Uu
ISC
EPC
IMS
Emulated Interface
Internal Interface
Media Path
VoLTE Solution 2015 © Ascom
27
Measuring E2E delivery network QoS
• E2E testing using a simulated device under a known, controlled environment.
• Focus on testing the service delivery network (QoS), by eliminating the UE from the test.
TOOL
SUT (System Under Test)
HSS
Rx
S6a
UE
simulation
PCRF
MME
Rx
P-CSCF
S-CSCF
S1-U
TMM
SW
S5/S8
SGW
Mr’
MRF
PGW
RTP
EnodeB
LTE
TAS
Gx
Gm
Uu
ISC
EPC
IMS
Emulated Interface
Internal Interface
Media Path
VoLTE Solution 2015 © Ascom
28
Measuring IMS core network QoS
• Active testing of IMS core network with simulated traffic.
• Focus on the QoS delivered by the IMS core network
TOOL
SUT (System Under Test)
Rx
PCRF
Rx
P-CSCF
S-CSCF
ISC
TAS
Gx
Mr’
Gm
PGW simulation
Test
MRF
agent
RTP
LTE
EPC
IMS
Emulated Interface
Internal Interface
Media Path
VoLTE Solution 2015 © Ascom
29
[ ]
CONCLUSIONS
Cyril Lau
ITU-T, Athens, September 2015
ASCOM APPROACH WITHIN THE ITU-T PERSPECTIVE ON QoS/QoE CYCLE
(ITU-T G.1000)
Network/service provider
Subscriber/user
QoS Requirements of Subscriber/User
OR
QoE Requirements
Execution gap
Value gap
QoS Perceived by Subscriber/User
OR
QoE of the User
QoS Targeted by Provider
(Target SLA)
Alignment gap
QoE/QoS
Cycle
Perception gap
KPI, QoS
TEMS
QoS Delivered/Achieved by Provider
(Achieved SLA)
QoE
Subscriber-centric
Quality (QoE)
Cyril Lau
ITU-T, Athens, September 2015
TEMS
Network-centric
Quality (QoS)
CONCLUSIONS
 PLEASE VISIT OUR WEBSITE:
http://www.ascom.com/nt/en/index-nt/about-us-network-testing/nt-about-usresources.htm/
White papers: VoLTE, Video Streaming, HetNets, Carrieir Aggregation
and...watch the space: eMBMS testing to come soon
Informa Webinar: Advanced testing with Ascom in LTE networks
Webinars: VoLTE, Video Streaming
Cyril Lau
ITU-T, Athens, September 2015
[ ]
THANK YOU
Cyril Lau
ITU-T, Athens, September 2015