ITU Workshop on “Performance, QoS and QoE of
Emerging Networks and Services
Athens, Greece, September 2015
QoE Evaluation and Enforcement Framework
for Internet Services
Marcus Eckert, Thomas M. Knoll
Research Assistant, TU-Chemnitz
marcus.eckert@etit.tu-chemnitz.de
Contents
• Motivation
• Architecture Overview
• Architecture Components
• QMON
–
QoE Monitoring
• QRULE –
QoE Policy and Rules
• QEN
QoE Enforcement
–
• Results
• Summary
Page 2
Motivation
• Experienced quality of Internet services is crucial for customer
satisfaction
• QoE monitoring and enforcement is thus required for business success
• Aim: application specific differentiated handling of traffic flows for major
Internet services
• 3GPP standard based procedures using dedicated bearers are hardly
used today
• Default bearer differentiated flow handling is missing
 Improved QoE measurement and enforcement framework required
 ISAAR Framework (ISAAR = Internet Service quality Assessment
and Automatic Reaction)
ISAAR augments existing QoS functions by flow based network centric
QoE monitoring and enforcement functions
Page 3
Architecture Overview
• Modular service specific QoE management architecture
• 3 functional components:
• QoE Monitoring (QMON) – flow detection and measurement,
• QoE Rules (QRULE) – policy rules and permission checking and
• QoE Enforcement (QEN) – respective flow manipulation
• Interworking with existing QoS mechanisms
•
•
•
•
3GPP PCC
Priority marking (DiffServ, Ethernet prio, MPLS prio)
Proprietary router QoS support (queueing, scheduling, shaping)
SDN based QoS support (e.g. through OpenFlow Action Sets)
Page 4
Architecture Overview
Page 5
Architecture Components – QMON
QMON operation
• Flow classification
• With and without DPI
• Centralized / distributed
• With SDN match and action rules
• Flow capturing
• SDN support to tee out flows
• Flow Monitoring
• Application specific KPI calculation
• Standardization: G.102y
QMON output
• Flow Information and QoE estimation
• currently implemented: “Video QoE
estimation”
Page 6
Measurement Procedure
Measurement at end device
• Most precise
• Firmware / device specific
• User involvement
• Tampering possible
Measurement within operator network
• User and device independent
• End device (buffer) model required  estimation
• Reliable results at scale
• Challenges: constant changes in video streaming (encoding + media
container formats)  MPEG DASH
Page 7
Measurement Procedure
Flow Detection and Classification
• Deep Packet Inspection (DPI)
• Built-in or external using
3GPP PCC Gx interface
Video Quality Measurement
• TCP = reliable transport
 no longer fine grained pixel
and block structure errors
• Video stall events, duration and
inter-stall timing is important
• Buffer fill level estimation as
measurement result
Page 8
Measurement Procedure
Buffer fill level estimation
(exact method)
• Difference between TCP segment
timestamp and playout time
encoded in video data within the
segment
• Each segment is traced and processed
• Use TCP ACKs to increase precision
(segment loss, RTT measurement)
• Buffer model required for fill level estimation
(initial buffering, re-buffering and play-out
thresholds)
• Buffer depletion / stall events,
re-buffering times and inter-stall timing
as measurement results
Page 9
Measurement Improvements (speed & accuracy)
Buffer fill level estimation
(estimation method)
• Speed-up by chunk based throughput like measurement to avoid
decoding every packet (“jump through the stream” method)
• Full header decoding needed and limited to suitable formats e.g. MP4
MP4
Header Chunk 1
Chunk 2
Chunk 3 ...
Chunk i
Chunk i+1
• Variable look-up interval  trade off between processing speed-up and
accuracy
• Loss of fill level estimation precision especially when high delays or
even losses occur due to congestion
Page 10
Measurement Improvements (speed & accuracy)
Buffer fill level estimation
(combined method)
• Automatic switching between exact and estimation mode of operation
to gain the speed-up during good times and to keep the precision
during bad networking conditions.
• There is hardly any difference between the exact and the combined
method result.
• However, the processing load increases (speed-up decreases) for bad
case video streaming conditions
Page 11
MOS calculation for Video QoE
MOS
• Mean Opinion Score (MOS) derived from P.862 (ITU-T: Perceptual
evaluation of speech quality = pesq)
• 0 = worst quality / 4.5 = highest quality
• Assumption: initial 4.5 decreased by negative impact (NI) factor
Initial buffering (e.g. 10s)
does not raise NI
5
4.5
• Each following stall
4
3.5
decreases the MOS; follows
3
an e-function (exponential
2.5
2
function)
1.5
1
0.5
• 𝑀𝑂𝑆 = 𝑒
𝑥
−5+1,5
0
0
1
2
3
4
5
6
7
8
9 10 11 12 13 14 15
stalling events
where x = # of stall events
Page 12
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
0
7
14
21
28
35
42
49
56
63
70
77
84
91
98
105
112
119
126
133
140
147
MOS
MOS calculation for Video QoE
time
• Example video with 5 stalling events
• Resulting in a bad MOS value
Page 13
MOS calculation for Video QoE
• Memory effect also influences the negative impact of each stall (D1) as
well and dampens the impact the longer the play-out has run smoothly
before
𝑒
𝑀𝑂𝑆 = 𝑒
𝑥 = 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠𝑡𝑎𝑙𝑙𝑠
𝑡 = 𝑡𝑖𝑚𝑒 𝑠𝑖𝑛𝑐𝑒 𝑙𝑎𝑠𝑡 𝑠𝑡𝑎𝑙𝑙
𝛼 = 𝑚𝑒𝑚𝑜𝑟𝑦 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
0
6
12
18
24
30
36
42
48
54
60
66
72
78
84
90
96
102
108
114
120
126
132
138
144
150
MOS
•
•
•
•
𝑥
−5+1,5−𝑎∙ 𝑡
time
Page 14
Architecture Components – QRULE
QRULE input
• Flow information and corresponding QoE
estimation from QMON
QRULE operation
• Mapping input flow to service flow classes
• Check whether QoE enhancement is allowed
by general operator policy
• Determine Per Flow Behaviour (PFB) based on
the Enforcement Database of QEN
QRULE output
• PFB specific commands for QEN for 3GPP PCC
triggering and/or marking, shaping, dropping
and even (SDN/LSP) path selection
Page 15
Architecture Components – QRULE / PFB Determination
Example: PFB commands for marking rules and SDN flow based path
selection in high contention situations
Page 16
Architecture Components – QRULE
/
PFB
Example
LTE Test 720p
70
60
buffered video time in s
50
BE/LE marking
low priority
40
normal priority
30
CS5 marking
20
Buffer Level
Threshold 1
Threshold 2
high priority
10
EF or equivalent class marking
0
0
50
100
packet time in s
Page 17
150
200
Architecture Components – QEN
QEN input
•
Flow information and QRULE action
command set
QEN operation
•
•
•
Register enforcement capabilities
Execute flow manipulation via:
• 3GPP – PCC (PCRF / PCEF)
• IETF & IEEE priority marking
• Automated router configuration
with vendor specific QoS
capabilities and settings
• SDN capabilities for marking and
Traffic Engineering (TE)
Granularity: per-flow or per-class
• PFB & class PHB / flow & class TE
Page 18
Architecture Components – QEN
QEN flow manipulation options
•
•
•
•
3GPP – PCC (PCRF / PCEF)
• QCI marking and/or dedicated bearer setup
IETF & IEEE priority marking
• IP Diffserv, Ethernet priority, MPLS traffic class priority
without the need to change the configuration of network elements
• Synchronized inside/outside GTP tunnel & IPSec tunnel marking
Automated router configuration with vendor specific capabilities and settings
• Cisco / Juniper specific router configuration with flow-specific rules for
scheduling, shaping, dropping as well as path (LSP) selection
SDN capabilities for marking and TE
• marking via OpenFlow switch action list configuration
• flow-specific traffic engineering (LSP selection or flow-specific forwarding
paths)
Page 19
Results
Demonstrator setup additionally to the field trials
• Field trials using packet traces at SGi interface of an operator
• Demonstrator Lab setup using 2 Laptops
• Online Buffer fill level estimation and MOS calculation
Page 20
Results
Results for
exact vs.
estimation vs.
combined method
of operation
estimation
reprocessing # re-buffering buffering
interval
time
events
stepping
time
Both algorithms - good case video
Human
0
0s
Exact
6s
0
0s
10 packets
3s
0
0s
50 packets
3s
0
0s
100 packets
3s
0
0s
150 packets
3s
0
0s
250 packets
3s
0
0s
Estimation algorithm - bad case video
Human
10
58 s
Exact
12 s
10
56,6 s
10 packets
6s
10
56,0 s
50 packets
6s
10
54,4 s
100 packets
6s
10
53,7 s
150 packets
6s
9
51,3 s
250 packets
5s
6
49,1 s
Combined algorithm - bad case video
Human
10
58 s
exact
12 s
10
56,6 s
10 packets
8s
10
56,6 s
50 packets
8s
10
56,6 s
100 packets
8s
10
56,6 s
150 packets
8s
10
56,6 s
250 packets
7s
10
56,6 s
Page 21
Results
Results for exact vs. combined method of operation
Good case video
Bad case video
Page 22
Results
5
4.5
4
3.5
3
2.5
2
1.5
1
0.5
0
0
7
14
21
28
35
42
49
56
63
70
77
84
91
98
105
112
119
126
133
140
147
• Each stall is sharply impacting the
experienced quality (MOS score)
• Re-buffering times gradually impact
the MOS score
• Periods of smooth play-out lead to
slight MOS recovery
• Full recovery is possible if only
a few stalls occur and a long
smooth play-out follows
MOS
Results for buffer fill level to
MOS QoE calculation
time
Page 23
Summary
• ISAAR addresses QoE management for Internet based services
• 3 components QMON, QRULE, QEN to monitor and manipulate flows
• Location aware service flow observation and steering
• Automated network based QoE estimation is feasible and produces
accurate results in terms of MOS score calculations and underlying
buffer fill level estimations.
• Aware of 3GPP standardized PCC
•
•
Interworking with PCRF/PCEF
3GPP interfaces are supported (Sd, UD/Sp, Rx, Gx/Gxx)
• ISAAR is also able to work independently of 3GPP QoS functionality
Page 24