Translated from Bosnian to English - www.onlinedoctranslator.com
Quality of Service / Quality of Experience
Overview and Best Practices
Deutsche
TV platform
Quality of Service /
Quality of Experience:
Overview and Best Practices
Introduction................................................. .................................................. .................................................. 4
application scenarios
IP Streaming .................................................................................................................................................6
Encoding Analysis and Optimization ........................................ ....................................... 17
Methods for assessing video quality ..................................................... .................................................. 9
Infrastructure and network analysis ..................................................... ....................................... 17
Objective quality assessment methods................................................. ....................... 10
Network Optimization ................................................ .................................................. ........ 19
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
• SAND SRA ............................................................................................................................................. 19
Measurement of streaming video quality................................................. .................................................. ....... 12
• CMCD ..................................................................................................................................................... 19
Adaptive streaming metrics and models ........................................ ....................... 12
KPI / KQI analysis in the player ............................................ .................................................. ... 20
• MPEG DASH Annex D........................................................................................................................... 13
• SAND...................................................................................................................................................... 13
QoE measurement based on integrated KQIs under consideration
• CTA-2066 –Streaming Quality of Experience Events, Properties and Metrics............................ 13
of the streaming session available on the player side......................................... ................... 20
• CTA-5004 – Common Media Client Data (CMCD) ............................................................................. 13
• Streaming Video Alliance – Key Network Delivery Metrics........................................................... 13
Summary ................................................. .................................................. ...................................21
• ITU-T Rec. P.1203, P.1204 .......................................... ................................................ ...................... 14
List of abbreviations................................................. .................................................. ............................21
Streaming metrics (KPIs)................................................. .................................................. ..... 15
• Parameters for the reliability of the stream playback ........................................ ............... 15
Appendix .....................................................................................................................................................22
• Video Quality Parameters (KQIs) ................................................ ................................................ 16
Imprint ................................................. .................................................. ................................................24
side
2
side
3
Quality of Service / Quality of Experience
Overview and Best Practices
Deutsche
TV platform
Quality of Experience:
„The degree of delight or annoyance of the
user of an application or service.“
(Definition of the ITU-T)
introduction
The Media over IP working group is primarily
concerned with the possibilities and challenges
of new technologies for television. A central
technology here is (unicast) video streaming, i.e.
the packet-wise, individual transmission of linear
and non-linear video content over the Internet.
Services like Netflix and YouTube use this
technology to deliver content directly and
individually to their users.
The advantages of video streaming are also
becoming increasingly relevant for TV: Content
can be consumed at any time and on a large
number of Internet-enabled devices and a
special TV network infrastructure (satellite,
terrestrial, cable) is no longer necessary. In
order to achieve broad user acceptance for the
new transmission methods, it is essential that
stability and quality are on the same level as
with traditional radio (broadcasting).
side
4
In order to be able to make reliable statements
about stability and quality, this white paper
deals with the quality measurement of video
streaming:
• First, terminology is defined and, using the
streaming chain, it is schematically described
which components have an influence on the
quality (Section 2, p. 6ff)
• In Section 3 there is an introduction to the
relevant measurement methods (p. 9ff)
The aim of this white paper is to give the
members of the German TV platform a practiceoriented introduction to the relatively new but
highly relevant world of measuring quality in
video streaming.
of the user of the service.” (ITU-T Rec. P.10/
G.100). In addition to the QoS or the system,
user and context-specific factors also influence
the QoE (Le Callet et al., 2012; Möller & Raake,
2014).
In particular, it should be shown that what
matters in the end is the so-called Quality of
Experience (QoE), i.e. the quality of what the
users experience individually on their end
devices. The ITU-T defines QoE as "The degree
of delight or announcement of the user of an
application or service." (ITU-T Rec. P.10/G.100).
This white paper was created in cooperation
with leading German research institutions and
video streaming platforms and has a clear focus
on the TV streaming application scenario.
• Section 4 provides insight into the application of the
measurement methods and defines applicable KPIs
and benchmarks (p. 12ff)
• In Section 5, the practical application of the
measurement methods and KPIs is explained
using case studies (p. 17ff).
The so-called Quality of Service (QoS), i.e. the
measurable interaction of the individual
components, is an important but not the only
basis of QoE. The ITU-T defines QoS accordingly
as "The totality of characteristics of a
telecommunications service that bear on its
ability to satisfy stated and implied needs
side
5
Quality of Service / Quality of Experience
Overview and Best Practices
Deutsche
TV platform
IP Streaming
access
network
IP-based media delivery can be multicast (e.g.
IPTV) or unicast. In this document, we focus on
adaptive unicast streaming based on the HTTP
protocol (HyperText Transfer Protocol), which is
also known as ABR (Adaptive Bitrate Streaming)
or HTTP-based Adaptive Streaming (HAS) and is
the basic technology for OTT streaming
represents.
These include the currently market-relevant
streaming technologies DASH (Dynamic
Adaptive Streaming over HTTP) and Apple's HLS
(HTTP Live Streaming) as well as CMAF (Common
Media Application Format), which has specified a
first step towards standardizing DASH and HLS.
There are two basic content formats carried over
ABR streaming that have different quality of
service requirements: linear TV and video-ondemand (VoD). Linear TV represents the highest
and most complex end-to-end service quality
requirements.
OTT streaming is also commonly referred to as
"Best Effort" streaming because the service
provider offers its service over the public
internet (e.g. YouTube) and thus the
side
6
Service quality for the customer can only
partially influence. If a network provider is also
a provider of a streaming service in its network,
this is referred to as "managed OTT" streaming,
since it can largely control the quality of the
service.
Video streaming is a complex process with
numerous influencing factors along the
streaming chain. This chain consists of the
components shown in Figure 1: transcoding /
compression, packaging, CDN and end device
including player, whereby only the optimal
interaction of all components ensures good
service quality.
During the transcoding/compression step,
different versions of the original videos are
created and divided into short segments
according to a predefined configuration ("bitrate
ladder"). These versions (representations) differ
in resolution and compression compared to the
original according to the bit rate ladder. The
ABR (Adaptive Bitrate) algorithm in the player
dynamically selects a representation appropriate
to the currently available network capacity
during streaming in order to obtain the best
possible video quality with the available
bandwidth.
TV Linears
transcoding/
Compression
CDN
packing/
encryption
end device
VoD
Origin
Server
Caching
Server
Errors in the input signals (ingest) are passed on
directly, which is why quality control should be
carried out. Only good subsequent encoding/
transcoding enables good service quality for the
end customer. So e.g. For example, incorrect or
excessive compression of the video signal can
lead to visible artefacts in the encoded video.
Standard coding methods on the market are H.264
(MPEG-4 AVC), H.265 (MPEG-H Part 2 / HEVC), VP9
and AV1.
In packaging, the individual video elements are
packed into target formats such as fMP4
(fragmented MP4) or MPEG-TS (MPEG Transport
Streams) for DASH or HLS streaming, with fMP4
being used for DASH and MPEG-TS for HLS.
fMP4 is used for CMAF. DRM encryption (DRM:
Digital Rights Management) also takes place
here. One of the biggest challenges regarding
the QoE here is the end-to-end delay time
(latency) for the linear TV format, which arises
from the segmentation of the video stream. In
the
side
7
Quality of Service / Quality of Experience
Overview and Best Practices
In standard formats for HLS and DASH, this
delay is between 15 and 45 seconds compared
to the input signal, depending on the segment
length set. With the newer CMAF-based
method, this can be reduced to 3 - 6 seconds
and thus comparable values to those of classic
broadcasting can be achieved.
The CDN (Content Delivery Network) reliably
distributes the requested video streams in file
format to the large number of end devices. It
replicates the content from the origin server to a
variety of cache servers distributed across the
network. In order to achieve a high QoE for the
end customer, the cache servers should be
located as "close" to the customer as possible, ie
in the edge network. A CDN that operates cache
servers within the network of the access
network used by the end user provides a
possible advantage for the QoE.
The characteristics of the access network have a
decisive influence on the quality of service for the
customer. Fixed network (cable, DSL), WLAN or
mobile radio (3G, 4G 5G) can be differentiated here.
The end device used by the customer also plays
a significant role in the streaming chain for QoE.
A distinction must be made here as to whether
the customer is using their own device (e.g.
smartphone) or a device provided by the
provider (streaming box). With the latter, the
service provider has better control over the
quality of service. The properties of the end
devices (size of the screen, resolution, CPU
performance) and the performance of the video
player used are decisive for the presentation
and quality of the streams.
Quality of Service:
„… ability to satisfy stated and implied needs
side
8
TV platform
Methods for assessing
video quality
Two perspectives are useful for assessing the
quality of video streaming: The QoS perspective
reflects the collection of network or service
parameters, according to the definition given in
Section 1. These measurement results are
generally linked to the technical parameters of
the system, such as selected encoding and
representation, network settings, the
configuration of CDNs and the algorithms and
settings used in the player.
The QoE perspective addresses the quality
actually perceived by the user in the sense of
"quality experience". This can either be
expressed in terms of concrete ratings made by
users, such as on the widespread 5-point Mean
Opinion Score scale (MOS 1 ), or in metrics
related to user behavior, such as watch time
(duration of watching a video, or deviation from
the average watch time under certain network
conditions), or actions such as stopping and
restarting a stream.
There are two basic methods for collecting
corresponding measurements of QoE from the
user perspective: (1) methods for evaluating or
measuring the behavior of actual users (in the
laboratory, using crowd-sourcing or in the field),
or (2) instrumental methods based on aim to (1)
automatically and “objectively” predict measured
values collected from real users, ie to estimate
them on the basis of network or stream-based
measurements. These "objective" models and
methods can operate on the basis of data such
as media bit streams, video and audio signals
and/or metadata (network, media information
such as played segments, stallings, user data,
etc.). In accordance with the focus of this white
paper on technical measurements, the following
considerations are limited to instrumental,
"objective" methods (2).
1
of the user of the service.“
Deutsche
The so-called MOS scale is actually a 5-point absolute category rating
scale according to e.g. B. ITU-T Rec. P.800 or P.910, showing a mean
(Definition of the ITU-T)
opinion score.
side
9
Quality of Service / Quality of Experience
Overview and Best Practices
Deutsche
TV platform
Objective quality assessment methods
Objective quality assessment methods
establish a connection between measured QoS
streaming parameters (KPI's) and the
perceived QoE of users through subjective
tests. They are also known as QoE prediction
models.
There are three basic categories: metadatabased QoE models,
Bitstream-based QoE models and
Pixel-based QoE models.
.
• Metadata-based QoE models leverage
information from the metadata layer such as
image resolution, frame rate, and bitrate.
They can also be thought of as lightweight
variants of bitstream models that only
analyze the metadata portion. An example of
this is ITU-T
Rec. P.1203.1, „Mode 0“.
• Bitstream-based QoE models analyze the IP
video stream without decoding and do not
require access to the original bitstream of
the source signal. Examples are ITU-T Rec.
P.1203.1 (modes 1 and 3) and Rec. P.1204.3.
• Pixel-based QoE models (e.g. VMAF, Video
Multimethod Assessment Fusion, see Section
4) analyze the decoded frames of the video.
Here can different
Variants are distinguished:
• Full Reference (FR) where the original
content can be compared to the received,
perceived image for comparison, such as
B. ITU-T Rec. P.1204.4 or VMAF.
• Reduced Reference (RR), in which a
"reduced" representation of the reference
and the sequence to be evaluated is used,
e.g. B. ITU-T Rec. P.1204.4.
For live streaming, pixel-based RR or FR
models are not readily applicable due to their
complexity and the use of reference
information.
Due to their complexity, such models can
generally only be used with video-ondemand using a precalculation of the quality
on the server side, possibly with a
corresponding transmission of this
information as meta information.
• Hybrid models are based on an evaluation of
pixel information and additional bit stream or
metadata information, such as. B. ITU-T Rec.
P.1204.5.
• No Reference (NR), where the assessment is
made without access to the reference
content. There is currently no known NR
model that provides sufficiently good
prediction accuracy compared to actual user
ratings.
side
10
side
11
Quality of Service / Quality of Experience
Overview and Best Practices
Deutsche
TV platform
measurement of
streaming video quality
Adaptive streaming metrics and models
Modern OTT video players receive the video
segments to be played back via HTTP. A
manifest file describes the representations
available on the streaming server. The video
player's ABR (Adaptive Bitrate) algorithm
constantly measures various metrics and then
adjusts its further procedure accordingly. If, for
example, the video buffer cannot be filled
quickly enough due to poor data throughput, so
that the video would "stand still" or
"jerk" (stalling), the player calls up a
representation with a low bit rate. The metrics
available to the player at play time can also be
sent to an external service (“reporting”) to
perform retrospective analysis and optimize the
overall system. In addition, aggregated
information (e.g. end-to-end latency) that is not
directly available in the end device can also be
transmitted to the player in order to optimize
the end-to-end quality in real time.
side
12
In principle, streaming metrics can be collected
in all standard streaming environments and
players. The basic HTTP-based ABR
transmission mechanisms are the same
everywhere, whether it's live TV or VoD, or
playback in a native player or in a web browser.
The following sections provide an overview of
various industry standards with the aim of
harmonizing the streaming metrics defined
therein and simplifying their exchange. First of
all, different standards and measurement
protocols are listed, which address the
acquisition or communication of QoS and QoEspecific information.
The remainder of this section tabulates and
briefly explains specific Key Performance
Indicators (KPIs) as QoS-related measurement
indicators. In the last part of this section, socalled Key Quality Indicators (KQIs) are listed
and explained. They make it possible, based on
underlying KPIs, for example, to provide
statements on the QoE perceived by the
customer.
MPEG DASH Annex D
Metrics for the adaptive streaming format
MPEG-DASH are defined in Annex D of the
MPEG-DASH (ISO/IEC 23009-1) standard. This
includes the following raw metrics: TcpList (TCPlevel transaction metrics), HttpList (HTTP-level
transaction metrics), RepSwitchList (quality
switch metrics), BufferLevel (the size of the
buffer at a given point in time), and PlayList ( e.g.
play/pause events). The metrics are also
applicable to other streaming formats (e.g. HLS).
https://www.iso.org/standard/79329.html
SAND
SAND (Server and Network-assisted DASH),
specified in ISO/IEC 23009-5, offers a
standardized message format and protocol for
the transmission of streaming metrics. SAND
messages are packaged as XML documents and
sent via HTTP or WebSockets. SAND references
MPEG DASH Annex D for the content of the
SAND messages.
CTA-2066 –
Streaming Quality of Experience Events,
Properties and Metrics
CTA-2066 standardizes player events and
properties, QoE metrics, and related
terminology. The aggregated metrics are
broken down into four categories: Availability,
Startup Time, Continuity, Video & Audio Quality.
https://shop.cta.tech/collections/standards/products/ streamingquality-of-experience-events-properties-and-metrics
CTA-5004 –
Common Media Client Data (CMCD)
CMCD (not yet published, demo link)
standardizes the information exchange between
player and CDN. The information is used on the
CDN side for log analysis, QoS monitoring and
optimization of data transfer.
https://tinyurl.com/cmcdata
Streaming Video Alliance – Key
Network Delivery Metrics
https://www.iso.org/standard/69079.html
In this document, the Streaming Video Alliance
defines metrics that describe the effectiveness of
network transmission (streaming). For this
purpose, only the most common and universally
measurable metrics were included in the standard.
https://www.streamingvideoalliance.org/product/keynetworkdelivery-metrics/
side
13
Quality of Service / Quality of Experience
Overview and Best Practices
ITU-T Rec. P.1203, P.1204
ITU-T Rec. P.1203 consists of several subcomponents for monitoring the QoE of adaptive
streaming sessions lasting 1 to 5 minutes. The
partial standard P.1203.1 specifies four
differently complex quality models, each of
which provides a per-second estimate of the
video quality on a 5-point "MOS scale" based on
bit stream information: The simplest so-called
Mode 0 model uses input information similar to
CTA- 2066 – such as the codec used
– as well as the resolution, bit and frame rates
measured for the played segments. Mode 1
provides more accurate estimates using frame
types and frame sizes. "Mode 2" and "Mode 3"
are based on full access to the encoded video bit
stream, with different amounts of evaluated
data.
P.1203.2 describes a corresponding audio
quality component that also provides persecond MOS estimates applicable to different
audio codecs.
Deutsche
TV platform
Streaming Metrics (KPIs)
Together with information on the initial loading
delay and the times of stalling or rebuffering (cf.
CTA-2066 KPIs), the audio and video quality
values are integrated into a session QoE value
in the P.1203.3 module. The "ground truth" on
the subjective overall QoE has been collected in
extensive laboratory studies with test persons in
order to train and validate the models, see
Raake et al. (2017), Robitza et al. (2018). For
exemplary applications of P.1203 see Robitza et
al. (2018), Robitza et al. (2020) and
Schwarzmann et al. (2020).
The new standard series ITU-T Rec. P.1204 was
adopted at the end of 2019 for predicting video
quality with resolutions up to 4K / UHD. So far,
this has included three types of video quality
models based on different input information.
They provide an estimated video quality value
for sections up to 10 seconds in length and persecond estimates for H.264, HEVC/H.265 and
VP9 video codecs with different presets:
(i) bitstream-based model P.1204.3 (see
Ramachandra Rao, 2020), (ii) pixel- and
reference-based (RR/FR) model P.1204.4, (iii)
hybrid, metadata- and pixel-based "NoReference” model P.1204.5.
The following KPIs are a selection of metrics from the above standards. They can be broken down into
stream playback reliability parameters and video quality parameters (KQIs). The most important are
listed below.
Stream playback reliability parameters
Video Start-up Time
(sec) *
Time between triggering a video playback and when it first plays on
screen.
Alternative Begriffe: Initial Loading Time, Initial Loading Delay.
Playback Stalls *
Time and period of buffer-related stopping when playing a video
stream.
Alternative term: stalling.
Video Frame Drops
Number of video frames dropped by the player due to
performance issues
Video Start Failure
The first video segment is not delivered within 10 seconds after
initialization
(yes/no) *
(bits/sek) *
Time, period and size of the played bit rate (bit
rate progression over time)
Video Resolution
History of the resolution of the video stream (e.g. 4096?2160, 1920x1080)
Video Frame Rate
Frame rate (e.g. 25 fps)
Quality Switches *
Temporal progression of the respective display (codec, bit rate, resolution, frame
rate) according to the streaming bit rate ladder used.
Re-Buffer Ratio (%)
Ratio between the total playback time of the video stream and the
total playback time plus the time for rebuffering (stalls, stalling).
Average Bitrate (bits/sek)
Average bitrate of the player for video playback
Video Bitrate
* defined in SVA. https://www.streamingvideoalliance.org/product/key-network-delivery-metrics/
side
14
side
15
Quality of Service / Quality of Experience
Overview and Best Practices
Video Quality Parameters (KQIs)
PSNR (Peak Signal-toNoise-Ratio)
VMAF (Video Multimethod
Assessment Fusion)
PNSR describes the so-called Mean Squared Error (MSE), i.e. the mean
square deviation between two frames. The PSNR value is given in decibels,
whereby the higher the decibel value, the smaller the deviation between
the compressed image and the original.
VMAF is a video quality metric developed by Netflix to capture subjectively
perceived video quality. VMAF is based on machine learning methods.
The main difference between VMAF and other quality metrics like PSNR is
that the underlying model was trained on real viewer ratings. For this
purpose, values for various categories of films from the Netflix catalog
were collected in laboratory tests.
SSIM (Structural
Similiarity Index)
SSIM is an objective prediction model for perceived video quality. SSIM
measures the similarity between two images. An initial uncompressed image
(video) serves as a reference.
Audio and video quality
The standards ITU-T Rec. P.1203 and P.1204 contain model components for
audio and video quality estimation with output on a 5-point scale [1,5].
Rec. P.1203.1 describes four different “no reference” algorithms for
metadata or bitstream-based video quality estimation with a per-second
resolution, which can then be integrated for longer time periods. Modes 0
to 3 differ in the complexity of the input data considered, see Section 4,
“ITU-T Rec. P.1203, P.1204”. Rec. P.1203.2 specifies a metadata-based
audio quality model that
Provides per-second estimates for different audio codecs.
according to ITU-T Rec. P.1203
and P.1204
The Rec. P.1204 family of standards specifies three different, very accurate video
quality models for a per-second prediction for resolutions up to 4K/UHD, frame
rates up to 60 fps and the video codecs H.264, HEVC/H.265 and VP9. See Section 4,
“ITU-T Rec. P.1203, P.1204” for more details.
VMAF. https://github.com/Netflix/vmaf
side
16
Deutsche
TV platform
application scenarios
Encoding analysis and optimization
In principle, models such as VMAF or the P.1204
series (see Section 4) enable the quality analysis
of already encoded video material. They can
thus be used to calculate a so-called "bit rate
ladder" in terms of the best representations for
a sequence or individual scene in terms of
resolution, frame rate and bit rate. Accordingly,
such dimensions are used by Netflix, for
example
used for their "Dynamic Optimizer" as scenespecific "per-scene" encoding, i.e. taking scene
changes into account (Katsavounidis et al.,
2018). Artifact analyzes and the general
optimization of encoder settings can also be
carried out with the help of correspondingly
precise quality models.
Infrastructure and network analysis
The service infrastructure and the different
paths of the OTT stream from the origin via the
CDN to the edge network and finally to the user
have a great influence on QoS and QoE. Failures
of parts of the infrastructure lead to limitations
in the quality of service, the different
components in the streaming chain each have
different effects.
Encoders in live operation transmit unpredictable
content and can therefore also have scenes with
different computational intensity for processing at
times. This can lead to CPU
bottlenecks and potentially affects the resulting
video quality. The response times of
communicating components can also vary,
depending on the system load. Especially at
peak times, this can lead to limitations and
affect other components. The network
bandwidth within the own network and to IPSs
can also lead to bottlenecks and service
interruptions due to the enormous data volume
of video content. The delivery of the streams
from the CDN via HTTP generates status codes.
By monitoring and evaluating the response
codes, direct conclusions can be
side
17
Quality of Service / Quality of Experience
Overview and Best Practices
se on the QoS and indirectly conclusions about
negative influences of the QoE on the user. All
of the above examples can be measured with
appropriate QoS metrics and provide insight
into factors influencing the overall quality that is
possible for the user at any given time.
In addition to the QoS metrics of the
infrastructure and the network, conclusions can
also be drawn about the actual QoE as perceived
by customers during use by evaluating the
usage data. By combining the maximum
available quality and the quality actually
requested by the player, one can
how many users are currently streaming with
reduced quality. Above all, we see significant
differences between Managed OTT and OTT
over the free Internet (cf. Fig. 2 OTT and Fig. 3
Managed OTT) . Between 65% and 92% of users
receive maximum quality with unmanaged OTT
streaming. The rate depends on the one hand
on the time of day and on the other hand on the
ISP (Internet Service Provider) of the users.
Managed OTT, on the other hand, impresses
with a high percentage of approx. 98.5%-99.5%
of users with playback in maximum quality,
since there are hardly any external influencing
factors affecting the service. Furthermore, a
distinction can be made between use via mobile
networks and fixed Internet connections.
Deutsche
TV platform
Fig. 3: Example of percentage distribution managed OTT: users not at maximum quality, between 0.5% - 1.5%.
network optimization
Effective technologies are now available to improve the potential degradation of QoE caused by network
effects. The currently most promising approaches are described below.
Fig. 2: Example of percentage distribution of unmanaged OTT: users not at maximum quality, between 10% - 13%.
SAND SRA
The Shared Resource Allocation (SRA) function
defined in SAND (DASH Part 5, Server and
network assisted DASH, ISO/IEC 23009-5) allows
servers to allocate bandwidth to streaming
clients. For this purpose, the server collects
streaming metrics from the clients. The server
component then sends bit rate allocations to the
clients. This bit rate must be maintained in the
client, e.g. B. by setting a maximum bit rate in
the player or by traffic shaping. Overall, this
allows for fairness and/or prioritization of the
clients attached to the same network bottleneck
(WiFi AP).
side
18
are closed, e.g. B. to maximize the number of
clients that can play a video stream smoothly.
https://www.iso.org/standard/69079.html
CMCD
CMCD defines metrics that are collected by a
streaming client and sent to the CDN with each
segment or manifest request. In doing so,
CMCD helps the CDN provider to optimize the
overall performance of the CDN. The metrics
defined in the specification can be useful for
"protocol analysis, QoS monitoring and
transmission optimization". Except-
side
19
Quality of Service / Quality of Experience
Overview and Best Practices
CDNs can "control their traffic and correlate
performance issues with player software
versions or specific devices." Overall, a CDN
provider can improve the CDN performance and
thus the user's QoE as well.
KPI / KQI analysis in the player
Streaming metrics captured in the video player
reflect what happens at the end of the
streaming chain, or before the customer. These
metrics are especially valuable when analyzing
streaming issues. The streaming metrics are
usually sent to a server (often in real time),
which stores the data and prepares and
visualizes it for detailed analysis.
There are various commercial solutions on the
market that offer a complete package for
streaming analytics. An interoperable approach
is based on the SAND standard (demo).
For research purposes, implementations are
also available for measurements of audio and
video quality in accordance with ITU-T Rec.
P.1203 and P.1204.3, which also provide
information on the quality played out in addition
to KPIs.
QoE measurement based on
integrated KQIs considering the
streaming session on the player side
If corresponding KPIs and thus KQIs can be
measured on the server and / or client side such as audio and video quality over time as well
as quality-relevant events such as the "stalling"
of a video stream (see Section 4) - these can be
measured using an integration model as ITU-T
Rec. P.1203.3 can be integrated into a QoE
estimate. This is a measure of the QoE
perceived by the end customer in typical short
streaming sessions lasting a few minutes (see
Section 4). Other integration models for longer
periods of time are also conceivable, or models
for predicting the termination of a streaming
session by the customer.
Togetherversion
As mentioned at the beginning, the systematic
recording of the user experience (QoE) is an
essential concern of the operators of TV / video
streaming services. The more detailed analysis
of the topic and the overview given in this white
paper show the complexity and multifaceted
nature of today's possibilities.
It is difficult to give a uniform recommendation
for the members of the German TV platform
with regard to the metrics and technologies to
be used. The right choice depends heavily on
the objective, quality requirements, technical
possibilities and framework conditions.
This whitepaper should therefore serve as a
well-founded introduction and an overview of
the currently most relevant approaches. When
preparing the paper, it became clear that a close
integration of practice-oriented science and
future-oriented service providers is a very
efficient way of penetrating the matter and
quickly achieving results in practical use.
Deutsche
TV platform
abbreviation
directory
ABR
CDN
Adaptive Bitrate Streaming Content
CMAF
Application Format Common Media
CMCD
Client Data
CTA
Consumer Technology Association Dynamic
DASH
Adaptive Streaming over HTTP) Digital
DRM
FR
ISP
Rights Management
THAT
Delivery Network Common Media
Full reference
Internet Service Provider International
Telecommunication Union HTTP-based
HAS
HLS
KPI
KQI
MOS
NR
Adaptive Streaming HTTP Live Streaming
PSNR
Peak Signal-to-Noise Ratio
QoE
QoS
RR
Quality of Experience
SAND
Server and Network-assisted DASH
SRA
YES
Shared Resource Allocation
VMAF
Video Multimethod Assessment Fusion
Key Performance Indicator
Key Quality Indicator
Mean Opinion Score
No Reference
Quality of Service OTT = Over The Top
Reduced Reference
Structural Similiarity Index
https://www.iso.org/standard/69079.html http://
dash.fokus.fraunhofer.de/SANDLibrary/samples/ dash.js/
https://github.com/Netflix/vmaf
side
20
The authors of the document are available for
an in-depth exchange and discussion of best
practices.
side
21
Quality of Service / Quality of Experience
Overview and Best Practices
Deutsche
TV platform
Appendix
Complementary source ITU definitions and regulatory
A. Raake, M. Garcia, W. Robitza, P. List, S. Göring and
W. Robitza, A. Dethof, S. Göring, A. Raake, A. Beyer, T. Polzehl
ITU-T Rec. P.1203.3 (2020).
considerations:
B. Facts (2017).
(2020).
Parametric bitstream-based quality assessment of progressive
ITU-D „Quality of service regulation manual” (2017), Geneva,
„A bitstream-based, scalable video-quality model for HTTP
„Are you still watching? Streaming Video Quality and Engagement
download and adaptive audiovisual streaming services over
Switzerland: ITU Telecommunication Development Bureau
adaptive streaming: ITU-T P.1203.1,” Ninth International
Assessment in the Crowd”, Twelfth International Conference on
reliable transport – Quality integration module. Geneva,
Conference on Quality of Multimedia Experience (QoMEX),
Quality of Multimedia Experience (QoMEX), Athlone, pp. 1– 6
Switzerland: International Telecommunication Union.
ITU-T Rec. P.10/G.100 (2017).
Erfurt, 2017, pp. 1– 6, DOI:10.1109/QoMEX.2017.7965631
Vocabulary for performance, quality of service and quality of
https://ieeexplore.ieee.org/document/7965631
experience. Geneva, Switzerland: International Telecommunication
ITU-T Rec. P.1204 (2020).
S Schwarzmann, N Hainke, T Zinner, C Sieber, W Robitza, and
Video quality assessment of streaming services over reliable
Union.
W. Robitza, S. Göring, A. Raake, D. Lindegren, G. Heikkilä, J.
A Raake (2020).
transport for resolutions up to 4K. Geneva, Switzerland:
https://www.itu.int/dms_pub/itu-d/opb/pref/D-PREF-
Gustafsson, P. List, B. Feiten, U. Wüstenhagen, M.-N. Garcia,
Comparing fixed and variable segment durations for adaptive
International Telecommunication Union.
BB.QOS_REG01-2017-PDF-E.pdf
K. Yamagishi, and S. Broom (2018).
Video-Streaming: a holistic analysis. In Proceedings of the 11th
HTTP adaptive streaming QoE estimation with ITU-T Rec. P. 1203:
ACM Multimedia Systems Conference (MMSys ’20). Association for
ITU-T Rec. P.1204.3 (2020).
P. Le Callet, S. Möller, and A. Perkis, edts (2012). Qualinet white
Open databases and software. In Proceedings of the 9th ACM
Computing Machinery, New York, NY, USA, 38–53. DOI:https://
Video quality assessment of streaming services over reliable
paper on definitions of quality of experience. European network
Multimedia Systems Conference (MMSys ’18). Association for
doi.org/10.1145/3339825.3391858
transport for resolutions up to 4K with access to full bitstream
on quality of experience in multimedia systems and services
Computing Machinery, New York, NY, USA, 466–471. DOI:https://
(COST Action IC 1003), 3(2012).
doi.org/10.1145/3204949.3208124
information. Geneva, Switzerland: International
ITU-T Rec. P.1203 (2016).
Telecommunication Union.
Parametric bitstream-based quality assessment of
S. Möller, & A. Raake (Eds.) (2014).
W. Robitza, D. G. Kittur, A. M. Dethof, S. Göring, B. Feiten and
progressive download and adaptive audiovisual streaming
ITU-T Rec. P.1204.4 (2020).
Quality of experience: advanced concepts, applications and
A. Raake (2018).
services over reliable transport. Geneva, Switzerland:
Video quality assessment of streaming services over reliable
methods. Springer.
Measuring YouTube QoE with ITU-T P.1203 Under Constrained
International Telecommunication Union.
transport for resolutions up to 4K with access to full and reduced
reference pixel information. Geneva, Switzerland: International
Bandwidth Conditions. Tenth International Conference on Quality of
ITU-T Rec. P.800 (1996).
Multimedia Experience (QoMEX), Cagliari, 2018, pp. 1– 6,
ITU-T Rec. P.1203.2 (2017).
Methods for subjective determination of transmission quality.
DOI:10.1109/QoMEX.2018.8463363
Parametric bitstream-based quality assessment of progressive
Geneva, Switzerland: International Telecommunication Union.
https://ieeexplore.ieee.org/document/8463363
download and adaptive audiovisual streaming services over
ITU-T Rec. P.1204.5 (2020).
reliable transport – Audio quality estimation module. Geneva,
Video quality assessment of streaming services over reliable
Switzerland: International Telecommunication Union.
transport for resolutions up to 4K with access to transport and
Telecommunication Union.
ITU-T Rec. P.910 (2008).
R. R. Ramachandra Rao, S. Göring, W. Robitza, A. Raake, B.
Subjective video quality assessment methods for multimedia
Feiten, P. List, and U. Wüstenhagen (2020),
received pixel information. Geneva, Switzerland: International
applications. Geneva, Switzerland: International
„Bitstream-based Model Standard for 4K/UHD: ITU-T P.1204.3
Telecommunication Union.
Telecommunication Union.
– Model Details, Evaluation, Analysis and Open Source
Implementation”, Twelfth International Conference on Quality of
Multimedia Experience (QoMEX), Athlone, pp. 1– 6
side
22
side
23
Deutsche
TV platform
Quality of Service / Quality of Experience
Overview and Best Practices
imprint
Editor/Publisher:
About the German TV platform
German TV Platform eV
The German TV platform is an association of over 50
www.tv-plattform.de
members, including private and public broadcasters,
Association register no. 73VR9797
streaming providers, device manufacturers, Internet
companies, infrastructure operators, service and
Head of the Media over IP working group:
technology providers, research institutes and
dr Niklas Brambring, Zattoo Head of Task
universities, federal and state authorities and others
Force Delivery: Peter Pogabit, Deutsche
involved with digital media companies, associations
Telekom AG
and institutions. Since it was founded in 1990, the aim
of the registered association has been the introduction
Contact:
of digital technologies based on open standards.
Deutsche TV-Plattform eV c/
o ZVEI, Lyoner Str. 9 60528
Frankfurt am Main mail@tv-
Liability:
plattform.de
The information in this report has been researched
accurately and thoroughly and has been compiled
to the best of our knowledge, taking into account
the neutral approach of the Media over IP / Task
Force Delivery working group of the German TV
platform. All information reflects the current status
at the time of going to press. However, the
members of the working group and the German TV
platform cannot guarantee the topicality,
correctness, completeness and/or quality of the
information published here. Therefore, liability
claims against Deutsche TV-Platform eV as the
publisher for material and immaterial damage
caused by the use of this publication or the content
provided or by the use of incorrect or incomplete
information are fundamentally excluded.
Tel.: 0049-69-6302-311
Fax: 0049-69-6302-361
Editorial staff:
Task Force Delivery of the Media over IP
working group of the German TV platform
Author team:
Alexander Raake / TU Ilmenau Peter
Pogrzeba / Deutsche Telekom AG
Nimesh Karia / Sky Germany Stefan
Arbanowski / Fraunhofer Focus Stefan
Kaiser / Zattoo
Stefan Lietsch / Zattoo Stefan
Pham / Fraunhofer Focus