Corso di Reti di Calcolatori II
A case study:
IPTV SLA Monitoring
Giorgio Ventre
The COMICS Research Group
@
The University of Napoli Federico II,
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
1
Outline
 The general problem: SLA, who cares?
 A business case for QoS
 Defining Service Level Agreements
 A Real-Life SLA monitoring service
 A case study: IPTV SLA Monitoring
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
2
Recent trends in the industry
 New emerging multimedia services both in fixed
and wireless networks
 Traditional voice carriers are moving to NGN:
 Essential to control costs and drive up revenues
 Triple play services: Voice – Video – Data
 Video represents a key element of the service
portfolio
• Price/quality balance must attract/retain users
• TV quality must compete with satellite and cable
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
3
Challenges and quality issues
 Users are conditioned to expect high quality TV
pictures:
 Users unlikely tolerate poor/fair quality pictures in
IPTV
 Early delivery of broadband services is unfeasible
due to the limited bandwidth compared to cable and
satellite
 Compulsory data compression can potentially
degrade quality
 Need for robust transmission to minimize dataloss and delay
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
4
Why Quality Assurance is a major issue?
 Because otherwise we wouldn’t be here
 Quality Assurance adds a new perspective to the flatness
of the current market of triple-play services
 Quality measurement for service assurance
 End-to-end quality monitoring
 SLA based on quality delivered to end-user
 New business models and scenarios
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
5
QoS vs QoE
 Quality of Service (QoS) refers to the capability of
a network to provide better service to selected
network traffic over various technologies. QoS is
a measure of performance at the packet level
from the network perspective.
 Quality of Experience (QoE) describes the
performance of a device, system, service, or
application (or any combination thereof) from the
user’s point of view. QoE is a measure of end-toend performance at the service level from the
user perspective.
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
6
From QoS to MOS
 MOS: Mean Opinion Score
 Used in POTS to have a quantitative value for a
“qualitative” evaluation:
 How do you evaluate the quality you perceived
during your last service usage/access?
 Very easy for simple services: telephony
 Very complex for complex services: multimedia
(sound vs video vs data vs mix)
 Even more complex when quality of service
depends on the distribution network AND
terminals AND servers
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
7
QoS evaluation
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
8
Requirements
 Identify parameters contributing to a satisfactory QoE
 Define network performance requirements to achieve
target QoE
 Design measurement methods to verify QoE
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
9
Performance parameters
 IPTV service is highly sensitive to packet loss
 The impact of packet loss depends on several
factors:
 Compression algorithm (MPEG2, H.264)
 GOP structure
 Type of information lost (I, P, B frame)
 Codec performance (coding, decoding)
 Complexity of the video content
 Error concealment at STB
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
10
Quality Measurement
 Quality Measurement
 Objective
• Pure computational
• Network performance
 Objective perceptual
• Measurements representative of human perception
 Traditional metrics such as PSNR, PLR, BER are
inadequate
 Requirements for objective perceptual metrics
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
11
Why Quality-Monitoring is hard?
 Measures have to be:
 Time-based
 Remoted
 Distributed
 Sharp
 Highly etherogeneous environments (codecs,
CPEs, media-types, …)
 Sampled measures?
 SLAs are not sampled.
 In order to ensure quality, measures have to be
carried out with quality
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
12
Why Quality-Monitoring is hard?
High impact also of content based factors:
MPEG performance depends on content
“pattern” and scene changes
Highly variable (movements, colours, lights)
scenes generates more data
Stallone vs Bergman
or better
Rambo vs The Seventh Seal
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
13
Methods: state of the art
Full-Reference
Reduced-Reference
No-Reference
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
14
Full-reference
 Measures are performed at both the input to the encoder
and the output of the decoder
 Both the source and the processed video sequences are
available
 Requires a reliable communication channel in order to
collect measurement data
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
15
Reduced-Reference
 Extracts only a (meaningful) sub-set of features from both
the source video and the received video
 A perceptual objective assessment of the video quality is
made
 The transmitter needs to send extracted features in
addition to video data
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
16
No-Reference
 Perceptual video quality evaluation is made based solely
on the processed video sequence
 There is no need for the source sequence
 Measurements results are intrinsically based on a
predictive model
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
17
Standards for voice quality assessment
 ITU-T P.862 (Feb. 2001):
 Full-reference perceptual model (PESQ)
 Signal-based measurement
 Narrow-band telephony and speech codecs
 P.862.1 provides output mapping for prediction on
MOS scale
 ITU-T P.563 (May 2004):
 No-reference perceptual model
 Signal-based measurement
 Narrow-band telephony applications
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
18
Standards for voice quality assessment
 ITU-T P.862.2 (Nov. 2005):
 Extension of ITU-T P.862
 Wide-band telephony and speech codecs
(5 ~ 7Khz)
 ITU-T P.VQT (ongoing)
 Targeted at VoIP applications
 Uses P.862 as a reference measurement
 Models analyze packet statistics; speech payload is assumed
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
19
Standards for video quality assessment
 ITU-T J.144 and ITU-R BT.1683 (2004)
 Full reference perceptual model
 Digital TV
 Rec. 601 image resolution (PAL/NTSC)
 Bit rates: 768 kbps ~ 5 Mbps
 Compression errors
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
20
Standards for video quality assessment
 IETF RFC 4445 (April 2006): A proposed Media
Delivery Index (MDI)
 MDI can be used as a quality indicator for
monitoring a network intended to deliver
applications such as streaming media, MPEG
video, Voice over IP, or other information
sensitive to arrival time and packet loss.
 It provides an indication of traffic jitter, a measure
of deviation from nominal flow rates, and a data
loss at-a-glance measure for a particular video
flow.
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
21
Our research
 Objectives:
 Real-time computation of achieved quality level
 “Quality” as perceived by the user
 Per-single-user measurements
 Light computation (about +5% overhead)
 Approach:
 Media playout and measures are both part of an
integrated process
 Measurement subsystems exposes a consistent
abstract interface
 Measurements results are high-level quality
indicators
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
22
VQM (1/2)
 No-Reference
 Evaluates the video quality as perceived by the
user
 QoS  QoE
 Based on MPEG2
 Light parsing
 Doesn’t parse motion vectors, DCT coefficients, and
other macroblock-specific information
 degradation due to packet losses is estimated using
only the high-level information contained in Group of
Pictures, frame, and slice headers
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
23
VQM (2/2)
 Does not need to make assumptions concerning how the
decoder deals with corrupted information
 i.e. what kind of error concealment strategy it uses.
 Based on this information it determines exactly which slices
are lost
 GoP loss-rate
 Frame loss-rate
 Slice loss-rate
 Differentiation per frame type (I, P, B)
 It computes how the error from missing slices propagates
spatially and temporally into other slices
 Appropriate for measuring video quality in a real-time fashion
within a network
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
24
Parsing method (1/2)
GOP
I
B
B
X
P
B
B
P
B
B
P
B
B
X
Frame
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
25
Parsing method (2/2)
MPEG-2 video bitstream
001100101101011010111010101000010101
DECODER
Quality Measurement
HEADERS
Decoded video stream
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
RENDERING
26
QoE vs. MOS
 Mapping between Quality of Experience evaluation and
MOS (Mean Opinion Score – ITU/T P.800) value
MOS
5
4
3
2
1
QMAX
QoE
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
27
MOS vs SLAs
 Knowledge of the function MOS(t) directly enables SLAs
monitoring
DOWN TIME
5
4
MOS 3
2
1
SLA TRESHOLD
TIME
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
28
Experimental testbed
Controlled-Loss
Router
Video
Dropped
Video Client
Server
Packets
+
Quality Meter
Video Characteristics:
MPEG2-TS
Constant Bit Rate:
3.9Mbps
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
29
High Quality
Throughput: 5.0
Mbps
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
30
Medium Quality
Throughput: 3.9
Mbps
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
31
Low Quality
Throughput: 3.0
Mbps
COMICS (COMputer for Interaction and CommunicationS) Research Group – DIS, University of Napoli Federico II
32