International Telecommunication Union
Are Existing
Performance Metrics
Adequate?
Alan Clark
CEO, Telchemy
Workshop on End-to-End Quality of Service.What is it? How do we get it?
Geneva, 1-3 October 2003
Outline
ITU-T
o IP Performance Metrics
o Network Characteristics
o Packet Loss Characteristics and
Measurement
o Jitter Characteristics and Measurement
o Recommendations
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
2
Typical IP Performance Metrics
ITU-T
o Packet Loss
• Equipment problems, misrouting, buffer
overflow, transmission errors
o Jitter
• Network congestion, contention and
queuing delays
o Delay
• Transmission delays, quasi-stable
congestion levels
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
3
Core IP Network characteristics
ITU-T
o Class “A” networks
• High capacity optical fiber, high
throughput routers
• Very low jitter and packet loss,
occasional link failures
o Class “B” networks
• T1/E1 trunk connections
• Significant jitter and loss due to network
congestion
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
4
Corporate IP Networks
ITU-T
o Moving to 100BaseT Switched Ethernet
but still some 10BaseT and Hubs
o Access links often T1/E1/PRI,
sometimes fractional
o Mixture of digital leased line, frame
relay and IP VPN
o Small offices and teleworkers may use
low bandwidth links
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
5
Teleworkers and Residential IP
ITU-T
o Low-mid bandwidth DSL and Cable
Modem connections with little or no QoS
control
o Often 10BaseT and IEEE802.11 LAN
o Can experience heavy usage of access
links leading to high levels of jitter
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
6
Packet Loss
ITU-T
o Specifically, packets lost within the
network or discarded before reaching
the network API in the receiving system.
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
7
Example packet loss
distribution
ITU-T
Number of packets lost
100
80
60
40
20
0
0
20
40
60
80
100
120
140
160
Burst length
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
8
Example packet loss
distribution
ITU-T
Number of packets lost
100
Consecutive
Loss
80
60
20% Loss Rate
40
20
0
0
20
40
60
80
100
120
140
160
Burst length
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
9
Consecutive Loss Distribution
ITU-T
14000
Frequency
12000
10000
8000
6000
4000
2000
0
0
1
2
3
4
5
6
7
8
9
Consecutive Lost Packets
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
10
Lessons learned from analyzing
real
world
traces
ITU-T
o Packets are usually lost singly – much less
frequent to lose 2-3 consecutive packets
o Occasional very long consecutive loss periods
due to link failure
o Much more common to see high loss periods –
seconds in length – with loss densities of 30%
o Measurement of packet loss needs to be based
around a burst model – e.g. Gilbert-Elliott,
Markov Model with 3 or more states….
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
11
Jitter
ITU-T
o Specifically – short term variations in
transmission delay
o Real time systems often use a jitter
buffer to remove jitter – but increases
delay and packet loss
o The impact on performance is due to
the combined effect of jitter and the
jitter buffer
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
12
Example time series plot of
packet delay
Delay (seconds)
ITU-T
0.15
0.14
0.13
0.12
0.11
0.1
0.09
0.08
0.07
0.06
0.05
0
2
4
6
8
10
12
14
Time (seconds)
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
13
Jitter Characteristics
ITU-T
What should we measure?
- Packet to packet delay?
- Absolute delay?
- Delay with respect to ?
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
14
Jitter Characteristics
ITU-T
What really counts is what the end system would
do as a result of delay variations
Lesson: estimate (or count) which packets would
be discarded due to jitter
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
15
Timing Drift
ITU-T
Can occur between end systems and between end and
measuring system.
Effect on VoIP is usually small however can cause
significant measurement errors for delay and jitter
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
16
Estimating the effects of jitter
ITU-T
Delay (seconds)
Jitter Buffer Emulator
0.15
0.14
0.13
0.12
0.11
0.1
0.09
0.08
0.07
0.06
0.05
Discard
0
2
4
6
8
10
12
14
Time (seconds)
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
17
Measuring the combined effects
of
jitter
and
loss
midpoint
ITU-T
Jitter
Buffer
Emulator
Loss and
Discard
Events
Burst Loss Model
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
18
Measuring the combined effects
of
jitter
and
loss
endpoint
ITU-T
Burst Metrics API
- percent lost
- percent discarded
- gap length/density
- burst length/density
DSP – Jitter Buffer
Vocoder, Echo
Canceller
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
19
Delay
ITU-T
o Mid-stream systems often rely on RTCP
to estimate round trip delay
• RTCP is often not implemented by end
systems
• RTCP delay is only the VoIP packet path
delay and does not include external
circuit switched path delays
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
20
Sources of delay
ITU-T
Frame accumulation and encoding
RTP transmission delay
Receiving Jitter Buffer
Decoding, PLC, playout
External
End
End System Delays can be more substantial
than transmission delay, particularly with high
jitter levels and adaptive jitter buffers
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
21
RTCP XR VoIP metrics
ITU-T
o RTCP XR reports:• Proportion of packets discarded
• Proportion of packets lost
• Length and density of bursts (periods of high
loss/discard density)
• Length and density of gaps
• End system delay
• Packet path delay
• and more…………………
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
22
Recommendations
ITU-T
o Understand that packet path properties are time
varying
o Don’t measure jitter but do:• Measure packet discards resulting from jitter
• Count discards as lost packets
o Use a “proper” model for packet loss distribution (e.g.
Markov model with 3+ states, Gilbert-Elliott model…)
o Incorporate end system delays – if known
o Use RTCP XR!!!! (as this implements the above)
1-3 October 2003
Workshop on End-to-End Quality of Service. What is it? How do we get it?
23