MEASURING THE ARRIVAL QUALITY OF REAL-TIME
PACKET TRAINS - A GLOBAL PERSPECTIVE
Ulrich Speidel, 'Etuate Cocker, Nevil Brownlee
Department of Computer Science, The University of Auckland
ACADEMIC CONTRIBUTORS TO DATE






Niko Rebenich, Stephen Neville, Aaron Gulliver, University of
Victoria, B.C., Canada
Kashif Nisar, Zeeshan Aziz, Suhaidi Hassan, Universiti Utara
Malaysia
Ming-Chui Dong, Vincent Wong, University of Macau
Raimund Eimann, Zurich, Switzerland (in a private capacity)
Bernhard Platter, Bernhard Ager, ETHZ Zürich
ITS, University of Auckland
Ulrich Speidel – ulrich@cs.auckland.ac.nz
INDUSTRY CONTRIBUTORS TO DATE








Ministry of Lands, Survey, and Natural Resources, Tonga
Connect ISP, Fiji
Ministry of Infrastructure and Planning, Cook Islands
Oyster ISP, Rarotonga
Government of Kiribati
Government of Tuvalu
E.M. Jones Ltd., Tonga
Ministry of Revenue, Tonga
Ulrich Speidel – ulrich@cs.auckland.ac.nz
AGENDA









VoIP and other real-time protocols
Buffering
Causes of variability in packet arrival
Implications of explosive network growth
Possible implications for remote communication
The beacon network – what is it and how does it work?
Why the Pacific Islands are a great test bed – and why
other parts of the world are important, too
Measures of arrival quality: jitter and entropy
Preliminary observations
Ulrich Speidel – ulrich@cs.auckland.ac.nz
WHY ARE REAL-TIME PROTOCOLS IMPORTANT?

Voice over IP (VoIP) telephony / videoconferencing

Telemedicine (e.g. remote surgery / diagnostics /
consultation)

Remote control in industrial applications

Internet of Things

Financial applications (rapid trading)
Ulrich Speidel – ulrich@cs.auckland.ac.nz
VOICE OVER IP AND OTHER REAL-TIME PROTOCOLS
• analog voice signal
is digitally encoded
Transmitter
Packets
• encoded byte
stream is chopped
into a packet train
blah
A/D
Encoder
Internet
Buffer
blah
Decoder
D/A
Receiver
Ulrich Speidel – ulrich@cs.auckland.ac.nz
• packets are
transmitted via
Internet to receiver
• receiver buffers
packets to establish
constant-rate data
flow to decoder
• decoder recovers
analog signal and
replays it
VOICE OVER IP AND OTHER REAL-TIME PROTOCOLS

Delay between signal generation and replay is
undesirable – must be kept as small as possible

Replay delay is composed of three components:
1.
2.
3.
Physical latency of cables or satellite links along
packet paths (fixed for a given path)
Time spent in router queues (variable)
Size of buffer at receiver (governed by variation in
arrival times)
Ulrich Speidel – ulrich@cs.auckland.ac.nz
CAUSES OF VARIABILITY IN PACKET ARRIVAL

Length and number of router queues

Overflowing router queues cause packet drops

Load balancing at routers
Ulrich Speidel – ulrich@cs.auckland.ac.nz
LOAD BALANCING ROUTERS

Load balancing routers send packets to the same destination across
different links

If packets from the same stream are load balanced, it causes them to take
different paths and experience different latencies

In some cases, packets may overtake each other (out-of-order arrivals)
Ulrich Speidel – ulrich@cs.auckland.ac.nz
PACIFIC RIM SUBMARINE CABLES
Which way do packets
travel from Macau to NZ?
What are the possible
latencies?
http://www.submarinecablemap.com/
Ulrich Speidel – ulrich@cs.auckland.ac.nz
LONG-DISTANCE FIBRE OPTICS
"Long fat network"
 Expensive to put in and maintain – prohibitive
for some destinations
 Become a huge resource once installed
 Growing in number around the world
 Denser mesh

Ulrich Speidel – ulrich@cs.auckland.ac.nz
SATELLITE CONNECTIVITY






"Long thin network"
Only commercially viable technology for small remote
locations
Geostationary satellites have a latency of 240-280 ms
between ground stations (compared to around 70 ms for
a trans-Pacific fibre cable)
MEO and LEO satellites promise overall lower latency,
but with large variations
Significant cost given low bandwidth and high latency
Still a growth industry!
Ulrich Speidel – ulrich@cs.auckland.ac.nz
OBSERVATIONS: PING RTT








U. of Auckland to www.u-tokyo.ac.jp: 142-149 ms (20 hops, via
Australia)
U. of Auckland to www.umac.mo: >220 ms (>17 hops, via Australia
and Japan)
U. of Auckland to www.uj.ac.za: >505 ms (>26 hops, via US & UK)
U. of Auckland to www.uvic.ca: 158-160 ms (13 hops, via US)
U. of Auckland to www.ethz.ch: >298 ms (>21 hops, via US)
U. of Auckland to lands.gov.to: 803-1074 ms (21 hops, via US,
Canada and satellite)
U. of Auckland to oyster.co.ck: 585-593 ms (15 hops via Australia
and satellite)
Conclusion: geographic distance and even fibre routes are only a
guide!
Ulrich Speidel – ulrich@cs.auckland.ac.nz
NETWORK GROWTH







Increase in number of users
Users create more traffic (volume of traffic estimated to
increase by a factor of 1000 per 20 years)
New protocols and applications appear
Topology changes: links are added / upgraded, routers are
added / upgraded. Hop counts tend to increase over time.
Multiple path alternatives are increasingly common,
physically shortest paths aren't necessarily the ones chosen
for routing (commercial factors).
Network mesh grows denser
Load balancing used increasingly to improve reliability
Ulrich Speidel – ulrich@cs.auckland.ac.nz
IRREGULAR PACKET ARRIVAL
Need to buffer more
 Real-time services such as VoIP, video
conferencing, and remote manipulation may
become impractical
 Downloads and streaming video/audio would still
work
 Significant economic impact (outsourced call
centre industry)
 Effects are most likely to be felt between remote
endpoints first

Ulrich Speidel – ulrich@cs.auckland.ac.nz
WHICH EFFECT WILL WIN?

Technological progress or the forces of chaos it
unleashes?
more traffic
more powerful
routers
more congestion
extra bandwidth
new links
more malware
more users
more efficient codecs
Ulrich Speidel – ulrich@cs.auckland.ac.nz
more inefficient routing
less predictable packet
arrivals
CAN WE PREDICT A TREND?

Basic idea: Pick a sample application, e.g., VoIP

Repeatedly use the application over time between remote end
points where deterioration is likely to be seen early

See whether you observe any long-term changes

Do this for multiple pairs of endpoints so you can ascertain
whether a change is specific to a particular endpoint or pair of
endpoints
Ulrich Speidel – ulrich@cs.auckland.ac.nz
FUNDAMENTAL CHALLENGES

Applications don't stay stable


Network topology changes & observation points shift


Accept as part of the observable & observe between multiple endpoints
Effects seen may be local rather than remote


Use synthetic reproducible traffic instead, from a software whose behaviour we
can control
Compare observations of multiple endpoints in vicinity
Effects seen may apply to a particular path (pair of endpoints) only

Have more than just a few endpoints
Ulrich Speidel – ulrich@cs.auckland.ac.nz
THE BEACON NETWORK

Basic idea:
"Run a global network of beacon computers that act pair-wise
as endpoints"

To date (April 2013), 16 beacons operate in Canada, Cook
Islands, Fiji, Kiribati, Macau, Malaysia, New Zealand,
Switzerland, Tonga, and Tuvalu

…and the network is growing!
Ulrich Speidel – ulrich@cs.auckland.ac.nz
THE BEACON NETWORK
Ulrich Speidel – ulrich@cs.auckland.ac.nz
BEACON SOFTWARE

Software is a C program that runs one experiment and
then shuts down

Scheduled via cron jobs & configured via command
line parameters

Writes plain text logs

Requires sufficient privileges to run raw sockets (to
determine TTL of received packets)
Ulrich Speidel – ulrich@cs.auckland.ac.nz
BEACON UDP EXPERIMENTS

Beacon exchanges 10,000 UDP packets of 110 bytes with a partner beacon

Packets transmitted every 20 ms

Transmit time logged at transmitting end (2 timestamps)

Packets are timestamped and serial-numbered

At receiving end, packets are logged with arrival time stamp, serial number,
arrival sequence number, and TTL observed

This experiment typically runs 3 times a day between selected beacon pairs

Unidirectional option also exists
Ulrich Speidel – ulrich@cs.auckland.ac.nz
BEACON TCP EXPERIMENTS

Two beacons exchange a predefined set of data via TCP

Two modes available: VoIP (constant payload rate) and
Download (maximum payload rate)

Transmitting end logs nothing except fact that experiment took
place + config

Receiving end logs data arrival at application layer plus TTL and
size of packets involved in communication (from raw socket)
Ulrich Speidel – ulrich@cs.auckland.ac.nz
A QUESTION OF TIMING

Less interested in total latency than in its variability (jitter)

Sending and receiving beacons need clocks that are
sufficiently stable for the duration of an experiment

But: No need for perfect synchronisation (NTP is sufficient)
Ulrich Speidel – ulrich@cs.auckland.ac.nz
BEACON HARDWARE

PC or Mac platform with Ubuntu/BSD/OSX

Doesn't need to be overly fast – we use 600 MHz ALIX boards
among others

Must have real-time clock, but GPS / atomic clock
synchronisation is not required

VM's are not a good idea – tend to spend too much time away
from task at hand
Ulrich Speidel – ulrich@cs.auckland.ac.nz
LOCATION, LOCATION!

Would like to observe longer paths (both in hops and in
latency)

Higher likelihood of out of order arrivals if path latencies
differ substantially

Interested in "user experience", so beacons don't need
to be near the backbone

While doing so, want to avoid measuring purely local
effects
Ulrich Speidel – ulrich@cs.auckland.ac.nz
WHY HAVE BEACONS ALL OVER THE WORLD?

Want long-term global trend, not just local effects

Want a "developed world" baseline but also see what it is like in remote
places on the fringe – many of our beacons run in the Pacific for that reason
(need I mention Africa?)

Long paths generally are of interest – both in terms of latency and number
of hops

A lot of international traffic passes through "hub regions" (North America,
Europe, SE Asia). What effect do these regions have on traffic that passes
through them?

Last but not least: We're looking for input (and your own experiments)!
Ulrich Speidel – ulrich@cs.auckland.ac.nz
BEACON CHALLENGES

Proper timing without proper timebase

Precise timing on multitasking computers

Getting two beacons to communicate (firewalls, ports, policies/politics)

Scheduling

Data volume backhaul (each beacon contributes ~3MB / experiment / day)

Data processing & presentation (currently UDP only)

Coordination with many partners, each with a unique network setup

Making it longitudinal
Ulrich Speidel – ulrich@cs.auckland.ac.nz
PACKET TRAIN QUALITY

Subjective approaches, e.g., Mean Opinion Score (MOS) –
reliably replicable only with very large sample

Objective approaches, e.g., jitter measurements. But is jitter
really random?

Stop & recap: What do real-time protocols need again?
Ulrich Speidel – ulrich@cs.auckland.ac.nz
WHAT RTP'S NEED

Regular packet arrival

In-order arrival

Low packet loss

So how do we come to grips with the "regular"?

Answer: Information theory can help!

But first, a quick look at initial observations…
Ulrich Speidel – ulrich@cs.auckland.ac.nz
PACKET LOSS
Ulrich Speidel – ulrich@cs.auckland.ac.nz
OUT-OF-ORDER ARRIVALS
Ulrich Speidel – ulrich@cs.auckland.ac.nz
LOSS VS OUT-OF-ORDER
Ulrich Speidel – ulrich@cs.auckland.ac.nz
JITTER

Knowing (logged) transmit & receive times tt and tr, we can compute
average latency t for the experiment:
1 n 1
t   (t r  tt )
n i 0

Note this includes any clock offset between transmitter and receiver clock

Can then compute jitter as average absolute deviation from the average
latency:
1 n 1
J   | t r  tt  t |
n i 0

Clock offset disappears

Gives a guide as to buffer size required
Ulrich Speidel – ulrich@cs.auckland.ac.nz
JITTER ISN'T ALWAYS A CLEAR MEASURE

Consider the following simplified scenario:

Assume uncongested routers but differences in latency, blue router load balances

Have two paths/latencies -> high jitter, but not really queue-induced

Jitter ignores systematic patterns in arrivals

A good entropy estimator (i.e., not Shannon) can detect the difference
Ulrich Speidel – ulrich@cs.auckland.ac.nz
COMPLEMENT: ENTROPY (RATE)

Entropy: "measure of disorder in a system". Usually given in the form
of an entropy rate rather than absolute entropy

Entropy rate: number of bits of actual information required to
represent an object divided by number of bits actually used to
represent it.

Dimensionless ("bits/bit")

Examples: Shannon entropy, Lempel-Ziv compression ratios, Tentropy, …

"Really unpredictable arrivals" = high entropy
Ulrich Speidel – ulrich@cs.auckland.ac.nz
ENTROPY HOW-TO

Map inter-arrival times of successive UDP packets to symbol bins, e.g.:



t < 17 ms: "A"
17 ms < t < 19 ms: "B"
19 ms < t < 21 ms: "C"
…

Form string from these symbols: "CCCBDCACF…"

Determine entropy rate for string (e.g., as Lempel-Ziv compression ratio or Tentropy)

"Perfect" string will be "CCCCCCC…" – highly compressible, low entropy

Chaotic arrivals generate many new pattern combinations: harder to
compress, higher entropy
Ulrich Speidel – ulrich@cs.auckland.ac.nz
ENTROPY VS. JITTER
Ulrich Speidel – ulrich@cs.auckland.ac.nz
ENTROPY VS. JITTER
5 bin T-entropy TO2-NZ3 vs. transit jitter
1
0
0.2
0.4
0.6
0.8
1
Jitter [s]
0.1
0.01
0.001
T-entropy [bits/symbol]
Ulrich Speidel – ulrich@cs.auckland.ac.nz
1.2
1.4
1.6
BUT IT AIN'T ALWAYS THAT WAY
Ulrich Speidel – ulrich@cs.auckland.ac.nz
WHERE IS ENTROPY INTRODUCED?
Ulrich Speidel – ulrich@cs.auckland.ac.nz
CHANGING TTLS – THE SMOKING GUN
Ulrich Speidel – ulrich@cs.auckland.ac.nz
PICKING A TREND?
Ulrich Speidel – ulrich@cs.auckland.ac.nz
CONCLUSIONS

Real-time traffic and best-effort protocols are uneasy companions

We propose to use a large beacon network to measure arrival shape
of packet trains

Our beacons already see interesting effects

Effects are strongly path-specific and sometimes not easily explained

Jitter and entropy are both useful for arrival shape evaluation

A lot of work remains to be done!
Ulrich Speidel – ulrich@cs.auckland.ac.nz