Measurement and Modelling I:
End-to-End Bandwidth Measurements
6.829 Lecture 11
October 10, 2002
Jacob Strauss
1
End-to-End Measurements
Goal: Bandwidth-Aware Applications
• Is a path fast enough?
• Which path is faster?
What is bandwidth?
• Link Capacity: physical link speed (e.g. 100 Mbps Ethernet)
. for link i, link capacity ci.
• Path Capacity: minimum capacity link along a path
. C = mini=0...n(ci)
• Link Utilization: fraction of link capacity used over some time interval
. 0 ≤ ui ≤ 1
• Available Bandwidth: minimum spare capacity along a path over some interval
. A = mini=0...n(1 − ui)ci
2
Example Topology
X
C=100 Mbps, u=0.95
Capacity from X to Y?
C=20 Mbps, u=0.5
Y
C=10 Mbps, u=0.2
Example Topology
C=20 Mbps, u=0.5
X
Y
C=100 Mbps, u=0.95
Capacity from X to Y?
C=10 Mbps, u=0.2
C = 10M bps
Available bandwidth from X to Y?
Example Topology
C=20 Mbps, u=0.5
X
Y
C=100 Mbps, u=0.95
Capacity from X to Y?
C=10 Mbps, u=0.2
C = 10M bps
Available bandwidth from X to Y?
A = 5M bps
3
Problem: Capacity Estimation
C1
X
C2
C3
Y
How can we measure the path capacity between X and Y ?
4
Packet Pair on an unloaded path
C1
C3
C2
X
Y
t
t
Send two packets, of P bits each, back-to-back
Record the difference between arrival times, ∆t.
Path Capacity is
P
∆t
5
Packet Pair Complications: Cross Traffic
C1
C3
C2
Y
X
t
t
probe packet
other packet
P
∆t >
C
6
Packet Pair Complications: Multiple Queues
C1
C3
C2
Y
X
t
t
probe packet
other packet
P
∆t <
C
7
Delay Measurements Block Diagram
Packet Source
Network
Timestamps
Filters
Estimates
8
New Problem: Path capacity and packet size
distribution
Long train of packets with size P , sent every δ seconds
Record either one-way-delay or round-trip time for each packet
• RTT is easier to measure. Why?
Use phase plots:
• For each n, plot a point at (rttn, rttn + 1)
9
Delay Components & Model
Components
•
•
•
•
Propagation delays
Queuing delays
Processing delay (lookup & scheduling)
Transmission delay
Model assuming a single queue
• Fixed delay: D
• waiting time: wn
• service time: yn =
P
µ
• total delay: rttn = D + wn + Pµ
10
Explain the Plot: light load conditions
Assume that the tight link load is low:
• wn+1 = wn + n
⇒ rttn+1 = rttn + n
Explain the Plot: light load conditions
Assume that the tight link load is low:
• wn+1 = wn + n
⇒ rttn+1 = rttn + n
Points plotted near the line y = x above (x, y) = (D, D)
11
Explain the Plot: heavy load conditions
Assume B bits in between probe packets n and n + 1
wn+1 = wn + B
µ −δ
Explain the Plot: heavy load conditions
Assume B bits in between probe packets n and n + 1
wn+1 = wn + B
µ −δ
Now assume
B
µ
δ
• We have probe compression: k − 1 packets arrive before B clears the queue
• These k packets depart every P/µ seconds
for the k packets: wn+i − wn+i−1 = P/µ − δ
• why is P/µ − δ < 0?
Explain the Plot: heavy load conditions
Assume B bits in between probe packets n and n + 1
wn+1 = wn + B
µ −δ
Now assume
B
µ
δ
• We have probe compression: k − 1 packets arrive before B clears the queue
• These k packets depart every P/µ seconds
for the k packets: wn+i − wn+i−1 = P/µ − δ
• why is P/µ − δ < 0?
result is rttn+1 = rttn + (P/µ − δ)
Explain the Plot: heavy load conditions
Assume B bits in between probe packets n and n + 1
wn+1 = wn + B
µ −δ
Now assume
B
µ
δ
• We have probe compression: k − 1 packets arrive before B clears the queue
• These k packets depart every P/µ seconds
for the k packets: wn+i − wn+i−1 = P/µ − δ
• why is P/µ − δ < 0?
result is rttn+1 = rttn + (P/µ − δ)
We now know the bottleneck capacity µ from the plot.
12
Packet size distribution
Assume that between packets n and n + 1, bn bits from other flows
join queue
• Use Lindley’s recurrence:
wn+1 = (wn + yn − δn)+, where x+ means max(x, 0)
Packet size distribution
Assume that between packets n and n + 1, bn bits from other flows
join queue
• Use Lindley’s recurrence:
wn+1 = (wn + yn − δn)+, where x+ means max(x, 0)
Apply twice:
wbn = (wn + P/µ − δb)+
wn+1 =
+
(wn + P/µ − δb) + bn/µ − (δ − δb)
+
Packet size distribution
Assume that between packets n and n + 1, bn bits from other flows
join queue
• Use Lindley’s recurrence:
wn+1 = (wn + yn − δn)+, where x+ means max(x, 0)
Apply twice:
wbn = (wn + P/µ − δb)+
wn+1 =
+
(wn + P/µ − δb) + bn/µ − (δ − δb)
+
wn+1 = wn + (P + bn)/µ − δ
bn = µ(wn+1 − wn + δ) − P
Now we can find packet sizes from peaks in PDF of wn+1 − wn + δ
13
Example Distribution Plot
700
1500B
500B
b2b
800
sent gap
Squeezed Pair Histogram: ccicom.ron.lcs.mit.edu --> nyu.ron.lcs.mit.edu
600
pair count
500
400
300
200
100
0
0
50
100
150
received gap in usec (2 us buckets)
200
14
New Problem: Available Bandwidth Estimation
Can we use the same packet train to estimate A?
New Problem: Available Bandwidth Estimation
Can we use the same packet train to estimate A?
No, but we can determine whether P/δ > A or P/δ < A
• Later we’ll build on this test to estimate A
New Problem: Available Bandwidth Estimation
Can we use the same packet train to estimate A?
No, but we can determine whether P/δ > A or P/δ < A
• Later we’ll build on this test to estimate A
Basic idea: instead of just comparing wn+1 to wn, compare wn to
wn+1, wn+2, wn+3, . . .
Method: Plot evolution of wn versus n
15
Pathload Delay Model
The packet train consists of K packets of size L sent at a constant
rate R.
Path consists of H links, each with capacity Ci, available bandwidth
Ai, and queue length qik when the kth probe packet arrives
Pathload Delay Model
The packet train consists of K packets of size L sent at a constant
rate R.
Path consists of H links, each with capacity Ci, available bandwidth
Ai, and queue length qik when the kth probe packet arrives
The One-Way-Delay for probe k is:
Dk =
H X
L
i=1
Ci
+
qik
Ci
Pathload Delay Model
The packet train consists of K packets of size L sent at a constant
rate R.
Path consists of H links, each with capacity Ci, available bandwidth
Ai, and queue length qik when the kth probe packet arrives
The One-Way-Delay for probe k is:
Dk =
H X
L
i=1
Ci
+
∆Dk ≡ Dk+1 − Dk =
qik
Ci
H
X
∆q k
i
i=1
Ci
16
Increasing Trends
If R > A, then ∆Dk > 0 for k ≥ 1
If R ≤ A, then ∆Dk = 0 for k ≥ 1
Increasing Trends
If R > A, then ∆Dk > 0 for k ≥ 1
If R ≤ A, then ∆Dk = 0 for k ≥ 1
Intuition for a single link i with Ai < R:
• Queue grows longer with each probe packet that arrives
• ∆qik = (L + uiCiT ) − C1T = (R − Ai)T > 0
For a single link with Ai ≥ R:
• ∆qik = 0
By induction, show that ∆Dk > 0 when R > A
Increasing Trends
If R > A, then ∆Dk > 0 for k ≥ 1
If R ≤ A, then ∆Dk = 0 for k ≥ 1
Intuition for a single link i with Ai < R:
• Queue grows longer with each probe packet that arrives
• ∆qik = (L + uiCiT ) − C1T = (R − Ai)T > 0
For a single link with Ai ≥ R:
• ∆qik = 0
By induction, show that ∆Dk > 0 when R > A
Plot Dk – if ∆Dk > 0 then R > A
17
Complications
Ai not a constant
• varies on both short and long time scales
Need to choose K and L carefully
• Too short – can’t tell if ∆Dk > 0
• Too long – flood link
• Compromise – use multiple trains
How did we choose R initially?
18
Iterative Search for A
maintain two state variables: Rmax and Rmin.
always have Rmin ≤ A ≤ Rmax .
Pick a new R halfway between Rmax and Rmin
Test whether R <> A, update either Rmax or Rmin
Stop when Rmax and Rmin are close enough.
In practice the search is more complicated because the outcome of
testing R <> A is sometimes unsure
19
Pathload Block Diagram
Packet Source
Network
Timestamps
Filters
Estimates
20
Available Bandwidth isn’t the full story
BTC vs Pathload: 15 RON hosts -- one test per path
1
0.9
0.8
0.7
CDF
0.6
0.5
0.4
0.3
0.2
0.1
0
0.01
0.1
1
BTC / Pathload rate
10
100
21
Conclusion
There’s a lot you can learn with simple probes
• packet pair
• packet trains — regular interval, constant packet size
Just by looking at packet delay variations you can determine
• Path Capacity
• Common Packet Sizes
• Available Bandwidth
22