INF570
LEDBAT/uTP
dario.rossi
INF570
v08/2013
Dario Rossi
http://www.enst.fr/~drossi
Plan
• Bufferbloat
– Problem
– Extent
• LEDBAT/uTP
– Historical
– Congestion control
– BitTorrent swarm
– Bufferbloat
• References
Bufferbloat
• RTT Delay between two Internet hosts ?
Internet
~100ms
~300ms
~2.5s
~5s !?
Bufferbloat
• RTT Delay between two closeby Internet hosts ?
Bufferbloat!
RTT may grow to several seconds!
Nasty impact on interactive Web,
VoIP, gaming traffic, etc.
TCP Alice to Bob
TCP Bob to Alice
5
RTT earth-moon
3
2
RTT
1
Source:
[PAM’10]
0
100
200
300
Time [s]
400
500
600
0
RTT [s]
4
Bufferbloat
• Bufferbloat = very large queuing delay
– New word, old “persistently full buffer” problem
• Root causes
– Narrow cable, ADSL pipe capacities (few 100Kbps)
– Relatively large modem buffers (~100KB)
– TCP dynamics are loss driven, hence buffers fill
Internet
~100ms
~400ms
~2.5s
~5s !?
Bufferbloat: single household, US
J. Gettys home, Comcast cable line
Bufferbloat: single household, EU
D. Rossi home, Free DSL line
Bufferbloat: uplink
Netalyzr
uplink
Bufferbloat: downlink
Netalyzr
donwlink
Bufferbloat: where and why ?
dst
pc
– OS stack (and above)
hop
Box
nas
Dns Gw
ntp
End usr
ISP box ISP net
devices
PC1 pings to google while PC2 backup to NAS
Backup:
large
files
• Examples
Backup:
many small files
Backup ended
• (L2) Eth driver, (L3) pfifo_fast
queue, (L4) TCP/UDP buffers,
(L7) application buffer
– WiFi AP
• e.g. when speed changes from
54 to 2Mbps
– Interfering traffic
• e.g., PC to NAS backup use
WiFi AP while a smartphone
access the Internet
– DSL box
• Unless AQM is used (more
details later)
LEDBAT: BitTorrent to the rescue?
Peer
File chunks
Chunk transmission
LEDBAT
congestion
control
LEDBAT: BitTorrent to the rescue?
• BitTorrent announces
closed source code,
data transfer over UDP
• BitTorrent + UDP
= Internet meltdown!
• After BitTorrent denial
and some discussion...
• Everybody agreed that
Internet not gonna die
• But is LEDBAT really
the best approach ?
LEDBAT: BitTorrent to the rescue?
• BitTorrent solution
– Side effect of the Internet meltdown buzz, BT had to explain itself
– Two fora: IETF and BitTorrent Enhancement Proposal (BEP)
IEFT (LEDBAT)
•Low Extra Delay BAckground Transport
•Definition of a low priority delaybased congestion control algorithm
• Two names, same goals:
–
–
–
–
Efficiently use the available bandwidth
Keep delay low on the network path
Quickly yield to regular TCP traffic
LEDBAT/UTP used interchangeably
BEP29 (UTP)
•UDP framing for
interoperability
among BT clients
• Assumptions:
- Bottleneck = user
access link
- Congestion = self-induced
by users’ own traffic
Research timeline: the big picture
• Two solutions to bufferbloat
– Active Queue Management (AQM)
– Low-priority/Delay-based Congestion Control (CC)
SFQ
RED
DRR
Choke
Vegas
1990
1993
1995
2000
CoDel
NICE
2002
TCP-LP
2003
LEDBAT
2010
2012
Let expand
this further
LEDBAT/uTP evolution
TCP
v5.2.2
Oct ‘08
α1
v1.9-13485
Dec ‘08
α2
v1.9-15380
Mar ’09
β1
v1.9-16666
Aug ‘09
RC1
v2.0.1
Apr’10
Open source
Closed source
First LEDBAT draft
Draft as WG Item
After IETF 77
Testbed
Simulation
Analysis
[PAM’10]
Passive
monitoring
Packet size [Bytes]
1500
α2
β1
TCP
α1
vv1.9-15380
1.9-16666
v5.2.2
v1.9-13485
Mar
Aug
‘09
’09
Oct
Dec ‘08
‘08
Draft
First LEDBAT
assource
WG
Open
source
Closed
draft
Item
1250
1000
750
500
250
0
0
10
20 30
Time [s]
40
50
•TCP transfers, full playload
•a1 small packet overkill !!
•a2 variable framing (not in draft)
•b1 finer bytewise cwnd control
•The evolution continues at IETF
LEDBAT/uTP evolution
libUTP
open src
May’10
2.0.2
2.2
Griffin b
v2
v3
Jul’10
Oct’10
[ICCCN’10] [LCN’10]
2.2
3.1 a
3.0
v4
v5,6
v7
v8,9 ~RFC
Dec’10 Mar’11 Apr’11 May’11 Jul’11 Sep’11 Oct’11 Nov’12
[P2P’11] [TMA’12,
P2P’12,
CoNEXT’12]
[GLOB’10,TR1]
LEDBAT fast growing
•on most BT clients
•>50% BT traffic
LEDBAT almost RFC
• with some bugs!
Source: [TR1]
Viewpoints
• Congestion control viewpoint
– Latecomer unfairness [ICCCN’10] + solution via simulation
[GLOB’10] and analytical modeling [TR1]
– LEDBAT vs other low priority protocols + magic numbers
[LCN’10]
• BitTorrent swarm viewpoint
– Reverse engineering of the closed source protocol [PAM’10]
– Impact of LEDBAT on swarm completion time via simulation
[P2P’11] and experiments [TMA’12]
• Bufferbloat viewpoint
– LEDBAT vs AQM [CoNEXT’12]
– Exploiting LEDBAT to gauge Internet bufferbloat delays [P2P’12]
Congestion control
viewpoint
dario.rossi
``Because LEDBAT is an IETF protocol,
whose scope is larger than BitTorrent’’
Congestion control viewpoint plan
• High-level idea and protocol specification [ICCCN’10]
• Latecomer unfairness: problem, extent, solution
[GLOBECOM’10,TR’10]
• Protocol tuning and magic numbers [LCN’10]
• Comparison with others low-priority protocols
[LCN’10]
• Experiments with BitTorrent implementation
[PAM’10]
LEDBAT operations: high level
TCP
Detect congestion by losses
LEDBAT
Infer via delay measurement
• Increment the congestion window
(cwnd) by one packet per RTT
• Halve cwnd on loss events
• Senders measure minimum delay
• Evaluate offset from TARGET delay
• React with linear controller
Consequences
Aim
• The buffer always fills up
• High delay for interactive apps
• Users need to prioritize traffic !
cwnd
losses
time
•
•
•
•
At most TARGET ms of delay
Lower priority than TCP
Do not harm interactive application
Avoid self-induced congestion
cwnd
TARGET
reached
time
LEDBAT operations: gory details
•
Pseudocode in draft v9 (31/10/2011)
One-way delay
(affected by clock offset skew)
@RX:
remote_timestamp = data_packet.timestamp
ack.delay = local_timestamp() - remote_timestamp
ack.send()
@TX:
current_delay = ack.delay
base_delay = min(delay, base_delay)
Base delay = min delay seen
(hopefully finds an empty queue)
Queuing delay estimation
(clock offset cancels !)
queuing_delay = current_delay - base_delay()
off_target = (TARGET - queuing_delay)/TARGET
cwnd += GAIN * off_target * bytes_newly_acked * MSS / cwnd
•
•
Note: lots of other details in the draft (e.g., route
change, sample filtering, etc.) but not in today talk
TARGET = 25ms (“magic number” fixed in draft v1)
= 100ms (in BEP29), <= 100ms (from draft > v5)
Linear response to
distance from target
(at most grows by 1 MSS
after a full cwnd is acked)
LEDBAT vs TCP
CWND [pkts]
80
5,6
40
3
4
0
40
Queue [pkts]
2
TCP
LEDBAT
Total
1
20
0
Source
[ICCCN]
4
8
Time [s]
12
1. TCP and LEDBAT start together
2. As soon as q > 20pkt
-> queuing delay > 25ms
-> LEDBAT decreases and stops
3. TCP experiences a loss (due to
TCP AIMD) and halves its rate
4. Queue empties, LEDBAT restarts
5. TCP is more aggressive (larger
cwnd w.r.t. t=0), LEDBAT yields
6. Cyclic losses
• LEDBAT is lower priority than TCP
• Total link utilization increases
(w.r.t TCP alone)
LEDBAT vs LEDBAT
80
CWND [pkts]
Total
1. LEDBAT flows start together
2. Queue builds up
3. As soon as q = 20pkts
-> delay=25ms
-> linear controller settles
4. Queue keeps stable
5. No changes until some
other flows arrive
40
LEDBAT 1
LEDBAT 2
0
Queue [pkts]
40
20
0
Source
[ICCCN]
4
8
Time [s]
12
• LEDBAT flows efficiently use
the available bandwidth
• The bandwidth share is fair
between LEDBAT flows
Latecomer unfairness
80
• Second flow starts before queue
builds up: same target, but
latecomer gets a smaller share
∆T = 5s, B=40
40
0
5
CWND [pkts]
80
15
25
• Second flow starts after first has
reached its target: latecomer sets
higher target (delay due to first
flow ends up in base delay)
After loss, base delay is possibly
corrected (queue empties)
∆T = 10s, B=40
40
0
5
80
15
25
∆T = 10s, B=100
• In case of larger buffer, latecomer
starve the firstcomer
40
0
Source
[ICCCN]
5
15
Time [s]
25
Latecomer unfairness
• Implementation
– LEDBAT as a new TCP flavor in ns2
– (also works as a Linux kernel module)
• Scenario
– Single bottleneck
– Backlogged transfers
• Parameters
– Buffer size B
– Time between flows start ∆T
– Target = 100 ms
• Methodology
– Simulation [ICCCN’10]
– Experiments [TR1]
∆T
100Mbps
10Mbps
100Mbps
B
Latecome unfairness
Real BitTorrent code
in a testbed [TR1]
Latecomer overestimates
the base delay (includeshttps://github.com/bittorrent/libutp
queueing due to the first)
Firstcomer
senses!
Firstcomer
starvation
delay
(nowincreasing
stated in small prints
afterand
tons of disclaimers in draft v9)
slowdown sending rate
Source
[TR1]
Root cause of latecomer unfairness
• Intrinsic to AIAD dynamics!
– Known since the late 80s!!
– Very well known, over
1500 citations!
D. Chiu and R. Jain, "Analysis of the
Increase/Decrease Algorithms for
Congestion Avoidance in Computer
Networks," Journal of Computer
Networks and ISDN, Vol. 17, No. 1,
June 1989, pp. 1-14
AIMD
AIAD
Solution to latecomer unfairness
Source: [GLOB]
Several possibilities:
Random
Random
:( pacing
:/ cwnd drop
• Random pacing
•Proposed on the •Throughput
• Slow start
IETF mailing list
fluctuations
•Efficiency ok,
• Random cwnd drop •Doesn’t work
(more later)
fairness almost ok
• Proportional drop
fair-LEDBAT
(fLEDBAT)
•Reintroduces AIMD
•Use proportional drop
•Details, simulation,
fluid model: see [TR1]
:)
Slow
start
•A loss empties
the queue…
•..but hurts
interactive traffic
:(
:/
Proportional
cwnd drop
•Synchronization
•Fairness ok,
efficiency almost ok
Extent of latecomer unfairness
• Fairness issue affects backlogged connections
– If tx stops => queue empties => base delay measure is correct => all ok
• Relevance of latecomer unfairness with traffic models ?
– Chunk based: M=4 simultaneous connections out of N>M peers
– After each chunk, continue with the same peer with probability P
– Rough model of chocking, pipelining, optimistic unchoking, etc.
TX
RX1
RX2
RX3
fLEDBAT
LEDBAT
…
RXN
Source:
[TR1]
Persistence probability P
Sensitivity analysis + low-priority comparison
• LEDBAT and TCP-NICE as new flavors in ns2
• Bottleneck with capacity C=10 Mbps, other links at 100 Mbps
• RTT=50 ms, packet size pktS=1500 B, buffer Bmax=100 pkts
• Flows start at same time ΔT=0s (avoid latecomer unfairness)
Sender
...
Efficiency
(η)
x


i
C
i
Average queue
occupancy (B )
B
E B 
Bmax
2 to N
Receivers
Jain fairness index
(F )
2
 N 
  xi 
F   i 1 N 
N   xi2
i 1
F∈[1/N,1]
F=1 max fairness
F=1/N max unfairness
Sensitivity analysis
• Inter-protocol case : LEDBAT vs NewReno
– Target T∈[2,150]% of Bmax
– Gain G∈[1,10] as in draft
• Main findings
– Gain only minimal impact
– Target has profound impact; hard to tune the level of priority
Target T [%]
Target T [%]
f(G)
f(T)
Fairness F
Efficiency η
R1
Gain G
R2
R3
Gain G
R4
R1: unstable
R2: low priority
R3: transient
R4: loss-based
Sensitivity analysis
• Intra-protocol case : LEDBAT vs LEDBAT
– Target ratio T1/T2 ∈[1,10]
– Gain G=1 since minimal impact
• Main findings
η, B, F metrics
– Even small target differences yield to serious unfairness
Efficiency η
Fairness F
Avg queue B
R0
R2
R1
Target T1/T2
Fair: R0: T1=T2
Unfair: R1: T1+T2<100% Bmax
Lossy: R2: T1+T2>100% Bmax
Low-priority congestion control
• NICE
– Inherits from Vegas, measure RTT delay, decrease
window when half of samples > E[RTT]
• TCP-LP
– EWMA of the OWD delay value; if threshold exceeded,
halves cwnd and enters congestion inference phase
• LEDBAT
– OWD estimation, continuous AIAD response to OWD
offset from target delay
Low-priority congestion control
cwnd [pkt]
NewReno
NICE
Time [s]
NICE.0
NICE.1
NewReno
TCP-LP
Time [s]
TCP-LP.0
TCP-LP.1
LEDBAT.0
LEDBAT.1
NewReno
LEDBAT
Time [s]
Multiple-flows comparison
• NewReno vs lower priority flows
– N∈[1,10] low priority flows vs 1 NewReno flow
• Main finding
LEDBAT
TCP-LP
NICE
NewReno
N=
Fairness F
Efficiency η
– LEDBAT and NICE efficient in exploiting spare capacity
– TCP-LP more aggressive than LEDBAT, NICE
N=
Multiple-flows comparison
• Lower priority flows against each other
– N∈[1,5] low priority flows for each flavors, 3N flows total
• Main finding
Throughput
breakdown
– Relative low priority rank: LEDBAT, NICE, TCP-LP
LEDBAT
TCP-LP
NICE
Total number of low priority flows
Experiments
• [PAM’10]
– Carried out before the draft
– Still relevant to point out difference
between theory and practice
• Active testbed mesurements
– Single-flow LAN experiments
• Timeline of LEDBAT evolution
• Emulated capacity and delay
– Multi-flow LAN experiments
• Level of low-priority ?
– ADSL experiments
• In the wild
• Interaction with cross-traffic
LAN: single-flow experiments
• Application setup
Leecher
Switch
10/100
– uTorrent, official BitTorrent
client, closed source
• on native Windows (XP)
• on Linux with Wine
– Private torrent between PCs
Forward
Seed
Backward
• Network setup
– Linux based router with Netem
to emulate network conditions
– Capture traffic on both sides,
analyze traces
• Experiment setup
Router
+
Netem
– Flavors comparison
– Delay impairment
– Bandwidth limitation
LEDBAT evolution
• Throughput evolution
– Different LEDBAT versions
– Each curve in separate experiment
– Also TCP BitTorrent (Linux + Win)
Throughput [Mb/s]
10
α1
β1
6
• Observations
TCP Linux
8
– a1 unstable, small packets
• Small packets recently broke b1 too
α2
– a2 , ß1 stable smooth throughput
• 4 and 7 Mbps respectively
4
TCP WinXP
• Window limits
2
– LEDBAT and TCP influenced by the
default maximum receive window:
0
0
60
120
Time [s]
180
240
•
•
•
•
TCP WinXP = 17 KB
TCP Linux = 108 KB
LEDBAT a2 = 30 KBytes
LEDBAT ß1 = 45 KBytes
LAN: Constant delay
• LEDBAT is delay based
– constant delay for all packets
– delay = k20 ms, k \in [1,5]
– forward (or backward) path
• Observation
– same behavior on both directions
– due to upper bound of maximum
receiver window (a2=20, ß1=30)
– not a constraint since bottleneck
shared among many flows
β1
5
60
0
240
0
600
Backward path
10
5
480
0
Delay [ms]
• Experiment setup
120
α2
Throughput [Mb/s]
– it measures the one-way delay
– sensitive to forward delay
– Independent from backward
Forward path
10
120
60
Delay
profile
240
480
Time [s]
600
0
LAN: Variable delay
• Observations
– a2 greatly affected by varying
delay on the forward path
– ß1 probably implements some
mechanism to detect reordering
– Minor impact of varying delay on
backward path (ack delay only
affects when decisions are taken)
Throughput [Mb/s]
– random uniformly distributed
delay for each packet
– reordering is possible!
– range = 20 ± {0,5,10,20} ms
– forward (or backward) path
Forward path
10
120
β1
α2
8
60
6
0
0
120
360
Backward path
10
8
6
240
120
60
Delay
profile
0
Delay [ms]
• LEDBAT is delay based
• Experiment setup
120
240
Time [s]
360
0
LAN: multi-flow experiments
Leechers
Switch
10/100
• Application setup
– uTorrent b1 flavor only
– Private torrents
– Each couple seeder/leecher
shares a different torrent
• Network setup
– Simple LAN environment
– No emulated conditions
• Experiment setup
– Varying ratios of TCP/LEDBAT flows
– Different TCP network stack
Seeds
Router
+
Netem
• native Linux stack
• Windows settings emulated
over Linux (to measure losses)
LAN: Multiple TCP and LEDBAT flows
TCP windows
TCP linux
0.98 0.98 0.98 0.98 0.96
1.00 0.75 0.56 0.33 1.00
0
4-0 3+1
3-1 2+2
2-2 1+3
1-3 0+4
0-4
4+0
TCP+LEDBAT
LEDBAT
0.5
TCP
LEDBAT
0.5
0.67 0.94 0.93 0.92 0.96
1.00 0.64 0.74 0.87 1.00
1
1
TCP
Bandwidth Breakdown
Efficiency
Fairness
4-0 3+1
3-1 2+2
2-2 1+3
1-3 0+4
0-4
4+0
TCP+LEDBAT
Observations
•
•
•
•
Throughput breakdown between X+Y competing TCP+LEDBAT competing flows
Efficiency is always high
Fairness among flows of the same type (intra-protocol) is preserved
TCP-friendliness depends on TCP parameters
– TCP Linux stronger than LEDBAT, LEDBAT stronger than TCP WinXP
ADSL experiments
• Application setup
Leecher
ADSL
– uTorrent b1 flavor only
– native Windows (XP) only
– private torrent
• Network setup
Internet
– real ADSL modems
– wild Internet
• uncontrolled delay, bandwidth
• Interfering traffic
• Experiment setup
ADSL
Seed
– LEDBAT alone in the wild
– LEDBAT with cross TCP traffic
• On forward path
• On backward path
ADSL experiments
TCP forward
0.6
5
DSL
capacity
4
3
0.4
Throughput
UTP β1
0.2
0
2
RTT [s]
Throughput [Mb/s]
0.8
TCP backward
RTT
1
100
200
300
Time [s]
400
500
600
0
• UTP alone: stable throughput, close to DSL capacity
• TCP on forward/backward paths
– Forward TCP: UTP correctly yields
– Backward TCP: nasty influence
• shacky RTT , competition with bursty ACK on foward
• RTT > 1sec affects the time at which decisions are effectively applied !
Source: [PAM]
Congestion control summary
• IETF LEDBAT
– Latecomer unfairness intrinsecally due to AIAD dynamics
[ICCCN’10,GLOBECOM’10] the bug almost on an IETF RFC !
– Unfairness due to heterogeneous TARGET [LCN’10]
– Ledbat lowest priority w.r.t. TCP-LP, NICE [LCN’10]
– Target delay–driven AIMD solves latecomer + keep queue
bounded[TR’10]
– Chunk-mode transmission: all flows in transient phase,
latecomer does not show up [TR’10]
– Implementation details can easily break the performance
[PAM’10]
– Meaning of low-priority depends on meaning of best effort
(i.e., TCP flavor) [PAM’10]
– Bufferbloat in the reverse direction can still harm performance
[PAM’10]
BitTorrent swarm
viewpoint
dario.rossi
``Because LEDBAT is used by BitTorrent,
whose users couldn’t care less for IETF’’
BitTorrent swarm viewpoint plan
• Simulation with arbitrary connection-level
semantic [P2P’11]
• Experiments with real application [TMA’12]
Simulation scenario
• Bottleneck at the access links (limited capacity C, large buffer B)
• Population: 1 seed, 100 leechers in flash-crowd scenario
• Seed policy: leave immediately, stay forever or for a random time
• Leechers use either TCP or uTP to TX chunk (but are able to RX both flavors)
• Swarm: homogeneous (100% TCP or 100% uTP) or heterogeneous (50%-50%)
Simulation results
• Homogeneous scenario:
same completion time for TCP and uTP
• Heterogeneous scenario:
swarm shows longer download time
• Heterogeneous scenario:
uTP peers have shorter completion time
w.r.t. TCP peers... unexpected !
Results explanation
Completion time
• Opposite uTP/TCP behavior for
growing buffer size
• Affected by congetion control
choice in the uplink direction
•
TCP fills the buffer, up to 2s
•
uTP keeps buffer occupancy low: faster
signalling due to shorter queuing delay
•
uTP peers opportunistically steal chunk
download slots to self-congested TCP peers
Experimental scenario
• Grid’5000 controlled testbed [TMA’12]
– uTorrent application
– Gbps LAN, emulated capacity & delay
– One peer per host (no shared queues)
• Experimental scenario
1
2
Seed
– Flash crowd, single provisioned seed
– 75 peers, mixed LEDBAT/TCP transport prefs
– Homogeneous delay and capacity
3
• Performance
– Torrent completion time (QoE)
– Linux kernel queue length (QoS)
– Closed source, so no chunk level log :(
N=75
Experimental results
– Homogeneous swarms: ( 75 UTP ) XOR ( 75 TCP ) peers,
separate experiments at different times
– Several repetitions (envelope report min & max)
– As expected, UTP limits queue length
E[TCP]=385ms
E[UTP]=
108ms
Source:
[TMA’12]
Experimental results
– Homogeneous swarms: ( 75 UTP ) XOR ( 75 TCP ) peers
– Shorter queue length leads to faster signaling:
awareness of content availability propagates faster!
– In turn, this assist in reducing the completion time
E[UTP]=
1325 sec
Source:
[TMA’12]
E[TCP]=
1421 sec
Experimental results
– Heterogeneous swarms: H TCP + (75-H) UTP peers
(+other scenarios, details in [TMA’12])
– Completion time increases linearly with TCP byte share!
(exception: all UTP scenario, details in [TMA’12])
– Impact of TCP flavors studied in [P2P’13]
s
BitTorrent swarm summary
• LEDBAT nice interplay with BitTorrent
– No latecomer unfairness due to chunk-based transmission [TR’10]
– Lower queue occupancy translates into faster signaling [P2P’11, TMA’12]
– Completion times of swarm possibly decreases, actual results depend on
application-layer connection management policies [P2P’11,TMA’12]
– LEDBAT nice interplay with other applications without hurting BitTorrent
performance either
– Similar interplay observed with NICE [P2P’13], additional benefits follow from
changes in the application (but uTorrent is closed source)
• Hidden details
– uTorrent implements an aggregated bandwidth shaper that ensures the set of
LEDBAT flows to take as much as the set of TCP flows (otherwise, LEDBAT
starves)
• Further complexity
– Dual TCP/LEDBAT stack: uTorrent attempts opening both connections in
parallel; if LEDBAT open succeeds, drop TCP connection
Bufferbloat
viewpoint
AQM ?
VoIP ? Games ?
dario.rossi
``Because LEDBAT is an important Internet
player, but only one of the players’’
Bufferbloat solution: LEDBAT ?
LEDBAT achieves its goals
LEDBAT not perfect
• Efficient and low queuing delay
[ICCCN’10,PAM’10]
• Lowest-priority w.r.t low-priority
protocols TCP-LP, NICE [LCN’10]
• Latecomer advantage for
backlogged flows can be
solved [GLOB’10,TR1]
• Fixed 100ms target does not
scale with link capacity (not
future proof)
• Heterogeneous target other
source of unfairness [LCN’10]
• Troubles with reverse bufferbloat
[PAM’10]
• ...but good interplay with
BitTorrent [P2P’11, TMA’12]
LEDBAT not enough
• A single TCP connection can
generate bufferbloat
• LEDBAT must be ubiquitously
replace TCP to solve bufferbloat
• ...but this ain’t going to happen
Bufferbloat viewpoint
• What is the actual extent
of Bufferbloat?
– Maximum bufferbloat easy
to measure (eg Netalyzer)
– Typical bufferbloat
unknown [P2P’12]
• Interesting questions
– What is the queuing delay
probability distribution (i.e.
user X sees a queuing delay
larger than Y) ?
– What % of RTT is due to
queuing ?
• What about LEDBAT
interoperability ?
– 1 TCP flow can still
create bufferbloat
– Hence, AQM possibly
deployed in parallel
• Interesting questions
– Combination of LEDBAT
and AQM ?
– More generally , of any
low-priority protocol and
AQM ? [CoNEXT’12]
Passive bufferbloat inference
• Methodology
– Passive traffic observation
– Gauge remote peer buffer
even with hidden B->C traffic
– Simple idea : queuing delay =
current OWD – minimum OWD
• Timestamps from payload
– Exploit LEDBAT headers
– TimeStamp Options
for TCP (RFC1323)
– Very accurate with kernel and
application-level ground truth
– Demo at [P2P’12]
Passive bufferbloat inference
• LEDBAT: modal
– Queuing 100ms
as expected (target)…
– Unless competing TCP
• TCP: bi-modal
– very low delay <=>
application-level
bandwidth limit
– very high delay <=>
bufferbloat
• Biased/coupled sampling
– Number of samples =
congestion window
– Congestion window =
~ inversely proportional to delay !
~2000 BitTorrent peers
Passive bufferbloat inference
~2000 BitTorrent peers
Removing sampling bias
• Bias due to sampling process
synchronous with LEDBAT dynamics
• Make the process asynchronous by
batching packets into fixed duration
windows
Performance analysis
• 1% of the 1-sec windows see a
queuing delay larger than 1 sec
• Breakdown per access type (AT),
BitTorrent client (BC) and operating
system (OS)
• Order of importance AT, BC, OS
AQM + LEDBAT
AQM alone
• AQM non ubiquitous
• “Dark” buffers in
OS, DSL, application
• AQM hard to tune
AQM+LEDBAT
• Reprioritization!
• LEDBAT as
aggressive as
NewReno TCP
AQM + Low-priority congestion control
AQM + Low-priority CC
• Reprioritization for any
AQM+CC combination!
• Simulation [CoNEXT’12]
student workshop
• Experimental results
confirm the trends
[TMA’13b]
• Fluid model to explain
dynamics [ITC’13]
Wrap up
dario.rossi
``Because nobody in the audience could
possibly stand any further picture !!’’
Conclusions
• IETF LEDBAT
– The protocol may perform poorly under some (not-so-weird) conditions
(i.e., backlogged transfers)
– Fixed TARGET delay not future-proof !
– Heterogeneous TARGET cause of further unfairness !
– Bufferbloat due to LEDBAT low unless TCP in the background.
– Hence, LEDBAT alone is not enough to solve the bufferbloat
– Dilemma: distributed E2E or local AQM solution ?
Both at the same time have poor interaction
• BitTorrent UTP
– When connections are not backlogged, conditions for latecomer unfairness
are not met, hence good performance
– Good interplay with both BT completion time (lower completion time)
– Good interplay with the interactive traffic of the same users (bounded
queuing delay, unlike TCP-LP, NICE)
– May not be needed if AQM properly configured on the DSL box ?
References
Optional readings:
[P2P’13] C. Testa, D. Rossi, A. Rao and A. Legout, Data Plane Throughput vs Control Plane Delay: Experimental Study of BitTorrent
Performance . In IEEE P2P'XIII, Trento, Italy, September 2013.
[P2P’11] C. Testa and D. Rossi, The impact of uTP on BitTorrent completion time. In IEEE P2P'11, 2011
For your thirst of knowledge:
[ITC-13] Gong, YiXi, Rossi, Dario and Leonardi, Emilio, Modeling the interdependency of low-prioritycongestion control and active queue
management . In The 25th International Teletraffic Congress (ITC25), sSptember 2013
[TMA-13a] C. Chirichella and D. Rossi, To the Moon and back: are Internet bufferbloat delays really that large Traffic Measurement and
Analysis (TMA'13), 2013.
[TMA-13b] Y. Gong, D. Rossi, C. Testa, S. Valenti and D. Taht, Fighting the bufferbloat: on the coexistence of AQM and low priority
congestion control . In Traffic Measurement and Analysis (TMA'13), 2013.
[CoNEXT’12] Y.Gong, D. Rossi, C. Testa, S. Valenti and D.That, Interaction or Interference: can AQM and low priority congestion cotnrol
succesfully collaborate ? In ACM CoNEXT, Student Session, Dec 2012
[P2P’12] Chiara Chirichella, Dario Rossi, Claudio Testa, Timur Friedman, Antonio Pescapé, Inferring the buffering delay of remote
BitTorrent peers under LEDBAT vs TCP . In IEEE P2P'XII, 2012.
[TMA’12] C. Testa, D. Rossi, A. Rao and A. Legout, Experimental Assessment of BitTorrent Completion Time in Heterogeneous TCP/uTP
swarms . In Traffic Measurement and Analysis (TMA), 2012.
[TR1] G. Carofiglio, L. Muscariello, D. Rossi, C. Testa, S. Valenti, Rethinking low extra delay backtround transport protocols . Elsevier
Computer Networks, 2013
[GLOBECOM’10] G. Carofiglio, L. Muscariello, D. Rossi and S. Valenti, The quest for LEDBAT fairness. In IEEE Globecom, 2010
[LCN’10] G. Carofiglio, L. Muscariello, D. Rossi and C. Testa, A hands-on Assessment of Transport Protocols with Lower than Best Effort
Priority . In 35th IEEE Conference on Local Computer Networks (LCN'10), Denver, CO, USA, October 10-14 2010.
[ICCCN’10] D. Rossi, C. Testa, S. Valenti and L. Muscariello, LEDBAT: the new BitTorrent congestion control protocol . In International
Conference on Computer Communication Networks (ICCCN'10), Zurich, Switzerland, August 2-5 2010.
[PAM’10] D. Rossi, C. Testa and S. Valenti, Yes, we LEDBAT: Playing with the new BitTorrent congestion control algorithm . In Passive and
Active Measurement (PAM'10), Zurich, Switzerland, April 2010.