Evaluating CoDel, FQ_CoDel and PIE:
how good are they really?
Naeem Khademi <naeemk@ifi.uio.no>
David Ros <David.Ros@telecom-bretagne.eu>
Michael Welzl <michawe@ifi.uio.no>
ICCRG – IETF 88
Vancouver, BC, Canada
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
1 / 23
Outline
Outline
New AQM Kids
Experimental Setup
A Basic Test
Parameter Sensitivity
AQM on 802.11 WLANs
FQ_CoDel: Blending SFQ and AQM
ECN
Conclusions and Future Work
Q&A
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
2 / 23
New AQM Kids
The New AQM Kids on the Block...
I
Two very recent proposals:
I
I
I
Some older AQMs dating back to early 90’s/00’s (*RED, REM,
BLUE, CHOKe,...)
I
I
(FQ_)CoDel (IETF 84)
PIE (IETF 85)
Designed to be better than RED, just like CoDel and PIE
Little academic literature available on CoDel and PIE
Literature (bold = peer-reviewed)
Wired, sim
Wired, real-life
Wireless (any)
CoDel
[NJ12][GRT+ 13][WP12]
[Whi13]
[GRT+ 13] X
X
PIE
[Whi13]
FQ_CoDel
[Whi13]
X
X
X
-
NOTE: [WP12] and [Whi13] are on DOCSIS 3.0 while [GRT+ 13] has tests with LP CC.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
3 / 23
New AQM Kids
The New AQM Kids on the Block (cont.)
AQM Deployment Status
I
(W)RED is available on plenty of HW but mostly "turned off"
Mentioned Reasons for Lack of Deployment
I
Bad implementation (?)
I
Hard to tune RED params
I
Sally Floyd’s ARED (2001 draft, available in Linux) adaptively
tunes RED params aiming for a certain target queuing => with
fixed BW maps to a "target delay"
I
Target delay can be set in ARED, CoDel and PIE
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
4 / 23
Experimental Setup
Experimental Setup
I
I
I
I
Traffic: 60/180/300 sec (wired/wireless/RTT=500 ms) of iperf, repeated for 10 runs
AQM iface: GSO TSO off, BQL=1514, txqueuelen=1000
TCP: Linux default with reno
Topology: Dumbbell with 4 sender-receiver pairs
Model
CPU
RAM
Ethernet
Ethernet driver
802.11 b/g
802.11 driver
OS kernel
Dell OptiPlex GX620
Intel(R) Pentium(R) 4 CPU 3.00 GHz
1 GB PC2-4200 (533 MHz)
Broadcom NetXtreme BCM5751
RTL-8139 (AQM interface)
RTL8111/8168B (Dummynet router)
tg3
8139too (AQM interface)
r8168 (Dummynet router)
D-Link DWL-G520 AR5001X+
ath5k
Linux 3.8.2 (FC14)
Linux 3.10.4 (AQM router) (FC16)
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
100 Mbps
100 Mbps
AQM
100 Mbps
100 Mbps
Dummynet router
100 Mbps
100 Mbps
100 Mbps
100 Mbps
100 Mbps
10 Mbps
AQM
100 Mbps
Dummynet router
100 Mbps
100 Mbps
Bottleneck router
100 Mbps
November 5, 2013
5 / 23
Experimental Setup
Experimental Setup (cont.)
I
AQM parameters used unless otherwise noted.
CoDel
interval=100 ms
target=5 ms
PIE
ARED
parameters in [pie].
PIE Parameter Default value
tupdate
30 ms
Ttarget
20 ms
α
0.125
β
1.25
N. Khademi, D. Ros, M. Welzl ()
parameters in [FGS01].
ARED Parameter
Default value
interval
500 ms
α
min(0.01, pmax /4)
β
0.9
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
6 / 23
Experimental Setup
Experimental Setup (cont.)
I
RTT is measured on per-packet basis using Synthetic Packet
Pairs (SPP) tool [spp]
I
I
Gives a very precise distribution of perceived RTT on the path
Goodput is measured per 5-sec intervals
I
long-term throughput/goodput does not reflect AQM performance
over time (e.g. bursts of packet drops are not desired)
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
7 / 23
A Basic Test
A Basic Test
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
Goodput (Mbps)
RTT (ms)
Single TCP Flow (RTTbase =100 ms)
1
1
1
CoDel
5
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
10
9.5
9
8.5
8
7.5
7
6.5
6
5.5
5
4.5
4
3.5
1
1
1
CoDel
(a) Per-packet RTT
5
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
(b) Goodput
Per-packet RTT and goodput. Bottom and top of whisker-box plots show 10th and 90th
percentiles respectively.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
8 / 23
A Basic Test
A Basic Test (cont.)
CoDel vs. RED from K. Nichols, “Controlling Queue
Delay” [NJ12]
I
A similar trend can
be observed between
CoDel and RED in a
different test in
[NJ12]
FTP traffic mix w/ and w/o web-browsing and CBR
applications and RTTs from 10∼500 ms.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
9 / 23
Parameter Sensitivity
Parameter Sensitivity (cont.)
1
1
1
5
CoDel
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
(c) Light
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
RTT (ms)
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
RTT (ms)
RTT (ms)
Target Delay (Per-packet RTT)
1
1
1
CoDel
5
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
(d) Moderate
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
1
1
1
5
CoDel
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
(e) Heavy
Per-packet RTT. Light, moderate and heavy congestion scenarios (4 senders and
RTTbase =100 ms).
Light, moderate and heavy congestion correspond to 4, 16 and 64 concurrent TCP flows
respectively.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
10 / 23
Parameter Sensitivity
Parameter Sensitivity (cont.)
Goodput (Mbps)
Target Delay (Goodput)
10
9.5
9
8.5
1
8.5
1
8
1
5
5
CoDel
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
7.5
(g) Moderate
7
6.5
6
5.5
1
1
1
CoDel
5
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
(f) Light
Goodput (Mbps)
Goodput (Mbps)
9
10
9.5
10
9.5
9
8.5
1
1
1
ARED
CoDel
5
5
5 10 10 10 20 20 20 30 30 30
Target Delay (ms)
PIE
ARED
(h) Heavy
Goodput. Light, Moderate and Heavy congestion scenarios (4 senders and RTTbase =100 ms).
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
11 / 23
AQM on 802.11 WLANs
AQM on 802.11 WLANs
I
802.11 is a challenging environment for AQM deployment
I
Varying MCS (BW) by RA and MAC retries
I
I
Shared channel with various active STAs
I
I
We use TCP max achievable BW for testing (e.g. ∼27 Mbps in .11g)
CoDel and PIE use delay
I
I
I
Direction-based unfairness (uplink vs. downlink)
Hard to predict the BW as input to ARED
I
I
Some drivers still use SampleRate instead of Minstrel (e.g. in FBSD)
CoDel: queuing delay by timestamping
PIE: estimated queuing delay (queue_length / departure_rate)
Public Wi-Fi e.g. at airports, hotels, corporations with bottleneck
on wlanX interface
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
12 / 23
AQM on 802.11 WLANs
AQM on 802.11 WLANs (cont.)
22
280
270
260
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
21.5
21
20.5
Goodput (Mbps)
RTT (ms)
802.11 Downlink Traffic Scenario – Target Delay=5 ms
20
19.5
19
18.5
18
17.5
17
16.5
1
1
1
4
4
4
8
8
8
16
16
16
1
1
1
Flows per sender #
CoDel
PIE
ARED
CoDel
(i) Per-packet RTT
4
4
4
8
8
Flows per sender #
PIE
8
16
16
16
ARED
(j) Goodput
4 senders, RTTbase =100 ms.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
13 / 23
AQM on 802.11 WLANs
AQM on 802.11 WLANs (cont.)
I
I
AQM on AP’s wlanX interface
∼65%/∼70% ACK loss at AQM (ARED) router for 16/32 flows.
802.11 Mixed Traffic – Target Delay=5 ms (Uplink’s Stats)
2400
2200
2000
Goodput (Mbps)
RTT (ms)
1800
1600
1400
1200
1000
27
26.5
26
25.5
25
24.5
24
1
800
1
1
600
CoDel
400
1
1
1
4
4
4
8
8
8
16
16
16
4
4
4
8
8
Flows per sender #
PIE
8
16
16
16
ARED
(l) Goodput
Flows per sender #
CoDel
PIE
ARED
(k) Per-packet RTT
4 senders, RTTbase =100 ms.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
14 / 23
FQ_CoDel: Blending SFQ and AQM
FQ_CoDel: Blending SFQ and AQM
SFQ is highly likely to improve
the performance when
combined with any AQM
I
I
I
Flow isolation/protection
with non-responsive traffic
Close to 100% flow-level
fairness on the edge
Significantly Lower 10th
percentile and median RTTs
but with longer (upper)
distribution tail
N. Khademi, D. Ros, M. Welzl ()
Comparison between CoDel and FQ_CoDel
– target_delay =5 ms
210
200
190
180
170
RTT (ms)
I
160
150
140
130
120
110
100
1
1
4
CoDel
4
8
Flows per sender #
8
16
16
FQ-CoDel
Per-packet RTT. 4 senders and
RTTbase =100 ms.
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
15 / 23
FQ_CoDel: Blending SFQ and AQM
FQ_CoDel: Blending SFQ and AQM
FQ_CoDel: Lower median latency at the expense of higher jitter
than CoDel (with the increase of congestion level)
I
Per-packet RTT Samples of a Single Flow – target_delay =5 ms
230
230
fq-codel
codel
220
210
200
200
200
190
190
190
180
180
180
160
RTT (ms)
210
170
170
160
170
160
150
150
150
140
140
140
130
130
130
120
120
110
110
100
120
110
100
20
25
30
Time (Sec)
35
(m) Light Congestion
40
fq-codel
codel
220
210
RTT (ms)
RTT (ms)
230
fq-codel
codel
220
100
20
25
30
Time (Sec)
35
(n) Moderate Congestion
40
20
25
30
Time (Sec)
35
40
(o) Heavy Congestion
4 senders, RTTbase =100 ms.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
16 / 23
ECN
ECN
I
Proposed two decades ago
I
Still turned off in clients
I
Can give a lot of benefits (out of this talk’s scope)
RFC 3168
For a router, the CE codepoint of an ECN- Capable packet SHOULD
only be set if the router would otherwise have dropped the packet as
an indication of congestion to the end nodes.
I
RFC 3168 is the only guideline for general use
I
Marking affects the marking/dropping probability since it
changes the metrics AQMs use (e.g. queue length and delay)
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
17 / 23
ECN
ECN (cont.)
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
RTT (ms)
RTT (ms)
Per-packet RTT. 4 senders, target_delay =5 ms, RTTbase =100 ms
1
1
1
4
4
4
8
8
8
16
16
250
240
230
220
210
200
190
180
170
160
150
140
130
120
110
100
16
1
1
1
4
Flows per sender #
CoDel
PIE
4
4
8
8
Flows per sender #
CoDel
ARED
(p) ECN disabled
PIE
8
16
16
16
16
16
16
ARED
(q) ECN enabled
10
9.5
9
8.5
8
7.5
7
6.5
6
Goodput (Mbps)
Goodput (Mbps)
Goodput
1
1
1
4
4
4
8
8
8
16
16
16
10
9.5
9
8.5
8
7.5
7
6.5
6
1
1
1
Flows per sender #
CoDel
PIE
4
4
8
8
8
Flows per sender #
ARED
(r) ECN disabled
N. Khademi, D. Ros, M. Welzl ()
4
CoDel
PIE
ARED
(s) ECN enabled
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
18 / 23
ECN
ECN (cont.)
I
I
Constant CE-marking under heavy
congestion
with ECN, ARED’s goodput drops
significantly as congestion level
decreases
I
I
Aggressive reaction to the
increase in average queue length
CE-marking (w/ ECN) 3.5∼6.6
times more than dropping (w/o
ECN)
N. Khademi, D. Ros, M. Welzl ()
pmarking|ecn /pdrop|noecn
Flows
4
16
32
64
Evaluating CoDel, FQ_CoDel and PIE
CoDel
1.256
1.356
1.719
6.117
PIE
1.156
1.106
1.591
6.569
ARED
6.621
3.465
4.303
3.873
November 5, 2013
19 / 23
ECN
ECN (cont.)
Our Recommendation
I
AQMs should modify their dropping/marking decision process to
incorporate the impact of CE-marked packets on their
measurements.
Possible Solutions
I
Exclude CE-marked packets from queue size (ARED), or exclude
delay caused by CE-marked packets (PIE)
I
Set lower ECN thresholds for AQMs
(fairness against non-ECN flows? dynamic threshold?)
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
20 / 23
Conclusions and Future Work
Conclusive Remarks
I
ARED only performed worse than CoDel or PIE in few scenarios
1. With very small number of flows
2. Public Wi-Fi with mixed (up- and down-link) traffic
3. With the common ECN implementation that CE-marks only when it
would otherwise drop
I
ARED outperforms CoDel and PIE in all other studied scenarios,
most notably regarding delay
Future Work
I
More realistic traffic types (here, only bulk TCP traffic)
I
Implementing and testing SFQ_ARED
I
Delay-based ARED
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
21 / 23
Q&A
Bibliography
[FGS01]
Sally Floyd, Ramakrishna Gummadi, and Scott Shenker.
Adaptive RED: An Algorithm for Increasing the Robustness of RED’s Active Queue Management.
Technical report, 2001.
[GRT+ 13] Y. Gong, D. Rossi, C. Testa, S. Valenti, and D. Taht.
Fighting the Bufferbloat: On the Coexistence of AQM and Low Priority Congestion Control.
2013.
[NJ12]
Kathleen Nichols and Van Jacobson.
Controlling Queue Delay.
Queue, 10(5):20:20–20:34, May 2012.
[pie]
PIE Linux code (Cisco).
ftp://ftpeng.cisco.com/pie/linux_code/.
[spp]
Synthetic Packet Pairs.
http://caia.swin.edu.au/tools/spp/.
[Whi13]
Greg White.
A Simulation Study of CoDel, SFQ-CoDel and PIE in DOCSIS 3.0 Networks.
Technical Report, CableLabs, April 2013.
[WP12]
Greg White and Joey Padden.
Preliminary Study of CoDel AQM in a DOCSIS Network.
Technical Report, CableLabs, November 2012.
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
22 / 23
Q&A
Q&A
More on experimental results: N. Khademi, D. Ros, and M. Welzl,
“The New AQM Kids on the Block: Much Ado About Nothing?”,
Technical Report 434, Department of Informatics, University of Oslo,
23 October 2013, available at http://urn.nb.no/URN:NBN:no-38868
N. Khademi, D. Ros, M. Welzl ()
Evaluating CoDel, FQ_CoDel and PIE
November 5, 2013
23 / 23