L15 - Fun with Congestion Control

CSCI-1680

Transport Layer III

Congestion Control Strikes Back

Theophilus Benson

Based partly on lecture notes byRodrigo Foncesa, David Mazières, Phil Levis, John Jannotti

This Week

• Congestion Control Continued

– Quick Review

• TCP Friendliness

– Equation Based Rate Control

• TCP’s Inherent unfairness

• TCP on Lossy Links

• Congestion Control versus Avoidance

– Getting help from the network

• Cheating TCP

Glossary of Terms

• RTT = Round Trip Time

• MSS = Maximum Segment Size

– Largest amount of TCP data in a packet

• Sequence Numbers

• RTO = Timeout

• Dup-Ack = a duplicate Acknowledgement

•

Transmission Rounds

Dup-Ack

Dup-Ack

Dup-Ack

Duplicate ACK example

each segment contains 1460 bytes

Dropped Pkt

Receiver sends ACKs for the last inorder accepted packet.

Seg2 re-transmitted after 3-dupacks

ACK_5 after re-transmission acknowledges all packets

4








•


MSS + Sequence Numbers

Your Code wants to send 146KB over the network

• Network MTU = 1500B

• Protocol header = IP(20B)+TCP(20B) = 40B

• What about link layer header? (ethernet or

ATM?)

MSS =1500B-40B = 1460B

# of TCP data segments = 146KB/1460B = 100

Each TCP Data segment has a sequence number:

• Location of first byte in the data segment.








•


Timeout (RTO) example

Wait for ACK … if no ACK then packet is lost

How long to wait?

**some function of RTT

1. duplicate

2. duplicate

3. duplicate

1K SeqNo=0

AckNo=1024

1K SeqNo=1024

1K SeqNo=2048

AckNo=1024

1K SeqNo=3072

AckNo=1024

1K SeqNo=4096

AckNo=1024

1K SeqNo=1024

1K SeqNo=5120

8








•



25

20

15

10

5

0

Transmission Round (in RTT)

• From perspective of the sending host

• At start of round= All data packets are sent

• At end of round = All ACKs received

• TCP tracks Window (CWND) in bytes but we may discuss it in MSS to make things simpler

Life of a TCP Connection

TCP Connection Handshake

• Agree TCP connection options

• Conversation starter: ‘Hi’..’hello’

SYN/ACK

SYN

ACK





SYN/ACK

SYN

Data ACK

ACK

*Flow Control

*Congestion Control ACK

Data

Data

Sequence number (in bytes) ..

Data .. Sequence number of first byte in the packet

ACK = sequence number of last byte in data packet +1

ACK

Data





• Allocate resources

SYN/ACK

SYN

ACK

Data ACK

Data

ACK

*Flow Control

*Congestion Control Data

ACK

Data

FIN

TCP Connection Teardown

• Conversation end: ‘good bye’

• De-allocate resources

FIN/ACK

ACK

Quick Review

• Flow Control:

– Receiver sets Advertised Window

– Prevents sender from overfilling receiver buffer

• Congestion Control

– Two states: Slow Start (SS) and Congestion

Avoidance (CA)

– A window size threshold governs the state transition

• Window <= ssthresh: SS

• Window > ssthresh: Congestion Avoidance

States differ in how they respond to acks

• Slow start: double w in one RTT

– There are w/MSS segments (and acks) per RTT

– Increase w per RTT  how much to increase per ack?

• w / (w/MSS) = MSS

• AIMD: Add 1 MSS per RTT

– MSS/(w/MSS) = MSS 2 /w per received ACK

Putting it all together

cwnd

Slow

Start

Timeout

Slow

Start

AIMD

Timeout ssthresh

Slow

Start

AIMD

Time

Fast Recovery and Fast Retransmit

cwnd

Slow Start

Fast retransmit

AI/MD

Time

This Week


– Quick Review








TCP Friendliness

• Can other protocols co-exist with TCP?

– E.g., if you want to write a video streaming app using UDP, how to do congestion control?

6

5

4

3

2

1

0

1

10

9

8

7

3 5 7

RED

9 11 13 15 17 19 21 23 25 27 29 31

Flow Number

1 UDP Flow at 10MBps

31 TCP Flows

Sharing a 10MBps link



– E.g., if you want to write a video streaming app using

UDP, how to do congestion control?

• Equation-based Congestion Control

– Instead of implementing TCP’s CC, estimate the rate at which TCP would send. Function of what?

– RTT, MSS, Loss

• Measure RTT, Loss, send at that rate!



– E.g., if you want to write a video streaming app using

UDP, how to do congestion control?



– RTT, MSS, Loss

Fair Througput= function( ?, ?, ?)

• Measure RTT, Loss, send at that rate!



– E.g., if you want to write a video streaming app using UDP, how to do congestion control?



Fair Throughput= function( RTT, MSS, Loss_rate)

• Measure RTT, Loss, MMSS

– Calculate and send at fair share rate!

TCP Throughput

• W is congestion window

• Assume a TCP congestion of window W (segments), round-trip time of RTT, segment size MSS

– Sending Rate S = W / RTT (1)

• Drop: W = W/2

– grows by MSS for W/ 2 RTTs , until another drop at W ≈ W

• Steady-state AMID: Average window then 0.75xS

– From (1), S = 0.75 W / RTT (2)

• Loss rate is 1 in number of packets between losses:

– Loss = 1 / ( 1 + ( W /2 + W /2+1 + W /2 + 2 + … + W )

= 1 / (3/8 W 2 ) (3)

TCP SawTooth Behavior

60

50

40

30

20

10

0

1 35 69 103 137 171 205 239 273 307 341 375 409 443 477

TCP Throughput

• W is congestion window

• Assume a TCP congestion of window W (segments), round-trip time of RTT, segment size MSS

– Sending Rate S = W x MSS / RTT (1)

• Drop: W = W/2

– grows by MSS for W/ 2 RTTs , until another drop at W ≈ W

• Steady-state AMID: Average window then 0.75xS

– From (1), S = 0.75 W MSS / RTT (2)

• Loss rate is 1 in number of packets between losses:

– Loss = 1 / ( ( W /2 + W /2+1 + W /2 + 2 + … + W )

= 1 / (3/8 W 2 ) (3)

Loss happens here

25

20

15

10

5

0

Transmission Round (in RTT) w/2+4 w w/2

Loss rate is 1 in number of packets between losses:

Loss = 1 / ( (W/2 + W/2+1 + W/2 + 2 + … + W)

Loss = 1 / (3/8 W 2 ) (3)

Loss happens here

25

20

15

10

5

0

Transmission Round (in RTT) w/2+4 w w/2

Loss rate is 1 in number of packets between losses:

Loss = 1 / ( (W/2 + W/2+1 + W/2 + 2 + … + W)

Loss = 1 / (3/8 W 2 ) (3)

Calculating Fair Rate

(Fair Throughput)

– Loss = 8/(3W 2

3

×

8

)

Loss

(4)

– Substituting (4) in (2), S = 0.75 * W / RTT ,

Throughput ≈

1.22

´

MSS

RTT

×

Loss

• Equation-based rate control can be TCP friendly and have better properties, e.g., small jitter, fast rampup…

This Week


– Quick Review








Assumptions Made by TCP

• Loss == congestion

– Loss only happens when:

– Aggregate sending Rate > capacity

• All flows in the network have the same:

– RTT

– MSS

– Same loss rates

• Everyone Plays Nicely


Fair Throughput= function( RTT, MSS, Loss_rate)

• Why would you have different RTTs or

MSS?

• What happens when you have different

RTT/MSS?

Throughput =

1.22

´

MSS

RTT

×

Loss

Implication of Buffering on TCP

• W in Bytes

• Recall: (1) S = 0.75 * W / RTT

• Where 0.75 is average of W and W/2

Network capacity

BW*RTT

• If No buffer then max W= BW*RTT (BW delay product)

BW*RTT

Avg

0.5*BW*RTT



• If Buffer = ½*BW*RTT

• Then max W= BW*RTT +½*BW*RTT

• Avg W~ ((1.5+.75)/2)*BW*RTT = 7.8*BW*RTT

• S = 7/8* W / RTT

• If No buffer then W*MSS = capacity

– S = 0.75* Capacity/RTT

Network capacity

BW*RTT

• What is ideal Sending rate?

– (2) S = Capacity/RTT

Avg

• What is ideal buffering?



• In ideal, min window = bandwidth*RTT

• What must buffer be to ensure this?




??*BW*RTT



Avg = ??



• In ideal, min = bandwidth*RTT

• What must buffer be to ensure this?

• Buffer = BW*RTT

• Max Window = 2*BW*RTT

• Min Window = BW*RTT

•

If No buffer then W*MSS = capacity





Network capacity

BW*RTT

Avg = 1.5BW*RTT


This Week


– Quick Review








Buffering and TCP

• If Buffer = ¼*BW*RTT

• What is TCP throughput

• Max Window =

• Min Window =

• TCP Throughput =










• Max Window =(1+1/4)*BW*RTT

• Min Window = (MaxWindow/2)=1/2(5/4)*BW*RTT

• TCP Throughput ~ ½(min+max) = 15/16*BW*RTT

– What’s wrong with this throughput?










• Max Window =(1+1/4)*BW*RTT

• Min Window = (MaxWindow/2)=1/2(5/4)*BW*RTT

• TCP Throughput ~ ½(min+max) = 15/16 *BW*RTT

– What’s wrong with this throughput?

– Less than Network capacity.








• If Buffer = 3*BW*RTT


• Max Window =

• Min Window =

• TCP Throughput =










• Max Window =(1+3)*BW*RTT

• Min Window = (MaxWindow/2)=1/2(4)*BW*RTT

• TCP Throughput ~ ½(min+max) ~ 2*BW*RTT

– What is wrong with this throughput?










• Max Window =(1+3)*BW*RTT

• Min Window = (MaxWindow/2)=1/2(4)*BW*RTT

• TCP Throughput ~ ½(min+max) ~ 2*BW*RTT

– What is wrong with this throughput?

– Greater than network capacity

•

If No buffer then W*MSS = capacity






How long is Fin-wait-1?

ACK

Data

ACK

Data

FIN

FIN/ACK

ACK

• How long until you get the Fin/ACK?

– If Max-Window is less than network capacity.

•

– If Max-window great than network capacity

•

How long is Fin-wait-1?

ACK

Data

ACK

Data

FIN

FIN/ACK

ACK

• How long until you get the Fin/ACK?

– If Max-Window is less than network capacity.

• Then 1 RTT (No buffering)

– If Max-window great than network capacity

• Then > 1 RTT. How many RTTs?

What Happens When Link is Lossy?

• Throughput ≈ 1 / sqrt(Loss) p = 0

60

50

40

30

20

10

0

1 26 51 76 101 126 151 176 201 226 251 276 301 326 351 376 401 426 451 476 p = 1% p = 10%

Assumptions Made by TCP

• TCP works extremely well when its assumptions are valid

• Loss == congestion

– Loss only happens when:

– Aggregate sending Rate > capacity

• All flows in the network have the same:

– RTT

– MSS

– Same loss rates

• Everyone Plays Nicely

– Everyone implements TCP correctly

What can we do about it?

• Two types of losses: congestion and corruption

• One option: mask corruption losses from

TCP

– Retransmissions at the link layer

– E.g. Snoop TCP: intercept duplicate acknowledgments, retransmit locally, filter them from the sender

• Another option:

– Network tells the sender about the cause for the drop

– Requires modification to the TCP endpoints

Congestion Avoidance

• TCP creates congestion to then back off

– Queues at bottleneck link are often full: increased delay

– Sawtooth pattern: jitter

• Alternative strategy

– Predict when congestion is about to happen

– Reduce rate early

• Two approaches

– Host centric: TCP Vegas

– Router-centric: RED, ECN, DECBit, DCTCP

Insight

Congestion  Longer queues  sending rate flattens out

Host based Approach

• Estimate Rate

• If flat then congestion

• Slow down

• Else no congestion

• Speed up

TCP Vegas

• Idea: source watches for sign that router’s queue is building up (e.g., sending rate flattens)

TCP Vegas

• Compare Actual Rate (A) with Expected Rate (E)

– If E-A > β, decrease cwnd linearly : A isn’t responding

– If E-A < α, increase cwnd linearly : Room for A to grow

Vegas

• Shorter router queues

• Lower jitter

• Problem:

– Doesn’t compete well with Reno. Why?

– Reacts earlier, Reno is more aggressive, ends up with higher bandwidth…

Insight

Congestion  Longer queues  sending rate flattens out

Network Approach

• Monitor Queue length

• If Q > threshold

• Drop more packets

• Host slows down

• If Q < threshold

• Drop no packets

• Hosts speeds up

Host based Approach

• Estimate Rate

• If flat then congestion

• Slow down

• Else no congestion

• Speed up

Help from the network

• What if routers could tell TCP that congestion is happening?

– Congestion causes queues to grow: rate mismatch

• TCP responds to drops

• Idea: Random Early Drop (RED)

– Rather than wait for queue to become full, drop packet with some probability that increases with queue length

– TCP will react by reducing cwnd

– Could also mark instead of dropping: ECN

RED Details

• Compute average queue length (EWMA)

– Estimated Weighted Moving Average

• Avg = Avg(x) + new_value (1-X)

– Don’t want to react to very quick fluctuations

RED Drop Probability

• Define two thresholds: MinThresh, MaxThresh

• Drop probability:

• Improvements to spread drops (see book)

RED Advantages

• Probability of dropping a packet of a particular flow is roughly proportional to the share of the bandwidth that flow is currently getting

• Higher network utilization with low delays

• Average queue length small, but can absorb bursts

• ECN

– Similar to RED, but router sets bit in the packet

– Must be supported by both ends

– Avoids retransmissions optionally dropped packets

More help from the network

• Problem: still vulnerable to malicious flows!

– RED will drop packets from large flows preferentially, but they don’t have to respond appropriately

• Idea: Multiple Queues (one per flow)

– Serve queues in Round-Robin

– Nagle (1987)

– Good: protects against misbehaving flows

– Disadvantage?

– Flows with larger packets get higher bandwidth

Example

A Network with Two flows

• Flow 1: 6 packets. In blue.

• Flow 2: 2 packets. In yellow

Red Drop Probability = 0.5

• 50% of 6 > 50% of 2

Queue 1 2 1 3 2 4 5 6 time

More help from the network

• Problem: still vulnerable to malicious flows!

– RED will drop packets from large flows preferentially, but they don’t have to respond appropriately

• Idea: Multiple Queues (one per flow)

– Serve queues in Round-Robin

– Nagle (1987)

– Good: protects against misbehaving flows

– Disadvantage?

– Flows with larger packets get higher bandwidth

Example

Queues for Flow 1

(arrival traffic)

Queue for Flow 2

(arrival traffic)

1

1

2 3

2 3 4

4 5

5 6 time time

Round-Robin combination

1 2 1 3 2 3 4 4 5 5 6 time

Big Picture

• Network Queuing doesn’t eliminate congestion: just manages it

• You need both, ideally:

– End-host congestion control to adapt

– Router congestion control to provide isolation and ensure sharing

This Week


– Quick Review








What happens if not everyone cooperates?

• Possible ways to cheat

– Increasing cwnd faster

– Large initial cwnd

– Opening many connections

– Ack Division Attack

C y

Increasing cwnd Faster

x increases by 2 per RTT y increases by 1 per RTT

Limit rates: x = 2y x

Figure from Walrand, Berkeley EECS 122, 2003

A

D

Larger Initial Window

x y

B

E x starts SS with cwnd = 4 y starts SS with cwnd = 1


Open Many Connections

• Web Browser: has to download k objects for a page

– Open many connections or download sequentially?

x

A B y

D E

• Assume:

– A opens 10 connections to B

– B opens 1 connection to E

• TCP is fair among connections

– A gets 10 times more bandwidth than B


Exploiting Implicit Assumptions

• Savage, et al., CCR 1999:

– “ TCP Congestion Control with a Misbehaving Receiver ”

• Exploits ambiguity in meaning of ACK

– ACKs can specify any byte range for error control

– Congestion control assumes ACKs cover entire sent segments

• What if you send multiple ACKs per segment?







• Evaluating Incentives and trade-offs of cheating

ACK Division Attack

• Receiver: “upon receiving a segment

2.1

with N bytes, divide the bytes in M

While a detailed description of TCP' s error and congestion conseparately” rudiments of their behavior below to allow those unfamiliar with

TCP to understand the vulnerabilities explained later. For simplic-

• ity, we consider TCP without the Selective Acknowledgment op-

Sender will grow window M times

TCP is a connection-oriented, reliable, ordered, byte-stream

– Because Old algorithms are naïve

Sender Maximum Segment Size (SMSS) determined during connection establishment. Each segment is labeled with explicit se-

• Could cause growth to 4GB in 4 receives an in-sequence segment it sends a cumulative acknowl-

RTTs!

can be retired from the sender' s retransmission buffers. If an outof-sequence segment is received, then the receiver acknowledges the next contiguous sequence number that was expected . If outstanding data is not acknowledged for a period of time, the sender will timeout and retransmit the unacknowledged segments.

TCP uses several algorithms for congestion control, most notably slow start and congestion avoidance [Jac88, Ste94, APS99].

Each of these algorithms controls the sending rate by manipulating a congestion window ( cwnd ) that limits the number of outstanding unacknowledged bytes that are allowed at any time. When a connection starts, the slow start algorithm is used to quickly increase cwnd to reach the bottleneck capacity. When the sender infers that a segment has been lost it interprets this has an implicit signal of network overload and decreases cwnd quickly. After roughly approximating the bottleneck capacity, TCP switches to the congestion avoidance algorithm which increases the value of cwnd more slowly to probe for additional bandwidth that may become available.

We now describe three attacks on this congestion control procedure that exploit a sender' s vulnerability to non-conforming receiver behavior.

2.2

ACK division

TCP uses a byte granularity error control protocol and consequently each TCP segment is described by sequence number and acknowledgment fields that refer to byte offsets within a TCP data stream.

However, TCP' s congestion control algorithm is implicitly defined in terms of segments rather than bytes. For example, the most recent specification of TCP' s congestion control behavior, RFC 2581, states:

During slow start, TCP increments cwnd by at most

SMSS bytes for each ACK received that acknowledges new data.

...

During congestion avoidance, cwnd is incremented by 1 full-sized segment per round-trip time (RTT).

The incongruence between the byte granularity of error control and the segment granularity (or more precisely, SMSS granularity) of congestion control leads to the following vulnerability:

Attack 1 :

Upon receiving a data segment containing N bytes, the receiver divides the resulting acknowledgment into M, where M N, separate acknowledgments – each covering one of M distinct pieces of the received data segment.

RTT

Sender

Data 1:14

61

ACK 4

87

ACK 9

73

ACK 1

461

Data 1461

:2921

Data 2921

:4381

Data 4381

:5841

Data 5841

:7301

Receiver

Figure 1: Sample time line for a ACK division attack. The sender begins with cwnd =1, which is incremented for each of the three valid ACKs received. After one round-trip time, cwnd =4, instead of the expected value of cwnd =2.

This attack is demonstrated in Figure 1 with a time line. Here, each message exchanged between sender and receiver is shown as a labeled arrow, with time proceeding down the page. The labels indicate the type of message, data or acknowledgment, and the sequence space consumed. In this example we can see that each acknowledgment is valid, in that it covers data that was sent and previously unacknowledged. This leads the TCP sender to grow the congestion window at a rate that is M times faster than usual. The receiver can control this rate of growth by dividing the segment at arbitrary points – up to one acknowledgment per byte received

(when M = N). At this limit, a sender with a 1460 byte SMSS could theoretically be coerced into reaching a congestion window in excess of the normal TCP sequence space (4GB) in only four roundtrip times!

1 Moreover, while high rates of additional acknowledgment traffic may increase congestion on the path to the sender, the penalty to the receiver is negligible since the cumulative nature of acknowledgments inherently tolerates any losses that may occur.

2.3

DupACK sp oofing

TCP uses two algorithms, fast retransmit and fast recovery , to mitigate the effects of packet loss. The fast retransmit algorithm detects loss by observing three duplicate acknowledgments and it immediately retransmits what appears to be the missing segment. However, the receipt of a duplicate ACK also suggests that segments are leaving the network. The fast recovery algorithm employs this information as follows (again quoted from RFC 2581):

Set cwnd to ssthresh plus 3*SMSS. This artificially “inflates” the congestion window by the number of segments (three) that have left the network and which the receiver has buffered.

..

For each additional duplicate ACK received, increment cwnd by SMSS. This artificially inflates the congestion window in order to reflect the additional segment that has left the network.

1

Of course the practical transmission rate is ultimately limited by other factors such as sender buffering, receiver buffering and network bandwidth.

TCP Daytona! (A Malicious version of

TCP)

Defense

• OLD TCP

– In slow start, cwnd++

• Appropriate Byte Counting

– [RFC3465 (2003), RFC 5681 (2009)]

– In slow start, cwnd += min (N, MSS) where N is the number of newly acknowledged bytes in the received ACK







• Evaluating Incentives and trade-offs of cheating

Cheating TCP and Game Theory

A

D x y

B

E

A

D  Increases by 1 Increases by 5

22 , 22 10 , 35

Increases by 1

35 , 10 15 , 15

Increases by 5

Individual incentives : cheating pays

Social incentives : better off without cheating

( x , y)

Too aggressive

 Losses

 Throughput falls

Classic Prisoner Dilemma: resolution depends on accountability

74

An alternative for reliability

• Erasure coding

– Assume you can detect errors

– Code is designed to tolerate entire missing packets

• Collisions, noise, drops because of bit errors

– Forward error correction

• Examples: Reed-Solomon codes, LT Codes, Raptor

Codes

• Property:

– From K source frames, produce B > K encoded frames

– Receiver can reconstruct source with any

K’ frames, with K’ slightly larger than K

– Some codes can make B as large as needed, on the fly

Advances to TCP

• Two main trends:

– Network access speeds have become better

– Many web-objects are much smaller now

• Advances to TCP

– TCP-Cubic

– Larger Initial Congestion Window

– TCP-FEC

• Problem:

TCP Cubic( 2008)

– TCP is not fair to flows with different RTTs

– AMID is very very slow to grow window size from

50k MSS to 100K MSS will take 50000 RTTS

• Replace the AI in AIMD with a CUBIC function

NEW RENO  AIMD CUBIC

Larger Initial Window ( 2010)

• Problem: web response are too slow

– Initial window was 4MSS  takes too long

– Proposed value:10MSS (15KB)

Image from [An Argument for Increasing TCP’s Initial Congestion Window. ACM SIGCOMM Computer Communication Review 27

Volume 40, Number 3, July 2010]

TCP+FEC( 2014)

• Problem: high probability of dropping 1 of the last data packets

– When this happens you may have a time-out (RTO)

– No duplicate ACKS because no more data to generate Dup-Acks

• Important because:

– Most objects fit in the initial congestion window

• So less than 15KB

– So can be sent in one RTT

– If Time-out occurs, you wait an 2 RTTs (1 to learn, 1 to retransmit)

• So now response time triples!!!

• Solution:

– Use FEC, send N+1 packet (assume only one packet is dropped)

– Use last packet for correction

Assignment #3 out

• Groups of 3 or 4

• Extra-credit ensure fairness+high utilization

– Fairness = Jain’s Fairness metric

– Fairness = (Sum of flow-utilization) 2

N * sum of (flow-utilization 2 )

Next Time

• Move into the application layer

• DNS, Web, Security, and more…

• Next week Guest lectures

– DNS + CDN

• Bruce Maggs

– Co-Founder of Akamai (largest public CDN)