How to get good TCP
performance over
asymmetric networks
Hari Balakrishnan
MIT Lab for Computer Science
http://wind.lcs.mit.edu/
hari@lcs.mit.edu
With: Venkata Padmanabhan (Microsoft
Research) and Randy Katz (UC Berkeley)
Motivation
• Many emerging (in some cases,
“emerged!”) networks are asymmetric
– Cable modems, “wireless” cable, (A)DSL
– Direct Broadcast Satellites
– Packet radio networks
• How does TCP behave over
asymmetric networks?
– TCP relies on ACK feedback to work,
which might be an issue
When is a network asymmetric?
Sender
Router
Forward
Router
Reverse
Receiver
The network characteristics in one direction
significantly affect performance in the other
Bandwidth: 10-1000X more in forward direction
Latency: Due to MAC protocol interactions
Loss-rate: Higher loss-rate in one direction
Why is this a problem for TCP?
6
7
5
4
3
2
1
4
2
3
Cumulative Acknowledgments (ACK)
• TCP relies on ACKs for reliability and for
congestion control
• New, cumulative ACKs “slide” and “bump up”
window, allowing new transmissions (ACK clocking)
• Smooth ACK flow  smooth transmits;
flaky ACK flow  bursty transmits
TCP Overview
1. Loss recovery
7
8
10
9
6
5
4
3
1
0
1
1
1
1
2
0
lost
Timeout = srtt + 4 * rtt_deviation (coarse-grained)
Fast retransmissions when duplicate ACKs arrive
2. Congestion control
– Window-based algorithm to
determine sustainable rate
– Upon congestion, reduce window
– “ACK clocking” sends data smoothly
TCP Dynamics
4.19E+06
Sequence number (bytes)
Data
4.18E+06
4.17E+06
Fast retransmission
4.16E+06
RTT
Window
4.15E+06
Duplicate ACKs
ACKs
4.14E+06
4.13E+06
33.52
33.54
33.56
33.58
33.6
33.62
Time (s)
33.64
33.66
33.68
33.7
33.72
Typical Performance
Technology
Lucent
WaveLAN
Metricom
Ricochet
Hybrid
wireless cable
Rated
Typical TCP
Bandwidth Throughput
2 Mbps
50 Kbps-1.5 Mbps
100 Kbps
10-35 Kbps
10 Mbps
0.5-3.0 Mbps
Goal: To bridge the gap between perceived
and rated performance
Bandwidth Asymmetry
• Shows up in asymmetric access
technologies
– ADSL, (wireless) cable modems, DBS
• Low-bandwidth ACK channel
• ACKs don’t come back fast enough or
get lost
Problem: Imperfect ACK feedback
degrades TCP performance
Problems
Data 11
Sender
Router
0
Data 10
ACKs
Forward
Data 9
Data 8
Bottleneck
Router
Receiver
1. Acks arrive slowly (large
1
4
7
buffer)
2. Acks are dropped (small
3
6
2
5
buffer)
3. Acks are queued behind data
Data
Data 1
packets
Performance depends on
1. Normalized asymmetry ratio
Lakshman & Madhow 97
Kalampoukas et al. 97
2. Bottleneck buffer sizes
Balakrishnan et al. 97
3. Competing reverse traffic
1 2 3 4 5 6 7
Hybrid Wireless Cable
TCP Throughput (Mbps)
6
5
10 Mbps Ethernet
4
3
28.8 C-SLIP
2
9.6 C-SLIP
1
0
28.8 SLIP
9.6 SLIP
0
20
40
60
80
100
120
140
160
Receiver socket buffer size (KB)
Return channel speed and latency
affects performance
180
200
Latency Asymmetry: Packet
Radio Networks
RTS
Fixed Host
Ethernet Radios
FH
ER
PT
PT
CTS
Mobile Host
PT
Internet
GW
ER
PT
Modem PR
PT
ER
MH
PT Poletop Radios PT
Half-duplex radios
Synchronization before communication
MAC Protocol Interactions
Pkt
Fixed Host
Ethernet Radios
FH
ER
PT
PT
Ack
No response
PT
Internet
GW
ER
Mobile Host
RTS
PT
Modem PR
PT
ER
MH
PT Poletop Radios
Exponential
PT backoff
Problem: Large, variable communication latency
Problem: Large Round-Trip Time
Variations
Metricom Ricochet Wireless Network
Sequence Number trace
RTT Estimate
Fast retransmissions
250000
Timeouts
200000
6000
RTT Estimate (msec)
150000
100000
50000
5000
4000
3000
2000
1000
0
20
40
60
Time (sec)
80
100
Mean rtt = 2.45s, std deviation = 1.5s  long timeout!
Long idle periods after multiple losses (~ 20 Kbps)
In contrast, UDP throughput = 50-64 Kbps
ACK flow affects data latency
19
17
15
13
11
9
7
Sample number
0
•
•
•
•
5
1
0
3
Sequence Number (bytes)
300000
Solutions
• Problems arise because of imperfect
ACK feedback
• 1. Reduce frequency of acks
– ACK Filtering (AF)
– ACK Congestion Control (ACC)
• 2. Handle infrequent acks
– Sender Adaptation (SA)
– ACK Reconstruction (AR)
General solutions for different asymmetric situations
ACK Filtering (AF)
Router
Forward
Router
Server
1
3 5 7 9
11
13
• Purge all redundant, cumulative ACKs
from constrained reverse queue
• No persistent filter state
• Used in conjunction with sender
adaptation or ACK reconstruction
Client
ACK Congestion Control (ACC)
Data 20
Data 21
Router
Data 22
8
Server
Forward
Data 19
16
10
14
18
Client
Delack factor = 2
Adaptive extension of TCP delayed ACKs
ACK Congestion Control (ACC)
Data
Data
Data
Router
Data
Forward
Data
Receiver
12
Sender
Delack factor = 2
RED marking of ECN bit
(Explicit Congestion
Notification)
22
ACK Congestion Control (ACC)
Data
Echo ECN marking
to receiver
Data 40
Data
Router
Forward
Data
Receiver
22
Sender
Delack factor = 2
ACK Congestion Control (ACC)
Data 42
Data 43
Router
Data 41
Forward
Data 40
36
Sender
40
Receiver
Delack factor = 4
Sender Adaptation (SA)
• Infrequent ACKs slow window growth
• Sender tends to be bursty
Forward
Receiver
Server
1
9
15
1. cwnd += 8
cwnd += 8/cwnd
Incr window by
amount acked
2.
. .
Regulation: pace packets. out at
rate estimated by cwnd/srtt
19
20
21
22
This reduces burstiness
ACK Reconstruction (AR)
Forward
Server
1
1
3
Client
9
5
7
ACK reconstructor
3 5 7 9
11
13
ACK filter
• Regenerate ACKs at other end of reverse link
• Shield sender from gaps in ACK sequence (no TCP
changes needed)
• AR rate determined by input ACK rate and
target ACK spacing
• Reconstructor state is soft
Performance: Bandwidth
Asymmetry
Throughput
(Mbps)
– TCP transfers in the forward direction alone
– Maximum window size 100 KB; no forward losses
10
8
6
4
2
0
Reno
ACC
AF
10 C/10 50 C/50
pkt
pkt
pkt
AF+AR
pkt
– Header compression helps
– Large reverse buffer hurts for Reno and ACC
– Fairness greatly improves using AF and ACC
Multihop Wireless Simulations
10 Mbps
1 ms
ER
Sender
–
–
–
–
100 Kbps,
10 ms
PT
PT
PT
Receiver
1 to 3 wireless hops on path
Radio turnaround time = 3-12 ms
Radio queue size = 10 packets
Exponential backoff in multiples of 20 ms
slots
Single TCP Transfer
• AF reduces chances that peer radio is busy
– MAC backoffs less frequent
• RTT std deviation reduces from 1.5s to 0.6s
Throughput (Kbps)
60
50
40
Reno
30
Reno+ACC
20
Reno+AF
10
0
1 hop
2 hops
3 hops
AF: 20-35% throughput improvement compared to Reno
Concurrent Transfers
• Metrics: utilization and fairness
• Simultaneous connections over 2-hop network
– Predictable and consistent performance with AF
• Long timeouts cause unpredictable performance
Jain's fairness index
1
0.8
0.6
AF
0.4
Reno
0.2
0
AF: 25% improvement in fairness over Reno
2
4
6
8
10
12
Number of connections
Combining Technologies
Wireless
transmitter
Web
data
Internet
server
Internet
Hybrid PoP
ER
10 Mbps, 2 ms
Requests &
ACKs
Client
PT
Cable forward channel with packet radio reverse channel
Workload: Multiple concurrent Web-like transfers
Result: Ack filtering greatly improves scaling behavior
(average completion time vs. # of concurrent transactions)
Status
• Extensive simulation results under
various bandwidth and latency
asymmetric conditions
• Implementation and evaluation under
BSD/OS 3.0 (code available)
• Documentation
– Paper to appear ACM MONET (extended,
revised version from MOBICOM 97
– Internet-draft submitted (PILC WG)
Cross-layer optimizations
• A way of performing protocol-specific
optimizations at link-layer
• Lightweight, soft-state agents in
network infrastructure
• State is soft: not essential for
correctness; improves performance
• Easy to implement!
• Examples include ACK filters and
reconstructors
Summary
• Asymmetry affects TCP performance
– ACK channel affects forward throughput
• Asymmetry comes in various flavors
– Bandwidth asymmetry (technology)
– Latency asymmetry (MAC interactions)
• Fortunately, good performance can be
obtained using a two-pronged approach
– Reduce frequency of ACKs (AF, ACC)
– Handle infrequent ACKs (SA, AR)
• Cross-layer optimizations are
A Good Thing!