Multipath TCP
Jennifer Rexford
Fall 2014 (TTh 3:00-4:20 in CS 105)
COS 561: Advanced Computer Networks
http://www.cs.princeton.edu/courses/archive/fall14/cos561/
Multipath
• Mobile user
– WiFi and cellular at the same time
• High-end servers
– Multiple Ethernet cards
• Data centers
– Rich topologies with many paths
• Benefits of multipath
– Higher throughput
– Failover from one path to another
– Seamless mobility
2
Bringing Multipath to the End Host
• Multiple addresses
– One or more addresses at an end host
– E.g., one per interface card
• Multiple paths
– Sequence of links between sender and receiver
– E.g., four-tuple of source and dest address and port
• Multiple subflows
– Flow of TCP segments over an individual path
– All associated with a single TCP connection
3
Review of TCP Protocol
4
Establishing a TCP Connection
A
B
Each host tells its Initial
Sequence Number (ISN)
to the other host.
• Three-way handshake to establish connection
– Host A sends a SYN (open) to the host B
– Host B returns a SYN acknowledgment (SYN ACK)
– Host A sends an ACK to acknowledge the SYN ACK
5
Initial Sequence Number (ISN)
• Sequence number for the very first byte
– E.g., Why not a de facto ISN of 0?
• Practical issue: reuse of port numbers
– Port numbers must (eventually) get used again
– … and an old packet may still be in flight
– … and associated with the new connection
• Security issue: adversary injecting packets
– Adversary may try to inject packets in a connection
– … by guessing the Initial Sequence Number
– … to send counterfeit packets to the receiving host
– … e.g., counterfeit packets that reset the connection
– Some firewalls change the ISN to further randomize
6
Step 1: A’s Initial SYN Packet
A’s port
B’s port
A’s Initial Sequence Number
Flags: SYN
FIN
RST
PSH
URG
ACK
Acknowledgment
20
Flags
0
Checksum
Advertised window
Urgent pointer
Options (variable)
A tells B it wants to open a connection…
7
Step 2: B’s SYN-ACK Packet
B’s port
A’s port
B’s Initial Sequence Number
Flags: SYN
FIN
RST
PSH
URG
ACK
A’s ISN plus 1
20
Flags
0
Checksum
Advertised window
Urgent pointer
Options (variable)
B tells A it accepts, and is ready to hear the next byte…
… upon receiving this packet, A can start sending data
8
Step 3: A’s ACK of the SYN-ACK
A’s port
B’s port
Sequence number
Flags: SYN
FIN
RST
PSH
URG
ACK
B’s ISN plus 1
20
Flags
0
Checksum
Advertised window
Urgent pointer
Options (variable)
A tells B it is okay to start sending
… upon receiving this packet, B can start sending data
9
Sequence Number
Host A
ISN (initial sequence number)
Sequence
number = 1st
byte
Host B
TCP Data
TCP Data
10
TCP Header
Source port
Destination port
Sequence number
Flags: SYN
FIN
RST
PSH
URG
ACK
Acknowledgment
HdrLen 0
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
Data
11
Receive Buffering: Flow Control
• Receive window size
– Amount that can be sent without acknowledgment
– Receiver must be able to store this amount of data
• Receiver tells the sender the window
– Tells the sender the amount of free space left
Window Size
Data ACK’d
Outstanding
Un-ack’d data
Data OK
to send
Data not OK
to send yet
12
TCP Header: Receive Window
Source port
Destination port
Sequence number
Flags: SYN
FIN
RST
PSH
URG
ACK
Acknowledgment
HdrLen 0
Flags
Advertised window
Checksum
Urgent pointer
Options (variable)
Data
13
Tearing Down the Connection
B
A
time
• Closing (each end of) the connection
– Finish (FIN) to close and receive remaining bytes
– And other host sends a FIN ACK to acknowledge
– Reset (RST) to close and not receive remaining bytes
14
Extending TCP: TCP Options
• TCP header
– Ten mandatory fields
– Optional extension field (usually during handshake)
• Examples
– Maximum segment size (MSS)
– Window scaling
– Support for Selected ACKs
But, some middleboxes:
• Unknown options
– Ignored by receiving host
(i) strip TCP options from
some packets or (ii) drop
packets with TCP options
• Routers and TCP options
– Should ignore them, passing them through unchanged
15
Incremental Deployment
Challenges
16
Keeping the Same Socket API
• Backwards compatibility with existing apps
– Present the same socket API and expectations
• Establish the TCP connection in the same way
– Create a socket to a single remote IP address/port
– … and then add more subflows to the connection
• Work in all scenarios where regular TCP works
– If a subflow fails, the connection should continue
– … as long as some other subflow has connectivity
17
MPTCP in the Network Stack
From http://queue.acm.org/detail.cfm?id=2591369
18
Negotiating MTTCP Capability
• How do end-points know they both speak
MPTCP?
– During the 3-way SYN/SYN-ACK/ACK handshake
• What if middleboxes strip the TCP option?
– On the SYN? On the SYN-ACK?
19
Negotiating MTTCP Capability
• Include capability on the ACK of the SYN-ACK?
– What if the ACK is lost?
– Carry on all subsequent packets
• What if the middlebox drops SYN packets with
unfamiliar options?
– Sender can retransmit lost SYN without the option
– … and fall back to regular TCP behavior
20
Adding Subflows, Idealized
• How to associate a new subflow with the
connection?
– Use the source/destination IPs and ports
• How to start using the new subflow?
– Simply start sending packets with new IP/port pairs
– … and associate them with the existing connection
• How could two end-points learn about extra IP
addresses for establishing new subflows?
– Implicitly: one end-point establishes a new subflow, to
already-known address(es) at the other end-point
21
Challenges: NAT
• Network Address Translators (NAT)
– Problem: NAT changes the IP address and port number
WiFi
LTE
NAT1
NAT2
• How to identify a connection?
– Using a token established during connection set-up
• How to establish new subflows?
– Allow one end-point to tell another about its addresses
22
Challenges: Security
• Security
– Malicious parties creating subflows
– To highjack (part of) the connection
• How to bootstrap security?
– Include a random key during connection set-up
– … and use it to verify authenticity of new subflows
• How to identify the connection on new subflows?
– A token generated from the key
• How to authenticate the addition of subflows?
– Exchanging nonces and computing message
authentication codes using the keys
23
Establishing New Flows, Reality
24
Sequence Numbers
• Challenges across subflows
– Out-of-order packets due to RTT differences
– Access networks that rewrite sequence numbers
– Middleboxes upset by discontinuous TCP byte stream
– Need to retransmit lost packets on a different subflow
• Two levels of sequence numbers
– Sequence numbers per subflow
– Sequence numbers for the entire connection
• Enables
– Efficient detection of loss on each subflow
– Retransmission of lost packet on a different subflow
25
Sending the Second Sequence #
• Mapping of subflow bytes to data sequence space
– To associate segments in a subflow
– … with their position in the connection
• Encoding the information
– Send the additional data in TCP options
– Carry the additional data in the TCP payload
• Problems with using TCP payload
– Flow control may keep receiver from sending a data
ACK
– Simply exclude the data-ACKs from flow control?
– … but middleboxes won’t know to do this… 
26
Sending the Second Sequence #
• So, encode in TCP options
• Encode as a mapping
– Subflow sequence # to connection sequence #?
• But, middleboxes may rewrite subflow sequence #
• Instead, map as an offset
– Offset from the subflow’s initial sequence number
– … robust to middleboxes that shift the sequence
numbers
• Middleboxes that modify contents (and length)?
– Include checksum in the mapping
– … and terminate the subflow if modifications occur
27
Receive Buffer Space
• Each TCP connection has a receive buffer
– Buffer space to store incoming data
– … until it is read by the application
• TCP flow control
– Receiver advertises the available buffer space
– … using the “receive window”
• Should each subflow have its own receive window?
– Starvation of some subflows in a connection?
– Fairness relative to other TCP connections?
– Fragmentation of the available buffer space?
• Instead, use a common receive window
28
Use of Multipath TCP in iOS 7
• Multipath TCP in iOS 7
– Primary TCP connection over WiFi
– Backup TCP connection over cellular data
• Failover
– If WiFi becomes unavailable…
– … iOS 7 will use the cellular data connection
• For destinations controlled by Apple
– E.g., Siri
29
Multipath Congestion Control
Slides from Damon Wischik
30
Goal #1: Fair at Shared Bottlenecks
A multipath
TCP flow with
two subflows
Regular TCP
To be fair, Multipath TCP should take as much capacity
as TCP at a bottleneck link, no matter how many paths
it is using.
31
Use Efficient Paths
12Mb/s
12Mb/s
12Mb/s
Each flow has a choice of a 1-hop and a 2-hop path.
How should split its traffic?
32
Use Efficient Paths
12Mb/s
8Mb/s
8Mb/s
12Mb/
s
12Mb/
s
8Mb/s
If each flow split its traffic 1:1 ...
33
Use Efficient Paths
12Mb/s
9Mb/s
9Mb/s
12Mb/s
9Mb/s
12Mb/s
If each flow split its traffic 2:1 ...
34
Use Efficient Paths
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
Better: Each connection on a one-hop path
Each connection should send all traffic on the leastcongested paths
35
Use Efficient Paths
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
12Mb/s
Better: Each connection on a one-hop path
Each connection should send all traffic on the leastcongested paths
But keep some traffic on the alternate paths as a probe
36
Discussion
37