Circuit switching in the
Internet
Ph.D. oral examination
Pablo Molinero-Fernández
21 st May 2002

Traffic doubles every year
Relative preformance increase
1000
750
500
250
1 0
Traffic growth x2/1 year
Router capacity x2.2/1.5 years
Moore’s law x2/1.5 years
2002 2004 2006
GAP OF 5x!!
2008 2010 2012
Cost and complexity of five times as many central offices is prohibitive
2

Optics removes bandwidth constraints
But we cannot buffer light!
3

Circuit switching
• Link bandwidth is reserved
• Need signaling
• No buffering
• No processing in data path 4

Packet switching
• Link bandwidth is shared
• Buffering
• Per-packet processing
5

Contributions
The
Internet needs circuit switchin g in the core
TCP
Switching: how to integrate circuit switching in the core
Provision ing of fat pipes in an alloptical backbon e
6

How we think the
Internet is
7

Why the Internet uses packet switching
• Efficient use of expensive links:
– “Circuit switching is rarely used for data networks, ... because of very inefficient use of the links” – Bertsekas & Gallager ‘92
• Resilience to failure of links & routers:
– ”For high reliability, ... [the Internet] was to be a datagram subnet, so if some lines and
[routers] were destroyed, messages could be ... rerouted” – Tanenbaum ‘96
8

What the Internet is really like today
SONET/SDH
DWDM
9

What the Internet is really like today
Modems,
DSL
10

Performance criteria
Users
• Response time
• Service Level Agreement (QoS)
Carriers
• Cost: Bandwidth efficiency
• Reliability and stability
• Low complexity
11

Users
What users want
• Response time
• Service Level Agreement (QoS)
Carriers
• Cost: Bandwidth efficiency
• Reliability and stability
• Low complexity
12

Response time of packets and circuits
File = 10Mbit
100 clients
1 server
1 Gb/s x 100
Flow bandwidth
Avg response time
Max response time
Circuit sw
1 Gb/s
0.51 s
1 s
Packet sw
10 Mb/s
1 s
1 s
All but one of circuits finish earlier
13

Response time when blocking occurs File = 10Gbit /
10Mbit
100 clients
1 server
1 Gb/s x 99
Flow bandwidth
Avg response time
Max response time
Circuit sw
1 Gb/s
10.50 s
10.99 s
Packet sw
10Mb/s;1Gb/s
1.10 sec
10.99 sec
A big flow can kill CS if it blocks the link
14

Response time with flow rate limits File = 10Gbit/
10Mbit
100 clients
1 server
1 Gb/s x 99
1 Mb/s
Flow bandwidth
Avg response time
Max response time
Circuit sw
1 Mb/s
109.9s
10,000 s
Packet sw
1 Mb/s
109.9sec
10,000 sec
No difference between
CS and PS in core 15

Analytical modeling
• Fluid model:
• Many independent arrivals => Poisson
• Service policies:
– Packets: Processor Sharing
– Circuits: FCFS
• Service time distribution:
– Flow size variance: Bimodal
– Realistic flow size distrib: Pareto
P ( A ) f ( x )
 p ;
 x
P ( B )
 

1

16 p

Fluid model: M/BiModal/N
FCFS (load = 0.5)
100
10
1
N=1
• Access link is as fast as the core link
• Hogging by long flows
0.1
0
Flow size variance
0.5
17

Fluid model: M/BiModal/N
FCFS (load = 0.5)
100
10
1
1
0.1
0
Flow size variance
0.5
32
N=2 k
• Flow rate limited by access link
(1/N)
• Same response time regardless of flow size variance, if N is large
18

Fluid model: M/Pareto/N
FCFS (alpha = 1.3)
10000
N=1
1000
100
10
Link hogging becomes very bad with heavy-tailed traffic, if ratio N=1
1
0 0.5
load
1
19

Fluid model: M/Pareto/N
FCFS (alpha = 1.3)
10000
1000
100
10
N =2 k
1
2
4
8 16
256
• Response time similar to IP if core/access ratio N is large
• Typically
N >> 1,000
1
512 load
0 0.5
1
20

Users see little difference in response time
1000
100
• Simulation of full networks
10
1
1E+2
Circuit switching
Packet switching
1E+4 flow size (bytes)
1E+6
• N is large => same response time
21

Service Level Agreements
• Packet switching:
– Algorithms (WFQ, DRR, …) => not used
– Thus we must overprovisioning => used and it works
• Circuit switching:
– Simple QoS: guaranteed BW => no jitter
22

What carriers want
Users
• Response time
• Service Level Agreement (QoS)
Carriers
• Cost: Bandwidth efficiency
• Reliability and stability
• Low complexity
23

Cost: Bandwidth efficiency
• Argument: packets share all link BW => statistical multiplexing gain => more throughput with bursty traffic
• Reality:
– Internet avg. link utilization: 5-20%
[Coffman&Odlyzko’02]
– Phone avg. link utilization: ~33% [Odlyzko’99]
– There is a glut of BW in the core [WSJ’00]
• Result:
– Packets more efficient, but BW is no longer a scarce resource
24

Carriers peak allocate their network
OC-3 OC-12 OC-48
• When overprovisioning, link BW is virtually peak allocated
• That is exactly what circuit switching does
Source: Chuck Fraleigh ‘02
25

Reliability and stability (I)
• Argument: because of the state, rerouting a circuit is more costly than with packets
• Reality:
– Internet average availability:
1220 min/year down time [Labovitz’99]
– Phone average availability:
5 min/year down time [Kuhn’97]
26

Reliability and stability (II)
• Reality (cont.):
– IP recovers in about 3 min (median), sometimes it takes over 15 min
[Labovitz’01]
– SONET required to recover in less than
50 ms
• Result:
– No evidence packet switching is more robust
27

Low complexity (I)
• Argument: No per-flow state => packet switching is simpler
• Reality:
– PS: 8M lines of code in core router
[Cisco’s IOS ‘00]
– CS: 18M lines of code in telephone switch
[AT&T/Lucent 5ESS ‘96]
– CS: 3M lines of code in transport switch [’01]
• Result:
– Packet switching does not seem inherently less complex than circuit switching
28

Functions in a packet switch
Ingress linecard Interconnect Egress linecard
Framing Route lookup
TTL process ing
Buffer ing
Buffer ing
QoS schedul ing
Framing
Interconnect scheduling
Control plane
Control path
Data path
Scheduling path
29

Functions in a circuit switch
Ingress linecard Interconnect
Egress linecard
Framing Framing
Interconnect scheduling
Control plane
Control path
Data path
30

Low complexity (II)
• Argument: IP does not have the signaling of circuits switches => Routers go faster
• Reality:
– IP does almost same operations on every packet as a circuit switch on the circuit establishment
– CS has no work to do once circuit is established
• Result:
– The fastest commercially-available circuit switches [Ciena ’01, Lucent ‘01] have 5x the capacity of the fastest routers [Cisco ’01,
Juniper ’02]
31

Network architecture
LANs & wireless
MANs
WANs
• Use packet switching
• Better response time (ratio N small)
• Efficient use of the spectrum
• If metro-to-access BW ratio (N) is small => use packets
• Otherwise use what costs less
• Use circuit switching
• More capacity, reliability
• Similar response time & QoS
32

Contributions
The
Internet needs circuit switchin g in the core
TCP
Switching: how to integrate circuit switching in the core
Provision ing of fat pipes in an alloptical backbon e
33

Integration of circuits and packets
TCP
Switching
• Create a separate circuit for each user flow
• IP controls circuits
• Optimize for the most common case
– TCP (90-95% of traffic)
– Data (9 out of 10 pkts)
34

TCP Switching exposes circuits to IP
IP routers
TCP Switches
35

Source
SYN
TCP “creates” a connection
Router Router Router
Destination
SYN+ACK
DATA
Packets Packets Packets Packets
36

Source
Let TCP leave state behind
Boundary
TCP-SW
Core
TCP-SW
Boundary
TCP-SW
Destination
SYN
Create circuit
Create circuit
SYN+ACK
DATA
Packets One Circuit Packets
37

What is a typical flow?
• Most traffic are TCP connections:
– Taking less than 10 s, 12 packets and 4
KBytes
– Obtaining less than 100 Kbps
– ~40% of the flows continue transmitting
ACKs after sending a FIN (asymmetrical closures)
38

State management feasibility
• Amount of state
– Minimum circuit = 56 Kb/s.
– 178,000 circuits for OC-192.
• Update rate
– About 51,000 entries per sec for
OC-192
• Implementable in hardware or software.
39

TCP Switching can be implemented in software
TCP Switching boundary router:
• Kernel module in Linux 2.4 1GHz PC
• Forwarding latency
– Forward one packet: 21 m s.
– Compare to: 17 m s for IP.
– Compare to: 95 m s for IP + QoS.
• Time to create new circuit: 57 m s.
Source: Byung-Gon Chun ‘01
40

Bandwidth inefficiencies rcuit BW
Slow
Start
Congestion
Avoidance
Flow duration
Inactivity timeout time
Compromise: inactivity timeout of few seconds
41

Related work
• IP Switching
– Uses ATM virtual circuits (i.e. packets)
– Became MPLS (but no longer user flows)
• Generalized Multi-Protocol Label
Switching (GMPLS)
– Coarse circuits
– Heavy weight signaling
42

Contributions
The
Internet needs circuit switchin g in the core
TCP
Switching: how to integrate circuit switching in the core
Provision ing of fat pipes in an alloptical backbon e
43

New networking scenario
IP routers
Optical Switches
44

New networking scenario
IP Linecards
Optical Crossconnect
45

Circuit creation is slow
• We need a safeguard to avoid running out of BW => inefficiency
A slow signaling requires a larger BW safeguard
46

Controlling coarse circuits with user flows
1
0.1
1 s
100 ms
0.01
0.001
Circuit creation 1 ms latency 100us
10 us
10 ms
0.0001
1E+04 1E+06
Bandwidth safeguard (bps)
1E+08
• Should use the fastest optical switching elements
• Should avoid
ACKs => no RTT
47

Conclusions
• Circuits should be used in the core, packets in the edges
• TCP Switching integrates circuits and packets in an evolutionary way
• User flows can be used to control an all-optical network
48

Papers
• PMF, Nick McKeown, "TCP Switching: Exposing
Circuits to IP“, IEEE Micro, 2002
• PMF, NM, "TCP Switching: Exposing Circuits to
IP“, Hot Interconnects, 2001
• PMF, NM, “Study of routing behavior trough traffic analysis and traceroute measurements”,
NAT Times, 2001
• PMF, NM, Hui Zhang, "Is IP going to take over the world (of communications)?“, submitted
49

Thank you