Data Center Networking
CS 6250: Computer Networking
Fall 2011
1
Cloud Computing
• Elastic resources
– Expand and contract resources
– Pay-per-use
– Infrastructure on demand
• Multi-tenancy
– Multiple independent users
– Security and resource isolation
– Amortize the cost of the (shared) infrastructure
• Flexibility service management
2
– Resiliency: isolate failure of servers and storage
– Workload movement: move work to other locations
Trend of Data Center
By J. Nicholas Hoover , InformationWeek June 17, 2008 04:00 AM
http://gigaom.com/cloud/google-builds-megadata-center-in-finland/
200 million Euro
http://www.datacenterknowledge.com
Data centers will be larger and larger in the future cloud
computing era to benefit from commodities of scale.
Tens of thousand  Hundreds of thousands in # of servers
Most important things in data center management
- Economies of scale
- High utilization of equipment
- Maximize revenue
- Amortize administration cost
- Low power consumption (http://www.energystar.gov/ia/partners/prod_development/downloads/EPA_Datacenter_Report_Congress_Final1.pdf)
Cloud Service Models
• Software as a Service
– Provider licenses applications to users as a service
– e.g., customer relationship management, email, …
– Avoid costs of installation, maintenance, patches, …
• Platform as a Service
– Provider offers software platform for building applications
– e.g., Google’s App-Engine
– Avoid worrying about scalability of platform
• Infrastructure as a Service
– Provider offers raw computing, storage, and network
– e.g., Amazon’s Elastic Computing Cloud (EC2)
– Avoid buying servers and estimating resource needs
4
Multi-Tier Applications
• Applications consist of tasks
– Many separate components
– Running on different machines
Front end
Server
• Commodity computers
– Many general-purpose computers
– Not one big mainframe
– Easier scaling
Aggregator
Aggregator
Aggregator
……
Aggregator
…
5
Worker
Worker
…
Worker
Worker
Worker
Enabling Technology:
Virtualization
• Multiple virtual machines on one physical machine
• Applications run unmodified as on real machine
•6 VM can migrate from one computer to another
Status Quo: Virtual Switch in Server
7
Top-of-Rack Architecture
• Rack of servers
– Commodity servers
– And top-of-rack switch
• Modular design
– Preconfigured racks
– Power, network, and
storage cabling
• Aggregate to the next level
8
Modularity
• Containers
• Many containers
9
Data Center Challenges
•
•
•
•
•
•
Traffic load balance
Support for VM migration
Achieving bisection bandwidth
Power savings / Cooling
Network management (provisioning)
Security (dealing with multiple tenants)
10
Data Center Costs (Monthly Costs)
• Servers: 45%
– CPU, memory, disk
• Infrastructure: 25%
– UPS, cooling, power distribution
• Power draw: 15%
– Electrical utility costs
• Network: 15%
– Switches, links, transit
http://perspectives.mvdirona.com/2008/11/28/CostOfPowerInLargeScaleDataCenters.aspx
11
Common Data Center Topology
Internet
Core
Aggregation
Access
Data Center
Layer-3 router
Layer-2/3 switch
Layer-2 switch
Servers
12
Data Center Network Topology
Internet
CR
AR
AR
S
S
S
S
S
…
~ 1,000 servers/pod
13
CR
...
S
…
AR
AR
...
Key
• CR = Core Router
• AR = Access Router
• S = Ethernet Switch
• A = Rack of app. servers
Requirements for future data center
• To catch up with the trend of mega data center, DCN technology
should meet the requirements as below
–
–
–
–
–
–
High Scalability
Transparent VM migration (high agility)
Easy deployment requiring less human administration
Efficient communication
Loop free forwarding
Fault Tolerance
• Current DCN technology can’t meet the requirements.
– Layer 3 protocol can not support the transparent VM migration.
– Current Layer 2 protocol is not scalable due to the size of forwarding table
and native broadcasting for address resolution.
Problems with Common Topologies
• Single point of failure
• Over subscription of links higher up in the
topology
• Tradeoff between cost and provisioning
15
Capacity Mismatch
CR
~ 200:1
AR
S
CR
AR
AR
AR
S
S
S
S
S
~ 40:1
S
S
~ 5:1
…
16
S
S
…
...
S
…
S
…
Data-Center Routing
Internet
CR
DC-Layer 3
AR
AR
SS
SS
SS
SS
CR
...
AR
AR
DC-Layer 2
SS
…
SS
…
~ 1,000 servers/pod == IP subnet
17
...
Key
• CR = Core Router (L3)
• AR = Access Router (L3)
• S = Ethernet Switch (L2)
• A = Rack of app. servers
Reminder: Layer 2 vs. Layer 3
• Ethernet switching (layer 2)
– Cheaper switch equipment
– Fixed addresses and auto-configuration
– Seamless mobility, migration, and failover
• IP routing (layer 3)
– Scalability through hierarchical addressing
– Efficiency through shortest-path routing
– Multipath routing through equal-cost multipath
• So, like in enterprises…
18
– Data centers often connect layer-2 islands by IP
routers
Need for Layer 2
• Certain monitoring apps require server with
same role to be on the same VLAN
• Using same IP on dual homed servers
• Allows organic growth of server farms
• Migration is easier
19
Review of Layer 2 & Layer 3
• Layer 2
– One spanning tree for entire network
• Prevents loops
• Ignores alternate paths
• Layer 3
– Shortest path routing between source and destination
– Best-effort delivery
20
FAT Tree-Based Solution
• Connect end-host together using a fat-tree topology
– Infrastructure consist of cheap devices
• Each port supports same speed as endhost
– All devices can transmit at line speed if packets are distributed
along existing paths
– A k-port fat tree can support k3/4 hosts
21
Fat-Tree Topology
22
Problems with a Vanilla Fat-tree
• Layer 3 will only use one of the existing equal
cost paths
• Packet re-ordering occurs if layer 3 blindly takes
advantage of path diversity
23
Modified Fat Tree
• Enforce special addressing scheme in DC
– Allows host attached to same switch to route only
through switch
– Allows inter-pod traffic to stay within pod
– unused.PodNumber.switchnumber.Endhost
• Use two level look-ups to distribute traffic and
maintain packet ordering.
24
Two-Level Lookups
• First level is prefix lookup
– Used to route down the topology to endhost
• Second level is a suffix lookup
– Used to route up towards core
– Diffuses and spreads out traffic
– Maintains packet ordering by using the same ports for
the same endhost
25
Diffusion Optimizations
• Flow classification
– Eliminates local congestion
– Assign to traffic to ports on a per-flow basis instead of
a per-host basis
• Flow scheduling
– Eliminates global congestion
– Prevent long lived flows from sharing the same links
– Assign long lived flows to different links
26
Drawbacks
• No inherent support for VLAN traffic
• Data center is fixed size
• Ignored connectivity to the Internet
• Waste of address space
– Requires NAT at border
27
Data Center Traffic
Engineering
Challenges and Opportunities
28
Wide-Area Network
...
Servers
Router
DNS
Server
DNS-based
site
selection
29
Data
Centers
...
Servers
Router
Internet
Clients
Wide-Area Network: Ingress Proxies
...
Servers
Router
...
Data
Centers
Servers
Router
Proxy
Proxy
Clients
30
Traffic Engineering Challenges
• Scale
– Many switches, hosts, and virtual machines
• Churn
– Large number of component failures
– Virtual Machine (VM) migration
• Traffic characteristics
– High traffic volume and dense traffic matrix
– Volatile, unpredictable traffic patterns
• Performance requirements
31
– Delay-sensitive applications
– Resource isolation between tenants
Traffic Engineering Opportunities
• Efficient network
– Low propagation delay and high capacity
• Specialized topology
– Fat tree, Clos network, etc.
– Opportunities for hierarchical addressing
• Control over both network and hosts
– Joint optimization of routing and server placement
– Can move network functionality into the end host
• Flexible movement of workload
32
– Services replicated at multiple servers and data
centers
– Virtual Machine (VM) migration
PortLand: Main Idea
Add a new host
Key features
- Layer 2 protocol based on tree topology
- PMAC encode the position information
- Data forwarding proceeds based on PMAC
- Edge switch’s responsible for mapping between
PMAC and AMAC
- Fabric manger’s responsible for address resolution
- Edge switch makes PMAC invisible to end host
- Each switch node can identify its position by itself
- Fabric manager keep information of overall topology.
Corresponding to the fault, it notifies affected nodes.
Transfer a packet
Questions in discussion board
• Question about Fabric Manager
o How to make Fabric Manager robust ?
o How to build scalable Fabric Manager ?
 Redundant deployment or cluster Fabric manager could be solution
• Question about reality of base-line tree topology
o Is the tree topology common in real world ?
 Yes, multi-rooted tree topology has been a traditional topology in data center.
[A scalable, commodity data center network architecture
Mohammad La-Fares and et el, SIGCOMM ‘08]
Discussion points
o Is the PortLand applicable to other topology ?
 The Idea of central ARP management could be applicable.
 To solve forwarding loop problem, TRILL header-like method would be necessary.
 The benefits of PMAC ? It would require larger size of forwarding table.
• Question about benefits from VM migration
o VM migration helps to reduce traffic going through aggregate/core switches.
o How about user requirement change?
o How about power consumption ?
• ETC
o Feasibility to mimic PortLand on layer 3 protocol.
o How about using pseudo IP ?
o Delay to boot up whole data center
Status Quo: Conventional DC Network
Internet
CR
DC-Layer 3
AR
AR
S
S
CR
...
AR
AR
DC-Layer 2
Key
•
•
S
S
S
S
...
•
•
…
CR = Core Router (L3)
AR = Access Router (L3)
S = Ethernet Switch (L2)
A = Rack of app. servers
…
~ 1,000 servers/pod == IP subnet
Reference – “Data Center: Load balancing Data Center Services”, Cisco
2004
Conventional DC Network Problems
CR
S
CR
AR
AR
AR
AR
S
~ 200:1
S
S
S
S
S
~ S
~
40:1
5
…
:
1
S
S
…
...
S
…
S
…
Dependence on high-cost proprietary routers
Extremely limited server-to-server capacity
And More Problems …
CR
AR
S
S
S
…
CR
AR
S
~ 200:1
S
S
…
IP subnet (VLAN) #1
S
…
AR
AR
S
S
S
S
S
…
IP subnet (VLAN) #2
• Resource fragmentation, significantly lowering
utilization (and cost-efficiency)
And More Problems …
CR
AR
S
S
S
…
CR
AR
Complicated manual
L2/L3 re-configuration
S
~ 200:1
S
S
…
IP subnet (VLAN) #1
S
…
AR
AR
S
S
S
S
S
…
IP subnet (VLAN) #2
• Resource fragmentation, significantly lowering
cloud utilization (and cost-efficiency)
All We Need is Just a Huge L2
Switch,
or an Abstraction of One
...
CR
S
…
CR
AR
AR
AR
AR
S
S
S
S
S
S
S
S
S
…
S
S
...
…
…
VL2 Approach
• Layer 2 based using future commodity switches
• Hierarchy has 2:
– access switches (top of rack)
– load balancing switches
• Eliminate spanning tree
– Flat routing
– Allows network to take advantage of path diversity
• Prevent MAC address learning
– 4D architecture to distribute data plane information
– TOR: Only need to learn address for the intermediate switches
– Core: learn for TOR switches
• Support efficient grouping of hosts (VLAN replacement)
41
VL2
42
VL2 Components
• Top-of-Rack switch:
– Aggregate traffic from 20 end host in a rack
– Performs ip to mac translation
• Intermediate Switch
– Disperses traffic
– Balances traffic among switches
– Used for valiant load balancing
• Decision Element
– Places routes in switches
– Maintain a directory services of IP to MAC
• End-host
– Performs IP-to-MAC lookup
43
Routing in VL2
• End-host checks flow cache for MAC of flow
– If not found ask agent to resolve
– Agent returns list of MACs for server and MACs for intermediate
routers
• Send traffic to Top of Router
– Traffic is triple-encapsulated
• Traffic is sent to intermediate destination
• Traffic is sent to Top of rack switch of destination
44
The Illusion of a Huge L2 Switch
1. L2 semantics
2. Uniform high
capacity
…
…
3. Performance
isolation
…
…
Objectives and Solutions
Objective
Approach
Solution
1. Layer-2 semantics
Employ flat addressing
Name-location
separation & resolution
service
2. Uniform
high capacity between
servers
Guarantee bandwidth
for
hose-model traffic
Flow-based random
traffic indirection
(Valiant LB)
Enforce hose model
using existing
mechanisms only
TCP
3. Performance
Isolation
Name/Location Separation
Cope with host churns with very little overhead
Directory
Service
Switches run link-state routing and
maintain only switch-level topology
• Allows to use low-cost switches
…
x  ToR2
• Protects network and hosts from host-state churn
y  ToR3
• Obviates host and switch reconfiguration
z  ToR34
ToR1
…
. . . ToR2
. . . ToR3
. . . ToR4
ToR3 y
payload
ToR34 z
payload
47
x
yy, z
Servers use flat names
z
Lookup &
Response
Clos Network Topology
Offer huge aggr capacity & multi paths at modest cost
Int
D
(# of 10G ports)
Aggr
48
96
144
...
...
48
TOR
20
Servers
...
Max DC size
(# of Servers)
11,520
...
46,080
K aggr switches with D ports
103,680
......
........
20*(DK/4) Servers
Valiant Load Balancing: Indirection
Cope with arbitrary TMs with very little overhead
IANY
IANY
IANY
Links used
for up paths
Links used
for down paths
[ ECMP + IP Anycast ]
•
•
•
•
Harness huge bisection bandwidth
Obviate esoteric traffic engineering or optimization
Ensure robustness to failures
Work with switch mechanisms available today
T1
IANY T35 zy
T2
payload
x
49
T3
T4
T5
T6
1. Must
spread
Equal
Cost
Multitraffic
Path Forwarding
y2. Must ensure dst
z independence
Properties of Desired Solutions
• Backwards compatible with existing
infrastructure
– No changes in application
– Support of layer 2 (Ethernet)
• Cost-effective
– Low power consumption & heat emission
– Cheap infrastructure
•
•
•
•
Allows host communication at line speed
Zero-configuration
No loops
Fault-tolerant
50
Research Questions
• What topology to use in data centers?
– Reducing wiring complexity
– Achieving high bisection bandwidth
– Exploiting capabilities of optics and wireless
• Routing architecture?
– Flat layer-2 network vs. hybrid switch/router
– Flat vs. hierarchical addressing
• How to perform traffic engineering?
– Over-engineering vs. adapting to load
– Server selection, VM placement, or optimizing routing
• Virtualization of NICs, servers, switches, …
51
Research Questions
• Rethinking TCP congestion control?
– Low propagation delay and high bandwidth
– “Incast” problem leading to bursty packet loss
• Division of labor for TE, access control, …
– VM, hypervisor, ToR, and core switches/routers
• Reducing energy consumption
– Better load balancing vs. selective shutting down
• Wide-area traffic engineering
– Selecting the least-loaded or closest data center
• Security
52
– Preventing information leakage and attacks