SEATTLE
- A Scalable Ethernet Architecture
for Large Enterprises
M.Sc. Pekka Hippeläinen
IBM
phippela@gmail
1.10.2012
T-110.6120 – Special Course in Future Internet Technologies
1
SEATTLE
 Based on and pictures borrowed
from:Changhoon,K;Caesar,M;Rexford,J.
Floodless in SEATTLE: A Scalable Ethernet
Architecture for Large Enterprises
 Is it possible to build a protocol that maintains the
same configuration-free properties as Ethernet
bridging, yet scales to large networks?
1.10.2012
2
Contents
 Motivation: network management challenge
 Ethernet features: ARP and DHCP broadcasts
 1) Ethernet Bridging
 2) Scaling with Hybrid networks
 3) Scaling with VLANs
 Distributed Hashing
 SEATTLE approach
 Results
 Conclusions
1.10.2012
3
Network management challenge
 IP Networks require massive effort to configure




and manage
Even 70% of an enterprise network’s cost goes
to maintenance and configuration
Ethernet is much simpler to manage
However Ethernet does not scale well beyond
small LANs
SEATTLE architecture aims to provide scalability
of IP with simplicity of Ethernet management
1.10.2012
4
Why Ethernet is so wonderful ?
 Easy to setup, easy to manage
 DHCP server, some hubs, plug’n play
1.10.2012
5
Flooding query 1: DHCP
requests
 Lets say node A joins the ethernet
 To get IP / confirm IP – node A sends a DHCP
request as a broadcast
 Request floods through the broadcast domain
18.09.2012
6
Flooding query 2: ARP
 In order for node A to communicate to node B in




the same broadcast domain, the sender needs
MAC address of the node B
Lets assume that node B IP is know
Node A sends and Address Request Protocol
(ARP) broadcast – to find out MAC address of
node B
Similarly to DHCP broadcast – the request is
flooded through the whole broadcast domain
This is basically {IP -> MAC} mapping
1.10.2012
7
Why flooding is bad ?
 Large Ethernet deployments contain vast




number of hosts and thousands of bridges
Ethernet was not designed to such a scale
Virtualization and mobile deployments can
cause many dynamic events – causing control
traffic
Broadcast messages need to be processed in the
end hosts – interrupting cpu
The bridges forwarding tables grow roughly
linearly with number of hosts
1.10.2012
8
1) Ethernet bridging
 Ethernet consists of segments each comprising




a single physical layer
Ethernet bridges are used to interconnect
segments to multi-hop network i.e. LAN
This forms a single broadcast domain
Bridge learns how to reach a host – by
inspecting the incoming frames and associating
the source MAC with the incoming port
A bridge stores this information to a forwarding
table – using the table to forward packets to
correct direction
1.10.2012
9
Bridge spanning tree
 One bridge is configured to be the root bridge
 Other bridges collectively compute a spanning
tree based on the distance to the root
 Thus traffic is not routed through shortest path
but along the spanning tree
 This approach avoids broadcast storms
1.10.2012
10
1.10.2012
11
2) Hybrid IP/Ethernet
 In this approach multiple LANs are




interconnected with IP routing
In hybrid networks each LAN contains at most a
few hundred of hosts that form IP subnet
IP subnet is associated with the IP prefix
Assigning IP prefixes to subnet and associating
subnets with router interfaces is a manual
process
Unlike MAC which is host identifier – IP address
denotes the hosts current location in the
network
1.10.2012
12
1.10.2012
13
Drawbacks of Hybrid approach
 Biggest drawback is the configuration overhead
 Router interfaces must be configured
 Host must have correct IP address corresponding to
the subnet it is located (DHCP can be used)
 Networking policies are defined usually per
network prefix i.e. topology
 When network changes the policies must be updated
 Limited mobility support
 Mobile users & virtualized hosts at datacenters
 If IP is constant – the user should stay on the same
subnet
1.10.2012
14
3) Virtual LANs
 Overcomes some problems of Ethernet and IP




Networks
Administrators can logically groups hosts into
same broadcast domain
VLANS can be configured to overlap –
configuring bridges not the hosts
Now broadcast overhead can be reduced by the
isolates domains
Mobility is simplified – IP address can be
retained while moving between bridges
1.10.2012
15
Virtual LANs
 Traffic from B1 to B2 can be ‘trunked’ over
multiple bridges
 Inter domain traffic needs to be routed
1.10.2012
16
Drawbacks of VLANs
 Trunk configuration overhead
 Extending VLAN across multiple bridges requires
VLAN to be configured at each of the bridges
participating. Often manual work.
 Limited control plane scalability
 Forwarding table entries and broadcast traffic for
every active host and every VLAN visible
 Insufficient data plane efficiency
 Single spanning tree is still used within each VLAN
 Inter-VLAN traffic must be routed via IP gateways
1.10.2012
17
Distributed Hash Tables
 Hash tables are used to store {key -> value} pairs
 In case of multiple nodes there is nice way to
make
 Nodes symmetric
 Distribute the hash table entries evenly among nodes
 Keep reshuffling of entries small in case of
adding/removing nodes
 Idea is to calculate H(key) that is mapped to a
host – one can visualize this to mapping to an
angle (or to a point on a circle)
1.10.2012
18
Distributed Hash Tables
 Each node is mapped to randomly distributed
points on the circle
 Thus each node is mapped to multiple buckets
 One calculates the H(key) – and stores the entry
to the node owning this bucket
 If node is removed – the values are now assigned
to next buckets
 If node is added – entries are moved to the new
buckets
1.10.2012
19
SEATTLE approach 1/2
 1) Switches calculate shortest
path among themselves
 This is link state protocol – basically Dijkstra
 Switch level discovery protocol – Ethernet hosts do
not respond
 Switch topology much more stable than at host level
 Much more scalable than at host level
 Each switch has an ID – one MAC address of the
switch interfaces
1.10.2012
20
SEATTLE approach 2/2
 2) DHT used in switches
 {IP->MAC} mapping
 This is essentially ARP request avoiding flooding
 {MAC->location} mapping
 When switch is located – routing along the shortest path
can be used
 DCHP Service location can also be stored
 SEATTLE thus reduces flooding, allows usage of
shortest path and offers a nice way to locate
DHCP service
1.10.2012
21
SEATTLE
 Control overhead reduced with consistent
hashing
 When set of switches changes due to network failure
or recovery – only some entries must be moved
 Balancing load with virtual switches
 If some switches are more powerful – the switch can
represent itself as many – getting more load
 Enabling flexible service discovery
 This is mainly DHCP – but could be something like
{“PRINTER”->location}
1.10.2012
22
Topology changes
 Adding and removing switches/links can alter
topology
 Switch/link failures and recoveries can also lead
to partitioning events (more rare)
 Non-partitioning link failures are easy to handle
– the resolver for hash entry is not changed
1.10.2012
23
Switch failures
 If switch fails or recovers hash entries need to be
moved
 The switch that published value – monitors the
liveliness of resolver. Republishing entry when
needed
 The entries have TTL
1.10.2012
24
Partitioning events
 Each switch has to book keep also locally-stored
location entries
 If switch s_old is removed / not reachable – all the
switches need to remove these location entries
 This approach correctly handles partitioning
events
1.10.2012
25
Scaling:
location
 Hosts use directory service to publish and
maintain {mac->location} mappings
 When host a with mac_a arrives – it accesses
switch S_a (steps 1-3)
 Switch s_a publishes {mac_a,location}, by calculating
the correct bucket F(mac_a) i.e. switch/resolver
 When node b wants to send message to node a
 F(mac_a) is calculated to fetch the location
 ’Reactive resolution’ – also cache misses do not
lead flooding
1.10.2012
26
Scaling:
ARP
 When node b makes ARP request – SEATTLE
converts this to a {F(IP_a) -> mac_a} request
 The resolver/switch for F(IP_a) is usually
different from F(mac_a)
 Optimization for hosts making ARP request
 F(IP_a) address resolver can also store mac_a and S_a
 When node b makes F(IP_a) ARP request also mac_a-
>S_a mapping is cached to S_b
 Shortest path (-> path 10) can now be used
1.10.2012
27
Handling host dynamics
 Location change
 Wireless handoff
 VM moved but retaining MAC
 Host MAC address changes
 NIC card replaced
 Failover event
 VM migration forcing MAC change
 Host changes IP
 DHCP leave expires
 Manual reconfiguration
1.10.2012
28
Insert, delete and update
 Location change
 Host h moves from s_old to s_new
 s_new updates the existing mac-to-location entry
 MAC change
 IP-to-MAC update
 MAC-to-location deletion (old) and insertion (new)
 IP change
 S_h deletes old IP-to-MAC and inserts new IP-to-MAC
1.10.2012
29
Ethernet: Bootstrapping
hosts
 Host discovered by access switches
 SEATTLE switches snoop ARP requests
 Most OSes generate ARP request at boot up / if up
 Aldo DHCP messages or host down can be used
 Host configuration without broadcast
 DHCP_SERVER hashes string “DHCP_SERVER” and
stores the location to the switches
 The “DHCP_SERVER” string is used to locate service
 No need to broadcast for ARP or DHCP
1.10.2012
30
Scalable and flexible VLANs
 To support broadcasts – the authors suggest
using groups
 Similar to VLAN - groups is defined as a set of
hosts who share the same broadcast domain
 The groups are not limited to layer-2
reachability
 Multicast-based group-wide broadcasting
 Multicast tree with broadcast root for each group
 F(group_id) used for broadcast root location
1.10.2012
31
Simulations
 1) Campus ~40 000 students
 517 routers and switches
 2) AP-Large (Access Provider)
 315 routers
 3) Datacenter (DC)
 4 core routes with 21 aggregation switches
 Routers were converted to SEATTLE switches
1.10.2012
32
Cache timeout and AP-large
with 50k hosts
 Shortest path cache timeout
has impact on number of
location lookups
 Even with 60s time out 99.98%
packets were forwarded without lookup
 Control overhead (blue) decreases very fast – where as the
table size increases only moderately
 Shortest path is used in majority of routing in these
simulations
1.10.2012
33
Table size increase in DC
 Ethernet bridges stores entry
for each destination ~ O(sh)
behavior across network
 SEATTLE requires only ~O(h) state since only access
and resolver switches need to store and location
information for each hosts
 With this topology the table size was reduced by factor of 22
 In AP-large case the factor was increased to 64
1.10.2012
34
Control overhead in AP-large
 Number of control messages
over all links in the topology
divided by the number switches
and duration of the trace
 SEATTLY significantly reduces control overhead in
the simulations
 This is mainly because Ethernet generates network
wide floods for a significant number of packets
1.10.2012
35
Effect of switch failure in
DC
 Switches were allowed to fail
randomly
 The average recover time was
30 seconds
 SEATTLE can use all the links in the topology, where
as Ethernet is restricted to the spanning tree
 Ethernet must re-compute the tree causing outages
1.10.2012
36
Effect of host mobility in
Campus
 Hosts were randomly moved
between access switches
 For high mobility rates,
SEATLLES loss rate was
lower than Ethernet
 On Ethernet it takes sometime for switches to evict
the stale information location information and relearn the new location
 SEATTLE provided low loss and broadcast overhead
1.10.2012
37
What was omitted
 Authors suggest multi-level one-hop DHTs
 With large dynamic networks – it can be beneficial that
entries are stored close
 This is achieved with regions and backbone – border
switches connect to the backbone switches
 With topology changes
 Approach to seamless mobility is described in the paper
 Updating remote host caches is required with switch
based MAC revocation lists
 Some simulation results
 Authors also made sample implementation
1.10.2012
38
Conlusions
 Operators today face challenges in managing
and configuring large networks. This is largely to
complexity of administering IP networks.
 Ethernet is not a viable alternative
 poor scaling and inefficient path selection
 SEATTLE promises scalable self-configuring
routing
 Simulations suggest efficient routing, low latency
with quick recovery
 Host mobility supported with low control overhead
 Ethernet stacks at end hosts are not modified
1.10.2012
39
Thank you for your attention!
Questions? Comments?
1.10.2012
40