©2013 Avaya Inc. All rights reserved
February 26-28, 2013 | Orlando, FL
Deployment Options for
Avaya DToR
Randy Cross
Paul Unbehagen
Director, PLM
@cross_randy
Chief Architect
@punbehagen
#AvayaATF
©2013 Avaya Inc. All rights reserved
February 26-28, 2013 | Orlando, FL
Agenda
• Introduction
• History Discussion
• A new DNA for the Data Center
• Design around real traffic flows
©2013 Avaya Inc. All rights reserved
3
February 26-28, 2013 | Orlando, FL
Evolution of the Data Center
Once, Campus-class was good enough
Traditional networks are
designed for north/south
traffic – ToR Switches
interconnected by the
Core or Aggregation…
Rack 1
©2013 Avaya Inc. All rights reserved
• 20-30 microseconds
for every hop
• Modern applications
have an average of
8 transactions
Rack 2
Rack 3
4
Rack 4
February 26-28, 2013 | Orlando, FL
Evolution of the Data Center
Traffic patterns were traditional
• What this meant:
– Application traffic
traverses multiple
Switch hops – Access
/ Core / ToR / Core /
Access
– Uplinks were more
important than InterRack capacity
Traditionally:
The North-South
to East-West
ratio has been
80:20
Top-of-Rack
Switches
Racked Servers
©2013 Avaya Inc. All rights reserved
5
February 26-28, 2013 | Orlando, FL
What’s the approach of the “Big 2”..?
• Cisco’s Vblock “marketecture” was the first salvo
• HP responded by purchasing 3COM/Huawei assets
• Both offered “end-to-end” architectures
• Server/Storage/ToR/EoR/Core
• The explosion of Virtual Machines has changed
everything
• What’s next..?
• Collapsed 2-Tier architecture and “Fabric”
• Why..?
• Better L2 scale, overcoming STP & VLAN limits
• Scalable support for L3 & VRFs
©2013 Avaya Inc. All rights reserved
6
February 26-28, 2013 | Orlando, FL
Cisco’s “Classic” Hierarchical Model
Spot the real difference between these two designs..?
Enterprise Campus
Data Center
Courtesy: Cisco Systems
Enterprise Campus 3.0 Architecture
Courtesy: Cisco Systems
Data Center Architecture Overview
All Access traffic flows via Aggregation/Distribution and/or Core
©2013 Avaya Inc. All rights reserved
7
February 26-28, 2013 | Orlando, FL
HP’s 2-Tier Network Design
An Improvement..?
Switch Clustering has done this for the last 12 years..!
2-Layer may reduce cost, but all traffic still goes to Core
©2013 Avaya Inc. All rights reserved
8
February 26-28, 2013 | Orlando, FL
Avaya Fabric DToR
The new DNA for your Data Center
• Avaya Fabric Connect provides
the new DNA for your Data
Centre Network.
• Enabling unparalleled scalability
and low latency interconnect to
address East-West traffic
requirements
• Fabric Connect Stacking
supports 2-32 switches offering
up to 512 10GE ports with
20Tbs Fabric.
• Fabric Connect Mesh supports
4-480 switches offering up to
15,360 10GE ports with
262.5Tbs Fabric.
Sidney Duffy
Avaya Fabric Connect – the new DNA for your Data Centre Network
©2013 Avaya Inc. All rights reserved
9
February 26-28, 2013 | Orlando, FL
Introducing the Virtual Services Platform 7000
Overview & Highlights
•
Perfect for Today
•
•
•
•
•
•
•
•
Versatile support for 1 or 10
Gigabit Ethernet
Distributed Top-of-Rack delivers
Industry’s fastest virtual backplane
Fabric networking delivered directly
to the Server
Media Dependent Adaptor flexibility
Lossless hardware & software
architecture
Front-back or back-to-front cooling
Small form-factor & energy-efficient
Highlights
Wire-speed performance
Delivering mass 1/10 Gigabit
today
Optimizes application performance
Future-proofed for 40/100
Gigabit & Storage convergence
(SDSN)
Future-Ready for Tomorrow
•
•
Seamless integration of 40/100G
Data Centre Bridging-ready to
integrate Storage Convergence
©2013 Avaya Inc. All rights reserved
10
February 26-28, 2013 | Orlando, FL
Distributed Top-of-Rack
Two Operational modes for true deployment flexibility
Stack-mode DToR
Fabric-mode DToR
Structured Interconnect: 8 Switches
(256 10GbE Ports & 5.12Tbps)
Flexible Interconnect: up to 200 Switches
(6,400 10GbE Ports & 112Tbps)
Row A
Row B
Row C
Row D
From small to very large—D-ToR can handle it!
©2013 Avaya Inc. All rights reserved
11
February 26-28, 2013 | Orlando, FL
Avaya Fabric DToR
Winning on Throughput & Latency
• Avaya Fabric Connect provides more throughput than
competing solutions.
• Competitors offer 128Gbps stacking or 4x40GE
(160Gbps) links.
• FI Stacking up to 20Tbs and Fabric Connect Mesh
supports up to 262.5Tbs.
• Avaya Fabric Connect provides lower latency than
competitors.
• “Ultra Low Latency” competitor provides
7.9uS latency (ToR-Distribution-ToR)
• Fabric Connect Stacking provides latency of 2.1uS –
5.3uS (4 hop maximum in stack of 8)
©2013 Avaya Inc. All rights reserved
12
February 26-28, 2013 | Orlando, FL
Avaya Fabric DToR
Winning on Price
•
•
Avaya Fabric DToR provides cost effective means to interconnect ToRs vs
typical Fat Tree designs of 4 - 20x lower cost.
Typical Fabric Connect deployment costs with 640Gbps for $6,780
•
•
•
4 x 3m Fabric Connect cables (640Gbps) per VSP 7000 with MSRP $6,780.
Fabric Connect cables range from $995 for 0.46m to $8,995 for 100m.
Competing designs with 160Gbp from $27,150 - $142,150
•
•
Competing ToR limited to 160Gbps due to 4 x 40GE ports per ToR.
Due to placement of ToR to Core or Distribution switches require longer cable
runs
• 8 x 40GBASE-SR4 @ $1,100ea + 8 x Fibre @ $250ea = $12,150
•
Require 8 modular ports at costs from $1,875 ea to $ 16,250ea
•
•
•
•
Arista (standalone) $1,875/port
Cisco Nexus 7000: $9,940/port
Juniper (standalone) $2,306/port, QFX3008 $7,226 - $16,250/port
HP (no offering)
Avaya Fabric Connect – the new DNA for your Data Centre Network
©2013 Avaya Inc. All rights reserved
13
February 26-28, 2013 | Orlando, FL
VSP 7000 Fabric DToR
Fabric Interconnect Stacking
•
•
Fabric Stack Mode, in which up to 8 units create a DToR (distributed Top
of Rack) delivering up to 5Tbps using two Fabric Connect cables in
parallel between each switch.
For Fabric DToR Stack operation, the stack operates in the same manner
as other Avaya stackable products and features many of the associated
benefits of Stacking
•
•
•
•
•
Single IP Address Management for Fabric Connect Stack.
Hot Swap Unit replacement with AUR, DAUR, AAUR.
Hot Unit Insertion with NUQC.
Distributed uplinks with DMLT and LAGs.
For Fabric Connect Stack operation each pair of the Up/Down ports
should be thought of as a single connection between each switch.
•
You will need 2 Fabric DToR Cables between each switch to form full adjacency
and the top port on the Up must be connected to the top Down port on an
adjacent switch and similarly for the bottom connector
©2013 Avaya Inc. All rights reserved
14
February 26-28, 2013 | Orlando, FL
Distributed ToR
How to address the new traffic needs
Distributed ToR is
specifically designed for
east/west traffic – ToR
Switches directly
interconnected…
Web
VM
©2013 Avaya Inc. All rights reserved
DToR costs just 1
microseconds Switch
hop
App
VM
D/B
VM
15
Web
VM
February 26-28, 2013 | Orlando, FL
VM Evolution –
The Need For Scale & Flexibility
Members of the same VM domain
must remain in the same subnet
Physical
Servers
Virtual
Servers
APPs
I-P
Per
Rack
APPs
A-H
Per
Rack
APP A
APP B
Rack 1
Rack 1
While application scale
& server utilization is
poor, network
complexity is simple
Rack 2
Server virtualization brings scale
and efficiency to applications. VMs
become essentially “jailed” and
network flexibility is reduced.
VM Mobility is seriously limited. What is the alternative…?
©2013 Avaya Inc. All rights reserved
16
February 26-28, 2013 | Orlando, FL
Scaling Concerns
ERS 8800
VSP 9000
Fabric Connect Core
North-South/CoreToR Interconnects
VSP 7000
SDSN
Distributed
Data Center
SDSN
Distributed Top-of-Rack
Stack-mode DToR
Fabric-mode DToR
Structured Interconnect: 8 Switches
256 10GbE Ports
Flexible Interconnect: up to 200 Switches
6,400 10GbE Ports
©2013 Avaya Inc. All rights reserved
17
SDSN
February 26-28, 2013 | Orlando, FL
VSP 7000
Avaya Fabric Connect Fabric DToR Mesh
•
•
•
•
Avaya’s FI Mesh delivers unmatched scalability
to address low latency East-West requirements in
the Data Centre
Switches in the Mesh are managed through COM
& VPS providing unified provisioning.
Fabric Connect Mesh is scalable from 4 to 480
switches per region.
• Providing low latency fabric.
• Fabric is scalable up to 262.5Tbps.
• Extensible up to 15,360 10GE ports.
• Designed to enable either 24/32/48/56/64
10GE ports per rack.
Fabric DToR ports interconnect each VSP7000:
• Can use mix of different Fabric DToR cable
lengths.
• Optimum performance will be achieved
when the same colour designated ports are
connected to one another.
Sidney Duffy
Avaya Fabric Connect Mesh – Low Latency Data Centre Grade Fabric Scalable to 262.5Tbps
©2013 Avaya Inc. All rights reserved
18
February 26-28, 2013 | Orlando, FL
VSP 7000 - VSP 9000
Avaya Fabric Connect DToR Mesh
•
•
•
Extending the FI Mesh of the
VSP 7000 can be easily
achieved through adding VSP
9000 (or ERS 8800).
VSP 9000 are connected to the
VSP 7000 via high speed
connections to handle traditional
North-South traffic.
VSP 9000 provide advanced
services to the Fabric Connect
Mesh.
•
•
Sidney Duffy
Routing Services for Fabric
Connect Mesh VSNs.
Connectivity between different
Fabric Connect Mesh regions.
Extending Avaya Fabric Connect Mesh with VSP 9000
©2013 Avaya Inc. All rights reserved
19
February 26-28, 2013 | Orlando, FL
VSP 7000 Fabric DToR
Fabric DToR Mesh
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
Rack
©2013 Avaya Inc. All rights reserved
20
February 26-28, 2013 | Orlando, FL
VSP 7000 Fabric Interconnect Mesh
Mesh Architecture Reference
Back Interconnect (240Gbps)
Interconnect top switch in rack N to top switch in adjacent (N-1) rack
Interconnect bottom switch in rack N to bottom switch in N+1 rack
Red Interconnect (160-240Gbps)
Interconnect top switch in rack N to top switch in adjacent (N+1) rack
Interconnect bottom switch in rack N to bottom switch in N-1 rack
Blue Interconnect (80Gbps)
Interconnect 2 switches in same rack (could be leveraged for IST in future release)
Interconnect top switch to top switch is adjacent row in DC
Interconnect bottom switch to bottom witch is adjacent row in DC
(The connection to adjacent rows are shown in parallel in this reference design. These connections could be run from top switch in switch
in row N to top switch in row N-1 and then bottom switch in row N to bottom switch in row N+1)
Rack
Rack
Rack
Logical Pod of 16 Racks (32 switches, 1024 10GE ports) 17.5Tbps Interconnect
©2013 Avaya Inc. All rights reserved
21
February 26-28, 2013 | Orlando, FL
VSP 7000 vToR (Virtual ToR)
Fabric DToR Stacking Reference
FI Stacking (10.1)
ToR
32
Servers
per Rack
Non-Resilient Server connections 32 x10G ports per Rack / 256 10GE ports per Pod
5Tbps East-West Bandwidth
©2013 Avaya Inc. All rights reserved
22
February 26-28, 2013 | Orlando, FL
VSP 7000 DToR
Fabric DToR Stacking Reference
 FI Stacking (10.1)
 Switch Clustering (10.2)
ToR
Cluster
1
ToR
Cluster
2
`
IST
32
Servers
per Rack
Dual
Attached
Fully Resilient 32 x10G ports per Rack / 512 10GE ports per Pod
10Tbps East-West Bandwidth
©2013 Avaya Inc. All rights reserved
23
February 26-28, 2013 | Orlando, FL
Technology Evolution
Completely unique solution to the real DC problem
Unshackle the DC
•
A modern 3-D design, instead
of the antiqued 2-D approach
•
From islands to virtual
modularity
•
Meeting the traffic model of
modern cloud computing
•
Bandwidth where you need it,
when you need it
•
True, extensible virtualization
for Networking, Compute, and
Storage
©2013 Avaya Inc. All rights reserved
24
February 26-28, 2013 | Orlando, FL
Thank you!
#AvayaATF
@cross_randy
@punbehagen
©2013 Avaya Inc. All rights reserved
25
February 26-28, 2013 | Orlando, FL