SP Datacenter fabric technologies
Brian Kvisgaard
System Engineer – CCIE SP #41039
VMDC 2.1 DC Container Architecture
• Simplified architecture
Core
Cisco
Nexus 7000
• Services on the stick design modification (Core/Agg handoff)
• Enterprise centric services integration
• Enterprise multi-tenancy SLA with QoS and alignment with
WAN/Campus QoS requirements
Services
Aggregation
Cisco
Nexus 7000
• Functional multicast integration with multi-tenancy
• Nexus 1010 integration and Network analysis and monitoring
(NAM) capability validation
vPC
Access
Cisco
Nexus 5000
Cisco
UCS 6100
Fabric
Interconnect
Compute
• Jumbo MTU support and jumbo frame validation
• Compute and Storage Components
Nexus
1010
VMware
vCenter
VMware
vSphere
– UCS Blade Server
4x10GE
4x10G
E
NAS
Storage
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
4x10G
E
4x10G
E
UCS
5100
Blade
Server
vPC to N5K
Traditional Networking
Management options:
Limitations:
•
•
•
•
•
•
CLI
Cut/Paste
Limited automation
Disparate management platforms
© 2014 Cisco and/or its affiliates. All rights reserved.
•
•
•
•
•
Cisco Public
Box by box approach
Lack of consistent configuration (no
network wide policies)
Leftover/unknown configuration
Open “any to any” connectivity*
Separate virtual and physical networks
Separate L4-7 device management
ACI Networking
APIC
APIC
APIC
Management options:
Benefits:
•
•
•
•
•
•
•
•
•
•
•
•
GUI (basic/advanced)
CLI
XML/JSON
Scripting
Open API
Automation
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Distributed, Centralised Management
Full traffic visibility*
Self documenting
Integrated virtual and physical network
Integrated L4-7 device management
Policy defined network
New Concept: Endpoint Groups
Endpoint Groups are quite simply groups of endpoints on the network.
The endpoints are identified by their connectivity Domain (virtual/physical/outside) and their connectivity method e.g.
•
Virtual machine portgroups (VLAN, VXLAN)
•
Physical interfaces / VLANs
•
External VLANs
•
External subnets
Devices within the same Endpoint group can communicate irrespective of their VLAN/VXLAN backing/ID,
provided that they have IP reachability.
Communication between Endpoint groups is, by default, not permitted (similar to PVLAN).
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Secure Networking with ACI End Point Groups
APIC
APIC
APIC
VRF: 01 (Anycast gateway)
BD: storage
Hardware Proxy: No
ARP Flooding: Yes
Unknown Unicast Flooding: Yes
IP Routing: No
vPC_to_UCS_a
vlan-12
ANP:
ESXi-Hosts
vPC_to_UCS_b
vlan-12
EPG: vmk-storage
Security Zone
Communication allowed within EPG
BD: vMotion
BD: Host-Mgmt
Hardware Proxy: No
ARP Flooding: Yes
Unknown Unicast Flooding: Yes
IP Routing: No
Hardware Proxy: No
ARP Flooding: Yes
Unknown Unicast Flooding: Yes
IP Routing: No
vPC_to_UCS_a
vlan-10
EPG: vMotion
Security Zone
Communication allowed within EPG
Tenant: ESXi-Hosts
© 2014 Cisco and/or its affiliates. All rights reserved.
vPC_to_UCS_b
vlan-10
Cisco Public
vPC_to_UCS_a
vlan-8
vPC_to_UCS_b
vlan-8
EPG: Host-Mgmt
Security Zone
Communication allowed within EPG
Endpoints in EPG identified by
Interface and VLAN ID
Hypervisor Integration
Network
Admin
APIC
APIC
ACI Fabric
• Integrated gateway for VLAN,
VxLAN, NVGRE networks from
virtual to physical
VLAN
VXLAN
• Normalization for NVGRE, VXLAN,
and VLAN networks
ESX
• Customer not restricted by a choice
of hypervisor
Hyper-V
VLAN
VXLAN
VLAN
KVM
PHYSICAL
SERVER
• Fabric is ready for multi-hypervisor
Application
Admin
© 2014 Cisco and/or its affiliates. All rights reserved.
VLAN
NVGRE
Cisco Public
Hypervisor
Management
New concept: Contracts (ACLs)
Contracts are “directional” Access Lists between Provider and Consumer EPGs. They comprise of one or more Filters
(ACEs) to identify traffic, e.g:
•
Contract: Any-to-Any | Filter: Any-Traffic
•
Contract: Web | Filter: 80, 443, 8000
•
Contract: DNS | Filter: 53
Provider
ANP:
My-Web-App
EPG: Web
Consumer
External
Subnet
Contract: Clients-to-Web
Any-to-Any
Filter: 80, 443 etc
Filter: none
EPG:
Clients
L3out:
Clients
Flags :
© 2014 Cisco and/or its affiliates. All rights reserved.
•
IP Protocol
•
Ports
•
Stateful
•
Etc.
Cisco Public
Flags :
•
Apply in both directions (single contract
which allows return traffic)
•
Reverse filter ports (dynamically permits
return flow based on src/dst ports)
Contracts are Required for Inter EPG Connectivity
APIC
APIC
APIC
VRF: 01 (Anycast gateway)
BD: ESXi
Primary Gateway: 192.168.10.1/24
Secondary Gateway: 192.168.20.1/24
Hardware Proxy: Yes
ARP Flooding: No
Unknown Unicast Flooding: No
IP Routing: 192.168.10.1/24
: 192.168.20.1/24
vPC Node104_105/1/50
vlan-40
192.168.20.10
ANP:
ESXi-Storage
EPG: Shared-storage
ANP:
ESXi-Hosts
Contract = Allow Communication
Tenant: ESXi-Hosts
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
vPC_to_UCS_a
vlan-30
vPC_to_UCS_b
vlan-30
192.168.20.11
192.168.20.12
EPG: vmk-storage
vPC_to_UCS_a
vlan-8
vPC_to_UCS_b
vlan-8
192.168.10.11
EPG: Host-Mgmt
No Contract = No Communication
192.168.10.10
Contracts Scope
Contracts are “scoped” at:
•
Global
•
Tenant
•
Context (aka Private Network, aka VRF)
EPG: Web
EPG: DB
EPG: App
ANP: 01
BD: 01
Hardware Proxy: Yes
IP Routing: Yes
Web_to_App
•
Application Profile
EPG: DB
EPG: Web
App_to_DB
EPG: App
ANP: 02
VRF: 01
Tenant: Web_Hosting
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Application Centric
Infrastructure
DB
DB
Web
Web
App
Web
App
Turnkey integrated solution with
security, centralized management,
compliance and scale
Automated application centric-policy
model with embedded security
Broad and deep ecosystem
Mass Market
© 2014 Cisco and/or
its affiliates.
All rights reserved.
(commercial, enterprises,
public
sector)
Cisco Public
Programmable
Network across the Nexus portfolio
Programmable Network
Starting with programmability boost on N3K /N9K
Programmable
Open APIs
3rd Party DevOps
Automation Tools
Managing Switch with
Linux Tools
Custom Application
Development
DC
Repository
3rd party/custom apps
integration
Nexus
Open, Modular
Operating System
Toolset Integration in
Open NX-OS
Extensible
Open NX-OS
Enhancements to existing
NX-API to support objectbased, model driven APIs
Pre-developed RPMs from
Cisco and Partners
(RESTful XML/JSON)
Leverage same software tools
and expertise across different
IT departments
New SDK enables custom
application development with
option for securelxc
containers
Leverage Linux Toolchain for
Switch Management
CPU, memory, priority controls
*Deliverables and Timelines
for Nexus platforms varies*
Cisco Public
© 2014 Cisco and/or its affiliates. All rights reserved.
Leverage tcpdump, ifconfig
ethtool, iproute, BASH shell
commands for config and
troubleshooting
Application Centric
Infrastructure
DB
Programmable Network
DB
Web
Web
App
Web
App
Turnkey integrated solution with
security, centralized management,
compliance and scale
Modern NX-OS with enhanced
NX-APIs
Automation Ecosystem
Automated application centric-policy
model with embedded security
(Puppet, Chef, Ansible etc.)
Common NX-API
across N2K-N9K
Broad and deep ecosystem
Mass Market
© 2014 Cisco and/or
its affiliates.
All rights reserved.
(commercial, enterprises,
public
sector)
Cisco Public
Mega Scale Datacenters
Underlay Network:
IP Transport
Network
•
IP routing – proven, stable, scalable
•
ECMP – utilize all available network paths
Overlay Network:
VXLAN VNI
VTE
P
VTE
P
Local LAN
VTEP
LocalLAN
LANSegment Local LAN
© 2014 Cisco and/or its affiliates. All rights reserved.
VTEP
Local LAN
Cisco Public
•
Standards-based overlay
•
Layer-2 extensibility and mobility
•
Expanded Layer-2 name space (16M)
•
Scalable network domain
•
Multi-Tenancy
Outer
IP Header
Outer
Mac Header
VXLAN
Header
UDP Header
Original L2 Frame
FCS
FCS
8 Bytes
For next-hop transport in
the underlay network
16
16
16
Source and Destination
VTEP addresses,
allowing transport across
the underlay IP network
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
16
Reserved
32
VNID
32
Reserved
UDP Length
16
Checksu
m
0x0000
UDP
Dst Port
8
UDP
Src. Port
72
Outer
Dst. IP
16
Outer
Src. IP
Ether Type
0x0800
16
Header
Checksum
VLAN ID
Tag
16
Protocol
0x11
VLAN Type
0x8100
48
IP Header
Misc Data
Src.
MAC Addr.
Dst.
MAC Addr.
48
20 Bytes
VXLAN
RRRR1RRR
8 Bytes
10 or 14 Bytes
8
24
24
8
Allows for possible
The well known VXLAN
16M segments
port 4789. Indicates a
VXLAN packet
Hash of the internal L2/L3/L4 header
of the original frame. Can be used as
entropy for better ECMP/LACP load
sharing
VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).
Each VTEP has two interfaces, one is to provide bridging function for local hosts, the
other has an IP identification in the core network for VXLAN
encapsulation/decapsulation.
Transport IP Network
VTEP
VTEP
IP Interface
Local LAN Segment
End System
Local LAN Segment
End System
© 2014 Cisco and/or its affiliates. All rights reserved.
IP Interface
End System
Cisco Public
End System
 No VXLAN control plane
 Data driven flood-&-learn
 Multicast transport for VXLAN BUM (Broadcast, Unknown Unicast and Multicast) traffic.
End System
End
System
End System
A
MAC-A
IP-A
VTEP 1
IP-1
© 2014 Cisco and/or its affiliates. All rights reserved.
VTEP
-3
VTEP
3
IP-3
VTEP-1
Multicast
Group
IP Network
Cisco Public
VTEP-2
VTEP 2
IP-2
End System B
MAC-B
IP-B
The Secret Sauce is the Control
Plane, not the Encapsulation
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
MP-BGP with MPLS VPN Route Distribution
Exchange of VPN Policies Among PE Routers
• Full mesh of BGP sessions
among all PE routers
– BGP Route Reflector
• Multi-Protocol BGP extensions
(MP-iBGP) to carry VPN policies
CE
• PE-CE routing options
Blue VPN
Policy
Red VPN
CE Policy
–
–
–
–
BGP Route Reflector
PE-CE
Link
Static routes
eBGP
OSPF
IS-IS
PE
P
PE
Cisco Public
PE
CE
BlueVPN
Policy`
Red VPN
Policy CE
P
Label Switched Traffic
© 2014 Cisco and/or its affiliates. All rights reserved.
P
PE-CE
Link
P
PE
VPN Control Plane Processing
VRF Parameters
Make customer routes unique:
• Route Distinguisher (RD):
8-byte field, VRF parameters; unique value to make VPN IP routes unique
• VPNv4 address: RD + VPN IP prefix
Selective distribute VPN routes:
• Route Target (RT): 8-byte field, VRF parameter, unique value to define the
import/export rules for VPNv4 routes
• MP-iBGP: advertises VPNv4 prefixes + labels
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Ethernet VPN
Highlights
• Next generation solution for Ethernet
multipoint connectivity services
Data-plane address
learning from Access
– Leverage similarities with L3VPN
• PEs run Multi-Protocol BGP to advertise &
learn MAC addresses over Core
• Learning on PE Access Circuits via dataplane transparent learning
Control-plane address
advertisement / learning
over Core
PE1
PE3
VID 100
SMAC: M1
DMAC: F.F.F
CE1
CE3
MPLS
• No pseudowire full-mesh required
– Unicast: use MP2P tunnels
– Multicast: use ingress replication over MP2P tunnels
or use LSM
• Under standardization at IETF – draft-ietfl2vpn-evpn
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
PE2
PE4
BGP MAC adv. Route
E-VPN NLRI
MAC M1 via PE1
EVPN – Ethernet VPN
VXLAN Evolution
ControlPlane
DataPlane
EVPN MP-BGP
draft-ietf-l2vpn-evpn
Multi-Protocol Label Switching
(MPLS)
Provider Backbone Bridges
(PBB)
Network Virtualization Overlay
(NVO)
draft-ietf-l2vpn-evpn
draft-ietf-l2vpn-pbb-evpn
draft-sd-l2vpn-evpn-overlay
 EVPN over NVO Tunnels (VXLAN, NVGRE, MPLSoE) for
Data Center Fabric encapsulations
 Provides Layer-2 and Layer-3 Overlays over simple IP
Networks
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
22
Fabric Design
3-Tier Design
DC Spine
DC Core
DC Aggregation
DC Access
DC Leaf
DC Interconnect
Collapsed Core/Aggregation
2-Tier Design
DC-2
DC-1
DC Core/ Aggregation
DC Access
WAN
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Flood-&-Learn
EVPN Control Plane
Overlay Services
L2+L3
L2+L3
Underlay Network
IP network with ECMP
IP network with ECMP
Encapsulation
MAC in UDP
MAC in UDP
Peer Discovery
Data-driven flood-&-learn
MP-BGP
Peer Authentication
Not available
MP-BGP
Host Route Learning
Local hosts: Data-driven flood-&-learn
Remote hosts: Data-driven flood-&-learn
Local Host: Data-driven
Remote host: MP-BGP
Host Route Distribution
No route distribution.
MP-BGP
L2/L3 Unicast Forwarding
Unicast encap
Unicast encap
BUM Traffic forwarding
Multicast replication
Unicast/Ingress replication
Multicast replication
Unicast/Ingress replication
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
MP-BGP for EVPN
 MP-BGP is the routing protocol for EVPN
 Multi-tenancy construct using VRF (Rout
Distinguisher, Route Targets)
 New address-family “l2vpn evpn” for distributing
EVPN routes
evpn
vni 20000 l2
rd auto
route-target import auto
route-target export auto
router bgp 100
router-id 10.1.1.11
log-neighbor-changes
address-family ipv4 unicast
address-family l2vpn evpn
neighbor 10.1.1.1 remote-as 100
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community extended
vrf evpn-tenant-1
address-family ipv4 unicast
advertise l2vpn evpn
 EVPN routes = [MAC] + [IP]
 iBGP or eBGP support
© 2014 Cisco and/or its affiliates. All rights reserved.
vrf context evpn-tenant-1
vni 39000
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
Cisco Public
AC
MAC-
Install host info to RIB/FIB:
Install host info to RIB/FIB:
H-MAC-1  MAC table
H-IP-1  VRF IP host table
Host
IP
VNI
H-IP-1
VNII-1
4
VTEP
VTEP-1
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
Route
Reflector
3
3
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
H-MAC-1  MAC table
4
H-IP-1  VRF IP host table
MAC
Host
IP
VNI
VTEP
H-MAC-1
H-IP-1
VNII-1
VTEP-1
2
BGP Update:
H-MAC-1
H-IP-1
VTEP-1
VNI-1
VTEP-3
1
MAC
Host
IP
VNI
H-MAC-1
H-IP-1
VNII-1
VTEP-1
Local learning of host info:
VTEP
H-MAC-1 (MAC table)
H-IP-1 (VRF IP host table )
© 2014 Cisco and/or its affiliates. All rights reserved.
VTEP-1
Cisco Public
H-MAC-1
H-IP-1
VLAN-1 /VNI-1
VTEP-2
BGP Update
RD: Route distinguisher
MAC address length: 6 bytes
MAC address: Host MAC address
IP address length: 32 or 128
IP address: Host IP address (IPv4 or IPv6)
L2 VNI: VNI of the bridge domain to which the end host belongs
L3 VNI: VNI associated with the tenant VRF routing instance
VXLAN
BGP
Control
Plane
EVPN Control Plane --- Host Movement
NLRI:
• Host MAC1, IP1
• NVE IP 1
• VNI 5000
• Next-Hop: VTEP-3
NLRI:
• Host MAC1, IP1
• NVE IP 1
• VNI 5000
• Next-Hop: VTEP-1
Ext. Community:
• Encapsulation: VXLAN
• Cost/Sequence: 1
Ext. Community:
• Encapsulation: VXLAN
• Cost/Sequence: 0
VTEP-1
VTEP-2
VTEP-3
VTEP-4
Host 1
MAC1
IP 1
VNI 5000
1.
2.
3.
4.
MAC
IP
VNI
Next-Hop
Encap
Seq
MAC
IP
VNI
Next-Hop
Encap
Seq
MAC-1
IP-1
5000
VTEP-1
VXLAN
0
MAC-1
IP-1
5000
VTEP-3
VXLAN
1
VTEP-1 detects Host1 and advertise an EVPN route for Host1 with seq# 0
Host1 Moves behind VTEP-3
VTEP-3 detects Host1 and advertises an EVPN route for Host1 with seq #1
VTEP-1 sees more recent route and withdraws its advertisement
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
SVI
GW IP
GW MAC
VTEP
VTEP
Host 1
MAC1
IP 1
VLAN A
VXLAN A
© 2014 Cisco and/or its affiliates. All rights reserved.
Host 2
MAC2
IP 2
VLAN A
VXLAN A
Cisco Public
VTEP
VTEP
Host 3
MAC3
IP 3
VLAN A
VXLAN A
Host 4
MAC4
IP 4
VLAN A
VXLAN A
# VLAN to VNI mapping
vlan 200
vn-segment 5200
# Anycast Gateway MAC, identically configured on all VTEPs
fabric forwarding anycast-gateway-mac 0002.0002.0002
# Distributed IP Anycast Gateway (SVI)
# Gateway IP address needs to be identically configured on all
VTEPs
interface vlan 200
no shutdown
vrf member Tenant-A
ip address 20.0.0.1/24
fabric forwarding mode anycast-gateway
The same anycast gateway virtual IP
address and MAC address need to
be configured on all VTEPs in the
VNI
SVI
GW IP
GW MAC
SVI
GW IP
GW MAC
VTEP
Host 1
MAC1
IP 1
VLAN A
VXLAN A
© 2014 Cisco and/or its affiliates. All rights reserved.
Host 2
MAC2
IP 2
VLAN A
VXLAN A
Cisco Public
SVI
GW IP
GW MAC
VTEP
Host 3
MAC3
IP 3
VLAN A
VXLAN A
SVI
GW IP
GW MAC
VTEP
Host 4
MAC4
IP 4
VLAN A
VXLAN A
VTEP
ARP Suppression in MP-BGP EVPN
ARP suppression reduces network flooding due to host learning
IP Address
MAC Address
VLAN
Physical Interface
Index (ifindex)
Flags
IP-1
MAC-1
10
E1/1
Local
IP-2
MAC-2
10
Null
Remote
IP-3
MAC-3
10
Null
Remote
VTEP-1 intercepts the ARP request and checks in its
ARP suppression cache. It finds a match for IP-2 in its
ARP suppression cache.*
VTEP-1 sends an ARP response back to Host-1 with
MAC-2.*
2
VTEP
1
VTEP
2
Host 1
MAC1
IP 1
VLAN 10
VXLAN 5000
3
VTEP
3
VTEP
4
Host 1
MAC1
IP 2
VLAN 10
VXLAN 5000
1
4
Host-1 in VLAN 10 sends an
ARP request for Host-2’s IP-2
address.
Host-1 learns the IP-2 and MAC-2 mapping.
* If VTEP-1 doesn’t have a match for IP-2 in its ARP suppression cache table, it will flood the ARP request to all other VTEPs in this VNI
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
30
ARP Suppression in MP-BGP EVPN (Cont’ed)
•
ARP Suppression can be enabled on
a per-VNI basis under the interface
nve1 configuration.
VTEP
1
VTEP
2
VTEP
3
VTEP
4
interface nve1
no shutdown
source-interface loopback0
host-reachability protocol bgp
member vni 20000
suppress-arp
mcast-group 239.1.1.1
member vni 21000
suppress-arp
mcast-group 239.1.1.2
member vni 39000 associate-vrf
member vni 39010 associate-vrf
n9396-vtep-1.sakommu-lab.com# sh ip arp suppression topo-info
ARP L2RIB Topology information
Topo-id ARP-suppression mode
100
L2 ARP Suppression
200
L2/L3 ARP Suppression
201
L2/L3 ARP Suppression
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Head-end Replication
Head-end Replication (aka. Ingress replication):
Eliminate the need for underlay multicast to transport overlay BUM
traffic
Spine
Multicast-Free
Underlay
VTEP-1 receives the overlay BUM traffic,
encapsulates the packets into unicast VXLAN
packets, sends one copy to each remote VTEP
peer in the same VXLAN VNI
VTEP
1
2
1
© 2014 Cisco and/or its affiliates. All rights reserved.
VTEP
2
Host-1 sends BUM
traffic into the
VXLAN VNI
Cisco Public
32
VTEP
3
VTEP
4
Leaf
Different integrated Route/Bridge (IRB) Modes
VXLAN Routing
• Overlay Networks do follow two
slightly different integrated
Route/Bridge (IRB) semantics
Routing ?
• Asymmetric
– Uses different “path” from Source to
Destination and back
• Symmetric
SVI
B
SVI
A
VTEP-1
– Uses same “path” from Source to
Destination and back
• Cisco follows Symmetric IRB
© 2014 Cisco and/or its affiliates. All rights reserved.
IP Transport Network
Cisco Public
VTEP-2
Host 1
H-MAC-1
H-IP-1
VNI-A
VTEP-3
VTEP-4
Host 2
H-MAC-2
H-IP-2
VNI-B
Asymmetric
•
Routing and Bridging on the ingress VTEP
•
Bridging on the egress VTEP
•
Both source and destination VNIs need to reside on the ingress VTEP
Ingress VTEP
routes packets
from source VNI to
destination VNI. DMAC in the inner
header is the
destination host
MAC
S-IP: VTEP-1
D-IP: VTEP-4
VNI: VNI-B
1
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VNI
A
VNI
B
VTEP-1
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VNI
A
VTEP-2
VTEP-3
Host 1
H-MAC-1
H-IP-1
VNI-A
© 2014 Cisco and/or its affiliates. All rights reserved.
VNI
B
VTEP-4
Host 2
H-MAC-2
H-IP-2
VNI-B
Cisco Public
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
2
Egress VTEP
bridges packets in
the destination
VNI
VXLAN
BGP
Control
Plane
VTEP VNI Membership Asymmetric IRB
Every VTEP needs to be in all
VNIs
Every VTEP needs to maintain
MAC tables for all VNIs, including
those they don’t have local hosts
for.
SVI 100
SVI 200
VTEP
Host 1
MAC1
IP 1
VLAN 100
VXLAN 5100
1.
2.
SVI 100
SVI 200
SVI 100
SVI 100
SVI 200
VTEP
VTEP
VTEP
Host 2
MAC2
IP 2
VLAN 100
VXLAN 5100
SVI 200
Host 3
MAC3
IP 3
VLAN 200
VXLAN 5200
All VTEPs in a VNI can be the virtual IP gateway for the local hosts
Optimized south-north bound forwarding for routed traffic without hair-pinning
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
• Routing on both ingress and egress VTEPs
• Layer-3 VNI
• Tenant VPN indicator
• One per tenant VRF
• VTEP Router MAC
• Ingress VTEP routes packets onto the Layer-3 VNI
• Egress VTEP routes packets to the destination Layer-2 VNI
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Ingress VTEP
routes packets
from source VNI to
L3 VNI. D-MAC in
the inner header is
the egress VTEP
router MAC
S-IP: VTEP-1
D-IP: VTEP-4
VNI: L3 VNI
1
VNI
A
L3
VNI
VTEP-1
Router MAC-1
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
2
S-MAC: Router-MAC-1
D-MAC: Router-MAC-4
S-IP: H-IP-1
D-IP: H-IP-2
L3
VNI
VTEP-2
VTEP-3
Host 1
H-MAC-1
H-IP-1
VNI-A
© 2014 Cisco and/or its affiliates. All rights reserved.
VNI
B
VTEP-4
Router MAC-4
Host 2
H-MAC-2
H-IP-2
VNI-B
Cisco Public
Egress VTEP
routes packets
from L3 VNI to the
destination
VNI/VLAN
S-MAC: H-MAC-1
D-MAC: H-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VXLAN
BGP
Control
Plane
VTEP VNI Membership Symmetric IRB
Every VTEP only needs to be in VNIs
that it has local hosts for.
VTEPs don’t need to maintain MAC
tables for VNIs that they don’t have local
hosts for.
SVI
100
Host 1
MAC1
IP 1
VLAN 100
VXLAN 5100
1.
2.
VTEP
SVI
100
VTEP
VTEP
Host 2
MAC2
IP 2
VLAN 100
VXLAN 5100
VTEP
SVI
200
Host 3
MAC3
IP 3
VLAN 200
VXLAN 5200
Optimal utilization of ARP and MAC tables
A VTEP only needs to be in the VNIs which it has local hosts for.
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Use VTEP addresses in the outer
header to route encapsulated
packets to the egress VTEP
S-IP: VTEP-1
D-IP: VTEP-2
VNI: L3-VNI-A
IP Transport
Network
S-MAC: Router-MAC-1
D-MAC: Router-MAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VTEP-1
S-MAC: HMAC-1
D-MAC: HMAC-2
S-IP: H-IP-1
D-IP: H-IP-2
VTEP
S-IP: VTEP-1
D-IP: VTEP-2
VNI: L3 –VNI-A
© 2014 Cisco and/or its affiliates. All rights reserved.
VTEP-2
VTEP
S-MAC: HMAC-1
D-MAC: HMAC-2
S-IP: H-IP-1
D-IP: H-IP-2
Host 1
H-MAC-1
H-IP-1
VNI-A
L3-VNI-A
VRF-A
Cisco Public
S-MAC: Router-MAC-1
D-MAC: Router-MAC-4
S-IP: H-IP-1
D-IP: H-IP-2
Host 2
H-MAC-2
H-IP-2
VNI-B
L3-VNI-A
VRF-A
Tenant
Tenant AA
VRF-A
VRF-A
L3-VNI-A
L3-VNI-A
H-IP-2
H-IP-2
Tenant B
VRF-B
L3-VNI-B
Use L3-VNI to identify
the tenant VRF
Tenant C
VRF-C
L3-VNI-C
• Symmetric IRB has optimal utilization of ARP and MAC tables on a VTEP
• Symmetric IRB scales better for end hosts
• Symmetric IRB scales better in terms of the total number of VNIs a VXLAN
overlay network can support
Multi-vendor interoperability:
• Some vendors implemented Asymmetric IRB
• It’s been agreed upon among multiple vendors that Symmetric IRB is the
ultimate solution
• Cisco implemented Symmetric IRB
• Cisco will introduce backward compatability with asymmetric IRB by adding the
support for it.
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Local Scoping of VLANs –ToR Local
16 million possible VNIs global scope
VNI 5000 maps to VLAN 10
VNI 5000 maps to VLAN 60
VLANS are Locally
Scoped at Top of
Rack/ Gateway
VLANS are Locally
Scoped at Top of Rack/
Gateway
Possible VLAN IDs 1-4K
© 2014 Cisco and/or its affiliates. All rights reserved.
Possible VLAN IDs 1-4K
Cisco Public
41
Local Scoping of VLANs – Port Local*
* Available in
Q2CY2015
16 million possible VNIs global scope
(Eth1/1, Vlan10) => VNI 10000
(Eth1/2, Vlan10) => VNI 10001
(Eth1/2, Vlan11) => VNI 10000
VNI 5000 maps to (E1/1, VLAN 10)
VNI 5000 maps to (E1/2, VLAN 60
VLANS are Locally Scoped
VLAN to VNI mapping is per-port
significant
Possible VLAN IDs 1-4K
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VLANS are Locally Scoped
VLAN to VNI mapping is per-port
significant
Possible VLAN IDs 1-4K
42
Underlay IP Network
BGP Router ID 1
BGP Router ID 2
vPC VTEP-1
vPC VTEP with
Anycast VTEP
Address
vPC VTEP-2
interface loopback0
ip address 10.1.1.13/32
ip address 10.1.1.134/32 secondary
Virtual
PortChannel
Layer 2 Link
Layer 3 Link
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
EVPN Control Plane Advantages
A multi-tenant fabric solution with host-based forwarding
•
Industry standard protocol for multi-vendor interoperability
•
Build-in multi-tenancy support
•
•
Truly scalable with protocol-driven learning
•
•
Leverage MP-BGP to deliver VXLAN with L3VPN characteristics
Host MAC/IP address advertisement through EVPN MP-BGP
Fast convergence upon host movements or network failures
•
MP-BGP protocol driven re-learning and convergence
•
Upon host movement, the new VTEP will send out a BGP update to advertise
the new location of the host
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
EVPN Control Plane Advantages (Cont’ed)
A multi-tenant fabric solution with host-based forwarding
•
•
Optimal traffic forwarding supporting host mobility
•
Anycast IP gateway for optimal forwarding for host generated traffic
•
No need for hair-pinning to to reach the IP gateway
ARP suppression
•
•
Minimize ARP flooding in overlay
Head-end Replication with dynamically learned remote-VTEP list
•
Head-end replication enables multicast-free underlay network
•
Dynamically learned remote-VTEP list minimizes the operational overhead of
head-end replication
•
VTEP peer authentication via MP-BGP authentication
•
Added security to prevent rogue VTEPs or VTEP spoofing
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Application Centric
Infrastructure
DB
Programmable Fabric
Programmable Network
DB
Web
Web
App
Web
App
Turnkey integrated solution with
security, centralized management,
compliance and scale
VxLAN-BGP EVPN
standard-based
Modern NX-OS with enhanced
NX-APIs
3rd party controller support
Automation Ecosystem
Automated application centric-policy
model with embedded security
(Puppet, Chef, Ansible etc.)
Common NX-API
across N2K-N9K
Broad and deep ecosystem
Mass Market
© 2014 Cisco and/or
its affiliates.
All rights reserved.
(commercial, enterprises,
public
sector)
Service Providers
Cisco Public
Mega Scale Datacenters
Application Centric
Infrastructure
Programmable Fabric
Programmable Network
VTS
DB
DB
Web
Web
App
Web
App
Turnkey integrated solution with
security, centralized management,
compliance and scale
VxLAN-BGP EVPN
standard-based
Integrated Overlay and
Underlay optimizations
Overlay optimizations
Mass Market
© 2014 Cisco and/or
its affiliates.
All rights reserved.
(commercial, enterprises,
public
sector)
Service Providers
Cisco Public
Modern OS with enhanced APIs
Mega Scale Datacenters
Cisco Virtual Topology System (VTS)
Overlay Provisioning & Management System
Cisco Network Services
Orchestrator
VMware vCenter
GUI
REST API
Automated
Flexible Overlays
Physical and Virtual Overlays
Seamless Integration with Orchestrators
Bare-metal and Virtualized Workloads
Automated Overlay Provisioning
Service Chaining
Automated DCI/WAN Integration
Cisco Virtual Topology
System
Open and Programmable
Scalable VXLAN Mgmt.
REST-Based Northbound APIs
MP-BGP EVPN Control Plane
Multi-protocol Support
Virtual Tenant Networks
High Performance Virtual Forwarding
Multi-hypervisor Support
YANG
CLI
NX-API
Nexus Portfolio
Nexus 2k – 9k
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
BGP-EVPN
VTS Architecture
Cisco Network Services
Orchestrator (Tail-f)
VMware vCenter
GUI
Unified Information Model (REST API)
Virtual Topology System
Inventory
Database
Resource Management
YANG
CLI
IOS XRv
NX-API
Cisco Nexus 2000, 3000,
5000, and 7000 Series
Virtual Compute Environment
OVS
VTF
© 2014 Cisco and/or its affiliates. All rights reserved.
DVS
Cisco Public
Control
Plane
Device Management
Policy Plane
Service and Infrastructure Policy
Control Plane Federation
MP-BGP
BGP-EVPN
Cisco Nexus 9000 Series Cisco ASR 9000 Series
EVPN Control Plane
S1
MAC, IP Address
VTEP1
S2
MAC, IP Address
VTEP2
S3
MAC, IP Address
VTEP3
S4
MAC, IP Address
VTEP4
Industry standard protocol for multi-vendor support
MP-BGP
EVPN RR
IP Transport Network
Restconf/YANG
Built in multi-tenancy support
Scalable, protocol driven control plane architecture
VXLAN VNI
VTEP 1
VTEP 2
Fast convergence upon network failures and host movements
VTEP 3
VTF
Local LAN
LAN Segment
Local LAN
VTEP 4
Local LAN
Local LAN
Minimize flooding through ARP suppression
S3
S2
S1
S4
Security through VTEP peer-authentication
Overlay Forwarding Table
S1
MAC, IP Address
P1/2
S2
MAC, IP Address
VTEP2
S3
MAC, IP Address
VTEP3
S4
MAC, IP Address
VTEP4
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
VTS Architecture – Hardware Switches
Spine
REST
API
Cisco VTS
Spine
NX-API, CLI,
YANG
VTEP
ToR
ToR
VTEP
ToR
VTEP
VMware vCenter
Hypervisor
Hypervisor
Hypervisor
VM
VM
x86 Server
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
x86 Server
51
VM
VM
x86 Server
VTS Architecture – Integrated DCI
Simpler Configuration –
Single MP-BGP session
for all tenants
Spine
REST
API
Cisco VTS
Spine
VTEP
L3 VNIs (Route)
•
VRF Route-Leaking
•
L3PVN Stitching
ToR
VTEP
ToR
VTEP
VMware vCenter
Hypervisor
Hypervisor
Hypervisor
VM
VM
x86 Server
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
x86 Server
52
VM
VTEP
•
NX-API, CLI,
YANG
ToR
DCI
VM
x86 Server
VTS Architecture - VTF
User space packet forwarder, Multi
tennant
DCI
Uses Cisco Vector Packet Processing
technology
Border Leaf
VTEP
Integrated with Intel DPDK
Supports VXLAN, extend to e.g SR,
MPLS, MPLSoGRE, L2TPv3 ..
VTF
Programmed
(VM) by VTS using
Tenant VM
VTF
(VM)
Spine
Tenant VM
Spine
Tenant VM
Restconf/YANG
REST
API
Cisco VTS
NX-API,
CLI,
YANG
vSwitch
vSwitch
ESXi
vSwitch
VTEP
ToR
ToR
VTEP
ToR
KVM
VTEP
NIC
NIC
VMware vCenter
Hypervisor
Hypervisor
Hypervisor
VM
VM
x86 Server
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
x86 Server
53
VM
VM
x86 Server
Tenant VM
VTS 2.0 - Hardware and Software overlay management and provisioning
NX-OS mode based VXLAN fabric with MP-BGP EVPN & ToR-based anycast
gateway
BGP-EVPN
VXLAN Overlay
Hardware Underlay
(standards-based)
VTEP
VTEP
VTEP
VTEP
VTEP
Hardware-based Overlay (standards-based)
VTS
ESX
Bare Metal
ESX
Bare Metal
Software-based Overlay (standards-based)
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
ESX
VTS – OpenStack Workflow
VTS provisions VTEP,
VLAN for each VTEP and
EVPN on ToR/VTF
7
Create
Tenant
Networks
1
2
Tenent and Tenant
Networks Created
Spine
Spine
NX-API,
CLI, YANG
REST API
Cisco VTS
3
VXLAN
VNID assigned
for each
network
OpenStack Tenant View
VXLAN
VTEP
ToR
VTEP
ToR
ToR
4
Attach VM
to Network
5
VM Host info captured
by VTS and mapped to
the right ToR & ToR
port using topology
database
VLAN
Hypervisor
Hypervisor
VM
VM
x86 Server
Neutron agent modified
to request VLAN
information from VTS
before programming
vSwitch
© 2014 Cisco and/or its affiliates. All rights reserved.
VTEP
Hypervisor
x86 Server
6
VLAN
VLAN
Cisco Public
55
VM
VLAN
VM
x86 Server
VTS – OpenStack Workflow
9
VTS provisions L3 VXLAN
(distributed L2/L3) , Anycast
gateway with EVPN
Spine
Spine
NX-API,
CLI, YANG
REST API
Cisco VTS
VXLAN
VXLAN
OpenStack Tenant View
VTEP
ToR
VTEP
ToR
VLAN
VLAN
VLAN
Hypervisor
ToR
VTEP
Hypervisor
Hypervisor
8
Create router
and attach
interfaces to
tenant
networks
© 2014 Cisco and/or its affiliates. All rights reserved.
VM
VM
x86 Server
Cisco Public
x86 Server
56
VM
VLAN
VM
x86 Server
Admin Domain d1
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Administration BGP Route Reflector
The Administrator can choose to install BGP RR
configuration on
1. Virtualized XR
2. Inline RR on Nexus9k Spine
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
View Virtual Forwarding Group
compute2
compute1
BOTH XRv and VTFs
register w/ VTS
automatically
Control Plane is xrv02 running IOS-XR
© 2014 Cisco and/or its affiliates. All rights reserved.
Cisco Public
What does VTS provide Infrastructure Providers
Abstracted view of a network-wide topology
Automate VM discovery in topology and provision virtual
network attachment.
Make it simple for the end-user
Tenant selfprovisioning
Neutron
SW
Forwarder
WA
N
SW forwarder for brownfield deployment
HW forwarder for performance
Virtual Appliance inter-working w/ Physical appliance
HW
Forwarder
Seamless P2V
Stitch Provider L3VPN to Tenant DC virtual network(s)
Tenants attach to External networks via Provider
Network
Connect Tenant
networks to Provider
Networks
VTE
P
VTE
P
VTE
P
© 2014 Cisco and/or its affiliates. All rights reserved.
VTE
P
VTE
P
Cisco Public
VTE
P
Application Centric
Infrastructure
DB
Programmable Fabric
Programmable Network
DB
Web
Web
App
Web
App
Turnkey integrated solution with
security, centralized management,
compliance and scale
Automated application centric-policy
model with embedded security
Broad and deep ecosystem
Mass Market
© 2014 Cisco and/or
its affiliates.
All rights reserved.
(commercial, enterprises,
public
sector)
VxLAN-BGP EVPN
standard-based
Modern NX-OS with enhanced
NX-APIs
3rd party controller support
Automation Ecosystem
(Puppet, Chef, Ansible etc.)
VTS for software overlay
provisioning and management
across N2K-N9K
Service Providers
Cisco Public
Common NX-API
across N2K-N9K
Mega Scale Datacenters