BRKDCT-3060
Deployment Considerations
with Interconnecting Data Centers
Patrice Bellagamba
Distinguished SE
Cisco Europe
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Session Objectives
The main goals of this session are:
• Highlighting the main business requirements driving Data Center
Interconnect (DCI) deployments
• Understand the functional components of the holistic Cisco DCI
solutions
• Get a full knowledge of Cisco LAN extension technologies and
associated deployment considerations
• Integrate routing aspect induced by the emerging application
mobility offered by DCI
This session does not include:
• Network services integration (Firewall / Load Balancer)
‒ This is the objective of BRK… session
• Storage extension considerations associated to DCI
deployments
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
3
Related Cisco Live 2011 Events
DCI Sessions
Session-ID
Session Name
TECIPM-3191
Advanced LISP Techtorial
BRKDCT-2049
Overlay Transport Virtualization
BRKDCT-2081
Cisco FabricPath Technology and Design
BRKDCT-2131
Mobility and Virtualization in the Data Center with LISP and OTV
An important companion to this session is:
BRKDCT-2312 - Design consideration for Network and Security services stretched
over multiple locations
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
4
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
IP Based Solutions
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
5
Data Center Interconnect
Business Drivers
DCI
 Data Centers are extending beyond traditional
boundaries
 Virtualization applications are driving DCI across PODs
(aggregation blocks) and Data Centers
Drivers
Business Solution
Constraints
Business
Continuity
 Disaster Recovery
 Stateless
 Network Service Sync
 Process Sync
 GSLB
 Geo-clusters
 HA Cluster
Operation Cost
Containment
 Data Center Maintenance /
Migration / Consolidation
 Host Mobility
 Distributed Virtual
Data Center
Business
Resource
Optimization
 Disaster Avoidance
 VLAN Extension
 Statefulness
 Bandwidth & Latency
 VM Mobility
Cloud Services
 Inter-Cloud Networking
 Flexibility
 Application mobility
 VM Mobility
 Automation
 HA Framework
 Workload Mobility
 XaaS
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
IT Technology
Cisco Public
6
Data Center Interconnect
LAN Extension Model
Path Optimization
Any type of links
Dual-Homing
STP Domain isolation + Storm-control
STP
domain
GW
STP
domain
GW
ALT
Si
DC1
GW
ALT
Si
DC2
Si
STP
domain
ALT
Si
DC3
Storage extension
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
7
LAN Extension for DCI
VLAN Types
Type T0
Limited to a single access layer
Type T1
Extended inside an aggregation block (POD)
OTV/VPLS
Type T2
Extended between PODs part of the same
DC site
OTV/VPLS
Type T3
T4
Extended between twin DC sites connected
via dedicated dark fiber links
Fabric-path
/ vPC
Type T4
T3
Extended between twin DC sites using non
5*9 connection
Type T5
Extended between remote DC sites
BRKDCT-3060
T2
T1
Fabric-path
/ vPC
T0
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
8
LAN Extension for DCI
Technology Selection Criteria
Over dark fiber or protected D-WDM
Ethernet
VSS & vPC
 Dual site interconnection
FabricPath (TRILL)
Campus style
MPLS Transport
 EoMPLS
 Transparent point to point
MPLS
 A-VPLS
SP style
 Enterprise style MPLS
 H-VPLS
 Large scale & Multi-tenants
IP Transport
IP
BRKDCT-3060
 OTV
IP style
 Enterprise style Inter-site MAC Routing
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
9
LAN Extension for DCI
Technology Selection Criteria
• Transport
Fiber
LOS report / Protected DWDM
Ethernet only for 5*9 HA link
MPLS/IP for WAN quality link
L2 SP offer (HA=99.7+)
IP
• Scale
Site
VLAN (102 or 103 or 104)
MAC (103 or 104 or 105)
Ethernet for medium scale
IP for low scale
MPLS for high scale
• Multi-tenants
Tagging (VLAN / 2Q / VRF)
Overlapping / Translation
• Multi-point or point to point
• Greenfield vs. Brownfield
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
MPLS for multi-tenancy features
Cisco Public
10
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
VSS, vPC and FabricPath
MPLS Based Solutions
IP Based Solutions
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
11
Dual Site Interconnection
Leveraging EtherChannel between Sites
On DCI Etherchannel:
 STP Isolation (BPDU Filtering)
 Broadcast Storm Control
 FHRP Isolation
Primary Root
interface port-channel10
desc DCI point to point connection
switchport
switchport mode trunk
vpc 10
switchport trunk allowed vlan 100-600
spanning-tree port type edge trunk
spanning-tree bpdufilter enable
storm-control broadcast level 1
storm-control multicast level x
 Link utilization with MultiChassis EtherChannel
 DCI port-channel
- 2 or 4 links
 Requires protected
DWDM or Direct fibers
BRKDCT-3060
Primary Root
L
2
WAN
L
3
Si
Server Cabinet Pair 1
L
2
L
3
Si
Server Cabinet Pair N
Server Cabinet Pair 1
Server Cabinet Pair N
 vPC does not support L3 peering:
Use dedicated L3 Links for Inter-DC routing!
 Validated design:
200 Layer 2 VLANs + 100 VLAN SVIs
1000 VLAN + 1000 SVI (static
routing)
Cisco Public
© 2012 Cisco and/or its affiliates. All rights reserved.
12
FabricPath is primarily positioned for Clos-based architectures
L2 DCi is NOT LAN Switching! Is FabricPath a valid solution for DCi ?
DC Site 1
DC Site 3
DC Site 4
DC Site 2
• Perception on FabricPath DCi
‒
‒
‒
‒
‒
‒
Plug and play
No Spanning Tree events shared between DC sites
Can do IP routing over FP DCi
One single protocol to manage end to end
One single Fabric end to end
Works also with N5K only scenarios
13
FabricPath DCI - Lessons learned
Site A
Site B
Root MDT1
S1
• Dependencies with L1 WAN links
40
10
-
Requires point to point high quality connections
-
Golden rule : WAN links must support Remote Port Shutdown and micro flapping protection
40
40
• Multidestination traffic impacts
-
Must tune multicast tree to avoid local traffic to fly over root tree site
-
Cannot avoid multicast to fly over root tree site for DCI multicast
R1
20
40
5
Site C
• IP routing over FabricPath
-
Ship in the night effect
-
OSPF hellos are multicast and will fly over root site
R2
• STP interactions with FabricPath DCI
-
The Fabric becomes STP root for all propagated VLAN, means that twin site vPC will be blocking
• FabricPath & HSRP Localization
-
HSRP Control-plane can be isolated with mismatching authentication key
-
But HSRP data-plane cannot be isolated when DC is also FP, leading to flapping vMAC
• High Availability
-
L2 ISIS fine tuning is required: allocate-delay timer, transition-delay, linkup-delay, spf-interval, lsp-gen-interval
-
Sub second convergence, except node recovery in 3s
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
FabricPath DCI - Key Takeaways
Customer
references
Operations
simplicity
Domino effect
prevention
DCi link quality
mgmt
3+ Sites
optimization
High Availability
L2 functions
L3 Unicast
functions
Multicast
functions
Scalability
FabricPath
• On DCi, FabricPath is not so Plug and Play actually
‒ No specific DCI functions compared to OTV, VPLS
‒ Several designs gotchas but do not impact all customers
‒ Multidestination Trees capacity planning may be very complex
‒ Multiple Topologies will enhance the overall solution
• By default, OTV/VPLS should be the first solutions to promote
‒ Cisco Validated Designs (CVDs)
‒ Specific DCi features
‒ Offer an efficient independence between DC
• FabricPath is a valid DCi solution when :
‒ Short distances between DCs (tromboning is not a issue)
‒ Multicast is not massively used
15
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
EoMPLS
VPLS
H-VPLS
IP Based Solutions
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
16
EoMPLS
Port Mode xconnect
interface
PE1
T-LDP
PE2
interface g1/1
description EoMPLS port mode connection
no switchport
no ip address
xconnect 2.2.2.2 vcid 1 encapsulation mpls
DA’
SA’
0x8847
LSP Label
VC Label
interface
Ethernet
Header
8
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Ethernet
Payload
FCS’
1518
Cisco Public
17
EoMPLS Usage for DCI
End-to-End Loop Avoidance using Edge to Edge LACP
On DCI Etherchannel:
 STP Isolation (BPDU Filtering)
 Broadcast Storm Control
 FHRP Isolation
Active PW
MPLS Core
DCI
Aggregation
Layer DC1
Active PW
DCI
Aggregation
Layer DC2
Encryption Services with 802.1AE
 Requires a full meshed vPC  4 PW
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
18
EoMPLS Usage for DCI
Over IP Core
Active PW
IP Core
DCI
Aggregation
Layer DC1
Active PW
DCI
Aggregation
Layer DC2
crypto ipsec profile MyProfile
set transform-set MyTransSet
interface Tunnel100
ip address 100.11.11.11 255.255.255.0
ip mtu 9216
mpls ip
tunnel source Loopback100
tunnel destination 12.11.11.21
tunnel protection ipsec profile MyProfile
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
19
Dealing with PseudoWire (PW) Failures
Remote Ethernet Port Shutdown
PE receives the PW down
notification and shutdown its
transmit signal toward
aggregation
Active PW
X
X
X
Si
MPLS Core
Si
DCI
Aggregation
Layer DC1
Active PW
DCI
Aggregation
Layer DC2
Failover
(msec)
Fallback
(msec)

281
54

453
300
ASR1000 / ASR903 feature configuration:
interface GigabitEthernet1/0/0
Bridged
traffic
xconnect 1.1.1.1 1 pw-class eompls
remote link failure notification
BRKDCT-3060
! (default)
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
20
EoMPLS Deployment on VSS
Point to Point EoMPLS with Port-Channel xconnect
Local LACP
Local LACP
One PW
Si
Si
MPLS
Si
Aggregation
Layer DC1
VSS
Si
VSS
Aggregation
Layer DC2
 Instead of xconnecting physical port, xconnect port-channel
 LACP is kept local, no more extended over EoMPLS
 PW is virtual on both VSS members
SSO protection in 12.2(33)SXJ
 Requires VSS or Nexus as DC device
Limited support of L3 routing with vPC
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
21
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
EoMPLS
VPLS
H-VPLS
IP Based Solutions
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
22
Multi-Point Topologies
What is VPLS?
PW
VFI
VLAN
VLAN
SVI VFI
MPLS
Core
PW
One extended bridge-domain built using:
 VFI = Virtual Forwarding Instance
( VSI = Virtual Switch Instance)
 PW = Pseudo-Wire
PW
VFI
SVI
Mac address table population
 is pure Learning-Bridge
SVI
 SVI = Switch Virtual Interface
 xconnect
VLAN
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
23
VPLS Cluster Solutions
SUP720+ES
• Using clustering mechanism
‒ Two devices in fusion as one
• VSS Sup720
SUP2T
• VSS Sup2T
• ASR9K nV virtual cluster
 One control-plane / two data-planes
• Dual node is acting as one only device
ASR9K nV
• Native redundancy (SSO cross chassis)
• Native load balancing
• Capability to use port-channel as attachment circuit
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
24
VPLS Redundancy
Making Usage of Clustering
X
Si
Si
Si
Si
mpls ldp session protection
mpls ldp router-id Loopback100 force
Failover
(msec)
VSS
Bridged
traffic
Fallback
(msec)

258
218

162
174
 LDP session protection & Loopback usage allows
PW state to be unaffected
 LDP + IGP convergence in sub-second
Fast failure detection on Carrier-delay / BFD
 Immediate local fast protection
Traffic exit directly from egress VSS node
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
25
VPLS Redundancy
Making Usage of Clustering
X
Si
Si
mpls ldp graceful-restart
Si
Si
 If failing slave node: PW state is unaffected
 If failing master node:
Failover
(msec)
VSS
Bridged
traffic
BRKDCT-3060
•
•
Fallback
(msec)

224
412

326
316




PW forwarding is ensured via SSO
PW state is maintained on the other side using
Graceful restart
Edge Ether-channel convergence in sub-second
Traffic is directly going to working VSS node
Traffic exits directly from egress VSS node
Quad sup SSO for SUP2T in 1QCY13
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
26
VPLS Deployment Considerations
“Symmetry is Good”
Problem / Solution
10.100.1.1
sh ip route 10.100.1.1
Known via "ospf 2”
via GigabitEthernet1/3/0/1 Route metric is 2
via GigabitEthernet2/3/0/1 Route metric is 2
X
Remote VSS are having two un-equal cost path to others, so one only route is put in RIB
 Stops forwarding traffic for 2mn when primary route is removed
(there is no control-plane to insert backup route)
Build a symmetric core with two ECMP paths between each VSS
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
27
VSS - A-VPLS CLI
#sh mpls l2 vc
SUP2T in 15.1SY
Local intf
------------VFI VFI_610_
VFI VFI_610_
VFI VFI_611_
VFI VFI_611_
Si
transport vpls mesh
neighbor 10.100.2.2 pw-class Core
neighbor 10.100.3.3 pw-class Core
Si
Si
Si
interface Virtual-Ethernet1
switchport
switchport mode trunk
switchport trunk allowed vlan 610-619
BRKDCT-3060
Dest address VC ID Status
------------ ----- -----10.100.2.2
610
UP
10.100.3.3
610
UP
10.100.2.2
611
UP
10.100.3.3
611
UP
Rem: One PW per VLAN per destination
Si
pseudowire-class Core
encapsulation mpls
Local circuit
------------VFI
VFI
VFI
Si
VFI
 Any card type facing edge
 SUP720 + SIP-400 facing core (5Gbps) or
 SUP720 + ES-40 (40Gbps) support with
12.2(33)SXJ
 SUP2T
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
28
VSS - A-VPLS over GRE
SUP2T in 15.1SY
Si
Si
One GRE tunnel per site
Si
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Si
int tunnel 1
tunnel mode gre ip
mpls ip
tunnel source 10.1.1.1
tunnel destination 10.2.2.2
…
interface virtual-ethernet 1
transport vpls mesh
neighbor 10.2.2.2 pw-class cl1
…
switchport
switchport mode trunk
switchport trunk allowed vlan 10, 20
ip route 10.2.2.2 255.255.255.255
Tunnel1
…
Si
Si
 Native on SUP2T
Cisco Public
29
ASR9K VPLS Set-up
PW
l2vpn
router-id 10.0.1.1
bridge group BG
bridge-domain BD
interface TenGigE0/0/0/4
interface TenGigE0/0/0/5
!
vfi VFI
vpn-id 4003
neighbor 10.0.1.2 pw-id 4003
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
30
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
EoMPLS
VPLS
H-VPLS
IP Based Solutions
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
31
DC Access Multi-Homing
Inter Chassis Communication Protocol - ICCP
Redundancy
Group
draft-ietf-martini-pwe3-iccp
C7600 SRE with ES facing edge
ASR9K XR4.0
ASR903 Q3CY13
Active POA
DHD
ICCP
 ICCP synchronizes event/states between multiple
chassis in a redundancy group
 ICCP runs over reliable LDP / TCP
 ICCP relies on BFD/IP route-watch as keepalive
 ICCP message to synch state
Ex: LACP, IGMP query …
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
MPLS
Standby POA
Terminology:
mLACP :
Protocol
MC-LAG :
DHD :
DHN :
POA :
Multi-Chassis Link Aggregation Control
Multi-Chassis Link Aggregation Group
Dual Homed Device (Customer Edge)
Dual Homed Network (Customer Edge)
Point of Attachment (Provider Edge)
Cisco Public
32
DC Access Multi-Homing
Inter Chassis Communication Protocol - ICCP
Multi-Chassis LACP synchronization:
 LACP BPDUs (01:80:C2:00:00:00) are exchanged on each
Link
 System Attributes: Priority + bundle MAC Address
 Port Attributes: Key + Priority + Number + State
Redundancy
Group
Active POA
DHD
ICCP
redundancy
iccp
group <ig-id>
mlacp node <node id>
mlacp system mac <system mac>
mlacp system priority <sys_prio>
member
neighbor <mpls device>
interface <bundle>
mlacp iccp-group <ig-id>
mlacp port-priority <port prio>
interface <physical interface>
bundle id <bundle id> mode active
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
MPLS
Standby POA
Terminology:
mLACP :
Protocol
MC-LAG :
DHD :
DHN :
POA :
Multi-Chassis Link Aggregation Control
Multi-Chassis Link Aggregation Group
Dual Homed Device (Customer Edge)
Dual Homed Network (Customer Edge)
Point of Attachment (Provider Edge)
Cisco Public
33
MC-LAG to VPLS Testing
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DCI/vpls/vpls_asr9k.html
2
1
3
4
5
8
Si
MPLS
core
6
7
Si
Only error 2/3/4 are leading to ICCP convergence
Rem: 2 & 4 are dual errors
 500 VLAN Unicast: Link error sub-1s & Node error sub-2s
 1200 VLAN unicast: Link error sub-2s & Node error sub-4s
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
34
Flexible VLAN Handling
Ethernet Virtual Circuit - EVC
1. Selective Trunk Support
Group multiple VLAN in one only core bridge domain
• QinQ model or PBB Model
• VLAN overlapping
2. VLAN translation 121 / 222 / …
Inter-DC VLAN numbering independency
3. Scale to 4000 * 4000 VLAN
Scale above 4000 VLAN
4. Routing for multi-TAG
Multi-tenant default gateway
IRB - IP routing / VRF routing for QinQ tagged frames
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
35
E-VPN (aka Routed VPLS)
Main Principles
 Control-Plane Distribution of Customer MACAddresses using BGP
BGP
PE continues to learn C-MAC over AC
When multiple PEs announce the same C-MAC, hash
to pick one PE
 MP2MP/P2MP LSPs for Multicast Traffic
Distribution
PE
PE
PE
PE
 MP2P (like L3VPN) LSPs for Unicast Distribution
 Full-Mesh of PW no longer required !!
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
36
Nexus 7000 - Data Center Interconnect with VPLS
Q3 CY 2013
Layer 2 switchport Trunk Portchannel
Primary N7K WAN Edge
Vlan X
Virtual Port
Channel (vPC)
VLAN X
VLAN tied to Active VFI
with neighbors to remote
DC sites
VFI
VLAN tied to Standby
VFI with neighbors to
remote DC sites
MCT
VFI
Vlan X
VLAN X
MCT = Multi-chassis Trunk Interface
VFI = Virtual Forwarding Instance
Secondary N7K WAN Edge
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Data Center Interconnect with VPLS
PE1
10.0.0.1
Sample Configuration – Nexus 7000
PE 1
• Primary VFI owner for
EVEN vlans
• Secondary owner for
ODD vlans
80
vlan 80-81
!
vlan configuration
member vfi vpls-80
!
vlan configuration 81
member vfi vpls-81
!
l2vpn vfi context vpls-80
vpn id 80
redundancy primary
member 10.0.0.3 encapsulation mpls
member 10.0.0.4 encapsulation mpls
!
l2vpn vfi context vpls-81
vpn id 81
redundancy secondary
member 10.0.0.3 encapsulation mpls
member 10.0.0.4 encapsulation mpls
!
interface port-channel50
switchport mode trunk
switchport trunk allowed vlan 80,81
PE 2
• Primary VFI owner for
ODD vlans
• Secondary owner for
EVEN vlans
vlan 80-81
!
vlan configuration 80
member vfi vpls-80
!
vlan configuration 81
member vfi vpls-81
!
l2vpn vfi context vpls-80
vpn id 80
redundancy secondary
member 10.0.0.3 encapsulation mpls
member 10.0.0.4 encapsulation mpls
!
l2vpn vfi context vpls-81
vpn id 81
redundancy primary
member 10.0.0.3 encapsulation mpls
member 10.0.0.4 encapsulation mpls
!
interface port-channel50
switchport mode trunk
switchport trunk allowed vlan 80,81
Note: Virtual Port Channel (vPC) configuration not shown
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
PE3
EVEN VLANs
10.0.0.3
VFI
VFI
VFI
VFI
PE2
PE4
10.0.0.2
10.0.0.4
PE1
PE3
vPC
10.0.0.1
ODDVLANs
10.0.0.3
VFI
VFI
VFI
VFI
PE2
PE4
10.0.0.2
10.0.0.4
vPC
Nexus 7000 - Layer 3 + Layer 2 Extension
• Dual VDC or additional VPLS PE
layer required for L3 and L2
extension for the same vlan
‒ Double-sided vPC design (dual vPC
peer links)
PE2
PE1
vPC
Operational Primary
VFI
100
VFI
200
VFI
100
Active VFI
Vlan 100
VFI
200
vPC
Operational Secondary
Standby VFI
Vlan 100
Peer Link
‒ No VPLS and IRB support
AGG A
vPC
Operational Primary
Primary Root
Vlan 100
AGG B
CFS
AGG
A
DP
AGG
B
RP
Peer Link
DP
DP
100, flow 1
RP
vPC
Operational Secondary
Secondary Root
Vlan 100
100, flow 2
Access
X
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
39
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
IP Based Solutions
OTV Technology Overview
OTV Deployment Considerations
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
40
Overlay Transport Virtualization
Technology Pillars
OTV is a “MAC in IP” technique to
extend Layer 2 domains
OVER ANY TRANSPORT
Dynamic Encapsulation
No Pseudo-Wire State
Maintenance
Optimal Multicast
Replication
Protocol Learning
Nexus 7000
First platform to support OTV
(since 5.0 NXOS Release)
Built-in Loop Prevention
Multipoint Connectivity
Point-to-Cloud Model
BRKDCT-3060
Preserve Failure Boundary
Automated Multi-homing
ASR 1000
Now also supporting OTV
(since 3.5 XE Release)
© 2012 Cisco and/or its affiliates. All rights reserved.
Site Independence
Cisco Public
41
Overlay Transport Virtualization
OTV Control Plane
 Edge Device (ED): connects the site to the (WAN/MAN) core and responsible for
performing all the OTV functions
 Internal Interfaces: L2 interfaces (usually 802.1q trunks) of the ED that face the site
 Join Interface: L3 interface of the ED that faces the core
 Overlay Interface: logical multi-access multicast-capable interface. It encapsulates
Layer 2 frames in IP unicast or multicast headers
OTV
L2
Internal
Interfaces
BRKDCT-3060
Overlay
Interface
L3
Join
Interface
© 2012 Cisco and/or its affiliates. All rights reserved.
Core
Cisco Public
42
OTV Data Plane
Inter-Site Packet Flow
4
Transport
Infrastructure
MAC TABLE
VLAN
MAC
100
IF
MAC 1
Eth 2
MAC 2
Eth 1
100
MAC 3
IP B
100
MAC 4
IP B
100
OTV
MAC 1  MAC 3
1
BRKDCT-3060
Server 1
IP A
OTV
3
Encap
MAC 1  MAC 3
IP A IP B
West
Site
MAC TABLE
Decap
5
MAC 1  MAC 3
VLAN
IP B
OTV
IP A  IP B
MAC
IF
100
MAC 1
IP A
100
MAC 2
100
MAC 3
Eth 3
100
MAC 4
Eth 4
OTV
IP A
6
Layer 2
Lookup
MAC 1  MAC 3
East
Site
Server 3
7
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
43
Overlay Transport Virtualization
OTV Control Plane
• Neighbor discovery and adjacency over
Multicast (Nexus 7000 and ASR 1000)
Unicast (Adjacency Server Mode currently available with Nexus 7000 from 5.2 release)
• OTV proactively advertises/withdraws MAC reachability (control-plane learning)
• IS-IS is the OTV Control Protocol - No specific configuration required
4
1
3 New MACs are
learned on VLAN 100
Vlan 100
MAC A
Vlan 100
MAC B
Vlan 100
MAC C
OTV updates exchanged via
the L3 core
3
VLAN
MAC
IF
100
MAC A
IP A
100
MAC B
IP A
100
MAC C
IP A
East
2
IP A
VLAN
West
3
MAC
IF
100
MAC A
IP A
100
MAC B
IP A
100
MAC C
IP A
4
South
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
44
OTV Failure Domain Isolation
Spanning-Tree Site Independence
 Site transparency: no changes to the STP topology
 Total isolation of the STP domain
 Default behavior: no configuration is required
 BPDUs sent and received ONLY on Internal Interfaces
OTV
The BPDUs
stop here
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
OTV
The BPDUs
L3
stop here
L2
Cisco Public
45
OTV Failure Domain Isolation
Preventing Unknown Unicast Storms
 No requirements to forward unknown unicast frames
 Assumption: end-host are not silent or uni-directional
 Default behavior: no configuration is required
MAC TABLE
VLAN
OTV
OTV
100
100
-
L3
L2
MAC
IF
MAC 1
Eth1
MAC 2
IP B
-
-
No MAC 3 in the
MAC Table
MAC 1  MAC 3
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
46
OTV Multi-homing
VLANs Split Across AEDs
• Automated and deterministic algorithm (not
configurable)
Remote OTV Device MAC
Table
• In a dual-homed site:
VLAN
Lower IS-IS System-ID (Ordinal 0) = EVEN VLANs
Higher IS-IS System-ID (Ordinal 1) = ODD VLANs
MAC
IF
100
MAC 1
IP A
101
MAC 2
IP B
• Future functionality will allow to tune the behavior
OTV-a# show otv vlan
OTV Extended VLANs and Edge Device State Information (* - AED)
VLAN
---100
101*
102
Auth. Edge Device
-----------------East-b
East-a
East-b
Vlan State
---------inactive(Non AED)
active
inactive(Non AED)
Overlay
------Overlay100
Overlay100
Overlay100
OTV-b# show otv vlan
OTV Extended VLANs and Edge Device State Information (* - AED)
VLAN
---100*
101
102*
Auth. Edge Device
-----------------East-b
East-a
East-b
Vlan State
---------active
inactive(Non AED)
active
Overlay
------Overlay100
Overlay100
Overlay100
AED
ODD VLANs
IP A
OTV
OTV-a
Overlay Adjacency
Site Adjacency*
IP B
OTV
AED
EVEN VLANs
OTV-b
Internal peering for AED
election
*Supported from 5.2 NX-OS release
47
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
IP Based Solutions
OTV Technology Overview
OTV Deployment Considerations
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
48
OTV and SVI Routing
Introducing the OTV VDC
• Guideline: The current OTV implementation on the Nexus 7000 enforces the
separation between SVI routing and OTV encapsulation for any extended VLAN
• This separation can be achieved with having two separate devices to perform
these two functions
• An alternative cleaner and less intrusive solution is the use of Virtual Device
Contexts (VDCs) available with Nexus 7000 platform:
A dedicated OTV VDC to perform the OTV functionalities
The Aggregation-VDC used to provide SVI routing support
L3
L2
BRKDCT-3060
OTV
VDC
OTV
VDC
© 2012 Cisco and/or its affiliates. All rights reserved.
Aggregation
Cisco Public
49
Placement of the OTV Edge Device
OTV in the DC Aggregation
 L2-L3 boundary at aggregation
 DC Core performs only L3 role
 STP and L2 broadcast Domains
isolated between PODs
 Intra-DC and Inter-DCs LAN extension
provided by OTV
Requires the deployment of dedicated
OTV VDCs
SVIs
SVIs
SVIs
SVIs
vPC
vPC
 Ideal for single aggregation block
topologies
 Recommended for Green Field
deployments
Nexus 7000 required in aggregation
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
50
Single Homed OTV VDC
Simple Model
N7K-B
N7K-A
Link-1
OTV
VDC
OTV
VDC
Routing VDC
OTV VDC
 May use a single physical link for Join and
Internal interfaces
Link-2
Po1
Minimizes the number of ports required to
interconnect the VDCs
Routing VDC
OTV VDC
Logical View
N7K-A
N7K-B
 Single link or physical node (or VDC)
failures lead to AED re-election
50% of the extended VLANs affected
Link-1
Link-2
Po1
Link-3
Link-4
 Failure of the routed link to the core is not
OTV related
Recovery is based on IP convergence
Physical View
Layer 3
Layer 2
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
51
Dual Homed OTV VDC
Improving the Design Resiliency
N7K-B
N7K-A
Links 1-2
Links 3-4
Po1
Routing VDC
OTV VDC
Routing VDC
Link 5
Links 3-4
Po1
Link 7
Link 6
Physical View
Layer 3
Layer 2
Increases the number of physical interfaces
required to interconnect the VDCs
 Traffic recovery after single link failure
event based on port-channel re-hashing
N7K-B
Link 8
BRKDCT-3060
OTV VDC
Logical View
N7K-A
Links 1-2
 Logical Port-channels used for the Join
and the Internal interfaces
OTV
VDC
OTV
VDC
No need for AED re-election
 Physical node (or VDC) failure still
requires AED re-election
In the current implementation may cause few
seconds of outage (for 50% of the extended
VLANs)
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
52
OTV in the DC Aggregation
Site Based Per-VLAN Load Balancing
Simple Appliance Model
Aggregation
OTV
VDC
OTV
VDC
 AED role negotiated between the two OTV VDCs (on a
per VLAN basis)
Internal IS-IS peering on the site VLAN
Recommended to carry the site VLAN on vPC links and vPC
peer-link
AED
 For a given VLAN all traffic must be carried to the AED
Device
Part of the flows carried across the vPC peer-link
Optimized traffic flows is achieved in the most resilient model
leveraging Port-Channels as Internal Interfaces
Most Resilient Model
Aggregation
OTV
VDC
AED
OTV
VDC
 The AED encapsulates the original L2 frame into an IP
packet and send it back to the aggregation layer device
 The aggregation layer device routes the IP packet
toward the DC Core/WAN edge
 L3 routed traffic bypasses the OTV VDC
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
53
OTV in the DC Aggregation
Using F-Series Linecards
• F1 and F2 linecards do not support OTV natively
• As of today, the OTV VDC must use only M-series ports for
both Internal and Join Interfaces
Recommendation is to allocate M1 only interfaces to the OTV VDC
• Native OTV support on F-series is targeted for 6.2 release
(Q2CY13)
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
55
OTV in the DC Aggregation
Configuration (Multicast Transport)
Routing VDC
PIM enabled interfaces
OTV
VDC
Routing VDC
N7K-Agg1
OTV
VDC
e2/1
e2/2
e1/1
e1/2
Routing VDC
N7K-Agg2
OTV VDC
L3 Link
L2 Link
BRKDCT-3060
Establish L3 peering
on a dedicated VLAN
**All
Could
use
static default route or ospf stub
© 2012 Cisco and/or its affiliates.
rights
reserved.
hostname routing-vdc
!
interface Ethernet1/1
switchport
switchport mode trunk
switchport trunk allowed vlan 100,600-700
!
interface Ethernet2/1
ip address 3.3.3.1/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
ip pim sparse-mode
ip igmp version 3
!
ip pim rp-address 33.33.33.33 group-list 224.0.0.0/4
ip pim ssm range 232.0.0.0/8
hostname otv-vdc
feature otv
!
otv site-vlan 100
!
interface Ethernet1/2
description Internal Interface
switchport
switchport mode trunk
switchport trunk allowed vlan 100,600-700
!
interface Ethernet2/2
description Join Interface
ip address 3.3.3.2/24
ip igmp version 3
!
interface Overlay100
otv join-interface Ethernet2/2
otv control-group 239.1.1.2
otv data-group 232.1.1.0/24
otv extend-vlan 600-700
!
ip route 0.0.0.0 0.0.0.0 3.3.3.1
Cisco Public
56
Release 5.2
and above
OTV in the DC Aggregation
Configuration (Unicast Transport)
Routing VDC
OTV
VDC
OTV
VDC
Routing VDC
N7K-Agg1
L3 Link
L2 Link
BRKDCT-3060
e2/1
e2/2
e1/1
e1/2
Routing VDC
N7K-Agg2
Establish L3 peering
on a dedicated VLAN
© 2012 Cisco and/or its affiliates. All rights reserved.
OTV VDC
hostname routing-vdc
!
interface Ethernet1/1
switchport
switchport mode trunk
switchport trunk allowed vlan 100,600-700
!
interface Ethernet2/1
ip address 3.3.3.1/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
hostname otv-vdc
feature otv
!
otv site-vlan 100
!
interface Ethernet1/2
description Internal Interface
switchport
switchport mode trunk
switchport trunk allowed vlan 100,600-700
!
interface Ethernet2/2
description Join Interface
ip address 3.3.3.2/24
!
interface Overlay100
otv join-interface Ethernet2/2
otv adjacency-server*
otv use-adjacency-server 10.1.1.1 11.1.1.1
otv extend-vlan 600-700
!
ip route 0.0.0.0 0.0.0.0 3.3.3.1
Cisco Public
* Needed only on the Adjacency Server
57
Placement of the OTV Edge Device
Connecting Brownfield and Greenfield Data Centers
OTV
Greenfield
OTV
Brownfield
ASR 1K
Greenfield
L3
OTV
OTV
L2
Si
Si
Nexus 7K
L3
L3
OTV
OTV
Nexus 7K
OTV
OTV
Nexus 7K
L2
Nexus 7K
Si
Si
L2
L3
L2
FabricPath
OTV Virt. Link
 Leverage OTV capabilities on Nexus 7000 (Greenfield) and ASR 1000 (Brownfield)
 Build on top of the traditional DC L3 switching model (L2-L3 boundary in Agg, Core
is pure L3)
 Possible integration with the FabricPath/TRILL model
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
60
6.2 (Q2CY13) OTV – New functionality
Key features for larger DCI usage
• Selective Unicast Flooding
(for unidirectional MACs & silent hosts)
• OTV VLAN translation
• Dedicated Data Broadcast Multicast Group
• Multiple Uplinks / Loopback ED IP
• Scalability to large deployment
• Fast convergence
‒ AED synchronization
‒ Fast remote convergence using Site-ID
‒ Fast local convergence using pre-population
‒ Fast ED failure detection using BFD & route tracking
• F1 and F2e as internal interfaces (proxy mode)
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
IP Based Solutions
• LISP for DCI Deployments
LISP and Path Optimization
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
62
Path Optimization and DCI
Avoid Suboptimal Traffic Path After Workload Motion
144.254.100.0/25 & 144.254.100.128/25
EEM or RHI can be used to get very granular
ISP B
Layer 3 Core
ISP A
Ingress Path Optimization:
Clients-Server
DC A
DC B
Server-Server
Path Optimization
Public Network
Agg
Agg
VLAN A


Access
DB
Front-End Data-Base
Egress
BRKDCT-3060
Move the whole application tier
Optimize the whole path:
 Client to Server
 Server to Server
 Server to Client
Access
L2 Links (GE or 10GE)
L3 Links (GE or 10GE)
Path Optimization: Server-Client
© 2012 Cisco and/or its affiliates. All rights reserved.
Egress Path
Server-Client
CiscoOptimization:
Public
63
Inbound Path Optimization
LISP Host Mobility
3
EID-prefix: 10.2.0.1/32
Mapping Cache
1
Locator-set:
Entry
DNS Entry:
D.abc.com A 10.2.0.1
2.1.1.1, priority: 1, weight: 50 (D1)
2.1.2.1, priority: 1, weight: 50 (D2)
10.1.0.0/24
LISP Site
2
S
ITR
1.1.1.1
IP Network
10.1.0.1 -> 10.2.0.1
5.3.3.3
Mapping DB
4
5.2.2.2
5.1.1.1
1.1.1.1 -> 2.1.1.1
10.1.0.1 -> 10.2.0.1
2.1.1.1
2.1.2.1
3.1.1.1
3.1.2.1
ETR
5
10.1.0.1 -> 10.2.0.1
BRKDCT-3060
LAN Extension
West-DC
D10.2.0.1
© 2012 Cisco and/or its affiliates. All rights reserved.
East-DC
10.2.0.0/24
Cisco Public
64
Inbound Path Optimization
LISP Host Mobility
EID-prefix: 10.2.0.1/32
Locator-set:
DNS Entry:
D.abc.com A 10.2.0.1
3.1.1.1,
2.1.1.1, priority: 1, weight: 50 (D1)
3.1.2.1,
2.1.2.1, priority: 1, weight: 50 (D2)
7
Mapping Cache
Entry Update
10.1.0.0/24
8
LISP Site
S
1.1.1.1 -> 3.1.1.1
ITR
10.1.0.1 -> 10.2.0.1
1.1.1.1
IP Network
5.3.3.3
Mapping DB
5.2.2.2
5.1.1.1
2.1.1.1
2.1.2.1
3.1.1.1
ETR
3.1.2.1
9
LAN Extension
West-DC
East-DC
10.2.0.0/24
D10.2.0.1
6
10.2.0.1
Workload Move
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
65
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
IP Based Solutions
• LISP for DCI Deployments
L3 Host Mobility using LISP
LISP as L3 DCI
• Summary and Conclusions
• Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
67
Use Case Focus for LISP as L3 DCI
• Data-Center migration
‒ Capability to install a device in a brownfield DC to ensure subnet continuity for
migration to a greenfield DC
‒ VM migration / Physical migration with no change on workload IP address
• Hybrid Cloud
‒ Insertion of SP resource in customer local subnet (like SaaS)
‒ Cloud bursting (provisioning of resource in Cloud)
‒ Migration
• Backup services
‒ Partial Disaster Recovery
‒ This require capability of moving back resource to the original site
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
Enable LISP on a stick
MS/MR
PiTR with Mobility on a stick
NOT default gateway
Does not receive any traffic before move
PxTR
10.17.0.0/24
Brownfield DC
LISP ETR (using M1-32) on a stick
Default gateway for the moved traffic
Does not receive any traffic before the move
ETR
10.17.0.0/24
Greenfield DC
This is ASM mode with same subnet value both side
Home subnet is Greenfield (registers with MS 10.17.0.0/24), Dynamic part is Brownfield (detects and registers any /32)
69
Packet Flow from Client & Server in Brownfield
North-South Traffic
Traffic to a non moved resource does not reach LISP nodes on a stick
70
Symmetric Packet Flow from Client & Server in Greenfield
North-South Traffic
PeTR
Existence of a Firewall between WAN
edge & PxTR requires
symmetrical flow
 Use PeTR
PeTR allows return flow to go thru LISP Path,
nevertheless it requires ETR to work with default routing
71
Convergence Considerations
All Failures are Leading to Sub 3s Convergence with IGP/BGP tuning
• There are three mechanisms to handle convergence
‒Route watch / Route notification
Mandate RLOC /32 to be received remotely
This /32 must not be part of an aggregated route
As fast convergence as the routing protocol
 The one used in this solution
‒EID/RLOC probing
Probes every EID
60s convergence
‒LSB bits
Data-plane bits indicating local RLOC status
Not supported with M1-32
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
72
Agenda
• DCI Business Drivers and Solutions Overview
• LAN Extension Deployment Scenarios
Ethernet Based Solutions
MPLS Based Solutions
IP Based Solutions
• LISP for DCI Deployments
L3 Host Mobility using LISP
LISP as L3 DCI
• Summary and Q&A
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
73
Data Center Interconnect - DCI Model
Connecting Virtualized Data Centers
Path Optimization
L2 Domain Elasticity





- LANisExtension
STP Isolation
the key element
Multipoint
Loop avoidance + Storm-Control
Unknown Unicast & Broadcast control
Link sturdiness
Scale & Convergence
OTV
OTV
- Optimal Routing
Considerations
 Network-and
SecurityPortability
services deployment
Route
 Server-Client Flows
 Server-Server Flows
Path Optimization Options
 Egress
 Addressed by FHRP Filtering
 Ingress:
 Addressed by LISP
OTV
VN-link
notifications
OTV
Storage Elasticity

- SAN
Extensions
Sync or Async replication
modes
are driven by the applications, hence the
distance/latency is a key component to select the choice

Localization of Active Storage is key
 Distance can be improved using IO accelerator or caching
 Virtual LUN is allowing Active/Active
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
74
Data Center Interconnect
Where to Go for More Information
http://www.cisco.com/go/dci
http://www.cisco.com/en/US/netsol/ns749/networking_solutions_sub_program_home.html
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
75
Recommended Reading for BRKDCT-3060
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
76
Call to Action
• Visit the Cisco Campus at the World of Solutions
to experience Cisco innovations in action
• Get hands-on experience attending one of the Walk-in Labs
• Schedule face to face meeting with one of Cisco’s engineers
at the Meet the Engineer center
• Discuss your project’s challenges at the Technical Solutions Clinics
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public
BRKDCT-3060
© 2012 Cisco and/or its affiliates. All rights reserved.
Cisco Public