Architecture & Solutions Group
US Public Sector Advanced Services
Mark Stinnette, CCIE Data Center #39151
Date 9 September 2013
Version 1.13.2
© 2013 Cisco and/or its affiliates. All rights reserved.
1
This Quick Start Guide (QSG) is a Cookbook style guide to Deploying Data Center
technologies with end-to-end configurations for several commonly deployed architectures.
This presentation will provide end-to-end configurations mapped directly to commonly
deployed data center architecture topologies. In this cookbook style; quick start guide;
configurations are broken down in an animated step by step process to a complete end-toend good clean configuration based on Cisco best practices and strong recommendations.
Each QSG will contain set the stage content, technology component definitions,
recommended best practices, and more importantly different scenario data center
topologies mapped directly to complete end-to-end configurations. This QSG is geared for
network engineers, network operators, and data center architects to allow them to quickly
and effectively deploy these technologies in their data center infrastructure based on
proven commonly deployed designs.
© 2013 Cisco and/or its affiliates. All rights reserved.
2
FabricPath Design :: 2 SPINE (Routing at Aggregation)
•
•
•
•
Simplest design option :: traditional Aggregation / Access designs
Simplified configuration
Removal of STP
Traffic distribution over all uplinks without vPC port-channels
•
•
•
•
Active / Active gateways (via vPC+ or Anycast HSRP)
VLAN anywhere (no trunk ports)
Option for vPC+ for legacy access switches and computer connectivity
Easily deploy L4-7 services
Natural Evolution of the vPC Design
© 2013 Cisco and/or its affiliates. All rights reserved.
3
FabricPath Design :: 4 SPINE (Routing at Aggregation w/ Anycast HSRP)
•
•
•
Scale out; n-way Active HSRP in FabricPath (up to 4 today)
No longer need vPC+ at SPINE for active/active HSRP
• No peer-link or peer-keepalive link required
Leaf software needs to understand Anycast HSRP in FabricPath
© 2013 Cisco and/or its affiliates. All rights reserved.
4
FabricPath Design :: Dedicated SPINE (Centralized Routing)
© 2013 Cisco and/or its affiliates. All rights reserved.
5
Alternative View
FabricPath Design :: Dedicated SPINE (Centralized Routing)
•
•
•
•
•
•
Paradigm shift with respect to typical designs (CLOS Fabric topology)
Simplifies SPINE design
Traditional “Aggregation” layer becomes pure FabricPath SPINE
Design helps ensure that any application node are at most only two hops away
FabricPath LEAF switches provide server connectivity like traditional designs
FabricPath LEAF switches also provide L2/L3 boundary, inter-VLAN routing, North  South routing
FabricPath Deployment in Preparation For Dynamic Fabric Automation (DFA)
© 2013 Cisco and/or its affiliates. All rights reserved.
6
FabricPath Design :: Multi POD (w/ FP Multi-Topology)
NX-OS 6.2
•
•
•
•
Provides DC wide vs. POD local VLAN segmentation / isolation
Can support VLAN ID reuse in multiple PODs
Define FabricPath VLANs :: map VLANs to topology :: map topology
to FabricPath core ports
Optional design for “disconnected” PODs
Each POD can use same non-default FP topology; don’t need
FabricPath Core since each POD is on its own island
© 2013 Cisco and/or its affiliates. All rights reserved.
•
•
Where to place DC wide L2/L3 boundary (vPC+ or Anycast HSRP)
FabricPath Core
Pick a any Aggregation POD
Routed Sub-interfaces on Routed Core / WAN Edge via CE edge ports
Default topology always includes all FabricPath core ports
Map DC Wide VLANs to default topology
POD local core ports also mapped to POD local topology
Map POD local VLANs to POD local topology
7
FabricPath Terminology
© 2013 Cisco and/or its affiliates. All rights reserved.
8
FabricPath Encapsulation
© 2013 Cisco and/or its affiliates. All rights reserved.
9
Benefits Overview
FabricPath is a next generation Layer 2 technology from Cisco that provides multi-path Ethernet capabilities in L2 switching
networks. FabricPath combines the benefits of L2 switching such as easy configuration and workload flexibility with greater
scalability and availability. Specifically, FabricPath adds to L2 switching some routing type capabilities such as all active links,
fast convergence, and loop avoidance mechanisms in the data plane. It allows Layer 2 networking without Spanning Tree
Protocol.
FabricPath provides the following benefits:
• Eliminates Spanning Tree Protocol (STP) with built-in loop prevention and mitigation (TTL & RPF)
• Single control plane for unknown unicast, unicast, broadcast, and multicast traffic
• VLAN anywhere
• FP is transparent to L3 protocols
• Easy to configure
• Easy to manage
• Flexibility
• Create arbitrary any topology
• Multiple designs to integrate L2/L3 boundaries
• Start small and expand as needed (bandwidth growth)
• Efficient and Scalable
• Layer 3 availability similar features
• Leverage parallel paths
• Expanding available bandwidth at L2/L3 Default Gateway level
• MAC address table scale (conversational learning) :: all FabricPath VLANs use conversation MAC address learning
• Fast Convergence and low latency
• Enhances mobility and virtualization in the FabricPath network
• Capable of running vPC (called vPC+) to connect devices to the edge in a port channel
• Multi-tenant support, traffic engineering, meet security separation requirements via FabricPath topologies
© 2013 Cisco and/or its affiliates. All rights reserved.
10
Feature Configuration
Feature
Benefit
Overview
fabricpath VLAN
mode & switchport
mode
Eliminate STP
protocol from the
infrastructure fabric
The FabricPath ports carry traffic only for those VLANs configured as FabricPath
VLANs. It is mandatory to enable the same FP mode VLAN EVERYWHERE on all
switches in the FP fabric (otherwise, FP multidestination trees will be incorrectly
built). VLAN pruning is performed automatically on FP core ports for FP traffic only.
fabricpath
forwarding tables
Service Continuity
FabricPath uses 3 HW forwarding tables to switch frames (1) MAC address table,
(2) Switch-ID table, (3) Multidestination tree table
fabricpath switch IDs
Service Continuity
Each switch in the FP fabric is allocated with a global switch ID value; this is
allocated automatically or manually set (recommended). The switch ID information
will be used in the MAC address-table for L2 forwarding. The vPC+ system also
uses an emulated switch ID; which you assign on both peer devices.
fabricpah IS-IS link
metric
Increase HighAvailability
FP will always take the path with the lowest metric. Its recommended to use the
default reference bandwidth.
fabricpath timers
Improve
Convergence Time
On a case by case basis, if convergence time needs to be improved upon switch
reload, modify lsp-gen-interval and spf-interval timers.
fabricpath root
priority
Service Continuity
FP uses two Multi destination Trees, Tree 1 (ftag 1) for broadcast, unknown unicast,
multicast & Tree 2 (ftag 2) –multicast. Recommend to use on SPINE switches for
primary and secondary root.
STP for Classical
Ethernet (CE)
Service Continuity
The FP fabric must be the root of the L2 domain when connected to other legacy L2
domains / switches. Make sure STP priority is the lowest for the entire FP fabric.
vPC+
Increase HighAvailability
FabricPath & vPC+ combined provides two main purposes, (1) dual attach a host to
the FP fabric, (2) Leverage Active/Active HSRP capability
© 2013 Cisco and/or its affiliates. All rights reserved.
11
Feature Configuration
Feature
Benefit
Overview
FabricPath multicast
load-balance
Service Continuity
Cisco NX-OS provides a way to control two peers to be partial designated
forwarders when both vPC paths are up. When this control is enabled, each peer
can be the designated forwarder for multi destination southbound packets for a
disjoint set of RBHs/FTAGs (depending on the hardware). The designated
forwarder is negotiated on a per-vPC basis.
There are three designated forwarder states for a vPC port:
 All—If the local vPC leg is up and the peer vPC is not configured or down, the
local switch is the designated forwarder for all RBHs/FTAGs for that vPC.
 Partial—If the vPC path is up on both sides, each peer is the designated
forwarder for half the RBHs or FTags. For the latter, the vPC port allows only the
active FTags on that peer. This mode is used in a FEX with vPC+ topology.
 None—If the local vPC path is down or not configured, the local switch does not
forward any multi destination packets from this vPC path.
The fabricpath multicast load-balance command is required for configuring
vPC+ with FEX ports.
© 2013 Cisco and/or its affiliates. All rights reserved.
12
Feature Configuration
Feature
Benefit
Overview
Overload Bit
Improve
Convergence Time
RFC 3277 based Overload bit is advertised in updates to prevent a corner case
when a single switch restarts causing temporary loops or traffic black holing. This
feature also prevents neighbors from using a switch as a transit during initial
convergence as well as lowering impact insertion or removal of a switch to the FP
domain.
Multiple Topologies
Design Separation
With multiple topologies, we can create up to 16 topologies where a subset of
VLANs are mapped to a particular topology; allowing more design possibilities.
Anycast HSRP
Increase HighAvailability
Provides up to 4 active Default Gateways for the network which lowers the risk of
disruption for routed and Inter-VLAN traffic and provides bandwidth capacity at
L2/L3 boundaries. The Anycast HSRP feature removes the reliance on vPC+ to
provide the Active/Active HSRP feature at the L2/L3 boundary.
fabricpath static
routes
Traffic Engineering
The static route feature gives users capabilities to enter routes directly in the
forwarding tables, ensuring predictable operation of the network.
Certain uses cases where users want to override the routes computed by IS-IS.
Users might want to route traffic to a particular switch using a particular link, better
load balancing or routing traffic through a firewall (policing) in the network.
© 2013 Cisco and/or its affiliates. All rights reserved.
13
Install license bootflash:///enchanced_layer2_pkg.lic
show license usage
Initial Baseline (Only 4 Commands !!)
feature lacp
install feature-set fabricpath
feature-set fabricpath
feature lacp
install feature-set fabricpath
feature-set fabricpath
Default / Admin
VDC Only
vlan 1 – 200
mode fabricpath
vlan 1 – 200
mode fabricpath
interface po2
switchport mode fabricpath
interface po2
switchport mode fabricpath
interface e3/1, e4/1
channel-group 2 mode active
interface e3/1, e4/1
channel-group 2 mode active
interface e5/1, e5/2
switchport mode fabricpath
interface e5/1, e5/2
switchport mode fabricpath
feature lacp
install feature-set fabricpath
feature-set fabricpath
feature lacp
install feature-set fabricpath
feature-set fabricpath
vlan 1 – 200
mode fabricpath
vlan 1 – 200
mode fabricpath
interface po2
switchport mode fabricpath
interface e1/1, e1/2
channel-group 2 mode active
interface e1/3, e1/4
switchport mode fabricpath
© 2013 Cisco and/or its affiliates. All rights reserved.
Step 1 :: install | validate Enhanced L2 License
Step 2 :: install FabricPath
Step 3 :: enable FabricPath
Step 4 :: configure FabricPath VLANs
Step 5 :: configure FabricPath core ports
Beginning with the Cisco NX-OS Release 5.1 and when you
use an F Series modules and NX-OS Release 5.1(3) N1(1)
with 5500 you can use the FabricPath feature
Default / Admin
VDC Only
interface po2
switchport mode fabricpath
interface e1/1, e1/2
channel-group 2 mode active
interface e1/3, e1/4
switchport mode fabricpath
14
Manually Set the FabricPath Switch-ID & Root
fabricpath switch-id 10
fabricpath switch-id 11
Root for FTAG 1
fabricpath domain default
root-priority 255
Root for FTAG 2
SW 10
SW 11
fabricpath switch-id 100
fabricpath switch-id 101
SW 100
SW 101
Each peer devices will have a unique global switch
ID value – make the FP network more deterministic
Suggested switch ID scheme:
SPINE :: 2 digit ID
LEAF :: 3 digit ID
Emulated Switch (vPC+) :: 4 digit ID
Step 1 :: set the FP Switch-ID
Step 2 :: set the FP Root
Multi destination Tree 1 (ftag 1) – broadcast,
unknown unicast, multicast
Multi destination Tree 2 (ftag 2) –multicast
Recommend to use on SPINE switches
Higher Number the better !!
F2/F2E uses both trees for UU/Bcast/Mcast
F1 uses MDT 2 for Mcast only
© 2013 Cisco and/or its affiliates. All rights reserved.
fabricpath domain default
root-priority 254
(start at 255 and go backwards)
-or(start at 200 in case you need to introduce another
MDT at a later time; ie expanded SPINE x 4)
15
Manually Set the Spanning-Tree :: Single Virtual Root Bridge)
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
optional
vlan 1 – 200
mode fabricpath
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
The entire FabricPath domain will look like one
virtual bridge to the CE domain – set best (lowest)
STP root priority on the vPC+ peers (recommended
at least at the access edge leaf switches); just make
sure the priority is lower than anything else in the
network (classical Ethernet)
optional
vlan 20, 40
spanning-tree vlan 20, 40 priority 8192
FP will use the same bridge ID c84c.75fa.6000
The root and sender bridge MAC addresses of this
pseudo-information are the same on every switch in the
Cisco FabricPath domain
All ports at the edge of a Cisco FabricPath network are
configured with the equivalent of root guard (don’t need to
configure this feature), a feature that would block a port
should it receive superior Spanning Tree Protocol BPDUs
© 2013 Cisco and/or its affiliates. All rights reserved.
Step 1 :: set FP domain to be root bridge
Note that the spanning-tree priority command
would also work; however, it would change the
priority for the spanning tree regardless of whether
the switch were sending regular BPDUs (when
Cisco FabricPath is not running) or sending BPDUs
with the pseudo-information (when Cisco
FabricPath is operational on the switch). In some
scenarios, this change can have undesirable side
effects.
16
Tune Timers for Fast Convergence
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
fabricpath timers linkup-delay 60
fabricpath timers linkup-delay 60
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
fabricpath timers linkup-delay 60
fabricpath timers linkup-delay 60
Problem Set: The IS-IS adjacency is established and
the access-edge started sending traffic to aggregationedge, but the control plane was not ready to forward the
traffic to the next hop. The default spf and lsp-gen
intervals are 8sec (default) and it attributes to the long
convergence. To address this issue, the default spf and
lsp-gen intervals of {max-wait, initial-wait, second-wait}
are brought down to 50msec, with this configuration, the
aggregation-edge restoration will yield sub-second
convergence for Layer 2 traffic
Note: Future enhancements such as Layer 2 IS-IS
overload bit support in 6.2 will help to improve unicast
and multicast convergence during FabricPath node
failure scenarios when default IS-IS timers are used.
Step 1 :: tune the IS-IS timers in FabricPath
Step 2 :: (optional) tune the FabricPath linkup-delay
To achieve fast convergence during node failures and recovery scenarios, it is
recommended to tune the IS-IS timers in Cisco FabricPath. This tuning is
particularly important when a switch is inserted in the topology.
This configuration is recommended for all switches in the network
© 2013 Cisco and/or its affiliates. All rights reserved.
Optional, to provide better network convergence
upon a Cisco FabricPath switch restart, you should
set a Cisco FabricPath linkup-delay timer to 60
17
Enable vPC+ :: Dual Attachment & Active/Active HSRP
feature vpc
feature vpc
vpc domain 1
role priority 1
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath multicast load-balance
fabricpath switch-id 1000
dual-active exclude interface vlan 20
vPC+
SW 1000
interface po2
switchport mode fabricpath
vpc peer-link
interface po2
switchport mode fabricpath
vpc peer-link
With vPC+, a FabricPath switch is emulated between
the CE and FabricPath domain. All packets originating
behind the Emulated Switch will be marked with the
source Switch ID of the emulated switch
vPC+ is an extension of vPC for FabricPath. It allows
dual-homed connections from Classical Ethernet
(CE) switches and hosts capable of port channels. It
also provides for active-active HSRP.
Assign the same emulated switch ID on both vPC peers;
but the emulated switch ID must be unique between
different vPC domains
The configuration of peer-link and peer-keepalive
links are required – as traditional vPC
Enable IP ARP Synchronization of ARP entries
between vPC Peers improves convergence for
North-South and East-West Layer 3 traffic when one
of the vPC+ peers is brought back up
Note: Since FabricPath does not rely on Spanning
Tree Protocol, and the vPC+ peer link is a FabricPath
Core port, so the peer-switch command is not
needed under the vpc domain [x] configuration
© 2013 Cisco and/or its affiliates. All rights reserved.
vpc domain 1
role priority 2
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath multicast load-balance
fabricpath switch-id 1000
dual-active exclude interface vlan 20
Step 1 :: enable vPC+
Step 2 :: set the emulated switch-id
Step 3 :: enable dual-active exclude for vPC SVIs
In a vPC environment, the Secondary vPC switch will
bring down the SVIs by default when the peer-link is
brought down. This behavior is fine in CE environment
as the vPC legs are also brought down on the
secondary vPC switch. However in the vPC+
environment, the down links to the Access-Edge
switches are FabricPath Core ports; in the absence of
the vPC+ peer-link, the SVIs can still communicate
through the FabricPath core ports.
The vPC dual-active exclude vlan command helps to
configure a VLAN list such that the SVI can continue to
stay up on the secondary vPC switch even if the vPC+
peer-link is down.
18
Note: In a FabricPath vPC+ environment both HSRP
peers are actively forwarding, there is no need to
configure preemption, different priorities, and fast hello
timers.
Enable vPC+ :: Active/Active HSRP @ SPINE (Full Configuration)
feature interface-vlan
feature hsrp
feature lacp
feature vpc
vPC+
vlan 1 – 200
mode fabricpath
SW 1000
feature interface-vlan
feature hsrp
feature lacp
feature vpc
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
------------------------
------------------------
vpc domain 1
role priority 1
system-priority 4096
peer-keepalive destination [….] source [….]
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
fabricpath multicast load-balance
fabricpath switch-id 1000
dual-active exclude interface vlan 20
vpc domain 1
role priority 2
system-priority 4096
peer-keepalive destination [….] source [….]
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
fabricpath multicast load-balance
fabricpath switch-id 1000
dual-active exclude interface vlan 20
interface po2
switchport mode fabricpath
vpc peer-link
interface e3/1, e4/1
channel-group 2 mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
Step 1 :: enable vPC+
Step 2 :: set the emulated switch-id
Step 3 :: enable dual-active exclude for vPC+ SVIs
interface vlan 20
ip address 20.20.20.5/24
no ip redirect
hsrp 20
ip 20.20.20.254
interface vlan 20
ip address 20.20.20.6/24
no ip redirect
hsrp 20
ip 20.20.20.254
interface po2
switchport mode fabricpath
vpc peer-link
interface e3/1, e4/1
channel-group 2 mode active
19
Enable vPC+ :: Dual Attachment @ LEAF
feature lacp
feature vpc
vlan 1 – 200
mode fabricpath
vPC+
SW 1000
feature lacp
feature vpc
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
vpc domain 10
role priority 1
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath multicast load-balance
fabricpath switch-id 1001
vpc domain 10
role priority 2
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath multicast load-balance
fabricpath switch-id 1001
vPC+
1000
SW 1001
interface po2
switchport mode fabricpath
vpc peer-link
interface e1/1, e1/2
channel-group 2 mode active
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 20 – 40
vpc 20
interface e1/5
channel-group 20 force mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
interface po2
switchport mode fabricpath
vpc peer-link
vPC 20
Step 1 :: enable vPC+
Step 2 :: set the emulated switch-id
Step 3 :: add devices redundantly with vPC+
VLANs carried on vPC+ member ports must be
FabricPath mode VLANs
interface e1/1, e1/2
channel-group 2 mode active
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 20 – 40
vpc 20
interface e1/5
channel-group 20 force mode active
20
FabricPath Authentication
interface port-channel2
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FP-KEYS
interface port-channel2
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FP-KEYS
fabricpath domain default
authentication-type md5
authentication key-chain FP-KEYS
fabricpath domain default
authentication-type md5
authentication key-chain FP-KEYS
key chain FP-KEYS
key 0
key-string Cisc0!
accept-lifetime 00:00:00 Sep 1 2012 infinite
send-lifetime 00:00:00 Sep 1 2012 infinite
key chain FP-KEYS
key 0
key-string Cisc0!
accept-lifetime 00:00:00 Sep 1 2012 infinite
send-lifetime 00:00:00 Sep 1 2012 infinite
global level authentication ::
authenticates and controls the FP LSPs and PSNPs
interfaces level authentication ::
authenticates the HELLO; the FP ISIS adjacency
FabricPath provides 2 levels of authentication
1. Authentication at the interfaces level
2. Authentication at the global level
The Key chain is used in both forms of authentication
Supported combinations:
Step 1 :: configure the key chain
Step 2 :: configure global FabricPath authentication
Step 3 :: configure FabricPath core port authentication
You can configure the accept lifetime and send lifetime for a key. By default, accept
and send lifetimes for a key are infinite, which means that the key is always valid.
accept-lifetime [local] start-time duration duration-value | infinite | end-time]
send-lifetime [local] start-time duration duration-value | infinite | end-time]
© 2013 Cisco and/or its affiliates. All rights reserved.
21
NX-OS 6.2(2)
HSRP
CP :: Control Plane
DP :: Data Plane
AnyCast HSRP
feature-set fabricpath
feature interface-vlan
feature hsrp
Step 1 ::
Step 2 ::
Step 3 ::
Step 4 ::
Step 5 ::
Step 6 ::
Step 7 ::
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
fabricpath switch-id 10
fabricpath domain default
root-priority 255
hsrp anycast 100 ipv4
switch-id 1000
vlan 20
priority 110
SW 10
feature-set fabricpath
feature interface-vlan
feature hsrp
enable required features
configure SVI
configure hsrp
configure anycast bundle
associate anycast switch id
associate a set vlans
designate active HSRP router
Active
HSRP CP
Active
HSRP DP
HSRP DP
Virtual FP-ID 1000
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
SW 11
fabricpath switch-id 11
fabricpath domain default
root-priority 254
hsrp anycast 100 ipv4
switch-id 1000
vlan 20
interface vlan20
ip address 20.20.20.2/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
no ip redirect
hsrp version 2
hsrp 100
ip 20.20.20.1
interface vlan20
ip address 20.20.20.3/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
no ip redirect
hsrp version 2
hsrp 100
ip 20.20.20.1
interface e5/1, e5/2
switchport mode fabricpath
interface e5/1, e5/2
switchport mode fabricpath
SW 100





You don’t need to enable vPC+ to achieve active/active HSRP
No vPC domain configuration required
No peer-link required
The FabricPath feature-set has to be enabled before configuring HSRP anycast
4 gateways are supported in an HSRP Anycast bundle for a common VLAN
HSRPv2 is required (IPv4/IPv6 address-families supported)
An Anycast bundle can reference multiple VLANs
© 2013 Cisco and/or its affiliates. All rights reserved.
SW 101



Downstream switches use the virtual FP-ID to equal cost route traffic destined to
all HSRP anycast devices; the active HSRP router advertises the anycast switch
ID as the source switch ID in FabricPath IS-IS
All Leaf devices need to support and be aware of the Anycast functionality
Nexus 7000 :: NX-OS 6.2(2) and later releases
Nexus 5500 & 6000 :: NX-OS 6.0(2)N2(1) and later releases
Nexus 5500 & 6000 can support Anycast HSRP Gateway functionality in
6.0(2)N3(1) and later releases
22
NX-OS 6.2(2)
HSRP CP
HSRP DP
HSRP DP
HSRP DP
HSRP DP
SW 10
SW 11
SW 12
SW 13
AnyCast HSRP :: 4 SPINE
Common
Configuration
feature-set fabricpath
feature interface-vlan
feature hsrp
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
Anycast HSRP Capable::
A FP switch can work as an Anycast HSRP Router / Gateway
Anycast HSRP Aware::
Same as "Anycast HSRP Leaf".
Can send traffic to multiple Anycast HSRP capable switches
Can recognize Anycast TLV sent from Anycast HSRP capable switches
Virtual FP-ID 1000
N7K is Anycast HSRP Capable & Aware :: 6.2(2)
N6K/N5K is Anycast HSRP Aware :: 6.0(2)N2(1)
----------------N6K/N5K is Anycast HSRP Capable & Aware :: 6.0(2)N3(1)
interface e5/1, e5/2
switchport mode fabricpath
SW 100
SPINE 7k-1
SPINE 7k-2
SPINE 7k-3
SPINE 7k-4
fabricpath switch-id 10
fabricpath switch-id 11
fabricpath switch-id 12
fabricpath switch-id 13
fabricpath domain default
root-priority 255
fabricpath domain default
root-priority 254
fabricpath domain default
root-priority 253
fabricpath domain default
root-priority 252
hsrp anycast 100 ipv4
switch-id 1000
vlan 20, 100-120
priority 110
hsrp anycast 100 ipv4
switch-id 1000
vlan 20, 100-120
hsrp anycast 100 ipv4
switch-id 1000
vlan 20, 100-120
hsrp anycast 100 ipv4
switch-id 1000
vlan 20, 100-120
interface vlan20
ip address 20.20.20.1/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
no ip redirect
hsrp version 2
hsrp 100
ip 20.20.20.254
interface vlan20
ip address 20.20.20.2/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
no ip redirect
hsrp version 2
hsrp 100
ip 20.20.20.254
interface vlan20
ip address 20.20.20.3/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
no ip redirect
hsrp version 2
hsrp 100
ip 20.20.20.254
interface vlan20
ip address 20.20.20.4/24
ip router ospf 1 area 0.0.0.0
ip ospf passive-interface
no ip redirect
hsrp version 2
hsrp 100
ip 20.20.20.254
© 2013 Cisco and/or its affiliates. All rights reserved.
SW 101
HSRP
CP :: Control Plane
DP :: Data Plane
23
NX-OS 6.2(2)
Optional :: Overload Bit & VLAN Pruning
SW 10
Common
Configuration
feature-set fabricpath
feature interface-vlan
feature hsrp
vlan 1 – 200
mode fabricpath
interface e5/1, e5/2
switchport mode fabricpath
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
Overload bit :: You can configure the overload bit for FabricPath ISIS. You achieve consistent routing behavior in conditions where a
node reboots or gets overloaded.
always—The overload bit is always on; out of service.
on-startup—The overload bit is set upon system startup and
remains set for the specified number of seconds.
SW 11
VLAN pruning :: The switch will only attract data traffic for the VLANs
that have active Classic Ethernet (CE) ports on an F1 Series module,
F2 Series module, or switch virtual interfaces (SVIs) for those
VLANs. Optional command and is only mentioned for informational
purposes only; use appropriately.
SPINE 7k-2
SPINE 7k-3
SW 101
SPINE 7k-4
fabricpath switch-id 10
fabricpath switch-id 11
fabricpath switch-id 12
fabricpath switch-id 13
fabricpath domain default
root-priority 255
fabricpath domain default
root-priority 254
fabricpath domain default
root-priority 253
fabricpath domain default
root-priority 252
fabricpath domain default
set-overload-bit on-startup [sec]
vlan pruning enabled
fabricpath domain default
set-overload-bit on-startup [sec]
vlan pruning enabled
fabricpath domain default
set-overload-bit on-startup [sec]
vlan pruning enabled
fabricpath domain default
set-overload-bit on-startup [sec]
vlan pruning enabled
© 2013 Cisco and/or its affiliates. All rights reserved.
SW 13
Virtual FP-ID 1000
SW 100
SPINE 7k-1
SW 12
24
NX-OS 6.2(2)
FabricPath Static Routes :: Traffic Engineering
Common
Configuration
feature-set fabricpath
feature interface-vlan
feature hsrp
vlan 1 – 200
mode fabricpath
FabricPath uses Layer 2 Integrated Intermediate System-to-System
(IS-IS) as a link state protocol to compute unicast topologies. You
can configure unicast static routes in the forwarding tables to ensure
a predictable operation of the network or to override the routes
computed by dynamic protocols such as IS-IS in FabricPath. For
example, you might want to route traffic to a particular device using a
specific link to ensure better load balancing or to route traffic through
a firewall in the network.
interface e5/1, e5/2
switchport mode fabricpath
SW 10
SW 11
SW 12
SW 13
Virtual FP-ID 1000
SPINE 7k-1
fabricpath switch-id 10
fabricpath domain default
root-priority 255
interface e5/3
switchport mode fabricpath
fabricpath route switch-id 100 e5/3
-----------------------fabricpath topology 1
fabricpath route switch-id 100 e5/4
© 2013 Cisco and/or its affiliates. All rights reserved.
SW 100
SW 101
25
NX-OS 6.2(2)
Multiple Topologies
SW 1
SW 2
Virtual FP-ID 1000
SPINE 7k-1
SPINE 7k-2
SPINE 7k-5
feature-set fabricpath
feature interface-vlan
feature hsrp
feature-set fabricpath
feature interface-vlan
feature hsrp
feature-set fabricpath
feature interface-vlan
feature hsrp
SW 10
vlan 10 – 20, 50 – 60, 100 – 200
mode fabricpath
vlan 10 – 20, 50 – 60, 100 – 200
mode fabricpath
vlan 100 – 200
mode fabricpath
Virtual FP-ID 1100
fabricpath switch-id 10
fabricpath switch-id 11
fabricpath switch-id 1
fabricpath domain default
topology 1
root-priority 255
topology 2
root-primary 255
fabricpath domain default
topology 1
root-priority 254
topology 2
root-primary 254
fabricpath domain default
root-priority 255
fabricpath topology 1
member vlan 10 – 20
fabricpath topology 1
member vlan 10 – 20
fabricpath topology 2
member vlan 50 – 60
fabricpath topology 2
member vlan 50 – 60
interface e5/1, e5/2
switchport mode fabricpath
interface e5/1, e5/2
switchport mode fabricpath
interface e6/1
fabricpath topology-member 1
switchport mode fabricpath
interface e6/1
fabricpath topology-member 1
switchport mode fabricpath
interface e6/2
fabricpath topology-member 2
switchport mode fabricpath
interface e6/2
fabricpath topology-member 2
switchport mode fabricpath
SW 13
Virtual FP-ID 1101
SW 101
SW 102
interface e5/1, e5/2, e5/3, e5/4
switchport mode fabricpath
SPINE 7k-6
feature-set fabricpath
feature interface-vlan
feature hsrp
vlan 100 – 200
mode fabricpath
The FabricPath domain can now consist of multiple logical
topologies. By default all VLANs belong to Topology 0, which
consisted of a single pair of multi-destination trees. Now with
NX-OS 6.2(2) a total of 16 FabricPath topologies can be
configured and assigned to a FabricPath domain; allowing
VLANs to be assigned to different topologies.
fabricpath switch-id 2
fabricpath domain default
root-priority 255
interface e5/1, e5/2, e5/3, e5/4
switchport mode fabricpath
© 2013 Cisco and/or its affiliates. All rights reserved.
SW 100
SW 12
SW 11




All VLANs by default belong to topology 0
(VLANs anywhere)
A VLAN can only belong to a one topology
All interfaces by default belong to topology 0
An interface can belong to multiple topologies
26
NX-OS 6.2(2)
Multiple Topologies
SW 1
SW 2
Virtual FP-ID 1000
SPINE 7k-3
SPINE 7k-4
feature-set fabricpath
feature interface-vlan
feature hsrp
feature-set fabricpath
feature interface-vlan
feature hsrp
vlan 80 – 90, 100 – 200
mode fabricpath
vlan 80 – 90, 100 – 200
mode fabricpath
fabricpath switch-id 12
fabricpath switch-id 13
fabricpath domain default
topology 1
root-priority 255
fabricpath domain default
topology 1
root-priority 254
fabricpath topology 1
member vlan 80 – 90
fabricpath topology 1
member vlan 80 – 90
interface e5/1, e5/2
switchport mode fabricpath
interface e5/1, e5/2
switchport mode fabricpath
interface e6/1
fabricpath topology-member 1
switchport mode fabricpath
interface e6/1
fabricpath topology-member 1
switchport mode fabricpath
LEAF 5k-3
feature-set fabricpath
SW 10
vlan 80 – 90, 100 – 200
mode fabricpath
SW 13
Virtual FP-ID 1101
Virtual FP-ID 1100
fabricpath switch-id 102
fabricpath topology 1
member vlan 80 – 90
interface e1/3, e1/4
fabricpath topology 1
switchport mode fabricpath
SW 100
SW 101
SW 102
spanning-tree pseudo-information
vlan 80 – 90, 100 – 200 root priority 0
Be aware of the FabricPath
topology scaling limits across
the different NX-OS versions
on the Nexus 5000 & 6000
switching platforms.
The FabricPath domain can now consist of multiple logical
topologies. By default all VLANs belong to Topology 0, which
consisted of a single pair of multi-destination trees. Now with
NX-OS 6.2(2) a total of 16 FabricPath topologies can be
configured and assigned to a FabricPath domain; allowing
VLANs to be assigned to different topologies.




© 2013 Cisco and/or its affiliates. All rights reserved.
SW 12
SW 11
All VLANs by default belong to topology 0
(VLANs anywhere)
A VLAN can only belong to a one topology
All interfaces by default belong to topology 0
An interface can belong to multiple topologies
27
FabricPath is Easy & Simple !!
vPC
interface e1/5
ip address 192.168.1.1/24
vrf membership vpc-keepalive
vpc domain 1
peer-keepalive destination 192.168.1.2
source 192.168.1.1 vrf vpc-keepalive
interface port-channel 1000
switchport mode trunk
vpc peer-link
interface e1/1-2
switchport mode trunk
channel-group 1000 mode active
interface e1/3
switchport mode trunk
channel-group 1 mode active
interface port-channel1
vpc 1
interface e2/5
ip address 192.168.1.2/24
vrf membership vpc-keepalive
vpc domain 1
peer-keepalive destination 192.168.1.1
source 192.168.1.2 vrf vpc-keepalive
interface port-channel 1000
switchport mode trunk
vpc peer-link
interface e2/1-2
switchport mode trunk
channel-group 1000 mode active
interface e2/3
switchport mode trunk
channel-group 1 mode active
interface port-channel1
vpc 1
FabricPath + vPC+
Advantages
Active/active path at L2
Active/active for HSRP
Works with all LC
Advantages
Active/active path at L2
Active/active for HSRP
Ease of configuration
No more STP
Extensibility
Drawbacks
Need dedicated infrastructure (PL,
PKL)
Configuration on both peer devices
Consistency check to care about
STP still here (but runs as fail safe
mechanism)
Drawbacks
Need dedicated infrastructure (PL,
PKL)
Need F1 (+M1) or F2
interface e1/1-3
switchport mode fabricpath
e1/5
interface e2/1-3
switchport mode fabricpath
e2/5
e1/1-2
e2/1-2
e1/3
e2/3
VPC
e3/1-2
interface e3/1-2
switchport mode trunk
channel-group 1 mode passive
VPC Configuration
© 2013 Cisco and/or its affiliates. All rights reserved.
e1/1-2
e1/3
e2/1-2
FabricPath
e2/3
e3/1-2
interface e3/1-2
switchport mode fabricpath
FabricPath Configuration
28
Common Design Migration Starting Point
7k – Aggregation
5k/2k – Access Pods
Dual Layer vPC
Mix F1 / M1 line cards
After Migration Completion
7k – SPINE role
5k – LEAF role
vPC converted to FabricPath core ports
Peer-Link also FP core port = vPC+
(only F1/F2 support FabricPath)
Additional Reading Here :: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-709336.html
© 2013 Cisco and/or its affiliates. All rights reserved.
29
Strong Recommendations and Key Notes
• FabricPath VLANs must be configured on all switches in the FP domain
• It is recommended to configure the switch ID manually on all FabricPath switches
• For Active-Active HSRP capability, it is recommended to configure vPC+ on the Aggregation-Edge switches even if there
are no vPC legs. Note: subject to vPC rules; so no dynamic routing over vPC to firewalls, Core layer, WAN edge
• The Nexus 6000 will supports a routing protocol over vPC+ with the 6.0(2)N2(1) release
• Implement Layer 3 routing backup path
Separate L3 port channel; point-to-point links
Separate L2 port channel; use dedicated VLAN in Classical Ethernet (CE) mode as transit VLAN inside this L2 trunk
• Disable IP redirects on SVIs and configure passive interface to avoid any routing adjacency over SVIs
• ARP sync feature with vPC+ is recommended for improved traffic convergence during Aggregation-Edge failure and
restoration
• It is recommended to configure highest and second highest MDT root priority on the Aggregation-Edge switches
• Have option of choosing single links or port-channels between Aggregation-Edge and Access-Edge for ECMP. If port
channels are used, configuring IS-IS metric is preferred. With path costing, member link failure is transparent to IS-IS
protocol so that the traffic would continue to use the same path
© 2013 Cisco and/or its affiliates. All rights reserved.
30
Strong Recommendations and Key Notes
• It is recommended to have lowest path cost for the links between Aggregation devices so the multicast hello packets
always take the peer-link which is direct link between the AGG devices
• It is recommended to tune Layer 2 IS-IS SPF and LSP generation timers to achieve better convergence during failure and
restoration scenarios. These timers should be tuned to 50 msec with 50msec initial wait and second wait. This is a
requirement until the overload bit support is available with Layer 2 IS-IS
• Use default reference BW (its 400 Gbps default)
fabricpath domain default
reference-bandwidth ?
• IS-IS metric cost (1Gb = cost 400, 10Gb = cost 40, 20Gb = cost 20)
• IS-IS link metric for port-channel depends on NX-OS version
Up to NX-OS 6.0: IS-IS metric for port-channel is calculated based on number of configured member ports; meaning
you may need to use LACP min-link feature to tear down port-channel if number of active member ports goes below a
specific limit
Since NX-OS 6.1: IS-IS metric for port-channel is calculated based on number of active ports
• Dual-active exclude VLAN configuration is recommended so that the SVIs can continue to be active on the secondary
vPC+ peer in the event of peer-link failure. This also helps to stay with default HSRP timers there by reducing control plane
load associated with aggressive HSRP timers
• Do not use dual-active exclude command for VLANs if you have vPC attached devices, for example at the access (leaf)
© 2013 Cisco and/or its affiliates. All rights reserved.
31
Strong Recommendations and Key Notes
• In typical vPC deployments it is not necessary to tune the HSRP hello timers from the defaults (3/10s). In a Classic
Ethernet environment, in which in a single VLAN only one HSRP gateway can be active at a time, fast failover between the
remaining peers is essential. Typically, fast hello timers and preemption is configured to enforce the required behavior. But
in a Cisco FabricPath vPC+ environment, both HSRP peers are actively forwarding, fast hello timers and preemption is no
longer required, and configurations can be left at their defaults.
• In CE-FabricPath hybrid networks, it is recommended to configure the lowest Spanning-tree root priority on all FabricPath
Edge switches
• The MAC timer should be consistent on all devices in the Layer 2 topology. The MAC and ARP aging timers can be left at
defaults, 1800sec & 1500sec respectively
• The M1/F1 mixed VDC currently supports up to 16K MAC/ARP entries. This limitation will be lifted with the Layer 2 proxy
learning feature in the upcoming NX-OS release
• The M1, M1-XL, M2 & F2E in a mixed VDC topology; meaning when F2E is placed in a chassis with M-series it will
operate in Layer 2 mode only leveraging the M for Layer 3 (proxy L3 forwarding); this will enable 128K MAC/ARP scale.
• If an ASA cluster is attached to the Nexus 7000 series Aggregation-edge switches, source-dest-ip or src-dst ip-l4port is the
recommended load balance algorithm if the ASA cluster is in single context mode or if the VLANs are fewer in multi-context
mode. This is to prevent traffic polarization on links towards ASA cluster member
© 2013 Cisco and/or its affiliates. All rights reserved.
32
Strong Recommendations and Key Notes
• Better use port-channel instead of individual links for the 2 following reasons
Decrease the number of direct IS-IS adjacency (1 for the whole port-channel instead of X IS-IS adjacencies if X
individual links are used between the 2 switches)
Allows to use the whole port-channel capacity for multidestination tree #1 or #2 (if multiple parallel individual links exist
between 2 switches, only 1 link will be selected for tree #1 and potentially 1 another link for tree #2)
• ECMP vs. Port Channel
Can use ECMP, port-channel, or both simultaneously
Port-channels have one main advantage over ECMP – treated as single logical link in FabricPath IS-IS. Individual link
failure invisible to upper layer protocols. Also allows more bandwidth for branches of Multidestination trees
With 4 member port channel, whole interface becomes single branch of tree with 40G BW
With 4 parallel ECMP paths, only one of the 4 interfaces becomes part of the tree
ECMP with port-channel : 2 levels of load-balancing decision :
First level : FP Core Link selection (based on L3/L4 fields by default)
Second level : Port-Channel member selection (based on src-dst ip by default)
© 2013 Cisco and/or its affiliates. All rights reserved.
33
Strong Recommendations and Key Notes
• Do not use UDLD with FabricPath
UDLD (normal or aggressive) does not bring any benefits on single physical link and port channels with FP enabled
(for port channel, activate LACP instead of relying on UDLD to detect member port issues)
Physical link level protection and the bi-directional IS-IS hellos should take care of all (or near all) potentially link level
issue
• HSRP preemption does not add any value but may hurt at large VLAN scale, when you need to maintain HSRP adjacency
for each of the VLANs. Control plane will just be burning cycles with no efficient and positive impact on data path. Consider
not using HSRP preemption in the FabricPath design.
• FabricPath and Jumbo MTU Interoperability with N5k/N6k and N7k; its recommended to disable ISIS hello padding on N7k
with the “no fabricpath isis hello-padding always” command when jumbo MTU is enabled
© 2013 Cisco and/or its affiliates. All rights reserved.
34
Building FabricPath Routing Tables :: Control Plane Operation
FabricPath Routing
Table on S10
FabricPath Routing
Table on S11
Switch
IF
Switch
IF
S10
-
S10
L2,L4,L6
S11
L1,L3,L5
S11
-
S100
L1
S100
L2
S101
L3
S101
L4
S140
L5
S140
L6
FabricPath Routing
Table on S100
FabricPath Routing
Table on S140
Switch
IF
Switch
IF
S10
L1
S10
L5
S11
L2
S11
L6
S100
-
S100
L5,L6
S101
L1,L2
S101
L5,L6
S140
L1,L2
S140
-
Step 1 :: Enable FabricPath on desired interfaces
Step 2 :: L2 IS-IS hello are sent out on all FabricPath Ports
Step 3 :: Establish L2 IS-IS Adjacency
Step 4 :: Send L2 IS-IS updates to exchange local link-states
Step 5 :: All FabricPath switches calculate unicast paths to all other switches in
the L2 fabric and create the ‘FabricPath Routing Table’ based on the results
© 2013 Cisco and/or its affiliates. All rights reserved.
 Forwarding path selection based on destination Switch-ID
 Switch Table basically contains (Switch-ID, Output Interface)
 Up to 16 ‘Next-Hop’ Interfaces (ECMP) per Switch-ID
35
FTAG/tree 1 handles
unknown unicast, broadcast
and some multicast
FabricPath Forwarding :: Broadcast (ARP Request)
Multidestination
Trees on S10
Tree
IF
1
L1,L3,L5
2
Root for Tree 1
Root for Tree 2
decap
encap
DSID→FF
Ftag→1
DSID→FF
Ftag→1
L5
SSID→100
SSID→100
ftag
FTAG/tree 2 handles
multicast only
DMAC→FF
DMAC→FF
SMAC→A
SMAC→A
Payload
Multidestination
Trees on S100
Tree
IF
1
L1,L2
2
L2
ftag
Payload
Multidestination
Trees on S140
Tree
IF
SMAC→A
1
L5
DMAC→FF
2
L5,L6
Payload
DMAC→FF
Broadcast
SMAC→A
Payload
FabricPath MAC Table
on S100
Switch
IF
A
e1/1 (local)
© 2013 Cisco and/or its affiliates. All rights reserved.
Step 1 :: Host A communicates to Host B for the first time – Sends ARP request to B
Step 2 :: S100 adds A into MAC table as the result of new source learning on CE port
Step 3 :: Since destination MAC is all ‘F’; S100 floods this frame out all CE ports
[Learn MACs of directly-connected devices unconditionally]
Step 4 :: Meanwhile, S100 selects ‘Tree 1’, marks this in the FabricPath header and
floods this frame out all FabricPath ports (L1, L2) that are part of Tree 1
Step 5 :: S10 floods this frame further, out (L3, L5) based on local info about Tree 1
Step 6 :: S101 and S140 remove the FabricPath header and flood the frame out all
local CE ports.
FabricPath MAC Table
on S140
Switch
IF
Don’t Learn Remote MAC
since DMAC is unknown / is a
Flooded Frame
36
FTAG/tree 1 handles
unknown unicast, broadcast
and some multicast
FabricPath Forwarding :: Unknown Unicast (ARP Reply)
Multidestination
Trees on S10
Tree
IF
1
L1,L3,L5
2
Root for Tree 1
Root for Tree 2
encap
decap
DSID→MC1
Ftag→1
DSID→MC1
Ftag→1
L5
SSID→140
SSID→140
ftag
FTAG/tree 2 handles
multicast only
DMAC→A
DMAC→A
SMAC→B
SMAC→B
Payload
Multidestination
Trees on S100
Tree
IF
1
L1,L2
2
L2
Unknown
Payload
ftag
FabricPath MAC Table
on S100
Switch
IF
A
e1/1 (local)
B
S140 (remote)
If DMAC is Known then Learn
Remote MAC
© 2013 Cisco and/or its affiliates. All rights reserved.
Multidestination
Trees on S140
Payload
DMAC→A
SMAC→B
SMAC→B
DMAC→A
Payload
Step 1 :: Host B sends ARP Reply back to Host A
Step 2 :: S140 adds B into the MAC Table from source learning on CE port
A 
Step 3 :: Since A is unknown, S140 floods the frame out all CE ports
Step 4 :: Meanwhile, S140 selects Tree 1, marks this in the FabricPath header and
floods this frame out all FabricPath ports (L5) that are part of Tree 1
Step 5 :: S10 floods this frame further (L1, L3) along Tree 1
Step 6 :: S100 floods this frame further (L2) along Tree 1. Also, upon removing the
FabricPath header, S100 finds host A was learned locally. Therefore adds B to the
MAC Table as remote, associated with S140
Tree
IF
1
L5
2
L5,L6
FabricPath MAC Table
on S140
Switch
IF
B
e2/2 (local)
MAC A is Unknown
37
FabricPath Forwarding :: Known Unicast (Data)
encap
DSID→140
Ftag→1
SSID→100
DMAC→B
SMAC→A
Destination Switch ID is used to
make routing decisions through
the FabricPath core & no MAC
learning or lookups required
inside the FP core
FabricPath Routing
Table on S11
decap
Switch
IF
S10
L2,L4,L6
S11
-
DSID→140
Ftag→1
S100
L2
SSID→100
S101
L4
DMAC→B
S140
L6
SMAC→A
Payload
Payload
FabricPath Routing
Table on S100
Switch
FabricPath Routing
Table on S140
IF
S10
L1
S11
L2
S100
-
S101
L1,L2
S140
L1,L2
Hash L1,L2
Switch
IF
Payload
S10
L5
SMAC→A
S11
L6
DMAC→B
S100
L5,L6
S101
L5,L6
S140
-
DMAC→B
SMAC→A
Payload
FabricPath MAC Table
on S100
Switch
IF
A
e1/1 (local)
B
S140 (remote)
© 2013 Cisco and/or its affiliates. All rights reserved.
Step 1 :: Host A starts sending traffic to Host B after ARP resolution
Step 2 :: S100 finds B was learned as remote; associated with S140, encap all
subsequent frames to B with S140 as destination in FP header
Step 3 :: S100 Routing Table indicates multiple paths to S140; runs ECMP hash
and this time S100 selects L2 as next-hop
Step 4 :: Routing Table lookup at S11 indicates L6 as next hop for S140
Step 5 :: S140 finds itself as destination in FabricPath header and B is also known
locally; decaps FP header, adds A as remote & associates with S100
FabricPath MAC Table
on S140
Switch
IF
A
S100 (remote)
B
e2/2 (local)
38
FabricPath Loop Mitigation
TTL=2
TTL=1
When the frame is originally encapsulated, the
system sets the TTL to 32; on each hop through
the FabricPath network, each switch decrements
the TTL by 1. If the TTL reaches 0, that frame is
discarded. This feature prevents the continuation
of any loops that may form in the network.
TTL=3
TTL=0
Loop prevention and mitigation is available in the data plane, helping ensure safe forwarding unmatched by
any transparent bridging technology. Cisco FabricPath frames include a time-to-live (TTL) field similar to the
one used in IP, and an applied reverse-path forwarding (RPF) check for multicast based on ‘Tree’ information
© 2013 Cisco and/or its affiliates. All rights reserved.
39
Mixed Chassis Mode :: Supported Topologies
Interop F2 & F2E VDC
With NX-OS 6.1 and Prior Releases ::
• Always use identical line cards on either side of the vPC+ Peer Link, vPC member
ports, and FabricPath core member ports (legs to downstream device)
• The F1-series line cards can mix with M-series line cards
• The F2-series line cards have to be in their own VDC; VDC type [F2] meaning they
can’t mix with F1 or the M-series in the same VDC
© 2013 Cisco and/or its affiliates. All rights reserved.
40
Mixed Chassis Mode :: Supported Topologies
Starting in NX-OS 6.2 and Later Releases ::
• VDC type [F2, F2E, F2 F2E] must match between the 2 vPC+
peer devices when F2 & F2E are used in same VDC; meaning
its ok to have F2 on vPC peer device 1 and F2E on vPC peer
device 2 for the vPC Peer Link, vPC member ports, or
FabricPath core member ports
• Note: in a F2 & F2E type of design; only features related to
F2 apply (lowest common denominator)
• Always use identical line cards on either side of the vPC Peer Link, vPC member ports, and
FabricPath core member ports when M1, M1-XL, M2 & F2E in same VDC [M-F2E] or system
• When F2E is placed in a chassis with M-series it will operate in Layer 2 mode only leveraging the
M for Layer 3 (proxy L3 forwarding); this will provide 128K MAC scale
© 2013 Cisco and/or its affiliates. All rights reserved.
41
© 2013 Cisco and/or its affiliates. All rights reserved.
42
Additional Resources & Further Reading
External (public)
Great External
Resources
Cisco FabricPath Best Practices
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c07-728188.pdf
Scale Data Centers with Cisco FabricPath
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-605488.html
Cisco FabricPath for Cisco Nexus 7000 Series Switches
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-687554.html
Nexus 7000/6000/5000 Configuration Guides
http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html
http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html
http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html
FabricPath Scaling limits
http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NXOS_Verified_Scalability_Guide.html#reference_3AD0536C32FF4B499A0936409729951D
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/configuration_limits/b_N5500_Config_Limits_602N11_
chapter_01.html
© 2013 Cisco and/or its affiliates. All rights reserved.
43
Additional Resources & Further Reading
Quick Start Guide :: Virtual Port Channel (vPC)
https://communities.cisco.com/docs/DOC-35728
© 2013 Cisco and/or its affiliates. All rights reserved.
44
© 2013 Cisco and/or its affiliates. All rights reserved.
45