Architecture & Solutions Group
US Public Sector Advanced Services
Mark Stinnette, CCIE Data Center #39151
Date 1 August 2013
Version 1.7.2
© 2013 Cisco and/or its affiliates. All rights reserved.
1
This Quick Start Guide (QSG) is a Cookbook style guide to Deploying Data Center
technologies with end-to-end configurations for several commonly deployed architectures.
This presentation will provide end-to-end configurations mapped directly to commonly
deployed data center architecture topologies. In this cookbook style; quick start guide;
configurations are broken down in an animated step by step process to a complete end-toend good clean configuration based on Cisco best practices and strong recommendations.
Each QSG will contain set the stage content, technology component definitions,
recommended best practices, and more importantly different scenario data center
topologies mapped directly to complete end-to-end configurations. This QSG is geared for
network engineers, network operators, and data center architects to allow them to quickly
and effectively deploy these technologies in their data center infrastructure based on
proven commonly deployed designs.
© 2013 Cisco and/or its affiliates. All rights reserved.
2
•
•
•
•
Simplest design option :: traditional Aggregation / Access designs
Simplified configuration
Removal of STP
Traffic distribution over all uplinks without vPC port-channels
•
•
•
•
Active / Active gateways (via vPC+ or Anycast HSRP)
VLAN anywhere (no trunk ports)
Option for vPC+ for legacy access switches and computer connectivity
Easily deploy L4-7 services
Natural Evolution of the vPC Design
© 2013 Cisco and/or its affiliates. All rights reserved.
3
•
•
•
Scale out; n-way Active HSRP in FabricPath (up to 4 today)
No longer need vPC+ at SPINE for active/active HSRP
• No peer-link or peer-keepalive link required
Leaf software needs to understand Anycast HSRP in FabricPath
© 2013 Cisco and/or its affiliates. All rights reserved.
4
© 2013 Cisco and/or its affiliates. All rights reserved.
5
•
•
•
•
•
•
Paradigm shift with respect to typical designs (CLOS Fabric topology)
Simplifies SPINE design
Traditional “Aggregation” layer becomes pure FabricPath SPINE
Design helps ensure that any application node are at most only two hops away
FabricPath LEAF switches provide server connectivity like traditional designs
FabricPath LEAF switches also provide L2/L3 boundary, inter-VLAN routing, North  South routing
FabricPath Deployment in Preparation For Dynamic Fabric Automation (DFA)
© 2013 Cisco and/or its affiliates. All rights reserved.
6
NX-OS 6.2
•
•
•
•
Provides DC wide vs. POD local VLAN segmentation / isolation
Can support VLAN ID reuse in multiple PODs
Define FabricPath VLANs :: map VLANs to topology :: map topology
to FabricPath core ports
Optional design for “disconnected” PODs
Each POD can use same non-default FP topology; don’t need
FabricPath Core since each POD is on its own island
© 2013 Cisco and/or its affiliates. All rights reserved.
•
•
Where to place DC wide L2/L3 boundary (vPC+ or Anycast HSRP)
FabricPath Core
Pick a any Aggregation POD
Routed Sub-interfaces on Routed Core / WAN Edge via CE edge ports
Default topology always includes all FabricPath core ports
Map DC Wide VLANs to default topology
POD local core ports also mapped to POD local topology
Map POD local VLANs to POD local topology
7
© 2013 Cisco and/or its affiliates. All rights reserved.
8
© 2013 Cisco and/or its affiliates. All rights reserved.
9
FabricPath is a next generation Layer 2 technology from Cisco that provides multi-path Ethernet capabilities in L2 switching
networks. FabricPath combines the benefits of L2 switching such as easy configuration and workload flexibility with greater
scalability and availability. Specifically, FabricPath adds to L2 switching some routing type capabilities such as all active links,
fast convergence, and loop avoidance mechanisms in the data plane. It allows Layer 2 networking without Spanning Tree
Protocol.
FabricPath provides the following benefits:
• Eliminates Spanning Tree Protocol (STP) with built-in loop prevention and mitigation (TTL & RPF)
• Single control plane for unknown unicast, unicast, broadcast, and multicast traffic
• VLAN anywhere
• FP is transparent to L3 protocols
• Easy to configure
• Easy to manage
• Flexibility
• Create arbitrary any topology
• Multiple designs to integrate L2/L3 boundaries
• Start small and expand as needed (bandwidth growth)
• Efficient and Scalable
• Layer 3 availability similar features
• Leverage parallel paths
• Expanding available bandwidth at L2/L3 Default Gateway level
• MAC address table scale (conversational learning) :: all FabricPath VLANs use conversation MAC address learning
• Fast Convergence and low latency
• Enhances mobility and virtualization in the FabricPath network
• Capable of running vPC (called vPC+) to connect devices to the edge in a port channel
• Multi-tenant support, traffic engineering, meet security separation requirements via FabricPath topologies
© 2013 Cisco and/or its affiliates. All rights reserved.
10
Feature
Benefit
Overview
fabricpath VLAN mode
& switchport mode
Eliminate STP
protocol from the
infrastructure fabric
The FabricPath ports carry traffic only for those VLANs configured as FabricPath
VLANs. It is mandatory to enable the same FP mode VLAN EVERYWHERE on all
switches in the FP fabric (otherwise, FP multidestination trees will be incorrectly
built). VLAN pruning is performed automatically on FP core ports for FP traffic only.
fabricpath forwarding
tables
Service Continuity
FabricPath uses 3 HW forwarding tables to switch frames (1) MAC address table,
(2) Switch-ID table, (3) Multidestination tree table
fabricpath switch IDs
Service Continuity
Each switch in the FP fabric is allocated with a global switch ID value; this is
allocated automatically or manually set (recommended). The switch ID information
will be used in the MAC address-table for L2 forwarding. The vPC+ system also
uses an emulated switch ID; which you assign on both peer devices.
fabricpah IS-IS link
metric
Increase HighAvailability
FP will always take the path with the lowest metric. Its recommended to use the
default reference bandwidth.
fabricpath timers
Improve
Convergence Time
On a case by case basis, if convergence time needs to be improved upon switch
reload, modify lsp-gen-interval and spf-interval timers.
fabricpath root priority
Service Continuity
FP uses two Multi destination Trees, Tree 1 (ftag 1) for broadcast, unknown
unicast, multicast & Tree 2 (ftag 2) –multicast. Recommend to use on SPINE
switches for primary and secondary root.
STP for Classical
Ethernet (CE)
Service Continuity
The FP fabric must be the root of the L2 domain when connected to other legacy L2
domains / switches. Make sure STP priority is the lowest for the entire FP fabric.
vPC+
Increase HighAvailability
FabricPath & vPC+ combined provides two main purposes, (1) dual attach a host to
the FP fabric, (2) Leverage Active/Active HSRP capability
© 2013 Cisco and/or its affiliates. All rights reserved.
11
Feature
Benefit
Overview
Overload Bit
Improve
Convergence Time
RFC 3277 based Overload bit is advertised in updates to prevent a corner case
when a single switch restarts causing temporary loops or traffic black holing. This
feature also prevents neighbors from using a switch as a transit during initial
convergence as well as lowering impact insertion or removal of a switch to the FP
domain.
Multiple Topologies
Design Separation
With multiple topologies, we can create up to 16 topologies where a subset of
VLANs are mapped to a particular topology; allowing more design possibilities.
Anycast HSRP
Increase HighAvailability
Provides up to 4 active Default Gateways for the network which lowers the risk of
disruption for routed and Inter-VLAN traffic and provides bandwidth capacity at
L2/L3 boundaries. The Anycast HSRP feature removes the reliance on vPC+ to
provide the Active/Active HSRP feature at the L2/L3 boundary.
fabricpath static routes
Traffic Engineering
The static route feature gives users capabilities to enter routes directly in the
forwarding tables, ensuring predictable operation of the network.
Certain uses cases where users want to override the routes computed by IS-IS.
Users might want to route traffic to a particular switch using a particular link, better
load balancing or routing traffic through a firewall (policing) in the network.
© 2013 Cisco and/or its affiliates. All rights reserved.
12
Install license bootflash:///enchanced_layer2_pkg.lic
show license usage
feature lacp
install feature-set fabricpath
feature-set fabricpath
feature lacp
install feature-set fabricpath
feature-set fabricpath
Default / Admin
VDC Only
vlan 1 – 200
mode fabricpath
7K-1
interface po2
switchport mode fabricpath
interface e3/1, e4/1
channel-group 2 mode active
7K-2
3/1
4/1
5/1
Po2
5/2
5/1
interface e5/1, e5/2
switchport mode fabricpath
feature lacp
install feature-set fabricpath
feature-set fabricpath
vlan 1 – 200
mode fabricpath
interface po2
switchport mode fabricpath
interface e1/1, e1/2
channel-group 2 mode active
interface e1/3, e1/4
switchport mode fabricpath
© 2013 Cisco and/or its affiliates. All rights reserved.
vlan 1 – 200
mode fabricpath
interface po2
switchport mode fabricpath
3/1
4/1
5/2
Default / Admin
VDC Only
interface e3/1, e4/1
channel-group 2 mode active
interface e5/1, e5/2
switchport mode fabricpath
1/3
1/4
1/4
1/1
1/2
5K-1
1/3
1/1
1/2
Po2
5K-2
Step 1 :: install | validate Enhanced L2 License
Step 2 :: install FabricPath
Step 3 :: enable FabricPath
Step 4 :: configure FabricPath VLANs
Step 5 :: configure FabricPath core ports
Beginning with the Cisco NX-OS Release 5.1 and when you
use an F Series modules and NX-OS Release 5.1(3) N1(1)
with 5500 you can use the FabricPath feature
feature lacp
install feature-set fabricpath
feature-set fabricpath
vlan 1 – 200
mode fabricpath
interface po2
switchport mode fabricpath
interface e1/1, e1/2
channel-group 2 mode active
interface e1/3, e1/4
switchport mode fabricpath
13
fabricpath switch-id 10
fabricpath switch-id 11
fabricpath domain default
root-priority 255
Root for FTAG 1
Root for FTAG 2
7K-1
7K-2
3/1
4/1
SW 10
5/1
3/1
4/1
5/2
SW 11
Po2
5/2
5/1
fabricpath switch-id 100
fabricpath switch-id 101
1/3
SW 100
Each peer devices will have a unique global switch
ID value – make the FP network more deterministic
1/4
1/4
1/1
1/2
5K-1
1/3
1/1
1/2
Po2
Step 1 :: set the FP Switch-ID
Step 2 :: set the FP Root
Suggested switch ID scheme:
SPINE :: 2 digit ID
LEAF :: 3 digit ID
Emulated Switch (vPC+) :: 4 digit ID
5K-2
SW 101
Multi destination Tree 1 (ftag 1) – broadcast,
unknown unicast, multicast
Multi destination Tree 2 (ftag 2) –multicast
Recommend to use on SPINE switches
Higher Number the better !!
F2/F2E uses both trees for UU/Bcast/Mcast
F1 uses MDT 2 for Mcast only
© 2013 Cisco and/or its affiliates. All rights reserved.
fabricpath domain default
root-priority 254
(start at 255 and go backwards)
-or(start at 200 in case you need to introduce another
MDT at a later time; ie expanded SPINE x 4)
14
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
vlan 1 – 200
mode fabricpath
optional
7K-1
7K-2
3/1
4/1
vlan 1 – 200
mode fabricpath
5/1
optional
3/1
4/1
5/2
Po2
5/2
5/1
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
vlan 1 – 200
mode fabricpath
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
1/3
The entire FabricPath domain will look like one
virtual bridge to the CE domain – set best (lowest)
STP root priority on the vPC+ peers (recommended
at least at the access edge leaf switches); just make
sure the priority is lower than anything else in the
network (classical Ethernet)
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
1/4
1/4
1/1
1/2
5K-1
1/3
1/1
1/2
Po2
5K-2
vlan 20, 40
spanning-tree vlan 20, 40 priority 8192
FP will use the same bridge ID c84c.75fa.6000
The root and sender bridge MAC addresses of this
pseudo-information are the same on every switch in the
Cisco FabricPath domain
All ports at the edge of a Cisco FabricPath network are
configured with the equivalent of root guard (don’t need to
configure this feature), a feature that would block a port
should it receive superior Spanning Tree Protocol BPDUs
© 2013 Cisco and/or its affiliates. All rights reserved.
Step 1 :: set FP domain to be root bridge
Note that the spanning-tree priority command
would also work; however, it would change the
priority for the spanning tree regardless of whether
the switch were sending regular BPDUs (when
Cisco FabricPath is not running) or sending BPDUs
with the pseudo-information (when Cisco
FabricPath is operational on the switch). In some
scenarios, this change can have undesirable side
effects.
15
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
fabricpath timers linkup-delay 60
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
7K-1
7K-2
3/1
4/1
5/1
3/1
4/1
5/2
Po2
5/2
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
5/1
fabricpath timers linkup-delay 60
1/4
1/4
1/1
1/2
5K-1
1/3
1/1
1/2
Po2
5K-2
Note: Future enhancements such as Layer 2 IS-IS
overload bit support in 6.2 will help to improve unicast
and multicast convergence during FabricPath node
failure scenarios when default IS-IS timers are used.
Step 1 :: tune the IS-IS timers in FabricPath
Step 2 :: (optional) tune the FabricPath linkup-delay
To achieve fast convergence during node failures and recovery scenarios, it is
recommended to tune the IS-IS timers in Cisco FabricPath. This tuning is
particularly important when a switch is inserted in the topology.
This configuration is recommended for all switches in the network
© 2013 Cisco and/or its affiliates. All rights reserved.
fabricpath domain default
spf-interval 50 50 50
lsp-gen-interval 50 50 50
fabricpath timers linkup-delay 60
1/3
Problem Set: The IS-IS adjacency is established and
the access-edge started sending traffic to aggregationedge, but the control plane was not ready to forward the
traffic to the next hop. The default spf and lsp-gen
intervals are 8sec (default) and it attributes to the long
convergence. To address this issue, the default spf and
lsp-gen intervals of {max-wait, initial-wait, second-wait}
are brought down to 50msec, with this configuration, the
aggregation-edge restoration will yield sub-second
convergence for Layer 2 traffic
fabricpath timers linkup-delay 60
Optional, to provide better network convergence
upon a Cisco FabricPath switch restart, you should
set a Cisco FabricPath linkup-delay timer to 60
16
vpc domain 1
role priority 1
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath switch-id 1000
dual-active exclude interface vlan 20
interface po2
switchport mode fabricpath
vpc peer-link
vPC+ is an extension of vPC for FabricPath. It allows
dual-homed connections from Classical Ethernet
(CE) switches and hosts capable of port channels. It
also provides for active-active HSRP.
The configuration of peer-link and peer-keepalive
links are required – as traditional vPC
Enable IP ARP Synchronization of ARP entries
between vPC Peers improves convergence for
North-South and East-West Layer 3 traffic when one
of the vPC+ peers is brought back up
Note: Since FabricPath does not rely on Spanning
Tree Protocol, and the vPC+ peer link is a FabricPath
Core port, so the peer-switch command is not
needed under the vpc domain [x] configuration
© 2013 Cisco and/or its affiliates. All rights reserved.
vPC+
7K-1
7K-2
SW 1000
3/1
4/1
5/1
3/1
4/1
5/2
Po2
5/2
5/1
vpc domain 1
role priority 2
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath switch-id 1000
dual-active exclude interface vlan 20
interface po2
switchport mode fabricpath
vpc peer-link
With vPC+, a FabricPath switch is emulated between
the CE and FabricPath domain. All packets originating
behind the Emulated Switch will be marked with the
source Switch ID of the emulated switch
1/3
1/4
1/4
1/1
1/2
5K-1
1/3
1/1
1/2
Po2
5K-2
Step 1 :: enable vPC+
Step 2 :: set the emulated switch-id
Step 3 :: enable dual-active exclude for vPC SVIs
Assign the same emulated switch ID on both vPC peers;
but the emulated switch ID must be unique between
different vPC domains
In a vPC environment, the Secondary vPC switch will
bring down the SVIs by default when the peer-link is
brought down. This behavior is fine in CE environment
as the vPC legs are also brought down on the
secondary vPC switch. However in the vPC+
environment, the down links to the Access-Edge
switches are FabricPath Core ports; in the absence of
the vPC+ peer-link, the SVIs can still communicate
through the FabricPath core ports.
The vPC dual-active exclude vlan command helps to
configure a VLAN list such that the SVI can continue to
stay up on the secondary vPC switch even if the vPC+
peer-link is down.
17
feature interface-vlan
feature hsrp
feature lacp
feature vpc
feature interface-vlan
feature hsrp
feature lacp
feature vpc
vPC+
7K-1
vlan 1 – 200
mode fabricpath
vpc domain 1
role priority 1
system-priority 4096
peer-keepalive destination [….] source [….]
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
fabricpath switch-id 1000
dual-active exclude interface vlan 20
interface po2
switchport mode fabricpath
vpc peer-link
interface e3/1, e4/1
channel-group 2 mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
vlan 1 – 200
mode fabricpath
SW 1000
3/1
4/1
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
------------------------
7K-2
5/1
1/3
3/1
4/1
5/2
Po2
5/2
1/4
1/4
1/1
1/2
5K-1
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
5/1
1/3
1/1
1/2
Po2
5K-2
Step 1 :: enable vPC+
Step 2 :: set the emulated switch-id
Step 3 :: enable dual-active exclude for vPC SVIs
interface vlan 20
ip address 20.20.20.5/24
no ip redirect
hsrp 20
preempt
priority 110
ip 20.20.20.254
interface vlan 20
ip address 20.20.20.6/24
no ip redirect
hsrp 20
preempt
ip 20.20.20.254
-----------------------vpc domain 1
role priority 2
system-priority 4096
peer-keepalive destination [….] source [….]
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
fabricpath switch-id 1000
dual-active exclude interface vlan 20
interface po2
switchport mode fabricpath
vpc peer-link
interface e3/1, e4/1
channel-group 2 mode active
18
feature lacp
feature vpc
feature lacp
feature vpc
vlan 1 – 200
mode fabricpath
vlan 1 – 200
mode fabricpath
vPC+
7K-1
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
interface po2
switchport mode fabricpath
vpc peer-link
interface e1/1, e1/2
channel-group 2 mode active
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 20 – 40
vpc 20
interface e1/5
channel-group 20 force mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
spanning-tree pseudo-information
vlan 1 – 200 root priority 0
3/1
4/1
-----------------------vpc domain 10
role priority 1
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath switch-id 1001
7K-2
SW 1000
5/1
3/1
4/1
5/2
------------------------
Po2
5/2
5/1
1/4
1/3
vPC+
1/3
1/4
1000
SW 1001
1/1
1/2
1/5
1/1
1/2
Po2
5K-1
vPC 20
1/5
5K-2
Step 1 :: enable vPC+
Step 2 :: set the emulated switch-id
Step 3 :: add devices redundantly with vPC+
VLANs carried on vPC+ member ports must be
FabricPath mode VLANs
vpc domain 10
role priority 2
peer-keepalive destination [….] source [….]
….
ip arp synchronize
fabricpath switch-id 1001
interface po2
switchport mode fabricpath
vpc peer-link
interface e1/1, e1/2
channel-group 2 mode active
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 20 – 40
vpc 20
interface e1/5
channel-group 20 force mode active
19
interface port-channel2
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FP-KEYS
interface port-channel2
switchport mode fabricpath
fabricpath isis authentication-type md5
fabricpath isis authentication key-chain FP-KEYS
7K-1
fabricpath domain default
authentication-type md5
authentication key-chain FP-KEYS
key chain FP-KEYS
key 0
key-string Cisc0!
accept-lifetime 00:00:00 Sep 1 2012 infinite
send-lifetime 00:00:00 Sep 1 2012 infinite
FabricPath provides 2 levels of authentication
1. Authentication at the interfaces level
2. Authentication at the global level
7K-2
3/1
4/1
3/1
4/1
5/1
5/2
1/3
1/4
key chain FP-KEYS
key 0
key-string Cisc0!
accept-lifetime 00:00:00 Sep 1 2012 infinite
send-lifetime 00:00:00 Sep 1 2012 infinite
Po2
1/1
1/2
5K-1
fabricpath domain default
authentication-type md5
authentication key-chain FP-KEYS
5/2
5/1
1/4
1/3
1/1
1/2
Po2
5K-2
global level authentication ::
authenticates and controls the FP LSPs and PSNPs
interfaces level authentication ::
authenticates the HELLO; the FP ISIS adjacency
The Key chain is used in both forms of authentication
Supported combinations:
Step 1 :: configure the key chain
Step 2 :: configure global FabricPath authentication
Step 3 :: configure FabricPath core port authentication
You can configure the accept lifetime and send lifetime for a key. By default, accept
and send lifetimes for a key are infinite, which means that the key is always valid.
accept-lifetime [local] start-time duration duration-value | infinite | end-time]
send-lifetime [local] start-time duration duration-value | infinite | end-time]
© 2013 Cisco and/or its affiliates. All rights reserved.
20
NX-OS 6.2
7K-1
7K-2
3/1
4/1
3/1
4/1
5/1
5/2
1/3
1/4
Po2
1/1
1/2
5K-1
5/2
5/1
1/4
1/3
1/1
1/2
Po2
5K-2
Step 1 ::
Step 2 ::
Step 3 ::
Step 4 ::
Step 5 ::
© 2013 Cisco and/or its affiliates. All rights reserved.
21
NX-OS 6.2
7K-1
7K-2
3/1
4/1
3/1
4/1
5/1
5/2
1/3
1/4
Po2
1/1
1/2
5K-1
5/2
5/1
1/4
1/3
1/1
1/2
Po2
5K-2
Step 1 ::
Step 2 ::
Step 3 ::
Step 4 ::
Step 5 ::
© 2013 Cisco and/or its affiliates. All rights reserved.
22
NX-OS 6.2
7K-1
7K-2
3/1
4/1
3/1
4/1
5/1
5/2
1/3
1/4
Po2
1/1
1/2
5K-1
5/2
5/1
1/4
1/3
1/1
1/2
Po2
5K-2
Step 1 ::
Step 2 ::
Step 3 ::
Step 4 ::
Step 5 ::
© 2013 Cisco and/or its affiliates. All rights reserved.
23
NX-OS 6.2
7K-1
7K-2
3/1
4/1
3/1
4/1
5/1
5/2
1/3
1/4
Po2
1/1
1/2
5K-1
5/2
5/1
1/4
1/3
1/1
1/2
Po2
5K-2
Step 1 ::
Step 2 ::
Step 3 ::
Step 4 ::
Step 5 ::
© 2013 Cisco and/or its affiliates. All rights reserved.
24
vPC
interface e1/5
ip address 192.168.1.1/24
vrf membership vpc-keepalive
vpc domain 1
peer-keepalive destination 192.168.1.2
source 192.168.1.1 vrf vpc-keepalive
interface port-channel 1000
switchport mode trunk
vpc peer-link
interface e1/1-2
switchport mode trunk
channel-group 1000 mode active
interface e1/3
switchport mode trunk
channel-group 1 mode active
interface port-channel1
vpc 1
interface e2/5
ip address 192.168.1.2/24
vrf membership vpc-keepalive
vpc domain 1
peer-keepalive destination 192.168.1.1
source 192.168.1.2 vrf vpc-keepalive
interface port-channel 1000
switchport mode trunk
vpc peer-link
interface e2/1-2
switchport mode trunk
channel-group 1000 mode active
interface e2/3
switchport mode trunk
channel-group 1 mode active
interface port-channel1
vpc 1
FabricPath + vPC+
Advantages
Active/active path at L2
Active/active for HSRP
Works with all LC
Advantages
Active/active path at L2
Active/active for HSRP
Ease of configuration
No more STP
Extensibility
Drawbacks
Need dedicated infrastructure (PL,
PKL)
Configuration on both peer devices
Consistency check to care about
STP still here (but runs as fail safe
mechanism)
Drawbacks
Need dedicated infrastructure (PL,
PKL)
Need F1 (+M1) or F2
interface e1/1-3
switchport mode fabricpath
e1/5
interface e2/1-3
switchport mode fabricpath
e2/5
e1/1-2
e2/1-2
e1/3
e2/3
VPC
e3/1-2
interface e3/1-2
switchport mode trunk
channel-group 1 mode passive
VPC Configuration
© 2013 Cisco and/or its affiliates. All rights reserved.
e1/1-2
e1/3
e2/1-2
FabricPath
e2/3
e3/1-2
interface e3/1-2
switchport mode fabricpath
FabricPath Configuration
25
Common Design Migration Starting Point
7k – Aggregation
5k/2k – Access Pods
Dual Layer vPC
Mix F1 / M1 line cards
After Migration Completion
7k – SPINE role
5k – LEAF role
vPC converted to FabricPath core ports
Peer-Link also FP core port = vPC+
(only F1/F2 support FabricPath)
Additional Reading Here :: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-709336.html
© 2013 Cisco and/or its affiliates. All rights reserved.
26
• FabricPath VLANs must be configured on all switches in the FP domain
• It is recommended to configure the switch ID manually on all FabricPath switches
• For Active-Active HSRP capability, it is recommended to configure vPC+ on the Aggregation-Edge switches even if there
are no vPC legs. Note: subject to vPC rules; so no dynamic routing over vPC to firewalls, Core layer, WAN edge
• Implement Layer 3 routing backup path
• Separate L3 port channel; point-to-point links
• Separate L2 port channel; use dedicated VLAN in Classical Ethernet (CE) mode as transit VLAN inside this L2 trunk
• Disable IP redirects on SVIs and configure passive interface to avoid any routing adjacency over SVIs
• ARP sync feature with vPC+ is recommended for improved traffic convergence during Aggregation-Edge failure and
restoration
• It is recommended to configure highest and second highest MDT root priority on the Aggregation-Edge switches
• Have option of choosing single links or port-channels between Aggregation-Edge and Access-Edge for ECMP. If port
channels are used, configuring IS-IS metric is preferred. With path costing, member link failure is transparent to IS-IS
protocol so that the traffic would continue to use the same path
• Raise the FP IS-IS metric for VPC+ Peer-Link to prefer other FP core links
• interface po2
fabricpath isis metric 200
• It is recommended to have lowest path cost for the links between AGG devices so the multicast hello packets always take
the peer-link which is direct link between the AGG devices
• It is recommended to tune Layer 2 IS-IS SPF and LSP generation timers to achieve better convergence during failure and
restoration scenarios. These timers should be tuned to 50 msec with 50msec initial wait and second wait. This is a
requirement until the overload bit support is available with Layer 2 IS-IS
• Use default reference BW (its 400 Gbps default)
• fabricpath domain default
reference-bandwidth ?
• IS-IS metric cost (1Gb = cost 400, 10Gb = cost 40, 20Gb = cost 20)
© 2013 Cisco and/or its affiliates. All rights reserved.
27
• IS-IS link metric for port-channel depends on NX-OS version
• Up to NX-OS 6.0: IS-IS metric for port-channel is calculated based on number of configured member ports; meaning
you may need to use LACP min-link feature to tear down port-channel if number of active member ports goes below
a specific limit
• Since NX-OS 6.1: IS-IS metric for port-channel is calculated based on number of active ports
• Dual-active exclude VLAN configuration is recommended so that the SVIs can continue to be active on the secondary
vPC+ peer in the event of peer-link failure. This also helps to stay with default HSRP timers there by reducing control plane
load associated with aggressive HSRP timers
• Do not use dual-active exclude command for VLANs if you have vPC attached devices, for example at the access (leaf)
• In typical vPC deployments it is not necessary to tune the HSRP hello timers from the defaults (3/10s). In a FabricPath
deployment it is recommended to use aggressive timers (1/3) to minimize flooding of South to North traffic from the
edge switch. This allows the active HSRP virtual MAC to be learned faster at all edge switches
• hsrp 1
preempt delay minimum 180
timers 1 3
ip ….
• In CE-FabricPath hybrid networks, it is recommended to configure the lowest Spanning-tree root priority on all FabricPath
Edge switches
• The MAC timer should be consistent on all devices in the Layer 2 topology. The MAC and ARP aging timers can be left at
defaults, 1800sec & 1500sec respectively
• The M1/F1 mixed VDC currently supports up to 16K MAC/ARP entries. This limitation will be lifted with the Layer 2 proxy
learning feature in the upcoming NX-OS release
• The M1, M1-XL, M2 & F2E in a mixed VDC topology; meaning when F2E is placed in a chassis with M-series it will
operate in Layer 2 mode only leveraging the M for Layer 3 (proxy L3 forwarding); this will enable 128K MAC/ARP scale.
• If an ASA cluster is attached to the Nexus 7000 series Aggregation-edge switches, source-dest-ip or src-dst ip-l4port is the
recommended load balance algorithm if the ASA cluster is in single context mode or if the VLANs are fewer in multi-context
mode. This is to prevent traffic polarization on links towards ASA cluster member
© 2013 Cisco and/or its affiliates. All rights reserved.
28
• Better use port-channel instead of individual links for the 2 following reasons
• Decrease the number of direct IS-IS adjacency (1 for the whole port-channel instead of X IS-IS adjacencies if X
individual links are used between the 2 switches)
• Allows to use the whole port-channel capacity for multidestination tree #1 or #2 (if multiple parallel individual links
exist between 2 switches, only 1 link will be selected for tree #1 and potentially 1 another link for tree #2)
• ECMP vs. Port Channel
• Can use ECMP, port-channel, or both simultaneously
• Port-channels have one main advantage over ECMP – treated as single logical link in FabricPath IS-IS. Individual
link failure invisible to upper layer protocols. Also allows more bandwidth for branches of multidestination trees
• With 4 member port channel, whole interface becomes single branch of tree with 40G BW
• With 4 parallel ECMP paths, only one of the 4 interfaces becomes part of the tree
• ECMP with port-channel : 2 levels of load-balancing decision :
• First level : FP Core Link selection (based on L3/L4 fields by default)
• Second level : Port-Channel member selection (based on src-dst ip by default)
• Do not use UDLD with FabricPath
• UDLD (normal or aggressive) does not bring any benefits on single physical link and port channels with FP enabled
(for port channel, activate LACP instead of relying on UDLD to detect member port issues)
• Physical link level protection and the bi-directional IS-IS hellos should take care of all (or near all) potentially link
level issue
© 2013 Cisco and/or its affiliates. All rights reserved.
29
Interop F2 & F2E VDC
With NX-OS 6.1 and Prior Releases ::
• Always use identical line cards on either side of the vPC+ Peer Link, vPC member
ports, and FabricPath core member ports (legs to downstream device)
• The F1-series line cards can mix with M-series line cards
• The F2-series line cards have to be in their own VDC; VDC type [F2] meaning they
can’t mix with F1 or the M-series in the same VDC
© 2013 Cisco and/or its affiliates. All rights reserved.
30
Starting in NX-OS 6.2 and Later Releases ::
• VDC type [F2, F2E, F2 F2E] must match between the 2 vPC+
peer devices when F2 & F2E are used in same VDC; meaning
its ok to have F2 on vPC peer device 1 and F2E on vPC peer
device 2 for the vPC Peer Link, vPC member ports, or
FabricPath core member ports
• Note: in a F2 & F2E type of design; only features related to
F2 apply (lowest common denominator)
• Always use identical line cards on either side of the vPC Peer Link, vPC member ports, and
FabricPath core member ports when M1, M1-XL, M2 & F2E in same VDC [M-F2E] or system
• When F2E is placed in a chassis with M-series it will operate in Layer 2 mode only leveraging the
M for Layer 3 (proxy L3 forwarding); this will provide 128K MAC scale
© 2013 Cisco and/or its affiliates. All rights reserved.
31
© 2013 Cisco and/or its affiliates. All rights reserved.
32
External (public)
Great External
Resource
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c07-728188.pdf
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/white_paper_c11-605488.html
Nexus 7000/6000/5000 Configuration Guides
http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html
http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html
http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html
FabricPath Scaling limits
http://www.cisco.com/en/US/docs/switches/datacenter/sw/verified_scalability/b_Cisco_Nexus_7000_Series_NXOS_Verified_Scalability_Guide.html#reference_3AD0536C32FF4B499A0936409729951D
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5500/sw/configuration_limits/b_N5500_Config_Limits_602N11_
chapter_01.html
© 2013 Cisco and/or its affiliates. All rights reserved.
33