Architecture & Solutions Group
US Public Sector Advanced Services
Mark Stinnette, CCIE Data Center #39151
Date 1 August 2013
Version 1.7.2
© 2013 Cisco and/or its affiliates. All rights reserved.
1
This Quick Start Guide (QSG) is a Cookbook style guide to Deploying Data Center
technologies with end-to-end configurations for several commonly deployed architectures.
This presentation will provide end-to-end configurations mapped directly to commonly
deployed data center architecture topologies. In this cookbook style; quick start guide;
configurations are broken down in an animated step by step process to a complete end-toend good clean configuration based on Cisco best practices and strong recommendations.
Each QSG will contain set the stage content, technology component definitions,
recommended best practices, and more importantly different scenario data center
topologies mapped directly to complete end-to-end configurations. This QSG is geared for
network engineers, network operators, and data center architects to allow them to quickly
and effectively deploy these technologies in their data center infrastructure based on
proven commonly deployed designs.
© 2013 Cisco and/or its affiliates. All rights reserved.
2
© 2013 Cisco and/or its affiliates. All rights reserved.
3
Double-Sided vPC
© 2013 Cisco and/or its affiliates. All rights reserved.
4
© 2013 Cisco and/or its affiliates. All rights reserved.
5
© 2013 Cisco and/or its affiliates. All rights reserved.
6
vPC Peer-Keepalive Link
vPC Peer-Link
Dedicated Layer 3 Infrastructure
vPC Peer Device
vPC Member Port
vPC
vPC Domain
Orphan Port
vPC
vPC VLAN ::
VLAN(s) carried over the vPC peer-link and used to
communicate via a vPC; As soon as a VLAN is
defined on vPC peer-link it becomes a vPC VLAN
non-vPC VLAN ::
VLAN(s) that is not part of any vPC and not present
on the vPC Peer-Link
© 2013 Cisco and/or its affiliates. All rights reserved.
7
vPC is a virtualization technology that presents paired or two Nexus devices as a unique Layer 2 logical node to the access
layer devices or endpoints. vPC belongs to Multichassis EtherChannel [MCEC] family of technology.
A virtual port channel (vPC) allows links that are physically connected to two different Cisco Nexus 7000 or 5000 Series
devices to appear as a single port channel to a third device. The third device can be a switch, server, firewall, load balancer
or any other networking device that supports link aggregation technology.
vPC provides the following technical benefits:
• Eliminates Spanning Tree Protocol (STP) blocked ports
• Uses all available uplink bandwidth; Layer 2 hashing algorithm
• Allows dual-homed servers to operate in active-active mode
• Provides fast convergence upon link or device failure
• Offers dual active/active FHRP (default gateways) for servers
• Each peer device in the vPC domain runs its own control plane, and both devices work independently
Using vPC; you gain immediate operational and architectural advantages:
• Simplifies network design
• Build highly resilient and robust Layer 2 network
• Enables seamless virtual machine mobility and server high-availability clusters
• Scales available Layer 2 bandwidth, increasing bisectional bandwidth
• Grows the size of the Layer 2 network
• vPC feature is included in the base NX-OS software license
vPC also leverages native split horizon/loop management provided by port-channeling technology; meaning a packet
entering a port-channel cannot immediately exit that same port-channel.
© 2013 Cisco and/or its affiliates. All rights reserved.
8
© 2013 Cisco and/or its affiliates. All rights reserved.
9
Feature
Benefit
Overview
vPC auto-recovery
(reload restore)
Increase HighAvailability
(1) Provides a backup mechanism in case a vPC peer-link failure followed by a vPC
primary peer device failure; (2) Both vPC peer devices reload or DC power outage;
but only one vPC peer comes up - this allows one vPC device to assume STP /
vPC primary role and bring up all local vPCs in case (auto-recovery reload-delay)
vPC Peer-Gateway
Service Continuity
Allows a vPC switch to act as the active gateway for packets addressed to the peer
router MAC (ie. NAS)
vPC orphan-ports
suspend
Increase HighAvailability
When vPC peer-links go down, vPC secondary shuts down all the vPC member
ports as well as orphan ports. It avoids single attached devices like FW, LB or NIC
teamed devices when isolated during vPC peer-link failure
vPC ARP SYNC
Improve Convergence
Time
Improve Convergence for Layer 3 flows after vPC peer-link is UP or recovers from
a failure
vPC Peer-Switch
Improve Convergence
Time
Virtualize both vPC peer devices so they appear as a unique STP root bridge
vPC Role & System
Priority
Service Continuity
Manually set vPC system priority to ensure vPC peer devices are the primary
devices on LACP. Manually set the vPC role as primary and secondary –
deterministic
vPC Peer-keepalive
Increase HighAvailability
Option 1 :: use SUP mgmt int on dedicated OOB
Option 2 :: use separate L3 Po in dedicated VRF
vPC Delay Restore
Service Continuity
Delays vPCs member links from bring up on the recovering vPC peer device. This
allows for the Layer 3 routing protocols to converge before allowing any traffic on
vPC member links; resulting in a more graceful restoration and zero packet loss
during the recovery phase. (This feature is enabled by default – 30 seconds)
© 2013 Cisco and/or its affiliates. All rights reserved.
10
Option 1
Dedicated link(s) in a Layer 3 Port-Channel in its own dedicated VRF (ie. PKAL VRF)
Use separate line cards & 1Gig ports are enough, else you burn 10Gig interfaces
Option 2
Use Mgmt0 interfaces off Supervisors to dedicated routable OOB network + use management VRF
Peer-Keepalive traffic along with management traffic
Option 3
As a last resort, route the peer-keepalive traffic over the Layer 3 infrastructure + use default VRF
© 2013 Cisco and/or its affiliates. All rights reserved.
11
Option 1
Use Mgmt0 interfaces to dedicated routable OOB network + use management VRF
Peer-Keepalive traffic along with management traffic
Option 2 (Nexus 5000 with L3 module)
Dedicated link(s) in a Layer 3 Port-Channel in its own dedicated VRF (ie. PKAL VRF)
Use separate interfaces & will burn 10Gig interfaces
Option 2 (Nexus 5000 without L3 module)
Dedicated link(s) in a separate Layer 2 Port-Channel , have the peer-keepalive peer across to the
SVI’s, manually prune those VLANs off the peer-link making those VLANs (non-vPC VLANs), only
trunk the peer-keepalive VLAN across this Layer 2 Port-Channel
Due to ISSU checks via the show spanning issue-impact, ISSU will fail, workaround is to disable
STP on this dedicated Layer 2 Port-Channel via the spanning-tree port type edge trunk command,
assuming you have the global command spanning-tree port type edge bpdufilter default enabled
Will burn 10Gig interfaces
© 2013 Cisco and/or its affiliates. All rights reserved.
12
feature lacp
feature lacp
vlan 1 – 200
vlan 1 – 200
vrf context PKAL
7K-1
7K-2
vrf context PKAL
Po1
interface port-channel 1
vrf member PKAL
ip address [….]/30
1/1
2/1
1/1
2/1
3/1
4/1
3/1
4/1
Po2
interface e1/1 , e2/1
channel-group 1 mode active
interface port-channel 1
vrf member PKAL
ip address [….]/30
interface e1/1 , e2/1
channel-group 1 mode active
------------------------------------------------------
------------------------------------------------------
interface port-channel 2
switchport
switchport mode trunk
interface port-channel 2
switchport
switchport mode trunk
interface e3/1 , e4/1
channel-group 2 force mode active
interface e3/1 , e4/1
channel-group 2 force mode active
5K-1
5K-2
Create a dedicated VRF for the vPC peer-keepalive link
(best practice)
Step 1 :: turn on LACP feature
Step 2 :: define your vlans
Step 3 :: build peer-keepalive
Step 4 :: build L2 port channel for peer-link
© 2013 Cisco and/or its affiliates. All rights reserved.
Building a vPC peer-link, follow these guidelines ::
(1) Must have Peer-keepalive link up first; ensure the peerlink member ports are 10 Gig interfaces
(2) Use a minimum of two 10 Gig ports (M1 up to 8 member
ports & F1/F2 up to 16 member ports)
(3) Use at least two different line cards to increase high
availability of peer-link
13
(Optional Config) – when using vPC peer-switch in a ‘hybrid’
environment use the spanning-tree pseudo-information to
load balance VLANs across the 2 peer devices
feature vpc
feature vpc
vlan 1 – 200
vlan 1 – 200
spanning-tree pathcost method long
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
no spanning-tree loopguard default
7K-1
7K-2
spanning-tree pathcost method long
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
no spanning-tree loopguard default
spanning-tree vlan 1-200 priority 0
spanning-tree pseudo-information
vlan 1-200 root priority 4096
vlan 1-100 designated priority 8192
vlan 101-200 designated priority 16384
spanning-tree vlan 1-200 priority 0
spanning-tree pseudo-information
vlan 1-200 root priority 4096
vlan 1-100 designated priority 16384
vlan 101-200 designated priority 8192
vpc domain 1
role priority 1
system-priority 4096
peer-keepalive destination [….] source [….] vrf
PKAL
peer-switch
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
vpc domain 1
role priority 2
system-priority 4096
peer-keepalive destination [….] source [….] vrf
PKAL
peer-switch
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
Hard set the Nexus 7K on the left vPC role primary and
Nexus 7K on the right vPC role secondary (deterministic)
Make the Nexus 7Ks control LACP establishment for all
port-channels; (lowest) vpc domain id + system priority
Setup the peer-keepalive; use the correct VRF accordingly
© 2013 Cisco and/or its affiliates. All rights reserved.
5K-1
5K-2
Step 1 :: turn on vpc feature
Step 2 :: configure spanning-tree defaults
Step 3 :: configure spanning-tree vlan root priorities
Step 4 :: configure vpc domain (per best practices)
Enable peer-switch; when activated both vPC peer devices
must have the same STP priority set for all vPC VLANs –
making them appear as a unique STP root bridge
Enable peer-gateway, auto-recovery, delay restore, and ip
arp synchronize (per best practice) … see Strong
Recommendations & Key Notes sections
14
feature lacp
feature vpc
feature lacp
feature vpc
vlan 1 – 200
vlan 1 – 200
7K-1
7K-2
vrf context PKAL
vrf context PKAL
interface port-channel 1
vrf member PKAL
ip address [….]/30
interface port-channel 1
vrf member PKAL
ip address [….]/30
peer-link
interface e1/1 , e2/1
channel-group 1 mode active
interface e1/1 , e2/1
channel-group 1 mode active
------------------------------------------------------
------------------------------------------------------
interface port-channel 2
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type network
vpc peer-link
interface port-channel 2
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type network
vpc peer-link
5K-1
5K-2
interface e3/1 , e4/1
channel-group 2 force mode active
interface e3/1 , e4/1
channel-group 2 force mode active
Step 1 :: enable vPC peer-link on the L2 port channel
Always perform VLAN pruning on vPC peer-link with the
allowed list of vPC VLANs; vPC VLANs must also be
pruned on the vPC member port s as well
© 2013 Cisco and/or its affiliates. All rights reserved.
Bridge Assurance is enabled by default when configuring
vPC peer-link (spanning-tree port type network); Do NOT
disable it on the vPC peer-link
15
7K-1
7K-2
peer-keepalive link
Use Mgmt0 interfaces to dedicated routable OOB
network + use management VRF (configured during
initial device setup); includes Peer-Keepalive traffic
along with management traffic
feature lacp
feature lacp
vlan 1 – 200
vlan 1 – 200
vrf context management
ip route 0.0.0.0/0 [….]
interface mgmt0
ip address [….]/24
interface port-channel 1
switchport
switchport mode trunk
interface e1/1 - 2
channel-group 2 force mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
1/1
1/2
mgmt0
1/1
1/2
mgmt0
5K-1
5K-2
OOB
Step 1 :: turn on LACP feature
Step 2 :: define your vlans
Step 3 :: build L2 port channel for peer-link
vrf context management
ip route 0.0.0.0/0 [….]
interface mgmt0
ip address [….]/24
interface port-channel 1
switchport
switchport mode trunk
interface e1/1 - 2
channel-group 2 force mode active
16
Always use a different domain ID in a double-sided
vPC topology; once configured, both peer devices
use the vPC domain ID to automatically assign a
unique vPC system MAC address; which is used as
part of the LACP protocol
vpc domain 1
role priority 1
system-priority 4096
peer-keepalive destination [….] source [….] vrf
management
peer-switch
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
7K-1
7K-2
vpc domain 1
role priority 2
system-priority 4096
peer-keepalive destination [….] source [….] vrf
management
peer-switch
peer-gateway
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
feature lacp
feature vpc
feature lacp
feature vpc
vlan 1 – 200
vlan 1 – 200
spanning-tree pathcost method long
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
no spanning-tree loopguard default
spanning-tree pathcost method long
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
no spanning-tree loopguard default
vpc domain 10
role priority 1
system-priority 8096
peer-keepalive destination [….] source [….] vrf
management
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
© 2013 Cisco and/or its affiliates. All rights reserved.
5K-1
5K-2
OOB
Manually set vPC system priority to ensure vPC peer
devices are the primary devices on LACP at the
aggregation layer or not the primary devices on LACP at
the access layer
Step 1 :: turn on vpc feature
Step 2 :: configure spanning-tree defaults
Step 3 :: configure vpc domain (per best practices)
vpc domain 10
role priority 2
system-priority 8096
peer-keepalive destination [….] source [….] vrf
management
auto-recovery
auto-recovery reload-delay
delay restore 30
ip arp synchronize
17
7K-1
7K-2
feature lacp
feature vpc
feature lacp
feature vpc
vlan 1 – 200
vlan 1 – 200
peer-link
interface port-channel 1
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type network
vpc peer-link
5K-1
interface port-channel 1
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type network
vpc peer-link
5K-2
OOB
interface e1/1 - 2
channel-group 2 force mode active
interface e1/1 - 2
channel-group 2 force mode active
Step 1 :: enable vPC peer-link on the L2 port channel
Always perform VLAN pruning on vPC peer-link with the
allowed list of vPC VLANs; vPC VLANs must also be
pruned on the vPC member port s as well
© 2013 Cisco and/or its affiliates. All rights reserved.
Bridge Assurance is enabled by default when configuring
vPC peer-link (spanning-tree port type network); Do NOT
disable it on the vPC peer-link
18
Configure vPC member port as spanning-tree port type normal
Keep Spanning Tree protocol root function on the
aggregation layer of the network; For each vPC peer device,
configure root guard on ports connected to access devices
interface port-channel 10
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type normal
spanning-tree guard root
vpc 10
7K-1
7K-2
interface e1/13 , e2/13
channel-group 10 force mode active
interface port-channel 10
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type normal
spanning-tree guard root
vpc 10
interface e1/13 , e2/13
channel-group 10 force mode active
1/13 2/13
1/13 2/13
port-channel load-balance src-dst ip-l4port-vlan
port-channel load-balance src-dst ip-l4port-vlan
vPC 10
interface port-channel 10
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type normal
vpc 10
interface e1/9 , e1/10
channel-group 10 force mode active
port-channel load-balance src-dst ip-l4port-vlan
1/9
1/10
5K-1
1/10
5K-2
Step 1 :: enable vPC on the member ports
Step 2 :: enable spanning-tree port configurations
Step 3 :: change port channel load balancing method
The configuration of the vPC member port must match on both vPC peer devices. If
there is a inconsistency, a VLAN or the entire port channel may suspend (depending
on type-1 or type-2 consistency check for the vPC member port). Use the same vPC
ID as the port channel ID for ease of configuration, monitoring, and troubleshooting
© 2013 Cisco and/or its affiliates. All rights reserved.
1/9
interface port-channel 10
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type normal
vpc 10
interface e1/9 , e1/10
channel-group 10 force mode active
port-channel load-balance src-dst ip-l4port-vlan
Use source-destination, IP, L4 port and VLAN as fields
for the port channel load balancing hashing algorithm;
this improves fair usage of all member ports forming in
the port channel
19
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type normal
spanning-tree port guard root
vpc 20
7K-1
7K-2
interface e3/13
channel-group 20 force mode active
interface e3/13
channel-group 20 force mode active
3/13
3/14
3/13
interface port-channel 30
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type edge trunk
vpc 30
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
3/14
interface port-channel 30
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type edge trunk
vpc 30
vPC 20
interface e3/14
channel-group 30 force mode active
interface port-channel 20
switchport
switchport mode trunk
switchport trunk allowed vlan 1-200
spanning-tree port type normal
spanning-tree port guard root
vpc 20
1/25 1/26
vPC 30
interface e3/14
channel-group 30 force mode active
Step 1 :: enable vPC on the member ports + enable spanning-tree port configurations accordingly
interface e1/25 , e1/26
channel-group 20 force mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
20
feature lacp
feature fex
Notice in the 5k/2k EvPC topology you DON’T need the vPC
command under the port channel towards the server
fex 100
pinning max-links 1
fex 199
pinning max-links 1
7K-1
feature lacp
feature fex
fex 100
pinning max-links 1
fex 199
pinning max-links 1
7K-2
interface port-channel 100
switchport mode fex-fabric
vpc 100
fex associate 100
interface port-channel 100
switchport mode fex-fabric
vpc 100
fex associate 100
vPC 10
interface port-channel 199
switchport mode fex-fabric
vpc 199
fex associate 199
interface e1/28
channel-group 100
interface e1/29
channel-group 199
interface port-channel 1000
switchport mode trunk
switchport trunk allowed vlan 10, 20
spanning-tree port type edge trunk
interface e100/1/1 , e199/1/1
channel-group 1000 force mode active
© 2013 Cisco and/or its affiliates. All rights reserved.
5K-1
5K-2
1/28
1/29
FEX 199
interface e1/29
channel-group 199
1/28
FEX 100
100/1/1
1/29
interface e1/28
channel-group 100
vPC 199
vPC 100
Po 1000
interface port-channel 199
switchport mode fex-fabric
vpc 199
fex associate 199
199/1/1
interface port-channel 1000
switchport mode trunk
switchport trunk allowed vlan 10, 20
spanning-tree port type edge trunk
interface e100/1/1 , e199/1/1
channel-group 1000 force mode active
21
install feature-set fex
Default VDC Only
feature lacp
feature-set fex
Straight-Through Topology
(only supported topology between 7k & 2k FEX)
7K-1
7K-2
install feature-set fex
feature lacp
feature-set fex
fex 199
pinning max-links 1
fex 199
pinning max-links 1
interface port-channel 199
switchport mode fex-fabric
fex associate 199
interface port-channel 199
switchport mode fex-fabric
fex associate 199
interface e5/28, e6/28
switchport mode fex-fabric
fex associate 199
channel-group 199
interface port-channel 1000
switchport mode trunk
switchport trunk allowed vlan 10, 20
spanning-tree port type edge trunk
vpc 1000
5/28 6/28
5/28 6/28
Po 199
Po 199
FEX 199
FEX 199
199/1/1
199/1/1
vPC 1000
interface e199/1/1
channel-group 1000 force mode active
Default VDC Only
interface e5/28, e6/28
switchport mode fex-fabric
fex associate 199
channel-group 199
interface port-channel 1000
switchport mode trunk
switchport trunk allowed vlan 10, 20
spanning-tree port type edge trunk
vpc 1000
interface e199/1/1
channel-group 1000 force mode active
Notice in the 7k/2k Straight-through topology you need the
vPC command under the port channel towards the server
FET is an optical transceiver that provides a highly cost-effective solution for connecting FEX to its parent switch (7k, 5k, 6k). Note that FET can only
be used to connect Fabric links between the Fabric Extender and the parent switch; FET-10G must be connected to another FET-10G)
© 2013 Cisco and/or its affiliates. All rights reserved.
22
feature interface-vlan
feature hsrp
feature interface-vlan
feature hsrp
interface port-channel 80
switchport mode trunk
switchport trunk allowed vlan 100, 200
spanning-tree port type edge trunk
vpc 80
interface e6/13
channel-group 80 force mode active
7K-1
7K-2
interface e6/13
channel-group 80 force mode active
6/13
6/13
vPC 80
interface vlan 200
ip address 20.20.20.5/24
no ip redirect
hsrp 200
preempt
priority 110
ip 20.20.20.254
0/0
See VMDC Architecture
for more virtual firewall
configuration use cases
and best practices
0/1
ASA-5585-X
interface vlan 200
ip address 20.20.20.6/24
no ip redirect
hsrp 200
preempt
ip 20.20.20.254
ip route 10.10.10.0/24 20.20.20.1
ip route 10.10.10.0/24 20.20.20.1
Subnet 10.10.10.0 /24 is
serviced by the ASA in
this example
interface port-channel 80
switchport mode trunk
switchport trunk allowed vlan 100, 200
spanning-tree port type edge trunk
vpc 80
interface GigabitEthernet0/0, Ge0/1
channel-group 80 mode active
no nameif
no secruity-level
no ip address
interface port-channel 80
port-channel load-balance vlan-src-dst-ip
no nameif
no secruity-level
no ip address
© 2013 Cisco and/or its affiliates. All rights reserved.
interface port-channel 80.100
vlan 100
nameif inside
secruity-level 99
ip address 10.10.10.1 255.255.255.0 standby 10.10.10.2
interface port-channel 80.200
vlan 200
nameif outside
secruity-level 1
ip address 20.20.20.1 255.255.255.0 standby 20.20.20.2
route outside 0.0.0.0 0.0.0.0 20.20.20.254
23
Separate Layer 3 (routed traffic) and Layer 2 (bridged traffic)
infrastructure. Use dedicated Layer 3 point-to-point link between the
vPC peer devices for backup path to core
Use a dedicated Layer 2 port-channel trunk for non-vPC
VLAN and create dedicated VLAN/SVI to established a
Layer 3 relationship (note those VLANS are not on the
peer-link – manually pruned off)
CAN’T Dynamically route over a vPC – road mapped in version 7.x
Firewalls attached in a vPC; use static routing
1. ASA static route to HSRP on Nexus
2. Nexus static route to ASA VIP
© 2013 Cisco and/or its affiliates. All rights reserved.
Firewalls attached in a VRF sandwich; separate vPC attachment
24
featue lacp
feature ospf
feature interface-vlan
feature hsrp
featue lacp
feature ospf
feature interface-vlan
feature hsrp
vlan 1 – 200
7K-1
interface loopback0
ip address [….]/32
3/32
7K-2
1/32
2/32
1/32
2/32
3/32
router ospf 1
router-id [….]
log-adjacency-changes detail
auto-cost reference-bandwidth 100Gbps
vPC 10
interface port-channel 5
ip address [….]/30
ip router ospf 1 area 0.0.0.0
ip ospf network point-to-point
interface e1/32, e2/32
channel-group 5 force mode active
interface vlan 100
ip address [10.10.10.2]/24
no ip redirects
ip router ospf 1 area 0.0.0.10
ip ospf passive-interface
hsrp 100
preempt
priority 110
ip [10.10.10.1]
© 2013 Cisco and/or its affiliates. All rights reserved.
interface loopback0
ip address [….]/32
router ospf 1
router-id [….]
log-adjacency-changes detail
auto-cost reference-bandwidth 100Gbps
interface port-channel 5
ip address [….]/30
ip router ospf 1 area 0.0.0.0
ip ospf network point-to-point
interface e3/32
ip address [….]/30
ip router ospf 1 area 0.0.0.0
ip ospf network point-to-point
vlan 1 – 200
interface e1/32, e2/32
channel-group 5 force mode active
5K-1
5K-2
Use dedicated Layer 3 point-to-point link between the vPC
peer devices for backup path to core
Define the SVI associated with HSRP as passive routing
interface in order to avoid forming routing adjacency over
vPC peer-link
Define vPC primary peer device as the active HSRP
instance and vPC secondary peer device as the standby
HSRP (from control plane standpoint) for ease of operations
Disable ip redirect (no ip redirect) on the interface VLAN
where HSRP is configured
interface e3/32
ip address [….]/30
ip router ospf 1 area 0.0.0.0
ip ospf network point-to-point
interface vlan 100
ip address [10.10.10.3]/24
no ip redirects
ip router ospf 1 area 0.0.0.10
ip ospf passive-interface
hsrp 100
preempt
ip [10.10.10.1]
25
Failure 1 :: Peer-Keepalive fails
Failure 2 :: Peer-Link fails on Aggregation
Failure 3 :: Peer-Link fails on Access Layer
Failure 4 :: Peer-Keepalive fails + Peer-Link fails (Split Brain)
nothing happens – no traffic loss
Role
Primary
Role
Operational
Secondary Primary
vPC member ports are shut down and all the vPC VLAN
interfaces (SVIs) are shut down; meaning no more L3
advertisements – all this happens on the secondary vPC peer
device
When PKL link fails and PL fails ( in this order ) , you have a dual
active situation , while both links are down , the primary vPC peer
device remains primary and your secondary vPC device becomes
operational primary
In a vPC environment only operational primary switch behaves as
STP root and processes BPDU and your secondary switch do not
process BPDUs ( this is regardless of whichever switch is configured
as STP root )
Existing flows continue to be forwarded as before the failure; but new
flows learning are impaired and uncertain forwarding (or broken state)
for new flows will be observed.
So when the links comes back up, the originally primary switch will
see that, there is an existing operational primary switch (originally
secondary) which is behaving like an STP root and processing
BPDUs
If the originally primary switch tries to reclaim the primary role at this
point, that would mean more convergence time while operational root
role is being switched, hence we do not try to reclaim the vpc primary
(and acting STP root ) role back to avoid more convergence times
© 2013 Cisco and/or its affiliates. All rights reserved.
Role
Primary
Role
Secondary
Black hole traffic to single attached
devices connected to vPC Peer
device with secondary role
26
Failure 1 :: Peer-Link fails on FEX Parent Switch at Access Layer
Failure 2 :: Single FEX Fails or Power Loss
Role
Primary
Role
Secondary
No traffic loss –
Only the vPC members are shut down northbound facing the
Aggregation Layer and the NIF interfaces are lost on the FEX
facing the secondary vPC peer device – all traffic will be
forward from both FEXs to the primary vPC peer device
Single Attached hosts connected to the FEX are unaffected
Role
Primary
Role
Secondary
5K Parent switches have lost communication to the failed FEX;
resulting all host traffic will forward out the secondary FEX.
Minimal to no traffic loss when hosts are dual attached in
LACP; Active / Standby NIC teaming will failover over to the
secondary FEX
Black hole traffic to devices
connected to single FEX
Active / Standby NIC teaming will
failover over to the secondary FEX
© 2013 Cisco and/or its affiliates. All rights reserved.
27
Without Bridge Assurance
Bridge Assurance prevents a spanning-tree domain from failing in an “open” state. When a
port configured for Bridge Assurance stops receiving BPDU’s, the port transitions into a
“blocking” state as opposed to remaining in a “forwarding” state. This “closed” state reduces
the likelihood for mis-configured devices from creating STP loops.
‘spanning-tree bridge assurance’ is enabled by default for all ‘network’ port types
Specifies bi-directional transmission of
BPDUs on all ports of type “network”.
With Bridge Assurance
Protects against unidirectional links and peer
switch software issues
Provides IGP like hello-dead timer behavior
for Spanning Tree
In all versions of NX-OS, available in IOS on
the Catalyst 6500 beginning 12.2(33) SXI
Recommended in STP topologies
Not recommend in vPC topologies; only on
the peer-link (default)
© 2013 Cisco and/or its affiliates. All rights reserved.
28
There are two types of consistency checks :
Type-1 :: Puts peer device or interface into a suspended state to prevent invalid packet forwarding behavior. With vPC Graceful
Consistency check, suspension occurs only on the secondary peer device.
Type-2 :: Peer device or Interface still forward traffic; however they are subject to undesired packet forwarding behavior.
Type 1 and Type 2 consistency check apply both for global configuration and for vPC interface configuration.
show vpc consistency-parameters global – (displays global type-1 consistency parameters)
Parameter Name
Value
Spanning Tree Protocol (STP) mode
RPVST or MST
STP Enable/Disable state per VLAN
Yes / No
STP region configuration for MST
Region name, revision, instance to VLAN mapping
STP global settings
Bridge Assurance settings
Port type settings
Loop guard settings
BPDU filter settings
MST simulate PVST enable / disable
show vpc consistency-parameters interface port-channel [id] – (displays interface type-1 consistency parameters)
Parameter Name
Value
Port channel LACP mode
ON, ACTIVE, PASSIVE
Link speed & duplex per port channel
Speed in mpbs & Half / Full duplex
Switchport mode per port channel
Trunk / Access, native VLAN
STP interface settings
Port type setting
Loop Guard
Root Guard
MST Simulate PVS
Enable / Disable
MTU per port channel
Maximum transmission Unit (MTU) value
© 2013 Cisco and/or its affiliates. All rights reserved.
29
© 2013 Cisco and/or its affiliates. All rights reserved.
30
If any of the vPC Type-2 parameters listed in the table below are not configured identically on both vPC peer devices, the inconsistent
configuration can cause undesirable behavior in the traffic flow
Type-2 consistency check parameters
Parameter Name
Value
MAC aging timers
MAC aging timer for a particular VLAN should be the same on both vPC peer devices
Static MAC entries
Static MAC entries in a particular VLAN should be applied on both vPC peer devices
VLAN interface (switch virtual interface [SVI])
Each peer device must have a VLAN interface configured for the same VLAN on both ends, and this VLAN
interface must be in the same operational state
ACL Configuration and parameters
ACL configuration should be identical on both vPC peer devices
QoS Configuration and parameters
QoS configuration should be identical on both vPC peer devices
STP interface settings
BPDU filter, Link type (auto, point-to-point, shared), Cost, Port-priority, STP interface setting should be identical
on both vPC peer devices
VLAN Database
You must create all VLANs on both the primary and secondary vPC peer devices, or the VLAN will be
suspended. Those VLANs configured on only one peer device do not pass traffic using the vPC or vPC peer-link
Port security
NAC, Dynamic ARP Inspection, IP source guard, port security must be identical on both vPC peer devices
Cisco TrustSec
Cisco TrustSec configuration should be identical on both vPC peer devices
DHCP snooping
DHCP snooping configuration should be identical on both vPC peer devices
IGMP snooping
IGMP snooping configuration should be identical on both vPC peer devices
HSRP
HSRP configuration should be identical on both vPC peer devices
PIM
PIM configuration should be identical on both vPC peer devices
GLBP
GLBPconfiguration should be identical on both vPC peer devices
All routing protocol configurations
Routing configuration should be consistent on both vPC peer devices
© 2013 Cisco and/or its affiliates. All rights reserved.
31
• Always use different domain ID in double-sided vPC topology
• Operations perspective, define vPC primary on the left Nexus and vPC secondary on the right Nexus (role priority)
• When configuring large number of VLANs in a vPC environment, use the range command (vlan x-z) vs. individually
configuring one at a time
• Create a dedicated VRF for the vPC peer-keepalive link (ie. vrf context PKAL)
• When building a vPC peer-link, follow these guidelines
• Must have Peer-keepalive link up first; ensure the peer-link member ports are 10 Gig interfaces
• Use a minimum of two 10 Gig ports (M1 up to 8 member ports & F1/F2 up to 16 member ports)
• Use at least two different line cards to increase high availability of peer-link
• Use dedicated mode 10 Gig ports with M1 32 line card vs. shared mode ports
• Split vPC and non-vPC VLANs on different interswitch port channels
• Don’t insert any device between vPC peers; a peer-link is a point-to-point link
• Any vPC VLAN allowed on the vPC member port MUST be allowed on the vPC peer-link
• Always perform VLAN pruning on vPC peer-link with allowed list of vPC VLAN; vPC VLAN must have been pruned on the
vPC member port previously
• If the M1 32 is used for both the vPC peer-link and L3 uplinks to L3 Core, use vPC object tracking feature
• When building a vPC member port, follow these guidelines
• The configuration of the vPC member port must match on both vPC peer devices
• If there is a inconsistency, a VLAN or the entire port channel may suspend (depending on type-1 or type-2
consistency check for the vPC member port)
• Use the same vPC ID as port channel ID for ease of configuration, monitoring, and troubleshooting
• With the M1 Series line card :: there can be up to 8 active ports bundled – resulting a 16-way port channel to be
built for the whole vPC
• With the F1/F2 Series line card :: there can be up to 16 active ports bundled – resulting a 32-way port channel to be
built for the whole vPC
• Do not mix different port types (M1, F1, F2) in the same vPC member port; this is not allowed by the software
• Both sides of the vPC member ports must be of the same port type
© 2013 Cisco and/or its affiliates. All rights reserved.
32
• The vPC peer-keepalive link carries periodic heartbeat (UDP 3200) between vPC peer devices. It is used at the boot up of
the vPC systems to guarantee both peer devices are up before forming vPC domain and also when vPC peer-link fails to
down state; in the latter case, vPC peer-keepalive link is leveraged to detect split brain scenario (both vPC peer devices
are active–active) [when vPC peer-link is down, there is no more real time synchronization between the 2 peer devices so
vPC systems must react to this active-active situation; this is done by shutting down vPC member ports on secondary peer
device].
• The vPC peer-keepalive link is a pre-requisite for the vPC domain to form initially (ie. prior to the vPC peer-link
configuration + if peer-link is initial up before peer-keepalive is up)
• vPC has 3 timers; hold-timeout (default 3 sec), timeout (default 5s), hello interval (default 1s). The hold-timeout starts
once the vPC peer link goes to a down state; during this time period the secondary vPC peer will ignore any peerkeepalive hello messages. During the timeout period, the secondary vPC peer device will look for vPC peer-keepalive
hello messages from the primary vPC peer device. If a single hello is received, the secondary vPC peer concludes that
there must be a dual active scenario and therefore will disable all its vPC member ports (that is, all port-channels that carry
the keyword vpc). Command line configuration to modify vPC timers is (under vPC domain configuration context): peerkeepalive destination ipaddress [source ipaddress | hold-timeout secs | interval msecs {timeout secs}] The default
values are ok in most situations.
vPC peer link is down !
Keepalive Hold Timeout
Keepalive Timeout
© 2013 Cisco and/or its affiliates. All rights reserved.
33
• Always enable vPC peer-gateway in the vPC domain (on both peer devices), even if there is no end device using this
feature (devices that don’t perform standard ARP request for their default IP gateway), there is no side effect enabling it
• (Corner Case) always use vPC peer-gateway exclude-vlan when a transit VLAN (over vPC peer-link) is used in the vPC
domain, this is applicable only for mixed chassis mode (M1/F1) with peer-link on F1 ints; note only static routing supported
• Always enable vPC ARP sync on both vPC peers; performs a bulk ARP sync, improves convergence time for L3 flows
• Always enable vPC delay restore on both vPC peer devices and tune the timer according based on the network profile
• Always enable vPC graceful type-1 check on both vPC peer devices; graceful consistency-check; (enabled by default)
• Always enable vPC auto-recovery on both vPC peer devices
• Always enable vPC auto-recovery reload-delay on both vPC peer devices (note the vPC auto-recovery reload-delay
deprecates the previous feature called vPC reload restore)
• Use vPC orphan port suspend when single-attached devices connected to a vPC domain need to be disconnected from
the network when vPC peer-link fails
• Always use a different domain ID in a double-sided vPC topology; once configured, both peer devices use the vPC domain
ID to automatically assign a unique vPC system MAC address; which is used as part of the LACP protocol
• vPC role is non-preemptive so vPC operational role is the most relevant of the information per table below
• With NX-OS 6.1 and prior releases, always use identical line cards on either side of the vPC Peer Link and vPC member
ports (legs to downstream device)
• Starting in NX-OS 6.2, always use identical line cards on either side of the vPC Peer Link and vPC member ports (legs to
downstream device) when M1/M2 & F2E
• Starting in NX-OS 6.2, VDC type must match between the 2 vPC peer devices when F2 & F2E are used in same VDC;
meaning its ok to have F2 on vPC peer device 1 and F2E on vPC peer device 2 for the vPC Peer Link or vPC member
ports. Note: in a F2 & F2E type of design; only features related to F2 apply (lowest common denominator)
© 2013 Cisco and/or its affiliates. All rights reserved.
34
•
•
•
•
•
•
•
•
•
•
•
•
•
Use LACP protocol when connecting access devices to vPC domain (channel-group [x] mode active
Use LACP when available for graceful failover and misconfiguration protection
LACP mode active on both sides of the port channel
If access device does not support LACP, use manual bundling (channel-group [x] mode on)
If the downstream access switch is a Cisco Nexus device, enable LACP graceful-convergence (its on by default)
If the downstream access switch is NOT is a Cisco Nexus device, disable LACP graceful-convergence
Use source-destination, IP, L4 port and VLAN as fields for the port channel load balancing hashing algorithm; this improves
fair usage of all member ports forming in the port channel
When possible, always dual-attach access devices to a vPC domain using a port channel
When connecting a single-attached access device to a vPC domain using a vPC VLAN, always connect it to the vPC
primary peer device; reason is when if the vPC peer-link fails down any single attached device connected to the secondary
peer device (and using vPC VLAN) will become completely isolated with the rest of the network
Single Attached Recommendations (descending order of priority):
• Connect access device to an intermediate switch which is dual-attached to a vPC domain
• Connect single-attached device to a vPC domain using non-vPC VLAN (must also create an inter-switch link
between the 2 peer devices to transport non-vPC VLAN
• Connect single-attached device to a vPC domain using vPC VLAN and leveraging vPC peer-link
In a double-sided vPC topology, all interconnect links between the 2 vPC domains MUST belong to the same vPC ID; all
links form a unique vPC (on both sides of the 2 vPC domains)
LACP port suspend :: By default, LACP sets a port to the suspended state if it does not receive an LACP PDU from the
peer (ie a server or host). In some cases, although this feature helps in preventing loops created due to misconfigurations,
it can cause servers to fail to boot up because they require LACP to logically bring up the port. You can put a port into an
individual state by using the lacp suspend-individual command.
On the Nexus 5000 this feature is disabled (no lacp suspend-individual) for servers connecting via LACP; on the Nexus
7000 this feature is enabled by default (lacp suspend-individual)
© 2013 Cisco and/or its affiliates. All rights reserved.
35
• Recommended Spanning Tree Protocol Configuration with vPC
• Spanning Tree protocol must remain enabled for all VLANs (even if all access devices are vPC attached to the vPC
domain); Do NOT disable spanning-tree protocol
• Use MST with vPC if you need to build a large L2 domain; Plan ahead to avoid future configuration changes that
can trigger vPC type-1 consistency failure
• Implement consistent STP mode in the same L2 domain, ensure that all switch in your L2 domain are running with
Rapid-PVST+ (default) or MST to avoid slow Spanning Tree convergence (30 seconds or more)
• Perform VLAN pruning on vPC member ports to reduce internal resource consumption
• Keep Spanning Tree protocol root function on the aggregation layer of the network (aggregation vPC domain)
• For each vPC peer device, configure root guard on ports connected to access devices
• Bridge Assurance is enabled by default when configuring vPC peer-link (spanning-tree port type network); Do NOT
disable it on the vPC peer-link
• It is not necessary to enable Bridge Assurance on the vPC (members ports in the vPC) – configure vPC member
port as spanning-tree port type normal
• Configure port fast (edge or edge trunk port type) on the host facing interfaces to avoid slow Spanning Tree protocol
convergence (30 seconds or more) when port transitions to an up state
• Configure BPDU guard on host facing interfaces to block any BPDU sent from the host (access switch port
receiving the BPDU will be put in errdisable mode) – enable BPDU guard globally
• Always define the vPC domain as the STP root for all VLANs in that domain (configure aggregation vPC peer
devices as the STP root primary and STP root secondary) – enforce this rule with root guard on vPC peer device
ports connected to another L2 switch
• IF the vPC peer-switch is activated, both vPC peer devices MUST have the SAME spanning tree configuration
(same priority for all vPC VLANs) – recommendation to activate vPC peer-switch in the environment
• Do not enable Loop guard on vPC (disabled by default)
• When using vPC peer-switch in a hybrid environment use the spanning-tree pseudo-information to load balance
VLANs across the 2 peer devices
• Enable UDLD in normal mode on vPC peer-link and vPC member ports
© 2013 Cisco and/or its affiliates. All rights reserved.
36
• Layer 3 and vPC Guidelines and Recommendations
• Use separate Layer 3 link(s) to connect to L3 devices (like a router or firewall in routed mode) to a vPC domain; use
individual Layer 3 links for routed traffic and a separate Layer 2 port-channel for bridged traffic if both routed and
bridged traffic are required
• Always build a Layer 3 backup routed path for the vPC domain in order to increase network resilience and
availability; use an OSPF point-to-point adjacency (or equivalent L3 protocol) between the 2 vPC peer devices to
establish a L3 backup path to the core in case of uplink failure
• Do NOT use a Layer 2 vPC to attach Layer 3 devices to a vPC domain unless the Layer 3 device can statically
route to the HSRP address configured on the vPC peer devices
• You can’t dynamically route over a vPC
• Layer 3 backup routing path options (descending order of preference)
• Use dedicated Layer 3 point-to-point link between the vPC peer devices for backup path to core
• Use a dedicated Layer 2 port-channel trunk for non-vPC VLAN and create dedicated VLAN/SVI to
established a Layer 3 relationship (note those VLANS are not on the peer-link)
• HSRP / VRRP Guidelines and Recommendations
• When running HSRP/VRRP in active-active mode (data plane standpoint), aggressive timers can be relaxed; use
the default HSRP/VRRP timers
• Define the SVI associated with HSRP/VRRP as passive routing interface in order to avoid forming routing adjacency
over vPC peer-link
• Define vPC primary peer device as the active HSRP/VRRP instance and vPC secondary peer device as the
standby HSRP/VRRP (from control plane standpoint) for ease of operations
• Disable ip redirect (no ip redirect) on the interface VLAN where HSRP/VRRP is configured.
• Do NOT use HSRP/VRRP object tracking in a vPC domain
© 2013 Cisco and/or its affiliates. All rights reserved.
37
• Recommendations for Multilayer vPC for DCI Solution
•
•
•
•
•
•
•
•
•
•
•
Use different vPC domain-id for each vPC domain (DC1: vPC domain for aggregation, vPC for DCI. DC2: vPC domain for aggregation,
vPC for DCI)
For each data center, interconnect the aggregation vPC domain to the DCI vPC domain using a vPC (double-sided topology)
Interconnect the 2 data centers using a vPC (vPC between DCI vPC domain in site 1 and site 2)
Enable BPDU filter on the vPC used for DCI (under the port-channel configuration, activate the following command: spanning-tree
bpdufilter enable) to avoid BPDU propagation
Configure the vPC used for DCI as spanning-tree port type edge (i.e port fast) to fasten port state forwarding mode when port is
operationally up
Remember by default vPC peer-link runs in spanning-tree port type network i.e bridge assurance is activated on the link
Configure root guard on aggregation vPC domain (more exactly on vPC between this vPC domain and DCI vPC domain). STP root must
remain on aggregation vPC domain on each side of the data center
No loop must exist outside the vPC domains.
Do not use Layer 3 peering between data centers (in other words, there is no Layer 3 over vPC).
Do not use bridge assurance for interconnect vPC (DCI vPC) – use spanning-tree port type edge trunk
Use M1 ports for DCI vPC if flows between the 2 data centers need to be encrypted using 802.1ae MACsec
© 2013 Cisco and/or its affiliates. All rights reserved.
38
• Best Practices for Network Services / Appliances and vPC
• Configure vPC to the inside and outside interfaces for ASA firewalls – use spanning-tree port type edge trunk
• If needed, use multiple VRF instances for the inside interfaces – intra data center nets (see VMDC architecture)
• Be aware of the following Layer 3 over vPC design caveat
• Use dedicated Layer 2 port-channel for the service appliances state and keepalive VLANs (recommend don’t use
the vPC peer-link)
• Recommended the ASA port channel hashing algorithm and the Nexus vPC hashing algorithm are the same
• Connect ASA in routed mode to a vPC – must use static routing
• ASA static route to HSRP on Nexus
• Nexus static route to ASA VIP
• If Connected ASA in routed mode and use dynamic routing
• Single attach ASA to vPC domain
• Create separate non-vPC interswitch link
• Peer with non-vPC VLAN/SVIs
SLB attached via vPC
SLB attached via Po
with orphan port suspend
Firewall attached via vPC
& Static Routes
Firewall attached via
non-vPC Po & Dynamic
Routing
Bandwidth reduced during
certain failure scenarios
© 2013 Cisco and/or its affiliates. All rights reserved.
Bandwidth
maintained
during certain
failure scenarios
39
Interop F2 & F2E VDC
With NX-OS 6.1 and Prior Releases ::
• Always use identical line cards on either side of the vPC Peer Link
and vPC member ports (legs to downstream device)
• The F1-series line cards can mix with M-series line cards
• The F2-series line cards have to be in their own VDC; VDC type [F2]
meaning they can’t mix with F1 or the M-series in the same VDC
© 2013 Cisco and/or its affiliates. All rights reserved.
40
Starting in NX-OS 6.2 and Later Releases ::
• VDC type [F2, F2E, F2 F2E] must match between the 2 vPC
peer devices when F2 & F2E are used in same VDC; meaning
its ok to have F2 on vPC peer device 1 and F2E on vPC peer
device 2 for the vPC Peer Link or vPC member ports
• Note: in a F2 & F2E type of design; only features related to
F2 apply (lowest common denominator)
•
•
© 2013 Cisco and/or its affiliates. All rights reserved.
Always use identical line cards
on either side of the vPC Peer
Link and vPC member ports
when M1, M1-XL, M2 & F2E in
same VDC [M-F2E] or system
When F2E is placed in a
chassis with M-series it will
operate in Layer 2 mode only
leveraging the M for Layer 3
(proxy L3 forwarding)
41
External (public)
Great External
Resource
Nexus vPC best practices design guide
http://www.cisco.com/en/US/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf
Nexus 7000/6000/5000 Configuration Guides
http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html
http://www.cisco.com/en/US/products/ps9670/products_installation_and_configuration_guides_list.html
http://www.cisco.com/en/US/partner/products/ps12806/products_installation_and_configuration_guides_list.html
Nexus 5000 Enhanced vPC Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_enhanced_vpc.html
© 2013 Cisco and/or its affiliates. All rights reserved.
42