Everyone
in this
room is a
GENIUS
2
What are Best Practices ?
Learning from Others
Mistakes
3
Learning from your
mistakes makes you
SMART
Learning from others
mistakes makes you
GENIUS
4
vPC Best Practices and
Design on NXOS
Nazim Khan, CCIE#39502 (DC/SP)
Network Consulting Engineer, Data Center Group
BRKDCT-2378
Session Focus
• Best Practices and Designs for vPC
• Nexus 2000 (FEX) will only be
addressed from vPC standpoint
• Fabricpath / vPC+ Overview
• vPC with FCOE
• vPC with VXLAN
• vPC with ACI
.
Pick the great from the good
We Are Not Covering
• vPC troubleshooting
• Scalability
• Fabricpath
• vPC+
• VXLAN
• FCoE
• ACI
Related Sessions at Cisco Live Berlin
Session Id
Session Name
BRKDCT-2404
VXLAN deployment models - A practical perspective
BRKDCT-3313
Fabricpath Operations and Troubleshooting
BRKDCT-2458
Nexus 9000/7000/6000/5000 Operations and
Maintenance Best Practices
BRKACI-2601
Real World ACI Deployment and Migration
BRKDCT-2333
Data Centre Network Failure Detection
9
Agenda
•
Feature Overview
•
Configuration Best Practices
•
Design Best Practices
•
vPC Operations and Upgrade
•
vPC with Fabric Technologies
•
Reference Material
10
Data Center Technology Evolution
MPLS, OTV,
LISP
MPLS, OTV,
LISP
ACI
VXLAN
FabricPath with vPC+
FEX with vPC
VPC
2014-2015
STP
2013-2014
2010
2010
2009
2008
11
Why vPC?
13
there’s
something
about
vPC
14
Role of vPC in the Evolution of Data Center
•
vPC launched in 2009
•
Deployed by almost 95% of Nexus customers
•
Used to redundantly connect network entities at the
edge of the Fabric
Unified Fabric
−
Dual-homed servers (bare metal, blades, etc.)
− Network services (Firewalls, Load Balancers, etc.)
15
Agenda
•
Feature Overview
−
−
Concepts and Benefits
Terminology
16
vPC Feature Overview
vPC Concept & Benefits
S1
S2
S1
S3
STP
•
•
S2
S2
S1
S3
vPC Physical Topology
S3
vPC Logical Topology
No Blocked Ports, More Usable Bandwidth, Load Sharing
Fast Convergence
17
Feature Overview
vPC Terminology
Layer 3 Cloud
vPC
Peer
Orphan
Port
vPC Peer
Keepalive Link
vPC Domain
Peer-Link
S1
CFS
S2
vPC Member
Port
vPC
Orphan
Device
S3
18
For Your
Reference
vPC Failure Scenario
vPC Peer-Keepalive Link up & vPC Peer-Link down
vPC peer-link failure (link loss):
P
vPC Peer-keepalive
S
• vPC peer-keepalive up
• Status of other vPC peer known
S1
S2
vPC_PLink
• Both peers Active
Suspend secondary
vPC Member Ports
• Secondary vPC peer disables all vPC’s
• Traffic from vPC primary.
• Orphan devices connected to secondary peer will
be isolated
vPC1
vPC2
SW4
SW3
Keepalive Heartbeat
P
Primary vPC
S
Secondary vPC
19
vPC Failure Scenario – Dual Active
For Your
Reference
vPC Peer-Keepalive down followed by vPC Peer-Link down
1. vPC peer-keepalive DOWN
2. vPC peer-link DOWN
3. DUAL-ACTIVE or SPLIT BRAIN
P
S1
• vPC primary peer remains primary and
secondary peer becomes operational primary
role
• Result in traffic loss / uncertain traffic behavior
• When links are restored, the operational
primary (former secondary) keeps the primary
role & former primary becomes operational
secondary
P
S
vPC Peer-keepalive
S2
vPC_PLink
Traffic Loss / Uncertain Traffic
Behavior
vPC1
vPC2
SW3
SW4
P Primary vPC
S
Secondary vPC
20
Agenda
•
vPC Configuration Best Practices
−
−
−
−
−
−
−
−
−
Building a vPC domain
Domain-ID
Peer-Link
Peer-Keepalive Link
Spanning-Tree
Peer-switch
Private VLAN (PVLAN)
Auto-recovery
Object tracking
21
vPC Configuration Best Practices
Building a vPC domain – Configuration Steps
1. Define domains
S1
S2
2. Establish Peer Keepalive connectivity
3. Create a Peer link
CFS
4. Create vPCs
5. Make Sure Configurations are Consistent
(Order does Matter!)
S3
22
vPC Configuration Best Practices
vPC Domain-ID
•
•
The vPC peer devices use the vPC domain ID to
automatically assign a unique vPC system MAC
address
vPC Domain 10
S1
S2
You MUST use unique Domain id’s for all vPC
pairs defined in a contiguous layer 2 domain
! Configure the vPC Domain ID – It should be unique within the layer 2
domain
NX-1(config)# vpc domain 20
vPC Domain 20
S4
S3
! Check the vPC system MAC address
NX-1# show vpc role
<snip>
vPC system-mac
: 00:23:04:ee:be:14
S5
23
vPC Configuration Best Practices
vPC Peer-Link
S1
S2
S3
•
•
•
•
S2
S1
S3
vPC Peer-link should be a point-to-point connection
Peer-Link member ports can be 10/40/100GE interfaces
Peer-Link bandwidth should be designed as per the vPC
vPC imposes the rule that peer-link should never be blocking
24
vPC Configuration Best Practices
vPC Peer-Keepalive link
Preference
Recommendations
(in order of
preference):
Nexus 7X00 /
9500 series
Nexus 9300 /6000 /
5X00 / 3X00 series
1
Dedicated link(s)
(1GE/10GE LC)
mgmt0 interface
2
mgmt0 interface
Dedicated link(s)
(1GE/10GE LC)
3
L3 infrastructure
L3 infrastructure
25
vPC Configuration Best Practices
For Your
Reference
vPC Peer-Keepalive link – Dual Supervisors
Management Switch
•
When using dual supervisors and mgmt0 interfaces
to carry the vPC peer-keepalive, DO NOT connect
them back to back between the two switches
Management
Network
vPC_PKL
vPC_PKL
vPC_PL
•
Only one management port will be active a given point
in time and a supervisor switchover may break keepalive connectivity
•
Use the management interface when you have an outof-band management network (management switch in
between)
vPC1
vPC2
Standby Management Interface
Active Management Interface
26
vPC Configuration Best Practices
Spanning Tree (STP)
STP is running to manage
loops outside of vPC domain,
or before initial vPC
configuration !
S1
S2
S4
S3
S5
•
•
All switches in Layer 2 domain should run either Rapid-PVST+ or MST
Do not disable spanning-tree protocol for any VLAN
•
Always define the vPC domain as STP root for all VLAN in that domain
27
vPC Configuration Best Practices
vPC Peer-Gateway
• Allows a vPC switch to act as the active
gateway for packets addressed to the peer
router MAC
S1
S2
• Keeps forwarding of traffic local to the vPC node
and avoids use of the peer-link
• Allows Interoperability with features of some NAS
or load-balancer devices
S3
S4
N7k(config-vpc-domain)# peer-gateway
28
vPC Configuration Best Practices
vPC Peer-switch
Primary
vPC
Secondary
vPC
Without Peer-switch
BPDUs
•STP for vPCs controlled by vPC primary.
•vPC primary send BPDU’s on STP designated ports
•vPC secondary device proxies BPDU’s to primary
With Peer-switch
• Peer-Switch makes the vPC peer devices to appear as a
single STP root
• BPDUs processed by the logical STP root formed by the 2
vPC peer devices
Primary
vPC
Secondary
vPC
N7k(config-vpc-domain)# peer-switch
29
vPC Configuration Best Practices
PVLAN on vPC
•
PVLAN configuration across both VPC switches
should be identical
•
PVLAN configuration not supported on Peer-Link
•
Type-1 Compatibility Check
• Port mode is a type-1 check
• vPC leg brought down if PVLAN port mode
different on vPC legs
•
Type-2 Compatibility Check
• PVLAN will bring down mismatched tuple
vPC Primary
S1
vPC Secondary
P
P
PVLANPROMISC
(3500, 3501)
S2
PVLANPROMISC
(3500, 3501)
C
Community
VLAN
Note : This feature is currently not supported on N9X00
30
vPC Configuration Best Practices
PVLAN VPC type 1 Consistency Check
vPC Primary
vPC Secondary
vPC Primary
S1
S2
S1
Pvlan
Promiscuous
trunk
S2
I
P
P
Pvlan Isolated trunk
S3
vPC Primary
I
S3
vPC Secondary
S1
S2
I
Type 1 Consistency
Failure
vPC Secondary
T
S3
31
vPC Configuration Best Practices
PVLAN VPC type 2 Consistency Check
vPC Primary
vPC Secondary
vPC Primary
S1
S2
S1
PVLANPROMISC
(10, 201)
I
S3
PVLANPROMISC
(10, 201)
vPC Primary
Type 2 Consistency
Failure
S2
P
P
vPC Secondary
Secondary
Trunk (2,31)
(3,30), (4,100)
I
S3
Secondary
Trunk (2,31)
(3,30), (4,100)
vPC Secondary
S1
S2
I
Secondary
Trunk (3,31)
(2,30), (4,100)
I
S3
Secondary
Trunk (2,31)
(3,30), (4,100)
32
vPC Configuration Best Practices
vPC auto-recovery
P
Operational
Primary
S
S
P
P
S2
S1
S3
S1
S2
S1
S3
S2
S3
1. vPC peer-link down : S2 - secondary shuts all its vPC member ports
2. S1 down : vPC peer-keepalive link down : S2 receives no keepalives
3. After 3 keepalive timeouts, S2 changes role and brings up its vPC
P vPC Primary
S vPC Secondary
33
vPC Configuration Best Practices
vPC auto-recovery
For Your
Reference
Auto-recovery addresses two cases of single switch behavior
•Peer-link fails and after a while primary switch (or keepalive link) fails
•Both VPC peers are reloaded and only one comes back up
How it works
•If Peer-link is down on secondary switch, 3 consecutive missing peer-keepalives will
trigger auto-recovery
•After reload (role is ‘none established’) auto-recovery timer (240 sec) expires while
peer-link and peer-keepalive still down, autorecovery kicks in
•Switch assumes primary role
•VPCs are brought up bypassing consistency checks
Nexus(config)# vpc domain 1
Nexus(config-vpc-domain)# auto-recovery
34
vPC Configuration Best Practices
Why Object-Tracking ?
•
S5
S4
Modules hosting peer-link and uplink fail on
the vPC primary
Primary
•
Peer-Link is down and vPC Secondary
shut all its vPC
•
Auto-Recovery does not kick in as peerkeepalive link is active
•
Traffic is black holed
Secondary
S1
S2
S3
35
vPC Configuration Best Practices
Object-tracking
•
vPC object tracking, tracks both peer-link and
uplinks in a list of Boolean OR
• Object Tracking triggered when the track object
goes down
• Suspends the vPCs on the impaired device
• Traffic forwarded over the remaining vPC peer
! Track
track 1
! Track
track 2
track 3
the vpc peer link
interface port-channel11 line-protocol
the uplinks
interface Ethernet1/1 line-protocol
interface Ethernet1/2 line-protocol
S4
S5
S1
S2
! Combine all tracked objects into one.
! “OR” means if ALL objects are down, this object will go down
track 10 list boolean OR
object 1
object 2
object 3
! If object 10 goes down on the primary vPC peer,
! system will switch over to other vPC peer and disable all local vPCs
vpc domain 1
track 10
S3
36
vPC Configuration Best Practices
Spanning Tree Bridge Assurance
Stopped receiving
BPDUS!
Root
Malfunctioning
switch
BPDUs
Network
Network
BA Inconsistent
Network
Network
BPDUs
BPDUs
Network
Network
BA Inconsistent
Blocked
Stopped receiving
BPDUS!
Edge
Edge
%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port Ethernet2/48
VLAN0700
switch# show spanning vl 700 | in -i bkn
Eth2/48
Altn BKN*4
128.304 Network P2p *BA_Inc
Spanning Tree Bridge Assurance
For Your
Reference
Almost like a routing protocol…
•
Turns STP into a bidirectional protocol
•
Ensures spanning tree fails “closed” rather than “open”
•
All ports with “network” port type send BPDUs regardless of state
•
If network port stops receiving BPDUs, port is placed in BA-Inconsistent state
(blocked)
%STP-2-BRIDGE_ASSURANCE_BLOCK: Bridge Assurance blocking port
Ethernet2/48 VLAN0700.
switch# sh spanning vl 700 | in -i bkn
Eth2/48
Desg BKN*4
128.304 Network P2p *BA_Inc
38
vPC Configuration Best Practices
vPC & Bridge Assurance (BA)
• STP Bridge Assurance is enabled by default on vPC Peer-Link
• DON’T disable Bridge Assurance on vPC Peer-link
• NO Bridge Assurance on vPC member ports (even with peer-switch)
39
For Your
Reference
vPC Configuration Best Practices
Unidirectional Link Detection (UDLD)
•
Light-weight Layer 2 failure detection protocol
•
Designed for detecting:
•
One-way connections due to physical or soft failure
• Mis-wiring detection (loopback or triangle)
•
Cisco proprietary, but listed in informational RFC 5171
•
Runs on any single Ethernet link, even inside bundle
•
Centralized implementation in switching platforms
•
Message interval: 7 - 90 sec (default: 15 seconds)
•
Detection: 2.5 x interval + timeout value (4 sec)  ~ 41 sec
Rx
Tx
Rx
Tx
40
vPC Configuration Best Practices
UDLD with vPC
• UDLD NOT recommended on vPC peer-link
• UDLD NOT recommended on vPC member ports if LACP is used
• UDLD only in normal mode on vPC member ports if required
41
Agenda
•
vPC Design Best Practices
−
Mixed Hardware across vPC Peers
−
FHRP with vPC
−
Hybrid topology (vPC and non-vPC)
−
vPC and Network Services
−
vPC Fex Supported Topologies
−
Physical port vPC
−
vPC as Data Center Interconnect (DCI)
−
Dynamic Routing over VPC
−
vPC and Multicast
42
Design Best Practices
Mixed Hardware across vPC Peers : Line Cards
Always use identical line cards on either sides of the peer link and VPC legs !
Examples
vPC Primary
vPC Secondary
vPC Peer-link
S1
N7000
F2E
S2
N7700
F2E
F3
vPC Primary
vPC Secondary
vPC Peer-link
S1
S2
M2
M1
F3
vPC
vPC
43
Design Best Practices
Mixed Hardware across vPC Peers : Nexus 9500
X
vPC Primary
vPC Secondary
vPC Peer-link
S1
N9500
X
S2
N9500
Y
X
Y
vPC
Y
N9K-X9636PQ
N9K-X9432PQ
N9K-X9564PX
N9K-X9464PX
N9K-X9564TX
N9K-X9464TX
N9K-X9536PQ
N9K-X9736PQ
vPC
44
Design Best Practices
Mixed Hardware across vPC Peers : Chassis & Supervisors
•
•
•
•
N7000 and N7700 in same vPC Construct -Supported
VDC type should match on both peer device
vPC peers can have mixed SUP version* (SUP1, SUP2, SUP2E)
N5500 and N5600 in same vPC Construct –Not Supported
vPC Primary
S1
N7000
vPC Secondary
S2
N7700
vPC Primary
vPC Secondary
S2
S1
N5500
N5600
*Recommended only for short period such as migration
45
Design Best Practices
FHRP with vPC
FHRP
“Active”:
Active for
shared L3 MAC
•
•
•
FHRP
“Standby”:
Active for
shared L3 MAC
S1
S2
S3
S4
FHRP in Active/Active mode with vPC
No requirement for aggressive FHRP timers
Best Practice : Use default FHRP timers
46
Design Best Practices
Use one transit vlan to establish L3 routing
backup path over the vPC peerlink in case L3
uplinks were to fail, all other SVIs can use
passive-interfaces
Backup Routing Path
•
•
•
•
Point-to-point dynamic routing protocol
adjacency between the vPC peers to
establish a L3 backup path to the core
through PL in case of uplinks failure
Define SVIs associated with FHRP as
routing passive-interfaces in order to avoid
routing adjacencies over vPC peer-link
A single point-to-point VLAN/SVI (aka
transit vlan) will suffice to establish a L3
neighbor
Alternatively, use an L3 point-to-point link
between the vPC peers to establish a L3
backup path
S3
S4
P
P
OSPF/EIGRP
L3
P
VLAN 99
P
OSPF/EIGRP
L2
S1
Primary
vPC
Secondary
vPC
S2
S5
P
Routing Protocol Peer
47
Hybrid topology (vPC and non-vPC)
STP Root
VLAN 1
VLAN 2
STP Root
VLAN 1
Bridge Priority
VLAN 1  4K
VLAN 2  8K
vPC Primary
STP Root
VLAN 2
vPC Secondary
S1
S2
peer-switch
VLAN 1
(blocked)
vPC1
S3
Bridge Priority
VLAN 1  8K
VLAN 2  4K
S4
VLAN 2
(blocked)
•
Supports hybrid topology where vPC and non-vPC are connected to the same vPC domain
•
Need additional configuration parameters : spanning-tree pseudo-information
•
STP pseudo configuration takes precedence over global STP configuration
48
Design Best Practices
ASA Cluster
Cluster
Control Link
Cluster
Data Link
ASA Cluster Mode
•
Use unique vPC for ASA Cluster Data Links to vPC domain
•
Use vPC per ASA device for Cluster Control Link (CCL) to vPC domain
•
Leverage peer-switch configuration
49
Nexus 2000 (FEX) Straight-Through Deployment with VPC
• Port-channel connectivity from the server
• Two Nexus switches bundled into a vPC
pair
S1
S2
Fabric Links
• Suited for servers with Dual NIC and
capable of running Port-Channel
HIF
Fex 100
HIF
Fex 101
VPC
50
Nexus 2000 (FEX)Active-Active Deployment with VPC
S1
•
•
•
Fabric Extender connected to two Nexus
5X00 / 6000
Suited for servers with Single NIC or
Dual NIC not having port-channel
capability.
Scale implications of less FEX per
system and less VPC
Note :
• This design is currently not supported on Nexus 9X00
• Nexus 7X00 will support this from release 7.2
S2
Fabric Links
Fex 101
Fex 100
HIF
HIF
51
Nexus 2000 (FEX) Active-Active Scale &
Limitations (N7X00)
•
N7X00 can support up to 64 FEXs
•
N7X00 supports only 15 Active-Active FEX in 7.2(0)D1(1)
•
Straight-Through FEX and Active-Active FEX cannot exist on the
same ASIC instance
•
Layer 3 HIF ports are not supported with Active-Active FEX
•
Active-Active FEX is not supported with vPC+
Nexus 2000 (FEX) - Enhanced VPC
•
•
•
•
Port-channel connectivity to dual-homed
FEXs
From the server perspective a single access
switch with port-channel support – each line
card supported by redundant supervisors
Ideal design for a combination of single
NIC and Dual NIC servers with portchannel capability
Scale implications of less FEX per
system and less VPC
S2
S1
Fabric Links
Fex 100
Fex 101
HIF
HIF
Note :
This design is currently not supported on N7000 / N7700 and
N9X00
53
Nexus 2000 (FEX) Active-Active (Unsupported)
54
Physical Port vPC
vPC domain
vPC domain
FEX101
e101/1/1
Port-channel vPC
FEX101
FEX102
Po1
VPC1
VPC1
Po1
e101/1/1
e102/1/1
FEX102
VPC1
interface e101/1/1
switchport
vpc 1
lacp mode active
VPC1
e102/1/1
Physical port vPC
• vPC configuration on a physical Layer 2 port as opposed to a port-channel
• Front panel ports and FEX ports connected to F2/F2e/F3 only
• Improves scaling as separate PC interface not created for single-link VPC leg
• Key benefit: more than 1000 host facing VPCs with FEX
55
vPC - Data Center Interconnect(DCI)
DC 1
DC 2
E
-
F
Long Distance
Dark Fiber
F
E
CORE
CORE
vPC domain 11
vPC domain 21
-
N
N
N
N
N
Network port
E
Edge or portfast
-
Normal port type
B
BPDUguard
F
BPDUfilter
R
Rootguard
802.1AE (Optional)
- R
F E
R
R
-
-
N
N
-
N
R
R
N
-
vPC domain 10
vPC domain 20
R
R
-
-
E
E
B
B
Server Cluster
ACCESS
ACCESS
E F
AGGR
AGGR
-
R
Server Cluster
56
Design Best Practices
vPC as Data Center Interconnect (DCI)
PROS
• vPC is easy to configure and it provides robust and resilient interconnect solution
CONS
• Maximum of only two Data Centers can be interconnected
• Layer 3 peering between Data Centers cannot be done through vPC and separate
links are required
57
Design Best Practices
vPC -Data Center Interconnect (DCI)
•
vPC Domain id for vPC layers should be UNIQUE
•
BPDU Filter on the edge devices to avoid BPDU propagation
•
STP Edge Mode to provide fast Failover times
•
No Loop must exist outside the vPC domain
•
No L3 peering between Nexus 7000 devices (i.e. pure layer 2)
58
Dynamic routing over vPC ?
59
Dynamic routing over vPC
Use Case 1 : Firewall at Aggregation layer
L3 Cloud
•
Peering Firewalls in routed mode over vPC
•
Firewalls may be in active-standby mode
•
Static routing / L3 P2P links NOT required
•
S1
External and internal traffic traverse same
port channel to firewall.
S2
FW-A
FW-B
Dynamic Peering Relationship
60
Dynamic routing over vPC
Use Case 2 : Remote Orphan Site Peering in DCI Deployment
• vPC as Data Center Interconnect (DCI)
• Each Switch has routing adjacency with both
vPC device in other DC
Remote Site 1
S1
Remote Site 2
S2
• Each DC connected to a remote site by
orphan port
• Remote sites forms routing adjacency with
both peers of its directly connected DC
S3
S4
61
Dynamic Routing over vPC
New Supported Designs
Dynamic routing over vPC
Supported Designs
Layer 3 over DCI - vPC
Layer 3 services devices with vPC
P
P
P
P
P
P
P
Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2.
Supported on Nexus 9X00 in ACI mode
Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card
63
Dynamic routing over vPC
Supported Designs
STP inter-connection using a vPC VLAN
P
P
Orphan device with vPC peers over vPC VLAN
P
P
P
P
Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2.
Supported on Nexus 9X00 in ACI mode
Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card
64
Dynamic routing over vPC
Supported Designs
Peering with vPC peers over FEX vPC host interfaces
P
P
P
Note : Supported only in Nexus 7X00 on F3 and F2E Line Cards starting from release 7.2(0)D1(1)
Currently not supported on Nexus 5X00, Nexus 3X00, Nexus 9X00 (standalone mode), Nexus 7000 M-series Line card
65
Dynamic Routing over vPC
Unsupported Designs
66
Dynamic routing over vPC
Unsupported Design
B
Peering across vPC interfaces with unequal L3
metrics
•
•
•
SVI
Router2
Int VLAN 20
The routing metric on S1 is less than the routing
metric on S2 (preferred path using S1).
Traffic from A to B may hash to S2. This traffic
will need to traverse to peer-link to get to B
through S1.
Po2
Int VLAN 20
S2
Int VLAN 20
S1
Po100
Int VLAN 10
Metric 10
Int VLAN 10
Metric 20
Due to the vPC loop avoidance rule S1 will not
allow traffic to flow to B.
Po1
Router1
Int VLAN 10
SVI
A
67
Dynamic routing over vPC
Configuration
L3 over vPC Configuration on Nexus 7x00 platform
Command: Layer3 peer-router
Mode: config-vpc-domain
Default: Disabled
Need to configure on BOTH the
peers
•Requirements
•
•
•
•
Command configured on both the peers.
“Peer-Gateway” should be enabled.
Peer link should be up.
Both peer should run image supporting L3 over vPC feature.
•Auto Enabling “Peer-Gateway”
• If “Layer3 peer-router” command is enabled without “Peer-Gateway” a syslog will be
displayed to enable “Peer-Gateway”.
68
Dynamic routing over vPC
Example Configuration and Verification on Nexus 7x00
vpc domain 200
peer-keepalive destination
10.10.12.42 source 10.10.12.52
peer-gateway
layer3 peer-router
P
P
show vpc brief
Peer Gateway : Enabled
Operational Layer3 Peer : Enabled
(output truncated for display)
vpc domain 200
peer-keepalive destination
10.10.12.52 source 10.10.12.42
peer-gateway
layer3 peer-router
show vpc brief
Peer Gateway : Enabled
Operational Layer3 Peer : Enabled
(output truncated for display)
P
69
Benefits of Dynamic Routing over vPC
• No Static routes
• No Parallel links
• No design changes and loss of business
• Route peering across vPC’s over existing infrastructure
• Routing between vPC DCI
• Most wanted by majority vPC customers
70
Dynamic Routing over vPC
Devices without L3 over vPC support
•
Don’t attach routers to VPC domain via L2 port-channel
•
Common workarounds:
• Individual L3 links for routed traffic
• Static route to FHRP VIP
A
SVI 1
IP Z
VIP A
SVI 1
IP Y
VIP A
S2
S1
SVI 2
IP X
B
SVI 1
IP Z
VIP A
SVI 1
IP Y
VIP A
S1
L3 ECMP
SVI 2
IP X
Router
S2
Router
SVI 1
IP Z
VIP A
SVI 1
IP Y
VIP A
S1
S2
SVI 2
IP X
Router
Static Route to VIP A
71
Design Best Practices
vPC and Multicast
• vPC supports PIM-SM only
Source
• vPC uses CFS to sync IGMP state
• Sources in vPC domain
− both vPC peers are forwarders
− Duplicates avoided via vPC loop-avoidance logic
S1
Source
S2
• Sources in Layer 3 cloud
− Active forwarder elected on unicast metric
− vPC Primary elected active forwarder in case metric
are equal
Receivers
72
Agenda
•
vPC Operations and Upgrade
−
−
−
−
vPC Self Isolation
vPC Shutdown
Graceful Insertion and Removal
ISSU / ISSD with vPC
73
vPC Configuration Best Practices
vPC Self-Isolation
Error
Triggered
Operational
Primary
ISOLATED
P
S
P
Self- Isolate
S
P
S2
S1
S3
S1
S2
S1
S2
S3
S3
1. Error Triggered : All Line cards Fail or All Vlans’s down on peer-link
2. S1 sends “self-isolation” message through the peer-keepalive
3. S2 takes over as operational Primary and S1 is isolated from the vPC domain
P
vPC Primary
S vPC Secondary
74
vPC Configuration Best Practices
Example Configuration and Verification on Nexus 7x00
vPC domain 100
peer-keepalive destination
10.126.216.44
peer-gateway
self-isolation
vPC domain 100
peer-keepalive destination
10.126.216.41
peer-gateway
self-isolation
sh vPC brief
vPC domain id
: 100
Self-isolation
: Enabled
(output truncated for display)
sh vPC brief
vPC domain id
: 100
Self-isolation
: Enabled
(output truncated for display)
75
vPC Configuration Best Practices
vPC Self-Isolation
• vPC self-isolation is turned OFF by default
• No Impact on vPC operation if sellf-isolation enabled
• Functional only when enabled on both vPC peers.
• Not part of vPC type-1 and type-2 consistency checks
76
vPC Configuration Best Practices
vPC Shutdown
•
Isolates a switch from the vPC complex
•
Isolated switch can be debugged, reloaded, or
even removed physically, without affecting the
vPC traffic going through the non-isolated switch
Primary
Secondary
vPC
S2
S1
switch# configure terminal
switch(config)# vpc domain 100
switch(config-vpc)# shutdown
S3
77
Graceful Insertion and Removal
Change window begins
vPC
vPC
system mode maintenance
One command!
Pre-change System Snapshot
78
Graceful Insertion and Removal
Change window complete
vPC
vPC
system mode normal
One command!
Pre/Post-change Snapshot Comparison
79
Graceful Insertion and Removal
•
Flexible framework providing a comprehensive, systemic method to isolate a
node.
•
Configuration profile foundation in NX-OS
•
Initial support for:
•
•
•
•
•
•
•
vPC/vPC+
ISIS
OSPF
EIGRP
BGP
Interface
Per VDC on Nexus 7x00
Platform
Release
Nexus 5x00/6000
NX-OS 7.1
Nexus 7x00
NX-OS 7.2
Nexus 9000
NX-OS 7.X
80
ISSU / ISSD with vPC
•
ISSU is the recommended system upgrade in a
multi-device vPC environment
•
vPC system can be independently upgraded with
no disruption to traffic
•
Upgrade is serialized and must be run one peer at
a time (config lock will prevent synchronous
upgrades)
•
Configuration is locked on “other” vPC peer during
ISSU
•
Similar process of downgrades (ISSD)
•
Check ISSU / ISSD compatibility matrix & ensure
ISSU is supported from current to target release
5.2(x) / 6.2(x)
81
Agenda
•
vPC with Fabric Technologies
−
vPC with Fabricpath (vPC+)
−
vPC with FCOE
−
vPC with VXLAN
−
vPC with ACI
82
FabricPath: an Ethernet Fabric
Shipping on Nexus 7x00, Nexus 600x and Nexus 5x00
FabricPath
•
•
•
•
•
Eliminates Spanning tree limitations
High resiliency, fast network re-convergence
Any VLAN, Anywhere in the Fabric
Connect a group of switches using an arbitrary topology
With a simple CLI, aggregate them into a Fabric
N7K(config)# interface ethernet 1/1
N7K(config-if)# switchport mode fabricpath
83
VPC vs VPC+
Architecture of vPC and FabricPath with vPC+
CE
FP
CE Port
FP Port
CE VLAN’s
FP VLAN’s
vPC
vPC+
• Physical architecture of vPC and vPC+ is the same from the access edge
• Functionality/Concepts of vPC and vPC+ are the same
• Key differences are addition of Virtual Switch ID and Peer Link is a FP Core Port
• vPC+ is not supported on Nexus 9X00 & Nexus 3X00 Series
84
Dynamic Routing over vPC+
• Layer 3 devices can form routing adjacencies with
both the vPC+ peers over vPC
Fabricpath Core
• The peer link ports and VLAN are configured in
FabricPath mode.
• N55xx, N56xx, N6000 support this design with
IPv4/IPv6 unicast and PIM-SM multicast
vPC
P
P
• This design is not supported on N7X00
N55xx, N56xx,
N6000
Router/ Firewall
P
Fabricpath Link
Dynamic Peering Relationship
Routing Protocol Peer
P
85
vPC with FCoE
Unified Fabric Design
•
vPC with FCoE is ONLY supported between hosts and
N5X00 or N5X00 & N2232 pairs.
•
Must follow specific rules:
•
•
A ‘vfc’ interface can only be associated with a
single-port port-channel.
•
While the port-channel configurations are the
same on both switches, the FCoE VLANs are
different.
FCoE VLANs are ‘not’ carried on the vPC peer-link
(automatically pruned):
•
LAN Fabric
Fabric A
VLAN 10 ONLY HERE!
Nexus 5000
FCF-A
Nexus 5000
FCF-B
VLAN 10,20
STP Edge Trunk
FCoE and FIP ethertypes are ‘not’ forwarded
over the vPC peer link.
•
vPC carrying FCoE between two FCF’s is NOT
supported.
•
Best Practice: Use static port channel rather than LACP
with vPC and boot from SAN.
[If NX-OS is prior to 5.1(3)N1(1)]
Fabric B
VLAN 10,30
vPC contains only 2 X 10GE
links – one to each Nexus 5X00
86
Why VXLAN ?

Problems being addressed:
•
VLAN scale – VXLAN extends the L2 segment ID field to 24-bits, potentially
allowing for up to 16 million unique L2 segments over the same network
•
Layer 2 segment elasticity over Layer 3 boundary – VXLAN encapsulates L2
frame in IP-UDP header

High Level Technology Overview:
•
MAC-in-UDP encapsulation.
•
Leverages multicast in the transport network to simulate flooding behavior for
broadcast, unknown unicast and multicast in the same segment
•
Leverage ECMP to achieve optimal path usage over the transport network
87
For Your
Reference
VXLAN Packet Format
16
16
Reserved
16
VNID
16
Reserved
32
8 Bytes
VXLAN
RRRR1RRR
32
Checksum
0x0000
UDP
Src. Port
16
VXLAN Port
Outer
Dst. IP
8
Outer
Src. IP
72
FCS
8 Bytes
Header
Checksum
Protocol
0x11
16
Original
FCS L2 Frame
20 Bytes
IP Header
Misc Data
16
Ether Type
0x0800
16
VLAN ID
Tag
VLAN Type
48
0x8100
Src.
MAC Addr.
Dst.
MAC Addr.
14 Bytes
(4 bytes optional)
48
VXLAN
Header
UDP Header
UDP Length
Outer
IP Header
Outer
Mac Header
8
24
24
8
•
VXLAN is a Layer 2 overlay scheme over a Layer 3 network.
•
VXLAN uses Ethernet in UDP encapsulation
•
VXLAN uses a 24-bit VXLAN Segment ID (VNI) to identify Layer-2 segments
88
VXLAN Terminology
VTEP – Virtual Tunnel End Point
Transport IP Network
VTEP
•
•
VTEP
IP Interface
IP Interface
Local LAN Segment
Local LAN Segment
End System
End System
End System
End System
VXLAN terminates its tunnels on VTEPs (Virtual Tunnel End Point).
VTEP has two interfaces :
1. Bridging functionality for local hosts
2. IP identification in the core network for VXLAN encapsulation / de-encapsulation.
89
vPC VTEP
•
When vPC is enabled an ‘anycast’ VTEP
address is programmed on both vPC
peers
•
Multicast topology prevents BUM traffic
being sent to the same IP address across
the L3 network (prevents duplication of
flooded packets)
•
vPC peer-gateway feature must be
enabled on both peers
•
VXLAN header is ‘not’ carried on the vPC
Peer link
VXLAN
vPC VTEP
vPC VTEP
VLAN
90
VXLAN & VPC
For Your
Reference
VPC Configuration
VTEP1
vlan 10
vn-segment 10000
Map VNI to VLAN
interface loopback 0
ip address <VTEP individual IP – orphan)
ip address <VTEP anycast IP – per VPC domain> secondary
!
interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
Source Interface
individual IP is used for single attached Hosts
anycast IP is used for VPC attached Hosts
VXLAN Tunnel Interface
vtep
1
vtep
2
vtep
3
vtep
4
VTEP2
vlan 10
vn-segment 10000
interface loopback 0
ip address <VTEP individual IP - orphan>
ip address <VTEP anycast IP – per VPC domain> secondary
!
interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
H1
10.10.10.10
VLAN 10
(vpc)
H2
10.10.10.20
VLAN 10
(vpc)
91
VXLAN & VPC
For Your
Reference
VPC Configuration
VTEP1
vlan 10
vn-segment 10000
VTEP3
vlan 10
vn-segment 10000
interface loopback 0
ip address 1.1.1.1/32
ip address 1.1.1.201/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
interface loopback 0
ip address 1.1.1.3/32
ip address 1.1.1.202/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
vtep
1
vtep
2
vtep
3
vtep
4
VTEP2
vlan 10
vn-segment 10000
VTEP4
vlan 10
vn-segment 10000
interface loopback 0
ip address 1.1.1.2/32
ip address 1.1.1.201/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
interface loopback 0
ip address 1.1.1.4/32
ip address 1.1.1.202/32 secondary
!
Interface nve1
source-interface loopback0
member vni 10000 mcast-group 235.1.1.1
H1
10.10.10.10
VLAN 10
(vpc)
H2
10.10.10.20
VLAN 10
(vpc)
92
VXLAN & VPC
Dual attached Host to dual attached Host (Layer-2)
•
Host 1 (H1) and Host 2 (H2) are dual
connected to a VPC domain
•
As H1 is behind a VPC interface, the
anycast VTEP IP is the source for
the the VXLAN encapsulation
•
vtep
1
vtep
2
vtep
20
vtep
3
vtep
4
vtep
30
As H2 is behind a VPC interface, the
anycast VTEP IP is the target
H1
10.10.10.10
VLAN 10
(vpc)
H2
10.10.10.20
VLAN 10
(vpc)
93
Nexus 9000 + APIC = ACI
APIC
APIC
APIC
94
ACI uses a
policy based approach
that focuses on
the application.
QoS
QoS
QoS
Filter
Service
Filter
Web
App
DB
External
Network
95
vPC and ACI
ACI fabric utilised for control-plane
•
vPC
peers
vPC
Domains
No dedicated peer-link between vPC peers:
Fabric itself serves as the MCT
•
No out-of-band mechanism to detect peer
liveliness:
Due to rich fabric-connectivity (leaf-spine), it is
very unlikely that peers will have no active
path between them
•
CFS (Cisco Fabric Services) is replaced by
Zero Message Queue (ZMQ)
•
As ACI fabric is VXLAN-based, an anycast
VTEP is shared by both leaf switches in a
vPC domain
ACI
fabric
vtep
1
vtep
2
vtep
3
vPC
vPC
96
Agenda
•
Reference Material
97
Reference Material
For Your
Reference
•
vPC Best Practices Design Guide:
http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guid
e.pdf
•
vPC design guides:
http://www.cisco.com/en/US/partner/products/ps9670/products_implementation_design_guides_list.html
•
vPC and VSS Interoperability white Paper:
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/white_paper_c11_589890.html
•
VXLAN Overview :
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.html
•
Fabrcipath whitepaper :
http://www.cisco.com/c/en/us/products/collateral/switches/nexus-7000-series-switches/white_paper_c11-687554.html

ACI Overview
http://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/aci-fabric-controller/white-paper-c11-729587.html
98
Key Take-Aways
vPC in 2016
VXLAN, ACI, Fabricpath
VXLAN
• L2 segment scalability
• VTEP redundancy with
vPC
vPC Benefits
ACI
• No Blocked Ports
• High availability
• Fast Convergence
• Policy Based
• Fabric for vPC control
plane
Fabricpath
FCoE
•
•
•
Eliminates Spanning-Tree *
High resiliency
vPC+ for legacy switches,
servers, hosts
• Unified Fabric for LAN &
SAN
99
Call to Action
•
Visit the World of Solutions for
•
Cisco Campus
• Walk in Labs
• Technical Solution Clinics
•
Meet the Engineer
•
Lunch and Learn Topics
•
DevNet zone related sessions
100
Complete Your Online Session Evaluation
•
Please complete your online session
evaluations after each session.
Complete 4 session evaluations
& the Overall Conference Evaluation
(available from Thursday)
to receive your Cisco Live T-shirt.
•
All surveys can be completed via
the Cisco Live Mobile App or the
Communication Stations
101
Many Things
there’s Something About
vPC
102
Thank you
103