#CLMEL
Your First Seven Days
Of ACI
Takuya Kishida – Technical Leader, Service
BRKACI-1001
#CLMEL
Agenda
•
Day 1: Why ACI?
•
Day 2: Infrastructure and Policies
•
Day 3: Forwarding Overview
•
Day 4: Network Centric Migrations
•
Day 5: Multi Location Deployments
•
Day 6: Troubleshooting Tools
•
Day 7: Additional Resources
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
3
Cisco Webex Teams
Questions?
Use Cisco Webex Teams (formerly Cisco Spark)
to chat with the speaker after the session
How
1 Open the Cisco Events Mobile App
2 Find your desired session in the “Session Scheduler”
3 Click “Join the Discussion”
4 Install Webex Teams or go directly to the team space
5 Enter messages/questions in the team space
cs.co/ciscolivebot#BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
4
Reference Slide Icon ➔
Acronyms/Definitions
Acronyms
Definitions
Acronyms
Definitions
ACI
Application Centric Infrastructure
SVI
Switch Virtual Interface
ACL
Access Control List
VIC
Virtual Interface Card
API
Application Programming Interface
VNID
Virtual Network Identifier
APIC
Application Policy Infrastructure Controller
VPC
Virtual Port-Channel
BD
Bridge Domain
VRF
Virtual Routing and Forwarding
COOP
Council of Oracle Protocol
VTEP
VXLAN Tunnel Endpoint
ECMP
Equal Cost Multi Pathing
VXLAN
Virtual Extensible LAN
EP
Endpoint
EPG
Endpoint Group
KVM
Keyboard, Video, and Mouse
MP-BGP
Multi Protocol BGP
pcTag
Policy Control Tag
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
5
Day 1: Why ACI?
Challenges of Traditional Network
Complicated
Topology
Core/Dist/Access
layer separation
CLI
to every Device
Harder
as we scale
Lots of copy and paste
#CLMEL
Spanning Tree for
Loop Free Topology
No default
security isolation
Static Configuration
No Automation
Coordination between Network
and Server Team
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
No such challenges with ACI !!
Spine1# show module
Mod Ports Module-Type
Model
Status
--- ----- ----------------------------------- --------------- -----2
32
32p 40/100G Ethernet Module
N9K-X9732C-EX
ok
Spine
Complicated
Topology
22
23
24
26
27
28
Simple
Core/Dist/Access
Topology
layer separation
SPINE/LEAF
Leaf
0
0
0
0
0
0
Fabric Module
Fabric Module
Fabric Module
Fabric Module
Supervisor Module
Supervisor Module
Spanning Tree for
Loop Free Topology
N9K-C9504-FM-E
ok
N9K-C9504-FM-E
ok
N9K-C9504-FM-E
ok
N9K-C9504-FM-E
ok
N9K-SUP-A
Active
N9K-SUP-A
Standby
No default
security isolation
Leaf4# show module
Mod
--1
CLI
to every Device
Ports
----54
Module-Type
Model
Status
---------------------------------- ------------------ -----48x10/25G+6x40/100G Switch
N9K-C93180YC-EX
ok
ACI Switches are all Nexus 9000 with
ACI-OS instead of NX-OS
Harder
as we scale
Static Configuration
No Automation
Lots of copy and paste
#CLMEL
BRKACI-1001
Coordination between Network
and Server Team
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
8
No such challenges with ACI !!
Spine
Simple
Topology
SPINE/LEAF
Leaf
CLI
to every Device
Infra
VRF
No STP
Spanning
Tree
for
ECMP
Routing
between
Loop
Free
LEAF
& Topology
SPINE
No default
security isolation
Static Configuration
No Automation
Coordination between Network
and Server Team
Harder
as we scale
Lots of copy and paste
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
No such challenges with ACI !!
Spine
Simple
Topology
SPINE/LEAF
Leaf
CLI
to every Device
Infra
VRF
EPG
No STP
EPG
Spanning
Tree
for
ECMP
Routing
between
Loop
Free
LEAF
& Topology
SPINE
No List
default
White
Model
security
isolation
with contracts
Static Configuration
No Automation
Coordination between Network
and Server Team
Harder
as we scale
Lots of copy and paste
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
10
No such challenges with ACI !!
Spine
Simple
Topology
SPINE/LEAF
Leaf
APIC
Cluster
APIC
APIC
Infra
VRF
EPG
No STP
EPG
Spanning
Tree
for
ECMP
Routing
between
Loop
Free
LEAF
& Topology
SPINE
No List
default
White
Model
security
isolation
with contracts
Static Configuration
No Automation
Coordination between Network
and Server Team
APIC
Centralised
CLI
Controller
Harder
to every Device
as we scale
API Access
Lots of copy and paste
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
No such challenges with ACI !!
Spine
Simple
Topology
SPINE/LEAF
Leaf
APIC
Cluster
APIC
APIC
Infra
VRF
EPG
No STP
EPG
ECMP Routing between
LEAF & SPINE
No List
default
White
Model
security
isolation
with contracts
REST
automation
StaticAPI
Configuration
Dynamic
Integration
No Automation
Coordination between Network
and Server Team
APIC
Centralised
Controller
API Access
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
No such challenges with ACI !!
Spine
Simple
Topology
SPINE/LEAF
Leaf
APIC
Cluster
APIC
APIC
Infra
VRF
EPG
No STP
EPG
ECMP Routing between
LEAF & SPINE
White List Model
with contracts
REST API automation
Dynamic Integration
Coordination
Network
Vmwarebetween
Integration
and Server
etc… Team
APIC
Centralised
Controller
API Access
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
R L3 Routing
ACI Overview
V VLAN
Fabric Discovery
GW
Gateway (SVI)
T Infra IP (Tunnel Endpoint: TEP)
T
T
ISIS/BGP Overlay
T
T
T
T
APIC
GW
GW
Bare Metal
T
APIC
V
T
T
T
APIC
R
T
External
L2 & L3
Network
APIC Cluster
Hypervisors
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
T Tunnel Endpoint (TEP)
ACI overlay network
ISIS
MP-BGP
T
T
T
T
IP reachability between TEPs
R
RR
BGP Route Reflector
RR
R
R
WAN
(L3OUT)
L3OUT Routes distribution
• APIC assigns TEP
• APIC assigns RR (Route Reflector)
• Automatically establish ISIS
• Automatically establish BGP
No manual ISIS config.
No ISIS knowledge is required.
Routes from L3OUT
VxLAN
RR
T
R
No manual config is required.
(except for RR)
#CLMEL
BRKACI-1001
VxLAN VNID
Switching / Routing Separation
• APIC assigns VNID
No manual VxLAN config.
No VxLAN knowledge is required
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
15
APIC Controller
APIC
ACI
Fabric
OOB
MGMT
A
A
S
S
APIC Controller
(UCS C220)
1)
Cisco VIC 1225 (Copper or Fiber)
A – Active
2)
3)
Two 10Gb port for connections to ACI Switches
1Gb Copper Ethernet port for CIMC
S - Standby
4)
Console Port
5)
Two 1Gb Copper Ethernet Ports for OOB MGMT
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
Infrastructure and Policies
Best Practice
OOB
Network
ACI Spine Switches
1 OOB MGMT per SUP
1 Console per SUP
40/100 Gb connections to Leafs
ACI Leaf Switches
1 OOB MGMT
1 Console
40/100 Gb connections to Spines
Console
Server
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
Infrastructure and Services
Required Addressing
1
2
1.
Infra Subnet
2.
3.
Infra VLAN
BD Multicast Range
4.
OOB Network IP’s (CIMC
included)
3
4
#CLMEL
BRKACI-1001
NOTE: Infrastructure subnet and BD
MCAST is used internally for APICs and
Switches!
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
Management - Required Addressing
Planning
Requirements
Notes
Fabric Name
Has to be consistent on all APICs
Fabric1
Fabric ID
Set to 1 (Default)
1
TEP Pool
Recommended a /19 network. APIC will assign IPs from this pool to
Leafs, Spines and other Fabric specific services. Avoid IP space which
APIC might have to communicate with. E.g.: vCenter or other
integrated services
10.0.0.0/16
GIPO Pool
Multicast network for flooding inside ACI. Not exposed to external
network unless using Multipod
225.0.0.0/15
Infra VLAN
VLAN will be reserved for internal ACI communication. Cannot be
deployed toward user servers
3967
APIC OOB IP
1 IP per APIC, has to be out of band. Inband can be configured later.
Switch
Management IP
1 IP per switch, can have inband, out of band or both.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
19
Infrastructure and Services
APIC UI
APIC Management
APIC
APIC Cluster
APIC
API
CLI (ssh)
APIC
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
Take a first look at APIC Controller (Inventory)
APIC
ACI Fabric Nodes
1. Discover – Only S/N and Model show up first
2. Register – Configure Node ID and name
3. Provision – TEP IP is auto assigned.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
Take a first look at APIC Controller (Inventory)
Topology for each pod
BRKACI-1001
#CLMEL
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
APIC
Take a first look at APIC Controller (Dashboard)
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
APIC
23
Take a first look at APIC Controller (Dashboard)
APIC
APIC Cluster Status
“Fully Fit” – All APICs are in sync
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
Take a first look at APIC Controller (Dashboard)
APIC
Faults
Faults are indications of mis-config or any
issues on ACI Fabric
※ This is a lab setup. Try to clear all faults whenever
a new one is raised in production.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
Take a first look at APIC Controller (Dashboard)
APIC
Looks like we
had an issue!
Health Score
Health scores are based on faults and events
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
Day 2: Infrastructure and
Policies
Checklist
❑
❑
❑
❑
❑
CIMC
Management
NTP
AAA/RBAC
Backups
Infrastructure and Services
CIMC
• Use for APIC Hardware
Diagnostics and Remote
Access
• Use to install the APIC
Software
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
29
Infrastructure and Services
CIMC
• CIMC KVM Provides
Remote Access
• Equivalent of Console
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
Checklist
✓
❑
❑
❑
❑
CIMC
Management
NTP
AAA/RBAC
Backups
Infrastructure and Services
spine 1
spine 2
Switch Management
ACI Fabric
APIC
leaf 1
leaf 2
leaf 3
leaf 4
leaf 5
Leaf and Spine Access
- Console
- SSH – via APIC or Direct
- REST API
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/basic-configuration/Cisco-APIC-Basic32
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
#CLMEL
Configuration-Guide-401/Cisco-APIC-Basic-Configuration-Guide-401_chapter_0100.html
Checklist
✓
✓
❑
❑
❑
CIMC
Management
NTP
AAA/RBAC
Backups
Infrastructure and Services
NTP & PTP
NTP
• APIC Cluster sync
(timestamp in control plane messaging)
• Certificates
• Tech Supports
• Atomic Counters!
APIC
PTP
APIC
APIC
• Gen 2 or newer (EX/FX) Spine can act
as a PTP master as well
• Allows user to measure latency
between EndPoints and leafs
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/basic-configuration/Cisco-APIC-Basic34
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
#CLMEL
Configuration-Guide-401/Cisco-APIC-Basic-Configuration-Guide-401_chapter_0101.html
Checklist
✓
✓
✓
❑
❑
CIMC
Management
NTP
AAA/RBAC
Backups
Infrastructure and Services
AAA / RBAC
Supports various AAA solutions
Easy to check your permissions
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/basic-configuration/Cisco-APIC-Basic36
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
#CLMEL
Configuration-Guide-401/Cisco-APIC-Basic-Configuration-Guide-401_chapter_011.html
Infrastructure and Services
AAA / RBAC
mgmt IP reachability
AAA Server
Config/Policy
Spine
Leaf
Which AAA server?
etc
APIC
Cluster
APIC
APIC
APIC
Deploy via APIC
Each node (both APIC and Switch) still needs their own
management
IP reachability to an AAA server
#CLMEL
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
37
Infrastructure and Services
AAA
“Oh no! We lost connectivity to
servers on February 12th at 3pm EST!?”
jristain@apic1:~> moquery -c aaaModLR | grep -C 12 "2017-02-13T15"
<snip>
# aaa.ModLR
id
: 8589940567
affected
: uni/tn-Joey-Tenant/BD-Joey-BD3
cause
: transition
changeSet
: arpFlood (Old: no, New: yes), unkMacUcastAct (Old: proxy, New: flood)
childAction :
clientTag
:
code
: E4206171
created
: 2017-02-13T15:06:07.249+00:00
descr
: BD Joey-BD3 modified
dn
: subj-[uni/tn-Joey-Tenant/BD-Joey-BD3]/mod-8589940567
ind
: modification
modTs
: never
rn
: mod-8589940567
sessionId
: Ld0sxAcCRfmb2Qb+W+XbUg==
severity
: info
status
:
trig
: config
txId
: 4611686018449066821
user
: remoteuser-jristain
Logs changes per user!!
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
Checklist
✓
✓
✓
✓
❑
CIMC
Management
NTP
AAA/RBAC
Backups
Infrastructure and Services
Backups – Configuration Export
• The current fabric configuration/policy in JSON/XML
• Best practice for DISASTER RECOVERY
The backup has
Enabled -> Encrypted password
Disabled -> No password
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/4-x/basic-configuration/Cisco-APIC-Basic40
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
#CLMEL
Configuration-Guide-401/Cisco-APIC-Basic-Configuration-Guide-401_chapter_0100.html
Infrastructure and Services
Backups - Snapshots
Creates a Config Backup that is stored on the APIC by default
Run on a Per Fabric or Tenant Basis
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
41
Infrastructure and Services
Backups - Snapshots
• Rollback feature allows config
rollback between 2 snapshots
• Can also compare differences
between a previous SS
Object
Changed To
Changed From
Changed From
Changed To
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
42
Infrastructure and Services
CIMC, NTP, AAA, and Backup Planning
Requirements
Notes
CIMC IP per
APIC
Unique IP address used for IP KVM built into APIC. Must
use dedicated port
NTP Server
NTP Server which all nodes inside fabric will use
User
Management
TACAS/ RBAC or RADIUS Server for accounting. Custom
local user account can be used too
Scheduled
backup
Multicast network for flooding inside ACI. Not exposed
to external network unless using Multipod
Backup Server
Server outside of ACI Fabric running FTP, SFTP or SCP
Server
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
43
Checklist
✓
✓
✓
✓
✓
CIMC
Management
NTP
AAA/RBAC
Backups
Fabric and Tenant Policies
Tenant and Fabric Policies
• Fabric Policy – Physical Concept
•
•
•
port-channel
I/F speed
LLDP/CDP etc
• Tenant Policy – Network Logical Concept
•
•
•
VLAN trunk
EPG/BD/VRF
Contract etc
#CLMEL
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
Fabric and Tenant Policies
Access Policies
S10
S20
vPC Domain
vPC Domain
L1
Server
L3
L2
L4
Server
Nexus 7000
vPC Port-Channel
vPC Port-Channel
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
47
Fabric and Tenant Policies
Access Policies
Access policies refer to the configuration that is applied for physical and virtual (hypervisors/VMs)
devices attached to the fabric.
Broken into a few major areas:
Global Policy
Switch Policy
Interface Policy
•
•
Pools
Domains
•
•
Policies
Policy Groups
•
Policies
•
Policy Groups
•
Attachable Access Entity Profiles
•
Profiles
•
Profiles
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
48
Fabric and Tenant Policies
vPC Domain Policy
• No Peer-Link
• No Peer-Keepalive
• Uses Fabric Links for
Communication
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
Fabric and Tenant Policies
Access Policies
S10
S20
101-102 vPC
103-104 vPC
L1
Server
L3
L2
L4
Server
vPC Port-Channel
Nexus 7000
vPC Port-Channel
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
Fabric and Tenant Policies
Port-Channels
Legacy NXOS Config
Nexus7710# show run int po 10
interface port-channel10
switchport mode trunk
vpc 10
Nexus7710# show run interface Ethernet1/10
interface Ethernet1/10
speed 10000
lldp transmit
lldp receive
channel-group 10 mode active
Unspecified fields use
default values
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
51
Fabric and Tenant Policies
Access Policies
S10
S20
101-102 vPC
103-104 vPC
L1
Server
L3
L2
L4
Server
BareMetal-vPC
Nexus 7000
N7000-vPC
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
52
AEP
The AEP is used to associate a domain to one or more interface policy groups. In
most deployments it is recommended to use a single AEP if VMM integration is
not being used. If the ACI Fabric will be integrated with n VMM domains, use 1 +
n to determine how many AEPs are needed
The Domain is used to specify what type of path (vlan) can be deployed on a
interface. If a AEP does not contain a “External Routed Domain”, the interface
can not be used to deploy a L3Out.
In Most deployments a single VLAN pool can be used with 1 Physical Domain and
1 External Routed Domain.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
Relationship View
Access Policies Workflow Example
Switch Profile
Leaf-101
vPC-101-102
Interface Profile
Leaf-101
vPC-101-102
P1-5_WinAD
P6-7-N7K-vPC
Interface Selector
Interface Block
Interface Policy Group
Interface Policies
1/6-7
1/1-5
1/6-7
Win2016Serv
CDP_On
LLDP_Off
#CLMEL
N7K-vPC
BPDU_Guard
BRKACI-1001
LACP
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
54
Management - Required Addressing
Planning
Requirements
Notes
Example
AEP
1 AEP for all Policy groups. Map all domains to this Policy group
Prod_AEP
Domain
1 Physical Domain, 1 External Routed Domain
phys
L3Out
VLAN Pool
1 VLAN pool for all statically deployed vlans. 1 VLAN pool for
Dynamically deployed VLANs. These pools should not overlap.
Static_VLANs
VMM_Domain
Switch Profile
1 Profile per switch for Orphan Ports, 1 Profile per vPC Domain
(Containing both switches)
vPC-101-102, Leaf101, Leaf102
Interface Profile
Create a 1 to 1 mapping to switch Profile
vPC-101-102, Leaf101, Leaf102
Interface Selector
Name after Server, Include Port ID.
P11-N7710-vPC
Policy Group
1 Policy Group per Port-Channel/ vPC. Policy Groups can be reused for
access ports. Assign AEP to Policy Group
N7710-vPC
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Global Policy
Pools (Vlan / VXLAN)
A resource pool of encapsulations that can be
allocated within the fabric.
Domains (Physical / VMM / External Bridged / External Routed)
Administrative domain which selects a vlan/vxlan pool for allocation of
encaps within the domain
Attachable Access Entity Profiles (AEP)
Selects one or more domains and is referenced/applied by
interface policy groups.
#CLMEL
Pool1
Pool2
DomPhy1
DomL2Ext1
AEP
TenantA
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
Global Policy - Attachable Entity Profiles
Configuration:
•
Create a VLAN/VXLAN pool with a range of
encapsulations
•
Create a domain (physical, l2/l3 external, or
VMM) and associate pool
•
Associate domain to AEP
•
Associate interface policy group to AEP
switch/interface selectors will apply the config
through the interface policy group assign to
specific ports
What have we accomplished?
•
1
Pool1
Pool2
Pool3
Pool4
DomPhy1
DomVm1
DomL2
DomL3
AEP
Statics
AEP
VMs
2
3
4
1
2
3
AEP
External
4
1
2
3
4
1
2
3
4
Specified what domains and corresponding
pools are allowed per interface in the fabric!
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Access Policies
SWITCH POLICY
Policies define protocol / feature configurations
Policy Groups select which policies should be applied
Profiles associate policy groups to switches or interfaces, through the
use of selectors
Switch Policy Types:
Interface Policy Types:
VPC Domain
Link-level
Storm Control
Spanning-tree (MST)
BFD
CDP
LLDP
Data plane policing
MCP
Fibre-channel SAN / Node
Port-channel / LAG
L2 (Vlan local / global)
Port-channel member
Spanning-tree
Firewall
#CLMEL
BRKACI-1001
INTERFACE POLICY
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
Interface Policy Groups
Used to specify which interface policies to be applied to a particular interface type.
It also associates an AEP (which defines which domains are allowed on the interface).
Types:
VPC Domain 1
Access port (EP1)
Access Bundle Groups
•
Virtual Port-channel (EP2)
•
Port-channel (EP3)
EP1
EP2
EP3
Note: Separate policy groups should be created for each port-channel (standard or VPC) that you need to
configure. All interfaces on leaf that are associated with a particular access bundle group reside in same channel.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
59
Port-Channel Policies
Classical vPC Domain configuration
ACI Port-Channel Policies
Required configuration of domain, peer-link, and
peer-keepalive link on both devices in domain
Specify mode, minimum / maximum links, and related
protocol options (relating to LACP)
interface Ethernet1/5-6
lacp port-priority 32768
lacp rate normal
channel-group 10 mode on
interface Ethernet1/10-11
lacp port-priority 32768
lacp rate fast
channel-group 20 mode active
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
60
Access Policy Example
General Configuration (reused for many interfaces):
AEP
CiscoLive
1)
2)
Configure a physical domain and vlan pool
Create an AEP and associate physical domain
3)
Create switch/interfaces profiles for leaf (LEAF101)
•
4)
Pool1
DomPhy1
Switch Profile
very easy to apply configurations if you create a
switch/interface profile for each leaf and one for each VPC
domain pair
Configure Interface policies (LACP / LLDP)
LEAF101
Leaf_101
Interface Profile
LACP Active
LEAF101
Policies
LLDP Rx / Tx enabled
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Creating Physical Domain / AEP / Vlan Pool
In dropdown:
Click Create Attachable Entity Profile
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
62
Creating Physical Domain / AEP / Vlan Pool
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
Creating Physical Domain / AEP / Vlan Pool
Click + to add vlan range
In dropdown:
Click Create VLAN Pool
Specify start and
end vlans in range
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
Create Interface Profile for each leaf / VPC domain
Enter name and submit
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
Create Switch Profile for each leaf / VPC domain
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
66
Create Switch Profile for each leaf / VPC domain
Enter name
Click + to add selector
Select the Interface Profile
created for this leaf earlier
Enter a name and choose
appropriate leaf or leafs (for
vpc pair)
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
Create common protocol configurations
Example demonstrates a common lacp port-channel policy
Use a descriptive name
Select the protocol
Configure options/knobs
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
68
Access Policy Example
Interface specific (each time you add a new interface):
1)
2)
Create policy group for device (VPC / PC / Access)
Within the policy group, select the desired policies / AEP
3)
Associate interfaces to policy group via desired leaf profile
•
•
AEP
CiscoLive
Pool1
DomPhy1
Switch Profile
use specific leaf profile if access or PC
use VPC leaf profile if policy group is VPC
LEAF101
Leaf_101
Interface Profile
LACP Active
PC_Server_1
LEAF101
Policies
Policy Groups
blk_1/1-2
LLDP Rx / Tx enabled
Access_Servers
blk_1/47-48
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
69
Create policy groups
Note:
A separate policy group
should be created for
each PC/VPC that you
will deploy
Descriptive name
Associate your desired
interface policies (otherwise
default)
Associate your AEP to select
which domains this interface can
deploy
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
70
Create interface selectors / associate policy group
Click + to add selector
Choose interface profile to
add selectors
Use a descriptive name
Specify interface/range
Associate the policy group to
deploy on interfaces
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
71
Example policy scheme
Switch Profile
Leaf101
Leaf101_102
Interface Profile
Leaf101
Leaf101_102
Interface Selector
Interface Block
linux
windows
n7k_pc10
1/20-25
1/30-35
1/10-11
asa_cl1_pc1
1/45-48
n7k1_pc10
n7k2_pc10
1/10
1/20
1/1-4
Interface Policy Group
linux-access
windows-access
#CLMEL
asa_vpc_ccl
BRKACI-1001
asa_vpc_data
n7k_vpc10
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
72
vPC Protection Group Policy
vPC Domain 1
vPC Domain 2
vPC Domain 1
Classical vPC Domain configuration
ACI vPC Domain configuration
Required configuration of domain, peer-link, and
peer-keepalive link on both devices in domain
Specify the Domain ID and the two Leaf switch IDs that
form the domain pair
vpc domain 1
peer-keepalive destination 172.168.1.2 /
source 172.168.1.1 vrf vpc-keepalive
peer-gateway
ip arp synchronize
VPC Protection Group
Name: vPC-Domain100
ID: 100
Switch1: 101
Switch2: 102
interface port-channel 20
vpc peer-link
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
73
VPC Protection Group (example configuration)
GUI sequence:
Tabs:
Fabric -> Access Policies
Navigation Tree:
Switch Policies -> Policies ->
VPC Domain -> Default
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
74
Fabric and Tenant Policies
Tenant Policies
Static Binding
VMM Integration
WAN Connectivity
ACI Fabric
ACI Fabric
ACI Fabric
Extend VLAN to legacy Net
Extend VLAN to legacy Net
Legacy
Extend VLAN to legacy Net
WAN
baremetal
server
Hypervisor Cluster
baremetal
server
baremetal
server
endpoints behind
legacy network
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
75
Fabric and Tenant Policies
Tenant Policies (Static Binding)
S10
Extend VLAN to legacy Net
Server
L1
S20
L3
L2
L4
Allow Layer 2
Connectivity to 7K
Server
Nexus 7000
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
76
Fabric and Tenant Policies
Tenant Policies – Key concepts
Tenants are a Logical Grouping containing Policies.
Resources in the Common Tenant can be used in User Tenants
VRFs are used to separate routing tables inside the ACI Fabric.
1 or more VRFs can be used.
Bridge Domains define your Broadcast/ Flood domain
Unique VXLAN VNID is used per Bridge Domain
Configure ARP Optimization and L2 Unknown Unicast Proxy
Subnet (SVI) can be defined under the BD and is mapped to a single VRF
Bridge Domain
Tenant
VRF
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
77
Fabric and Tenant Policies
Tenant Policies – Key concepts one EPG to another
EPGs defines a collation of policy assigned to a group of devices
Contracts, QoS, SPAN requirements
L4-L7 policies (PBR, Load balancing, Firewalls)
EPG is most commonly determined by ingress VLAN & Port
Contracts are a collection of filters which allow traffic to pass between EPGs
Contacts are similar to access-lists. Consumer is Source, Provider is Destination
Filters contain a list of protocols and ports
EPG
ICMP
Bridge
Domain
Contract
VRF
Tenant
EPG
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
78
Fabric and Tenant Policies
Tenant View
EPGs
Bridge Domains
VRFs
Contracts
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
79
Fabric and Tenant Policies
Deploying a VRF
Change the VRF from a WhiteList model to an “Allow All”
Model
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
80
Fabric and Tenant Policies
Deploying a Bridge Domain
Associate Bridge Domain to VRF
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
81
Fabric and Tenant Policies
N7710# configure terminal
Deploying an EndPoint Group
N7710(config-if)# switchport trunk allowed vlan add 100
N7710(config)# interface port-channel 1
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
82
Fabric and Tenant Policies
Tenant Policies
S10
Extend VLAN to legacy
Net
L1
S20
L3
L2
L4
Allow Layer 2
Connectivity to 7K
L2 Path
Server
Server
Nexus 7000
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
83
Fabric and Tenant Policies
Planning
Requirements
Notes
Example
Tenant
1 Tenant can be used company. Tenants can also separate functions of a business.
NOTE: Shorter names are easier when using CLI
Prod/Dev
VRF
1 or more VRFs per Tenant
PROD-MAIN
DEV-TEST,DEV-PROD
Bridge Domain
Recommended to have 1 BD per Legacy VLAN. For Network Centric Migrations, 1 BD
should be used for each EPG.
VLAN_100,VLAN_101
BD_vMotion
Application Profile
Logical Container for EPGs. 1 AP is sufficient in most installations. NOTE: This is
strictly a management entity. No policies are defined on this object.
Prod-AP
EndPoint Group
Ports/VLANs (static path bindings) are added to EPGs to define what Endpoints get
defined in what EPGs. QOS/Contracts, etc are added to EPGs. For Network Centric
Migrations, 1 EPG should be used for each Legacy VLAN.
VLAN_100
VLAN_101
vMotion
Contracts
Contracts can be re-used across multiple EPGs. If we compare this to an ACL, the
Consumer is the Source, and the Provider is the Destination.
Web
Filters
Add Required Ports and Protocols to allow communication. Only what is specified in
the filter → contract will be allowed between EPGs providing and consuming that
contract.
SRC: Any, DST:80
SRC: Any, DST:443
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
84
Fabric and Tenant Policies
Tenant Policies (VMM Integration)
S10
S20
VMM enabled EPGs
L1
L2
L3
L4
• Tenant + Virtual Networking Tab
Hypervisor Cluster
#CLMEL
BRKACI-1001
Server
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
85
Cisco ACI Hypervisor Integration (vmware example)
Red - Manual Operation
Create VMM Domain
APIC
vCenter A
VMM Domain
1
6
Associate EPGs to the
VMM Domain
End Point Group
(EPG)
EPG
WEB
EPG
APP
EPG
DB
APIC Admin
9
Push Policy
ACI Fabric
2
Cisco APIC and
VMware vCenter
Initial Handshake
7
Automatically Map EPGs
To Port Groups
Learn location of ESXi Host
through LLDP
5
Attach vmware ESXi
to VDS
4
3
HYPERVISOR (ESXi)
Create DVS
HYPERVISOR (ESXi)
VIRTUAL DISTRIBUTED SWITCH
8
VI/Server Admin
vmware
vCenter
10
WEB PORT GROUP
APP PORT GROUP
DB PORT GROUP
Web
App
DB
Create Port Groups
Instantiate VMs,
Assign to Port Groups
BRKACI-1001
#CLMEL
Web
DB
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
86
Fabric and Tenant Policies
Layer 3 (WAN) Connectivity
S10
S20
Layer 3 Access To Core
L1
L3
L2
L4
ACI
Server
WAN/Core
#CLMEL
BRKACI-1001
Provide External
Access to Server
Layer 2
Layer 3
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
87
Basic Connectivity
node-103
RID: #
node-104
RID: #
IP: A
IP: B
Layer3 Out: L3Out-1
VRF: VRF-V1
Layer-3 Domain: DomL3
vlan-x
Logical Node Profile: node-103-104
node: node-103
Router-ID: #
node: node-104
Router-ID: #
L3Out-1
VRF-V1
Logical Interface Profile: ipv4-lif
path: topology/pod-1/…vpcX
type: ext-svi, encap: vlan-x
IP-A, IP-B, MTU, MAC
Create the L3Out
• Associate VRF and L3 Domain
• Create Logical Node Profile and associate fabric nodes to
the L3Out.
• Create Logical Interface Profile
• Specify Path attributes containing physical interface,
encapsulation, and IPs
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
88
Fabric and Tenant Policies
Creating a Layer 3 Out
•
External Routed Networks allow us to
peer with external routers
•
Dynamic Protocols
•
#CLMEL
BRKACI-1001
•
EIGRP
•
OSPF
•
BGP
Static Routing
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
89
Fabric and Tenant Policies
Route Reflectors
• Fabric nodes communicate
using MP-BGP.
• BGP advertises routes from
Border Leaf to Compute Leafs.
• Runs in overlay-1 VRF
L1
L2
L3
L4
ACI
Server
0.0.0.0/0
10.0.0.0/24
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
90
Fabric and Tenant Policies
Route Reflectors
L1
S10
S20
L3
L2
L4
TEP 192.168.160.64 192.168.160.65
leaf3# show ip route vrf A:A
0.0.0.0/0, ubest/mbest: 1/0
*via 192.168.160.64%overlay-1,
*via 192.168.160.65%overlay-1,
10.0.0.0/24, ubest/mbest: 1/0
*via 192.168.160.64%overlay-1,
*via 192.168.160.65%overlay-1,
[200/1], 03w21d, bgp-90002, internal, tag 90002
[200/1], 03w21d, bgp-90002, internal, tag 90002
[200/1], 03w21d, bgp-90002, internal, tag 90002
[200/1], 03w21d, bgp-90002, internal, tag 90002
RR Config
• BGP AS number
• Pick 2 spines / pod
Routing Table points to two boarder leaves with ECMP
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
91
Fabric and Tenant Policies
Planning
Requirements
Notes
Example
BGP Route Reflector
Use an AS Number not already in your environment. The AS number is only
exposed to the external network when peering BGP with devices. Private AS
number can be used. NOTE: CHANGING THE AS NUMBER IS DISRUPTIVE!
65000
External Routed
Network
This is your Layer 3 Object. It contains the entire Layer 3 path configuration.
L3out-To-Core
Node Profile
Defines which nodes are part of the Layer 3 out Domain. Here is where you
define your Router ID’s and Static Routes.
Leaf101, Leaf102
Leaf101-102
Logical Interface
Profile
Defines which interfaces are used for peering. Support Types are Routed
Interfaces, Routes Sub-Interfaces, and SVIs. This is also where you define the
IP/MTU/VLAN is SVI or Sub-Interface.
Port10
vPC-To-Core
Networks (External
EPG)
This is where you define the external subnets you want to apply policy to. You
do this by listing the subnets and applying contracts. NOTE: multiple all 0’s
subnets should not be configured in the same VRF.
Ext_EPG → 0.0.0.0/0 subnet
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
92
Fabric and Tenant Policies
Layer 3 Connectivity
S10
S20
Layer 3 Access To Core
L1
L3
L2
L4
ACI
Server
Provide External
Access to Server
Layer 2
Layer 3
WAN/Core
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
93
Agenda
•
Day 1: Why ACI?
•
Day 2: Infrastructure and Policies
•
Day 3: Forwarding Overview
•
Day 4: Network Centric Migrations
•
Day 5: Multi Location Deployments
•
Day 6: Troubleshooting Tools
•
Day 7: Additional Resources
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
94
Day 3: Forwarding
Overview
What is an Endpoint?
Traditional Endpoint
L2 – MAC Table
- MAC Address
- VLAN
- Interface
Eth1/1
10
L3 – ARP Table
- IP / MAC
- Interface
- VRF
10
000a.000a.000a
dynamic
0
00:00:01 000a.000a.000a
000b.000b.000b
192.168.2.100/24
N5K# show mac address-table | grep 000b
Eth1/1
N5K# show ip arp vrf default | grep 000a
192.168.1.1
20
000a.000a.000a
192.168.1.100/24
N5K# show mac address-table | grep 000a
•
Eth1/2
•
20
000b.000b.000b
dynamic
0
Eth1/2
N5K# show ip arp vrf default | grep 000b
Vlan10
192.168.2.1
#CLMEL
BRKACI-1001
00:00:01 000b.000b.000b
Vlan20
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
96
What is an Endpoint?
Eth1/1
ACI Endpoint
-
MAC or MAC/IP → IP is /32 or /128
Route
VLAN → EPG (pcTag)
Interface
VRF
Flags → Local, vPC, static, etc.
Eth1/2
10
20
APIC
000a.000a.000a
192.168.1.100/24
EPG1
000b.000b.000b
192.168.2.100/24
EPG2
apic1# show endpoints ip 192.168.1.100
Dynamic Endpoints:
Tenant
: CL
Application : CL
AEPg
: EPG1
End Point MAC
----------------00:0A:00:0A:00:0A
IP Address
---------------------------------------192.168.1.100
Node
---------101 102
#CLMEL
Interface
-----------------------------eth1/1
BRKACI-1001
Encap
--------------vlan-10
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
97
What is an Endpoint?
Eth1/1
ACI Endpoint
-
MAC or MAC/IP → IP is /32 or /128
Route
VLAN → EPG (pcTag)
Interface
VRF
Flags → Local, vPC, static, etc.
10
000a.000a.000a
192.168.1.100/24
EPG1
Eth1/2
20
000b.000b.000b
192.168.2.100/24
EPG2
Leaf1# show endpoint mac 000a.000a.000a detail
Legend:
s - arp
O - peer-attached
a - local-aged
S - static
V - vpc-attached
p - peer-aged
M - span
L - local
B - bounce
H - vtep
+-----------------------------------+---------------+-----------------+--------------+-------------+----------------+
VLAN/
Encap
MAC Address
MAC Info/
Interface
Endpoint Group
Domain
VLAN
IP Address
IP Info
Info
+-----------------------------------+---------------+-----------------+--------------+-------------+----------------+
16
vlan-10
000a.000a.000a
L
eth1/1
CL:CL:EPG1
CL:17
vlan-10
192.168.1.100
L
eth1/1
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
98
Endpoint Learning - ARP
ACI Leafs learn via ARP!
Eth1/1
ARP Request
Eth1/2
Who has 192.168.1.101??
DMAC
FFFF.FFFF.FFFF
SMAC
000a.000a.000a
Eth: 0x0806
Eth: 0x0806
Hdr/Opcode
Hdr/Opcode
Sender MAC
000a.000a.000a
Sender IP
192.168.1.100
Target MAC
0000.0000.0000
Target IP
192.168.1.101
000a.000a.000a
192.168.1.100/24
EPG1
000b.000b.000b
192.168.1.101/24
EPG1
Frame
Unicast Routing?
EP Contents
ARP
No
MAC (Sender MAC)
ARP
Yes
MAC (Sender MAC), IP
(Sender-IP)
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
99
Endpoint Learning– Routed Frames
Routed Frame triggers an EP Learn
Eth1/1
000a.000a.000a
192.168.1.100/24
EPG1
Eth1/2
000b.000b.000b
192.168.2.100/24
EPG2
DMAC
BD MAC
SMAC
000a.000a.000a
802.1Q
10
Frame
Unicast Routing
EP Contents
SIP
192.168.1.100
IPV4/6
Yes
DIP
MAC (L2 SRC MAC), IP
(SRC IP)
192.168.2.100
Protocol
1
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
100
Pervasive Gateway
S10
•
To work as a default gateway for
endpoints
•
Gateway IP is programmed on all leafs
that need it
•
Deterministic Traffic Flow to Gateway
L1
L2
•
Consistent Latency across all Devices
Towards Gateway
BD1
BD1
EP1 –EPG1
BD1
#CLMEL
S20
L3
BD1
L4
BD2
EP2 - EPG1
BD1
BRKACI-1001
BD2
EP3 – EPG2
BD2
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
101
Proxy Routing
•
•
•
Leafs report EP’s to spine
once Learnt
Spines maintain a database
of all Endpoints Learnt in the
Fabric, and on what Leaf(s)
they exist.
Used for “Hardware Proxy” ✓
BD Mode.
3
✓
EP published
to Spine
S10
S20
✓
EP synced to
other Spines
2
L1
EP
learnt
on Leaf
L3
L2
L4
1
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
102
ARP Flooding
EP1 ARP’s for EP2
•
•
Behavior is the same as
Traditional Switches
ARP is flooded using BD
Multicast Group to all Leafs
that have the BD
S10
S20
3
✓
ARP is flooded
to all leafs that
have the BD
ARP is
2 ✓ Flooded
in
BD, copy to
Spine
1
✓
ARP
Received
on L1
L1
L2
L3
BD1
BD1
BD1
L4
✓
L2 sends
ARP out
ports in BD
✓
L3 sends
ARP to EP2
4
000b.000b.000b
192.168.1.101/24
EP2 - EPG1
BD1
000a.000a.000a
192.168.1.100/24
EP1 - EPG1
BD1
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
103
ARP Optimization – Unicast Routing
EP1 ARP’s for EP2
•
ACI can Unicast ARP to avoid
unnecessary Flood traffic. →
Requires Unicast Routing on
BD
S10
S20
3
✓
Spine knows Target
IP is on L3, Unicast
to L3
✓
L3 learns EP1 from
L1
2 ✓ L1 doesn’t
know Target
IP → Send to
Spine!
L1
1
✓
ARP
Received
on L1
L2
L3
L4
BD1
BD1
4
000a.000a.000a
192.168.1.100/24
EP1 - EPG1
#CLMEL
✓
L3 sends
ARP to EP2
000b.000b.000b
192.168.1.101/24
EP2 - EPG1
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
104
Known Unicast – Layer 2
EP1 pings EP2
Outer
Outer
SIP
L1
DIP
L3
VXLAN
BD1
S10
✓
2
from L1 directly
to L3 through
spines
L1 looks at the
DMAC and
knows it exists
on L3 in EPG1
L1
Inner
3 ✓ Packet is sent
S20
L2
L3
L4
Inner
BD1
BD1
DMAC
BBBB
SMAC
AAAA
SIP
192.168.1.100
DIP
192.168.1.101
Protocol
ICMP
1
✓
ICMP
Received
on L1
4 ✓ L3 sends
ICMP to EP2
000a.000a.000a
192.168.1.100/24
EP1 - EPG1
#CLMEL
BRKACI-1001
000b.000b.000b
192.168.1.101/24
EP2 - EPG1
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
105
Known Unicast – Layer 3
EP1 pings EP2
S10
S20
4
2 ✓ L1 looks at the
DST IP and
knows it exists
on L3 in EPG2
Subnet under BD acts as GW
If traffic is destined to the GW
MAC, we do an IP Lookup in the
VRF
EPG1
EPG2
✓
Packet is sent
from L1
directly to L3
through spines
X
L1
1
✓
ICMP
Received
on BD GW
L3
L2
BD1
L4
BD2
5
3
ICMP
✓
L3 sends
ICMP to EP2
ICMP
000a.000a.000a
192.168.1.100/24
EP1 - EPG1
#CLMEL
BRKACI-1001
000b.000b.000b
192.168.2.100/24
EP2 – EPG2
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
106
Day 4:
Network Centric Migrations
Physical Layer
S10
L1
S20
L3
L4
L2
vPC to allow L2 VLANs
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
108
Checklist
✓ Physical Layer ☺
❑ Layer 2
❑ Layer 3
Network Centric Design
L2 Migration Recommendations
BD_VLAN100
+
Each Legacy VLAN requires a unique Bridge Domain
Settings: Unicast Routing Disabled
Unknown L2 Flooding
ARP Flooding
EPG
VLAN_100
=
Each Legacy VLAN has a unique EPG
Legacy VLAN
100
What have we Accomplished?
Each Legacy VLAN maps to a unique Bridge Domain
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
110
Conceptual View
Legacy
ACI
VRF CiscoLive
BD_VLAN100
EPG
VLAN_100
VLAN100
VLAN101
BD_VLAN101
EPG
VLAN_101
BD_VLAN102
EPG
VLAN_102
VLAN102
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
111
Conceptual View
S10
S20
SVI/VLAN:100
192.168.100.1
L1
SVI/VLAN:101
192.168.101.1
SVI/VLAN:102
192.168.102.1
L3
L4
L2
L2 Extension
VLAN100
VLAN101
BD_100
BD_101
BD_102
EPG 100
EPG 101
EPG 102
VLAN102
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
112
Spanning-tree in ACI
•
ACI Fabric does not run Spanning-tree
•
BPDUs are flooded in ‘EPG VNID’ (use same VLAN pool for all ports deploying legacy VLANs)
•
ACI Fabric does snoop BPDUs and will flush Endpoints (Mac & IP) when TCNs are received
•
Learning is disabled when excessive BPDUs are received
•
External Spanning-tree devices should be configured with “spanning-tree linktype shared”
•
Use “show mcp internal info vlan encap_vlan” to see TCNs
Leaf101# show mcp internal info vlan 100
------------------------------------------------PI VLAN: 13 Up
Encap VLAN: 100
PVRSTP TC Count: 11
RSTP TC Count: 0
Last TC flush at Mon May 1 19:32:22 2017
on Tunnel13
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
113
Verification
APIC GUI shows connected Endpoints (MAC and or IP) per EPG and Path
E.g.: 5C:83:8F:69:BB:C9 (N7K) connected via Nodes-101-102/N7710-vPC
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
114
Checklist
✓ Physical Layer ☺
✓ Layer 2
❑ Layer 3
Network Centric Design
L3 Migration Requirements
Configure “Layer 3 Out” to create a routed connection to legacy network
Routed Interface
Routed subinterface
Subnet
Switched Virtual Interface (SVI)
Bride Domain
Bridge Domain with “Unicast Routing” enabled
Subnet defined on BD
L3Out associated with BD
EPG
VLAN_100
L3Out
EPG has contract to L3Out Network
Dynamic Routing
OSPF/ EIGRP/ BGP/ Static
Routing Protocol
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
116
Conceptual View
S10
S20
L3 Extension
SVI/VLAN:100
192.168.100.1
L1
SVI/VLAN:101
192.168.101.1
SVI/VLAN:102
192.168.102.1
L3
L4
L2
L2 Extension
VLAN100
VLAN101
BD_100
BD_101
BD_102
EPG 100
EPG 101
EPG 102
VLAN102
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
117
L3 Migration Considerations
1)
Disable External GW!
2)
Bridge Domain Settings
Unicast routing Enabled – Minor Service Impact
L2 Unknown Unicast H/W Proxy – Service Impact
ARP Flooding Optimized - In conjunction with L2 Unknown
Unicast
Limit IP learning to Subnet
Off Subnet Learns are cleared
Learning is disabled for 2 minutes
3) Global Settings
Enforce Subnet Check - adds prefix check to all BD’s
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
118
Verification
APIC GUI now shows IP information since UC Routing is enabled on BD
E.g.: 192.168.102.11 connected via Nodes-101-102/BareMetal02-vPC
Recommended Content! – ACI Endpoint Learning White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739989.html
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
119
Verification
GUI
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
120
Verification
Leaf101# show ip
OSPF Process ID
Total number of
Neighbor ID
192.168.255.255
SSH
ospf neighbors vrf CiscoLive:VRF1
default VRF CiscoLive:VRF1
neighbors: 1
Pri State
Up Time Address
1 FULL/BDR
02:27:05 192.168.255.2
Interface
Eth1/13
Leaf101# show ip route vrf CiscoLive:VRF1 10.0.0.0/8
IP Route Table for VRF "CiscoLive:VRF1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
10.0.0.0/8, ubest/mbest: 1/0
*via 192.168.255.2, eth1/13, [110/5], 01:45:34, ospf-default
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
121
Checklist
✓ Physical Layer ☺
✓ Layer 2
✓ Layer 3
Common Pitfalls
Old Gateway still Active!
S10
S20
L3 Extension
SVI/VLAN:100
192.168.100.1
L1
SVI/VLAN:101
192.168.101.1
SVI/VLAN:102
192.168.102.1
L3
L4
SVI/VLAN:100
192.168.100.1
SVI/VLAN:101
192.168.101.1
SVI/VLAN:102
192.168.102.1
L2
L2 Extension
VLAN100
VLAN101
BD_100
BD_101
BD_102
EPG 100
EPG 101
EPG 102
VLAN102
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
123
Common Pitfalls
S10
Windows Dynamic Load Balancing
S20
Problem:
Traffic is Sourced with the same IP but from
both NIC’s using different MACs
ACI Fabric sees frequent IP Move between
MAC’s when Routing is Enabled!
L1
L3
L2
L4
Solution:
Use “Hyper-V Port” to force single MAC to
IP Communication
NIC1: MAC A
NIC1: MAC A
NIC2: MAC B
IP: 192.168.100.11
IP: 192.168.100.10
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
124
Day 5: Multi-Location
Deployment Options
Stretched Fabric
S10
S20
L2
L3
L1
APIC
APIC
L4
IS-IS
L5
L6
APIC
S11
S21
L8
L9
L7
L10
APIC
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
126
Stretched Fabric
Advantages
•
•
•
All one Fabric
No Additional Routed Infrastructure
Simple Provisioning – If cabling is in
place
S10
S20
L2
L3
S11
S21
L8
L9
Limitations
•
•
•
Single APIC Failure Domain
L1 Connectivity between Transit Leafs
and spines (dark fiber)
Same Control Plane Instance Across
Sites
L1
L4
APIC
APIC
#CLUS
BRKACI-1001
L5
L6
L7
L10
APIC
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
127
IPN MTU Requirements: 9150 Bytes
Multipod
IPN
S10
S20
L2
L3
L1
APIC
APIC
IPv4 Multicast
Network
IS-IS
OSPF
IPN
L7
L4
APIC
S11
S21
L8
L9
L10
APIC
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
128
Multipod
Advantages
•
•
•
•
All one Fabric
Policy Stretched across sites
Separate Control Plane Instances per
site
Increases Leaf Scale to 400
Limitations
•
•
•
•
Single APIC Failure Domain
Need dedicated Routing Devices as
Inter-Pod Network (IPN) Routers.
Requires PIM BI-Dir to route BUM traffic
between sites.
50ms max latency between pods
IPN
S10
S20
L2
L3
L1
APIC
#CLUS
L4
APIC
BRKACI-1001
IPN
L7
S11
S21
L8
L9
L10
APIC
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
129
IPN MTU Requirements: 9150 Bytes
Remote Leaf
ISN
IPV4 “Inter-site”
Network
IS-IS
OSPF
ISN
Remote Office/ DC
Primary Site
S10
S20
L2
L3
L1
APIC
APIC
RL1
RL2
L4
APIC
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
130
Remote Leaf
Advantages
•
•
•
•
All one Fabric
Easy Addition of small site to existing
APIC
Spines not required in Remote Site.
Connects to existing routing
infrastructure
• No Multicast required
ISN
L1
S10
S20
L2
L3
ISN
RL1
RL2
L4
Limitations
•
•
•
All traffic goes to “main” site before
other sites.
140ms Latency Restriction
Port Count
APIC
APIC
#CLUS
APIC
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
131
IPN MTU Requirements: 9150 Bytes
Multi-Site
ISN
S10
S20
L2
L3
L1
IPV4 “Inter-site”
Network
IS-IS
OSPF
ISN
S10
S20
L2
L3
L1
L4
L4
ACI
Multi-Site
ACI
Controller
Multi-Site
ACI
Controller
Multi-Site
Controller
APIC
APIC
APIC
APIC
#CLUS
BRKACI-1001
APIC
APIC
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
132
Multi-Site
Advantages
•
•
•
ISN
Two Independent Fabrics (APIC Clusters)
Policy is synchronized using Multi-Site Controller
Connects to existing routing infrastructure
• No Multicast required
L1
S10
S20
L2
L3
•
500ms – 1s latency for OOB
MSC → APIC connectivity
Not all Site Specific Config can be done from
MSC
L1
L4
Limitations
•
ISN
S10
S20
L2
L3
L4
ACI
Multi-Site
ACI
Controller
Multi-Site
ACI
Controller
Multi-Site
Controller
APIC
#CLUS
APIC
APIC
BRKACI-1001
APIC
APIC
APIC
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
133
Day 6:
Troubleshooting Tools
Faults
Available in 2.2(2e)!
☺
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
135
EP Tracker
“We had a
problem at
14:21!!!”
Attach/Detach events
are logged for each EP
IP Was Moving???
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
136
Atomic Counters
S10
Used to measure packet loss in Overlay
Logs packet count between EP’s on different Leafs
Specific Filter can be set
Requires NTP!
Leaf
Direction
Filter
Packet Count
L1
Tx
ICMP
500
L2
Rx
ICMP
500
Rx
Tx
L1
L2
192.168.101.10
192.168.102.11
Ping –c 500 192.168.102.11 –c 500
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
137
Atomic Counters
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
138
Atomic Counters
NO Packet Loss In Overlay
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
139
S10
SPAN
•
•
•
ACI allows for SPAN of EPG
ERSPAN Destination must be an IP EP
Learnt in ACI
EP Can run Wireshark or Tshark
L1
L2
ERSPAN
10.10.10.10
EPG 100
SPAN Source
SPAN Destination
EPG
ERSPAN
Port
ERPSAN/Local
Port
EP Learnt
#CLUS
BRKACI-1001
Leaf101# show monitor session all
session 1
--------------description
: Span session 1
type
: erspan
version
: 2
oper version
: 1
state
: up (active)
erspan-id
: 1
granularity
:
vrf-name
: CiscoLive:VRF1
acl-name
:
ip-ttl
: 64
ip-dscp
: ip-dscp not
specified
destination-ip
: 10.10.10.10/32
origin-ip
: 1.1.1.1
mode
: access
source VLANs
:
rx
: 100
tx
: 100
both
: 100
filter VLANs
: filter not specified
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
140
Troubleshooting Wizard - Faults
Shows Faults
in the Path
Builds Topology of Flow
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
141
Troubleshooting Wizard – Drop Stats
Shows Drops on Every Hop. Green
Arrows portray no Drops
NOTE: Some Drops are expected.
Look for Drops like “Buffer” and “Error”!
Recommended Content! – Understanding Drop Faults in ACI
http://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policyinfrastructure-controller-apic/210539-Explanations-of-Packet-Drop-Faults-in-AC.html
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
142
Troubleshooting Wizard - Contracts
Shows Contracts for Flows
Implicit Deny
#CLMEL
Allow SSH
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
143
Troubleshooting Wizard – Atomic Counters
No Drops!
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
144
Troubleshooting Wizard – SPAN
Ability to SPAN to APIC or other devices
attached to the Fabric
User can select which ports to SPAN
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
145
Capacity Dashboard
Contract TCAM is Full!
Capacity Dashboard panel displays your usage by range and percentage.
Use this to plan your fabric Scale.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
146
App Center
Enhanced Endpoint Tracker
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
147
Enhanced Endpoint Tracker
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
148
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
149
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
150
App Center
Elam Assistant
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
151
ELAM Assistant
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
152
ELAM Assistant
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
153
Day 7: Additional
Resources
Support Forums
TAC Engineers are Subscribed
Easy Portal to Post Non Impacting
Questions or Concerns
Has Documentation written by CSE’s
and Technical Leaders
https://supportforums.cisco.com/t5/application-centric/bd-p/12206936-discussions-aci
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
155
Facebook Group
Many Customers and Cisco Employees
Great Real World Deployment Advice
Great way to meet others working
with ACI
Great Community ☺
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
156
Solutions Support
One TAC team to support all aspects of
ACI
Engineers are familiar with 3rd party
products like VMWare
Case does not get handed off when it
is a Switching vs. Routing issue.
ACI Team takes ownership
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
157
JumpStart
Program designed by TAC
Two 3-hour WebEx session with TAC
Talk to your Cisco Account team to get
scheduled for your JumpStart!
#CLUS
BRKACI-1001
© 2018 Cisco and/or its affiliates. All rights reserved. Cisco Public
158
Continue
your
education
Demos in
the Cisco
campus
Walk-in
self-paced
labs
#CLMEL
BRKACI-1001
Meet the
engineer
1:1
meetings
Related
sessions
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
159
Complete your online session evaluation
Give us your feedback to be entered into a
Daily Survey Drawing.
Complete your session surveys through the
Cisco Live mobile app or on
www.CiscoLive.com/us.
Don’t forget: Cisco Live sessions will be available for viewing on
demand after the event at www.CiscoLive.com/Online.
#CLMEL
BRKACI-1001
© 2019 Cisco and/or its affiliates. All rights reserved. Cisco Public
160
Thank you
#CLMEL
#CLMEL