ACI Troubleshooting
Layer 3 Out (L3Out)
Takuya Kishida – Technical Marketing, DCBU ACI
BRKACI-2642
Cisco Webex Teams
Questions?
Use Cisco Webex Teams to chat
with the speaker after the session
How
1 Find this session in the Cisco Events Mobile App
2 Click “Join the Discussion”
3 Install Webex Teams or go directly to the team space
4 Enter messages/questions in the team space
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
3
Agenda
•
•
L3Out Key Components
•
Learning routes (Routing Protocol)
•
Distributing routes within ACI (MP-BGP)
•
Advertising ACI subnet
•
Contract on L3Out (prefix based EPG)
L3Out Subnet scope options
•
Summary of all options
•
Export Route Control Subnet example
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
4
Acronyms/Definitions
Reference Slide
Acronyms
Definitions
ACI
Application Centric Infrastructure
APIC
Application Policy Infrastructure Controller
EP
Endpoint
EPG
Endpoint Group
BD
Bridge Domain
VRF
Virtual Routing and Forwarding
L3Out
Layer 3 Out (External Routed Network)
L3Out EPG
Layer 3 Out EPG, Prefix Based EPG (External Network Instance)
MP-BGP
Multi Protocol BGP
VPNv4
Virtual Private Network Version 4
RT
Route Target
RD
Route Distinguisher
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
5
Why L3Out?
Why L3Out?
• What is EPG for ?
➢ Endpoint (EP) = MAC & /32 IP (or /128)
VRF
BD
EPG
• What is BD Subnet for ?
➢ To be a default gateway
➢ For ACI Fabric to know a subnet for EPs in a
BD
Subnet A
This is for Spine-Proxy
Please check BRKACI-3545 for details
EPG
EP A1
EP A2
EP A3
MAC A1
IP A1
MAC A2
IP A2
MAC A3
IP A3
IP A4
R1
MAC R1
IP R1
A network device (ex. router,
loadbalancer) as an endpoint?
IP X.X.X.X/8
IP Y.Y.Y.Y/8
IP Z.Z.Z.Z/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
Why L3Out?
• What is EPG for ?
➢ Endpoint (EP) = MAC & /32 IP (or /128)
• What is BD Subnet for ?
VRF
BD
EPG
Subnet A
BD
EPG
EP A1
EP A2
EP A3
MAC A1
IP A1
MAC A2
IP A2
MAC A3
IP A3
IP A4
Subnet R, X, Y, Z
EPG
R1
MAC R1
IP R1
IP X1 – X999
IP Y1 – Y999
IP Z1 – Z999
➢ To be a default gateway
➢ For ACI Fabric to know a subnet for EPs in a
BD
A network device as an endpoint?
➢ All IPs as /32 in a single endpoint
leaf1# show endpoint vlan 84
84
vlan-5
0000.0000.R1R1
TK:VRF1
vlan-5
R.R.R.1
TK:VRF1
vlan-5
X.X.X.1
TK:VRF1
vlan-5
X.X.X.2
TK:VRF1
vlan-5
X.X.X.3
.....
TK:VRF1
vlan-5
Y.Y.Y.1
.....
TK:VRF1
vlan-5
Z.Z.Z.1
.....
L
L
L
L
L
po3
po3
po3
po3
po3
L
po3
L
po3
※ One endpoint can have up to 1024 IPs in ACI
IP X.X.X.X/8
IP Y.Y.Y.Y/8
IP Z.Z.Z.Z/8
This does not scale and efficient.
No need to learn each IP as /32.
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
8
Why L3Out?
• What is L3Out for ?
➢ To connect ACI with other network domain
= devices with multiple subnet behind it
• How is L3Out different from EPG?
VRF
BD
EPG
EP A1
MAC A1
IP A1
EP A2
MAC A2
IP A2
Subnet A
EPG
EP A3
L3Out
Routing Protocol
L3Out EPG
R1
MAC A3
IP A3
IP A4
MAC R1
IP R1
➢ Speak Routing Protocol
➢ No IP learning as endpoint
➢ Next-hop IP is stored in ARP table
= Same as normal routers
Next-hop MAC in endpoint table
leaf1# show endpoint vlan 84
84/TK:VRF1
vxlan-14876665
0000.0000.R1R1
L
po3
Next-hop IP in ARP table (only for L3Out)
leaf1# show ip arp vlan 84
Address
Age
MAC Address
R.R.R.1
00:07:51 0000.0000.R1R1
Interface
vlan84
Other routes via Routing Protocol
IP X.X.X.X/8
IP Y.Y.Y.Y/8
IP Z.Z.Z.Z/8
leaf1# show ip route vrf TK:VRF1
X.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
Y.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
Z.0.0.0/8, ubest/mbest: 1/0
*via R.R.R.1, vlan84, [110/5], 2d00h, ospf-default, intra
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
9
L3Out
Key Components
L3Out Key Components
VRF Overlay-1
1. Learn external routes
➢ Routing Protocol in L3Out
Distribute
External Routes
BLEAF
non-BLEAF
VRF1
BD
EPG
2. Distribute external routes to other
leaves
➢ MP-BGP
VRF1
3. Advertise internal routes (BD subnet)
to outside
L3Out
➢ Redistribution
L3Out
and
EPG
Advertise
Learn
➢ Contract
Internal Routes
External Routes
(Export)
Allow traffic
(Import)
4. Allow traffic with contracts
➢ L3Out EPG (Prefix Based EPG)
and
➢ Contract
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
L3Out Key Components
1. Learn External Routes = Routing Protocol
Configurations
External Routed Networks (L3Out)
•
•
VRF to deploy Routing Protocol
Routing Protocol parameters
Routing Protocol
Information
ex. OSPF area 0.0.0.1 nssa
Only on
configured nodes
(Border LEAF)
Node Profile
•
•
Node(s) to deploy Routing Protocol
Static Route (if any)
Interface Profile
•
•
I/F(s) to deploy Routing Protocol
Routing Protocol I/F parameters
ex. OSPF hello interval
Networks (L3Out EPG)
•
•
VRF1
VRF1
VRF1
BD
L3Out
L3Out
Contract
Advanced Route Control
ex. route-map
※ Details for L3Out EPG are in later sections
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
Verification Examples (OSPF)
1. Is OSPF enabled on a correct I/F?
border-leaf# show ip ospf int bri vrf TK:VRF1
Interface
ID
Area
Vlan58
134
backbone
border-leaf# show vlan id 58 extended
VLAN Name
---- -------------------------------58
TK:VRF1:l3outL3OUT_OSPF:vlan-1425
Cost
4
State
BDR
Neighbors Status
2
up
Same CLI verifications
are as useful in ACI too
If anything is not as
expected, check config
or any faults in APIC GUI.
Encap
Ports
---------------- ---------------------vxlan-15695748, Eth1/3, Po2
vlan-1425
2. Are OSPF parameters matching with neighbors?
border-leaf# show int vlan 58 | grep MTU
MTU 1500 bytes, BW 10000000 Kbit, DLY 1 usec
Is MTU matching?
Is Network Mask matching?
Is Area matching?
Is Timer matching?
Is Network Type expected?
etc.
border-leaf# show ip ospf int vlan 58 | egrep 'IP|State|Timer|auth'
IP address 15.0.0.3/24, Process ID default VRF TK:VRF1, area backbone
State BDR, Network type BROADCAST, cost 4
Timer intervals: Hello 10, Dead 40, Wait 40, Retransmit 5
No authentication
3. Are OSPF neighbors established correctly?
border-leaf# show ip ospf neighbors vrf TK:VRF1
Neighbor ID
Pri State
Up Time Address
4.4.4.4
1 FULL/DR
2d06h
15.0.0.4
9.9.9.9
1 FULL/DROTHER
2d06h
15.0.0.1
Interface
Vlan58
Vlan58
BRKACI-2642
Can they ping to each other?
leaf# iping –V <VRF> <target IP>
※OSPF DBD requires unicast reachability
etc.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
Verification Examples (EIGRP)
1. Is EIGRP enabled on a correct I/F?
border-leaf# show ip eigrp int bri vrf TK:VRF1
Xmit Queue
Mean
Pacing Time
Interface
Peers Un/Reliable SRTT
Un/Reliable
vlan92
2
0/0
1
0/0
border-leaf# show vlan id 92 extended
VLAN Name
---- -------------------------------92
TK:VRF1:l3outL3OUT_EIGRP:vlan-1426
Multicast
Flow Timer
50
Pending
Routes
0
Encap
Ports
---------------- ---------------------vxlan-14712828, Eth1/3, Po2
vlan-1426
Same CLI verifications
are as useful in ACI too
If anything is not as
expected, check config
or any faults in APIC GUI.
2. Are EIGRP parameters matching with neighbors?
border-leaf# show int vlan 92 | grep MTU
MTU 1500 bytes, BW 10000000 Kbit, DLY 1 usec
border-leaf# show ip eigrp vrf TK:VRF1 | egrep 'AS|K'
IP-EIGRP AS 1 ID 3.3.3.3 VRF TK:VRF1
Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0
border-leaf# show ip int vlan 92 | grep 'IP addr'
IP address: 16.0.0.3, IP subnet: 16.0.0.0/24
3. Are EIGRP neighbors established correctly?
border-leaf# show ip eigrp neighbors vrf TK:VRF1
H
Address
Interface
Hold Uptime
(sec)
0
16.0.0.4
vlan92
12
2d06h
1
16.0.0.1
vlan92
13
2d06h
SRTT
(ms)
1
1
RTO
50
50
Q Seq
Cnt Num
0
10
0
346
BRKACI-2642
Is MTU matching?
Is Network Mask matching?
Is AS matching?
Is K value matching?
etc.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
Verification Examples (BGP)
1. Is BGP neighbor session configured as expected?
border-leaf# show ip bgp neighbors vrf TK:VRF1 | egrep 'BGP nei|Using|Opens|hops'
BGP neighbor is 17.0.0.1, remote AS 65001, ebgp link, Peer index 1
Using Loopback6 as update source for this peer
Is it correct remote AS?
External BGP peer might be upto to 2 hops away
Is it using correct source I/F with
Opens:
1
1
correct IP?
Is enough multi-hop configured
for eBGP?
Is Open message exchanged?
border-leaf# show ip int lo6 | grep 'IP addr'
IP address: 3.3.3.3, IP subnet: 3.3.3.3/32
2. Is there IP reachability ?
border-leaf# iping -V TK:VRF1 17.0.0.1 -S 3.3.3.3
PING 17.0.0.1 (17.0.0.1) from 3.3.3.3: 56 data bytes
64 bytes from 17.0.0.1: icmp_seq=0 ttl=255 time=0.76 ms
64 bytes from 17.0.0.1: icmp_seq=1 ttl=255 time=0.639 ms
=== snip ===
Is there an IP reachability to the BGP neighbor
from the correct source IP?
--- 17.0.0.1 ping statistics --5 packets transmitted, 5 packets received, 0.00% packet loss
3. Are BGP neighbors established correctly?
border-leaf# show ip bgp summary vrf TK:VRF1
BGP router identifier 3.3.3.3, local AS number 65003
Neighbor
17.0.0.1
V
AS MsgRcvd MsgSent
4 65001
3300
3302
TblVer
78
Is it receiving BGP routes?
Is ACI BGP using expected local AS?
InQ OutQ Up/Down State/PfxRcd
0
0
2d06h 2
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
L3Out Key Components
2. Distribute External Routes = MP-BGP in infra
Configurations
Implement all steps except
for step 1 (user L3Out)
Pod Profile
BGP Route Reflector Policy
•
default
System Settings
BGP Route Reflector
•
ACI BGP AS number
•
MP-BGP Route Reflector Spines
(for both MP-BGP and L3Out BGP)
Route Reflectors
MP-BGP in
VRF Overlay-1
Pod Policy Group
5
Import
back to VRF1
from MP-BGP
3
4
To other LEAFs
10.0.0.0/8 (VRF1)
-> LEAF2
10.0.0.0/8
-> LEAF2
2
To Route Reflector
Export
to MP-BGP
10.0.0.0/8 (VRF1)
-> Local
VRF1
VRF1
EPG
L3Out
10.0.0.0/8
-> local
1
10.0.0.0/8
BRKACI-2642
user L3Out
(Routing Protocol
or Static Route)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
L3Out Key Components
2. Distribute External Routes = MP-BGP in infra
1. Select ACI BGP AS and Route Reflector SPINEs
2. Apply Route Reflector policy to Pod Policy Group
Use default
※ L3Out BGP share this same AS with the internal MP-BGP
3. Apply Pod Policy Group
to Pod Profile
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
※ Check appendix for MP-BGP details
CLI Verification
1. Do both border leaf and non-border leaf have BGP sessions with RR spines?
leaf# show bgp sessions vrf overlay-1
Neighbor
ASN
Flaps LastUpDn|LastRead|LastWrit St Port(L/R)
10.0.184.65
65003 0
2d07h
|never
|never
E 37850/179
10.0.184.66
65003 0
2d07h
|never
|never
E 45089/179
leaf# acidiag fnvread | grep spine
1001
1
spine1
1002
1
spine2
FGE10000000
SAL10000000
10.0.184.65/32
10.0.184.66/32
2. Is the external route learned on a border leaf?
Notif(S/R)
0/0
0/0
spine
spine
active
active
0
0
✓ BGP neighbors are RR spines TEP IPs
border-leaf# show ip route vrf TK:VRF1
10.0.0.0/8, ubest/mbest: 1/0
*via 15.0.0.1, Vlan58, [110/5], 2d08h, ospf-default, intra
3. Does non-border leaf show the expected border leaf as next-hop?
✓ Next-hops are border Leaf TEP IPs
✓ Learned via iBGP in ACI AS# (65003)
non-border-leaf# show ip route vrf TK:VRF1
10.0.0.0/8, ubest/mbest: 2/0
*via 10.0.184.67%overlay-1, [200/5], 2d08h, bgp-65003, internal, tag 65003
*via 10.0.184.64%overlay-1, [200/5], 2d08h, bgp-65003, internal, tag 65003
non-border-leaf# acidiag fnvread
ID
Pod ID
Name
Serial Number
IP Address
Role
State
LastUpdMsgId
-------------------------------------------------------------------------------------------------------103
1
leaf3
SAL10000003
10.0.184.64/32
leaf
active
0
104
1
leaf4
SAL10000004
10.0.184.67/32
leaf
active
0
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
L3Out Key Components
3. Advertise BD subnet
Configurations
Bridge Domain (BD)
BD Subnet
•
Subnet A
✓ “Advertised Externally”
VRF Overlay-1
Redistribution
Direct (Subnet A)
-> L3Out Protocol
Associated L3Out
•
non-BLEAF
BLEAF
Target L3Out(s)
to advertise BD subnets
VRF1
VRF1
L3Out
Subnet A
L3Out
EPG
EPG
BD
No BD Subnet A on BLEAF yet
➢ MP-BGP is only to distribute external routes
➢ MP-BGP never distributes BD subnets
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
19
L3Out Key Components
3. Advertise BD subnet
Configurations
Bridge Domain (BD)
BD Subnet
•
Subnet A
✓ “Advertised Externally”
VRF Overlay-1
Redistribution
Direct (Subnet A)
-> L3Out Protocol
Subnet
VRF1A
Associated L3Out
•
Target L3Out(s)
to advertise BD subnets
External Routed Networks (L3Out)
Networks (L3Out EPG)
•
non-BLEAF
BLEAF
Static Route (subnet A)
on BLEAF via MO (object)
Contract to EPG
BRKACI-2642
VRF1
L3Out
Subnet A
L3Out
EPG
EPG
BD
Pushed by APIC. Not MP-BGP.
Please check “pervasive gateway”
in BRKACI-3545
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
L3Out Key Components
3. Advertise BD subnet
1. L3Out Association from BD (for redistribution)
border-leaf# show ip route vrf TK:VRF1
192.168.0.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.0.184.64%overlay-1, [1/0], 04:32:27, static
border-leaf# show ip ospf vrf TK:VRF1
Redistributing External Routes from
direct route-map exp-ctx-st-2326530
2. Contract
ip prefix-list
➢ 192.168.1.0/24
Make sure the other end of contract is configured correctly as well
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
CLI Verification (OSPF, EIGRP)
1. Does the border leaf have BD subnet to advertise?
border-leaf# show ip route vrf TK:VRF1
192.168.1.0/24, ubest/mbest: 1/0, attached, direct,
pervasive
*via 10.0.184.64%overlay-1, [1/0], 04:32:27, static
If not, check the contract between the L3Out EPG
and the EPG for the BD.
This should be pushed by APIC. Not via MP-BGP.
2. Check a route-map name used by the routing protocol on the border leaf for redistribution
border-leaf# show ip ospf vrf TK:VRF1
Redistributing External Routes from
direct route-map exp-ctx-st-2097152
border-leaf# show ip eigrp vrf TK:VRF1
Redistributing:
direct route-map exp-ctx-st-2097152
3. Does the route-map have expected BD subnet?
Check next page for BGP
IP prefix-list should have the BD
border-leaf# show route-map exp-ctx-st-2097152
subnet.
route-map exp-ctx-st-2097152, deny, sequence 1
If not, check APIC config and any
Match clauses:
faults.
tag: 4294967295
✓ Is “Advertise Externally” on the
Set clauses:
route-map exp-ctx-st-2097152, permit, sequence 15804
BD subnet checked?
Match clauses:
✓ Is L3Out associated to the BD?
ip address prefix-lists: IPv4-st49158-2097152-exc-int-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
border-leaf# show ip prefix-list IPv4-st49158-2097152-exc-int-inferred-export-dst
ip prefix-list IPv4-st49158-2097152-exc-int-inferred-export-dst: 1 entries
seq 1 permit 192.168.1.254/24
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI Verification (BGP)
1. Does the border leaf have BD subnet to advertise?
--- snip ---
2. Check a route-map name used by BGP outbound rule for each neighbor
border-leaf# show bgp process vrf TK:VRF1
Information for address family IPv4 Unicast in VRF TK:VRF1
Redistribution
direct, route-map permit-all
BGP redistributes all direct routes first,
then limit the routes with an outbound route-map.
border-leaf# show ip bgp neighbors vrf TK:VRF1 | egrep '^BGP|Out'
BGP neighbor is 17.0.0.1, remote AS 65001, ebgp link, Peer index 1
Outbound route-map configured is exp-l3out-L3OUT_BGP-peer-2097152, handle obtained
3. Does the BGP outbound route-map have the expected BD subnet?
IP prefix-list should have the BD
border-leaf# show route-map exp-l3out-L3OUT_BGP-peer-2097152
subnet.
route-map exp-l3out-L3OUT_BGP-peer-2097152, permit, sequence 15801
If not, check APIC config and any
Match clauses:
faults.
ip address prefix-lists: IPv4-peer49157-2097152-exc-int-inferred-export-dst
✓ Is “Advertise Externally” on the
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
BD subnet checked?
route-map exp-l3out-L3OUT_BGP-peer-2097152, deny, sequence 16000
✓ Is L3Out associated to the BD?
Match clauses:
route-type: direct
Set clauses:
border-leaf# show ip prefix-list IPv4-peer49157-2097152-exc-int-inferred-export-dst
ip prefix-list IPv4-peer49157-2097152-exc-int-inferred-export-dst: 1 entries
seq 1 permit 192.168.1.254/24
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
L3Out Key Components
4. Prefix based Contract
Learning
10.0.0.0/8 and 20.0.0.0/8
through routing protocol
Should be able to
talk with 10.0.0.0/8
10.0.0.0/8
L3Out
?
EPG
20.0.0.0/8
Should NOT be able
to talk with 20.0.0.0/8
How do we accomplish this ??
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
L3Out Key Components
4. Prefix based Contract
10.0.0.0/8
Learning
10.0.0.0/8 and 20.0.0.0/8
through routing protocol
L3Out EPG A
Subnet 10.0.0.0/8
✓ External EPG
L3Out
EPG
L3Out EPG B
20.0.0.0/8
Subnet 20.0.0.0/8
✓ External EPG
Prefix Based EPG (= L3Out EPG)
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
L3Out Key Components
4. Prefix based Contract
Configurations
VRF
External Routed Networks (L3Out)
Node Profile
Interface Profile
Networks (L3Out EPG)
•
A subnet with scope
“External Subnets for
the External EPG”
EPG Classification
based on prefix
L3Out
BD
L3Out
EPG
EPG
EPG (Security Group)
Classification
Prefix Mapping
VLAN + I/F
This scope is VRF wide.
No overlapping with other L3Out EPGs in the same VRF
Traffic from LEAF front panel port
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
L3Out Key Components
4. Prefix based Contract
VRF1 – 10.0.0.0/8 => pcTag 49158
leaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Vrf|10.0.0.0'
Vrf-Vni VRF-Id Table-Id
Addr
Class Shared Remote Complete
2097152 8
0x8
10.0.0.0/8
49158 0
1
No
=== use this command from 3.2 ===
leaf# vsh –c ‘show sytem internal policy-mgr prefix’
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
L3Out Key Components
4. Prefix based Contract
“External Subnets for the External EPG” is
to declare this subnet belongs to this
L3Out EPG
➢ To create prefix to pcTag mapping
NOTE:
It has nothing to do with routing table or
routing protocol behavior unlike other
Route Control Subnet scopes
A common mistake is selecting both
“External Subnets for the External EPG”
and “Export Route Control Subnet” for the
same subnet, which implies a conflicting
situation where the subnet behind the
L3Out but the same L3Out is also
expected to advertise/redistribute the
subnet back to where it came from. It may
not cause an immediate issue but
unnecessary redistribution should always
be avoided.
Check L3Out Subnet scope section for
details.
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
CLI Verifications
1. Check if there is any contract drops
leaf# show logging ip access-list internal packet-log deny
192.168.1.1
EPG
L3Out
EPG 10.0.0.0/8
Contract Drop on this leaf shows up in this command.
Check both ingress/egress leaf just in case,
or see appendix for Policy Control Enforcement Direction
[ Wed May 8 18:34:31 2019 155907 usecs]: CName: TK:VRF1(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 26, SMac: 0x0050569185d1,
DMac:0x0022bdf819ff, SIP: 192.168.1.1, DIP: 10.0.0.1, SPort: 58968, DPort: 80, Src Intf: port-channel1, Proto: 6, PktLen: 74
2. Check VRF VNID
leaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
pcTag/contract is per VRF
except for shared service (VRF route leaking)
If your source/destination is an endpoint, it should
be in here.
sclass = pcTag = EPG ID for contract
leaf# show system internal epm endpoint ip 192.168.1.1 | egrep 'VRF|sclass'
3. Check source (or destination) EPG pcTag
Vlan id : 30 ::: Vlan vnid : 9025 ::: VRF name : TK:VRF1
BD vnid : 16318374 ::: VRF vnid : 2097152
Flags : 0x80005c04 ::: sclass : 49100 ::: Ref count : 5
EP Flags : local|IP|MAC|host-tracked|sclass|timer|
Make sure the external IP is not here.
This pcTag takes precedence over “prefix-pcTag
mapping table”. If it is, check the traffic path that
caused ACI to learn the external IP as an endpoint.
4. Check destination (or source) L3Out prefix based EPG pcTag
leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Vrf|10.0.0.0'
Vrf-Vni VRF-Id Table-Id
Addr
Class Shared Remote Complete
2097152 8
0x8
10.0.0.0/8
49200 0
1
No
=== use this command from 3.2 ===
leaf# vsh –c ‘show sytem internal policy-mgr prefix’
“External Subnet for the External EPG”
config is reflected here.
This is Longest Prefix Match.
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI Verifications
5. Check contracts between two pcTags
leaf# show
Rule ID
4165
4124
192.168.1.1
49100
L3Out
EPG 10.0.0.0/8
49200
scope = VRF VNID
zoning-rule scope 2097152 | egrep ‘Rule|49100|49200’
SrcEPG
DstEPG
FilterID
operSt
Scope
49100
49200
5
enabled
2097152
49200
49100
5
enabled
2097152
leaf# show zoning-filter
FilterId Name
EtherT
======== ====== ======
5
5_0
ip
EPG
Action
permit
permit
Priority
fully_qual(7)
fully_qual(7)
filter 5
ArpOpc
Prot
~snip~ SFromPort
SToPort
DFromPort
DToPort
~snip~
=========
======= ~snip~ =======
====
====
====
~snip~
unspecified icmp
~snip~ unspecified unspecified unspecified unspecified ~snip~
6. Check ELAM to see if the traffic is using correct src pcTag and dst pcTag
https://dcappcenter.cisco.com/elam-assistant.html
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
30
L3Out Contract
Common Issue (L3Out EPGs with 0.0.0.0/0)
VRF 1
L3Out A
L3Out EPG A
0.0.0.0/0
✓ External EPG
EPG X
10.0.0.0/8
0.0.0.0/0 should cover
10.0.0.0/8
L3Out B
L3Out EPG B
0.0.0.0/0
✓ External EPG
20.0.0.0/8
0.0.0.0/0 should cover
20.0.0.0/8
Both 10.0.0.0/8 and 20.0.0.0/8 can talk to EPG X
even though there is no contract between L3Out EPG B and EPG X
➢ Prefix-pcTag mapping is per VRF.
0.0.0.0/0 for L3Out A and B ends up in the same entry.
BRKACI-2642
Do not overlap
External EPG
subnets
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
31
L3Out Contract
192.168.1.1 EPG X
Common Issue (L3Out EPGs with 0.0.0.0/0)
L3Out
EPG A
10.0.0.0/8
L3Out
EPG B
20.0.0.0/8
1. Check VRF VNID
leaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
“0.0.0.0/0 -> 15” is the only pcTag entry
in this VRF.
2. Check source (or destination) EPG pcTag
➢ Both L3Out A & B will share it since
there is no other granular LPM entries
leaf# show system internal epm endpoint ip 192.168.1.1 | egrep 'VRF|sclass'
Vlan id : 30 ::: Vlan vnid : 9025 ::: VRF name : TK:VRF1
BD vnid : 16318374 ::: VRF vnid : 2097152
Flags : 0x80005c04 ::: sclass : 49100 ::: Ref count : 5
3. Check destination L3Out 0.0.0.0/0 EPG pcTag
NOTE:
• 0.0.0.0/0 always use pcTag 15
• This is not a routing table. It doesn’t matter even if
the routing table has more granular routes
leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep ‘Vrf|2097152'
Vrf-Vni VRF-Id Table-Id
Addr
Class Shared Remote Complete
2097152 8
0x8
0.0.0.0/0
15
0
0
No
4. Check contracts between pcTags
leaf# show zoning-rule scope 2097152 | egrep ‘Rule|49162’
Rule ID
SrcEPG
DstEPG
FilterID
operSt
4165
49100
15
5
enabled
This contract is due to “EPG X <-> L3Out A”
But any traffic that hits 0.0.0.0/0 in the prefix table
can use this rule
Scope
2097152
BRKACI-2642
Action
permit
Priority
fully_qual(7)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
32
L3Out Subnet Scope
L3Out Subnet Scope
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
34
L3Out Subnet Scope
Route Control for Routing Protocol
• Export Route Control Subnet
• Import Route Control Subnet
• Shared Route Control Subnet
Traffic Classification for Contract
• External Subnets for the External EPG
Grouping by
functionality
• Shared Security Import Subnet
Aggregate
• Aggregate Export
• Aggregate Import
• Aggregate Shared Routes
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
L3Out Subnet Scope Summary
Only for contracts
No impact in routing table
Route Control for Routing Protocol
Import Route Control Subnet
(Mainly for Transit Routing)
(Only for OSPF or BGP)
L3Out
Protocol
Database
L3Out
Protocol
Database
export-filter
Shared Route Control Subnet
VRF1
L3Out
Protocol
Database
import-filter
shared-filter
Export Route Control Subnet
Subnet Classification
VRF2
L3Out
EPG1
Leak the external route to
different VRF
aggregation
Aggregate Export
Aggregate Import
Aggregate Shared Route
0.0.0.0/0 le 32
0.0.0.0/0 le 32
Advertise all routes from ACI
to outside
Receive all routes from
outside
VRF1
L3Out
Protocol
Database
X.X.X.X/X le 32
Receive the route from outside
(by default, receive all)
aggregation
L3Out
Protocol
Database
L3Out
L3Out
EPG2
10.0.0.0/8
Advertise the route from ACI
to outside
aggregation
L3Out
Protocol
Database
External Subnet for
the External EPG
VRF2
Leak multiple external
© 2020
routes to different VRF
20.0.0.0/8
Group subnets into
each L3Out EPG (pcTag)
Shared Security Import
VRF1
L3Out
L3Out
EPG1
VRF2
L3Out
EPG1
10.0.0.0/8
Leak prefix-pcTag mapping
to different VRF
Cisco and/or its affiliates. All rights reserved. Cisco Public
Route Control Enforcement
Import is disabled by default.
➢ Receive all routes by
default.
No import route control.
Export is always enabled.
Route
Control for
Routing
Protocol
Available
only
when
enabled
Export Route Control Subnet
Import Route Control Subnet
Shared Route Control Subnet
(Only for OSPF or BGP)
export-filter
VRF1
L3Out
Protocol
Database
L3Out
Protocol
Database
import-filter
VRF2
shared-filter
L3Out
Protocol
Database
Receive the route from outside
(by default, receive all)
aggregation
Leak the external route to
different VRF
aggregation
Aggregate Export
Aggregate Import
Aggregate Shared Route
L3Out
L3Out
VRF1
Protocol
Database
Protocol
Database
L3Out
Protocol
Database
0.0.0.0/0 le 32
Advertise all routes from ACI
to outside
0.0.0.0/0 le 32
Receive all routes from
outside
X.X.X.X/X le 32
Advertise the route from ACI
to outside
aggregation
VRF2
Leak multiple external
routes to different VRF
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Export Route Control (OSPF)
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF
User VRF
Route Export
Route Import
Route-maps
BGP (IPv4)
Redistribute
permit-all
L3Out 1
permit-all
OSPF Protocol Database
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
Export Route Control (OSPF)
VRF overlay-1
MP-BGP (vpnv4)
Creates a route-map
ACI Border LEAF
User VRF
Route Export
Route Import
Route-maps
BGP (IPv4)
Redistribute
permit-all
L3Out 1
export
IP prefix-list
10.0.0.0/8
permit-all
OSPF Protocol Database
== NOTE ==
Be careful when deploying
multiple L3Outs in one VRF.
Route maps are shared with
other protocols (L3Out) in the
same VRF on the same LEAF.
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
39
10.0.0.0/8
Export Route Control
(OSPF)
User
VRF
L3Out
2
Protocol
Database
BGP (IPv4)
Another Border LEAF
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF
User VRF
From another L3Out
on different LEAF
using MP-BGP
Route Import
Route Export
BGP (IPv4)
Redistribute
permit-all
L3Out 1
Route-maps
export
export
Redistribute
permit-all
OSPF Protocol Database
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
40
10.0.0.0/8
Export Route Control
(OSPF)
User
VRF
L3Out
2
Protocol
Database
BGP (IPv4)
Another Border LEAF
VRF overlay-1
MP-BGP (vpnv4)
Advertise external routes
from other L3Out(s)
➢ Transit Routing
ACI Border LEAF
From another L3Out
on different LEAF
using MP-BGP
Route Import
Route Export
Route-maps
BGP (IPv4)
Redistribute
export
permit-all
L3Out 1
export
L3Out 3
Redistribute
OSPF Protocol Database
export
User VRF
Redistribute
or Area-filter
Protocol
Database
permit-all
From another L3Out
on same LEAF
10.0.0.0/8
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
41
10.0.0.0/8
Export Route Control
(OSPF)
User
VRF
L3Out
2
Protocol
Database
BGP (IPv4)
Another Border LEAF
VRF overlay-1
MP-BGP (vpnv4)
Advertise BD subnets
➢ 2nd method
NOTE:
ACI Border LEAF
From 3.0, “Advertised Externally”
on BD subnet is also required with
this method
User VRF
Advertise external routes
from other L3Out(s)
➢ Transit Routing
Route Import
Route Export
Route-maps
BGP (IPv4)
Redistribute
export
permit-all
L3Out 1
BD
export
L3Out 3
RIB
export
Subnets
OSPF Protocol Database
export
Redistribute
Protocol
Database
permit-all
Redistribute
or Area-filter
Redistribute
10.0.0.0/8
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
42
CLI Verification (OSPF/EIGRP)
1. OSPF/EIGRP Redistribution route-map
border-leaf# show ip ospf vrf TK:VRF1
Redistributing External Routes from
static route-map exp-ctx-st-2097152
direct route-map exp-ctx-st-2097152
bgp route-map exp-ctx-proto-2097152
eigrp route-map exp-ctx-proto-2097152
Area (backbone)
Area-filter in 'exp-ctx-proto-2097152'
border-leaf# show ip eigrp vrf TK:VRF1
Redistributing:
static route-map exp-ctx-st-2097152
ospf-default route-map exp-ctx-proto-2097152
direct route-map exp-ctx-st-2097152
bgp-65003 route-map exp-ctx-proto-2097152
2. route-map and ip prefix-list
It shares the same route-map with other
protocols in the same VRF on the same LEAF
route-map naming:
exp-ctx-st-<vrf vnid> or
exp-ctx-proto-<vrf vnid>
EIGRP doesn’t support Transit Routing on a same LEAF.
➢ No equivalent filter like OSPF area-filter in EIGRP
All Export Route Control subnet on a
same LEAF is added here
border-leaf# show route-map exp-ctx-proto-2097152
Same goes to exp-cxt-st-2097152
route-map exp-ctx-proto-2097152, permit, sequence 15801
Match clauses:
ip address prefix-lists: IPv4-proto49158-2097152-exc-ext-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
border-leaf# show ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst
tag 4294967295
ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst: 1 entries
seq 1 permit 10.0.0.0/8
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
CLI Verification (BGP)
1. BGP outbound route-map
BGP has a route-map per L3Out
➢ A bit more granular control
route-map naming:
exp-l3out-<bgp l3out name>-peer-<vrf vnid>
border-leaf# show ip bgp neighbors vrf TK:VRF1
BGP neighbor is 17.0.0.1, remote AS 65001, ebgp link, Peer index 1
Outbound route-map configured is exp-l3out-L3OUT_BGP-peer-2097152, handle obtained
2. route-map and ip prefix-list
All Export Route Control subnets from
border-leaf# show route-map exp-l3out-L3OUT_BGP-peer-2097152
the same BGP L3Out is added here
route-map exp-l3out-L3OUT_BGP-peer-2097152, permit, sequence 15804
Match clauses:
ip address prefix-lists: IPv4-peer49157-2097152-exc-ext-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
tag 4294967295
route-map exp-l3out-L3OUT_BGP-peer-2097152, deny, sequence 16000
Match clauses:
route-type: direct
Set clauses:
border-leaf# show ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-export-dst
ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-export-dst: 4 entries
seq 1 permit 10.0.0.0/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
44
BD Subnet and Export Route Control
border-leaf# show ip route vrf TK:VRF1
192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 11.0.248.0%overlay-1, [1/0], 00:00:05, static, tag 4294967295
“Advertised Externally” removes VRF
tag from BD subnet
border-leaf# show ip route vrf TK:VRF1
192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 11.0.248.0%overlay-1, [1/0], 00:00:05, static
IP prefix-list from
“Export Route Control”
for 192.168.1.0/24
Prior to 3.0
border-leaf# show route-map exp-ctx-st-2097152
route-map exp-ctx-st-2097152, permit, sequence 15804
Match clauses:
ip address prefix-lists: IPv4-st49158-2097152-exc-int-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
From 3.0
border-leaf# show route-map exp-ctx-st-2097152
route-map exp-ctx-st-2097152, deny, sequence 1
Match clauses:
IP prefix-list from
tag: 4294967295
“Export Route Control”
Set clauses:
route-map exp-ctx-st-2097152, permit, sequence 15804
for 192.168.1.0/24
Match clauses:
ip address prefix-lists: IPv4-st49158-2097152-exc-int-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
New rule to prevent BD subnets
without “Advertised Externally”
from being advertised
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
Import Route Control (OSPF)
VRF overlay-1
MP-BGP (vpnv4)
Creates a route-map
ACI Border LEAF
User VRF
Route Import
Route Export
Route-maps
BGP (IPv4)
Redistribute
export
permit-all
L3Out
import
IP prefix-list
10.0.0.0/8
permit-all
OSPF Protocol Database
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
46
Import Route Control (OSPF)
VRF overlay-1 If it’s not allowed to be imported, it should
not
be exported to other L3Outs.
MP-BGP
(vpnv4)
ACI Border LEAF
User VRF
Limit external routesRoute Import
to be used in RIB.
It is still in OSPF LSDB.
When L3Out2 is not OSPF
Redistribution is blocked by table-map
(only routes in RIB can be redistributed)
When L3Out2 is OSPF
Route Export
Block advertisement to another OSPF area
BGP (IPv4)
via area-filter
Redistribute
export
permit-all
import
permit-all
OSPF Protocol Database
Table-map
area-filter
out
External Router
BRKACI-2642
export
RIB
L3Out 2
import
Subnets
import
L3Out
BD
Routemaps
Protocol
Database
Still need export in L3Out2
(OSPF) on top of import in
L3Out1 (OSPF)
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
47
CLI Verification (OSPF)
border-leaf# show ip ospf vrf TK:VRF1
Table-map using route-map exp-ctx-2097152-deny-external-tag
Area (backbone)
Area-filter out 'imp-ctx-ospf-area20971520'
•
•
Table-map to prevent the routes from being
used in RIB
“Area-filter out” to prevent the routes from
being advertised to another OSPF area on a
same LEAF (Transit Routing)
border-leaf# show route-map exp-ctx-2097152-deny-external-tag
route-map for table-map
route-map exp-ctx-2097152-deny-external-tag, deny, sequence 1
1. blocks any routes with VRF tag
Match clauses:
2. allow routes with Import Route Control
tag: 4294967295
Set clauses:
subnet in OSPF area X
route-map exp-ctx-2097152-deny-external-tag, permit, sequence 15801
3. block any routes from OSPF area X
Match clauses:
ip address prefix-lists: IPv4-ospf-49158-2097152-exc-ext-inferred-import-dst-rtpfx
ipv6 address prefix-lists: IPv6-deny-all
ospf-area: backbone
Set clauses:
route-map exp-ctx-2097152-deny-external-tag, deny, sequence 19999
Match clauses:
ospf-area: backbone
Set clauses:
A prefix configured by “Import Route
route-map exp-ctx-2097152-deny-external-tag, permit, sequence 20000
Control Subnet”
Match clauses:
Set clauses:
border-leaf# show ip prefix-list IPv4-ospf-49158-2097152-exc-ext-inferred-import-dst-rtpfx
ip prefix-list IPv4-ospf-49158-2097152-exc-ext-inferred-import-dst-rtpfx: 1 entries
seq 1 permit 10.0.0.0/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
48
CLI Verification (OSPF) cont.
border-leaf# show ip ospf vrf TK:VRF1
Table-map using route-map exp-ctx-2097152-deny-external-tag
Area (backbone)
Area-filter out 'imp-ctx-ospf-area20971520'
•
•
Table-map to prevent the routes from being
used in RIB
“Area-filter out” to prevent the routes from
being advertised to another OSPF area on a
same LEAF (Transit Routing)
border-leaf# show route-map imp-ctx-ospf-area20971520
route-map for area-filter
route-map imp-ctx-ospf-area20971520, permit, sequence 15801
Match clauses:
ip address prefix-lists: IPv4-ospf-rt-ospf-import49158-2097152-exc-ext-inferred-import-dstipv6 address prefix-lists: IPv6-deny-all
Set clauses:
border-leaf# show ip prefix-list IPv4-ospf-rt-ospf-import49158-2097152-exc-ext-inferred-import-dstip prefix-list IPv4-ospf-rt-ospf-import49158-2097152-exc-ext-inferred-import-dst-: 1 entries
seq 1 permit 10.0.0.0/8
A prefix configured by “Import Route
Control Subnet”
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
CLI Verification (BGP)
BGP uses an inbound route-map (per L3Out) instead of table-map
border-leaf# show ip bgp neighbors vrf TK:VRF1
BGP neighbor is 17.0.0.1, remote AS 65001, ebgp link, Peer index 1
Inbound route-map configured is imp-l3out-L3OUT_BGP-peer-2097152, handle obtained
border-leaf1# show route-map imp-l3out-L3OUT_BGP-peer-2097152
route-map imp-l3out-L3OUT_BGP-peer-2097152, permit, sequence 15801
Match clauses:
ip address prefix-lists: IPv4-peer49157-2097152-exc-ext-inferred-import-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
border-leaf# show ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-import-dst
ip prefix-list IPv4-peer49157-2097152-exc-ext-inferred-import-dst: 1 entries
seq 1 permit 10.0.0.0/8
A prefix configured by “Import Route
Control Subnet”
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
Shared Route Control Subnet
(VRF Route Leaking) Configuration in VRF1 L3Out
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1
Route Export
BGP (IPv4)
Redistribute
permit-all
User Route Import
VRF2
Import RT
<AS>:<VRF2 VNID>
L3Out
Route Export
BGP (IPv4)
Route-maps
== default ==
Only import L3Out routes for
same VRF (VRF2) from other
LEAF
OSPF Protocol Database
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
51
Shared Route Control Subnet
(VRF Route Leaking) Configuration in VRF1 L3Out
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1
Route Export
BGP (IPv4)
Redistribute
permit-all
L3Out
User Route Import
VRF2
Import RT
<AS>:<VRF2 VNID>
<AS>:<VRF1 VNID>
OSPF Protocol Database
EPG
BRKACI-2642
Route Export
BGP (IPv4)
Route-maps
Contract across VRFs
➢ Import VRF1
routes as well
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
52
Shared Route Control Subnet
(VRF Route Leaking) Configuration in VRF1 L3Out
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1
shared
Route Export
BGP (IPv4)
Redistribute
permit-all
L3Out
User Route Import
VRF2
Route Export
BGP (IPv4)
Import RT
<AS>:<VRF2 VNID>
<AS>:<VRF1 VNID>
OSPF Protocol Database
EPG
Route-maps
shared
IP prefix-list
10.0.0.0/8
Limit routes to be
imported (leaked)
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
CLI Verification
1. MP-BGP Import rule with another VRF VNID route-target and a route-map
leaf# show bgp process vrf TK:VRF2
Information for address family IPv4 Unicast in VRF TK:VRF2
Import route-map 2588672-shared-svc-leak
Export RT list:
65003:2588672
Import RT list:
65003:2097152
65003:2588672
Label mode: per-prefix
leaf# show vrf TK:VRF1 detail extended | egrep 'RD|vxl'
RD: 10.0.184.64:2
Encap: vxlan-2097152
•
It always has Import and Export RT for its
own VRF2 VNID (65003:2588672)
•
VRF Route Leaking is handled by Import RT
and Import route-map (highlighted ones)
VRF VNID can be checked with this
command to confirm Import RT is correct
leaf# show vrf TK:VRF2 detail extended | egrep 'RD|vxl'
RD: 10.0.184.64:13
Encap: vxlan-2588672
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
54
CLI Verification
2. A route-map for shared service (VRF Route Leaking)
leaf# show route-map 2588672-shared-svc-leak
route-map 2588672-shared-svc-leak, deny, sequence 1
Match clauses:
pervasive: 2
Set clauses:
route-map 2588672-shared-svc-leak, permit, sequence 2
Match clauses:
extcommunity (extcommunity-list filter): 2588672-shared-svc-leak
Set clauses:
route-map 2588672-shared-svc-leak, permit, sequence 1000
Match clauses:
ip address prefix-lists: IPv4-2097152-32771-18-2588672-shared-svc-leak
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
leaf# show ip extcommunity-list 2588672-shared-svc-leak
Standard Extended Community List 2588672-shared-svc-leak
permit RT:65003:2588672
1. Prevent BD subnet (pervasive
route) from being imported via
MP-BGP.
BD subnet distribution should be
done by APIC instead of MP-BGP.
2. Allow importing any routes from
the same VRF.
Extended community list has RT
for the same VRF VNID.
3. Allow importing certain routes
from another VRF
RT for the same VRF VNID
Not for VRF Route Leaking
leaf# show ip prefix-list IPv4-2097152-32771-18-2588672-shared-svc-leak
ip prefix-list IPv4-2097152-32771-18-2588672-shared-svc-leak: 1 entries
seq 1 permit 10.10.10.0/8
BRKACI-2642
IP Prefix-List from Shared
Route Control Subnet
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Shared Security Import Subnet
Routing table is leaked via MP-BGP
and “Shared Route Control Subnet”
MP-BGP
User
VRF1
User
VRF2
Prefix <-> pcTag mapping is leaked
via APIC and “Shared Security Import
Subnet”
VRF1 RIB
10.0.0.0/8 -> Local
Prefix – pcTag mapping
VRF1: 10.0.0.0/8 -> pcTag X
VRF2 RIB
10.0.0.0/8 -> LEAF 1 in VRF 1
Prefix – pcTag mapping
10.0.0.0/8 -> pcTag X
Routing
Protocol
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
CLI Verification
1. Check VRF VNID
leaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
pcTag (class) for shared route
leaf# show vrf TK:VRF2 detail extended | grep vxlan
Encap: vxlan-2588672
prefix-pcTag mapping is sahred
to VRF2 (VNID 2588672)
2. Prefix – pcTag mapping table
1st-gen-leaf# vsh_lc –c ‘show system internal aclqos prefix’ | egrep '^Shared|52.52.52'
Shared Addr
Mask
Scope Class RefCnt
10.0.0.0
ffffff
0
18
1
2nd-gen-leaf# vsh_lc -c 'show system internal aclqos prefix' | egrep 'Shared|52.52.52'
Vrf-Vni VRF-Id Table-Id
Addr
Class Shared Remote Complete
2097152 8
0x8
10.0.0.0/8
18
0
1
No
2588672 11
0xb
10.0.0.0/8
18
1
1
No
leaf# vsh -c 'show system internal policy-mgr prefix'
Vrf-Vni VRF-Id Table-Id Table-State VRF-Name
Addr
2097152 8
0x8
Up
TK:VRF1
10.0.0.0/8
2588672 11
0xb
Up
TK:VRF2
10.0.0.0/8
Class Shared Remote Complete
18
True
True
False
18
True
True
False
From 3.2 release, use this command
regardless of leaf generations
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Aggregate Route Control
== Export ==
== Import ==
Route-maps
Route-maps
export
import
IP prefix-list
0.0.0.0/0 le 32
== Shared ==
Route-maps
shared
IP prefix-list
10.0.0.0/8 le
32
IP prefix-list
0.0.0.0/0 le 32
permit-all
permit-all
permit-all
Only “Aggregate Shared Routes” support non-0.0.0.0/0 aggregation
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
OSPF route-map
Export Route Control Subnet
xxxxxxxx
Import Route Control Subnet
xxxxxxxxx
Per
VRF/LEAF
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF
User VRF
Route Import
Route Export
Routemaps
export-proto
BGP (IPv4)
Redistribute
permit-all
exportstatic
Redistribute
OSPF Protocol Database
importopsf
denyexternal
RIB
area-filter
out
Table-map
External Router
BRKACI-2642
export-static
deny-external
import-ospf
exportproto
Subnets
Redistribute or
area-filter in L3Out 2
export
L3Out
BD
exportproto Redistribute
Protocol
Database
permit-all
Still needs export in
L3Out2 on top of
import in L3Out1
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
59
EIGRP route-map
Export Route Control Subnet
Import is not supported for EIGRP
Used only for VRF Tag
xxxxxxxx
deny-external
Per
VRF/LEAF
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF
User VRF
Route Import
Route Export
Routemaps
export-proto
BGP (IPv4)
Redistribute
permit-all
L3Out
Redistribute
denyexternal
RIB
export-static
L3Out 2
deny-external
Redistribute
EIGRP Protocol Database
exportproto
Subnets
exportstatic
BD
exportproto Redistribute
Protocol
Database
permit-all
Table-map
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
60
BGP route-map
Export Route Control Subnet
export-l3out
Import Route Control Subnet
import-l3out
Per
L3Out/LEAF
VRF overlay-1
MP-BGP (vpnv4)
ACI Border LEAF
User VRF
Route Import
Route Export
Routemaps
export-l3out
L3Out
outbound route-map
permit-all
BGP (IPv4)
export-l3out
import-l3out
Redistribute
L3Out 2
Protocol
Database
permit-all
import-l3out
Inbound route-map
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Routing Loop
Loop Avoidance for OSPF/EIGRP – VRF Tag
EIGRP and MP-BGP Redistribution Issue
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
Block routes with its own VRF tag
Set VRF tag
when exporting routes
User VRF
L3Out 1
(By default 4294967295)
It may overwrite the original route
“10.0.0.0/8 => Router 1”
(By default 4294967295 for all VRF)
L3Out 2
10.0.0.0/8 => Router 1
Router 1
export
Protocol Database
L3Out 3
Protocol Database
Router 2
Protocol Database
10.0.0.0/8 => Router 3
(tag 4294967295)
10.0.0.0/8 => L3Out 2
(tag 4294967295)
Router 3
10.0.0.0/8
※ VRF tagging for exported routes and blocking routes with VRF tag are always enabled
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
It will be blocked in other VRFs
as well since all VRF use same
VRF tag by default
Set VRF tag
when exporting routes
User VRF 1
L3Out 1
(By default 4294967295 for all VRF)
User VRF 2
L3Out 2
10.0.0.0/8 => Router 1
Router 1
export
Protocol Database
L3Out 3
Protocol Database
Router 2
Protocol Database
10.0.0.0/8 => Router 3
(tag 4294967295)
10.0.0.0/8 => L3Out 2
(tag 4294967295)
Router 3
10.0.0.0/8
※ VRF tagging for exported routes and blocking routes with VRF tag are always enabled
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
User VRF 1
L3Out 1
VRF2’s tag is 200 not 100.
➢ Routes are not blocked
Set VRF tag 100
when exporting routes
User VRF 2
L3Out 2
10.0.0.0/8 => Router 1
Router 1
export
Protocol Database
L3Out 3
Protocol Database
Router 2
Protocol Database
10.0.0.0/8 => Router 3
(tag 100)
10.0.0.0/8 => L3Out 2
(tag 100)
Router 3
10.0.0.0/8
VRF 1 tag – 100
VRF 2 tag – 200
BRKACI-2642
Do this when routes need to be
advertised back to ACI
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
VRF tag can be
configured per VRF
In this example VRF1’s tag is 100
--- snip ---
※ VRF tag is only for OSPF and EIGRP
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
66
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
leaf# show ip ospf vrf TK:VRF1 | egrep 'route-map|Redis'
Table-map using route-map exp-ctx-2097152-deny-external-tag
Redistributing External Routes from
static route-map exp-ctx-st-2097152
direct route-map exp-ctx-st-2097152
eigrp route-map exp-ctx-proto-2097152
bgp route-map exp-ctx-proto-2097152
Export routes with VRF tag
leaf# show route-map exp-ctx-proto-2097152
route-map exp-ctx-proto-2097152, permit, sequence 15802
Match clauses:
ip address prefix-lists: IPv4-proto49158-2097152-exc-ext-inferred-export-dst
ipv6 address prefix-lists: IPv6-deny-all
Set clauses:
tag 100
leaf# show ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst
ip prefix-list IPv4-proto49158-2097152-exc-ext-inferred-export-dst: 1 entries
seq 1 permit 10.0.0.0/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
Routing Loop Avoidance - VRF tag (OSPF/EIGRP)
note:
Always there with VRF tag
Import Route Control Subnet is added
here after VRF tag deny rule when
Import Route Control subnet is used.
leaf# show ip ospf vrf TK:VRF1 | egrep 'route-map|Redis'
Table-map using route-map exp-ctx-2097152-deny-external-tag
Redistributing External Routes from
static route-map exp-ctx-st-2097152
direct route-map exp-ctx-st-2097152
eigrp route-map exp-ctx-proto-2097152
bgp route-map exp-ctx-proto-2097152
Block routes with
VRF tag
leaf# show route-map exp-ctx-2097152-deny-external-tag
route-map exp-ctx-2097152-deny-external-tag, deny, sequence 1
Match clauses:
tag: 100
Set clauses:
route-map exp-ctx-2097152-deny-external-tag, permit, sequence 200
Match clauses:
Set clauses:
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
68
EIGRP & MP-BGP Redistribution Issue
1. 10.0.0.0/8 from L3Out 1 via EIGRP on two Border LEAFs
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1
Route Export
BGP (IPv4)
Redistribute
L3Out1
User
VRF1
Route Import
Route Export
BGP (IPv4)
Redistribute
L3Out1
EIGRP Topology
EIGRP Topology
10.0.0.0/8 FD 100000
10.0.0.0/8 FD 100000
10.0.0.0/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
69
EIGRP & MP-BGP Redistribution Issue
2. L3Out 1 exports all routes (including 10.0.0.0/8)
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1
L3Out1
Route Export
BGP (IPv4)
Redistribute
Redistribute
0.0.0.0/0 le 32
User
VRF1
Route Import
L3Out1
EIGRP Topology
Route Export
BGP (IPv4)
Redistribute
Redistribute
0.0.0.0/0 le 32
EIGRP Topology
10.0.0.0/8 FD 100000
10.0.0.0/8 FD 100000
10.0.0.0/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
70
EIGRP & MP-BGP Redistribution Issue
3. Redistributed routes have lower metric than the original
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1
L3Out1
Route Export
BGP (IPv4)
Redistribute
Redistribute
0.0.0.0/0 le 32
User
VRF1
EIGRP Topology
Route Export
BGP (IPv4)
Redistribute
L3Out1
Redistribute
0.0.0.0/0 le 32
EIGRP Topology
10.0.0.0/8 FD 51200
Overwrite the
original
Route Import
10.0.0.0/8 FD 51200
Overwrite the
original
10.0.0.0/8 FD 100000
10.0.0.0/8
BRKACI-2642
10.0.0.0/8 FD 100000
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
71
EIGRP & MP-BGP Redistribution Solution1
Export only necessary routes
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1Redistribute
L3Out1
Route Export
BGP (IPv4)
Redistribute
20.0.0.0/8
User
VRF1
Route Import
BGP (IPv4)
Redistribute
Redistribute
20.0.0.0/8
L3Out1
EIGRP Topology
Not Redistributed
back
Route Export
EIGRP Topology
Not Redistributed
back
10.0.0.0/8 FD 100000
10.0.0.0/8 FD 100000
10.0.0.0/8
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
72
EIGRP & MP-BGP Redistribution Solution2
Add “set metric” rule to export route-map
VRF overlay-1
MP-BGP (vpnv4)
User Route Import
VRF1Redistribute
L3Out1
Route Export
BGP (IPv4)
Set metric
Redistribute
0.0.0.0/0 le 32
User
VRF1
Route Import
BGP (IPv4)
Set metric
Redistribute
Redistribute
0.0.0.0/0 le 32
L3Out1
EIGRP Topology
Route Export
EIGRP Topology
10.0.0.0/8 FD 2588162
10.0.0.0/8 FD 2588162
Lower metric
Not used
Lower metric
Not used
10.0.0.0/8 FD 100000
10.0.0.0/8
BRKACI-2642
10.0.0.0/8 FD 100000
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
73
EIGRP & MP-BGP Redistribution Solution2
Add “set metric” rule to export route-map
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
L3Out Contract
deep dive
L3Out Contract
pcTag (policy control Tag) in normal EPG
Source EP
Learning
On APIC
EPG A
EPG B
ICMP
Forwarding
Lookup
Source EPG
Destination EPG
Check
Check
VLAN + I/F
On LEAF
source
pcTag A
Forwarding Result
get pcTag
destination
Filter
pcTag B
ICMP
pcTag A
pcTag B
Contract Filter Check
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
76
L3Out Contract
Src: Subnet A -> Dst: Subnet B
pcTag (policy control Tag) in L3Out EPG
Source EP
Learning
On APIC
L3Out EPG A
L3Out EPG B
Subnet A
✓ External EPG
Subnet B
✓ External EPG
ICMP
On LEAF
subnet
VRF1
VRF1
VRF1
subnet A
subnet B
0.0.0.0/0
pcTag A
default catch all
not used in this example
Lookup
Source EPG (pcTag)
Destination EPG (pcTag)
Check
Check
VLAN + I/F
VRF
source
Forwarding
pcTag
pcTag A
pcTag B
pcTag 15
destination
Filter
pcTag B
ICMP
Forwarding Result
pcTag VRF
Prefix To pcTag mapping for L3Out
pcTag A
Hit subnet A
pcTag B
Hit subnet B
Contract Filter Check
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
77
L3Out Contract
pcTag (policy control Tag) in L3Out EPG with 0.0.0.0/0
Source EP
Learning
On APIC
L3Out EPG A
L3Out EPG B
0.0.0.0/0
✓ External EPG
Subnet B
✓ External EPG
ICMP
On LEAF
subnet
VRF1
VRF1
subnet B
0.0.0.0/0
pcTag VRF
default catch all
not used in this example
Lookup
Source EPG (pcTag)
Check
VLAN + I/F
VRF
source
Forwarding
pcTag
pcTag B
pcTag 15
destination
Filter
pcTag B
ICMP
Destination EPG (pcTag)
Check
Forwarding Result
pcTag VRF
Prefix To pcTag mapping for L3Out
pcTag VRF
No Hit for subnet A
Keep pcTag from
VLAN + I/F
pcTag B
Contract Filter Check
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
78
L3Out Contract
pcTag (policy control Tag) in L3Out EPG with 0.0.0.0/0
Source EP
Learning
On APIC
L3Out EPG A
L3Out EPG B
Subnet A
✓ External EPG
0.0.0.0/0
✓ External EPG
ICMP
On LEAF
subnet
pcTag
VRF1
subnet A
pcTag A
VRF1
0.0.0.0/0
pcTag 15
pcTag A
Lookup
Source EPG (pcTag)
Check
VLAN + I/F
VRF
source
Forwarding
destination
Filter
pcTag 15
ICMP
Destination EPG (pcTag)
Check
Forwarding Result
pcTag VRF
Prefix To pcTag mapping for L3Out
pcTag A
No Hit for subnet B
Use default pcTag 15
pcTag
15
Contract Filter Check
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
79
L3Out Contract
Common Issue (L3Out EPGs with 0.0.0.0/0)
On APIC
VRF 1
Prefix-pcTag entry is per VRF.
Default catch all (0.0.0.0) is shared
with everyone in the VRF.
On LEAF
L3Out A
L3Out EPG A
0.0.0.0/0
✓ External EPG
EPG X
ICMP
10.0.0.0/8
L3Out B
subnet
pcTag
VRF1
0.0.0.0/0
pcTag 15
source
L3Out EPG B
0.0.0.0/0
✓ External EPG
VRF
pcTag X
pcTag VRF
20.0.0.0/8
destination
Filter
pcTag 15
pcTag X
ICMP
ICMP
These contracts are from EPG X and L3Out A
However, traffic from/to L3Out B (20.0.0.0/8) will also use
default pcTag (VRF or 15) due to 0.0.0.0/0 config.
No overlap of External EPG L3Out subnets in same VRF
Use 0.0.0.0/0 (External subnet for the external EPG) only for one L3Out EPG per VRF
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
80
How to get pcTag for L3Out
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
81
How to get pcTag for normal EPG
== Policy tab ==
➢ Check EPG’s pcTag
== Operational tab ==
➢ Check if the endpoint is learned on
the expected EPG
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
82
How to get VRF pcTag
• From APIC
admin@apic1:~> moquery -c fvCtx -f 'fv.Ctx.name=="VRF1"' | egrep '#|dn|pcTag'
# fv.Ctx
dn
: uni/tn-TK/ctx-VRF1
pcTag
: 49153
• From LEAF
leaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep sclass
scope:
4
:::
sclass:
49153
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
83
Under VRF
L3Out Contract
Policy Control Enforcement Direction
A feature to save contract TCAM usage on border LEAF
APIC
EPG A
L3Out EPG X
EPG B
Subnet X
✓ External EPG
ICMP
Egress Policy Enforcement
Non-Border LEAF(s)
pcTag A
Ingress Policy Enforcement
Border
LEAF(s)
with EPG A
source
Non-Border LEAF(s)
ICMP
with EPG B
source
destination Filter
pcTag B
pcTag X
ICMP
source
source
destination Filter
pcTag A
pcTag B
pcTag X
pcTag X
ICMP
ICMP
pcTag A
default
from 1.2
Border
LEAF(s)
with EPG A
destination Filter
pcTag X
No effects on
EPG <-> EPG traffic
destination Filter
pcTag X
ICMP
with EPG B
source
destination Filter
pcTag B
pcTag X
BRKACI-2642
source
destination Filter
- none -
ICMP
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
84
Under VRF
L3Out Contract
Policy Control Enforcement Direction
How does it affect traffic flow and contract?
Egress Policy Enforcement
EPG -> L3Out
L3Out EPG
EPG
Otherwise
Contract is applied
on Egress LEAF
EPG
Contract is applied
on Egress LEAF
EPG <- L3Out
if remote EP exists,
Contract is applied
on Ingress LEAF
L3Out EPG
Ingress Policy Enforcement
Contract is applied
on Ingress LEAF
EPG -> L3Out
L3Out EPG
EPG
Contract is applied
on Egress LEAF
EPG
BRKACI-2642
EPG <- L3Out
L3Out EPG
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
85
L3Out Contract
CLI Verification
Points to the
correct border leaf
TEP
EPG -> L3Out
Ingress Compute LEAF Verification
for
Egress Policy Enforcement
Contract is applied on
Egress LEAF
L3Out EPG
EPG
pcTag 49162
pcTag 16391
Prefix-pcTag mapping table
Routing Table
leaf# show ip route 52.52.52.0/24 vrf TK:VRF1
52.52.52.0/24, ubest/mbest: 1/0
*via 11.0.64.64%overlay-1, [200/5], 00:00:14, bgp-65003, internal,
tag 65003
recursive next hop: 11.0.64.64/32%overlay-1
leaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
leaf# vsh -c 'show system internal policy-mgr prefix' | grep 2097152
leaf#
---- no output ----
No prefix-pcTag for a pure compute LEAF
note: before 3.2 release, use
vsh_lc –c ‘show system internal aclqos prefix’
Hardware table
(2nd gen or later LEAF –EX, -FX, -FX2 etc.)
leaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
vrf_id:
8
:::
hw_vrf_idx:
4614
check hw_vrf index
leaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| VRF |
Prefix/Len| RT| RID |
LID
| Type| PID | FPID/| HIT |N| NB-ID |
| 4614| 52.52.52.0/ 24| UC| 1a2|
117| TCAM| 1823|
0| 1823|E|
1a|
NB Hw | PID | FPID/|
12|
NA|
NA|
TBI
|TRO|Ifindex|CLSS|CLP| AI |SH|DH| Flags |
NA| NA|
0|
1| 3|
0| 0| 0|
Zoning-rule on compute
LEAF in egress mode
(not used in this case)
Check contract on LEAF (zoning-rule)
leaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
Rule ID
SrcEPG
DstEPG
FilterID
4142
49162
16391
5
pcTag (CLSS) is 1 to
bypass contract
operSt
enabled
Scope
2097152
Action
permit© 2020
BRKACI-2642
Priority
fully_qual(7)
Cisco and/or its affiliates. All rights reserved.
Cisco Public
86
L3Out Contract
EPG -> L3Out
CLI Verification
Points to the
correct external
next-hop IP
L3Out EPG
EPG
Egress border LEAF Verification for
Egress Policy Enforcement
Contract is
applied on Egress
LEAF
pcTag 49162
pcTag 16391
Prefix-pcTag mapping table
Routing Table
Bleaf# show ip route 52.52.52.0/24 vrf TK:VRF1
52.52.52.0/24, ubest/mbest: 1/0
*via 15.2.2.1, vlan37, [110/5], 00:25:42, ospf-default, intra
Bleaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
Bleaf# vsh -c 'show system internal policy-mgr prefix' | grep 52.52.52
Vrf-Vni VRF-Id ~snip~ VRF-Name
Addr
Class Shared Remote
Complete
2097152 8
~snip~ TK:VRF1
52.52.52.0/24
16391
False False False
prefix-pcTag mapping for L3Out EPG
Hardware table
(2nd gen or later LEAF –EX, -FX, -FX2 etc.)
note: before 3.2 release, use
vsh_lc –c ‘show system internal aclqos prefix’
Bleaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
vrf_id:
8
:::
hw_vrf_idx:
4614
check hw_vrf index
Bleaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| VRF |
Prefix/Len| RT| RID |
LID
| Type| PID | FPID/| HIT |N| NB-ID |
| 4614| 52.52.52.0/ 24| UC| 121|
68| TCAM| 80f|
0| 80f|A|
7afd|
NB Hw | PID | FPID/|
80df|
NA|
NA|
Check contract on LEAF (zoning-rule)
Bleaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
Rule ID
SrcEPG
DstEPG
FilterID
4142
49162
16391
5
pcTag (CLSS) is
16391 (0x4007)
TBI
|TRO|Ifindex|CLSS|CLP| AI |SH|DH| Flags |
NA| NA|
0|4007| 2|
0| 1| 0|spi,dpi
Zoning-rule on border
LEAF in egress mode
operSt
enabled
Scope
2097152
Action
permit© 2020
BRKACI-2642
Priority
fully_qual(7)
Cisco and/or its affiliates. All rights reserved.
Cisco Public
87
L3Out Contract
Contract is applied on
Ingress LEAF
CLI Verification
Points to the
correct border leaf
TEP
pcTag 49162
pcTag 16391
Prefix-pcTag mapping table
Routing Table
leaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
leaf# vsh -c 'show system internal policy-mgr prefix' | grep 52.52.52
Vrf-Vni VRF-Id ~snip~ VRF-Name
Addr
Class Shared Remote
Complete
2097152 8
~snip~ TK:VRF1
52.52.52.0/24
16391
False True False
prefix-pcTag mapping for L3Out EPG
Hardware table
gen or later LEAF –EX, -FX, -FX2 etc.)
note: before 3.2 release, use
vsh_lc –c ‘show system internal aclqos prefix’
leaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
vrf_id:
8
:::
hw_vrf_idx:
4614
check hw_vrf index
leaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| VRF |
Prefix/Len| RT| RID |
LID
| Type| PID | FPID/| HIT |N| NB-ID |
| 4614| 52.52.52.0/ 24| UC| 1a2|
117| TCAM| 1823|
0| 1823|A|
7c82|
NB Hw | PID | FPID/|
830a|
NA|
NA|
Check contract on LEAF (zoning-rule)
leaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
Rule ID
SrcEPG
DstEPG
FilterID
4142
49162
16391
5
L3Out EPG
EPG
Ingress Compute LEAF Verification
for
Ingress Policy Enforcement
leaf# show ip route 52.52.52.0/24 vrf TK:VRF1
52.52.52.0/24, ubest/mbest: 1/0
*via 11.0.64.64%overlay-1, [200/5], 00:00:14, bgp-65003, internal,
tag 65003
recursive next hop: 11.0.64.64/32%overlay-1
(2nd
EPG -> L3Out
pcTag (CLSS) is
16391 (0x4007)
TBI
|TRO|Ifindex|CLSS|CLP| AI |SH|DH| Flags |
NA| NA|
0|4007| 2|
0| 0| 0|spi,dpi
Zoning-rule on compute
LEAF in egress mode
operSt
enabled
Scope
2097152
Action
permit© 2020
BRKACI-2642
Priority
fully_qual(7)
Cisco and/or its affiliates. All rights reserved.
Cisco Public
88
L3Out Contract
EPG -> L3Out
CLI Verification
Points to the
correct external
next-hop IP
L3Out EPG
EPG
Egress border LEAF Verification for
Ingress Policy Enforcement
Contract is applied on
Egress LEAF
pcTag 49162
pcTag 16391
Prefix-pcTag mapping table
Routing Table
Bleaf# show ip route 52.52.52.0/24 vrf TK:VRF1
52.52.52.0/24, ubest/mbest: 1/0
*via 15.2.2.1, vlan37, [110/5], 00:25:42, ospf-default, intra
Bleaf# show vrf TK:VRF1 detail extended | grep vxlan
Encap: vxlan-2097152
Bleaf# vsh -c 'show system internal policy-mgr prefix' | grep 52.52.52
Vrf-Vni VRF-Id ~snip~ VRF-Name
Addr
Class Shared Remote
Complete
2097152 8
~snip~ TK:VRF1
52.52.52.0/24
16391
False False False
prefix-pcTag mapping for L3Out EPG
Hardware table
(2nd gen or later LEAF –EX, -FX, -FX2 etc.)
note: before 3.2 release, use
vsh_lc –c ‘show system internal aclqos prefix’
Bleaf# vsh_lc -c 'show system internal eltmc info vrf TK:VRF1' | grep hw_vrf_idx
vrf_id:
8
:::
hw_vrf_idx:
4614
check hw_vrf index
Bleaf# vsh_lc -c 'show platform internal hal l3 routes’ | egrep ‘VRF|52.52.52’
| VRF |
Prefix/Len| RT| RID |
LID
| Type| PID | FPID/| HIT |N| NB-ID |
| 4614| 52.52.52.0/ 24| UC| 121|
68| TCAM| 80f|
0| 80f|A|
7afd|
NB Hw | PID | FPID/|
80df|
NA|
NA|
pcTag (CLSS) is
16391 (0x4007)
Check contract on LEAF (zoning-rule)
Bleaf# show zoning-rule scope 2097152 | egrep 'Rule|16391'
Rule ID
SrcEPG
DstEPG
FilterID
--- none ---
TBI
|TRO|Ifindex|CLSS|CLP| AI |SH|DH| Flags |
NA| NA|
0|4007| 2|
0| 1| 0|spi,dpi
No zoning-rule on border
LEAF in ingress mode
operSt
Scope
Action
BRKACI-2642
Priority
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
89
MP-BGP Deep Dive
MP-BGP
MP-BGP is automatically deployed once Route
Reflector (and MP-BGP AS) is configured
VRF overlay-1
MP-BGP
MP-BGP table (vpnv4)
Non-BLEAF
User
VRF
Route Import
BGP table (IPv4)
Route Export
Border LEAF
User
VRF
Route Import
Route Export
BGP table (IPv4)
Redistribute
permit-all
RIB
L3Out
export
Redistribute
Protocol Database
Route-maps
export
permit-all
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
91
MP-BGP with L3Out BGP
VRF overlay-1
MP-BGP is automatically deployed once Route
Reflector (and MP-BGP AS) is configured
MP-BGP
MP-BGP table (vpnv4)
Non-BLEAF
User
VRF
Route Import
BGP table (IPv4)
Route Export
Border LEAF
User
VRF
Route Import
Route Export
BGP table (IPv4)
Route-maps
RIB
L3Out
(BGP)
export
export
permit-all
permit-all
External Router
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
92
CLI Verifications
1. BGP process in your VRF with expected Redistribution and Route-Target
Automatically created regardless of routing
protocol used in L3Out. If not, check Route
Reflector policy on APIC
border-leaf# show bgp process vrf TK:VRF1
VRF RD
: 10.0.184.64:2
Information for address family IPv4 Unicast in VRF TK:VRF1
Redistribution
direct, route-map permit-all
static, route-map imp-ctx-bgp-st-interleak-2097152
eigrp, route-map permit-all
ospf, route-map permit-all
Export RT list:
65003:2097152
Import RT list:
65003:2097152
Information for address family IPv6 Unicast in VRF TK:VRF1
--- snip ---
•
VRF RD (Route Distinguisher) is based on TEP IP
•
BGP redistributes (almost) all external routes to
export them into MP-BGP vpnv4 by default.
Check a later page for the exception on BD subnets
(direct routes).
RT (Route Target) is based on ACI BGP AS and
VRF VNID.
•
2. External routes are redistributed/exported into VPNv4 in VRF overlay-1
border-leaf# show bgp vpnv4 unicast vrf overlay-1
Network
Next Hop
Route Distinguisher: 10.0.184.64:2
* i5.5.5.0/24
10.0.184.67
*>r
0.0.0.0
* i15.0.0.0/24
10.0.184.67
*>r
0.0.0.0
Metric
(VRF TK:VRF1)
5
5
0
0
LocPrf
100
100
100
100
MP-BGP VPNv4 table can be checked via
normal CLI in vrf overlay-1
Weight Path
0
32768
0
32768
NOTE:
This example shows two routes are learned
locally (r with next-hop 0.0.0.0) and also from
another leaf with TEP 10.0.184.67.
?
?
?
?
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
93
CLI Verifications
3. MP-BGP on all leaves should have all external routes in VPNv4 format in VRF overlay-1
non-border-leaf# show bgp vpnv4 unicast vrf overlay-1
Route Distinguisher: 10.0.184.64:2
*>i5.5.5.0/24
10.0.184.64
* i
10.0.184.64
Route Distinguisher: 10.0.184.67:1
*>i5.5.5.0/24
10.0.184.67
* i
10.0.184.67
5
5
5
5
100
100
100
100
5.5.5.0/24 is advertised from border-leaf1
(10.0.184.64) and border-leaf2
(10.0.184.67)
0 ?
0 ?
Two entries with the same next-hop LEAF TEP
means there are two Route Reflectors.
0 ?
0 ?
non-border-leaf# show bgp vpnv4 unicast 5.5.5.0/24 vrf overlay-1
Route Distinguisher: 10.0.184.67:1
BGP routing table entry for 5.5.5.0/24, version 598 dest ptr 0xaa7e840c
Each VPNv4 route in VRF overlay-1 has RT
with its original VRF VNID
AS-Path: NONE, path sourced internal to AS
10.0.184.67 (metric 3) from 10.0.184.65 (1.1.1.101)
Extcommunity:
RT:65003:2097152
4. BGP process is running also in your VRF on non-border-leaf for MP-BGP
non-border-leaf# show bgp process vrf TK:VRF1
Information for address family IPv4 Unicast in VRF TK:VRF1
Export RT list:
65003:2097152
Import RT list:
65003:2097152
Information for address family IPv6 Unicast in VRF TK:VRF1
--- snip ---
IPv4 BGP imports all the external routes for
its own VRF based on RT from VPNv4 table
If BGP is not running in your VRF on non-borderleaf, check Route Reflector Policy config on APIC
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
94
CLI Verifications
5. The external routes are imported in IPv4 BGP table based on RT
non-border-leaf# show bgp ipv4 unicast vrf TK:VRF1
Network
*>i5.5.5.0/24
*|i
Next Hop
10.0.184.64
10.0.184.67
Metric
5
5
LocPrf
100
100
Weight Path
0 ?
0 ?
6. The routing table shows border leaves as next-hop learned from iBGP
non-border-leaf# show ip route 5.5.5.0/24 vrf TK:VRF1
5.5.5.0/24, ubest/mbest: 2/0
*via 10.0.184.67%overlay-1, [200/5], 2d10h, bgp-65003, internal, tag 65003
recursive next hop: 10.0.184.67/32%overlay-1
*via 10.0.184.64%overlay-1, [200/5], 2d10h, bgp-65003, internal, tag 65003
recursive next hop: 10.0.184.64/32%overlay-1
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
95
CLI Verifications for BD subnet exception
border-leaf# show bgp vpnv4 unicast neighbors vrf overlay-1
BGP neighbor is 10.0.184.65,
remote AS 65003, ibgp link,
Peer index 1
For address family: VPNv4 Unicast
Outbound route-map configured is deny-pervasive, handle obtained
BGP neighbor is 10.0.184.66,
remote AS 65003, ibgp link,
BD subnets and Null0 I/F should not be
distributed via MP-BGP.
➢ Outbound route-map to BGP Route
Reflector (spines) limits BD subnets
(pervasive routes) and Null0.
Peer index 2
For address family: VPNv4 Unicast
Outbound route-map configured is deny-pervasive, handle obtained
border-leaf# show route-map deny-pervasive
route-map deny-pervasive, deny, sequence 1
Match clauses:
pervasive: 2
Set clauses:
route-map deny-pervasive, deny, sequence 2
Match clauses:
interface: Null0
Set clauses:
route-map deny-pervasive, permit, sequence 3
Match clauses:
Set clauses:
In this example, 10.0.184.65 and 10.0.184.66
are RR spines.
BD subnets are deployed based on object
policies from APIC instead of routing protocol
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
96
Reference
ACI Fabric L3Out Guide https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/guide-c07-743150.html
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
97
Complete your
online session
survey
•
Please complete your session survey
after each session. Your feedback
is very important.
•
Complete a minimum of 4 session
surveys and the Overall Conference
survey (starting on Thursday) to
receive your Cisco Live t-shirt.
•
All surveys can be taken in the Cisco Events
Mobile App or by logging in to the Content
Catalog on ciscolive.com/emea.
Cisco Live sessions will be available for viewing on
demand after the event at ciscolive.com.
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
98
Continue your education
Demos in the
Cisco campus
Walk-in labs
Meet the engineer
1:1 meetings
Related sessions
BRKACI-2642
© 2020 Cisco and/or its affiliates. All rights reserved. Cisco Public
99
Thank you