#CiscoLive
ACI Troubleshooting:
A Deep Dive into PBR
Michael Garcia, ACI Technical Leader, Customer Experience
BRKDCN-3615
#CiscoLive
Cisco Webex App
Questions?
Use Cisco Webex App to chat
with the speaker after the session
How
1
Find this session in the Cisco Live Mobile App
2
Click “Join the Discussion”
3
Install the Webex App or go directly to the Webex space
4
Enter messages/questions in the Webex space
Enter your personal notes here
Webex spaces will be moderated
by the speaker until June 9, 2023.
https://ciscolive.ciscoevents.com/ciscolivebot/#BRK3615
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
3
Agenda
•
Why do we need PBR?
•
How PBR Works
•
PBR Validation Tools
•
PBR Packet Walk w/ CLI
Verification
•
PBR Troubleshooting
Workflow
•
Common PBR Issues
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
4
Glossary of Acronyms
Acronyms
Definitions
ACI
Application Centric Infrastructure
APIC
Application Policy Infrastructure Controller
EP
Endpoint
EPG
Endpoint Group
BD
Bridge Domain
VRF
Virtual Routing and Forwarding
COOP
Council of Oracle Protocol
PBR
Policy-Based Redirect
VxLAN
Virtual eXtensible LAN
pcTag
Policy Class Tag
VxLAN packet acronyms
#CiscoLive
Acronyms
Definitions
dXXXo
Outer Destination XXX
(dIPo = Outer Destination IP)
sXXXo
Outer Source XXX
(sIPo = Outer Source IP)
dXXXi
Inner Destination XXX
(dIPi = Inner Destination IP)
sXXXi
Inner Source XXX
(sIPi = Inner Source IP)
VNID
Virtual Network Identifier
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
5
Why do we need
PBR?
Service Insertion without PBR
•
Requires extended ACLs and route-maps
•
More complex route configuration to redirect traffic to desired next-hop
Leaf-1
Leaf-2
Contract to allow APP EPG->FW
Leaf-3
Contract to allow WEB EPG->FW
APP EPG
WEB EPG
Customer has to
manually configure
contracts
#CiscoLive
BRKDCN-3615
Needs Extended
ACLs and possible
VRF leaking
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
7
Why do we service graphs with PBR in ACI?
•
Service graphs automate contract configuration
•
PBR gives us the ability to attach forwarding constructs to contracts
APP EPG
WEB EPG
Service graph tied to
subject of contract
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
8
How PBR Works
How PBR works
Service Graph Templates
defines HOW traffic should
flow
1
Contract selects
traffic to redirect
Leaf-1
Leaf-3
EP1
EP2
APP
EPG
EP2
WEB
EP2
EPG
Leaf-2
The Device tells us how many
interfaces and logical
connectors on the Service
Devices
The Device Selection Policy
defines how the Device will
communicate with the fabric
Shadow
EPG
Redirect policy defines
parameters about
service device we are
redirecting traffic
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
10
PBR Components (Review)
•
•
Devices
•
Physical Device & interfaces it connects to in fabric. Converted to Consumer Connector and
Provider Connector
•
The Device tells us how many interfaces and logical connectors on the Service Devices
Service Graph Template
•
•
•
•
Define the flow of traffic
Contract
•
Places Contract between Consumer & Provider and the shadow EPG
•
Selects traffic to redirect
Device Selection Policy
•
Ties the physical device to a Graph template and contract
•
Defines how the Device will communicate with the fabric
Redirect Policy
•
Defines information about the service node that is used for redirection
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
11
Where is policy enforced?
Scenario
VRF enforcement
mode
Consumer
Provider
Intra-VRF
Ingress/egress
EPG
EPG
Policy enforced on
If destination endpoint is learned: ingress leaf *
If destination endpoint is not learned: egress leaf
Ingress
EPG
L3Out EPG
Ingress
L3Out EPG
EPG
Egress
EPG
L3Out EPG
Egress
L3Out EPG
EPG
Consumer leaf (non-border leaf)
Provider leaf (non-border leaf)
Border leaf -> non-border leaf traffic
If destination endpoint is learned: border leaf
If destination endpoint is not learned: non-border leaf
Non-border leaf-> border leaf traffic : Border leaf
Inter-VRF
Ingress/egress
L3Out EPG
L3Out EPG
Ingress leaf*
Ingress/egress
EPG
EPG
Ingress/egress
EPG
L3Out EPG
Ingress/egress
L3Out EPG
EPG
Ingress leaf*
Ingress/egress
L3Out EPG
L3Out EPG
Ingress leaf*
Consumer leaf
Consumer leaf (non-border leaf)
*Policy
#CiscoLive
enforcement is applied on the first leaf hit by the packet.
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
12
What are shadow EPGs?
Leaf-1
A ‘two armed’ example
• Shadow EPGs connect to the service
Device
• External Interface is called the
“Consumer Connector”
• Internal interface is the “Provider
Connector”
• Each is represented by a VLAN and
has its own PcTag
• Kind of a way of stitching
EPG VLAN and service
Cons
node VLAN to ”steer”
traffic to service node
External and Internal
Interfaces
Leaf-2
EP2
EP1
EPG
APP
EPG
Web
Shadow
EPG
Provider Connector
Prov
Consumer Connector
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
13
Leaf-2
Leaf-1
Shadow EPGs & contracts
•
•
•
•
Leaf-3
All in same VRF
EPG App to EPG Web (Redirect)
Consumer Connection to App (uni-dir filter)
EPG Web to EPG App (Redirect)
Provider Connection to Web (uni-dir default)
EP1
EPG
APP
32771
EP2
Shadow
EPG
16389
EPG
Web
16386
a1-leaf1# show zoning-rule scope 2293762
| Rule ID | SrcEPG | DstEPG | FilterID |
Dir
| operSt | Scope |
Action
|
+---------+--------+--------+----------+----------------+---------+---------+-----------------+
|
4212 | 32771 | 16386 |
1
|
bi-dir
| enabled | 2293762 | redir(destgrp-3)|
|
4098 | 16389 | 32771 |
1
|
uni-dir
| enabled | 2293762 |
permit
|
|
4199 | 16386 | 32771 |
1
| uni-dir-ignore | enabled | 2293762 | redir(destgrp-3)|
+
4152 | 16389 | 16386 | default |
uni-dir
| enabled | 2293762 |
permit
+
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
14
How PBR
Forwarding
Works
4. Spine does
COOP lookup for
Service MAC
Spine
Spine
8. Normal EP
forwarding (Transit
or Proxy)
3. Traffic is sent
to MAC proxy
DST IP Is
LEARNED on
Leaf-1
2. *PBR POLICY APPLIED
HERE*
SCLASS: APP
DCLASS: WEB
SRC IP: 192.168.1.100
SMAC: APP
DST IP: 192.168.2.100
DMAC: FW-MAC
SEGMENT ID (BDVNID):
SVC_BD
1. EP1 Initiates Traffic to
EP2
SRC IP: 192.168.1.100
SMAC: APP MAC
DST IP: 192.168.2.100
DMAC: LEAF MAC
BD1
192.168.1.1
Leaf-2
FW
IP: 192.168.1.100
GW: 192.168.1.1
MAC: APP
Location
192.168.1.100
Leaf-1
192.168.2.200
Leaf-3
7. POLICY CHECK
SCLASS:
SVC_PROV
DCLASS: WEB
5. Traffic arrives on
service leaf
SCLASS: APP
DCLASS: SVC_CONS
Leaf-1
Endpoint
Leaf-3
SVC_BD
172.16.1.1
6. Traffic from
service node
BD2
192.168.2.1
9. Traffic goes to
provider (WEB EPG)
SRC IP:
192.168.1.100
SMAC: FW MAC
DST IP:
192.168.2.100
DMAC: LEAF MAC
IP: 172.16.1.100
VRF1 (ALL BDs in Same VRF)
EPG-APP
SRC IP: 192.168.1.100
SMAC: LEAF MAC
DST IP: 192.168.2.100
DMAC: WEB
IP: 192.168.2.100
GW: 192.168.2.1
MAC: WEB
EPG-WEB
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
16
8. Spines already has
APP EP location in COOP
and sends packet directly
to Leaf-1
* Return Traffic *
EPG WEB --> EPG
APP
3. Traffic is sent to
4. Spine does COOP
lookup for Service
MAC
Spine
Spine
MAC proxy
9. Traffic returns to
consumer (APP EPG)
SRC IP: 192.168.2.100
SMAC: LEAF MAC
DST IP: 192.168.1.100
DMAC: APP
5. Traffic arrives on
service leaf
SCLASS: WEB
DCLASS: SVC_PROV
7. POLICY CHECK
SCLASS:
SVC_CONS
DCLASS: APP
Leaf-1
Leaf-2
BD1
192.168.1.1
Leaf-3
SVC_BD
172.16.1.1
2. *PBR POLICY APPLIED
HERE*
SCLASS: WEB
DCLASS: APP
SRC IP: 192.168.2.100
SMAC: WEB
DST IP: 192.168.1.100
DMAC: FW-MAC
SEGMENT ID (BDVNID): SVC_BD
BD2
192.168.2.1
6. Traffic from service
node
SRC IP: 192.168.2.100
SMAC: FW MAC
DST IP: 192.168.1.100
DMAC: LEAF MAC
IP: 192.168.1.100
GW: 192.168.1.1
MAC: APP
FW
IP: 172.16.1.100
VRF1 (ALL BDs in Same VRF)
EPG-APP
IP: 192.168.2.100
GW: 192.168.2.1
MAC: WEB
EPG-WEB
#CiscoLive
BRKDCN-3615
1. EP2 sends traffic back to EP1
SRC IP: 192.168.2.100
SMAC: APP MAC
DST IP: 192.168.1.100
DMAC: LEAF MAC
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
17
So what about PBR multi-location deployments?
Please refer to BRKDCN-3982:
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
18
PBR Validation
Tools
Available since 3.2(2)
Contract Parser Script
Validates contract rules on the Leaf
Leaf# contract_parser.py –-help
--nz, --nonzero
display only entries with non-zero hits
--incremented
display only entries that have incremented since last checked
--node NODES [NODES ...]
display entries specific to one or more leaf nodes
--contract CONTRACT [CONTRACT ...]
display only rules that match a specific contract. The
name of the contract is in the form
uni/tn-<tenant>/brc-<contract>
--vrf VRF [VRF ...]
display entries for a specific vrf. The integer vnid
of the vrf can be provided or the vrf name in the form
<tenant>:<vrf>
--epg EPG [EPG ...]
display entires for specific EPG. The integer pcTag or
DN name can be provided. Note the dn is a partial dn
in the form
tn-<tenant>/ap-<applicationProfile>/epg-<epg>
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
20
Contract Parser Script Example
a1-leaf1# contract_parser.py --vrf mg-cisco-live:v1 --sepg tn-mg-cisco-live/ap-a1/epg-e1 -depg tn-mg-cisco-live/ap-a1/epg-e2
Rule ID
VRF
[7:4249] [vrf:mg-cisco-live:v1]
redir ip
Action+Filter
tn-mg-cisco-live/ap-a1/epg-e1(32771)
SRC EPG
tn-mg-cisco-live/ap-a1/epg-e2(16386)
DST EPG
[contract:uni/tn-mg-cisco-live/brc-ip]
CONTRACT
[hit=130028]
Service MAC and
Service BD (in
VXLAN header)
Contract Hit Count
destgrp-3 vrf:mg-cisco-live:v1 ip:172.16.1.100 mac:00:50:56:A8:48:97 bd:vxlan-16744311
For PBR gives
redirect group info
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
21
ELAM – Embedded
Logic Analyzer Module
•
It is a tripwire in hardware
•
The first frame to match a
specified condition ‘trips’ it
•
Dst – TCP
192.168.2.100:3000
Dst – TCP
192.168.2.100:3001
Dst – TCP
192.168.2.100:3002
vsh_lc
debug platform internal
trigger reset
trigger init in-select
set outer ipv4 dst_ip
set outer l4 dst-port
start
Report is created with vast
amount of data regarding
asic decisions
Frame was not
dropped in lookups!
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
tah elam asic 0
6 out-select 1
192.168.2.100
3001
Matching frame was
caught!
module-1(DBG-elam-insel6)# ereport | grep "drop reason"
RW drop reason
: no drop
LU drop reason
: no drop
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
22
What ASIC should be
set in the ELAM?
Model
vsh_lc
debug platform internal <asic> elam asic 0
Role
Asic for Elam
N9K-C*C
Fixed Spine
roc
N9K-C*GX
Fixed Spine
app
N9K-C*-EX
Leaf
tah
N9K-C*-FX/FXP/FX2
Leaf
roc
N9K-C*-GX
Leaf
app
N9K-C*-GX2
Leaf
cho
N9K-X97*-EX
Spine LC
tah
N9K-X97*-FX
Spine LC
roc
N9K-X97*-GX
Spine LC
app
N9K-C95*-FM-E
Spine FM
tah
N9K-C950*-FM-E2
Spine FM
roc
N9K-C95*-FM-G
Spine FM
app
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
23
Steps to Using Elam on Gen2+ Leaf or Fixed Spine
Elams are run from
the line card shell
Failing to reset the trigger
can cause past elam
configurations to take effect.
Always reset the trigger!
Refer to “What ASIC should
be set in the ELAM” slide
vsh_lc
debug platform internal
trigger reset
trigger init in-select
set outer ipv4 dst_ip
set outer l4 dst-port
start
Leafs and fixed spines are single
asic switches. Always use asic 0
tah elam asic 0
6 out-select 0
192.168.2.100
3001
Use 0 or 1
module-1(DBG-elam)# trigger init in-select ?
!ommitted
14 Outer(l2(vntag)|l3|l4)-inner(l2|l3|l4)-ieth
6
Outerl2-outerl3-outerl4
Determines which headers conditions
7
Innerl2-innerl3-innerl4
can be matched in. Use 14 or 7 when
!ommitted
matching vxlan encapsulated headers.
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
24
Steps to Using Elam on Gen2+ Leaf or Fixed Spine
Use “set outer” or
“set inner” depending
on in-select and if
matching outer or inner
headers in vxlan packet
Which headers to match
conditions for?
vsh_lc
debug platform internal
trigger reset
trigger init in-select
set outer ipv4 dst_ip
set outer l4 dst-port
start
tah elam asic 0
6 out-select 0
192.168.2.100
3001
Finally enable the elam!
What to match in the
header?
When running stat if Triggered is seen, this
means a matching packet was received
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
25
ELAM on Leaf CLI - PBR Verification
ereport available since 4.2
vsh_lc
debug platform internal tah elam asic 0
trigger reset
trigger init in-select 6 out-select 1
set outer ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
start
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
module-1(DBG-elam-insel6)# ereport | egrep "Source IP|Destination IP|Contract Applied|service_redir"
Destination IP
: 192.168.2.100
Source IP
: 192.168.1.100
Contract Applied
: yes
sug_luc_latch_results_vec.luc3_0.service_redir: 0x1
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
26
ELAM Assistant APP - PBR Verification
Set Parameters
Triggered!
and
Report is Ready.
Click to see report.
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
27
FTRIAGE - PBR Verification
End-to-End ELAMs run from the APIC:
a-apic1# ftriage help
usage: ftriage [-h] [-outdir OUTDIR] [-loglevel loglevel] [-user username] [-nsdropaction
nsdropaction] [-xt xt] [-wait WAIT] {bridge,example,route,infraping} ...
a-apic1# ftriage example
*snippet*
Examples:
# Bridge
> ftriage bridge -ii leaf1:Eth1/1 -ie 100 -ei leaf2:Eth1/2 -ee 101 -dmac 02:02:02:02:02:02
# Route
> ftriage route -ii leaf1:Eth1/1 -ie 100 -ei leaf2:Eth1/2 -ee 101 -dip 192.168.1.101
*snippet*
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
28
Before Service Device
FTRIAGE Example for PBR Flow
a-apic1# ftriage -user admin route -ii a1-leaf1:Eth1/44 -sip 192.168.1.100 -dip 192.168.2.100
2023-04-23 19:28:20,133 INFO
ftriage:
main:1295 L3 packet Seen on a1-leaf1 Ingress: Eth1/44
Egress: Eth1/52 Vnid: 16089032
2023-04-23 19:28:20,133 INFO
ftriage:
main:1337 a1-leaf1: Incoming Packet captured with
[SIP:192.168.1.100, DIP:192.168.2.100]
2023-04-23 19:29:13,080 INFO
ftriage: unicast:1543 a1-leaf1: traffic is redirected to vnid:16744311
mac:00:50:56:A8:48:97 via tenant:mg-cisco-live graph:pbr-one-arm-SG-template contract: ip
2023-04-23 19:34:33,920 INFO
ftriage: acigraph:743
found matching devicenode:N1 ldev:pbr-one-armCL2023 dev:asavasavuni/tn-mg-cisco-live/lDevVip-pbr-one-arm-CL2023/cDev-asav/cIf-[asav]
2023-04-23 19:34:33,921 INFO
ftriage: unicast:2755 a1-leaf2: PBR first pass is done and traffic is
sent to service device: node:N1 ldev:pbr-one-arm-CL2023 dev:asav
2023-04-23 19:34:33,921 INFO
ftriage: unicast:2757 a1-leaf2: expected traffic to return from:
topology/pod-1/paths-102/pathep-[eth1/43] encap:unknown
Coming back from Service Device
2023-04-23 19:34:53,165 INFO
ftriage:
main:1821 pbr return path, nxt_nifs {a1-leaf2:
['Eth1/43']}, nxt_dbg_f_n ig, nxt_inst ig, eg_ifs Eth1/43, Vnid: unknown
2023-04-23 19:35:06,803 INFO
ftriage:
fcls:2379 a1-leaf2: Valid ELAM for asic:0 slice:0
srcid:114 pktid:170
2023-04-23 19:35:07,823 INFO
ftriage:
main:1295 L3 packet Seen on a1-leaf2 Ingress: Eth1/43
Egress: Eth1/59 Vnid: 2293762
2023-04-23 19:35:07,824 INFO
ftriage:
main:1337 a1-leaf2: Incoming Packet captured with
[SIP:192.168.1.100, DIP:192.168.2.100]
29
BRKDCN-3615
BRKDCN-3915
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
#CiscoLive
PBR Packet Walk
with CLI Verification
PBR Packet Walk – Step 2 (Packet Rewrite)
rewrite
L4/Payload
L4/Payload
Proto
DIP
BD VNID VXLAN
SIP
00
802.1Q SMAC
DIP
SIP
FW MAC
Spine-2
Spine-1
After rewrite,
sends to MAC
Proxy on spine
802.1Q SMAC DMAC
2
Service BD
Leaf-1
Leaf-2
Leaf-3
Leaf-1 does policy lookup
and redirects packet to
Service BD/Service MAC
EP1
1
EP1 sends packet to EP2 via Leaf-1
#CiscoLive
EP2
192.168.1.100
BRKDCN-3615
172.16.1.100
192.168.2.100
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
31
2
Command Line
Verification
Confirm pcTags of traffic flow
Dest EP is
known by leaf-1
a1-leaf1# show system internal epm endpoint ip
192.168.1.100 | egrep "VRF vnid|sclass"
BD vnid : 16089032 ::: VRF vnid : 2293762
sclass for
Flags : 0x80004c04 ::: sclass : 32771
source EP
If destination EP is known :
redirect happens on ingress leaf
-----------------------------If destination EP is unknown:
redirect will happen on egress leaf
2
1
Leaf-1
EP1
a1-leaf1# show system internal epm endpoint ip
192.168.2.100 | egrep "VRF vnid|sclass"
BD vnid : 15826920 ::: VRF vnid : 2293762
dclass for
Flags : 0x80004c04 ::: sclass : 16386
dest EP
Verify zoning-rule has ‘redir’ action and matches desired traffic type (ex. ip traffic)
a1-leaf1# show zoning-rule scope 2293762 src-epg 32771 dst-epg 16386
+---------+--------+--------+----------+--------+---------+-----------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir
| Scope
| Action
|
----+--------+---------+---------+------+-----------------------------------+
|
4308 | 32771 | 16386 |
1
| bi-dir | 2293762 | redir(destgrp-3)|
Redirect happening
on ingress leaf since
destination is known
a1-leaf1# show zoning-filter filter 1
+----------+------+--------+-------------+-------------+-------------+-------------+
| FilterId | Name | EtherT | SFromPort |
SToPort
| DFromPort |
DToPort
|
+----------+------+--------+-------------+-------------+-------------+-------------+
|
1
| 1_0 |
ip
| unspecified | unspecified | unspecified | unspecified |
#CiscoLive
BRKDCN-3615
IP Filter
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
32
2
Command Line
Verification
2
1
Leaf-1
EP1
Check redirect policy to see how packets will be redirected
Service node
redirect IP
a1-leaf1# show service redir info group 3
==============================================================
GrpID
Name
destination
3
destgrp-3
dest-[172.16.1.100]-[vxlan-2293762]
Service node
VRF VNID
a1-leaf1# show service redir info destination ip 172.16.1.100 vnid 2293762
=======================================================================================================
Parameters used to build vrf
Name
bdVnid
vMac
redirected vxlan packet
====
======
====
====
dest-[172.16.1.100]-[vxlan-2293762]
vxlan-16744311 00:50:56:A8:48:97
mg-cisco-live:v1
0x1 – yes, redirected
0x0 – not redirected
module-1(DBG-elam-insel6)# report detail | grep service_redir
sug_luc_latch_results_vec.luc3_0.service_redir: 0x1
#CiscoLive
Packet is being
redirected
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
33
PBR Packet Walk – Step 3 (Spine COOP MAC
Lookup)
Spine MAC
COOP Lookup
for Service EP
Spine-1
Spine-2
3
Leaf-1
2
1
Leaf-3
Leaf-2
EP2
EP1
192.168.1.100
EP2
172.16.1.100
#CiscoLive
192.168.2.100
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
34
3
Command Line
Verification
Spine-1
3
Verify Spine has installed MAC of service node in COOP
Leaf-1
2
a1-spine1# show coop internal info repo ep key 16744311 00:50:56:A8:48:97 | egrep
"Tunnel|EP" | head -n 3
EP bd vnid : 16744311
Service Device
Service BD VNID
EP mac : 00:50:56:A8:48:97
MAC
Tunnel nh : 10.0.216.68
1
EP1
Map tunnel destination address to leaf
apic1# moquery -c ipv4Addr -f 'ipv4.Addr.addr=="10.0.216.68"' | grep dn
dn
: topology/pod-1/node-102/sys/ipv4/inst/dom-overlay-1/if-[lo0]/addr-[10.0.216.68/32]
Tunnel points to Leaf 102 where
service device is connected
Rewrite info!
Service BD VNID
ELAM ingress on Spine
sees Destination TEP
as MAC Proxy
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
35
PBR Packet Walk – Step 4 (Service Leaf)
Spine-1
Spine-2
3
Traffic is sent to Service Leaf
(Leaf-2) & Leaf-2 sends
traffic to Service Device
4
2
1
Leaf-1
Leaf-2
EP2
EP1
192.168.1.100
EP2
Leaf-3
172.16.1.100
#CiscoLive
192.168.2.100
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
36
4
Spine-1
Command Line
Verification
Spine-2
3
4
Verify Service Device programming on Service Node (Leaf-2)
a1-leaf2# show system internal epm endpoint mac 0050.56a8.4897
Checking Service
MAC Learning
2
EP1
1
Leaf-1
Leaf-2
EP1 EP2
MAC : 0050.56a8.4897 ::: Num IPs : 1
IP# 0 : 172.16.1.100 ::: IP# 0 flags : host-tracked
Vlan id : 57 ::: Vlan vnid : 10792 ::: VRF name : mg-cisco-live:v1
BD vnid : 16744311 ::: VRF vnid : 2293762
Shadow EPG
Phy If : 0x1a02a000 ::: Tunnel If : 0
pcTAG
Interface : Ethernet1/43
Flags : 0x80004c04 ::: sclass : 49155 ::: Ref count : 5
EP Create Timestamp : 03/28/2023 15:23:44.027077
EP Update Timestamp : 04/14/2023 13:52:41.683129
EP Flags : local|IP|MAC|sclass|timer|
a1-leaf2# show vlan id 57 extended
VLAN Name
Encap
Ports
FD_VLAN
---- -------------------------------- ---------------- --------------------57
mg-cisco-live:pbr-one-armvlan-1417
Eth1/43
CL2023ctxv1:provider:
Access Encap VLAN
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
37
PBR Packet Walk – Step 5 (Return from FW)
Spine-1
Spine-2
3
4
2
Leaf-1
policy lookup is
made (implicit
permit)
1
Leaf-3
Leaf-2
Service Device sends traffic
back to router MAC (19:FF).
Destination IP is EP2 and policy
lookup is made
5
EP2
EP1
EP2
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
38
5
Command Line Verification
Spine-1
3
Traffic is sent to 1-ARM FW device. After inspection, traffic
comes back to Leaf-2 via Service EPG VLAN
4
2
a1-leaf2# show system internal epm endpoint mac 0050.56a8.4897
| egrep "VRF vnid|sclass"
BD vnid : 16744311 ::: VRF vnid : 2293762
pcTAG of service EPG
Flags : 0x80004c04 ::: sclass : 16389
a1-leaf1# show system internal epm endpoint ip 192.168.2.100 |
egrep "VRF vnid|sclass "
pcTAG of dest EPG
BD vnid : 15826920 ::: VRF vnid : 2293762
(provider
of PBR flow)
Flags : 0x80000c80 ::: sclass : 16386
1
Leaf-2
Leaf-1
5
EP1 EP2
Shadow EPG (16389) to
provider (16386) is
implicitly allowed (default
filter) by service graph
a1-leaf2# show zoning-rule scope 2293762 src-epg 16389 dst-epg 16386
+---------+--------+--------+----------+---------+---------+---------+------+--------+----------------+
| Rule ID | SrcEPG | DstEPG | FilterID |
Dir
| operSt | Scope | Name | Action |
Priority
|
+---------+--------+--------+----------+---------+---------+---------+------+--------+----------------+
|
4152 | 16389 | 16386 | default | uni-dir | enabled | 2293762 |
| permit | src_dst_any(9) |
+---------+--------+--------+----------+---------+---------+---------+------+--------+----------------+
a1-leaf2# contract_parser.py --vrf mg-cisco-live:v1 --depg tn-mg-cisco-live/ap-a1/epg-e2
Contract hit
[9:4152] [vrf:mg-cisco-live:v1] permit any tn-mg-cisco-live/G-pbr-one-arm-CL2023ctxv1/C-provider(16389) tn-mgcisco-live/ap-a1/epg-e2(16386) [contract:uni/tn-mg-cisco-live/brc-ip] [hit=172]
*NOTE FOR RETURN TRAFFIC: EP2->EP1 PBR flow is the same
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
39
PBR
Troubleshooting
Workflow
PBR Troubleshooting Workflow
Is the service graph
deployed?
YES
YES
Is the service device
reachable from the
service leaf?
ELAM to understand
where packet is
forwarded/dropped
NO
NO
YES
Is there a provider and
consumer associated to
PBR contract? Validate
with contract_parser
Is the service VLAN
deployed on the service
leaf?
Are the service destination
parameters (IP/MAC) for
service device correct?
NO
Use FTRIAGE and
captures on service
device to see if packet
is seen
YES
-Check zoning-rules on leaf
-Look for “service_redir” flag
in ELAM report
-Check service device for
ACLs
TIPS
YES
NO
Make sure the correct
service graph template
is tied to subject of the
PBR contract and EPGs
respectively
NO
Correct configuration in
PBR redirect policy
Make sure cluster interface
config matches to
appropriate service VLAN
#CiscoLive
BRKDCN-3615
Check faults
regarding service
graph deployment
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
41
Common Issues
#1: Service Graph
is Not Deploying
Common PBR Problems
#1: Graph is not deploying
Things to check:
• Is the service graph template is correctly tied to the contract?
Nothing actually
deployed
• Does contract have a filter associated?
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
44
Common PBR Problems
#1: Graph is not deploying
Nothing actually
deployed
Things to check:
• Is the contract filter valid?
No actual filter entry configured!
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
45
Common PBR Problems
#1: Graph is not deploying
FIX: Associate proper filter or correct contract misconfiguration
Service Graph now
deployed
Additional
Config
“Gotcha” for
inter-VRF PBR
The contract has the filter
correctly configure but the
scope is set to “VRF” even
though it's for inter-VRF or
inter-tenant EPG
communication.
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
46
#2: Graph Shows
Deployed but
Traffic is Not
Redirected
Common PBR Problems
#2: Graph shows deployed but traffic is not directed
Things to check:
• Check the fault info from APIC (can give you hints to what could be wrong)
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
48
Common PBR Problems
#2: Graph shows deployed but traffic is not directed
Things to check:
• Check which EPG the service VLAN is mapped to
a1-leaf2# show vlan encap-id 1417
The service VLAN (1417) shows it
is already tied to an EPG called
“mg-cisco-live:FW:ASAV”
VLAN Name
Status
Ports
---- -------------------------------- --------- ------------------------------44
mg-cisco-live:FW:ASAV
active
Eth1/43
GUI validation shows a static binding
towards FW was deployed in an
application EPG called “ASAV”
VLAN Type Vlan-mode
---- ----- ---------44
enet CE
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
49
Common PBR Problems
#2: Graph shows deployed but traffic is not directed
Things to check:
• Check PBR function connectors to see VLAN encap configured
Since static binding was deployed
first this PBR VLAN deployment fails
due to overlapping VLAN encap ID:
Service Graph configuration using
the same VLAN as APP EPG
(VLAN-1417)
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
50
Common PBR Problems
#2: Graph shows deployed but traffic is not directed
FIX: Delete EPG static binding that maps to service node VLAN
a-apic1# moquery -c faultInst -f 'fault.Inst.code=="F0497"' | grep cisco-live
a-apic1#
Fault is now cleared!
a1-leaf2# show vlan encap-id 1417
VLAN Name
Status
Ports
---- -------------------------------- --------- -------------------------------Service VLAN is properly
62
mg-cisco-live:pbr-one-armactive
Eth1/43
associated to service EPG
CL2023ctxv1:provider:
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
51
#3: Traffic is
Working but Does
Not Hit Redirect
Rule
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
asav-cisco-live# capture pbr_traffic interface
inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
*PBR traffic not seen*
Things to check:
• Contract parser script on leaf
a1-leaf1# contract_parser.py --vrf mg-cisco-live:v1 --sepg tn-mg-cisco-live/ap-a1/epg-e1 --depg tnmg-cisco-live/ap-a1/epg-e2
Contract “ip”
[7:4351] [vrf:mg-cisco-live:v1] redir ip tn-mg-cisco-live/ap-a1/epg-e1(32771) tn-mg-cisco-live/apa1/epg-e2(16386) [contract:uni/tn-mg-cisco-live/brc-ip] [hit=348] destgrp-3 vrf:mg-cisco-live:v1
ip:172.16.1.100 mac:00:00:00:00:00:54 bd:vxlan-14974950
Rule ID
Contract “icmp”
[7:4330] [vrf:mg-cisco-live:v1] permit ip icmp tn-mg-cisco-live/ap-a1/epg-e1(32771) tn-mg-ciscolive/ap-a1/epg-e2(16386) [contract:uni/tn-mg-cisco-live/brc-icmp] [hit=796,+10]
a1-leaf1# contract_parser.py --vrf mg-cisco-live:v1 --sepg tn-mg-cisco-live/ap-a1/epg-e1 --depg tnmg-cisco-live/ap-a1/epg-e2
[7:4330] [vrf:mg-cisco-live:v1] permit ip icmp tn-mg-cisco-live/ap-a1/epg-e1(32771) tn-mg-ciscolive/ap-a1/epg-e2(16386) [contract:uni/tn-mg-cisco-live/brc-icmp] [hit=1294,+10]
Hit count is increasing
for “icmp” contract!
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
53
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
asav-cisco-live# capture pbr_traffic interface
inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
*PBR traffic not seen*
Things to check:
• Alternatively you can get an ELAM to see what rule is being hit (*Refer to
slide 12 to see where the PBR policy should be applied*)
module-1(DBG-elam-insel6)# trigger reset
module-1(DBG-elam)# debug platform internal tah elam asic 0
module-1(DBG-elam)# trigger init in-select 6 out-select 1
module-1(DBG-elam-insel6)# set outer ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# stat
ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Triggered
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
54
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
asav-cisco-live# capture pbr_traffic interface
inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
*PBR traffic not seen*
Things to check:
• Check rule being hit in ereport
Contract Lookup Key
----------------------------------------------------------------------------------------------------------------------------------------------------IP Protocol
: ICMP( 0x1 )
L4 Src Port
: 2048( 0x800 )
L4 Dst Port
: 8647( 0x21C7 )
sclass (src pcTag)
: 32771( 0x8003 )
 EPG consumer 192.168.1.100
dclass (dst pcTag)
: 16386( 0x4002 )
 EPG provider 192.168.2.100
Contract Result
Contract Drop
: no
Contract Logging
: no
Contract Applied
: yes
Contract Hit
: yes
Contract Aclqos Stats Index
: 81863
Use this command to get rule ID!
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81863" )
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
55
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
asav-cisco-live# capture pbr_traffic interface
inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
*PBR traffic not seen*
Things to check:
• Check rule being hit (command from previous slide)
module-1# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81863"
Rule ID: 4328 Scope 44 Src EPG: 32771 Dst EPG: 16386 Filter 5
| grep "Rule ID"
• Check zoning-rule output for the rule ID above (4238)
a1-leaf1# show zoning-rule scope 2293762 | egrep "Rule ID|4328"
| Rule ID | SrcEPG | DstEPG | FilterID | Scope
|
Name
|
4328 | 32771 | 16386 |
5
| 2293762 |
mg-cisco-live:icmp
• Map filter ID to type of traffic
|
|
No “redir” action
Action
permit
|
|
Priority
fully_qual(7)
VRF VNID
a1-leaf1# show zoning-filter filter 5 | grep -A 2 Protocol
| FilterId | Name | EtherT |
ArpOpc
| Prot | ApplyToFrag | Stateful | SFromPort |
SToPort
| DFromPort
|----------+------+--------+-------------+------+-------------+----------+-------------+-------------+------------|
5
| 5_0 |
ip
| unspecified | icmp |
no
|
no
| unspecified | unspecified | unspecified
Filter has specific protocol set
for ICMP traffic
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
56
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
asav-cisco-live# capture pbr_traffic interface
inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
*PBR traffic not seen*
Things to check:
• Look for redirect rule and compare to previous slide
a1-leaf1# show zoning-rule scope 2293762 | grep redir
|
4325 | 16386 | 32771 |
1
| uni-dir-ignore | enabled | 2293762 | redir(destgrp-3) | fully_qual(7)
|
4351 | 32771 | 16386 |
1
|
bi-dir
| enabled | 2293762 | redir(destgrp-3) | fully_qual(7)
• Validate contract filter
a1-leaf1# show zoning-filter filter 1 | grep -A 2 Protocol
| FilterId | Name | EtherT |
ArpOpc
| Prot | ApplyToFrag | Stateful | SFromPort |
SToPort
| DFromPort
|----------+------+--------+-------------+------+-------------+----------+-------------+-------------+------------|
1
| 1_0 |
ip
| unspecified | ip |
no
|
no
| unspecified | unspecified | unspecified
Filter has specific protocol set
for IP traffic
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
57
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
asav-cisco-live# capture pbr_traffic interface inside
real-time match icmp
Use ctrl-c to terminate real-time capture
0 packets shown.
Things to check:
• Check contract provider and consumer of both contracts
Contract “icmp” (standard contract)
a-apic1# moquery -c vzRtProv -f 'vz.RtProv.dn*"tn-mg-cisco-live/brc-icmp"'| grep dn
dn
: uni/tn-mg-cisco-live/brc-icmp/rtfvProv-[uni/tn-mg-cisco-live/ap-a1/epg-e2]
a-apic1# moquery -c vzRtCons -f 'vz.RtCons.dn*"tn-mg-cisco-live/brc-icmp"'| grep dn
dn
: uni/tn-mg-cisco-live/brc-icmp/rtfvCons-[uni/tn-mg-cisco-live/ap-a1/epg-e1]
/brc-icmp/rtfvProv-[uni/tn-mg-cisco-live/ap-a1/epg-e2]
Contract “ip” (PBR contract)
a-apic1# moquery -c vzRtProv -f 'vz.RtProv.dn*"tn-mg-cisco-live/brc-ip"'| grep dn
dn
: uni/tn-mg-cisco-live/brc-ip/rtfvProv-[uni/tn-mg-cisco-live/ap-a1/epg-e2]
Both EPGs have
contracts ”ip”
and “icmp”
associated.
More specific
filter is
preferred!
(ICMP in this
case)
a-apic1# moquery -c vzRtCons -f 'vz.RtCons.dn*"tn-mg-cisco-live/brc-ip"'| grep dn
dn
: uni/tn-mg-cisco-live/brc-ip/rtfvCons-[uni/tn-mg-cisco-live/ap-a1/epg-e1]
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
58
Common PBR Problems
#3 : Traffic is working but
does not hit redirect rule
FIXES:
1) Remove ICMP contract between EPGs so “IP” filter takes over and ping traffic is redirected
2) Reconfigure filters on contracts to match specific types of traffic to be redirected
asav-cisco-live# capture pbr_traffic interface inside real-time match icmp any any
1:
2:
3:
4:
17:40:29.668056
17:40:29.668224
17:40:29.668681
17:40:29.668712
192.168.1.100
192.168.1.100
192.168.2.100
192.168.2.100
>
>
>
>
192.168.2.100
192.168.2.100
192.168.1.100
192.168.1.100
#CiscoLive
icmp:
icmp:
icmp:
icmp:
BRKDCN-3615
echo
echo
echo
echo
request
request
reply
reply
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
59
#4: PBR Packet
Never makes it to
PBR Node with
Redirect Rule in
Place
Common PBR Problems
#4: Packet never makes it to PBR Node with redirect rule in place
Things to check:
• Validate with PBR capture you do not see traffic on service node
asav-cisco-live# capture pbr_traffic interface inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
0 packets shown.
• Check Service Node EP learning
a1-leaf2# show system internal epm endpoint ip 172.16.1.100
MAC : 0050.56a8.4897 ::: Num IPs : 1
IP# 0 : 172.16.1.100 ::: IP# 0 flags : host-tracked| ::: l3-sw-hit: Yes ::: flags2 :
Vlan id : 32 ::: Vlan vnid : 14214 ::: VRF name : mg-cisco-live:v1
BD vnid : 16744311 ::: VRF vnid : 2293762
• Validate if redirect rule is there on ingress leaf
a1-leaf1# show zoning-rule scope 2293762 | grep redir
|
4325 | 16386 | 32771 |
1
| uni-dir-ignore | enabled | 2293762 | redir(destgrp-3) | fully_qual(7)
|
4351 | 32771 | 16386 |
1
|
bi-dir
| enabled | 2293762 | redir(destgrp-3) | fully_qual(7)
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
61
Common PBR Problems
#4: Packet never makes it to PBR Node
Things to check:
• Can service node even reach the PBR NH?
a1-leaf2# iping -V mg-cisco-live:v1 172.16.1.100 -S 172.16.1.1
PING 172.16.1.100 (172.16.1.100) from 172.16.1.1: 56 data
bytes
64 bytes from 172.16.1.100: icmp_seq=0 ttl=255 time=1.044 ms
64 bytes from 172.16.1.100: icmp_seq=1 ttl=255 time=1.027 ms
64 bytes from 172.16.1.100: icmp_seq=2 ttl=255 time=0.979 ms
64 bytes from 172.16.1.100: icmp_seq=3 ttl=255 time=0.888 ms
64 bytes from 172.16.1.100: icmp_seq=4 ttl=255 time=0.955 ms
• FTRIAGE to see where packet is lost
2023-05-22 10:57:20,535 INFO
ftriage:
main:1295
Egress: LC-2/2 FC-22/0 Port-1 Vnid: 16744311
2023-05-22 10:58:37,770 ERROR
ftriage:
fib:727
16744311
2023-05-22 10:58:47,735 ERROR
ftriage: unicast:2167
exit but continue with further fault isolation
2023-05-22 10:58:47,735 INFO
ftriage: unicast:2207
local EP. Exiting!
#CiscoLive
We have reachability from service
node BD SVI
Spine gets the packet but EP lookup
for redirect MAC in COOP fails!
L3 packet Seen on
a1-spine1 Ingress: Eth2/24
a1-spine1: EP not found in COOP! for VRF VNID:
a1-spine1: EP is unknown in COOP. Ftriage will
a1-spine1: Egress node not provided. Cannot check
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
62
Common PBR Problems
#4: Packet never makes it to PBR Node
FW MAC
asav-cisco-live# show int gigabitEthernet MAC
address 0050.56a8.4897, MTU 1500
Things to check:
• Is the redirect IP and MAC of service device set correctly? *
*For releases later
than APIC Release
5.2, the MAC
address
configuration is not
mandatory for L3
PBR if IP-SLA
tracking is enabled)*
Redirect MAC is
configured
incorrectly!
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
63
Common PBR Problems
#4: Packet never makes it to PBR Node
FIX: Correct NH MAC address under Tenants -> Policies -> Protocol -> L4/L7 Policy-Based Redirect
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
64
Common PBR Problems
#4: Packet never makes it to PBR Node
After correcting last octet of FW MAC tracking is seen on PBR node as expected
mg-asav-cisco-live# capture pbr_traffic interface inside real-time match icmp any any
Use ctrl-c to terminate real-time capture
1:
2:
3:
4:
15:06:07.671870
15:06:07.671916
15:06:07.672297
15:06:07.672313
192.168.1.100
192.168.1.100
192.168.2.100
192.168.2.100
>
>
>
>
192.168.2.100
192.168.2.100
192.168.1.100
192.168.1.100
#CiscoLive
icmp:
icmp:
icmp:
icmp:
BRKDCN-3615
echo
echo
echo
echo
request
request
reply
reply
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
65
Resources
Packet Format
1 From EP1 to Leaf-1
SIP
192.168.1.100
Inner
Pod1
DIP
192.168.2.100
SMAC
EP1
Spine-1
DMAC
LEAF MAC
3
2 From Leaf-1 to fabric– PACKET REWRITE (EP -2 is known to Leaf-1)
SIP
192.168.1.100
Inner
Pod1
DIP
192.168.2.100
Outer
SMAC
EP1
DMAC
FW MAC
VNID
Service BD
SCLASS
EPG1
DCLASS
EPG2
SIP
Leaf101 PTEP
DIP
Spine Anycast MAC
2
3 After packet rewrite, Spine does MAC COOP Lookup in Service BD
SIP
192.168.1.100
Inner
Pod1
DIP
192.168.2.100
SIP
192.168.1.100
SPINE MAC
DMAC
LEAF MAC
VNID
Service BD
SCLASS
EPG1
DCLASS
Shadow EPG
SIP
SPINE PTEP
SIP
192.168.1.100
Leaf-2
Leaf-1
Leaf-3
DIP
LEAF-2 PTEP
SMAC
FW MAC
4
1
EP1
DMAC
LEAF MAC
EP1 – EPG1
192.168.1.100
5 After FW, between FW leaf and destination leaf 301
Inner
Pod1
DIP
192.168.2.100
5
Outer
4 Coming back from FW on Leaf-2
Inner
Pod1
DIP
192.168.2.100
Spine-2
EP2
EP2
FW
172.16.1.100
EP2 – EPG2
192.168.2.100
Outer
SMAC
LEAF MAC
DMAC
EP2
VNID
VRF
SCLASS
SHADOW EPG
DCLASS
EPG2
SIP
LEAF-2 PTEP
#CiscoLive
DIP
LEAF-3 PTEP
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
67
BR
68K D C N - 3 6 1 5
PBR Datapath ELAMs (via Leaf CLI)
On leaf from front panel Port (ingress from server, or back from firewall)
debug platform internal tah elam asic 0 trigger reset
trigger init in-select 6 out-select 1
set outer ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set outer l2 src_mac <SRC MAC EP>
Spine-1
Spine-2
On Spine or egress leaf Before Redirect (for example if EP is unknown in ingress
leaf)
debug platform internal roc elam asic 0 trigger reset
trigger init in-select 14 out-select 1
set inner ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set outer l4 tn-seg-id 0x2e001 (VRF VNID in HEX)
Leaf-2
Leaf-1
Leaf-3
On Spine or egress leaf after redirect and before Firewall
debug platform internal tah elam asic 0 trigger reset
trigger init in-select 14 out-select 1
set inner ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set outer l4 tn-seg-id 0xe27fef
(service BD VNID)
On leaf from front panel Port (ingress from server, or back from firewall)
debug platform internal tah elam asic 0 trigger reset
trigger init in-select 6 out-select 1
set outer ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set outer l2 src_mac <MAC FIREWALL>
On spine or egress leaf (ingress from server, or back from firewall)
debug platform internal tah elam asic 0 trigger reset
trigger init in-select 14 out-select 1
set inner ipv4 src_ip 192.168.1.100 dst_ip 192.168.2.100
set inner l2 src_mac <MAC FIREWALL>
EP1
EP1 – EPG1
192.168.1.100
EP2
FW
172.16.1.100
EP2 – EPG2
192.168.2.100
*Note for these ELAMs were on –EX leaf (“tah”), asic family
is different for other switches*
For FX and FX2 – Use “roc”
For GX = Use “app”
For GX2 = Use “cho”
PBR White Paper:
REFERENCES
https://techzone.cisco.com/t5/Application-Centric/ACI-Multisite-Policy-Based-Redirect-PBR-Configuration-for-East/ta-p/1581084#toc-hId293703813
ACI Multi-Pod and Service Node Integration White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-739571.html
ACI Multi-Site and Service Node Integration White Paper
https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-743107.html
Troubleshoot ACI Policy-Based Redirect
https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/218047-troubleshootaci-policy-based-redirect.html
Cisco APIC Layer 4 to Layer 7 Deployment Guide, Release 5.0(x)
https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/
5-x/l4-l7-services/cisco-apic-layer-4-to-layer-7-services-deploymentguide50x/m_selecting_a_layer_4_to_layer_7_device_to_render_a_graph.html
Cisco Live On-Demand Library
https://www.ciscolive.com/on-demand/on-demandlibrary.html?search=PBR#/
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
69
Fill out your session surveys!
Attendees who fill out a minimum of four session
surveys and the overall event survey will get
Cisco Live-branded socks (while supplies last)!
Attendees will also earn 100 points in the
Cisco Live Game for every survey completed.
These points help you get on the leaderboard and increase your chances of winning daily and grand prizes
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
70
Continue
your education
•
Visit the Cisco Showcase
for related demos
•
Book your one-on-one
Meet the Engineer meeting
•
Attend the interactive education
with DevNet, Capture the Flag,
and Walk-in Labs
•
Visit the On-Demand Library
for more sessions at
www.CiscoLive.com/on-demand
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
71
Thank you
#CiscoLive
Gamify your Cisco Live experience!
Get points for attending this session!
How:
1 Open the Cisco Events App.
2 Click on 'Cisco Live Challenge’ in the side menu.
3 Click on View Your Badges at the top.
4 Click the + at the bottom of the screen and scan the QR code:
#CiscoLive
BRKDCN-3615
© 2023 Cisco and/or its affiliates. All rights reserved. Cisco Public
73
#CiscoLive