FWSM: Troubleshoot Traffic Failures Due to Wrong
Xlates
Document ID: 115010
Contributed by Michael Robertson and Jay Johnston, Cisco TAC
Engineers.
Nov 02, 2012
Contents
Introduction
Prerequisites
Requirements
Components Used
Conventions
Symptoms
Logical Topology
Relevant Configuration
Observed Behaviors
Triggers
Solutions
Resolve Incorrect Routing Configurations
Disable same−security−traffic permit intra−interface
Drop Packets that Arrive on an Incorrect Interface (ACLs or uRPF)
Enable xlate−bypass
Summary
Related Information
Introduction
Due to the design of the Firewall Services Module's (FWSM) packet processing, xlates built by incorrectly
routed packets can cause traffic failures for connections through the firewall. In order to select an egress
interface for an inbound packet, the FWSM first checks to see if the destination IP of the inbound packet
matches any existing global IP/Network in a NAT translation (xlate) for that interface in its xlate table. If a
match is found, the egress interface is simply chosen based on the local interface in the xlate entry and the
firewall does not consult the routing table to make the egress interface decision.
The default behavior of the FWSM is to build an xlate entry for the source IP of any permitted packet that is
received on one of its interfaces. If a packet is routed through the network incorrectly (for any number of
reasons) and arrives inbound on the wrong interface of the FWSM, an xlate is built to reflect this. When this
occurs, entries in the xlate table can override entries in the routing table and cause traffic failures for the
affected destinations.
This document describes the symptoms and triggers for this issue, how to diagnose it, and provides solutions
for preventing it from occurring.
Prerequisites
Requirements
Cisco recommends that you have knowledge of FWSMs.
Components Used
This document is not restricted to specific software and hardware versions.
Conventions
Refer to Cisco Technical Tips Conventions for more information on document conventions.
Symptoms
Logical Topology
Relevant Configuration
interface Vlan1
nameif outside
security−level 0
ip address 192.168.100.50 255.255.255.0
!
interface Vlan10
nameif inside
security−level 100
ip address 10.10.1.50 255.255.255.0
!
interface Vlan20
nameif dmz
security−level 50
ip address 10.20.1.50 255.255.255.0
!
same−security−traffic permit intra−interface
access−list outside_in extended permit tcp any host 10.30.1.1 eq www
access−list inside_in extended permit ip any any
access−group inside_in in interface inside
access−group outside_in in interface outside
route outside 0.0.0.0 0.0.0.0 192.168.100.254
route dmz 10.30.1.0 255.255.255.0 10.20.1.254
Observed Behaviors
Connections from the client PC at 172.16.1.10 to the web server at 10.30.1.1 fail.
A packet capture on the outside interface shows a TCP SYN from the client PC arriving at the FWSM's
interface.
FWSM# show capture outside
3 packets seen, 3 packets captured
1: 13:58:09.280752960 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S
918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK>
2: 13:58:12.280755950 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S
918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK>
3: 13:58:18.280761960 802.1Q vlan#1 P0 172.16.1.10.57389 > 10.30.1.1.80: S
918518428:918518428(0) win 8192 <mss 1380,nop,nop,sackOK>
3 packets shown
A packet capture on the dmz interface does not show that packet leaving the firewall.
FWSM# show capture dmz
0 packet seen, 0 packet captured
0 packet shown
No entry is built in the FWSM's connection table and syslogs do not show any information related to the client
or server IP addresses.
Triggers
At a fundamental level, this issue is caused by an entry in the FWSM's xlate table that was built by an
incorrectly routed packet. Because of the way the FWSM's packet processing is designed, the firewall checks
the xlate table before it checks the routing table to determine the egress interface. As a result, if a packet
matches an existing xlate the egress interface will be selected based on that entry, even if the entry conflicts
with what is listed in the routing table. In other words, the xlate table takes precedence over the routing table.
In order to diagnose this issue, check the output of the show xlate debug command:
FWSM# show xlate debug
Flags: D − DNS, d − dump, I − identity, i − inside, n − no random,
o − outside, r − portmap, s − static
3 in use, 3 most used
NAT from inside:10.30.1.1 to outside:10.30.1.1 flags Ii idle 0:00:00 timeout 3:00:00 connec
NAT from inside:10.30.1.1 to inside:10.30.1.1 flags Ii idle 0:00:07 timeout 3:00:00 connect
NAT from dmz:10.30.1.1 to outside:10.30.1.1 flags Ii idle 0:00:10 timeout 3:00:00 connectio
Note: The debug keyword in the show xlate is crucial. Without it, the xlate entries will not include the
interface names that the entry is associated with.
The xlate table shows that there are 3 xlates built for the web server. The first xlate is built between the inside
interface and the outside interface. The second xlate is built as a hairpinned or u−turned xlate on the inside
interface. The third xlate is built between the dmz and the outside interface. The I flag indicates that this is an
identity xlate and the IP is not actually being translated.
The first interface listed in the entry is the "real" or "local" interface where the IP is supposed to actually exist.
The second interface listed is the "mapped" or "global" interface where the IP is being translated. Neither of
these xlates shown are correct. This is because the web server (10.30.1.1) actually exists behind the dmz
interface. The third xlate is correct for this network design.
The connection failure occurs because of the first xlate listed in the table. When the client's TCP SYN packet
arrives on the outside interface destined to 10.30.1.1, the FWSM checks the xlate table and matches the first
entry. This entry indicates that the packet should egress on the inside interface, which is incorrect, and the
packet is blackholed.
By default, the FWSM will automatically build an identity xlate for any traffic that does not match an
explicitly configured NAT rule. Because of this, even if a packet erroneously arrives on an incorrect interface,
an xlate will be built. Specifically for this case, packets sourced from 10.30.1.1 arrived inbound on the inside
interface instead of arriving on the dmz interface as is expected.
The first xlate (inside > outside) was built when the web server tried to ping a non−existent IP address
(10.199.199.1). The echo request left the web server destined to its default gateway (the DMZ router). The
DMZ router forwarded the packet toward the inside router, per its static route:
S
10.0.0.0/8 [1/0] via 10.50.1.254
Because the 10.199.199.0/24 network does not actually exist anywhere, the inside router simply follows its
default route and sends the packet to the FWSM's inside interface:
S*
0.0.0.0/0 [1/0] via 10.20.1.50
Likewise, the FWSM also does not have a route for the destination network. Therefore, it selects the outside
interface as the egress interface and builds an identity xlate from inside > outside:
S
0.0.0.0 0.0.0.0 [1/0] via 192.168.100.254, outside
The second xlate (inside > inside) was built when the web server tried to access the DNS server while the
inside router's 10.40.1.254 interface was temporarily down due to a link flap. The DNS request left the web
server destined to its default gateway (the DMZ router). The DMZ router forwarded the packet toward the
inside router, per its static route:
S
10.0.0.0/8 [1/0] via 10.50.1.254
However, the inside router's interface connected to the 10.40.1.0/24 network was temporarily down and its
directly connected route for this network was missing. Therefore, the only matching route in the routing table
was the default route back toward the FWSM:
S*
0.0.0.0/0 [1/0] via 10.20.1.50
The packet was routed to the FWSM's inside interface. The FWSM's routing table indicated that the
destination network of 10.40.1.0/24 existed behind the same inside interface:
S
10.40.1.0 255.255.255.0 [1/0] via 10.10.1.254, inside
Because the same−security−traffic permit intra−interface command is enabled, the FWSM will allow the
u−turned xlate to be built.
To summarize, the first xlate was triggered by:
• A broad 10.0.0.0/8 route configured on the DMZ router
• A permit ip any any ACL configured on the FWSM's inside interface
The second xlate was triggered by:
• A flapping interface on the Inside router
• same−security−traffic permit intra−interface configured on the FWSM
Solutions
There are many different possible solutions to this problem. First and foremost, deleting the xlate from the
table should allow traffic to start working again until the xlate is rebuilt. This can be done with the clear xlate
command. For example:
FWSM# clear xlate interface inside local 10.30.1.1 global 10.30.1.1
Note: Any connections that are using the deleted xlate(s) will also be torn down.
Once that is complete, the focus should be on preventing the xlates from returning. Often times, the most
preferred way to do this is to fix the routing configuration in the environment to prevent traffic from arriving
on the wrong FWSM interface. The FWSM also offers a handful of configuration options to address these
issues.
Resolve Incorrect Routing Configurations
This solution takes careful planning and a deep understanding of the network environment. In the first
example above, the 10.0.0.0/8 route on the DMZ router is technically incorrect since the entire /8 network
does not exist beyond its 10.50.1.253 interface. Instead, some options that exist are:
• Eliminate the 10.50.1.0/24 network all together and simply route all traffic through the FWSM. This
also provides better segmentation and security between the Inside and DMZ networks.
• Configure a static route on the DMZ for only 10.40.1.0/24 and remove the 10.0.0.0/8 route.
• Use a dynamic routing protocol between the Inside and DMZ routers to correctly advertise only the
networks that actually exist.
There are often many possibilities for adjusting the routing configuration, but the end goal is to ensure that
traffic from a given host is able to arrive only on a single FWSM interface.
Disable same−security−traffic permit intra−interface
The same−security−traffic permit intra−interface command allows the FWSM to u−turn or hairpin traffic
on an interface. This means that a packet can enter the firewall on the same interface it leaves on. This
functionality is disabled by default and has very little use in most FWSM designs. Because the FWSM uses
VLAN interfaces, traffic that stays within the same VLAN should never be processed by the FWSM.
In the second example above, the same−security−traffic permit intra−interface command allowed a packet
to both enter and leave the inside interface. Disabling same−security−traffic permit intra−interface would
prevent this behavior and drop the packet before an xlate was ever built:
FWSM(config)# no same−security−traffic permit intra−interface
Drop Packets that Arrive on an Incorrect Interface (ACLs or uRPF)
In both examples above, the xlates were built when a packet from the web server incorrectly arrived on the
inside interface. In order to prevent the problem all together, the FWSM can be configured to drop packets
that arrive on the wrong interface.
The FWSM requires that all traffic be permitted by an ACL before it can pass. Therefore, this functionality
can be achieved by only permitting traffic from appropriate source networks on each interface. In the
examples above, the inside interface permits all IP traffic:
access−list inside_in extended permit ip any any
Instead, this should be changed to only permit traffic from the 10.10.1.0/24 and 10.40.1.0/24 subnets:
access−list inside_in extended permit ip 10.10.1.0 255.255.255.0 any
access−list inside_in extended permit ip 10.40.1.0 255.255.255.0 any
In some environments, this is not a feasible option due to the size and/or scale of the different networks
passing through the FWSM. However, this functionality can be achieved more simply using a feature called
Unicast Reverse Path Forwarding (uRPF).
When the uRPF feature is enabled, the FWSM will compare the source IP address of the first packet of every
connection against its routing table. If the route that is found does not match up with the interface that the
packet arrived on, that packet will be dropped due to a RPF failure.
In the example above, the FWSM has a static route that uses the dmz interface to reach the 10.30.1.0/24
network. Therefore, if uRPF is enabled on the inside interface, packets sourced from the web server
(10.30.1.1) that arrive incorrectly on the inside interface will be dropped.
In order to enable uRPF, apply the ip verify reverse−path command to each interface in question. For
example:
FWSM(config)# ip verify reverse−path interface inside
Enable xlate−bypass
In both of the examples above, the xlates are created with the Ii flags. These flags indicate that the xlate is an
identity translation (I) that originated on a high security (i) interface. By default, the FWSM will build these
xlates for any traffic that does not match an explicit NAT/PAT rule. In order to disable this behavior, the
xlate−bypass command can be enabled in FWSM 3.2(1) and later:
FWSM(config)# xlate−bypass
This feature will prevent the FWSM from building identity xlates in the first place. Thus, the traffic in the
examples above would not be redirected to an incorrect interface due to an xlate table entry. However, the
traffic will still pass through the FWSM untranslated.
Summary
In order to determine the egress interface for a packet, the FWSM will always consult its xlate table before
looking at its routing table. If that packet matches an existing xlate, the egress interface is selected based on
the xlate's associated interface. This happens regardless of any contradictions that might be found in the
routing table. In this way, the xlate table takes precedence over the routing table.
Because the FWSM will always build an xlate entry for all new connections by default, this can cause traffic
failures in cases where incorrectly routed packets cause the FWSM to build an xlate. As outlined above, there
are many possible scenarios where this can occur but all relate back to a packet being received on an incorrect
interface. This document covered these possible issues:
• A broad routing config sends packets in an incorrect direction
• The FWSM is configured to permit traffic from incorrect source networks
• The FWSM is configured to hairpin/u−turn traffic
In order to quickly restore connectivity for connections that fail due to a wrong xlate, delete the entry with the
clear xlate command. This document also covered multiple solutions for preventing these xlates from
returning in the future, including:
• Resolve incorrect routing configurations using more specific routes
• Disable same−security−traffic permit intra−interface
• Drop packets that arrive on an incorrect interface using ACLs or uRPF
• Enable xlate−bypass
Related Information
• Command Reference: ip verify reverse−path
• Command Reference: xlate−bypass
• Technical Support & Documentation − Cisco Systems
Contacts & Feedback | Help | Site Map
© 2014 − 2015 Cisco Systems, Inc. All rights reserved. Terms & Conditions | Privacy Statement | Cookie Policy | Trademarks of
Cisco Systems, Inc.
Updated: Nov 02, 2012
Document ID: 115010