Expert Reference Series of White Papers
Cisco Security
Troubleshooting:
Part I – Connectivity
Through ASA or PIX
Firewalls
1-800-COURSES
www.globalknowledge.com
Cisco Security Troubleshooting: Part I –
Connectivity Through ASA or PIX Firewalls
Douglas B. McKillip, P.E., CCSI, CCSP, CCIE #1851
Introduction
This white paper is the first in a three-part series, each of which examines the challenge of implementing
network security on equipment from Cisco Systems® while maintaining the connectivity requirements of the
business or enterprise.
Cisco Security Troubleshooting
Part I: Connectivity Through ASA or PIX Firewall
Part II: Virtual Private Networks
Part III: Troubleshooting Intrusion Prevention Systems.
The focus in Part 1 will primarily be on the most effective use of both GUI-based and CLI-based troubleshooting
tools. We will examine the following troubleshooting tools available on both the ASA and PIX firewalls except
• debug packet (PIX OS 6.3 only)
• capture
• packet tracer
Troubleshooting Scenario
The following scenario will be used to illustrate the use of the troubleshooting tools outlined above. If the
names and IP Addresses look familiar to some of you, this is because the diagram represents a portion of the lab
topology used in Cisco® Security classes offered by Global Knowledge.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
2
Figure 1
Debug Packet
As mentioned previously, the debug packet command is only available on PIX Firewalls, having been introduced as early as the 5.x generation of code. While “debug packet” is no longer supported beginning in
version 7.0 of either the PIX or ASA code, we will later examine the use of the newer “real-time” option to the
capture command as an alternative.
The following 2 @ debug packet commands were used to verify the TCP 3-way handshake originating from
the Outside PC, 150.150.1.20 and the DMZ-Server, 172.16.1.15:
(initiated via
http://200.200.1.15)
debug packet dmz src 150.150.1.20 dst 172.16.1.15 proto tcp
dport 80
debug packet dmz src 172.16.1.15 dst 150.150.1.20 proto tcp
sport 80
Note below that the attempt to verify both debugs only shows the first!
LABPIX(config)#
show debug
debug packet dmz src 150.150.1.20 dst 172.16.1.15 proto tcp dport
80 both
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
3
Here is the live output (the raw hex and sequence #’s have been omitted).
--------- PACKET ---------- IP -150.150.1.20
==>
ver = 0x4
hlen = 0x5 tos = 0x0
id = 0xf40eflags = 0x40frag off=0x0
ttl = 0x7f proto=0x6
-- TCP --
172.16.1.15
tlen = 0x34
chksum = 0xc2eb
source port = 0xa8f
dest port = 0x50syn
--------- END OF PACKET ---------(output truncated)
--------- PACKET ---------- IP -172.16.1.15==>
150.150.1.20
ver = 0x4
hlen = 0x5 tos = 0x0
id = 0x501 flags = 0x40frag off=0x0
ttl = 0x20 proto=0x6
-- TCP --
tlen = 0x2c
chksum = 0x1102
source port = 0x50
dest port = 0xa8fsyn ack
--------- END OF PACKET ---------(output truncated)
--------- PACKET ---------- IP -150.150.1.20
==>
ver = 0x4
hlen = 0x5 tos = 0x0
id = 0xf410flags = 0x40frag off=0x0
ttl = 0x7f proto=0x6
-- TCP --
172.16.1.15
tlen = 0x28
chksum = 0xc2f5
source port = 0xa8f
dest port = 0x50ack
--------- END OF PACKET --------Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
4
Two points are worthy of note in the previous example. First, each debug packet command was used with as
much qualification as possible to target the output to be relevant for troubleshooting the specific web connection. Second, the use of the second simultaneous debug with the parameters carefully inverted results in the
ability to see all of the http client-server exchange.
Capture – Part I – Output to Terminal
The capture command first became available in PIX OS 6.2; however, the real-time option only recently became
available in release 8.0. In this next section we will examine the flexibility in how this valuable troubleshooting
command can be used. Again, for all the alternative implementations examined here, the same scenario with
the Outside-PC at 150.150.1.20 connecting to the DMZ-Server at 172.16.1.15 (http://200.200.1.15)
will be used.
- Commands entered:
access-list WWW_TRAFFIC permit tcp host 150.150.1.20 host
172.16.1.15 eq 80
access-list WWW_TRAFFIC permit tcp host 172.16.1.15
host 150.150.1.20
capture WWW-TEST-DMZ
real-time
access-list WWW_TRAFFIC
eq 80
interface
dmz
- Terminal output:
(occurred immediately after entering the capture command
above - actual HyperTerm
line wraps are displayed)
Warning: using this option with a slow console connection may
result in an excessive amount of non-displayed packets
due to performance limitations.
Use ctrl-c to terminate real-time capture
1: 18:58:19.230258 150.150.1.20.3010 > 172.16.1.15.80: S
839932538:839932538(
0) win 64240 <mss 1260,nop,wscale 0,nop,nop,nop,nop>
2: 18:58:19.231891 172.16.1.15.80 > 150.150.1.20.3010: S
954995:954995(0) ack
839932539 win 8820 <mss 1460>
3: 18:58:19.232882 150.150.1.20.3010 > 172.16.1.15.80: . ack
954996 win 64240
4: 18:58:19.235400 150.150.1.20.3010 > 172.16.1.15.80: P
839932539:839932983(
444) ack 954996 win 64240
5: 18:58:19.262635 172.16.1.15.80 > 150.150.1.20.3010: P
954996:955068(72) ac
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
5
k 839932983 win 8376
6: 18:58:19.307753 150.150.1.20.3010 > 172.16.1.15.80: P
839932983:839933408(
425) ack 955068 win 64168
7: 18:58:19.314894 172.16.1.15.80 > 150.150.1.20.3010: P
955068:955372(304) a
ck 839933408 win 7951
8: 18:58:19.315382 172.16.1.15.80 > 150.150.1.20.3010: F
955372:955372(0) ack
839933408 win 7951
9: 18:58:19.316481 150.150.1.20.3010 > 172.16.1.15.80: . ack
955373 win 63864
10: 18:58:19.317015 150.150.1.20.3010 > 172.16.1.15.80: F
839933408:839933408(
0) ack 955373 win 63864
11: 18:58:19.317778 172.16.1.15.80 > 150.150.1.20.3010: . ack
839933409 win 79
51
11 packets shown.
0 packets not shown due to performance limitations.
Capture – Part II – Output to HTML File
Assuming the same access-list as shown above, if the real-time option is omitted the same text display as
shown on the previous page can be derived by
show capture WWW-TEST-DMZ
A more readable format can be derived by securely viewing the capture buffer using a browser; however, before
this can be done, the following commands must be entered into the PIX or ASA security appliance: (using the
Admin-PC in our Lab Topology as an example)
http server enable
http 10.10.10.10 255.255.255.255 inside
To view the capture file in buffer memory, execute the following.
https://10.10.10.1/capture/WWW-TEST-DMZ
This will result in the following authentication prompt; the default credentials expected would be the enable
password for the appliance (Password: field) with the User Name: blank
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
6
Screen 1
The result is shown below.
Screen 2
Note that not only is the output sorted into much more readable aligned columns, but the TCP flags are easily
seen. (S - SYN, P - PSH, F - FIN)
Capture – Part III – Output to a Windows Pcap File
If even more granular examination of the packets contained in the capture file is needed, the buffered capture
file can be copied to a TFTP Server for later viewing using the following:
copy /pcap capture:WWW-TEST-DMZ tftp://10.10.10.10/WWW.cap
After confirming your selections in the TFTP CLI dialog, the resulting file can be viewed with the default protocol
analyzer (“sniffer”) application associated with the .cap suffix. In this case, Wireshark® was used to generate
the following display.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
7
Screen 3
Capture – Part IV – Using the Packet Capture Wizard in
ASDM
In this last sub-section on the capture command, we will examine how the Adaptive Security Device Manager
(ASDM) for the ASA and PIX can be used to not only analyze packets in and out of a particular interface (as we
have done previously), but also look at the overall flow through the security appliance as a whole. In the series
of screens below, the same http packet stream is analyzed, except that the “outside” interface is a logical one
with the name outside_lab and it is part of an 802.1q trunk. Note that in step 3 we need to specify the translated address (200.200.1.15) of the DMZ-Server.
Screen 4
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
8
Screen 5
Screen 6
Note that in screen 6 above, bidirectional flows are shown for both the dmz and the outside_lab interfaces.
Secondly, for the egress interface (outside_lab) the VLAN ID is seen, as the Ethernet frames entering and leaving use 802.1q trunk encapsulation. If the “Launch Network Sniffer Application” button is clicked, the following
is seen as a pop-up: (note that the 802.1q frame decode is clearly seen in the middle window)
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
9
Screen 7
Packet-tracer
The packet-tracer command was first introduced into ASA and PIX OS 7.2 and can be executed both from the
command-line and from within ASDM. The big advantage of this powerful and relatively new tool is that most
simple, unencrypted (non-VPN) flows can be simulated without having to physically access either of the two
connection endpoints. We will show in this final section of this whitepaper several uses of packet-tracer. First,
for consistency, we will show the simulated flow of the Outside-PC connecting to the DMZ-Server with http.
Later, we will show more complex translation scenarios for both inbound and outbound access.
Below is the simulated http://200.200.1.15 from the Outside-PC as shown in ASDM.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
10
Screen 8
If each of the Phase items shown above are expanded by clicking on the “+” sign, what is obtained is virtually
identical to performing this using the CLI command
packet-tracer input outside_lab tcp 150.150.1.20 1024 200.200.1.15 80
Note that the first phase item in the GUI display above titled “ACCESS-LIST” is missing from the output below;
the CLI does not display the null MAC-address access-list as a phase.
Phase: 1
Type: FLOW-LOOKUP
Subtype: Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
11
Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
static (dmz,outside_lab) 200.200.1.15 172.16.1.15 netmask
255.255.255.255
nat-control
match ip dmz host 172.16.1.15 outside_lab any
static translation to 200.200.1.15
translate_hits = 1, untranslate_hits = 3
Additional Information:
NAT divert to egress interface dmz
Untranslate 200.200.1.15/0 to 172.16.1.15/0 using netmask
255.255.255.255
Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group OUTSIDE_IN in interface outside_lab
access-list OUTSIDE_IN extended permit tcp any host 200.200.1.15
eq www
Additional Information:
Phase: 4
Type: IP-OPTIONS
Subtype: Result: ALLOW
Config:
Additional Information:
Phase: 5
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
12
class inspection_default
inspect http
service-policy global_policy global
Additional Information:
Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
static (dmz,outside_lab) 200.200.1.15 172.16.1.15 netmask
255.255.255.255
nat-control
match ip dmz host 172.16.1.15 outside_lab any
static translation to 200.200.1.15
translate_hits = 1, untranslate_hits = 3
Additional Information:
Phase: 7
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (dmz,outside_lab) 200.200.1.15 172.16.1.15 netmask
255.255.255.255
nat-control
match ip dmz host 172.16.1.15 outside_lab any
static translation to 200.200.1.15
translate_hits = 1, untranslate_hits = 3
Additional Information:
Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 9
Type: FLOW-CREATION
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
13
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 96, packet dispatched to next module
Phase: 10
Type: ROUTE-LOOKUP
Subtype: output and adjacency
Result: ALLOW
Config:
Additional Information:
found next-hop 172.16.1.15 using egress ifc dmz
adjacency Active
next-hop mac address 0080.c882.11dc hits 4
Result:
input-interface: outside_lab
input-status: up
input-line-status: up
output-interface: dmz
output-status: up
output-line-status: up
Action: allow
Several items are worthy of note in the display of the various “Phases” on the previous page.
1) UN-NAT (Phase 2) actually does the reverse address translation (global→local).
2)NAT (Phase 6) is checking for Reverse Path Forwarding, or the spoofing of a source address associated
with a different interface other than the one on which it arrived. While this could be enabled, it has not
been here. (default = disabled)
3)NAT (Phase 7) looks at the exceeding of specified maximum or embryonic connection limits, neither of
which are specified in this example.
The following ASDM screenshot shows the power of the integrated use of this tool with the GUI display of relevant access lists; it was generated by clicking on the Show rule in Access Rules Table link: (note that the
relevant access-list entry is highlighted.)
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
14
Screen 9
Our final example for the use of packet-tracer will be for a complex set of Network Address Translation (NAT)
rules as might be encountered for an organization requiring VPN connectivity as well as general clear-text connectivity to the Internet. To visualize this scenario, the following shows a revision to the troubleshooting
scenario presented previously – namely, the addition of a remote location to which a site-to-site VPN session
might be initiated. To accommodate this, what is frequently needed is a NAT exemption rule, one that won’t
translate the 10.10.10.0/24 network-sourced packets traversing the VPN tunnel to 10.20.10.10/24. The resulting
output shows what happens when provision was not made for that exemption.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
15
Perimeter
Router
INTERNET
.1
200.200.1.0/24
DMZ Subnet
172.16.1.0/24
.2
Outside-PC
150.150.1.20
10.20.10.0/24
ASA/PIX
.1
.1
DMZ-Srv
172.16.1.15
NAT:
200.200.1.15
10.10.10.0/24
XP
Admin-PC
(Syslog Server)
10.10.10.10
Figure 2
Below is the result for the command:
packet-tracer input inside icmp 10.10.10.10 0 0 10.20.10.10
Phase: 1
Type: FLOW-LOOKUP
Subtype: Result: ALLOW
Config:
Additional Information:
Found no matching flow, creating a new flow
Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
16
in
0.0.0.0
0.0.0.0
outside
Phase: 3
Type: IP-OPTIONS
Subtype: Result: ALLOW
Config:
Additional Information:
Phase: 4
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Additional Information:
Result:
input-interface: inside
output-status: up
input-status: up
output-line-status: up
input-line-status: up
output-interface: outside
Phase: 5
Type: NAT
Subtype: host-limits
Result: ALLOW
Config:
static (inside,outside) udp interface syslog 10.10.10.10 syslog
netmask 255.255.255.255
nat-control
match udp inside host 10.10.10.10 eq 514 outside any
static translation to 200.200.1.1/514
translate_hits = 0, untranslate_hits = 0
Additional Information:
Phase: 6
Type: NAT
Subtype:
Result: DROP
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
17
Config:
nat (inside) 1 0.0.0.0 0.0.0.0
nat-control
match ip inside any outside any
dynamic translation to pool 1 (200.200.1.2 [Interface PAT])
translate_hits = 1, untranslate_hits = 0
Additional Information:
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule
As can be seen above, instead of a NAT exemption rule being matched, the 10.10.10.10 IP Address would be
translated to the ASA outside interface via Port Address Translation (PAT). As a result, if users attempted to ping
across the secure tunnel, their attempts would fail. Further VPN troubleshooting techniques will be addressed
in the second whitepaper in this series, Part II: Virtual Private Networks. Part III of this series examines Troubleshooting Intrusion Prevention Systems.
Summary
One key concept common to all the preceding scenarios is an effort to “target” the troubleshooting as much
as possible. This minimizes the extraneous output, which usually has no relevance to the problem at hand. An
added bonus is minimizing the impact on the CPU of the appliance that would be understandably burdened by
a more generalized “debug any” approach and could negatively impact throughput. We have seen the effective
use of the debug packet tool on older operating system, and newer capture and packet tracer tools, the
last of which can be used without involving the end-user.
Learn More
Learn more about how you can improve productivity, enhance efficiency, and sharpen your competitive edge.
Check out the following Global Knowledge courses:
IINS (Introduction to IOS Network Security)
SNRS (Securing Networks with Cisco® Routers and Switches)
SNAF (Securing Networks with ASA Fundamentals)
SNAA (Securing Networks with ASA Advanced)
MARS (Cisco® Monitoring Analysis and Reporting System)
CANAC (Cisco® Appliance for Network Admission Control)
For more information or to register, visit www.globalknowledge.com or call 1-800-COURSES to speak with a
sales representative. Our courses and enhanced, hands-on labs offer practical skills and tips that you can immediately put to use. Our expert instructors draw upon their experiences to help you understand key concepts and
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
18
how to apply them to your specific work situation. Choose from our more than 700 courses, delivered through
Classrooms, e-Learning, and On-site sessions, to meet your IT and management training needs.
About the Author
Douglas B. McKillip, P.E., CCIE #1851 is the president and principal consultant of Innovative Integrators Incorporated, a Delaware Corporation actually based in Delaware.
In addition to a BS and MS in Chemical Engineering from M.I.T., Doug also later obtained an M.S. from the
University of Delaware in Computer and Information Science. After 15 years of experience at DuPont and a brief
stint with the original startup company associated with the Raptor EagleTM Firewall, Doug began his now 15+
year career of teaching and consulting, specializing in Internet Security with hardware from Cisco Systems®
since 1998.
Doug’s hobbies include playing piano at his local church and physical activities such as running, cycling, swimming, golf, skiing, and windsurfing. He and his wife Karen reside in Wilmington, Delaware. Doug can be reached
at innovativeint@mindspring.com.
Copyright ©2009 Global Knowledge Training LLC. All rights reserved.
19