Troubleshooting
Tools
Troubleshooting Tools
Inspecting HTTP Headers
• In many cases it is useful to view the HTTP headers when debugging various
problems, including:
ᵒ
ᵒ
ᵒ
ᵒ
Persistence failures when using cookie persistence
Performing caching and tuning cache headers
Verifying compression is occurring
Isolating authentication issues
• Many tools that are available are very useful for this task and are easy to use:
ᵒ
ᵒ
ᵒ
ᵒ
Live HTTP Headers for Mozilla/Firefox
IE HTTP Headers for Internet Explorer
IEHTTPAnalyzer
Fiddler
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
Inspecting HTTP Headers: Live HTTP Headers
• Live HTTP Headers is available at http://livehttpheaders.mozdev.org – free tool
• Once installed, go to Tools > livehttpheaders to start capturing headers.
• Once opened, all HTTP
request and response
headers will be displayed,
as shown.
• If desired, you can also
filter what requests will be
captured.
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
Inspecting HTTP Headers: IE HTTP Headers
• IE HTTP Headers can be downloaded from http://www.bluck.info/iehttpheaders
- free tool
• After installation, go to View > Explorer Bar >
ieHTTPHeaders to display the view pane.
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting
Inspecting HTTP Headers
• IEHTTPAnalyzer - must purchase
ᵒ (found on student Desktop) or download from http://www.ieinspector.com/httpanalyzer/
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting
Inspecting HTTP Headers
• Fiddler - Freeware
ᵒ Download from http://www.fiddler2.com/fiddler2/
© 2012 Citrix | Confidential – Do Not Distribute
Tracing
Troubleshooting Tools
Packet captures can be done on the NetScaler via the “new trace” option on the
diagnostics tab, as shown
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
• The dialog to perform a trace has several
options:
ᵒ Packet size: Specify 0 to capture the entire
packet
ᵒ Duration of data per file: By default 1 hr.
ᵒ Format: Select “tcpdump” to capture in pcap
format
ᵒ Filter expression: TCPDUMP capture filter to
only capture certain packets being processed
ᵒ Capturing mode: Additional options for capturing
packets
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
Packet Traces, continued
• Once a trace is performed, the
trace will have to be downloaded
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
Packet Traces, continued
• Select where you want the trace saved and open with your favorite trace tool
© 2012 Citrix | Confidential – Do Not Distribute
Network Traffic Capture
The nstrace utility:
• Is used for packet capture on the NetScaler system
• Has files stored in /var/nstrace
• Has names nstracexx.cap (Native) or nstracexx.pcap (tcpdump)
• Has syntax nstrace.sh –sz 0
• Prefers the native format for packet capture since it captures more information
• Can view the native NetScaler and pcap format with Wireshark
© 2012 Citrix | Confidential – Do Not Distribute
Sample nstrace.sh Output
© 2012 Citrix | Confidential – Do Not Distribute
SNMP
Troubleshooting Tools
SNMP Overview
• SNMP is a standard means of receiving information from a device by either
polling variables (OIDs) or receiving alerts (traps).
• The NetScaler system provides a list of over a thousand variables in a MIB file.
• In some devices, SNMP can both read data and write configurations, but with
the NetScaler system it is read only.
• Common tools that are used to keep track and plot long-term SNMP data
include NetScaler Command Center, HP OpenView and MRTG.
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
SNMP Community Configuration
• The first step in monitoring via SNMP is to configure a community and
permission (select ALL).
System -> SNMP -> Community
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
SNMP Manager Configuration
• To limit SNMP polling to a set of IPs or networks, configure managers as well.
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting Tools
Network Monitoring with SNMP
• Some of the most commonly polled SNMP OIDs
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
HA State:
Average CPU:
Memory Utilization (%):
Server Connections:
Client Connections:
HTTP Requests:
HTTP Responses:
Interface Stats (table):
.1.3.6.1.4.1.5951.4.1.1.6.0
.1.3.6.1.4.1.5951.4.1.1.41.1.0
.1.3.6.1.4.1.5951.4.1.1.41.2.0
.1.3.6.1.4.1.5951.4.1.1.46.1.0
.1.3.6.1.4.1.5951.4.1.1.46.2.0
.1.3.6.1.4.1.5951.4.1.1.48.67.0
.1.3.6.1.4.1.5951.4.1.1.48.53.0
.1.3.6.1.4.1.5951.4.1.1.54
Since many OIDs are dynamically generated, tools such as the iReasoning
MIB browser (http://www.iReasoning.com) are handy for locating them.
© 2012 Citrix | Confidential – Do Not Distribute
Troubleshooting
Most Common Issues
• Do not rule out basic problems.
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
Is there a duplex mismatch? #1 cause of poor performance
Is a feature enabled?
Check routing, in particular when USIP is enabled.
Do not assume coders follow HTTP? Does it work as TCP?
When installing a NetScaler system, do not assume the rest of the network is correct—
it usually is broken in subtle ways.
• Divide and conquer.
ᵒ Try to break the problem down into pieces, and verify each.
Always perform a trace when you have a problem. A trace tells you what
IS happening and not what you think should be happening.
© 2012 Citrix | Confidential – Do Not Distribute
Logs
Logged Information
The following components enable recording and extracting information for
troubleshooting the NetScaler:
• nsconmsg
• newnslog
© 2012 Citrix | Confidential – Do Not Distribute
Nsconmsg Command Output
© 2012 Citrix | Confidential – Do Not Distribute
Accessing Logged Information
• The log file and time parameters should be replaced with the necessary values
to obtain desired log information
• The log file and time parameters are available to:
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
Uncompress an archived log file
Discover the time period covered by the log
View load-balancing statistics from the archived log
Extract logging information for a shorter duration
Start log process for newnslog
© 2012 Citrix | Confidential – Do Not Distribute
Displaying Console Messages
© 2012 Citrix | Confidential – Do Not Distribute
Calendar Dialog
© 2012 Citrix | Confidential – Do Not Distribute
Sample nsconmsg Output Using grep
© 2012 Citrix | Confidential – Do Not Distribute
Displaying Event Information
© 2012 Citrix | Confidential – Do Not Distribute
Displaying CPU Information
© 2012 Citrix | Confidential – Do Not Distribute
Displaying NIC Statistics
© 2012 Citrix | Confidential – Do Not Distribute
Load Balancing Display Output
© 2012 Citrix | Confidential – Do Not Distribute
Statistics
Statistics
• Monitoring statistics of the NetScaler system or a feature can be obtained by
running the stat command
• Stat feature displays summary statistics for the feature
• Additional stat parameters are:
ᵒ
ᵒ
ᵒ
ᵒ
ᵒ
detail: for detailed feature statistics
fullValues: for number and strings displayed in full form
ntimes: for number of times statistics to be displayed
logFile: for reading stats from a log file
names internal: for viewing internal counter names
© 2012 Citrix | Confidential – Do Not Distribute
Displaying Statistics in the CLI
- stat command: Provides statistics
on certain components of a
NetScaler system
- E.g.:
- stat cpu
- stat system
- stat lb vserver
© 2012 Citrix | Confidential – Do Not Distribute
NetScaler System Statistical Utility
© 2012 Citrix | Confidential – Do Not Distribute
Work better. Live better.