CH A P T E R
1
General Troubleshooting Procedures
This chapter describes general troubleshooting techniques you can use to troubleshoot the
Cisco ASR 9000 Aggregation Series Router. It includes the following sections:
•
Prerequisite Documentation for Troubleshooting, page 1-1
•
Verifying and Troubleshooting CLI Access, page 1-2
•
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package, page 1-7
•
Validating and Troubleshooting Cisco IOS XR Software Configuration, page 1-16
•
Verifying the System, page 1-26
•
Troubleshooting the Backplane Ethernet Control System, page 1-41
•
Basic Cisco IOS XR Verification and Troubleshooting Commands, page 1-46
•
Displaying ASIC Errors, page 1-54
•
Using Trace Commands, page 1-56
•
MIB Location, page 1-57
•
Gathering Information Before You Call Cisco TAC, page 1-58
Prerequisite Documentation for Troubleshooting
As a starting point for troubleshooting, we strongly recommend that you have a system of maintaining
and accessing detailed information about your network and ASR 9000 router. This should include:
•
Current documentation about the system, including chassis numbers, serial numbers, installed cards,
and location of chassis details.
•
Diagrams illustrating the connectivity of the router control plane Ethernet network.
•
Detailed documentation about the network, including the following:
– Up-to-date internetwork map that outlines the physical location of all the devices on the network
and how they are connected, as well as a logical map of interfaces, network addresses, network
numbers, subnetworks, and so on
– List of all network protocols implemented in your network; and for each of the protocols
implemented, a list of the network numbers, subnetworks, zones, areas, and so on that are
associated with them
– All points of contact to external networks
– Routing protocol for each external network connection
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-1
Chapter 1
General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
– Established baseline for your network, that is, the normal network behavior and performance at
different times of the day so that you can compare any problems with a baseline
– Name of the device that is the spanning tree root bridge for the system control plane Ethernet
network
•
Captured output of all commands
Verifying and Troubleshooting CLI Access
Ensure that the system has been booted. If the system has not booted, see Cisco IOS XR Getting Started
Guide for the Cisco ASR 9000 Aggregation Services Router for information on booting a router running
Cisco IOS XR software. The following CLI access troubleshooting information is provided:
•
General CLI Access Information, page 1-2
•
User Access Privileges, page 1-2
•
Cisco-support Task ID, page 1-3
•
CLI Access Through a Console Port, page 1-3
•
CLI Access Through a Terminal Server, page 1-3
•
CLI Access Through the Management Ethernet Interface, page 1-4
General CLI Access Information
The following CLI access information applies to a console port, terminal server, and management
Ethernet interface connections.
Once the terminal emulation software is started and you press Enter, a router prompt should appear. If
no prompt appears, verify the physical connection to the console port and press Enter again. If the
prompt still does not appear, contact Cisco Technical Support. See the “Obtaining Documentation and
Submitting a Service Request” section on page xii for Cisco Technical Support contact information.
If a prompt appears, indicating that the CLI is accessible, but your login username and password are
invalid, you are prevented from accessing the router. Verify that you have the correct username and
password. If you have the correct username and password, but are locked out of the router, you may need
to perform password recovery to access the system again. See Cisco ASR 9000 Aggregation Services
Router ROM Monitor Guide for password recovery procedures.
User Access Privileges
When you log on to the router, use a username that is associated with a valid user group that has the
authorization to execute the required commands. If you suspect user group assignment is preventing you
from using a command, contact your AAA administrator for assistance.
See Cisco ASR 9000 Aggregation Services Router System Security Command Reference and
Cisco ASR 9000 Aggregation Services Router System Security Configuration Guide for information on
users, usernames, and user groups.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-2
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
Cisco-support Task ID
Many of the troubleshooting commands can be performed only by users who are assigned to a user group
that includes the cisco-support task ID. Users without the cisco-support task ID receive a “This
command is not authorized” response if they attempt to use those commands. The cisco-support
commands are normally reserved for use by Cisco Technical Support personnel, because there is some
risk that they may cause performance or other issues.
Caution
These Cisco support commands are normally reserved for use by Cisco Technical Support personnel
only. There is some risk that they may cause performance or other issues that impact products without
proper usage, and we highly recommend that you contact Cisco Technical Support prior to using any of
these commands. See the “Obtaining Documentation and Submitting a Service Request” section on
page xii for information on contacting Cisco TAC.
CLI Access Through a Console Port
The first time a router is started, you must use a direct connection to the console port to connect to the
router and enter the initial configuration. See Cisco ASR 9000 Aggregation Services Router Router
Getting Started Guide for information on connecting to the router through a console port. When you use
a direct connection to the Console port, CLI commands are entered at a terminal or at a computer running
terminal emulation software. A direct Console port connection is useful for entering initial
configurations and performing some debugging tasks.
CLI Access Through a Terminal Server
A terminal server connection provides a way to access the Console port from a remote location. A
terminal server connection is used when you need to perform tasks that require Console port access from
a remote location.
Connecting to a router through a terminal server is similar to directly connecting through the Console
port. For both connection types, the physical connection takes place through the Console port. The
difference is that the terminal server connects directly to the Console port, and you must use a Telnet
session to establish communications through the terminal server to the router.
If you are unable to access the CLI through a terminal server, perform the following procedure.
Step 1
Disable flow control (XON/XOFF) on the Terminal Server.
Step 2
Disable local echo mode on the Terminal Server.
Step 3
Verify the router name configured using the hostname command.
Step 4
Check whether the port address is configured correctly.
Step 5
Verify whether the address (interface) used for the reverse Telnet is up/up. The output of the show
interfaces brief command provides this information. Cisco recommends you to use loopbacks because
they are always up.
Step 6
Ensure that you have the correct type of cabling. For example, you must not use a crossover cable to
extend the length.
Step 7
Establish a Telnet connection to the IP address port to test direct connectivity. You must Telnet from both
an external device and the terminal server. For example, telnet 172.21.1.1 2003.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-3
Chapter 1
General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
Step 8
Ensure that you have the transport input telnet command under the line for the target device. The target
device is the device that is connected to the terminal server.
Step 9
Use a PC/dumb terminal to connect directly to the console of the target router. The target router is the
device connected to the terminal server. This step helps you identify the presence of a port issue.
Step 10
If you are disconnected, check timeouts. You can remove or adjust timeouts.
Note
If you encounter authentication failures, remember that the terminal server performs the first
authentication (if configured), while the device to which you try to connect performs the second
authentication (if configured). Verify whether AAA is configured correctly on both the terminal server
and the connecting device.
Step 11
Contact Cisco Technical Support. See the “Obtaining Documentation and Submitting a Service Request”
section on page xii for Cisco Technical Support contact information.
CLI Access Through the Management Ethernet Interface
The Management Ethernet interface allows you to manage the router using a network connection. Before
you can use the Management Ethernet interface, the interface must be configured. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on
configuring the interface.
Once configured, the network connection takes place between client software on a workstation computer
and a server process within the router. The type of client software you use depends on the server process
you use. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for
information on the client and server services supported by the Cisco IOS XR software.
If you are unable to access the CLI through a management Ethernet interface, perform the following
procedure.
SUMMARY STEPS
1.
show interface MgmtEth interface-instance
2.
show arp MgmtEth interface-instance
3.
show ipv4 interface type instance
4.
ping
5.
Contact Cisco Technical Support if the problem is not resolved
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-4
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
DETAILED STEPS
Step 1
Command or Action
Purpose
show interfaces MgmtEth interface-instance
Displays statistics for all interfaces configured on the
router.
Example:
Check the following:
RP/0/RSP0/CPU0:router#
0/RSP0/CPU0/0
show interfaces MgmtEth
•
MgmtEth interface is up
•
Line protocol (state of the Layer 2 line protocol) is up
•
Number of input and output errors
If an interface is administratively down, use the no
shutdown command to enable the interface.
If an interface is down (operationally down), input or output
errors are not within an acceptable range, the management
Ethernet interface is not enabled when the no shutdown
command is used, or the line protocol is down, see
Chapter 2, “Verifying and Troubleshooting Interface
Status,” for detailed information on troubleshooting
interfaces.
If the interface is up and the input and output errors are
within an acceptable range, proceed to Step 2.
Step 2
show arp MgmtEth interface-instance
Displays the Address Resolution Protocol (ARP) table for
the management Ethernet interface.
Example:
Ensure that the expected ARP entries exist for the
management Ethernet interface.
RP/0/RSP0/CPU0:router#
0/RSP0/CPU0/0
show arp MgmtEth
If the expected ARP entries exist, proceed to Step 3.
If the expected ARP entries do not exist, verify the physical
layer Ethernet interface connectivity. Use the show arp
trace command to display the ARP entries in the buffer. See
the Chapter 2, “Verifying and Troubleshooting Interface
Status,” for more information on troubleshooting interfaces.
Step 3
show ipv4 interface type instance
Displays the usability status of interfaces configured for
IPv4.
Example:
If the interface is in the expected state, proceed to Step 4.
RP/0/RSP0/CPU0:router#
MgmtEth 0/RSP0/CPU0/0
show ipv4 interface
If the status of the interface is not as expected, see
Chapter 2, “Verifying and Troubleshooting Interface
Status,” for more information on troubleshooting interfaces.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-5
Chapter 1
General Troubleshooting Procedures
Verifying and Troubleshooting CLI Access
Step 4
Command or Action
Purpose
ping
Checks host reachability and network connectivity on the IP
network.
Note
Example:
RP/0/RSP0/CPU0:router#
ping
Enter a specific IP address or follow the prompts to
send the ping message to the target address.
If no problems are detected, proceed to Step 5.
Step 5
Contact Cisco Technical Support.
If the problem is not resolved, contact Cisco Technical
Support. For Cisco Technical Support contact information,
see the “Obtaining Documentation and Submitting a
Service Request” section on page xii.
Examples
The output from the show interfaces MgmtEth command displays the status of the management
Ethernet interface. In the following example, the management Ethernet interface is up, and there are 20
input errors and 8 output errors.
RP/0/RSP0/CPU0:router# show interface MgmtEth 0/RSP0/CPU0/0
Tue Sep 14 14:21:07.496 DST
MgmtEth0/RSP0/CPU0/0 is up, line protocol is up
Interface state transitions: 1
Hardware is Management Ethernet, address is 001b.53ff.4a62 (bia 001b.53ff.4a62)
Description: Connected to Lab LAN
Internet address is 172.29.52.137/24
MTU 1514 bytes, BW 100000 Kbit (Max: 100000 Kbit)
reliability 73/255, txload 0/255, rxload 0/255
Encapsulation ARPA,
Half-duplex, 100Mb/s, THD, link type is autonegotiation
output flow control is off, input flow control is off
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters never
5 minute input rate 2000 bits/sec, 3 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
373082 packets input, 51028824 bytes, 239105 total input drops
62028 drops for unrecognized upper-level protocol
Received 2601 broadcast packets, 194653 multicast packets
10 runts, 0 giants, 0 throttles, 0 parity
20 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
45232 packets output, 3042775 bytes, 0 total output drops
Output 24 broadcast packets, 0 multicast packets
8 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
1 carrier transitions
The output from the show arp MgmtEth 0/RSP0/CPU0/0 command displays the ARP table for the
management Ethernet interface. Use the output from this command to verify that there are dynamic ARP
addresses in the table and that ARP is functioning over the interface. The output shows that ARP is
functioning over the management Ethernet interface 0/RSP0/CPU0/0.
RP/0/RSP0/CPU0:router# show arp MgmtEth 0/RSP0/CPU0/0
Tue Sep 14 14:24:03.962 DST
------------------------------------------------------------------------------0/RSP0/CPU0
------------------------------------------------------------------------------Address
Age
Hardware Addr
State
Type Interface
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-6
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
172.29.52.1
172.29.52.13
172.29.52.21
172.29.52.27
172.29.52.28
172.29.52.32
172.29.52.36
172.29.52.46
172.29.52.47
172.29.52.60
172.29.52.69
172.29.52.70
172.29.52.71
172.29.52.75
172.29.52.76
172.29.52.81
172.29.52.83
172.29.52.127
172.29.52.134
172.29.52.135
172.29.52.136
172.29.52.137
172.29.52.138
172.29.52.161
172.29.52.171
172.29.52.172
172.29.52.173
172.29.52.180
172.29.52.217
172.29.52.226
172.29.52.243
01:44:00
01:16:59
01:40:25
02:18:16
02:05:29
01:42:16
02:39:34
01:36:50
01:36:39
01:35:20
00:00:00
01:23:38
02:00:47
01:44:59
01:41:10
00:15:35
00:21:05
01:43:38
01:15:53
01:01:46
00:43:39
01:32:12
00:16:12
00:17:47
01:57:04
01:26:50
01:16:21
01:28:32
01:48:25
0000.0c07.ac01
0010.79e9.6038
0022.0d5a.a6c4
0012.7fd6.ba08
0012.7fd6.ba09
0022.0d26.3bc5
0026.527c.5341
0012.7fd6.b9aa
0012.7fd6.b9ab
0003.a099.8000
001b.7852.4bd1
0011.93ef.e8e6
0011.93ef.e8fe
5a59.0000.0202
0011.93ef.e8ea
001a.6c40.d89c
001a.6c40.d89c
0013.c4cb.a200
001f.6c26.7fc0
001f.6c25.c480
0022.5560.8840
001b.53ff.4a62
001b.53ff.4a62
0019.aaa3.3d48
001c.5838.5b28
001c.5838.5b29
0015.c75f.09f8
0015.c75f.0800
0019.aaa3.b5ff
0010.f60e.8400
001e.79c1.e0c1
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Interface
Interface
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
Dynamic
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
ARPA
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/0
The ping command checks to see if the neighbor is reachable.
RP/0/RSP0/CPU0:router# ping 172.16.52.28 count 10
Tue Sep 14 14:36:52.441 DST
Type escape sequence to abort.
Sending 10, 100-byte ICMP Echos to 172.16.52.28, timeout is 2 seconds:
!!!!!!!!!!
Success rate is 100 percent (10/10), round-trip min/avg/max = 1/1/2 ms
Validating and Troubleshooting Installation of the Cisco IOS XR
Software Package
The Cisco IOS XR software is divided into software packages allowing you to select which features run
on your router. Each package contains the components to perform a specific set of router functions, such
as routing, security, or Modular Services Card (MSC) support. Bundles are groups of packages that can
be downloaded as a set. For example, the Unicast Routing Core Bundle provides six packages for use on
every router.
This section provides information on how to validate and troubleshoot the Cisco IOS XR software
package installation. The following sections are provided:
•
Verifying the Software Version, page 1-8
•
Validating the Installation, page 1-10
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-7
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Verifying the Software Version
To verify the Cisco IOS XR software version, perform the following procedure.
SUMMARY STEPS
1.
show version
2.
show install
DETAILED STEPS
Step 1
Command or Action
Purpose
show version
Displays a variety of system information, including
hardware and software version, router uptime, boot settings
(configuration register), and active software.
Example:
RP/0/RSP0/CPU0:router#
show version
Determine if all expected packages are installed and the
current software versions are the expected versions.
If the expected packages are not installed or are not the
expected version, install the correct package. See
Cisco ASR 9000 Aggregation Series Router Getting Started
Guide for information on installing and upgrading
Cisco IOS XR software packages.
Step 2
Displays a list of all installed and active packages on each
node.
show install
Example:
RP/0/RSP0/CPU0:router#
show install
Determine if the expected packages are installed on each
node.
If the software or active package versions are not as
expected for a node, the package is not compatible with the
node for which it is being activated, or the package being
activated is not compatible with the current active software
set, install the correct software or package on the node. See
Cisco ASR 9000 Aggregation Series Router Getting Started
Guide for information on installing and upgrading
Cisco IOS XR software packages.
The following example shows that the Cisco IOS XR software and active packages are version 4.0.0.
RP/0/RSP0/CPU0:router# show version
Cisco IOS XR Software, Version 4.0.0
Copyright (c) 2010 by cisco Systems, Inc.
ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],
router uptime is 1 day, 18 hours, 34 minutes
System image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"
cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory.
MPC8641D processor at 1333MHz, Revision 2.2
2 Management Ethernet
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-8
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
12 DWDM controller(s)
12 TenGigE
40 GigabitEthernet
2 SONET/SDH
2 Packet over SONET/SDH
219k bytes of non-volatile configuration memory.
975M bytes of compact flash card.
33994M bytes of hard disk.
1605616k bytes of disk0: (Sector size 512 bytes).
1605616k bytes of disk1: (Sector size 512 bytes).
Configuration register on node 0/RSP0/CPU0 is 0x0
Boot device on node 0/RSP0/CPU0 is disk0:
Package active on node 0/RSP0/CPU0:
asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0
Built on Wed Sep 8 16:17:30 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0
Built on Wed Sep 8 16:12:40 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0
Built on Wed Sep 8 16:13:28 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0
Built on Wed Sep 8 16:16:57 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0
Built on Wed Sep 8 16:13:26 DST 2010
--More-.
.
.
The following example shows that the Cisco IOS XR software and active packages are version 4.0.0. If
there is an expected package missing or an active package is not an expected package, install and activate
the missing package or upgrade the unexpected package to the appropriate package. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for details on installing,
activating, and upgrading software packages.
RP/0/RSP0/CPU0:router# show install
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-doc-p-4.0.0
disk0:asr9k-k9sec-p-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mgbl-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/1/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-9
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/2/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/4/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/6/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Validating the Installation
Validate the Cisco IOS XR software package installation to ensure the packages were installed correctly.
The following commands are used to validate the currently installed software packages:
•
install verify Command, page 1-10
•
show install active Command, page 1-12
•
show install committed Command, page 1-14
install verify Command
Use the install verify command to verify the consistency of a previously installed software set with the
package file from which it originated.
This command can be used as a debugging tool to verify the validity of the files that constitute the
packages to determine if there are any corrupted files. The command is also used to check that the install
infrastructure is up and running and to determine if all files are expected. If there are corrupted files, see
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on
deactivating and removing software packages and adding and activating software packages.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-10
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Note
The install verify command can take up to two minutes per package to process.
Note
The install verify command ignores secure domain router (SDR) boundaries and performs the operation
in global scope.
The following example shows the output of the install verify command. The output is used to verify the
consistency of a previously installed software set with the package file from which it originated.
RP/0/RSP0/CPU0:router(admin)# install verify
Sat Sep 25 08:18:14.077 DST
Install operation 3 '(admin) install verify packages' started by user_A
'dwolman-r' via CLI at 08:18:14 DST Sat Sep 25 2010.
The install operation will continue asynchronously.
RP/0/RSP0/CPU0:router(admin)#Info:
This operation can take up to 2 minutes per package
being verified.
Info:
Please be patient.
Info:
0/0/CPU0 [LC] [SDR: Owner]
Info:
meta-data: [SUCCESS] Verification Successful.
Info:
/install/asr9k-optics-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/asr9k-scfclient-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-video-adv-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/iosxr-mcast-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-routing-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-infra-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-diags-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.
Info:
0/6/CPU0 [LC] [SDR: Owner]
Info:
meta-data: [SUCCESS] Verification Successful.
Info:
/install/asr9k-optics-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-cpp-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/asr9k-scfclient-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-video-adv-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-mpls-4.0.0: [SUCCESS] Verification Successful.
Info:
/install/iosxr-mcast-4.0.0: [SUCCESS] Verification
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-11
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Info:
Successful.
Info:
/install/iosxr-routing-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-infra-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-fwding-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/iosxr-diags-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-adv-video-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-diags-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-mcast-supp-4.0.0: [SUCCESS] Verification
Info:
Successful.
Info:
/install/asr9k-base-4.0.0: [SUCCESS] Verification Successful.
Info:
0/5/CPU0 [LC] [SDR: Owner]
.
.
.
Info:
Verification Summary:
Info:
0/0/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/6/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/5/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/7/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/1/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/4/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/2/CPU0: SUCCESSFUL. No anomalies found.
Info:
0/RSP0/CPU0: SUCCESSFUL. No anomalies found.
Info:
The system needs no repair.
Install operation 3 completed successfully at 08:19:48 DST Sat Sep 25 2010.
show install active Command
Use the show install active command to display active software packages. Verify that the command
output matches the output of the show install committed command. If the output does not match, when
you reload the router, the software displayed in the show install committed command output is the
software that will be loaded. For example, the following output shows two different software package
versions, one is the active version and the other is the committed version, so when the router reloads, the
3.9.1 version will be loaded even though 4.0.0 is the currently active version on 0/RSP0/CPU0.
RP/0/RSP0/CPU0:router(admin)# show install active location 0/RSP0/cpu0
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm << 4.0.0 is active, not committed
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-doc-p-4.0.0
disk0:asr9k-k9sec-p-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mgbl-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
RP/0/RSP0/CPU0:router(admin)# show install committed location 0/RSP0/cpu0
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-3.9.1/mbiasr9k-rp.vm<< 3.9.1 is committed
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-12
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Committed Packages:
disk0:asr9k-mini-p-3.9.1
disk0:asr9k-optic-3.9.1
disk0:asr9k-doc-p-3.9.1
disk0:asr9k-k9sec-p-3.9.1
disk0:asr9k-video-p-3.9.1
disk0:asr9k-mpls-p-3.9.1
disk0:asr9k-mgbl-p-3.9.1
disk0:asr9k-mcast-p-3.9.1
If the expected active software packages are not displayed, install the packages (if required) and activate
the packages. See Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for
information on installing and activating Cisco IOS XR software packages. The following example
output shows the active packages for all cards in a router. The output displays the disk on which each
package is located.
RP/0/RSP0/CPU0:router# show install active
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-doc-p-4.0.0
disk0:asr9k-k9sec-p-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mgbl-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/0/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/1/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/2/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/4/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-13
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/5/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/6/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/7/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Active Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
The output shows the name of the disk on which the packages are located. In the above example, the
active packages for each node are on disk0, and for all nodes, the composite package asr9k-os-mbi-4.0.0
is active. Additional packages shown are optional packages that have been activated after the initial
loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing Core Bundle.
show install committed Command
Use the show install committed command to display committed software packages. The committed
software packages are the software packages that will be booted on a router reload.
Committed packages are the packages that are persistent across router reloads. If you install and activate
a package, it remains active until the next router reload. If you commit a package set, all packages in that
set remain active across router reloads until the package set is replaced with another committed package
set. The show install committed command is useful to ensure software is installed and committed after
a router reload. If the expected software is not installed and committed, see
Cisco ASR 9000 Aggregation Services Router Getting Started Guide for information on installing and
committing Cisco IOS XR software packages.
The following command output shows the committed software packages on all cards in the router. The
output displays the disk on which each package is located.
RP/0/RSP0/CPU0:router# show install committed
Node 0/RSP0/CPU0 [RP] [SDR: Owner]
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-14
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Installation of the Cisco IOS XR Software Package
Boot Device: disk0:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-doc-p-4.0.0
disk0:asr9k-k9sec-p-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mgbl-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/0/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/1/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/2/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/4/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/5/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/6/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-15
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
Node 0/7/CPU0 [LC] [SDR: Owner]
Boot Device: mem:
Boot Image: /disk0/asr9k-os-mbi-4.0.0/lc/mbiasr9k-lc.vm
Committed Packages:
disk0:asr9k-mini-p-4.0.0
disk0:asr9k-optic-4.0.0
disk0:asr9k-video-p-4.0.0
disk0:asr9k-mpls-p-4.0.0
disk0:asr9k-mcast-p-4.0.0
The output shows the name of the disk on which the packages are located. In the above example, the
committed packages for each node are on disk0, and for all nodes, the composite package
asr9k-os-mbi-4.0.0 is committed. Additional packages shown are optional packages that have been
committed after the initial loading of the Cisco ASR 9000 Aggregation Series Router Unicast Routing
Core Bundle.
Validating and Troubleshooting Cisco IOS XR Software
Configuration
Validating the Cisco IOS XR software configuration includes collecting configuration information on
the router to determine configuration changes and verifying the current running configuration. When a
configuration fails during a commit, the failed configuration can be viewed to help determine why the
configuration was not committed.
The following sections are provided:
•
Local and Global Configurations, page 1-16
•
Collecting Configuration Information, page 1-19
•
Verifying the Running Configuration, page 1-20
•
Using the show configuration failed Command, page 1-24
Local and Global Configurations
To troubleshoot configurations, you need to determine whether the problem is in the local configuration
or the shared (global) configuration.
•
The local configuration is specific to the individual LC or RP to which it belongs. Every LC and RP
has a data store containing the local data for that node, including configuration and operational data
for the local interfaces. An example of a local configuration is the port designations on a particular
LC.
•
The shared (global) configuration applies to the entire router, and is shared with all of the LCs and
RPs. An example of a shared configuration is the routing protocol parameters.
To view the local configuration, use the show running-config interface * command. The output
displays all the configured interfaces on the node.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-16
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
RP/0/RSP0/CPU0:router# show running-config interface *
interface Bundle-Ether16
description Connect to router-S Port-Ch 16
mtu 9216
bundle maximum-active links 1
!
interface Bundle-Ether16.160 l2transport
description Connect to router-S Port-Ch 16 Service Instance 160
encapsulation dot1q 160
!
.
.
.
!
interface Loopback0
ipv4 address 10.144.144.144 255.255.255.255
!
interface tunnel-ip10
!
interface tunnel-te44190
description Primary GE Tunnel from router-S to router-T
ipv4 unnumbered Loopback0
priority 0 0
autoroute announce
signalled-bandwidth 100000
destination 10.19.19.19
fast-reroute
record-route
path-option 1 explicit name Primary_GE_Path_to_router-T ospf 100 area 0
!
.
.
.
interface MgmtEth0/RSP0/CPU0/0
description Connected to LAN
ipv4 address 172.29.52.137 255.255.255.0
!
interface MgmtEth0/RSP0/CPU0/1
shutdown
!
interface GigabitEthernet0/0/0/0
shutdown
!
interface GigabitEthernet0/0/0/1
shutdown
!
interface GigabitEthernet0/0/0/2
shutdown
!
.
.
.
interface TenGigE0/7/0/1
shutdown
!
interface TenGigE0/7/0/2
shutdown
!
interface TenGigE0/7/0/3
shutdown
!
interface POS0/2/0/0
description Connected to PE_router-2 POS 0/2/0/0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-17
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
!
interface POS0/2/0/1
description Connected to PE_router-3 POS 0/2/0/1
!
controller SONET0/2/0/0
clock source internal
!
controller SONET0/2/0/1
clock source internal
!
Use the show sysdb trace commands to display the contents of the system database after a configuration
change. The trace information includes a history of any changes to the running configuration. You can
specify either a local node or the shared plane.
The following example output shows the contents of the local database, that is, for a specific location
(node):
RP/0/RSP0/CPU0:router# show sysdb trace verification location 0/5/cpu0 reverse
.
.
.
Timestamp
nid
reqid
jid
tid reg_hndl connid
action
path
432 wrapping entries (8192 possible, 158 filtered, 590 total)
Sep 23 04:35:39.969
0/RSP0/CPU0 8168
354
1
94
4483
reply
'--'
Sep 23 04:35:39.960
0/RSP0/CPU0 8168
354
1
94
4505
called
'cfg/if/act/tunnel-ip10/v'
Sep 23 04:35:39.960
0/RSP0/CPU0 8168
354
1
94
4505
reply: accept
'--'
Sep 23 04:35:39.685
0/RSP0/CPU0 8168
354
1
94
4505
called
'cfg/if/act/tunnel-ip10/v'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/mtu/tunnel-ip'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/im/bw'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/keepalive'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/dfbit_disable'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/ttl'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/tos'
Sep 23 04:35:39.678
0/RSP0/CPU0 0
354
1
94
4505
register
'cfg/if/act/tunnel-ip[0-9]*/tunl_gre/mode'
.
.
.
apply
Apply
verify
Verify
The following example output shows the contents of the shared database, that is, the configuration data
that is shared with all LC and RP in the router:
RP/0/RSP0/CPU0:router# show sysdb trace verification shared-plane reverse
Config Shared Server
====================
Timestamp
nid
reqid
jid
tid reg_hndl connid
path
2259 wrapping entries (4096 possible, 0 filtered, 2259 total)
action
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-18
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Sep 23 19:34:40.202
unregister
Sep 23 19:34:40.197
unregister
Sep 23 19:34:40.196
unregister
Sep 23 19:14:45.076
register
Sep 23 19:14:41.679
register
Sep 23 19:14:41.593
register
Sep 23 19:12:36.472
unregister
Sep 23 19:12:36.471
unregister
Sep 23 19:12:36.470
unregister
Sep 23 19:07:56.914
register
.
.
.
0/3/CPU0
0
241
8
384
1430
'from-'
0/3/CPU0
0
241
15
385
1434
'from-'
0/3/CPU0
0
163
1
386
1440
'from-'
0/3/CPU0
0
163
1
386
1440
'cfg/gl/ipv4/cef/hardware/forwarding/update/synchronous'
0/3/CPU0
0
241
15
385
1434
'cfg/gl/dbgtrace/node/831/'
0/3/CPU0
0
241
8
384
1430
'cfg/gl/dbgtrace/node/831/'
0/3/CPU0
0
241
8
381
1375
'from-'
0/3/CPU0
0
241
15
382
1378
'from-'
0/3/CPU0
0
163
1
383
1383
'from-'
0/3/CPU0
0
163
1
383
1383
'cfg/gl/ipv4/cef/hardware/forwarding/update/synchronous'
The show processes location node-id | include sysdb command displays all active database processes
for a specified node.
RP/0/RSP0/CPU0:router# show process location 0/1/CPU0 | include sysdb
Thu Nov 4 14:06:30.191 DST
279
1
0
56K 10 Sigwaitinfo 739:28:22:0145
0:00:00:0057 sysdb_svr_local
279
2
1
56K 10 Receive
0:00:00:0779
0:00:02:0459 sysdb_svr_local
279
3
1
56K 10 Receive
0:03:34:0474
0:00:03:0285 sysdb_svr_local
279
4
1
56K 10 Receive
0:05:03:0006
0:00:02:0368 sysdb_svr_local
277
1
0
64K 10 Sigwaitinfo 739:28:21:0305
0:00:00:0046 sysdb_mc
277
2
0
64K 10 Receive
739:28:21:0274
0:00:00:0003 sysdb_mc
277
3
1
64K 10 Receive
166:59:14:0698
0:00:00:0038 sysdb_mc
277
4
1
64K 10 Receive
0:01:49:0941
0:00:00:0106 sysdb_mc
277
6
1
64K 10 Receive
739:15:22:0734
0:00:00:0058 sysdb_mc
See Chapter 1, “General Troubleshooting Procedures” for additional information on troubleshooting
processes.
Collecting Configuration Information
Collecting configuration information allows you to determine if changes to the system have occurred. It
also allows you to determine if these changes could impact the system. The following commands allow
you to determine if there was an unknown commit, if there was a commit that overwrote a previous
configuration, or there are configuration changes that should be removed from the running configuration.
•
show configuration commit changes {[since] commit-id | last number-of-commits} [diff]—the
command output displays changes made to the running configuration by previous configuration
commits.
RP/0/RSP0/CPU0:router# show configuration commit changes since 1000000319
Wed May 17 09:30:27.877 UTC
Building configuration...
no logging console
no domain ipv4 host ce1
no domain ipv4 host ce2
domain ipv4 host ce6 172.29.52.73
domain ipv4 host ce7 172.29.52.78
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-19
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
no domain ipv4 host pe1
no domain ipv4 host pe2
domain ipv4 host pe6 172.29.52.128
domain ipv4 host pe7 172.29.52.182
interface GigabitEthernet0/1/5/1
no negotiation
!
end
•
show configuration commit list [number-of-commits] [detail]—the command output displays a list
of the commit IDs (up to 100) available for rollback.
RP/0/RSP0/CPU0:router# show configuration commit list
Wed May 17 09:31:21.727 UTC
SNo. Label/ID
User
Line
~~~~ ~~~~~~~~
~~~~
~~~~
1
1000000324 userA
vty0
2
1000000323 userA
vty0
3
1000000322 userB
vty0
4
1000000321 userC
vty2
5
1000000320 userA
vty2
6
1000000319 userB
vty2
7
1000000318 userB
vty2
8
1000000317 userB
vty2
9
1000000316 userC
vty2
10
1000000315 userC
vty2
11
1000000314 userA
vty2
12
1000000313 userA
vty2
13
1000000312 userD
con0_RSP0_C
•
Client
~~~~~~
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
CLI
Time Stamp
~~~~~~~~~~
16:50:33 UTC Wed May 10 2006
16:49:51 UTC Wed May 10 2006
16:48:05 UTC Wed May 10 2006
19:11:26 UTC Wed May 03 2006
19:10:45 UTC Wed May 03 2006
18:03:01 UTC Wed May 03 2006
18:02:43 UTC Wed May 03 2006
18:02:38 UTC Wed May 03 2006
17:59:16 UTC Wed May 03 2006
17:46:38 UTC Wed May 03 2006
15:40:04 UTC Wed May 03 2006
13:05:09 UTC Wed May 03 2006
13:49:31 UTC Mon May 01 2006
commit confirmed minutes (executed from config mode)—This command commits the
configuration on a trial basis for a minimum of 30 seconds and a maximum of 300 seconds (5
minutes). During the trial configuration period, enter commit to confirm the configuration. If
commit is not entered, then the system will revert to the previous configuration when the trial time
period expires.
Verifying the Running Configuration
To verify the running configuration, perform the following procedure.
SUMMARY STEPS
1.
configure
2.
show running-config
3.
describe hostname hostname
4.
end
5.
show sysdb trace verification shared-plane | include path
6.
show sysdb trace verification location node-id
7.
show cfgmgr trace
8.
show configuration history commit
9.
show configuration commit changes {last | since | commit-id}
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-20
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
10. show config failed startup
11. cfs check
DETAILED STEPS
Step 1
Command or Action
Purpose
configure
Enters global configuration mode.
Example:
RP/0/RSP0/CPU0:router#
Step 2
configure
Displays the contents of the running configuration.
show running-config
Verify that the running configuration is as expected.
Example:
RP/0/RSP0/CPU0:router(config)# show
running-config
Step 3
Determines the path.
describe hostname hostname
Example:
RP/0/RSP0/CPU0:router(config)# describe
hostname router_A
Step 4
Saves configuration changes.
end
•
Example:
RP/0/RSP0/CPU0:router(config)# end
When you issue the end command, the system prompts
you to commit changes:
Uncommitted changes found, commit them before
exiting(yes/no/cancel)?
[cancel]:
– Entering yes saves configuration changes to the
running configuration file, exits the configuration
session, and returns the router to EXEC mode.
– Entering no exits the configuration session and
returns the router to EXEC mode without
committing the configuration changes.
– Entering cancel leaves the router in the current
configuration session without exiting or
committing the configuration changes.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-21
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Step 5
Command or Action
Purpose
show sysdb trace verification shared-plane |
include path
Displays details of recent verification sysDB transactions
and changes on the shared plane allowing you to verify
whether the configuration was verified correctly.
Example:
Specifying the path filters the data to display only the
sysDB path for the router.
RP/0/RSP0/CPU0:router# show sysdb trace
verification shared-plane | include
gl/a/hostname
Step 6
show sysdb trace verification location node-id
Displays details of recent verification sysDB transactions
and changes on local plane configurations.
Example:
Verify that changes to the SysDB were verified and
accepted.
RP/0/RSP0/CPU0:router# show sysdb trace
verification location 0/3/CPU0
Step 7
Verify that changes to the SysDB were verified and
accepted.
Displays cfgmgr trace information.
show cfgmgr trace
Example:
RP/0/RSP0/CPU0:router#
Step 8
show cfgmgr trace
show configuration history commit
Displays a list of historical changes to the configuration.
Verify that the timeline of changes is as expected.
Example:
RP/0/RSP0/CPU0:router#
history commit
Step 9
show configuration
show configuration commit changes {last | since
| commit-id}
Displays detailed committed configuration history
information.
Verify that the history information is as expected.
Example:
RP/0/RSP0/CPU0:router# show configuration
commit changes last 15
Step 10
show configuration failed startup
Displays information on any configurations that failed
during startup.
Example:
RP/0/RSP0/CPU0:router#
failed startup
Step 11
show configuration
Checks the current configuration to see if there are any
missing configurations.
cfs check
Example:
RP/0/RSP0/CPU0:router#
cfs check
Examples
The following example shows the output of the show running-config command:
RP/0/RSP0/CPU0:router# show running-config
Building configuration...
!! IOS XR Configuration 4.0.1.10I
!! Last configuration change at Thu Sep 23 04:35:38 2010 by user_A
!
hostname router
.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-22
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
.
.
logging suppress duplicates
telnet vrf default ipv4 server max-servers 100
domain name cisco.com
domain lookup disable
taskgroup default
!
.
.
.
interface preconfigure GigabitEthernet0/3/0/7
shutdown
!
interface preconfigure GigabitEthernet0/3/0/8
shutdown
!
interface preconfigure GigabitEthernet0/3/0/9
shutdown
!
interface preconfigure GigabitEthernet0/3/0/10
shutdown
!
.
.
.
The output is used to determine if the configuration is as expected.
In the following example, the path to SysDB where the configuration is stored in the database is
displayed.
RP/0/RSP0/CPU0:router(config)# describe hostname router
Package:
iosxr-infra
iosxr-infra V4.0.0 IOS-XR Infra Package Definition
Vendor : Cisco Systems
Desc
: IOS-XR Infra Package Definition
Build : Built on Wed Sep 8 16:07:48 DST 2010
Source : By sc-g-01 in /auto/ioxbuild8/production/4.0.0/asr9k/workspace for pie
Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE,
NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500
Restart information:
Default:
parallel impacted processes restart
Size Compressed/Uncompressed: 38MB/85MB (44%)
Component:
shellutil V[ci-401/7]
Common shell utility applications
User needs ALL of the following taskids:
host-services (READ) or root-lr (READ WRITE)
It will take the following actions:
Create/Set the configuration item:
Path: gl/a/hostname
Value: router
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-23
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
Using the show configuration failed Command
Use the show configuration failed command to browse a failed configuration. The configuration can be
classified as failed during startup or during a configuration commit.
•
Startup Failed Configuration, page 1-24
•
Commit Configuration Failed, page 1-25
Startup Failed Configuration
A configuration can be classified as failed during startup for three reasons:
•
Syntax errors
Syntax errors are generated by the parser and usually indicate that there is an incompatibility with
the CLI commands. Correct the syntax errors and reapply the configuration. A syntax error can be
an invalid CLI entry or a CLI syntax change. See the “Obtaining Documentation and Submitting a
Service Request” section on page xii in the Preface for information on obtaining
Cisco IOS XR software CLI documentation.
•
Semantic errors
Semantic errors are generated by the backend components when the configuration is being restored
by the configuration manager during startup of the router. Semantic errors include logical problems
(invalid logic).
•
Apply errors
Apply errors are generated when a configuration has been successfully verified and accepted as part
of running configuration but the backend component is not able to update its operational state. The
configuration shows both as the running configuration (since it was correctly verified) and as a failed
configuration because of the backend operational error. To find the component apply owner, use the
describe on the CLI that failed to be applied.
Note
You may browse startup failed configurations for up to the previous four router reloads.
Use the show configuration failed startup command and the load configuration failed startup
command to browse and reapply any failed configuration. The load configuration failed startup
command can be used in configuration mode to load the failed startup configuration into the target
configuration session, then the configuration can be modified and committed. See
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide for information on
committing a configuration.
RP/0/RSP0/CPU0:router# show configuration failed startup
!! CONFIGURATION FAILED DUE TO SYNTAX/AUTHORIZATION ERRORS
telnet vrf default ipv4
server max-servers 5 interface POS0/7/0/3 router static
address-family ipv4 unicast
0.0.0.0/0 172.18.189.1
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS
router bgp 217
!!% Process did not respond to sysmgr !
RP/0/RSP0/CPU0:router#
RP/0/RSP0/CPU0:router# config
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-24
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Validating and Troubleshooting Cisco IOS XR Software Configuration
RP/0/RSP0/CPU0:router(config)# load config failed startup noerror
Loading. 263 bytes parsed in 1 sec (259)bytes/sec
RP/0/RSP0/CPU0:mike3(config-bgp)#show configuration
Building configuration...
telnet vrf default ipv4 server max-servers 5 router static
address-family ipv4 unicast
0.0.0.0/0 172.18.189.1
!
!
router bgp 217
!
end
The failed configuration is loaded into the target configuration, minus the errors that caused the startup
configuration to fail.
RP/0/RSP0/CPU0:router(config-bgp)# commit
Use the show configuration failed command to display failed items in the last configuration commit,
including reasons for the error.
In any mode, the configuration failures from the most recent commit operation are displayed.
The show configuration failed command can be used in EXEC mode and configuration mode. The
command is used in EXEC mode when the configuration does not load during startup. The command is
used in configuration mode to display information when a commit fails.
The following example shows the show configuration failed command.
RP/0/RSP0/CPU0:router(config)# interface pos 0/6/0/4
RP/0/RSP0/CPU0:router(config-if)# no vrf
RP/0/RSP0/CPU0:router(config-if)# commit
% Failed to commit one or more configuration items during an atomic operation, no changes
have been made. Please use 'show configuration failed' to view the errors
RP/0/RSP0/CPU0:router(config-if)# exit
RP/0/RSP0/CPU0:router(config)# show configuration failed
Wed May 2 13:14:08.426 EST EDT
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS interface POS0/6/0/4 no vrf !!
% The interface's numbered and unnumbered IPv4/IPv6 addresses must be removed prior to
changing or deleting the VRF !
Note
The show configuration failed command in configuration mode only exists as long as the configuration
session is active. Once you exit configuration mode, the command cannot be used to display the failed
configuration.
Commit Configuration Failed
The following example shows an invalid task ID configuration that fails to commit. The show
configuration failed command provides information on why the configuration failed.
RP/0/RSP0/CPU0:router(config)# taskgroup isis
RP/0/RSP0/CPU0:router(config-tg)# commit
% Failed to commit one or more configuration items during an atomic operation, s
RP/0/RSP0/CPU0:router(config-tg)# show configuration failed
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-25
Chapter 1
General Troubleshooting Procedures
Verifying the System
!! CONFIGURATION FAILED DUE TO SEMANTIC ERRORS
taskgroup isis
!!% Usergroup/Taskgroup names cannot be taskid names
!
If a configuration commit fails, do not exit configuration mode (return to EXEC mode) as you will not
be able to view the failed configuration.
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# taskgroup bgp
RP/0/RSP0/CPU0:router(config-tg)# end
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:y
% Failed to commit one or more configuration items during an atomic operation, s
RP/0/RSP0/CPU0:router(config)# exit
Uncommitted changes found, commit them before exiting(yes/no/cancel)? [cancel]:n
RP/0/RSP0/CPU0:router# show configuration failed
RP/0/RSP0/CPU0:router#
Verifying the System
To verify the general status and state of a router using Cisco IOS XR software, perform the following
procedure.
SUMMARY STEPS
1.
admin
2.
show platform [node-id]
3.
show version
4.
show running-config
5.
show logging
6.
show environment
7.
show context
8.
exit
9.
show context
10. show memory summary detail location all
11. show memory heap summary {job-id | all}
12. top processes
13. show running-config
14. show system verify start
show system verify report
15. show {ipv4 | ipv6} interface brief
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-26
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
DETAILED STEPS
Step 1
Command or Action
Purpose
admin
Enters administration mode.
Example:
RP/0/RSP0/CPU0:router#
Step 2
admin
Displays information about the status of cards and modules
installed in the router.
show platform [node-id]
Example:
•
Some cards support a CPU module and service
processor (SP) module. Other cards support only a
single module.
•
A card module is also called a node. When all nodes are
working properly, the status of each node displayed in
the State column is IOS-XR RUN.
•
If you run the command without a node-id (show
platform as shown in the example), the output will
include all nodes in the system.
•
Type the show platform node-id command to display
information for a specific node. Replace node-id with a
node name from the show platform command Node
column.
RP/0/RSP0/CPU0:router(admin)# show platform
Step 3
show version
Displays information about the router, including image
names, uptime, and other system information.
Example:
Verify that the expected software version and images are
installed.
RP/0/RSP0/CPU0:router(admin)# show version
Step 4
show running-config
Example:
RP/0/RSP0/CPU0:router(admin)# show
running-config
Displays all of the nondefault commands currently running,
including hardware module power status, secure domain
router (SDR) configuration, and fabric configuration. The
output also displays the users defined in administration
mode with root-system access.
Verify that the serial numbers for the nodes in the current
running configuration are what you expected. The expected
rack numbers and serial numbers should be listed in the
current system documentation. See the “Prerequisite
Documentation for Troubleshooting” section on page 1-1.
Also verify that the hardware module power status is as
expected and the SDR and fabric configurations are as
expected.
Step 5
Displays all syslog messages stored in the buffer. The
command output displays the device operation history from
a system perspective.
show logging
Example:
RP/0/RSP0/CPU0:router(admin)# show logging
Analyze the logged events and their order of happening.
Check for anything out of the ordinary such as errors,
tracebacks, or crashes. Also check for any Severity 1 or
Severity 2 errors.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-27
Chapter 1
General Troubleshooting Procedures
Verifying the System
Step 6
Command or Action
Purpose
show environment
Displays environmental monitor parameters for the system.
Verify that the parameters are as expected.
Example:
RP/0/RSP0/CPU0:router(admin)# show environment
Step 7
show context
Example:
RP/0/RSP0/CPU0:router(admin)# show context
Step 8
Displays core dump context information on fabric cards,
alarm modules, fan controllers, and service processors
(system-owned cards). See the “show context Command”
section on page 1-50 for more information on the show
context command output.
Exits administration mode.
exit
Example:
RP/0/RSP0/CPU0:router(admin)# exit
Step 9
show context
Example:
RP/0/RSP0/CPU0:router#
Step 10
show memory summary detail location all
Displays information about the memory available on the
router after the system image decompresses and loads.
Example:
Verify that the expected memory is available or installed.
Ensure that all memory regions have adequate free space
available.
RP/0/RSP0/CPU0:router#
detail location all
Step 11
show context
Displays core dump context information on CPUs
responsible for routing and Cisco Express Forwarding
(CEF). See the “show context Command” section on
page 1-50 for more information on the show context
command output.
show memory summary
show memory heap summary {job-id | all}
Example:
RP/0/RSP0/CPU0:router#
summary all
show memory heap
Displays a summary of the information about the heap
space. The output displays each process and the amount of
memory allocated for each process.
Note
The job-id is the output of the show processes
command.
Verify if there are any processes using a large amount of
memory.
Step 12
Provides a live update of process resource consumption.
top processes
Press ‘M’ to sort by memory usage.
Example:
RP/0/RSP0/CPU0:router# top processes
Verify that the resource consumption is as expected.
Press q to exit the command.
Step 13
Displays the contents of the currently running
configuration.
show running-config
Example:
RP/0/RSP0/CPU0:router#
show running-config
Verify that the contents of the current running configuration
are what you expected.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-28
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
Step 14
Command or Action
Purpose
show system verify start
show system verify report
A two-step command that produces system reports.
Example:
RP/0/RSP0/CPU0:router#
start
RP/0/RSP0/CPU0:router#
report
show system verify
show system verify
•
show system verify start—Starts the system verify
process (creates the initial baseline file)
•
show system verify report—Generates a report for the
system verification process (report of the current
status)
The output of the show system verify report command
provides a comparison of the system at the time of the show
system verify start snapshot and the show system verify
report snapshot. The output provides a sanity check of the
system provided the show system verify start system
snapshot was taken when the system was healthy or before
an event.
Verify that the system parameters are as expected.
Step 15
show (ipv4 | ipv6} interface brief
Displays the usability status of interfaces.
Example:
Verify that all expected interfaces are listed, that they have
the correct assigned address, and that they are in the
expected states.
RP/0/RSP0/CPU0:router#
brief
show ipv4 interface
Examples
The output from the show platform command indicates that all expected nodes are in the run state. If
all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the
OK state. The example output shows that all expected nodes are in the run state.
RP/0/RSP0/CPU0:router(admin)# show platform
Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/FT0/SP
FAN TRAY
READY
0/FT1/SP
FAN TRAY
READY
0/1/CPU0
A9K-40GE-B
IOS XR RUN
PWR,NSHUT,MON
0/2/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,MON
0/2/0
SPA-2XOC48POS/RPR
OK
PWR,NSHUT,MON
0/3/CPU0
A9K-2T20GE-B
IN-RESET
PWR,NSHUT,MON
0/4/CPU0
A9K-8T/4-B
IOS XR RUN
PWR,NSHUT,MON
0/6/CPU0
A9K-4T-B
IOS XR RUN
PWR,NSHUT,MON
0/PM0/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM1/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM2/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
The output from the show version command indicates the version of software being run on the nodes
and from which location (disk or network). Check that the expected software version and images are
installed. The example output shows that the Cisco IOS XR software version is 4.0.0 and that the
installed pie versions are also 4.0.0.
RP/0/RSP0/CPU0:router(admin)# show version
Cisco IOS XR Software, Version 4.0.0[Default]
Copyright (c) 2010 by Cisco Systems, Inc.
ROM: System Bootstrap, Version 1.04(20100216:021454) [ASR9K ROMMON],
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-29
Chapter 1
General Troubleshooting Procedures
Verifying the System
router uptime is 1 day, 15 hours, 53 minutes
System image file is "bootflash:disk0/asr9k-os-mbi-4.0.0/mbiasr9k-rp.vm"
cisco ASR9K Series (MPC8641D) processor with 4194304K bytes of memory.
MPC8641D processor at 1333MHz, Revision 2.2
2 Management Ethernet
12 DWDM controller(s)
12 TenGigE
40 GigabitEthernet
2 SONET/SDH
2 Packet over SONET/SDH
219k bytes of non-volatile configuration memory.
975M bytes of compact flash card.
33994M bytes of hard disk.
1605616k bytes of disk0: (Sector size 512 bytes).
1605616k bytes of disk1: (Sector size 512 bytes).
Configuration register on node 0/RSP0/CPU0 is 0x102
Boot device on node 0/RSP0/CPU0 is disk0:
Package active on node 0/RSP0/CPU0:
asr9k-optics-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-optics-supp-4.0.0
Built on Wed Sep 8 16:17:30 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-fwding, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-fwding-4.0.0
Built on Wed Sep 8 16:12:40 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-cpp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-cpp-4.0.0
Built on Wed Sep 8 16:13:28 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9K-doc-supp, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9K-doc-supp-4.0.0
Built on Wed Sep 8 16:16:57 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
asr9k-scfclient, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:asr9k-scfclient-4.0.0
Built on Wed Sep 8 16:13:26 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
iosxr-security, V 4.0.0[DT_IMAGE], Cisco Systems, at disk0:iosxr-security-4.0.0
Built on Wed Sep 8 16:16:48 DST 2010
By sjc5-gf-021 in /auto/ioxbuild8/production/4.0.0.DT_IMAGE/asr9k/workspace for pie
--More--
The output from the show running-config command displays the current running configuration, that is,
all of the nondefault commands currently active. Verify that the contents of the current running
configuration are as expected.
Tip
The output of this command in exec mode is different from the output in admin mode. You should run
the command from each of these modes to locate all of the configuration information.
RP/0/RSP0/CPU0:router# show running-config
Building configuration...
!! Last configuration change at 18:56:31 UTC Tue Feb 28 2006 by user_A
!
hostname router
clock timezone PST 8.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-30
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
logging console informational
telnet vrf default ipv4 server max-servers 100
domain name cisco.com
domain lookup disable
taskgroup default
!
taskgroup igpadmin
task write rib
task write isis
task write ospf
!
taskgroup routeadmin
task write bgp
inherit taskgroup igpadmin
!
usergroup default
taskgroup default
!
usergroup igp_admin
taskgroup igpadmin
!
usergroup route_admin
taskgroup routeadmin
!
tacacs-server host 172.29.52.69 port 49
key 7 060506324F41
aaa group server tacacs+ DOC_LAB_TACACS+
server 172.29.52.69
server 172.29.52.68
!
aaa authorization exec LAB_AAA group DOC_LAB_TACACS+ local
aaa authorization exec CONSOLE_AAA group DOC_LAB_TACACS+ none
aaa authorization commands LAB_AAA group DOC_LAB_TACACS+ none
aaa authorization commands CONSOLE_AAA group DOC_LAB_TACACS+ none
aaa authentication login default group DOC_LAB_TACACS+ local
aaa authentication login CONSOLE_AAA group DOC_LAB_TACACS+ local
aaa default-taskgroup default
explicit-path name Primary_GE_Path_to_P19
index 1 next-address strict ipv4 unicast 10.114.4.44
index 2 next-address strict ipv4 unicast 10.114.4.11
index 3 next-address strict ipv4 unicast 10.119.4.11
index 4 next-address strict ipv4 unicast 10.119.4.19
index 5 next-address strict ipv4 unicast 10.19.19.19
!
line console
accounting exec CONSOLE_AAA
accounting commands CONSOLE_AAA
authorization exec CONSOLE_AAA
authorization commands CONSOLE_AAA
login authentication CONSOLE_AAA
exec-timeout 600 0
session-timeout 600
!
line default
exec-timeout 600 0
session-timeout 600
!
.
.
.
interface preconfigure GigabitEthernet0/3/0/18
shutdown
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-31
Chapter 1
General Troubleshooting Procedures
Verifying the System
!
interface preconfigure GigabitEthernet0/3/0/19
shutdown
!
interface preconfigure TenGigE0/3/0/0
shutdown
!
interface preconfigure TenGigE0/3/0/1
shutdown
!
router static
address-family ipv4 unicast
0.0.0.0/0 172.29.52.1 200
!
!
router isis 100
is-type level-2-only
net 49.0001.0000.0000.0044.00
nsf cisco
address-family ipv4 unicast
metric-style wide
mpls traffic-eng level-2-only
mpls traffic-eng router-id Loopback0
!
interface Loopback0
passive
address-family ipv4 unicast
!
!
interface TenGigE0/4/0/0
bfd minimum-interval 50
bfd multiplier 3
bfd fast-detect ipv4
address-family ipv4 unicast
!
!
!
router ospf 100
nsr
router-id 10.144.144.144
bfd minimum-interval 50
bfd multiplier 3
mpls ldp sync
nsf cisco
area 0
mpls ldp sync-igp-shortcuts
mpls traffic-eng
interface Loopback0
passive enable
!
.
.
.
http server
ssh server vrf default
igmp snooping profile default
system-ip-address 10.144.144.144
minimum-version 2
internal-querier
tcn query solicit
ttl-check disable
router-alert-check disable
!
igmp snooping profile mrouter
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-32
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
router-guard
mrouter
!
router pim
address-family ipv4
mofrr mofrr-acl
rp-address 10.11.11.11
rp-address 10.144.144.144 bidir-acl bidir
spt-threshold infinity
!
!
end
RP/0/RSP0/CPU0:router# admin
Wed Oct 27 14:52:07.000 DST
RP/0/RSP0/CPU0:router(admin)# show running-config
Wed Oct 27 14:52:12.766 DST
Building configuration...
!! IOS XR Admin Configuration 4.0.0
username doclabuser-c
group root-system
group cisco-support
secret 5 $1$RJVQ$6w7saUHgk16v5HXRWEp6m/
!
username doclabuser-r
group root-system
secret 5 $1$.uOF$O9N0aRRk.V1qe250IavLw1
!
alias cr copy run disk0a:/usr/base_config_admin
alias sa show alias
alias sc show config commit list
alias sd show diag
alias si show install req
alias sl show led
alias sp show platform
alias sr show run
alias sv show version
alias nda no debug all
end
The output from the show logging command displays the contents of the logging buffer. The output
displays details on syslog historical events. Analyze the logged events and the order in which they
happened. Check for anything out of the ordinary such as errors, tracebacks, or crashes. Also check for
any Severity 1 or Severity 2 errors.
RP/0/RSP0/CPU0:router(admin)# show logging
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
Console logging: level informational, 693 messages logged
Monitor logging: level debugging, 0 messages logged
Trap logging: level informational, 0 messages logged
Buffer logging: level debugging, 4467 messages logged
Log Buffer (307200 bytes):
LC/0/3/CPU0:Sep 13 23:58:03.272 : pfm_node_lc[230]: %PLATFORM-NP-0-NP_INIT_FAILURE :
Set|prm_server[110670]|Network Processor Unit(0x1007000)|Persistent Initialization
Failure.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-33
Chapter 1
General Troubleshooting Procedures
Verifying the System
LC/0/3/CPU0:Sep 13 23:58:03.276 : pfm_node_lc[230]: %PLATFORM-PFM-0-CARD_RESET_REQ :
pfm_dev_sm_perform_recovery_action, Card reset requested by: Process ID: 110670
(prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device Handle: 0x1007000,
CondID: 1027, Fault Reason: Persistent Initialization Failure.
LC/0/3/CPU0:Sep 13 23:58:03.276 : syslog_dev[85]: pfm_node_lc[230]: Request Graceful
Reboot via Sysmgr: Reason: pfm_dev_sm_perform_recovery_action, Card reset requested by:
Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device
Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure.
LC/0/3/CPU0:Sep 13 23:58:03.277 : sysmgr[87]: %OS-SYSMGR-2-REBOOT : reboot required,
process (pfm_node_lc) reason (pfm_dev_sm_perform_recovery_action, Card reset requested by:
Process ID: 110670 (prm_server), Fault Sev: 0, Target node: 0/3/CPU0, CompId: 0x1f, Device
Handle: 0x1007000, CondID: 1027, Fault Reason: Persistent Initialization Failure. )
LC/0/3/CPU0:Sep 13 23:58:03.467 : sysmgr[87]: %OS-SYSMGR-3-ERROR :
sysmgr_shutdown_cleanup_handler: shutdown script execution timed-out! Node will reset
LC/0/3/CPU0:Sep 13 23:58:03.467 : sysmgr[87]: %OS-SYSMGR-7-DEBUG :
sysmgr_shutdown_cleanup_handler: shutdown script execution timed-out! Node will reset
LC/0/3/CPU0:Sep 13 23:58:03.468 : sysmgr[87]: %OS-SYSMGR-3-ERROR :
sysmgr_shutdown_cleanup_handler: shutdown triggered by (pfm_node_lc) did not complete in
45 seconds, shutting down
RP/0/RSP0/CPU0:Sep 13 23:58:16.859 : shelfmgr[299]:
%PLATFORM-SHELFMGR-0-MAX_RESET_BRINGDOWN : Can not boot node 0/3/CPU0 A9K-2T20GE-B due to
multiple resets, putting it IN_RESET state. The probable cause is an unexpected event on
the node or a failure in communication with the node. Please refer to the Cisco ASR 9000
System Error Message Reference Guide for further information if needed.
--More--
The output from the show environment command displays environmental monitor parameters for the
system. Verify that the environment parameters are as expected. Environment parameter anomalies are
logged in the syslog, so if an environment parameter displayed in the show environment command
output is not as expected, check the syslog using the show logging command. The syslog provides details
on any logged problems.
RP/0/RSP0/CPU0:router(admin)# show environment
Wed Sep 15 09:48:27.178 DST
Temperature Information
--------------------------------------------R/S/I
Modules Sensor
(deg C)
host
host
Inlet0
Hotspot0
36.4
46.7
spa0
spa0
InletTemp
Hotspot
35.5
35.5
host
host
Inlet0
Hotspot0
34.5
61.0
host
host
Inlet0
Hotspot0
31.1
32.5
host
host
Inlet0
Hotspot0
31.3
42.0
0/1/*
0/2/*
0/3/*
0/RSP0/*
0/4/*
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-34
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
host
host
Inlet0
Hotspot0
34.9
45.2
host
host
Inlet0
Hotspot0
38.0
48.9
host
host
Inlet0
Hotspot0
31.9
33.5
host
host
Inlet0
Hotspot0
32.3
35.3
0/6/*
0/FT0/*
0/FT1/*
Voltage Information
--------------------------------------------R/S/I
Modules Sensor
(mV)
Margin
0/1/*
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
host
IBV
5.0V
VP3P3_CAN
3.3V
2.5V
1.8VB
1.2VB
1.8VA
0.9VB
1.2V_LDO_BRG0
1.2V_LDO_BRG1
1.8VC
1.5VB
1.5VA
1.1V(1.05V_CPU)
0.75VA
0.75VB_0.75VC
1.1VB
1.2V_TCAM0
1.2V_TCAM1
1.0V_Bridge_LDO
1.0VB
0.75VD_and_0.75VE
1.2V_TCAM2
1.2V_TCAM3
1.5VC
1.8VD
1.1VC
ZARLINK_3.3V
ZARLINK_1.8V
1.2V_DB
3.3V_DB
2.5V_DB
1.5V_DB
10592
4925
3289
3302
2516
1812
1193
1806
886
1193
1195
1811
1505
1503
1053
752
754
1103
1003
1000
999
1042
752
1006
1002
1504
1804
1100
3272
1807
1195
3318
2535
1509
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
.
.
.
LED Information
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-35
Chapter 1
General Troubleshooting Procedures
Verifying the System
--------------------------------------------R/S/I
Modules
0/RSP0/*
host
host
host
host
host
LED
Status
Critical-Alarm
Major-Alarm
Minor-Alarm
ACO
Fail
Off
Off
Off
Off
Off
Fan Information
--------------------------------------------Fan speed (rpm):
FAN0
FAN1
FAN10
FAN11
FAN2
FAN3
FAN4
FAN5
FAN6
FAN7
FAN8
FAN9
3510
3510
3480
3540
3540
3510
3390
3510
3510
3540
0/FT0/*
3510
3480
3510
3570
3540
3480
3480
0/FT1/*
3510
3510
3510
3540
3480
3450
3480
Power Supply Information
--------------------------------------------R/S/I
Modules
Capacity
(W)
Status
0/PM0/*
host
PM
3000
Ok
host
PM
3000
Ok
host
PM
3000
Ok
Voltage
(V)
53.7
53.8
54.1
Current
(A)
11.6
14.4
0.0
0/PM1/*
0/PM2/*
R/S/I
Power Draw
(W)
0/PM0/* 622.9
0/PM1/* 774.7
0/PM2/* 0.0
-------------Total: 1397.6
Power Shelves Type: AC
Total Power Capacity:
Usable Power Capacity:
Supply Failure Protected Capacity:
Feed Failure Protected Capacity:
Worst Case Power Used:
9000W
9000W
6000W
3000W
3170W
Slot
---0/1/CPU0
0/2/CPU0
0/RSP0/CPU0
0/RSP1/CPU0
0/4/CPU0
0/6/CPU0
0/FT0/SP
Max Watts
--------350
450
235
235
350
350
600
(default)
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-36
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
0/FT1/SP
600
Worst Case Power Available:
Supply Protected Capacity Available:
Feed Protected Capacity Available:
5830W
2830W
Not Protected
The output from the show context command displays core dump context information. See the “show
context Command” section on page 1-50 for more information on the show context command output.
RP/0/RSP0/CPU0:router# show context
node:
node0_1_CPU0
-----------------------------------------------------------------Crashed pid = 61524 (pkg/bin/tcam_mgr)
Crashed tid = 1
Crash time: Wed Apr 05, 2006: 18:27:26
Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
R0
R4
R8
R12
R16
R20
R24
R28
R32
R36
r0
00000000
r4
0000f054
r8
00000000
r12
4823be90
r16
00000048
r20
00000000
r24
00000003
r28
00000006
cnt
00000000
cnd
28004024
Registers info
r1
r2
481ff7b0 4824a55c
r5
r6
00000001 00000006
r9
r10
fc220000 481fffc0
r13
r14
4824a4a0 48230000
r17
r18
00000001 00000019
r21
r22
00000000 00000003
r25
r26
00000000 00000003
r29
r30
0000f054 48254064
lr
msr
fc1c6340 0000d932
xer
00000008
r3
00000000
r7
00000000
r11
00000000
r15
00000000
r19
48256520
r23
00000045
r27
4825dc34
r31
481ff810
pc
fc1d3fa0
DLL Info
DLL path
Text addr. Text size
Data addr. Data size
Version
/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8
0
/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000
0x00000bbc
0
Crash Package Infomation
Package: hfr-mgbl, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-37
Chapter 1
General Troubleshooting Procedures
Verifying the System
Package: hfr-mcast, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-mpls, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-rout, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-k9sec, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-lc, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/works
pace for c2.95.3-p8
Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-os-mbi, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/w
orkspace for c2.95.3-p8
node:
node0_6_CPU0
-----------------------------------------------------------------node:
node0_RSP0_CPU0
-----------------------------------------------------------------node:
node0_RSP1_CPU0
------------------------------------------------------------------
The example output shows that the pkg/bin/tcam_mgr process crashed.
The output from the show memory command displays information about the memory available on the
router after the system image decompresses and loads. Verify that the expected memory is available or
installed. Ensure that all memory regions have adequate free space available. The example output shows
that there is 2.003 gigabits of application memory available.
RP/0/RSP0/CPU0:router# show memory summary detail location all
Physical Memory: 4.000G total (2.003G available)
Application Memory : 3.826G (2.003G available)
Image: 48.725M (bootram: 48.725M)
Reserved: 128.000M, IOMem: 1.980G, flashfsys: 0
Shared window mfwdv6: 449.910K
Shared window mfwd_info: 701.910K
Shared window soasync-app: 242.402K
Shared window soasync: 242.402K
Shared window li: 3.359K
Shared window ipv4_fib: 1.003M
Shared window l2fib: 2.425M
Shared window statsd_db: 67.386K
Shared window mgid: 587.390K
Shared window ifc-protomax: 1.290M
Shared window ifc-mpls: 7.981M
Shared window ifc-ipv6: 7.212M
Shared window ifc-ipv4: 11.286M
Shared window infra_statsd: 3.402K
Shared window im_rd: 1.104M
Shared window im_db: 1.204M
Shared window infra_ital: 67.316K
Shared window netio_fwd: 292
Shared window vkg_bmp_adj: 211.371K
Shared window aib: 623.375K
Shared window rspp_ma: 3.351K
Shared window im_rules: 293.308K
Shared window aaa: 67.382K
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-38
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Verifying the System
Shared window pfm_node: 131.304K
Shared window atc_cache: 35.359K
Shared window spp: 619.312K
Shared window qad: 134.707K
Shared window pcie-server: 43.246K
Total shared window: 37.931M
Allocated Memory: 378.742M
Program Text: 46.175M
Program Data: 37.796M
Program Stack: 16.539M
The show system verify start command starts the system verification process and the show system
verify report generates the output from the system verification process. The output allows you to verify
that the system parameters are as expected.
RP/0/RSP0/CPU0:router# show system verify start
Storing initial router status ...
done.
The example output compares the system from the time the show system verify start command took the
first snapshot to the snapshot taken of the system when the show system verify report command took
the second snapshot and generated the comparison. If there are no changes, [OK] is displayed. If there
are changes between the first and second snapshot, the specific change is noted and marked with
[WARNING] or [FAIL].
RP/0/RSP0/CPU0:router# show system verify report
Getting current router status ...
System Verification Report
==========================
- Verifying Memory Usage
- Verified Memory Usage
- Verifying CPU Usage
- Verified CPU Usage
-
Verifying Blocked Processes
Verified Blocked Processes
Verifying Aborted Processes
Verified Aborted Processes
Verifying Crashed Processes
Verified Crashed Processes
- Verifying LC Status
- Verified LC Status
- Verifying QNET Status
Unable to get current LC status info
- Verified QNET Status
-
Verifying GSP Fabric Status
Verified GSP Fabric Status
Verifying GSP Ethernet Status
Verified GSP Ethernet Status
-
Verifying POS interface Status
Verified POS interface Status
Verifying TenGigE interface Status
Verified TenGigE interface Status
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
: [FAIL]
: [OK]
: [OK]
: [OK]
: [OK]
- Verifying TCP statistics
- Verified TCP statistics
: [OK]
- Verifying UDP statistics
tcp_udp_raw WARNING messages for router
UDP Packets sent has not increased during this period.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-39
Chapter 1
General Troubleshooting Procedures
Verifying the System
- Verified UDP statistics
- Verifying RAW statistics
- Verified RAW statistics
-
: [WARNING]
: [OK]
Verifying RIB Status
Verified RIB Status
Verifying CEF Status
Verified CEF Status
Verifying CEF Consistency Status
Verified CEF Consistency Status
Verifying BGP Status
Verified BGP Status
Verifying ISIS Status
Verified ISIS Status
Verifying OSPF Status
Verified OSPF Status
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
: [OK]
- Verifying Syslog Messages
- Verified Syslog Messages
: [OK]
System may not be stable. Please look into WARNING messages.
The show interface brief command displays the usability status of the configured interfaces. Verify that
all expected interfaces are listed. For an interface to be usable, both the interface hardware (Status) and
line protocol must be up. The protocol is Up if the interface can provide two-way communication. The
example output displays IP addresses, status, and protocol status for each interface. The output shows
that all assigned interfaces (interfaces that are configured with IP addresses) have an interface hardware
status and line protocol status of Up.
RP/0/RSP0/CPU0:router# show ipv4 interface brief
Interface
Bundle-Ether16
Bundle-Ether16.160
Bundle-Ether16.161
Bundle-Ether16.162
Bundle-Ether16.163
Loopback0
tunnel-te44190
tunnel-te44192
tunnel-te44194
tunnel-te44196
MgmtEth0/RSP0/CPU0/0
MgmtEth0/RSP0/CPU0/1
GigabitEthernet0/1/0/0
GigabitEthernet0/1/0/1
GigabitEthernet0/1/0/2
GigabitEthernet0/1/0/3
GigabitEthernet0/1/0/3.160
GigabitEthernet0/1/0/3.161
GigabitEthernet0/1/0/3.185
GigabitEthernet0/1/0/3.189
GigabitEthernet0/1/0/3.215
GigabitEthernet0/1/0/4
GigabitEthernet0/1/0/5
GigabitEthernet0/1/0/6
GigabitEthernet0/1/0/7
GigabitEthernet0/1/0/7.185
GigabitEthernet0/1/0/7.187
GigabitEthernet0/1/0/7.189
GigabitEthernet0/1/0/7.210
GigabitEthernet0/1/0/7.211
GigabitEthernet0/1/0/7.215
IP-Address
unassigned
unassigned
unassigned
10.194.8.44
10.194.12.44
10.144.144.144
10.144.144.144
10.144.144.144
10.144.144.144
10.144.144.144
172.29.52.137
unassigned
unassigned
unassigned
10.147.4.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
Status
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Shutdown
Shutdown
Shutdown
Up
Up
Up
Up
Up
Up
Up
Shutdown
Shutdown
Shutdown
Up
Up
Up
Up
Up
Up
Up
Protocol
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Up
Down
Down
Down
Up
Up
Up
Up
Up
Up
Up
Down
Down
Down
Up
Up
Up
Up
Up
Up
Up
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-40
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
GigabitEthernet0/1/0/8
GigabitEthernet0/1/0/9
GigabitEthernet0/1/0/10
GigabitEthernet0/1/0/11
GigabitEthernet0/1/0/12
GigabitEthernet0/1/0/13
GigabitEthernet0/1/0/14
GigabitEthernet0/1/0/15
GigabitEthernet0/1/0/16
GigabitEthernet0/1/0/17
GigabitEthernet0/1/0/18
GigabitEthernet0/1/0/19
GigabitEthernet0/1/0/19.2127
GigabitEthernet0/1/0/19.2130
GigabitEthernet0/1/0/20
GigabitEthernet0/1/0/20.2125
GigabitEthernet0/1/0/21
GigabitEthernet0/1/0/22
GigabitEthernet0/1/0/23
GigabitEthernet0/1/0/24
GigabitEthernet0/1/0/25
GigabitEthernet0/1/0/26
GigabitEthernet0/1/0/27
GigabitEthernet0/1/0/28
GigabitEthernet0/1/0/29
GigabitEthernet0/1/0/30
GigabitEthernet0/1/0/30.215
GigabitEthernet0/1/0/31
GigabitEthernet0/1/0/32
GigabitEthernet0/1/0/33
GigabitEthernet0/1/0/34
GigabitEthernet0/1/0/35
GigabitEthernet0/1/0/36
GigabitEthernet0/1/0/37
GigabitEthernet0/1/0/38
GigabitEthernet0/1/0/39
POS0/2/0/0
POS0/2/0/1
TenGigE0/4/0/0
TenGigE0/4/0/1
TenGigE0/4/0/2
TenGigE0/4/0/3
TenGigE0/4/0/4
TenGigE0/4/0/5
TenGigE0/4/0/6
TenGigE0/4/0/7
TenGigE0/6/0/0
TenGigE0/6/0/1
TenGigE0/6/0/2
TenGigE0/6/0/3
10.146.4.44
unassigned
unassigned
unassigned
10.194.16.44
unassigned
unassigned
unassigned
unassigned
unassigned
10.194.4.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
10.114.4.44
unassigned
unassigned
unassigned
10.145.4.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
10.114.8.44
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
unassigned
Up
Shutdown
Shutdown
Shutdown
Up
Shutdown
Shutdown
Shutdown
Up
Up
Up
Up
Up
Up
Up
Up
Shutdown
Shutdown
Up
Shutdown
Shutdown
Shutdown
Up
Shutdown
Shutdown
Up
Up
Up
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Up
Up
Up
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Shutdown
Up
Down
Down
Down
Up
Down
Down
Down
Up
Up
Up
Up
Up
Up
Up
Up
Down
Down
Up
Down
Down
Down
Up
Down
Down
Up
Up
Up
Down
Down
Down
Down
Down
Down
Down
Down
Up
Up
Up
Down
Down
Down
Down
Down
Down
Down
Down
Down
Down
Down
Troubleshooting the Backplane Ethernet Control System
This section describes techniques that you can use to troubleshoot the control plane Ethernet network on
routers using Cisco IOS XR software. The system control plane Ethernet network is used for processes
on different devices to communicate for functions such as system device discovery, image transfers,
heartbeat messages, alarms, and configuration management.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-41
Chapter 1
General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
All devices in a system using Cisco IOS XR software connect to the system control plane Ethernet
network, also called the Ethernet over backplane channel (EOBC). The control plane is provided using
Gigabit Ethernet (GE) links between nodes. The GE links are internal to the chassis and cannot be
removed.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-42
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
Figure 1-1 shows the control plane Ethernet network (the dotted line in the drawing).
Figure 1-1
Cisco ASR 9000 Aggregation Services Router Control Ethernet Topology
RSP 0
RSP 1
Fabric
Interface
Chip
System
Timing
Fabric
Chip
Fabric
Interface
Chip
CPU
VOQ
Scheduler
System
Timing
GE
Switch
CPU
Data Plane
Fabric
Chip
VOQ
Scheduler
Control Plane
GE
Switch
Backplane
40x1GE
Line Card
Fabric
Interface
Chip
8x10GE 2:1
Fabric
Oversubscribed Interface
Line Card
Chip
GE PHY
CPU
FPGA
NPU
NPU
10 x
SFP
10 x
SFP
GE PHY
FPGA
NPU
NPU
CPU
FPGA
NPU
NPU
FPGA
NPU
X
F
P
10 x
SFP
X
F
P
X
F
P
X
F
P
X
F
P
X
F
P
X
F
P
FPGA
NPU
NPU
NPU
NPU
10
GE
10
GE
10
GE
10
GE
X
F
P
X
F
P
X
F
P
X
F
P
10 10 10 10 10 10 10 10
GE GE GE GE GE GE GE GE
10 x
SFP
Fabric
Interface
Chip
CPU
FPGA
NPU
4x10GE
Line Card
GE PHY
X
F
P
Backplane
8x10GE 80G
Line Rate Card
2x10GE + 20x1GE
Combo Line Card
Fabric
Interface
Chip
Fabric
Interface
Chip
GE
PHY
FPGA
FPGA
CPU
FPGA
FPGA
CPU
NPU NPU NPU NPU
NPU NPU NPU NPU
GE
SW
NPU NPU
NPU NPU
GE
SW
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
10
GE
X
F
P
To
NPUs
To
FPGAs
10x 10x
S
F
P
S
F
P
GE
PHY
10
GE
X
F
P
10
GE
X
F
P
To
NPUs
To
FPGAs
247272
10
GE
X
F
P
Fabric
Interface
Chip
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-43
Chapter 1
General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
To verify and troubleshoot booting of the system control plane Ethernet network, perform the following
procedure.
SUMMARY STEPS
1.
show platform
2.
show controllers backplane ethernet clients all location node-id
3.
show controllers backplane ethernet clients 18 statistics location node-id
4.
Contact Cisco Technical Support if the problem is not resolved.
DETAILED STEPS
Step 1
Command or Action
Purpose
show platform
Displays information about the status of cards and modules
installed in the router.
Example:
Verify that the expected nodes display IOS XR RUN under
the State column of the command output.
RP/0/RSP0/CPU0:router#
Step 2
show platform
show controllers backplane ethernet clients all
location node-id
Displays information about all local client applications.
Each row contains the client Ethernet server ID and the
client process ID (PID).
Example:
The system allows client processes to send and receive
packets over the control Ethernet. It uses client IDs to
demultiplex packets that arrive at the node.
RP/0/RSP0/CPU0:router# show controllers
backplane ethernet clients all location
0/RSP0/CPU0
Two client IDs in the output are important for
troubleshooting boot problems:
Step 3
show controllers backplane ethernet clients 18
statistics location node-id
•
Client Ethernet server ID 18—used for boot requests
•
Client Ethernet server ID 22—used for heartbeats
Displays a list of client statistics for the specified client ID.
Check the values for:
Example:
RP/0/RSP0/CPU0:router# show controllers
backplane ethernet clients 18 statistics
location 0/RSP0/CPU0
•
Packets input
•
Packets delivered
If they contain values other than 0, boot requests have been
received and replies have been sent (packets output).
If they contain values of 0, check the system control plane
Ethernet network physical connectivity.
If there are no problems with the physical connectivity,
contact Cisco Technical Support. For Cisco Technical
Support contact information, see the “Obtaining
Documentation and Submitting a Service Request” section
on page xii.
Step 4
Contact Cisco Technical Support if the problem is not If the problem is not resolved, contact Cisco Technical
resolved.
Support. For Cisco Technical Support contact information,
see the “Obtaining Documentation and Submitting a
Service Request” section on page xii.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-44
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Troubleshooting the Backplane Ethernet Control System
Examples
The output from the show platform command indicates that all expected nodes are in the run state. If
all nodes in the system are active, the cards should be in the IOS XR RUN and the SPAs should be in the
OK state. The example output shows that all expected nodes are in the run state.
RP/0/RSP0/CPU0:router# show platform
Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/1/CPU0
A9K-40GE-B
IOS XR RUN
PWR,NSHUT,MON
0/2/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,MON
0/2/0
SPA-2XOC48POS/RPR
OK
PWR,NSHUT,MON
0/3/CPU0
A9K-2T20GE-B
IN-RESET
PWR,NSHUT,MON
0/4/CPU0
A9K-8T/4-B
IOS XR RUN
PWR,NSHUT,MON
0/6/CPU0
A9K-4T-B
IOS XR RUN
PWR,NSHUT,MON
The following example shows the current state of each Ethernet server client.
RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients all location
0/RSP0/CPU0
Intf
Client ethernet
Client
Description
Name
server id
Process Id
-----------------------------------------------------------------------------GE0_RSP0_CPU0
1
110639
QNX network manager
2
221272
Group services
3
0
Reserved for Attach
4
221274
Plugin controller
5
0
Designated SC
6
0
ASR9K H/W diags
7
221279
IP packet handler
8
217149
Redundancy controller
9
0
ASR9K Virtual console
10
110638
ASR9K Virtual terminal
11
49196
Control ethernet echo
12
0
Control eth echo reply
13
221274 Card Configuration Protocol
14
0
Reserved for Attach
15
0
Chassis controller
16
0
Forwarding driver
17
0
MBI hello
18
110640
MBI Boot Server Source
19
0
HSR ES client
20
0 Packets for ethernet server
21
0
For Diag application
22
233589
heartbeat request
23
0
heartbeat reply
24
221275
Async IPC client
25
0
Test application 1
26
0
Test application 2
27
0
Test client out-of-band
The following example shows that there are 18 nodes in the run state, which means that 12 boot requests
have been received by eth_server and 12 replies have been sent:
RP/0/RSP0/CPU0:router# show controllers backplane ethernet clients 18 statistics location
0/RSP1/CPU0
Client ShelfMgr, ES Client Id 18, PID 53338 running on FastEthernet0_RSP0_0
12 packets input, 8676 bytes
12 packets delivered, 8676 bytes
0 packets discarded (0 bytes) in garbage collection
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-45
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
0 (0 bytes) unicast packets filtered
0 (0 bytes) multicast packets filtered
0 (0 bytes) buffer mgmt policy discards
0 (0 bytes) locking error discards
12 packets output, 8676 bytes, 0 could not be transmitted
Basic Cisco IOS XR Verification and Troubleshooting
Commands
The following commands are used to collect information to aid in verifying the system and
troubleshooting problems:
•
man Command, page 1-46
•
describe Command, page 1-49
•
show platform Command, page 1-49
•
top Command, page 1-50
•
show context Command, page 1-50
•
show users Command, page 1-52
•
show history Command, page 1-52
•
show configuration Command, page 1-53
man Command
The man command provides online help for standard Cisco IOS XR CLI commands using manual (man)
pages. The command is used to display the manual pages for a specific command based on the command
name, a feature, or a keyword. Each man page contains the command name, syntax, command mode,
usage, examples, and related commands.
Note
To run the man command, you must have the Cisco IOS XR Documentation Package,
“asr9k-doc.pie-4.0.0, .man pages for Cisco IOS XR software on the Cisco ASR 9000 Series Router
chassis,” loaded. If you are running a release later than 4.0.0, the package installation envelope (PIE)
name might be different. For the appropriate PIE name and an explanation of PIE installation, see the
“Upgrading Cisco IOS XR Software” section of the Release Notes document for the IOS XR version you
are running
The following example shows the output from the man command show users command.
RP/0/RSP0/CPU0:router# man command show users
Tue Sep 14 14:39:16.409 DST
Building index table...
Total Number of Command Entries:2726
[OK]
DESCRIPTION
Displays information about the active lines on the router.
To display information about the active lines on the router, use the show users
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-46
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
command in EXEC mode.
show users
SYNTAX DESCRIPTION
This command has no arguments or keywords.
COMMAND DEFAULT
No default behavior or values
COMMAND MODES
EXEC
COMMAND HISTORY
Release
Modification
Release 3.7.2
This command was introduced.
USAGE GUIDELINES
To use this command, you must be in a user group associated with a task group
that includes the proper task IDs. If you suspect user group assignment is
preventing you from using a command, contact your AAA administrator for
assistance.
Use the show users command to display the line number, connection name, idle
time, hosts, and terminal location. An asterisk (*) indicates the current
terminal session.
===================================================================================
Note:
To display all user groups and task IDs associated with the currently logged-in
user, use the show user command in EXEC mode. See the Authentication,
Authorization, and Accounting Commands on Cisco IOS XR Software module in
Cisco^B^`ASR^B^`9000 Series Aggregation Services Router System Security Command
Reference.
===================================================================================
TASK ID
Task ID
Operations
tty-access
read
EXAMPLES
The following example shows sample output identifying an active vty terminal
session:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-47
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
* * * * * * * * * * * * * * START OF LISTING * * * * * * * * * * * * * *
RP/0/RSP0/CPU0:router# show users
*
Line
con0_RSP0_CPU0
vty0
vty1
User
cisco
cisco
cisco
* * * * * * * * * * * * * *
Service Conns
Idle
hardware
0 18:33:48
telnet
0 00:30:36
telnet
0 00:00:00
Location
10.33.54.132
10.33.54.132
END OF LISTING * * * * * * * * * * * * * * *
Table 1 describes the significant fields shown in the display.
show users Field Descriptions
Field
Description
Line
All current connections. An asterisk (*) indicates the active connection.
User
Username of the user logged into the line.
Service
Physical or remote login service used.
Conns
Number of outgoing connections.
Idle
Interval (in hours:minutes:seconds) since last keystroke.
Location
IP address of remote login host. For local (physical) terminal
connections, this field is blank.
RELATED COMMANDS
Command
Description
show line Displays the parameters of terminal lines.
Displays the parameters of a terminal line.
show user
Displays all user groups and task IDs associated with the currently
logged-in user.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-48
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
describe Command
The describe command provides a preview of a command without actually implementing it. This
command lists information about the package, component, and task ID for a specific command. You
must be in the appropriate configuration mode for the specific command. For example, to display the
package, component, and task ID information for the router bgp 1 command, you must be in global
configuration mode.
The following example shows the output from the describe router bgp 1 command.
RP/0/RSP0/CPU0:router(config)#describe router bgp 1
Package:
iosxr-routing
iosxr-routing V4.0.0[DT_IMAGE] IOS-XR Routing Package Definition
Vendor : Cisco Systems
Desc
: IOS-XR Routing Package Definition
Build : Built on Wed Sep 8 16:10:14 DST 2010
Source : By router-021 in /files/4.0.0.DT_IMAGE/asr9k/workspace fo8
Card(s): RP, NP24-4x10GE, NP24-40x1GE, NP40-40x1GE, NP40-4x10GE, NP40-8x10GE,
NP40-2_20_COMBO, NP80-8x10GE, NP80-16x10GE, A9K-SIP-700, A9K-SIP-500
Restart information:
Default:
parallel impacted processes restart
Size Compressed/Uncompressed: 8556KB/22MB (37%)
Component:
ipv4-bgp V[ci-401/18]
IPv4 Border Gateway Protocol (BGP)
User needs ALL of the following taskids:
bgp (READ WRITE)
show platform Command
The show platform command displays a high level overview of the entire physical system. Use the show
platform command in administration mode to display a summary of the nodes in the system, including
node type and status.
The following example shows the output from the show platform command in administration mode.
RP/0/RSP0/CPU0:router(admin)#show platform
Tue Sep 14 14:52:52.558 DST
Node
Type
State
Config State
----------------------------------------------------------------------------0/RSP0/CPU0
A9K-RSP-4G(Active)
IOS XR RUN
PWR,NSHUT,MON
0/FT0/SP
FAN TRAY
READY
0/FT1/SP
FAN TRAY
READY
0/1/CPU0
A9K-40GE-B
IOS XR RUN
PWR,NSHUT,MON
0/2/CPU0
A9K-SIP-700
IOS XR RUN
PWR,NSHUT,MON
0/2/0
SPA-2XOC48POS/RPR
OK
PWR,NSHUT,MON
0/3/CPU0
A9K-2T20GE-B
IN-RESET
PWR,NSHUT,MON
0/4/CPU0
A9K-8T/4-B
IOS XR RUN
PWR,NSHUT,MON
0/6/CPU0
A9K-4T-B
IOS XR RUN
PWR,NSHUT,MON
0/PM0/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM1/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
0/PM2/SP
A9K-3KW-AC
READY
PWR,NSHUT,MON
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-49
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
top Command
The top command is used to monitor CPU usage on the system through interactive process statistics.
The following example show the output from the top command.
RP/0/RSP0/CPU0:router# top
Computing times...
224 processes; 803 threads;
CPU states: 93.6% idle, 3.6% user, 2.7% kernel
Memory: 4096M total, 3504M avail, page size 4K
JID
91
256
340
294
1
65816
1
60
256
340
TID LAST_CPU PRI STATE HH:MM:SS
1
0
10 Rcv
0:16:29
10
0
10 Rcv
0:19:43
10
0
10 Rcv
0:03:21
1
1
10 Rcv
0:03:24
12
1
10 Rcv
0:00:03
1
0
10 Rply
0:00:00
11
1
10 Run
0:00:19
5
1
10 Rcv
0:00:43
11
1
10 Rcv
0:00:15
14
1
10 Rcv
0:01:06
CPU COMMAND
2.25% spp
2.16% netio
0.13% udp
0.09% sc
0.09% procnto-600-smp-instr
0.07% top
0.05% procnto-600-smp-instr
0.02% eth_server
0.02% netio
0.01% udp
Press q to exit the command.
show context Command
The show context command displays core dump context information for the last ten core dumps. The
command output is used for post-analysis in the debugging of processes (determine if any process
crashes have occurred).
If there are no crashed processes, the show context command displays no output for each node. The
following example shows the output of the show context command with no crashed processes.
RP/0/RSP1/CPU0:router# show context
node:
node0_1_CPU0
-----------------------------------------------------------------node:
node0_6_CPU0
-----------------------------------------------------------------node:
node0_RSP0_CPU0
-----------------------------------------------------------------node:
node0_RSP1_CPU0
------------------------------------------------------------------
The following example shows the output from the show context command where there is a crashed
process.
RP/0/RSP1/CPU0:router# show context
node:
node0_1_CPU0
-----------------------------------------------------------------Crashed pid = 61524 (pkg/bin/tcam_mgr)
Crashed tid = 1
Crash time: Wed Apr 05, 2006: 18:27:26
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-50
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
Core for process at harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
Stack Trace
#0 0xfc1d3fa0
#1 0xfc1c6340
#2 0xfc1c5364
#3 0xfc1c542c
#4 0x48210930
#5 0x482110b8
#6 0x48212ba4
#7 0x48203dd8
#8 0x4820c61c
#9 0xfc1557ec
#10 0xfc15573c
#11 0xfc152fb8
#12 0x4820d140
R0
R4
R8
R12
R16
R20
R24
R28
R32
R36
r0
00000000
r4
0000f054
r8
00000000
r12
4823be90
r16
00000048
r20
00000000
r24
00000003
r28
00000006
cnt
00000000
cnd
28004024
Registers info
r1
r2
481ff7b0 4824a55c
r5
r6
00000001 00000006
r9
r10
fc220000 481fffc0
r13
r14
4824a4a0 48230000
r17
r18
00000001 00000019
r21
r22
00000000 00000003
r25
r26
00000000 00000003
r29
r30
0000f054 48254064
lr
msr
fc1c6340 0000d932
xer
00000008
r3
00000000
r7
00000000
r11
00000000
r15
00000000
r19
48256520
r23
00000045
r27
4825dc34
r31
481ff810
pc
fc1d3fa0
DLL Info
DLL path
Text addr. Text size
Data addr. Data size
Version
/hfr-os-3.3.90/lib/libinfra.dll 0xfc142000 0x00034200 0xfc1343b8
0
/lib/libc.dll 0xfc1a8000 0x00079dd8 0xfc222000 0x00002000
0x00000bbc
0
Crash Package Infomation
Package: hfr-mgbl, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-mcast, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-mpls, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-rout, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-k9sec, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-lc, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/works
pace for c2.95.3-p8
Package: hfr-fwdg, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-admin, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wo
rkspace for c2.95.3-p8
Package: hfr-base, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/wor
kspace for c2.95.3-p8
Package: hfr-os-mbi, Source: By edde-bld1 in /vws/aga/production/3.3.90.1I/hfr/w
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-51
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
orkspace for c2.95.3-p8
node:
node0_6_CPU0
-----------------------------------------------------------------node:
node0_RSP0_CPU0
-----------------------------------------------------------------node:
node0_RSP1_CPU0
------------------------------------------------------------------
Use the show context command to locate the core dump file path. For example, the core dump file path
shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z. The
command output shows a crash on a node. The process is pkg/bin/tcam_mgr.
Collect the following information and send it to Cisco Technical Support. For Cisco Technical Support
contact information, see the “Obtaining Documentation and Submitting a Service Request” section on
page xii.
•
ppc.Z file—This file contains the binary core dump information. Use the path listed in the command
output to copy the contents of the ppc.Z file. The path shown in the command output is:
harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.Z
•
ppc.txt file—This file contains content on the core dump similar to the show context command
output. Use the path listed in the command output to copy the contents of the ppc.txt file. The path
shown in the command output is: harddisk:/dumper/first.tcam_mgr.abort.node0_1_CPU0.ppc.txt
•
Collect the show version or show install active command output.
show users Command
The show users command displays information on active lines on the router including the line number,
user, service, number of connections, idle time, and remote terminal location. An asterisk (*) indicates
the current terminal session.
The following example shows the output from the show users command.
RP/0/RSP0/CPU0:router# show users
*
Line
vty0
vty1
User
User_A
User-B
Service
telnet
telnet
Conns
0
0
Idle
00:00:00
00:00:03
Location
161.44.1925
161.44.1929
show history Command
The show history command displays a history of the command entered for the current command mode.
You can enter the show history command to display a history of commands entered in EXEC, ADMIN,
or CONFIG mode.
Examples
RP/0/RSP0/CPU0:router# show history
RP/0/RSP0/CPU0:router(admin)# show history
RP/0/RSP0/CPU0:router(config)# show history
The following example shows the output from the show history command in EXEC mode:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-52
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Basic Cisco IOS XR Verification and Troubleshooting Commands
RP/0/RSP0/CPU0:router# show history
RP/0/RSP0/CPU0:router# show history
Thu Oct 28 14:20:50.328 DST
show vrrp interface gigabitEthernet 0/1/0/0
show vrrp interface brief
show vrrp brief
show vrrp detail
show vrrp interface gigabitEthernet 0/1/0/1
show vrrp interface gigabitEthernet 0/1/0/1 statistics all
The detailed history provides a timestamp also:
RP/0/RSP0/CPU0:router# show history
Thu Oct 28 14:26:06.199 DST
1 Thu Oct 28 14:02:25.310 show
2 Thu Oct 28 14:03:34.854 show
3 Thu Oct 28 14:04:02.042 show
4 Thu Oct 28 14:04:08.167 show
5 Thu Oct 28 14:08:25.180 show
6 Thu Oct 28 14:09:03.402 show
detail
vrrp
vrrp
vrrp
vrrp
vrrp
vrrp
interface
interface
brief
detail
interface
interface
gigabitEthernet 0/1/0/0
brief
gigabitEthernet 0/1/0/1
gigabitEthernet 0/1/0/1 statistics all
show configuration Command
The show configuration command displays details on uncommitted configuration changes, that is, the
commands you are about to commit. You can enter the show configuration command to display the
changes in EXEC, ADMIN, or CONFIG mode.
Use the show configuration command with the running keyword to display the running (active)
configuration.
Prior to committing the target configuration, use the show configuration command with the merge
keyword from any configuration mode to display the result of merging the target configuration with the
running configuration.
Examples
RP/0/RSP0/CPU0:router# show configuration
RP/0/RSP0/CPU0:router# show configuration running-config
RP/0/RSP0/CPU0:router(admin)# show configuration running
RP/0/RSP0/CPU0:router(config)# show configuration
RP/0/RSP0/CPU0:router(config)# show configuration running
RP/0/RSP0/CPU0:router(config)# show configuration merge
In this example, the show configuration command displays uncommitted changes made during a
configuration session:
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3
RP/0/RSP0/CPU0:router(config-if)# description faq
RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0
RP/0/RSP0/CPU0:router(config-if)# show configuration
Building configuration...
interface TenGigE0/3/0/3
description faq
ipv4 address 10.10.11.20 255.0.0.0
end
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-53
Chapter 1
General Troubleshooting Procedures
Displaying ASIC Errors
The following example shows sample output from the show configuration command with the optional
merge keyword. The command is entered during a configuration session. The output displays the result
of merging the target and running configuration, without committing the changes.
RP/0/RSP0/CPU0:router# configure
RP/0/RSP0/CPU0:router(config)# interface tengige0/3/0/3
RP/0/RSP0/CPU0:router(config-if)# description faq
RP/0/RSP0/CPU0:router(config-if)# ipv4 address 10.10.11.20 255.0.0.0
RP/0/RSP0/CPU0:router(config-if)# show configuration merge
Building configuration...
hostname router
interface TenGigE0/0/0/0
ipv4 address 10.2.3.4 255.0.0.0
exit
interface TenGigE0/3/0/3
description faq
ipv4 address 10.1.1.1 255.0.0.0
shutdown
end
Displaying ASIC Errors
The following example shows how to display ASIC errors for each ASIC in a SIP-700 optical LC. If an
error is displayed, dump the individual ASIC instance number to obtain details on the ASIC error.
RP/0/RSP0/CPU0:router# show asic-errors all location 0/6/CPU0
Thu Oct 21 19:00:54.178 DST
************************************************************
*
Fia ASIC Error Summary
*
************************************************************
Instance
: 0
Number of nodes
: 0
SBE error count
: 0
MBE error count
: 0
Parity error count : 0
CRC error count
: 0
Generic error count : 0
Reset error count
: 0
-------------------************************************************************
*
Mace ASIC Error Summary
*
************************************************************
Instance
: 0
Number of nodes
: 0
SBE error count
: 0
MBE error count
: 0
Parity error count : 0
CRC error count
: 0
Generic error count : 0
Reset error count
: 0
-------------------************************************************************
*
Prm_np ASIC Error Summary
*
************************************************************
Instance
: 0
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-54
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Displaying ASIC Errors
Number of nodes
:
SBE error count
:
MBE error count
:
Parity error count :
CRC error count
:
Generic error count :
Reset error count
:
-------------------Instance
:
Number of nodes
:
SBE error count
:
MBE error count
:
Parity error count :
CRC error count
:
Generic error count :
Reset error count
:
-------------------Instance
:
Number of nodes
:
SBE error count
:
MBE error count
:
Parity error count :
CRC error count
:
Generic error count :
Reset error count
:
-------------------Instance
:
Number of nodes
:
SBE error count
:
MBE error count
:
Parity error count :
CRC error count
:
Generic error count :
Reset error count
:
--------------------
3
0
0
0
0
0
0
1
3
0
0
0
0
0
0
2
3
0
0
0
0
0
0
3
3
0
0
0
0
0
0
The following ASIC error types are supported:
•
FIA (Fabric Interface ASIC)
•
Mace ASIC
•
Prm_np ASIC
The following ASIC error classifications are supported:
•
Single Bit Errors (SBE)—Correctable ECC protected single bit errors in external or internal
memory.
Not reported to PM on each occurrence and reported to the platform manager (PM) as Minor when
software threshold rate is exceeded. Report alarm using Alarm Logging, and Debugging Event
Management System (ALDEMS).
Error data:
– Address—Address that encountered the SBE
– Syndrome—Syndrome if available
•
Multiple Bit Errors—Uncorrectable multiple bit error in memory.
Reported to PM as Major and ALDEMS for each occurrence.
Error data:
– Address—Address that encountered the SBE.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-55
Chapter 1
General Troubleshooting Procedures
Using Trace Commands
– Data—Actual error data.
•
PARITY Errors—Parity error in all applicable memory.
Reported to PM as Major.
•
Cyclic redundancy check (CRC) Errors—CRC errors in EIO other links.
Not reported for each occurrence. When the threshold is reached it is reported as Major to the PM.
•
GENERIC Errors—Errors that do not fall under any of the other classifications.
Threshold and alarm reporting is done.
•
RESET Errors—Logged for each reset instance of the ASIC.
Reported to PM when threshold is exceeded.
Error data:
– Interrupt status—Interrupt status bits due to ASIC reset.
– Halt status—Halt status bits.
– Reset node key—Key for the error node that causes the reset.
– Time—Reset time.
The following ASIC error fault severities are supported:
•
Critical—Affected component is unusable or card is reset if no redundant card exists.
•
Major—Partially service affecting fault, causing the card to run in degraded mode. For redundant
cards, consider performing a switchover.
•
Minor—Non-service affecting fault.
•
OK—No fault.
Using Trace Commands
Trace commands provide an ‘always on’ debug feature. Many major functions in Cisco IOS XR software
have “trace” functionality to show the last actions it conducted allowing you to analyze function events.
Use the show trace commands to display the trace data for a specific feature or process. Use the ? in the
CLI to determine if a command has the trace keyword. The following example shows that the show arp
command has the trace keyword.
RP/0/RSP0/CPU0:router# show arp ?
A.B.C.D
IP address or hostname of ARP entry
BVI
Bridge-Group Virtual Interface
Bundle-Ether
Aggregated Ethernet interface(s)
GigabitEthernet GigabitEthernet/IEEE 802.3 interface(s)
H.H.H
48-bit hardware address of ARP entry
MgmtEth
Ethernet/IEEE 802.3 interface(s)
TenGigE
TenGigabitEthernet/IEEE 802.3 interface(s)
api-stats
Show ARP API statistics data
client
ARP Client show commands
dagr
Show Direct Attached Gateway Redundancy group information
idb
Show the internal ARP interface data block
location
specify a node name
resolution
Show the ARP resolution history
trace
Show trace data for the ARP component
traffic
ARP traffic statistics
vrf
Specify a VRF
|
Output Modifiers
<cr>
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-56
OL-23591-02
Chapter 1
General Troubleshooting Procedures
MIB Location
The following example shows the last 20 events in the address resolution protocol (ARP) table.
RP/0/RSP0/CPU0:router# show arp trace tailf last 20
1349 wrapping entries (2048 possible, 0 filtered, 1349 total)
Apr 19 09:52:29.857 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for
address: 172.18.105.255
Apr 19 09:52:34.501 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
Apr 19 09:52:41.856 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
Apr 19 09:52:46.324 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
Apr 19 09:52:59.979 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted
from table
Apr 19 09:59:37.463 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
Apr 19 09:59:37.463 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for
address: 172.18.105.255
Apr 19 09:59:39.515 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
Apr 19 09:59:42.082 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
Apr 19 09:59:45.007 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted
from table
Apr 19 09:59:50.101 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
Apr 19 09:59:50.101 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for
address: 172.18.105.255
Apr 19 09:59:54.820 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
Apr 19 10:00:00.008 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted
from table
Apr 19 10:04:11.675 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
Apr 19 10:04:11.675 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for
address: 172.18.105.255
Apr 19 10:04:16.272 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
Apr 19 10:04:30.028 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: entry 172.18.105.255: deleted
from table
Apr 19 10:04:44.097 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: received address resolution
request for 172.18.105.255
Apr 19 10:04:44.097 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: creating incomplete entry for
address: 172.18.105.255
Apr 19 10:04:48.810 ipv4_arp/arp 0/RSP0/CPU0 t1 ARP-TABLE: address resolution failed for
172.18.105.255
MIB Location
To locate and download MIBs, use the Cisco MIB Locator found at the following URL and choose a
platform under the Cisco Access Products menu:
http://cisco.com/public/sw-center/netmgmt/cmtk/mibs.shtml.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-57
Chapter 1
General Troubleshooting Procedures
Gathering Information Before You Call Cisco TAC
Gathering Information Before You Call Cisco TAC
We recommend that you have a system of maintaining and accessing detailed information about your
network and ASR 9000 router, including system hardware and software, network diagrams, and captured
output from commands. For additional details, see the “Prerequisite Documentation for
Troubleshooting” section on page 1-1.
Before calling the Cisco Technical Assistance Center (TAC), you should gather the information
described in the following sections, if possible. This information will be helpful for troubleshooting.
Caution
Timesaver
We strongy recommend that, if possible, you gather the information described in this section before you
reset any cards. If you reset cards before you gather information, the system erases the information and
it will be more difficult to diagnose and repair the problem.
•
Gathering Information about Crashes and Core Dumps, page 1-58
•
Capturing Logs, page 1-58
•
Using Debug Commands, page 1-59
•
Using Diagnostic Commands, page 1-59
•
Commands Used to Display Process and Thread Details, page 1-59
Before contacting Cisco Technical Support, review the information provided at the following URL:
http://www.cisco.com/web/services/ts/access/index.html.
For information on contacting Cisco Technical Support, see the “Obtaining Documentation and
Submitting a Service Request” section on page xii.
Gathering Information about Crashes and Core Dumps
Gather system information with the following commands:
•
show install active summary
•
show version
•
show run
•
show context
•
show log
•
show inventory
•
show diagnostics
Upload any core dumps that were written to disk0, disk1, or harddisk directories.
Capturing Logs
See the “Prerequisite Documentation for Troubleshooting” section on page 1-1 in Chapter 1, “General
Troubleshooting Procedures,” for information on collecting current system information.
Collect system information using the following commands:
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-58
OL-23591-02
Chapter 1
General Troubleshooting Procedures
Gathering Information Before You Call Cisco TAC
•
Note
show tech-support—Displays system information for Cisco Technical Support and includes a
traditional dump of the configuration and show command outputs.
Some tech-support commands require the user to be assigned the cisco-support task ID. For a mapping
of commands to task IDs and allowed operations, see Cisco IOS XR Task ID Reference Guide.
•
show logging—Displays the contents of the logging buffers
•
show system verify—Displays system verification information
Using Debug Commands
For details on using debug commands, see Cisco IOS XR Using Debug Guide.
Using Diagnostic Commands
The Cisco ASR 9000 Aggregation Series Router diagnostic tests verify control Ethernet and fabric data
paths. If a diagnostic tests fails, it indicates a bad data path. The integrity of the covered data paths is
verified when the diagnostic tests pass.
The diagnostic tests generally test data paths between multiple nodes, therefore error reports need to be
analyzed to narrow down the possible points of failure in a system.
All diagnostic tests run within the 1 second to 1 minute range.
Note
On the Cisco ASR 9000 Aggregation Services Router, only online diagnostics are supported.
To run a specified on-demand diagnostic test or series of tests, use the diagnostic start location
command.
Example:
RP/0/RSP0/CPU0:router(admin)# diagnostic start location 0/RSP1/CPU0 test 1
RP/0/RSP0/CPU0:router(admin)# diagnostic stop location 0/RSP1/CPU0
For details on the diagnostic commands, see Cisco ASR 9000 Series Aggregation Services Router
Interface and Hardware Component Command Reference.
Commands Used to Display Process and Thread Details
For details on processes and threads, see the “Understanding Processes and Threads” section in
Cisco ASR 9000 Aggregation Services Router Router Getting Started Guide.
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
OL-23591-02
1-59
Chapter 1
General Troubleshooting Procedures
Gathering Information Before You Call Cisco TAC
Cisco IOS XR Troubleshooting Guide for the Cisco ASR 9000 Aggregation Services Router
1-60
OL-23591-02