Cisco 10000 Series Router
Troubleshooting Guide
Corporate Headquarters
Cisco Systems, Inc.
170 West Tasman Drive
San Jose, CA 95134-1706
USA
http://www.cisco.com
Tel: 408 526-4000
800 553-NETS (6387)
Fax: 408 526-4100
Text Part Number: OL-0439-02
(DRAFT LABEL) ALPHA DRAFT - CISCO CONFIDENTIAL
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL
STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT
WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.
THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT
SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE
OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.
The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public
domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.
NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS” WITH
ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT
LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF
DEALING, USAGE, OR TRADE PRACTICE.
IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING,
WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO
OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
CCVP, the Cisco Logo, and the Cisco Square Bridge logo are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn is a service mark of
Cisco Systems, Inc.; and Access Registrar, Aironet, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, Cisco, the Cisco Certified Internetwork Expert logo,
Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unity, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step,
Follow Me Browsing, FormShare, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, iQuick Study,
LightStream, Linksys, MeetingPlace, MGX, Networking Academy, Network Registrar, Packet, PIX, ProConnect, RateMUX, ScriptShare, SlideCast, SMARTnet, StackWise,
The Fastest Way to Increase Your Internet Quotient, and TransPath are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other
countries.
All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship
between Cisco and any other company. (0609R)
Cisco 10000 Series Router Troubleshooting Guide
Copyright © 2006, Cisco Systems, Inc.
All rights reserved.
C O N T E N T S
About This Guide
Audience
vii
vii
Document Organization
vii
Related Documentation
viii
Document Conventions
viii
Obtaining Documentation ix
Cisco.com ix
Product Documentation DVD
Ordering Documentation x
Documentation Feedback
x
x
Cisco Product Security Overview x
Reporting Security Problems in Cisco Products
xi
Obtaining Technical Assistance xi
Cisco Technical Support & Documentation Website
Submitting a Service Request xii
Definitions of Service Request Severity xii
CHAPTER
1
xii
Obtaining Additional Publications and Information
xiii
Basic Troubleshooting Tasks and Startup Issues
1-1
Basic Troubleshooting Tasks
1-1
Confirming the Hardware Installation
1-1
Displaying Cisco IOS Software Version and System Environment Information 1-2
Using the show version Command 1-2
Using the show environment Command for Basic Troubleshooting 1-3
Hardware Troubleshooting Flowchart
1-4
Cisco 10000 Series Router System Startup Sequence
CHAPTER
2
PEM Faults and Blower Failures
DC PEM Faults
2-1
AC PEM Faults
2-3
Other Electrical Problems
Blower Module Faults
1-5
2-1
2-4
2-5
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
iii
Contents
CHAPTER
3
Troubleshooting PREs
3-1
Troubleshooting PREs 3-1
PRE Module Faults 3-1
Ethernet Connection Problems 3-3
Console Port Serial Connection Problems
CHAPTER
4
Troubleshooting Line Cards
3-4
4-1
Troubleshooting the OC-12 Packet-Over-SONET Line Card
Troubleshooting the Channelized T3 Line Card
Troubleshooting the Gigabit Ethernet Line Card
CHAPTER
5
4-1
4-4
4-6
Cisco 10000 Series Router Alarms and Error Messages
5-1
OC-12 POS Line Card Alarms and Error Messages 5-1
OC-12 POS Line Card SONET Alarms 5-1
OC-12 POS Line Card Bit Error Rate Alarms 5-3
OC-12 POS Line Card Error Messages 5-4
Channelized T3 Line Card Alarms and Error Messages
Channelized T3 Line Card Alarms 5-6
Channelized T3 Line Card Error Messages 5-7
Gigabit Ethernet Line Card Error Messages
CHAPTER
6
Troubleshooting Serial Lines
5-6
5-8
6-1
Optical Signal Input/Output Problems 6-1
Fiber-Optic Connections 6-1
Evaluating the Power Budget 6-2
Managing Receiver Overload 6-2
Using Bit Error Rate Tests 6-3
Configuring a BER Test on a T1 Line 6-3
Sending a BER Test Pattern on a T1 Line 6-4
Viewing the Results of a BER Test 6-5
Terminating a BER test 6-6
Using Loopback Tests 6-6
Configuring the Loopback Mode for a T3 Controller 6-7
Configuring a T3 Controller to Respond to Remote Loopback Commands
Configuring the Loopback Mode for a Gigabit Ethernet Interface 6-8
Configuring the Loopback Mode for an OC-12 POS Interface 6-8
6-7
Cisco 10000 Series Router Troubleshooting Guide
iv
OL-0439-02
Contents
CHAPTER
7
Replacing or Recovering Passwords
7-1
Password Recovery Procedure Overview
Password Recovery Procedure
7-1
7-2
APPENDIX
A
Unsupported Commands
APPENDIX
B
Recommended Tools and Test Equipment
A-1
B-1
Testing with Digital Multimeters and Cable Testers
B-1
Testing with TDRs and OTDRs B-2
Testing with TDRs B-2
Testing with OTDRs B-2
Testing with Breakout Boxes, Fox Boxes, and BERTs/BLERTs
Testing with Network Monitors
B-3
Testing with Network Analyzers
B-3
B-3
INDEX
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
v
Contents
Cisco 10000 Series Router Troubleshooting Guide
vi
OL-0439-02
About This Guide
This guide documents processes and procedures for user-level troubleshooting on the Cisco 10000 series
router. These are the troubleshooting steps that you can take before intervention from higher level
support agencies become necessary. For complete configuration instructions, refer to the
Cisco 10000 Series Router Line Card Hardware Installation Guide and the documents listed in the
“Related Documentation” section on page viii.
The Cisco 10000 series router supports the aggregation of leased lines and many high-density and
Layer 3 services. You configure, manage, and maintain the router by using the Cisco IOS command line
interface (CLI).
Audience
To benefit from this guide, you must be experienced using Cisco IOS and have some responsibility for
installing, configuring, or operating the Cisco 10000 series router.
Document Organization
The sections of this guide are as follows:
Chapter
Description
Table of Contents
Chapter 1: Basic
Troubleshooting Tasks and
Startup Issues
Basic procedures that users should perform before undertaking a
detailed troubleshooting analysis of the Cisco 10000 series router or
logging a case with the Cisco Technical Assistance Center (TAC).
Chapter 2: PEM Faults and
Blower Failures
Methods for troubleshooting faults involving the Cisco 10000 series
router Power Entry Modules (PEMs) and blower modules.
Chapter 3: Troubleshooting
PREs
How to troubleshoot PREs. It provides information on
troubleshooting PRE fault states, the management Ethernet port, and
the serial port.
Chapter 4: Troubleshooting
Line Cards
Troubleshooting faults for all following Cisco 10000 series router
line cards.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
vii
About This Guide
Related Documentation
Chapter
Description
Chapter 5: Cisco 10000
Series Router Alarms and
Error Messages
Alarms and error messages that can be raised on the Cisco 10000
series router.
Chapter 6: Troubleshooting
Serial Lines
Methods for detecting and correcting data errors on the Cisco 10000
series router serial interfaces.
Chapter 7: Replacing or
Recovering Passwords
How to recover a lost enable or console login password, and how to
replace a lost enable secret password on the Cisco 10000 series router.
Appendix A: Unsupported
IOS Commands
A list of the commands that are not supported in IOS
Release 12.0(9)SL for the Cisco 10000 series router.
Appendix B: Recommended
Tools and Test Equipment
A list of basic tools and test equipment necessary to perform
maintenance and troubleshooting tasks on the Cisco 10000 series
router.
Index
Related Documentation
When troubleshooting the Cisco 10000 series router, use the Cisco 10000 Series Router
Troubleshooting Guide in conjunction with the following documents:
•
Release notes for the Cisco 10000 series router—Provides the most up-to-date information about
software version requirements for using the router. It also provides information about bugs and
workarounds.
•
Cisco 10000 Series Router Line Card Hardware Installation Guide and Cisco 10000 Series Router
Line Card Configuration Guide—Contain detailed information on line card configuration and
administration.
For additional information about the Cisco 10000 series router, refer to the Cisco 10000 Series Router
Hardware Installation Guide. For more information about the Cisco IOS software that runs on the
router, refer to the Cisco IOS command reference books and configuration guides.
Document Conventions
Command descriptions use the following conventions:
boldface font
Commands and keywords are in boldface.
italic font
Arguments for which you supply values are in italics.
[ ]
Elements in square brackets are optional.
{x|y|z}
Alternative keywords are grouped in braces and separated
by vertical bars.
[x|y|z]
Optional alternative keywords are grouped in brackets and
separated by vertical bars.
Cisco 10000 Series Router Troubleshooting Guide
viii
OL-0439-02
About This Guide
Obtaining Documentation
Screen examples use the following conventions:
screen font
Terminal sessions and information the system displays are in
screen font.
boldface screen
Information you must enter is in boldface screen font.
font
italic screen font
Arguments for which you supply values are in italic screen
font.
< >
Nonprinting characters, such as passwords, are in angle
brackets.
[ ]
Default responses to system prompts are in square
brackets.
!, #
An exclamation point (!) or a pound sign (#) at the
beginning of a line of code indicates a comment line.
Notes use the following conventions:
Note
Means reader take note. Notes contain helpful suggestions or references to material not covered in the
publication.
Cautions use the following conventions:
Caution
Means reader be careful. In this situation, you might do something that could result in equipment
damage or loss of data.
Obtaining Documentation
Cisco documentation and additional literature are available on Cisco.com. Cisco also provides several
ways to obtain technical assistance and other technical resources. These sections explain how to obtain
technical information from Cisco Systems.
Cisco.com
You can access the most current Cisco documentation at this URL:
http://www.cisco.com/techsupport
You can access the Cisco website at this URL:
http://www.cisco.com
You can access international Cisco websites at this URL:
http://www.cisco.com/public/countries_languages.shtml
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
ix
About This Guide
Documentation Feedback
Product Documentation DVD
The Product Documentation DVD is a comprehensive library of technical product documentation on a
portable medium. The DVD enables you to access multiple versions of installation, configuration, and
command guides for Cisco hardware and software products. With the DVD, you have access to the same
HTML documentation that is found on the Cisco website without being connected to the Internet.
Certain products also have .PDF versions of the documentation available.
The Product Documentation DVD is available as a single unit or as a subscription. Registered Cisco.com
users (Cisco direct customers) can order a Product Documentation DVD (product number
DOC-DOCDVD= or DOC-DOCDVD=SUB) from Cisco Marketplace at this URL:
http://www.cisco.com/go/marketplace/
Ordering Documentation
Registered Cisco.com users may order Cisco documentation at the Product Documentation Store in the
Cisco Marketplace at this URL:
http://www.cisco.com/go/marketplace/
Nonregistered Cisco.com users can order technical documentation from 8:00 a.m. to 5:00 p.m.
(0800 to 1700) PDT by calling 1 866 463-3487 in the United States and Canada, or elsewhere by
calling 011 408 519-5055. You can also order documentation by e-mail at
tech-doc-store-mkpl@external.cisco.com or by fax at 1 408 519-5001 in the United States and Canada,
or elsewhere at 011 408 519-5001.
Documentation Feedback
You can rate and provide feedback about Cisco technical documents by completing the online feedback
form that appears with the technical documents on Cisco.com.
You can submit comments about Cisco documentation by using the response card (if present) behind the
front cover of your document or by writing to the following address:
Cisco Systems
Attn: Customer Document Ordering
170 West Tasman Drive
San Jose, CA 95134-9883
We appreciate your comments.
Cisco Product Security Overview
Cisco provides a free online Security Vulnerability Policy portal at this URL:
http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html
From this site, you will find information about how to:
•
Report security vulnerabilities in Cisco products.
•
Obtain assistance with security incidents that involve Cisco products.
•
Register to receive security information from Cisco.
Cisco 10000 Series Router Troubleshooting Guide
x
OL-0439-02
About This Guide
Obtaining Technical Assistance
A current list of security advisories, security notices, and security responses for Cisco products is
available at this URL:
http://www.cisco.com/go/psirt
To see security advisories, security notices, and security responses as they are updated in real time, you
can subscribe to the Product Security Incident Response Team Really Simple Syndication (PSIRT RSS)
feed. Information about how to subscribe to the PSIRT RSS feed is found at this URL:
http://www.cisco.com/en/US/products/products_psirt_rss_feed.html
Reporting Security Problems in Cisco Products
Cisco is committed to delivering secure products. We test our products internally before we release
them, and we strive to correct all vulnerabilities quickly. If you think that you have identified a
vulnerability in a Cisco product, contact PSIRT:
•
For Emergencies only — security-alert@cisco.com
An emergency is either a condition in which a system is under active attack or a condition for which
a severe and urgent security vulnerability should be reported. All other conditions are considered
nonemergencies.
•
For Nonemergencies — psirt@cisco.com
In an emergency, you can also reach PSIRT by telephone:
Tip
•
1 877 228-7302
•
1 408 525-6532
We encourage you to use Pretty Good Privacy (PGP) or a compatible product (for example, GnuPG) to
encrypt any sensitive information that you send to Cisco. PSIRT can work with information that has been
encrypted with PGP versions 2.x through 9.x.
Never use a revoked or an expired encryption key. The correct public key to use in your correspondence
with PSIRT is the one linked in the Contact Summary section of the Security Vulnerability Policy page
at this URL:
http://www.cisco.com/en/US/products/products_security_vulnerability_policy.html
The link on this page has the current PGP key ID in use.
If you do not have or use PGP, contact PSIRT at the aforementioned e-mail addresses or phone numbers
before sending any sensitive material to find other means of encrypting the data.
Obtaining Technical Assistance
Cisco Technical Support provides 24-hour-a-day award-winning technical assistance. The Cisco
Technical Support & Documentation website on Cisco.com features extensive online support resources.
In addition, if you have a valid Cisco service contract, Cisco Technical Assistance Center (TAC)
engineers provide telephone support. If you do not have a valid Cisco service contract, contact your
reseller.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
xi
About This Guide
Obtaining Technical Assistance
Cisco Technical Support & Documentation Website
The Cisco Technical Support & Documentation website provides online documents and tools for
troubleshooting and resolving technical issues with Cisco products and technologies. The website is
available 24 hours a day, at this URL:
http://www.cisco.com/techsupport
Access to all tools on the Cisco Technical Support & Documentation website requires a Cisco.com user
ID and password. If you have a valid service contract but do not have a user ID or password, you can
register at this URL:
http://tools.cisco.com/RPF/register/register.do
Note
Use the Cisco Product Identification (CPI) tool to locate your product serial number before submitting
a web or phone request for service. You can access the CPI tool from the Cisco Technical Support &
Documentation website by clicking the Tools & Resources link under Documentation & Tools. Choose
Cisco Product Identification Tool from the Alphabetical Index drop-down list, or click the Cisco
Product Identification Tool link under Alerts & RMAs. The CPI tool offers three search options: by
product ID or model name; by tree view; or for certain products, by copying and pasting show command
output. Search results show an illustration of your product with the serial number label location
highlighted. Locate the serial number label on your product and record the information before placing a
service call.
Submitting a Service Request
Using the online TAC Service Request Tool is the fastest way to open S3 and S4 service requests. (S3
and S4 service requests are those in which your network is minimally impaired or for which you require
product information.) After you describe your situation, the TAC Service Request Tool provides
recommended solutions. If your issue is not resolved using the recommended resources, your service
request is assigned to a Cisco engineer. The TAC Service Request Tool is located at this URL:
http://www.cisco.com/techsupport/servicerequest
For S1 or S2 service requests, or if you do not have Internet access, contact the Cisco TAC by telephone.
(S1 or S2 service requests are those in which your production network is down or severely degraded.)
Cisco engineers are assigned immediately to S1 and S2 service requests to help keep your business
operations running smoothly.
To open a service request by telephone, use one of the following numbers:
Asia-Pacific: +61 2 8446 7411 (Australia: 1 800 805 227)
EMEA: +32 2 704 55 55
USA: 1 800 553-2447
For a complete list of Cisco TAC contacts, go to this URL:
http://www.cisco.com/techsupport/contacts
Definitions of Service Request Severity
To ensure that all service requests are reported in a standard format, Cisco has established severity
definitions.
Cisco 10000 Series Router Troubleshooting Guide
xii
OL-0439-02
About This Guide
Obtaining Additional Publications and Information
Severity 1 (S1)—An existing network is down, or there is a critical impact to your business operations.
You and Cisco will commit all necessary resources around the clock to resolve the situation.
Severity 2 (S2)—Operation of an existing network is severely degraded, or significant aspects of your
business operations are negatively affected by inadequate performance of Cisco products. You and Cisco
will commit full-time resources during normal business hours to resolve the situation.
Severity 3 (S3)—Operational performance of the network is impaired, while most business operations
remain functional. You and Cisco will commit resources during normal business hours to restore service
to satisfactory levels.
Severity 4 (S4)—You require information or assistance with Cisco product capabilities, installation, or
configuration. There is little or no effect on your business operations.
Obtaining Additional Publications and Information
Information about Cisco products, technologies, and network solutions is available from various online
and printed sources.
•
The Cisco Product Quick Reference Guide is a handy, compact reference tool that includes brief
product overviews, key features, sample part numbers, and abbreviated technical specifications for
many Cisco products that are sold through channel partners. It is updated twice a year and includes
the latest Cisco offerings. To order and find out more about the Cisco Product Quick Reference
Guide, go to this URL:
http://www.cisco.com/go/guide
•
Cisco Marketplace provides a variety of Cisco books, reference guides, documentation, and logo
merchandise. Visit Cisco Marketplace, the company store, at this URL:
http://www.cisco.com/go/marketplace/
•
Cisco Press publishes a wide range of general networking, training and certification titles. Both new
and experienced users will benefit from these publications. For current Cisco Press titles and other
information, go to Cisco Press at this URL:
http://www.ciscopress.com
•
Packet magazine is the Cisco Systems technical user magazine for maximizing Internet and
networking investments. Each quarter, Packet delivers coverage of the latest industry trends,
technology breakthroughs, and Cisco products and solutions, as well as network deployment and
troubleshooting tips, configuration examples, customer case studies, certification and training
information, and links to scores of in-depth online resources. You can access Packet magazine at
this URL:
http://www.cisco.com/packet
•
iQ Magazine is the quarterly publication from Cisco Systems designed to help growing companies
learn how they can use technology to increase revenue, streamline their business, and expand
services. The publication identifies the challenges facing these companies and the technologies to
help solve them, using real-world case studies and business strategies to help readers make sound
technology investment decisions. You can access iQ Magazine at this URL:
http://www.cisco.com/go/iqmagazine
or view the digital edition at this URL:
http://ciscoiq.texterity.com/ciscoiq/sample/
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
xiii
About This Guide
Obtaining Additional Publications and Information
•
Internet Protocol Journal is a quarterly journal published by Cisco Systems for engineering
professionals involved in designing, developing, and operating public and private internets and
intranets. You can access the Internet Protocol Journal at this URL:
http://www.cisco.com/ipj
•
Networking products offered by Cisco Systems, as well as customer support services, can be
obtained at this URL:
http://www.cisco.com/en/US/products/index.html
•
Networking Professionals Connection is an interactive website for networking professionals to
share questions, suggestions, and information about networking products and technologies with
Cisco experts and other networking professionals. Join a discussion at this URL:
http://www.cisco.com/discuss/networking
•
World-class networking training is available from Cisco. You can view current offerings at
this URL:
http://www.cisco.com/en/US/learning/index.html
Cisco 10000 Series Router Troubleshooting Guide
xiv
OL-0439-02
C H A P T E R
1
Basic Troubleshooting Tasks and Startup Issues
This chapter describes the basic procedures that users should perform before undertaking a detailed
troubleshooting analysis of the Cisco 10000 series router or logging a case with the Cisco Technical
Assistance Center (TAC).
These basic troubleshooting checks are organized as follows:
•
Basic Troubleshooting Tasks, page 1-1
•
Confirming the Hardware Installation, page 1-1
•
Displaying Cisco IOS Software Version and System Environment Information, page 1-2
•
Hardware Troubleshooting Flowchart, page 1-4
•
Cisco 10000 Series Router System Startup Sequence, page 1-5
Basic Troubleshooting Tasks
If you encounter a problem after you install the Cisco 10000 series router, perform the following
troubleshooting checks on the equipment before you contact the Cisco Technical Assistance Center
(TAC) or before you perform a detailed troubleshooting analysis:
1.
Is the power on?
2.
Is the PRE seated securely in the chassis?
3.
Are the other line cards firmly seated and securely screwed to the chassis?
4.
Are power leads and data cables firmly connected at both ends?
5.
Are the blower module and PEMs properly connected and secured to the chassis?
6.
Are the ports properly configured? Refer to the Cisco 10000 Series Router Line Card Hardware
Installation Guide for configuration examples.
Confirming the Hardware Installation
Start troubleshooting the installation by issuing the show hardware command. The show hardware
command displays all hardware components that are recognized by the system. These components can
include
•
Performance Routing Engines (PREs)
•
Gigabit Ethernet line cards (GEs)
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
1-1
Chapter 1
Basic Troubleshooting Tasks and Startup Issues
Displaying Cisco IOS Software Version and System Environment Information
•
OC-12 Packet Over SONET (POS) line cards
•
CT3 Subscriber line cards
If an installed item does not appear in the command output, make sure the item is properly installed. For
example, make sure the line cards are fully inserted into the slot and the captive screws are tightened. If
the problem persists, consult the Cisco 10000 series router release notes to confirm that this is not an
existing problem. Finally, you should consider replacing the component.
The following example is sample output from the show hardware command:
Router# show hardware
Cisco Internetwork Operating System Software
IOS (tm) 10000 Software (C10K-P6-M), Experimental Version
12.0(20000308:073152) [20000308:020003 104]
Copyright (c) 1986-2000 by Cisco Systems, Inc.
Compiled Wed 08-Mar-00 04:49 by chrel
Image text-base:0x60008900, data-base:0x60A44000
ROM:System Bootstrap, Version 12.0(20000303:180332)
[REL-omega_isp.ios-rommon 101], DEVELOPMENT SOFTWARE
BOOTFLASH:10000 Software (C10K-EBOOT-M), Experimental Version
12.0(20000307:074438) [20000307:020003 105]
omega-1 uptime is 9 minutes
System returned to ROM by power-on
Running default software
cisco C10000 (CRE-RP) processor with 229376K/32768K bytes of memory.
Processor board ID dt-omega-1_
R7000 CPU at 262Mhz, Implementation 39, Rev 1.0, 256KB L2, 2048KB L3 Cache
Unknown midplane, Version 1.0
Last reset from power-on
Toaster processor tmc0 is running.
Toaster processor tmc1 is running.
1 Ethernet/IEEE 802.3 interface(s)
1 FastEthernet/IEEE 802.3 interface(s)
1 GigabitEthernet/IEEE 802.3 interface(s)
257 Serial network interface(s)
1 Packet over SONET network interface(s)
12 Channelized T3 port(s)
509K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128KB).
40960K bytes of ATA PCMCIA card at slot 1 (Sector size 512 bytes).
16384K bytes of Flash internal SIMM (Sector size 256KB).
Configuration register is 0x2
Displaying Cisco IOS Software Version and System Environment
Information
Run the show version and show environment commands to determine if the Cisco 10000 series
router is operating within acceptable bounds.
Using the show version Command
Use the show version command to confirm that you are running the proper version of IOS and have a
sufficient amount of system memory. The command also reports the system uptime and the method by
which the system was powered up.
Cisco 10000 Series Router Troubleshooting Guide
1-2
OL-0439-02
Chapter 1
Basic Troubleshooting Tasks and Startup Issues
Displaying Cisco IOS Software Version and System Environment Information
In the following sample of output from the show version command, some of the information that may
be useful for troubleshooting appears in bold type:
Router# show version
Cisco Internetwork Operating System Software
IOS (tm) 10K Software (C10K-EBOOT-M), Experimental Version 12.0(19990916:085921)
[BLD-omega_isp.ios-weekly 101]
Copyright (c) 1986-1999 by Cisco Systems, Inc.
Compiled Thu 16-Sep-99 05:20 by
Image text-base: 0x60008900, data-base: 0x6047A000
ROM: System Bootstrap, Version 12.0(19990719:134935) [fyang-cre1 1004], DEVELOPM
ENT SOFTWARE
another-omega uptime is 6 hours, 7 minutes
System returned to ROM by power-on at 14:12:52 EST Fri Jan 7 2000
System image file is "bootflash:c10k-eboot-mz"
cisco C10000 (CRE-RP) processor with 114688K/16384K bytes of memory.
Processor board ID noserial
R7000 CPU at 262Mhz, Implementation 39, Rev 1.0, 256KB L2, 2048KB L3 Cache
Unknown midplane, Version 0.0
Last reset from register reset
1 Ethernet/IEEE 802.3 interface(s)
1 FastEthernet/IEEE 802.3 interface(s)
509K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128KB).
Configuration register is 0x0
Using the show environment Command for Basic Troubleshooting
Use the show environment command to
•
Make sure the system operating temperature is equal to or less than 40 degrees C or 104 degrees F
at the inlet.
•
Report the operational status of the PEMs and blower
If the operating temperature is not between 41 degrees F and 104 degrees F, refer to the “Blower Module
Faults” section on page 2-5.
The following example is sample output from the show environment command:
Router# show environment
Temperature normal:chassis inlet measured at 29C/84F
Temperature normal:chassis core measured at 39C/102F
Fan: OK
Power Entry Module 0 type DC status: OK
Power Entry Module 1 type AC status: Failure
The Cisco 10000 series router default temperature alarm thresholds are programmed to assert at
approximately these conditions:
•
Minor inlet and core alarms will assert when the ambient temperature is within a few degrees of 40
degrees C.
•
Major inlet and core alarms will assert when the ambient temperature is within a few degrees of 50
degrees C.
•
Critical inlet and core alarms will assert when the ambient temperature is within a few degrees of
70 degrees C.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
1-3
Chapter 1
Basic Troubleshooting Tasks and Startup Issues
Hardware Troubleshooting Flowchart
A malfunctioning blower or blocked air passages could result in these alarms asserting at lower ambient
temperatures.
For customer sites where ambient temperatures are frequently close to 40 degrees C, and/or for
customers who do not wish to receive these alarms, the minor and major thresholds can be adjusted.
Hardware Troubleshooting Flowchart
Use Figure 1-1 to determine which component of your Cisco 10000 series router is malfunctioning.
Figure 1-1 describes a series of hardware dependent startup events that must take place for a Cisco 10000
series router to allow the passage of IP traffic. At each main point of the flowchart, there are pointers to
the chapters in this guide that describe how to troubleshoot individual pieces of hardware.
Note
This flowchart does not address software configuration problems.
Figure 1-1
Hardware Troubleshooting Flowchart
Turn
on
PEM
Correct
input
wiring
No
PEM
No
Power OK
LED on
?
Troubleshoot
PEM
Yes
PRE
No
Status
LEDs OK
?
Yes
Reseat
PRE
and restart
No
Reseat
line card
and restart
Line card
LEDs on
?
Yes
System
startup
successful
PEM
No
Power OK
LED on
?
Yes
PRE
No
status LEDs
OK?
See section
"PEM Faults"
See section
"PRE Module
Faults"
Yes
See appropriate
line card fault
Line
No section
card LEDs
OK
?
Yes
Obtain
technical
assistance
38680
*Miswire
LED
off
?
Yes
Cisco 10000 Series Router Troubleshooting Guide
1-4
OL-0439-02
Chapter 1
Basic Troubleshooting Tasks and Startup Issues
Cisco 10000 Series Router System Startup Sequence
Cisco 10000 Series Router System Startup Sequence
Table 1-1 describes the visible sequence of events that occur during a typical Cisco 10000 series router
power up.
Table 1-1
Cisco 10000 Series Router System Startup Sequence
Startup Event
Event Description
PEM is switched off
The AC or DC PEM Fault LED is lit.
Power on the
router
1.
The yellow Critical, Major, and Minor alarm and Fail LEDs illuminate
for about 2 seconds.
2.
The alphanumeric display on the PRE counts up through a range of
numbers from 1111 to 9999 (1111, 2222, and so on).
3.
The alphanumeric display counts up through a sequence of letters from
AAA to CCC (AAA, BBB, and CCC).
4.
The message ROM DONE appears on the alphanumeric display.
If the system is not configured to auto boot, it stops at the ROM
DONE message. The console displays a rommon> prompt.
Note
IOS loads
1.
If the system is set to boot from the slot0: file system, the green slot 1
LED lights.
2.
The message BOOT IMGE appears on the alphanumeric display.
3.
The console displays a series of pound signs (#) as the IOS software
image is decompressed.
4.
The following messages appear on the alphanumeric display:
•
IOS STRT
•
IOS EXC
•
IOS FPGA
•
IOS FPOK
•
IOS FILE
•
IOS STBY
•
IOS DRVR
•
IOS LIB
•
IOS MGMT
•
IOS CONF
The console displays:
...
Press RETURN to get started!
5.
The following appears in the alphanumeric display: - - - In a redundant configuration, the message IOS STBY appears on the
alphanumeric display of the secondary PRE after a successful startup.
If the boot process fails, no console access is available. If you cannot
boot the Cisco 10000 series router, call the Cisco TAC.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
1-5
Chapter 1
Basic Troubleshooting Tasks and Startup Issues
Cisco 10000 Series Router System Startup Sequence
Cisco 10000 Series Router Troubleshooting Guide
1-6
OL-0439-02
C H A P T E R
2
PEM Faults and Blower Failures
The following sections provide methods for troubleshooting faults involving the Cisco 10000 series
router Power Entry Modules (PEMs) and blower modules.
This chapter contains the following major sections:
•
DC PEM Faults, page 2-1
•
AC PEM Faults, page 2-3
•
Other Electrical Problems, page 2-4
•
Blower Module Faults, page 2-5
DC PEM Faults
DC PEMs can be installed in two modes—redundant and nonredundant. It is recommended that you
install PEMs in a redundant configuration. A PEM failure in a nonredundant configuration causes a
system failure.
Tip
To quickly check the functional status of your PEMs, use the show environment command.
DC PEM faults can occur for the following reasons:
•
PEM failure
•
Reversed power cables
•
Backplane interface failures or damage
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
2-1
Chapter 2
PEM Faults and Blower Failures
DC PEM Faults
Figure 2-1 illustrates the DC PEM and its indicators. Table 2-1 describes the indicators.
DC PEM
30003
Figure 2-1
POWER
FAULT
MISWIR
E
Table 2-1
DC PEM LEDs
LED
Description
Power (green)
PEM is powered on and is operational.
Fault (yellow)
PEM is not operating correctly or the circuit breaker is in the off position.
Miswire (yellow)
Input power cables are wired incorrectly and should be reversed.
Table 2-2 lists the DC PEM fault symptoms and corrective actions.
Table 2-2
DC PEM Fault Symptoms and Corrective Action
Fault Symptom
Corrective Action
1.
Make sure the circuit breaker on the PEM is turned on.
2.
Make sure the PEM is properly seated and screwed in place.
3.
Make sure power leads are properly connected to power connectors
on the backplane. If connections are loose or their polarity is
reversed, chassis does not receive power.
4.
Check the external power source.
5.
Move the PEM to the other PEM slot. If the PEM still fails, replace it.
PEM experiences problems
in one slot but operates
normally in a different slot
1.
Ensure that the input power to both slots is correct.
2.
If the problem persists, contact the Cisco TAC.
Fault LED is lit yellow
1.
Check to see if the circuit breaker (on/off switch) has tripped. If it
has, return the switch to the on position.
2.
Replace PEM with a known good replacement.
3.
Contact the Cisco TAC.
Green LED on PEM fails to
light
Miswire LED is lit yellow
If the MISWIRE LED is on, the power cables are reversed. Power off the
PEM and the external power source and reconnect the wires correctly.
See the Cisco 10000 Series Router Hardware Installation Guide.
Cisco 10000 Series Router Troubleshooting Guide
2-2
OL-0439-02
Chapter 2
PEM Faults and Blower Failures
AC PEM Faults
Tip
Securely tighten the captive screws on your PEMs to prevent heightened levels of electromagnetic
interference.
AC PEM Faults
AC PEMs can be installed in two modes—redundant and nonredundant. It is recommended that you
install PEMs in a redundant configuration. A single PEM failure in a nonredundant configuration causes
a system failure.
Tip
To quickly check the status of your PEMs, use the show environment command.
AC PEM faults can occur for the following reasons:
•
PEM failure
•
Backplane interface failures or damage
Figure 2-2 shows the AC PEM and its LED indicators.
AC PEM
30004
Figure 2-2
POWER
FAULT
Table 2-3 describes the AC PEM LEDs located at the bottom of the front panel.
Table 2-3
AC PEM LEDs
LED
Status
Condition
POWER
Steady green
Power is available.
FAULT
Steady yellow
PEM has failed or is turned off.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
2-3
Chapter 2
PEM Faults and Blower Failures
Other Electrical Problems
Table 2-4 lists the AC PEM fault indications and recommended actions.
Table 2-4
AC PEM Fault Indications and Recommended Actions
Symptom
Power LED on PEM fails to light
Fault LED is lit yellow
Steps to Take
1.
Make sure the power switch is in the on position.
2.
Make sure the PEM is properly seated and screwed
into place.
3.
Make sure the power cable is plugged in properly.
4.
Check the power source.
5.
Move the PEM to the other PEM slot. If the PEM
still fails, replace it.
6.
If the problem persists, contact the Cisco TAC.
1.
Make sure that the PEM is switched on.
2.
Replace the PEM.
3.
Call the Cisco TAC.
Other Electrical Problems
If the electrical problem cannot be traced to a PEM, check the unit for:
•
Improper power cable connections to the Cisco 10000 series router
•
Improper installation of other field-replaceable units (FRUs)
Check the site for:
•
Improperly grounded equipment, particularly equipment racks and power grounds
•
Fluctuating voltage, which can result from excessive power drains caused by other equipment (such
as air conditioning units)
•
Cable corrosion or defective power panels, circuit breakers or fuses, or cable connections
•
Undersized power cables or excessive power cable lengths
•
Excessive power demand on backup power systems or batteries when alternate power sources
are used
Cisco 10000 Series Router Troubleshooting Guide
2-4
OL-0439-02
Chapter 2
PEM Faults and Blower Failures
Blower Module Faults
Blower Module Faults
The blower module is critical to the operation of the Cisco 10000 series router. Severe overheating can
result in system failure. The blower module contains a redundant array of fans in a single shelf unit.
Figure 2-3 shows the blower module front panel and its LED indicators.
Figure 2-3
Blower Module
FANS
OK
FAN
FAILURE
MULTIFAN
FAILURE
30002
When hot CAUTION
removal swapping
this fan
be done and replacem
tray,
system in under two ent must
shutdow
minu
n will occutes or
r.
Table 2-5 lists the blower module fault indications and recommended actions.
Table 2-5
Blower Module Fault Indications and Recommended Action
Symptom
Fans OK LED on blower
module fails to power on
Steps to Take
1.
Make sure the blower module is fully inserted into the chassis.
2.
If you use DC PEMs, make sure the wiring is not reversed.
3.
Replace the blower module.
FAN FAILURE LED is lit
Replace the blower module.
MULTI-FAN FAILURE
LED is lit
Replace the blower module.
Fans run but the system
overheats
1.
Make sure that all intake and exhaust vents on the front and rear
of the chassis are free of blockages.
2.
Make sure that the ambient temperature and other environmental
factors in the system area are within the ranges specified in the
“Using the show environment Command for Basic
Troubleshooting” section on page 1-3.
3.
Make sure all line cards and blank faceplates are in place. The
cooling system cannot operate effectively unless the chassis is
fully enclosed.
4.
Check the air filter, and if necessary clean or replace it.
5.
Reduce the ambient temperature of the area surrounding the router
chassis.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
2-5
Chapter 2
PEM Faults and Blower Failures
Blower Module Faults
Cisco 10000 Series Router Troubleshooting Guide
2-6
OL-0439-02
C H A P T E R
3
Troubleshooting PREs
Troubleshooting PREs
This section describes how to troubleshoot Performance Routing Engines (PREs). It provides
information on troubleshooting PRE fault states, the management Ethernet port, and the serial port.
PRE Module Faults
The PRE provides the IP routing and forwarding functionality in the Cisco 10000 series router. Thus, in
a nonredundant PRE configuration, a PRE failure is a system failure. A redundant PRE configuration is
recommended.
If the PRE fails, the yellow PRE STATUS LED lights. Try rebooting the Cisco 10000 series router,
moving the card to a different PRE slot, or replacing the PRE with another one. In any event, you should
capture any error messages that appear on the console, as well as the state of the PRE LEDs and
alphanumeric display. Then you should contact the Cisco Technical Assistance Center (TAC).
Figure 3-1 describes the LED indicators on the PRE faceplate. Use these descriptions to verify the
operation of the PRE.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
3-1
Chapter 3
Troubleshooting PREs
Troubleshooting PREs
Figure 3-1
CISCO
10000
CO
NS
OL
PRE Faceplate and LEDs
LED
Ethernet port LEDs
Status
Description
Activity
Green
Packets are being transmitted
and received.
Off
No activity.
Green
Carrier detected; the port is
able to pass traffic.
Off
No carrier detected; the port is
not able to pass traffic.
PCMCIA slot 0
Green
Slot 0 is active.
PCMCIA slot 1
Critical, Major, and
Minor LEDs
Green
Off
Slot 1 is active.
No alarm.
Yellow
Indicates an alarm condition.
E
AU
X
AC
T
ET
IV
HE
LIN
SL
SL
K
IT
Y
RN
ET
OT
OT
Link
0
1
Pressing this switch disables
–
Alarm Cut-off (ACO)
an audible alarm.
switch
Status
Flashing System is booting.
yellow
ALARMS
AC
O
IT
MA
Green
IC
JO
ST
AT
US
FA
IL
PRE is active (primary).
Flashing PRE is standby (secondary).
green
R
OR
PERFORMANCE ROUTING ENGINE
MIN
AL
Fail
Off
No power to PRE.
Yellow
A major failure has disabled
the PRE.
Off
The PRE is operating properly.
29995
CR
Cisco 10000 Series Router Troubleshooting Guide
3-2
OL-0439-02
Chapter 3
Troubleshooting PREs
Troubleshooting PREs
Table 3-1 lists the PRE fault indications and recommended actions. The information contained in the
table is based on the assumption that you have a nonredundant configuration.
Table 3-1
PRE Module Fault Indications and Recommended Action
Fault
STATUS LED is not lit
FAIL LED is yellow,
indicating that the PRE
failed
The PRE initializes, but
you cannot establish a
console connection
Card cannot be fully
inserted into its slot
An alarm LED is lit
Steps to Take
1.
Check LEDs on other modules and cards. If none are lit, refer to
Table 2-1 or Table 2-3 depending upon the PEM type in your chassis.
2.
If LEDs on other modules and cards are lit, remove the card from its
slot and check for bent or broken pins on the backplane. Return the card
to its slot and screw it firmly into place.
3.
Replace the card.
4.
If the problem persists, contact the Cisco TAC.
1.
Reinsert the PRE.
2.
Insert a new PRE.
3.
If the problem persists, contact the Cisco TAC.
1.
Ensure that the terminal settings are properly set.
2.
If you still cannot connect, check the console cable. Is it firmly
connected? Is it the correct type of cable with proper connectors?
3.
If the cable checks out and you cannot establish a console or Telnet
session, reinsert the PRE module. If the problem persists, replace
the PRE.
4.
Enter show log to review console messages recorded in the system log.
Make sure that you are using slot 0A or 0B for the PRE module.
1.
Enter the show facility-alarm status command and examine the output
to determine which system component raised the alarm.
2.
Troubleshoot using a procedure appropriate to the module or FRU
responsible for the alarm. Refer to the following sections for
information on interpreting alarms:
•
OC-12 POS Line Card Alarms and Error Messages, page 5-1
•
Channelized T3 Line Card Alarms and Error Messages, page 5-6
Ethernet Connection Problems
If the management Ethernet connection on the PRE fails to work properly, and the corresponding Link
LED is not lit (steady green):
•
Visually check that an Ethernet cable is connected to the correct Ethernet port on the
Cisco 10000 series router.
•
Verify that you are using the correct type of cable for a 100BASE-T Ethernet.
•
Check to see if the cable is bad or broken.
•
Make sure the PRE module booted up properly by checking the Status LED on its faceplate. This
LED should be steady green. If necessary, remove and reinsert the module and boot it up again.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
3-3
Chapter 3
Troubleshooting PREs
Troubleshooting PREs
If the Link LED is lit (steady green), but the Ethernet port is not working properly, make sure that the
port in question is configured properly and is not administratively shut down. If you have a working
console connection, perform the following steps:
Step 1
At the switch prompt, enter show interface fastethernet0/0/0. If the port is administratively down, enter
these commands to enable it:
c10000# configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
c10000(config)#interface fastethernet0/0/0
c10000(config-if)# no shut
c10000(config-if)# exit
c10000(config)# exit
c10000#
Step 2
Check that the Ethernet port in question is assigned a valid IP address.
For more information about configuring Ethernet ports, refer to the Cisco 10000 Series Router Line Card
Hardware Installation Guide.
If the cable, connections, power, and configuration all check out, and you still cannot connect to the
Ethernet port on the module, replace the module in question. If the problem persists, contact the Cisco
TAC for further assistance. Refer to the “Obtaining Technical Assistance” section on page -xi for
instructions on contacting the Cisco TAC.
Console Port Serial Connection Problems
If the console screen connected to a Cisco 10000 series router console port appears frozen or fails to
work properly
•
Refer to the “Cisco 10000 Series Router System Startup Sequence” section on page 1-5. If the
display stops responding during this process, there is no console output.
•
Check the console cable and make sure it is properly connected to the correct console port on the
Cisco 10000 series router system at one end and to your terminal equipment or terminal server at
the other end.
•
Verify that you are using the right type of cable and adapter. For information about pin-out
connections and installation instructions, refer to the Cisco 10000 Series Router Hardware
Installation Guide.
•
Make sure the cable is not defective or broken. Replace the cable with another high quality cable if
possible, and check to see if the console port starts working.
•
Check that the terminal equipment is configured with the correct settings for the console port. The
default console port settings are:
– 9600 baud
– 8 data bits
– 1 stop bit
– No parity
– No flow control
Cisco 10000 Series Router Troubleshooting Guide
3-4
OL-0439-02
Chapter 3
Troubleshooting PREs
Troubleshooting PREs
•
Check the LEDs on the PRE faceplate to make sure it has powered up properly. If necessary, remove
and reinsert both PRE modules to power them up again. Also, make sure the terminal equipment is
working properly.
If the cable, connections, power, and terminal settings all check out and you still cannot connect to the
console port on the module, replace the module in question. If the problem persists, contact the Cisco
TAC for further assistance.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
3-5
Chapter 3
Troubleshooting PREs
Troubleshooting PREs
Cisco 10000 Series Router Troubleshooting Guide
3-6
OL-0439-02
C H A P T E R
4
Troubleshooting Line Cards
This chapter discusses troubleshooting faults on the following Cisco 10000 series router line cards:
•
Troubleshooting the OC-12 Packet-Over-SONET Line Card, page 4-1
•
Troubleshooting the Channelized T3 Line Card, page 4-4
•
Troubleshooting the Gigabit Ethernet Line Card, page 4-6
Troubleshooting the OC-12 Packet-Over-SONET Line Card
Figure 4-1 describes the LEDs on the OC-12 Packet-Over-SONET (POS) line card faceplate. Use these
descriptions to verify the operation of the OC-12 POS line card.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
4-1
Chapter 4
Troubleshooting Line Cards
Troubleshooting the OC-12 Packet-Over-SONET Line Card
CISCO
10000
OC-12 POS line card LEDs
IL
FA
LED
Fail
Rx (receive)
Tx (transmit)
TX
RX
R
IE
RR
CA
Carrier
0
Status
Yellow
Description
A major failure has disabled the
line card.
Off
The line card is operating
properly.
Green
Receiving traffic.
Off
Not receiving traffic.
Green
Transmitting traffic.
Off
Not transmitting traffic.
Green
Off
Carrier detected.
Carrier not detected.
29993
Figure 4-1
OC–12/STM–4 POS SM–IR
Cisco 10000 Series Router Troubleshooting Guide
4-2
OL-0439-02
Chapter 4
Troubleshooting Line Cards
Troubleshooting the OC-12 Packet-Over-SONET Line Card
Table 4-1 describes fault conditions on the OC-12 POS line card and recommended corrective actions.
Table 4-1
OC-12 POS Line Card Fault Indications and Recommended Action
Fault
Corrective Action
1.
Reinsert the line card.
2.
Replace the line card.
3.
Insert the line card in another slot.
4.
Contact the Cisco TAC.
1.
Check the fiber optic cable, making sure that it is properly
attached at both ends.
2.
Make sure that you are using the proper fiber type.
3.
Check the cable connections on the remote equipment.
4.
Contact the Cisco TAC.
Fail LED blinks then lights
steadily repeatedly
1.
Check for bent pins on the backplane.
2.
If there are no bent pins, replace with a new line card.
or
3.
Try inserting the line card in a different slot.
Fail LED is lit indicating that
the line card has failed
Carrier LED is off and
interface is enabled
Card seems to be passing
traffic (Tx/Rx lights), but
cannot communicate with the
PRE
If the card works in a different slot, the Cisco 10000 series router
backplane may be defective. Contact the Cisco TAC.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
4-3
Chapter 4
Troubleshooting Line Cards
Troubleshooting the Channelized T3 Line Card
Troubleshooting the Channelized T3 Line Card
Figure 4-2 describes the LEDs on the channelized T3 line card faceplate to help you verify proper
operation.
Figure 4-2
CISCO
10000
LED States on the Channelized T3 Line Card
IL
FA
LED
Fail
Description
A major failure has disabled the
line card.
Off
The line card is operating
properly.
OP
LO
M
AR
AL R
IE
RR
CA
Status
Yellow
0
Carrier (carrier detect) Green
Carrier detected; the port is
able to pass traffic.
Off
No carrier detected; the port is
not able to pass traffic.
Yellow
Indicates an alarm condition at
DS1, AIS, DS1 remote,
DS1 OOF, DS3 OOF, DS3 AIS,
or DS3 FERF 1 level.
No alarm condition.
2
Alarm
3
4
Off
5
Loop (active loop)
Yellow
Port is in a loopback state and
not enabled for data traffic.
Off
Port is not in a loopback state.
29991
1
6XCT3–DS0
Cisco 10000 Series Router Troubleshooting Guide
4-4
OL-0439-02
Chapter 4
Troubleshooting Line Cards
Troubleshooting the Channelized T3 Line Card
Table 4-2 describes fault conditions on the channelized T3 line card and lists recommended corrective
actions.
Table 4-2
Channelized T3 Line Card Fault Indications and Corrective Action
Fault
Fail LED is lit indicating that
the line card has failed
Carrier LED is off and
interface is enabled
Corrective Action
1.
Reinsert line card.
2.
Replace line card.
3.
Insert the line card in another slot.
4.
Call the Cisco TAC.
1.
Check the cables, making sure that they are properly attached at
both ends.
2.
Check the cable connections at the remote equipment.
3.
Replace the card.
4.
Try the card in a different slot. If the card works in a different slot,
you may have a damaged backplane. Call the Cisco TAC.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
4-5
Chapter 4
Troubleshooting Line Cards
Troubleshooting the Gigabit Ethernet Line Card
Troubleshooting the Gigabit Ethernet Line Card
Figure 4-3 describes the LEDs on the gigabit Ethernet line card faceplate to help you verify correct
operation.
Make sure that the gigabit Ethernet Interface Converter (GBIC) type on the Cisco 10000 series router
matches the GBIC type at the other end of the fiber optic cable.
Figure 4-3
CISCO
10000
Gigabit Ethernet Line Card Faceplate and LED Descriptions
IL
FA
LED
Fail
Rx (receive)
Tx (transmit)
Link
K
TX
Description
A major failure has disabled the
line card.
Off
The line card is operating
properly.
Green
Receiving traffic.
Off
Not receiving traffic.
Green
Transmitting traffic.
Off
Not transmitting traffic.
Green
Carrier detected; the port is
able to pass traffic.
Off
No carrier detected; the port is
not able to pass traffic.
RX
LIN
Status
Yellow
0
29992
Tip
GIGABIT ETHERNET
Cisco 10000 Series Router Troubleshooting Guide
4-6
OL-0439-02
Chapter 4
Troubleshooting Line Cards
Troubleshooting the Gigabit Ethernet Line Card
Table 4-3 describes the gigabit Ethernet line card fault indications and suggests responses to each.
Table 4-3
Gigabit Ethernet Line Card Faults and Recommended Responses
Fault Type
Fail LED is lit yellow indicating that a major fault
has disabled the card
Fail LED blinks then lights steadily repeatedly
or
Card seems to be passing traffic (Tx/Rx lights),
but cannot communicate with the PRE
Response
1.
Reinsert the line card.
2.
Insert the line card into another slot.
3.
Replace the line card.
4.
If neither of the above responses to a card
failure succeeds, call the Cisco TAC.
1.
Check for bent pins on the backplane.
2.
If there are no bent pins, replace with a new
line card.
Try inserting the line card in a different slot.
If the card works in a different slot, the
Cisco 10000 series router backplane may be
defective. Call the Cisco TAC.
Fail LED blinks steadily
This is a user correctable problem. The steadily
blinking LED indicates a transmit failure.
To correct the problem:
1.
Reinsert the GBIC.
If reinsertion fails:
Link LED does not light but the port is enabled
2.
Replace the GBIC.
1.
Make sure the fiber optic cable is plugged in
properly, unbroken, and undamaged.
2.
Make sure that you are using the correct type
of fiber optic cable (see Table 6-1).
3.
If you have autonegotiation enabled on the
local gigabit Ethernet interface, make sure
that it is enabled on the remote interface also.
If autonegotiation is disabled, it must be
disabled at the remote interface as well.
4.
Replace the GBIC.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
4-7
Chapter 4
Troubleshooting Line Cards
Troubleshooting the Gigabit Ethernet Line Card
Cisco 10000 Series Router Troubleshooting Guide
4-8
OL-0439-02
C H A P T E R
5
Cisco 10000 Series Router Alarms and Error
Messages
This chapter describes the alarms and error messages that can be raised on the Cisco 10000 series router.
The Cisco 10000 series router interface, environment, and system alarms provide feedback on the
functional status of a chassis, PRE, and line cards. Furthermore, alarms and error messages alert users
to less serious conditions that can degrade the performance of networks.
The following topics are addressed in this chapter:
•
OC-12 POS Line Card Alarms and Error Messages, page 5-1
•
Channelized T3 Line Card Alarms and Error Messages, page 5-6
•
Gigabit Ethernet Line Card Error Messages, page 5-8
OC-12 POS Line Card Alarms and Error Messages
This section describes alarm and error messages associated with the OC-12 POS line card. Where
possible, the source of the error message is explained and a corrective action is suggested.
OC-12 POS Line Card SONET Alarms
The OC-12 POS line card provides standard SONET alarms to monitor the health of OC-12 POS trunk
links. Table 5-1 describes these alarms and suggests responses to each.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
5-1
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
OC-12 POS Line Card Alarms and Error Messages
Table 5-1
SONET Alarms
Alarm Type and Severity
Alarm Symptoms
Recommendation
1.
Alarm messages appear in the
CLI and logs.
1.
Check the fiber optic cable to
make sure it is plugged in.
2.
Service on the SONET line
card is lost.
2.
Verify that the local fiber optic
cable is not damaged.
3.
If APS 1+1 redundancy is
enabled, trunk service cuts
over to the alternate line card.
3.
Make sure that the remote end of
the fiber optic cable is
connected, undamaged and that
the remote port is configured
properly.
1.
Alarm messages appear in the
CLI and logs.
1.
Check the fiber optic cable to
make sure it is plugged in.
2.
Service on the SONET trunk is
lost.
2.
Verify that the fiber is not
damaged.
3.
If APS is enabled, trunk
service cuts over to the
alternate line card.
3.
Make sure that the remote end of
the fiber optic cable is connected
and the router properly
configured.
LAIS
1.
1.
Alarm Indicate Signal –
Line
This alarm is indicative of a
remote problem.
Verify that the remote
configuration is correct.
2.
Alarm messages appear in the
CLI and logs.
2.
Check the line status the remote
end of the link.
SLOS
Section Loss of Signal
Critical
SLOF
Section Loss of Frame
Critical
Major
LRDI
Remote Defect
Indication – Line
Alarm messages appear in the CLI RDI – Line problems arise from the
and logs.
remote interface.
Check the remote site for alarm
conditions.
Major
PAIS
1.
Alarm Indicate Signal –
Path
Alarm messages appear in the
CLI and logs.
2.
This is sent by a site that has
received LAIS.
Minor
This is sent by a site that has received
LAIS. This is a minor warning, and
no action needs to be taken except to
monitor the far end.
If the alarms are persistent, verify the
interface configurations on both ends
of the trunk.
Cisco 10000 Series Router Troubleshooting Guide
5-2
OL-0439-02
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
OC-12 POS Line Card Alarms and Error Messages
Table 5-1
SONET Alarms (continued)
Alarm Type and Severity
Alarm Symptoms
PLOP
Alarm messages appear in the CLI This is an error in the data. As with
and logs.
the PAIS, you should monitor for
more of the same messages.
Loss of Pointer – Path
Minor
Recommendation
If the alarms are persistent, verify the
interface configurations on both ends
of the trunk.
PRDI
Remote Defect
Indication – Path
Minor
Alarm messages appear in the CLI A PRDI alarm usually indicates a
and logs.
problem two sites away. If the alarm
is persistent, check the alarm status
of neighboring sites, beginning with
the nearest neighbor.
OC-12 POS Line Card Bit Error Rate Alarms
Table 5-2 describes a separate set of alarm messages raised when the bit error rate (BER) thresholds
exceed specified limits. The BER monitors help identify signal degrading BER levels.
Table 5-2
BER Alarms
Alarm Type and Severity
Alarm Symptoms
Recommendation
TCA_B1
For alarm types:
In all cases, test the quality of
the cables and connections.
Threshold crossing alarm – B1
•
TCA_B1
Minor
•
TCA_B2
•
TCA_B3
Alarm messages appear in the
CLI and logs.
TCA_B2
Threshold crossing alarm – B2
Minor
TCA_B3
Threshold crossing alarm – B3
Minor
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
5-3
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
OC-12 POS Line Card Alarms and Error Messages
Table 5-2
BER Alarms (continued)
Alarm Type and Severity
Alarm Symptoms
Recommendation
BER_SF
BER_SF and BER_SD alarms
result in APS cutovers.
In both cases, test the quality of
the cables and connections.
Signal Fail condition
Minor
For more information on signal
fail (BER_SF) and signal
degrade (BER_SD) alarms, refer
to the APS chapter in the
Cisco 10000 Series Router
Broadband Aggregation,
Leased-Line, and MPLS
Configuration Guide. These
BER thresholds can be specified
by the user.
BER_SD
Signal degrade condition
Minor
OC-12 POS Line Card Error Messages
Table 5-3 describes hardware-related error messages that the OC-12 POS line card can report. It also
defines the problem causing the error message and suggests a corrective action.
Table 5-3
OC-12 POS Hardware-Related Error Messages
Error Message
Cause of Error Message
LC_ERROR: Slot(x/0)
1oc12pos-1 DataPathControl:
Hardware Fatal Error
Corrective Action
These alarm messages announce In all cases, follow this
unrecoverable OC-12 POS line procedure:
card hardware failures.
1. Replace the OC-12 POS line
card.
2.
Call the Cisco TAC.
LC_ERROR: Slot(x/0)
1oc12pos-1 FromPRE: Hardware
Fatal Error
LC_ERROR: Slot(x/0)
1oc12pos-1 ToPRE: Hardware
Fatal Error
LC_ERROR: Slot(x/0)
1oc12pos-1 Port0: Transmit
Clock Hardware Fatal Error
LC_ERROR: Slot(x/0)
1oc12pos-1 ToPRE/FromPRE:
Hardware Download Fatal
Error
LC_ERROR: Slot(x/0)
1oc12pos-1 FromPRE: Hardware
Configuration Fatal Error
LC_ERROR: Slot(x/0)
1oc12pos-1 ToPRE: Hardware
Configuration Fatal Error
Cisco 10000 Series Router Troubleshooting Guide
5-4
OL-0439-02
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
OC-12 POS Line Card Alarms and Error Messages
Table 5-4 describes the software-related error messages that the OC-12 POS line card might generate. It
also defines the problem causing the error message and suggests a corrective action.
Table 5-4
OC-12 POS Software-Related Error Messages
Error Message
Cause of Error Message
Corrective Action
LC_ERROR: Slot(x/0)
1oc12pos-1 Software:
Received unknown command
from IOS (%d)
The most likely cause of these
messages is an incompatible
version of IOS.
In both cases:
1.
Type
Router#show version
2.
Verify that the IOS version
is a valid release for the
Cisco 10000 series router.
3.
If the version is wrong, get
the most recent IOS image
for the Cisco 10000 series
router.
4.
If the version is correct,
Call the Cisco TAC.
LC_ERROR: Slot(x/0)
1oc12pos-1 Software: Queue
timer expired
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
5-5
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
Channelized T3 Line Card Alarms and Error Messages
Channelized T3 Line Card Alarms and Error Messages
This section describes alarm and error messages that can be generated by the channelized T3 line card.
Where possible, the source of the error message is explained and a corrective action is suggested below.
Channelized T3 Line Card Alarms
Table 5-5 describes the standard T1 alarms that can be raised on the channelized T3 line card.
Table 5-5
CT3 Line Card Alarms: T1 Alarms
Alarm Type
Cause of Alarm
AIS
A received alarm indication signal
(AIS) indicates an alarm raised on
a line upstream from the
Cisco 10000 series router.
Receive Alarm
Indication Signal
Corrective Action
1.
Check the status of the adjacent
network element to determine if the
problem is there. If the problem is not
in the adjacent network element, go to
Step 2.
2.
Ask your T1 service provider to trace
the source of the AIS signal.
3.
Contact the Cisco TAC.
1.
Check to see if the framing format
configured on the port matches the
framing format on the line.
2.
Try the other framing format and see
if the alarm clears.
3.
Configure a remote loopback on the
affected interface, then run an
unframed T1 BERT. This will help
determine if there are problems on the
line.
Major
LOF
Loss of Frame
Major
A loss of frame (LOF) condition
typically happens in one of two
situations:
1.
The configuration settings on
the port are not correct for the
line.
2.
The port configuration is
correct but the line is
experiencing other errors that
result in an LOF alarm.
If you find evidence of a bad line, you
can isolate the problem using
loopbacks. (Refer to the “Using
Loopback Tests” section on
page 6-6.)
RAI
Receive Remote
Alarm Indication
Major
RAI indicates a problem between
the C10000 T3 transmitter and the
far end T3 receiver, but it may not
be in the segment between the
C10000 and the adjacent node.
1.
Connect an external loopback cable to
the port. If there are no alarms, the
problem is not with the
Cisco 10000 series router. (Refer to
the “Using Loopback Tests” section
on page 6-6)
2.
Examine adjacent network elements
and check for LOS or LOF alarms.
Cisco 10000 Series Router Troubleshooting Guide
5-6
OL-0439-02
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
Channelized T3 Line Card Alarms and Error Messages
Channelized T3 Line Card Error Messages
Table 5-6 lists the error messages that the channelized T3 line card can generate.
Table 5-6
Channelized T3 Line Card Error Messages
Error Message
Corrective Action
LC_ERROR:Slot[x/0] Received %s (%d:%d) command, for invalid t3 %d
from CRE-RP!
In all cases, contact
the Cisco TAC for
assistance.
LC_ERROR:Slot[x/0] Received %s (%d:%d) command, for invalid t3 %d,
t1 %d, from CRE-RP!
LC_ERROR:Slot[x/0] Received %s (%d:%d) command, for t3 %d, invalid
t1 %d, from CRE-RP!\n
LC_ERROR:Slot[x/0] Received %s (%d:%d) command, for invalid t3 %d,
t1 %d chan %d, from CRE-RP!
LC_ERROR:Slot[x/0] ct3_cmd_dispatcher:Unknown command %d!
LC_ERROR:Slot[x/0] c10k_lc_cmd_q timer expired!
LC_ERROR:Slot[x/0] ct3 i960 stats queue timeout!
LC_ERROR:Slot[x/0] Error getting channel stats array pointer
LC_ERROR:Slot[x/0] Error getting message unit array pointer
LC_ERROR:Slot[x/0] request_channel_stats challoc failed!!
LC_WARNING:Slot[x/0] Lunar Pmon data not latched for collection.
T3=%d, T1=%d
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
5-7
Chapter 5
Cisco 10000 Series Router Alarms and Error Messages
Gigabit Ethernet Line Card Error Messages
Gigabit Ethernet Line Card Error Messages
Table 5-7 describes the error messages associated with the gigabit Ethernet (GE) line card. Where
possible, the source of the error message is explained and a corrective action is suggested.
Table 5-7
Gigabit Ethernet Line Card Error Messages
Error Message
Possible Cause
Corrective Action
LC_ERROR:Slot[x/0] TX
fault
GBIC failure. GBIC transmit
failure is characterized by a
steadily blinking Fail LED on
the gigabit Ethernet line card.
1.
Replace the GBIC.
LC_ERROR:Slot[x/0]
Unknown GBIC type
There are several types of
GBICs. If the software cannot
determine the type, this error is
displayed. This does not mean
that the GBIC will not function.
1.
If this message is accompanied
by a GBIC failure, replace the
GBIC with a model supported
by the Cisco 10000 series
router.
2.
If the GBIC is Cisco supplied,
it may be faulty. Try replacing
the GBIC.
3.
If none of the suggestions
above work, call the Cisco
TAC.
See the Cisco 10000 Series Router
Hardware Installation Guide for a
list of GBIC part numbers.
LC_ERROR:Slot[x/0]
Unknown GBIC CC type
Occurs if a GBIC (containing an
EEPROM) type cannot be
determined.
1.
Reinsert the GBIC.
2.
Insert a new GBIC.
3.
Try the GBIC in a different
gigabit Ethernet line card.
LC_ERROR:Slot[x/0] Fatal
interrupt from x
Occurs if there is a problem with Call the Cisco TAC.
the hardware.
%C10KGE-3-GBIC_MISSING:In
terface
GigabitEthernet1/0/0,
Gigabit Interface
Converter (GBIC) missing
Error messages appear in the
logs and the CLI.
1.
Make sure the GBIC is
properly inserted.
2.
Replace the GBIC.
Cisco 10000 Series Router Troubleshooting Guide
5-8
OL-0439-02
C H A P T E R
6
Troubleshooting Serial Lines
This chapter describes methods for detecting and correcting data errors on the Cisco 10000 series router
serial interfaces.
Optical Signal Input/Output Problems
Signal input and output problems can occur at any point in the network and can be caused by mechanical
defects in cables or fiber, poor connections, or loss of signal caused by other equipment failures. Refer
to your site log and other facility records to isolate signal connections for your facility.
Fiber-Optic Connections
An optical signal I/O problem can be caused by:
•
Incorrect type of fiber
•
Defective fiber
•
Transmit (TX) and Receive (RX) fibers that are reversed
•
Insufficient power budget on the optical link
•
Receiver overload on the optical link
Be sure to use single-mode fiber for a single-mode interface and multimode fiber for a multimode
interface. Table 6-1 describes the fiber types appropriate for each Cisco 10000 series router line card.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
6-1
Chapter 6
Troubleshooting Serial Lines
Optical Signal Input/Output Problems
Table 6-1
Optical Fiber Types for Cisco 10000 Series Router Line Cards
Card Type
Appropriate Fiber Type
OC-12 Packet Over SONET line card
Single mode.
Gigabit Ethernet line card
The appropriate fiber type for the gigabit Ethernet
line card is dependent upon the installed GBIC.
1.
1000BaseSX, multimode.
2.
1000BaseLX/LH, single mode and
multimode1.
3.
1000BaseZX, single mode.
1. Mode-conditioning patch cord (CAB-GELX-625 or equivalent) is required. If you use an ordinary patch cord with MMF,
1000BaseLX/LH GBICs, and a short link distance (10s of meters), you can cause transceiver saturation, resulting in an
elevated bit error rate (BER). In addition, if you use the LX/LH GBIC with 62.5-micron diameter MMF, you must install a
mode-conditioning patch cord between the GBIC and the MMF cable on both the transmit and receive ends of the link. The
mode-conditioning patch cord is required for link distances greater than 984 ft. (300 m).
Evaluating the Power Budget
Use the following equation to ensure that an appropriate power budget has been allotted to optical links
terminating at the Cisco 10000 series router.
The power budget (PB) is the maximum possible amount of power transmitted. The following equation
shows the calculation of the power budget:
PB = PTmin – PRmin
Where:
PTmin = Minimum transmitter power
PRmin = Minimum receiver sensitivity
Insufficient power budget occurs when the power margin (PM) is less than 0. PM is equal to the power
budget minus the link loss (LL).
PM = PB – LL
Three factors contribute to link loss:
Note
•
Fiber attenuation (single mode) 0.5 dB/km
•
Connector 0.5 dB
•
Splice 0.5 dB
These are typical values; refer to the manufacturer for the actual values.
Managing Receiver Overload
Receiver overload can occur when (PRmax – (PTmax – LL)) is less than 0, where PRmax is maximum
receiver power and PTmax is maximum transmitter power. To prevent overloading the receiver, you can
use an attenuator on the link between any singlemode SONET transmitter and the receiver. Doing so
increases the value of LL.
Cisco 10000 Series Router Troubleshooting Guide
6-2
OL-0439-02
Chapter 6
Troubleshooting Serial Lines
Using Bit Error Rate Tests
Note
For the gigabit Ethernet line card, PR max is greater than or equal to PTmax, so an attenuator is
unnecessary.
Using Bit Error Rate Tests
This section discusses problem isolation using bit error rate (BER) tests. The topics discussed are:
•
Configuring a BER Test on a T1 Line, page 6-3
•
Sending a BER Test Pattern on a T1 Line, page 6-4
•
Viewing the Results of a BER Test, page 6-5
•
Terminating a BER test, page 6-6
Configuring a BER Test on a T1 Line
BER test circuitry is built into the CT3 line card. With BER tests, you can test cables and signal problems
in the field. You can configure individual T1 lines to run BER tests, but only one BER test circuit exists
for all 28 T1 lines. Hence, only one BER test can be run on a single T3 port at any given time.
There are two categories of test patterns that can be generated by the onboard BER test circuitry:
pseudorandom and repetitive. Pseudorandom test patterns are exponential numbers and conform to the
CCITT/ITU O.151 and O.153 specifications; repetitive test patterns are all zeros, all ones, or alternating
zeros and ones.
A description of each type of test pattern follows:
•
Pseudorandom test patterns:
– 2^11 (per CCITT/ITU O.151)
– 2^15 (per CCITT/ITU O.151)
– 2^20 (per CCITT/ITU O.153)
– 2^20 QRSS (per CCITT/ITU O.151)
– 2^23 (per CCITT/ITU O.151)
•
Repetitive test patterns:
– All zeros (0s)
– All ones (1s)
– Alternating zeros (0s) and ones (1s)
Both the total number of error bits received and the total number of bits received are available for
analysis. You can set the testing period from 1 minute to 14,400 minutes (240 hours), and you can also
retrieve the error statistics anytime during the BER test.
When running a BER test, your system expects to receive the same pattern that it is transmitting. To help
ensure this:
•
Use a loopback at a location of your choice in the link or network.
•
Configure remote testing equipment to transmit the same BER test pattern at the same time.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
6-3
Chapter 6
Troubleshooting Serial Lines
Using Bit Error Rate Tests
Sending a BER Test Pattern on a T1 Line
You can send a BER test pattern on a T1 line with the controller command. The unframed option causes
the BER test pattern to use the entire T1 bandwidth including the T1 framing as well as payload bits. If
“unframed” is omitted then the T1 will be either SF or ESF framed as configured by the T1 framing
command and the BER test pattern will occupy only the T1 payload bits.
t1 t1-line-number bert pattern pattern interval time [unframed]
where:
•
t1-line-number is 1–28.
•
time is 1–14400 minutes.
•
pattern is:
– 0s, repetitive test pattern of all zeros (as 00000...).
– 1s, repetitive test pattern of all ones (as 11111...).
– 2^11, pseudorandom test pattern (2,048 bits long).
– 2^15, pseudorandom O.151 test pattern (32,768 bits long).
– 2^20-O153, pseudorandom O.153 test pattern (1,048,575 bits long).
– 2^20-QRSS, pseudorandom QRSS O.151 test pattern (1,048,575 bits long).
– 2^23, pseudorandom O.151 test pattern (8,388,607 bits long).
– alt-0-1, repetitive alternating test pattern of zeros (0s) and ones (1s), as 01010101.
Examples:
•
Send a BER test pseudorandom pattern of 2^20 through T1 line 10 for 5 minutes.
The example that follows is for a T1, numbered 10, on a CT3 line card in slot 1:
Router(config)# controller T3 1/0/0
Router(config-controller)# t1 10 bert pattern 2^20 interval 5 unframed
•
Send a repetitive pattern of all ones through T1 line 10 for 14400 minutes (240 hours).
The example that follows is for a T1, numbered 10, on a CT3 line card in slot 1:
Router(config)# controller T3 1/0/0
Router(config-controller)# t1 10 bert pattern 1s interval 14400 unframed
Note
You can terminate a BER test during the specified test period with the no t1 bert command.
Cisco 10000 Series Router Troubleshooting Guide
6-4
OL-0439-02
Chapter 6
Troubleshooting Serial Lines
Using Bit Error Rate Tests
Viewing the Results of a BER Test
You can view the results of a BER test using the controller command:
show controllers T3 slot/port-adapter/port/t1-line-number
where: t1-line-number is 1–28.
You can view the results of a BER test at the following times:
•
After you terminate the test using the no t1 bert command.
•
After the test runs completely.
•
Anytime during the test (in real time).
You can view information about a BER test using the controller command:
show controllers T3 slot/subslot/port
where: t1-line-number is 1–28.
Examples follow:
•
The example that follows is for a CT3 line card:
Router# show controllers T3 1/0/0
T3 1/0/0 is up.
C2T3 H/W Version : 3, C2T3 ROM Version : 0.79, C2T3 F/W Version : 0.29.0
T3 1/0/0 T1 1
No alarms detected.
Clock Source is internal.
BERT test result (running)
Test Pattern : 2^11, Status : Sync, Sync Detected : 1
Interval : 5 minute(s), Time Remain : 5 minute(s)
Bit Errors(Since BERT Started): 6 bits,
Bits Received(Since BERT start): 8113 Kbits
Bit Errors(Since last sync): 6 bits
Bits Received(Since last sync): 8113 Kbits
Table 6-2 explains the output of the preceding command, starting at the arrow:
Table 6-2
Interpreting BER Test Results
Output Display Line
Explanation
BERT test result (running)
Indicates the current state of the test. In this case, “running”
indicates that the BER test is still in progress. After a test is
completed, “done” is displayed.
Test Pattern : 2^11, Status : Sync,
Sync Detected : 1
Indicates the test pattern you selected for the test (2^11), the
current synchronization state (sync), and the number of
times synchronization has been detected during this test (1).
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
6-5
Chapter 6
Troubleshooting Serial Lines
Using Loopback Tests
Table 6-2
Interpreting BER Test Results (continued)
Interval : 5 minute(s), Time Remain
: 5 minute(s)
Indicates the time the test takes to run and the time
remaining for the test to run.
If you terminate a BER test, you receive a message similar
to the following:
Interval : 5 minute(s), Time Remain : 2
minute(s) (unable to complete)
“Interval: 5 minutes” indicates the configured run time for
the test. “Time Remain : 2 minutes” indicates the time
remaining in the test prior to termination. “(Unable to
complete)” signifies that you interrupted the test.
Bit Errors(Since BERT Started):
6 bits,
Bits Received(Since BERT start):
8113 Kbits
Bit Errors(Since last sync): 6 bits
Bits Received(Since last sync):
8113 Kbits
Note
These four lines show the bit errors that have been detected
versus the total number of test bits that have been received
since the test started and since the last synchronization was
detected.
Unless unframed is selected, the BER test runs over the configured framing option for the specified T1
line (ESF or SF). Before running a BER test, you should configure the framing option that is appropriate
to your application.
Terminating a BER test
To terminate a BER test, type:
no t1 t1-line-number bert
where: t1-line-number is 1–28.
Examples:
•
Terminate the BER test running on T1 line 10 on the CT3 line card.
Router(config)# controller T3 1/0/0
Router(config-controller)# no t1 10 bert
Using Loopback Tests
The following sections describe the configuration and use of loopback tests in problem isolation:
•
Configuring the Loopback Mode for a T3 Controller, page 6-7
•
Configuring a T3 Controller to Respond to Remote Loopback Commands, page 6-7
•
Configuring the Loopback Mode for a Gigabit Ethernet Interface, page 6-8
•
Configuring the Loopback Mode for an OC-12 POS Interface, page 6-8
Cisco 10000 Series Router Troubleshooting Guide
6-6
OL-0439-02
Chapter 6
Troubleshooting Serial Lines
Using Loopback Tests
Configuring the Loopback Mode for a T3 Controller
You can configure the loopback modes for a T3 controller by using the loopback command:
loopback [local | network | remote]
The default loopback mode for the T3 controller is no loopback.
To return the T3 controller to its default condition, use the no form of the command.
Examples:
•
Configure a T3 controller for local loopback.
Router(config)# controller T3 1/0/0
Router(config-controller)# loopback local
Local loopback simultaneously loops all channels toward the router and transmits a T3 AIS to the
network. You can use local loopback to diagnose problems with the port when isolated from the
network cables.
•
Configure a T3 port for network loopback.
Router(config)# controller T3 1/0/0
Router(config-controller)# loopback network
Network loopback loops the T3 line back towards the network and can be used to diagnose problems
with cables from the central switching office to the port.
•
Configure a T3 port for remote loopback.
Router(config)# controller T3 1/0/0
Router(config-controller)# loopback remote
Remote loopback sends a command to loop the T3 line at the far end (central office). It can be used
to diagnose problems with cables from the port adapter to the switching office.
Configuring a T3 Controller to Respond to Remote Loopback Commands
The equipment customer loopback command allows a port to respond to loopback commands from
remote T3 equipment. The equipment network loopback causes a controller to ignore remote T3
loopback commands.
Syntax:
equipment [customer | network] loopback
Example:
To enable the controller’s ability to respond to remote loopback requests, type:
Router(config)# controller T3 1/0/0
Router(config-controller)# equipment customer loopback
To prevent a controller from responding to remote loopback commands, type:
Router(config)# controller T3 1/0/0
Router(config-controller)# equipment network loopback
Note
Remote loopbacks are only available when you use c-bit parity framing.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
6-7
Chapter 6
Troubleshooting Serial Lines
Using Loopback Tests
Configuring the Loopback Mode for a Gigabit Ethernet Interface
To set loopback mode on a gigabit Ethernet interface, use the loopback command in interface
configuration mode.
loopback [internal | external]
[no] loopback [internal | external]
Where:
•
external runs a loopback that requires a loopback connector.
•
internal runs a loopback at the MAC controller using a serializing/deserializing method (SERDES).
Use the no form of the command to stop the loopback.
In the following example, an internal loopback mode is defined for a gigabit Ethernet interface:
router(config)# interface GigabitEthernet 1/0/0
router(config-if)# loopback internal
Tip
If you are performing a hard plug loopback test on a gigabit Ethernet interface, be sure to set the
loopback type for the interface to external. Otherwise, no packets are transmitted onto the fiber optic
cable.
Configuring the Loopback Mode for an OC-12 POS Interface
To enable loopback testing of data transmitted from the PRE card to the OC-12 POS card and back, use
the loopback command in interface configuration mode:
loopback [line | internal]
[no] loopback [line | internal]
Where:
Both line and internal do the following:
•
Loop any data received at the OC-12 POS card’s network interface back into the network.
•
Loop any data received at the OC-12 POS card’s network interface back into the PRE card.
Use the no form of the command to stop the loopback test.
In the following example, a loopback is set for the OC-12 POS line card in slot 5:
Router(config)# interface pos 5/0/0
Router(config-if)# loopback line
Cisco 10000 Series Router Troubleshooting Guide
6-8
OL-0439-02
C H A P T E R
7
Replacing or Recovering Passwords
This chapter describes how to recover a lost enable or console login password, and how to replace a lost
enable secret password on the Cisco 10000 series router.
Note
It is possible to recover the enable or console login password. The enable secret password is encrypted,
however, and must be replaced with a new enable secret password.
Password Recovery Procedure Overview
Following is an overview of the steps in the password recovery procedure:
•
If you can log in to the router, enter the show version command to determine the existing
configuration register value.
•
Press the Break key to go to the bootstrap program prompt (ROM monitor). You might need to
reload the system image by power-cycling the router.
•
Change the configuration register so that the following functions are enabled:
– Break
– Ignore startup configuration
– Boot from Flash memory
Note
The key to recovering a lost password is to set the configuration register bit 6 (0x2142) so that
the startup configuration (usually in NVRAM) is ignored. This allows you to log in without
using a password and to display the startup configuration password.
•
Power cycle the router by typing reload at the
•
Log in to the router and enter the privileged EXEC mode.
•
Enter the show startup-config command to display the passwords.
•
Recover or replace the displayed passwords.
•
Change the configuration register back to its original setting.
Note
rommon>
prompt.
To recover a lost password if the break function is disabled on the router, you must have physical
access to the router.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
7-1
Chapter 7
Replacing or Recovering Passwords
Password Recovery Procedure
Password Recovery Procedure
To recover or replace a lost enable, enable secret, or console login password, use this procedure:
Step 1
Attach an ASCII terminal to the console port on the router.
Step 2
Configure the terminal to operate at 9600 baud, 8 data bits, no parity, and 1 stop bit.
Step 3
If you can log in to the router as a nonprivileged user, enter the show version command to display the
existing configuration register value, then go to Step 6. If you cannot log in to the router at all, go to the
next step.
Step 4
Press the Break key or send a break from the console terminal.
•
If break is enabled, the router enters the ROM monitor, indicated by the ROM monitor prompt
(rommon>). Go to Step 6.
•
If break is disabled, power cycle the router (turn off the router or unplug the power cord, and then
restore power). Then go to Step 5.
Step 5
Within 60 seconds of restoring the power to the router, press the break key or send a break. This action
causes the router to enter the ROM monitor and display the ROM monitor prompt ( rommon>).
Step 6
Set the configuration register using the configuration register utility. Enter the confreg command at the
ROM monitor prompt as follows:
rommon> confreg
Answer yes to the enable “ignore system config info?”
Step 7
Reboot the router by entering the reset command:
rommon> reset
The router initializes, the configuration register is set to 0x142, and the router boots the system image
from Flash memory and enters the system configuration dialog (setup):
--- System Configuration Dialog --
Step 8
Enter no in response to the system configuration dialog prompts until the following message appears:
Press RETURN to get started!
Step 9
Press Return. The user EXEC prompt appears:
Router>
Step 10
Enter the enable command to enter privileged EXEC mode. Then enter the show startup-config
command to display the passwords in the configuration file as follows:
Router# show startup-config
Step 11
Scan the configuration file display, looking for the passwords (the enable passwords are usually located
near the beginning of the file, and the console login or user EXEC password is near the end). The
passwords displayed appear similar to the following:
enable secret 5 $1$ORPP$s9syZt4uKn3SnpuLDrhuei
enable password 23skiddoo
.
.
line con 0
password onramp
Cisco 10000 Series Router Troubleshooting Guide
7-2
OL-0439-02
Chapter 7
Replacing or Recovering Passwords
Password Recovery Procedure
The enable secret password is encrypted and cannot be recovered; it must be replaced. Go to the next
step to replace an enable secret, console login, or enable password. If there is no enable secret password,
note the enable and console login passwords. If the enable and console login passwords are not
encrypted, go to Step 16.
Caution
Step 12
Do not execute the next step unless you have determined you must change or replace the enable, enable
secret, or console login passwords. Failure to follow the steps as shown might cause you to erase the
router configuration.
Enter the copy startup-config running-config command to load the startup configuration file into
running memory. This action allows you to modify or replace passwords in the configuration.
Router# copy startup-config running-config
Step 13
Enter the privileged EXEC command configure terminal to enter configuration mode:
Router# configure terminal
Step 14
Change all three passwords using the following commands:
Router(config)# enable secret newpassword1
Router(config)# enable password newpassword2
Router(config)# line con 0
Router(config-line)# password newpassword3
Change only the passwords necessary for your configuration. You can remove individual passwords by
using the no form of the above commands. For example, entering the no enable secret command
removes the enable secret password.
Step 15
You must configure all interfaces to avoid having the system be administratively shut down:
Router(config)# interface fastethernet 0/0
Router(config-int)# no shutdown
Enter the equivalent commands for all interfaces that were originally configured. If you omit this step,
all interfaces are administratively shut down and unavailable when the router is restarted.
Step 16
Use the config-register command to set the configuration register to the original value noted in Step 3
or Step 7, or to the factory default value 0x2102.
Router(config)# config-register 0x2102
Step 17
Caution
Press Ctrl-Z (hold down the Control key while you press Z) or enter end to exit configuration mode
and return to the EXEC command interpreter.
Do not execute the next step unless you have changed or replaced a password. If you skipped Step 12
through Step 15, go to Step 19. Failure to observe this caution causes you to erase the router
configuration file.
Step 18
Enter the copy running-config startup-config command to save the new configuration to NVRAM.
Step 19
Enter the reload command to reboot the router.
Step 20
Log in to the router using the new or recovered passwords.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
7-3
Chapter 7
Replacing or Recovering Passwords
Password Recovery Procedure
Cisco 10000 Series Router Troubleshooting Guide
7-4
OL-0439-02
A P P E N D I X
A
Unsupported Commands
Following is a list of the commands that are not supported in IOS release 12.0(9)SL for the
Cisco 10000 series router:
Unsupported Frame Relay Commands
•
FRF.12 fragmentation commands
•
FRF.11 VoFR commands
•
frame-relay adaptive-shaping {becn | foresight}
•
frame-relay bc {in | out} bits
•
frame-relay be {in | out} bits
•
frame-relay cir {in | out} bps
•
frame-relay custom-queue-list list-number
•
frame-relay de-group group-number dlci
•
frame-relay ip [rtp | tcp] header-compression [passive]
•
frame-relay map protocol protocol-address dlci [broadcast] [ietf | cisco] [payload-compress
{packet-by-packet | frf9 stac [hardware-options]}]
•
frame-relay map ip ip-address dlci [broadcast] [cisco | ietf] [nocompress] tcp
header-compression {active | passive}
•
frame-relay mincir {in | out} bps
•
frame-relay traffic-rate average [peak]
•
frame-relay traffic-shaping
•
show frame-relay ip [rtp | tcp] header-compression [interface]
Unsupported PPP Commands
•
compress [predictor | stac | mppc [ignore-pfc]]
•
debug ppp cbcp
•
ppp bridge appletalk
•
ppp bridge ip
•
ppp bridge ipx [novell-ether | arpa | sap | snap]
•
ppp reliable-link
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
A-1
Appendix A
Unsupported Commands
Unsupported MLPPP Commands
•
multilink load-threshold
•
multilink max-fragments
•
ppp multilink fragment-delay
•
ppp multilink fragmentation
•
ppp multilink idle-link
•
ppp multilink interleave
Cisco 10000 Series Router Troubleshooting Guide
A-2
OL-0439-02
A P P E N D I X
B
Recommended Tools and Test Equipment
Table B-1 lists the basic tools and test equipment necessary to perform general maintenance and
troubleshooting tasks on the Cisco 10000 series router.
Table B-1
Recommended Tools and Test Equipment
Equipment
Description
Number 2 Phillips and flat-head
screwdrivers
Small and medium-sized.
Voltage tester
Refer to the “Testing with Digital Multimeters and Cable
Testers” section on page B-1.
Optical fiber test equipment
Refer to the “Testing with Digital Multimeters and Cable
Testers” section on page B-1.
Cable testing equipment
Refer to the “Testing with Digital Multimeters and Cable
Testers” section on page B-1.
ESD-preventive wrist or ankle
strap with connection cord
—
The following sections describe advanced testing equipment to aid in complex problem isolation.
Testing with Digital Multimeters and Cable Testers
Use a digital multimeter to measure parameters such as AC and DC voltage, current, resistance,
capacitance, cable continuity. Use cable testers, also, to verify physical connectivity.
Use cable testers (scanners) to check physical connectivity. Cable testers are available for shielded
twisted pair (STP), unshielded twisted pair (UTP), 10BaseT, and coaxial and twinax cables. A given
cable tester might be able to perform any of the following functions:
•
Test and report on cable conditions, including near-end crosstalk (NEXT), attenuation, and noise.
•
Perform time domain reflectometer (TDR), traffic monitoring, and wire map functions.
•
Display Media Access Control (MAC) layer information about LAN traffic, provide statistics such
as network utilization and packet error rates, and perform limited protocol testing (for example,
TCP/IP tests such as ping).
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
B-1
Appendix B
Recommended Tools and Test Equipment
Testing with TDRs and OTDRs
Test fiber-optic cable both before installation (on-the-reel testing) and after installation. Continuity
testing of the fiber requires either a visible light source or a reflectometer. Light sources capable of
providing light at the three predominant wavelengths, 850 nanometers (nm), 1300 nm, and 1550 nm, are
used with power meters that can measure the same wavelengths and test attenuation and return loss in
the fiber.
Testing with TDRs and OTDRs
This section describes time domain reflectometers (TDRs) and optical time domain reflectometers
(OTDRs), which are typically used to detect cable defects.
Testing with TDRs
Use time domain reflectometers to test for the following cable defects:
•
Open and short circuits
•
Crimps, kinks, and sharp bends
•
Impedance mismatches
•
Other defects
A TDR works by “bouncing” a signal off the end of the cable. Open circuits, short circuits and other
problems reflect the signal back at different amplitudes, depending on the problem.
A TDR measures:
•
Amount of time it takes for the signal to reflect
•
Physical distance to a fault in the cable
•
Length of a cable
Some TDRs can also calculate the propagation rate based on a configured cable length.
Testing with OTDRs
Use optical time domain reflectometers to:
•
Locate fiber breaks
•
Measure attenuation
•
Measure the length of a fiber
•
Measure splice or connector losses
An OTDR can be used to identify the “signature” of a particular installation, noting attenuation and
splice losses. This baseline measurement can then be compared with future signatures if you suspect a
problem in the system.
Cisco 10000 Series Router Troubleshooting Guide
B-2
OL-0439-02
Appendix B
Recommended Tools and Test Equipment
Testing with Breakout Boxes, Fox Boxes, and BERTs/BLERTs
Testing with Breakout Boxes, Fox Boxes, and BERTs/BLERTs
Use breakout boxes, fox boxes, and bit/block error rate testers (BERTs/BLERTs) to measure the digital
signals present at:
•
PCs
•
Printers
•
Modems
•
CSU/DSUs
These devices can monitor data line conditions, analyze and trap data, and diagnose problems common
to data communication systems. Traffic from data terminal equipment (DTE) through data
communications equipment (DCE) can be examined to:
•
Isolate problems
•
Identify bit patterns
•
Ensure that the correct cabling is installed
These devices cannot test media signals such as Ethernet, Token Ring, or FDDI.
Testing with Network Monitors
Use network monitors to:
•
Track packets crossing a network
•
Provide an accurate picture of network activity at any moment
•
Provide a historical record of network activity over a period of time
Network monitors do not decode the contents of frames. Monitors are useful for baselining, in which the
activity on a network is sampled over a period of time to establish a normal performance profile, or
baseline.
Monitors collect information such as packet sizes, the number of packets, error packets, overall usage
of a connection, the number of hosts and their MAC addresses, and details about communications
between hosts and other devices. This data can be used to:
•
Create profiles of LAN traffic
•
Locate traffic overloads
•
Plan for network expansion
•
Detect intruders
•
Establish baseline performance
•
Distribute traffic more efficiently
Testing with Network Analyzers
Use network analyzers (also called protocol analyzers) to decode protocol layers in a recorded frame and
present the layers as readable abbreviations or summaries, detailing which layer is involved (physical,
data link, and so forth) and the function each byte or byte content serves.
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
B-3
Appendix B
Recommended Tools and Test Equipment
Testing with Network Analyzers
Most network analyzers can perform many of the following functions:
•
Filter traffic that meets certain criteria so that, for example, all traffic to and from a particular device
can be captured.
•
Time-stamp captured data.
•
Present protocol layers in an easily readable form.
•
Generate frames and transmit them onto the network.
•
Incorporate an “expert” system in which the analyzer uses a set of rules, combined with information
about the network configuration and operation, to diagnose and solve, or offer potential solutions
to, network problems.
Cisco 10000 Series Router Troubleshooting Guide
B-4
OL-0439-02
I N D EX
A
E
AC PEM faults
electrical problems
2-3 to 2-4
alphanumeric display
messages
other
2-4
see also PEM faults
1-5
error messages
Ethernet
B
5-1 to 5-8
3-3
Ethernet problems
BERT
6-3
bit error rate tests
using
3-3
F
6-3
blower failures
fail LEDs
2-5
channelized T3 line card
see also show environment
blower module faults
gigabit Ethernet line card
2-5
OC-12 POS line card
fan failures
C
4-7
4-3
2-5
faults
PRE
carrier LEDs
channelized T3 line card
OC-12 POS
console
4-5
fiber optic cables
power budget
cable types
6-2
receiver overload
3-4
FRUs
3-4
console problems
6-1
fiber optics
4-3
3-3
console port
4-5
3-3
6-2
2-4
3-4
G
D
GBIC
DC PEM faults
2-1 to 2-3
deserializing method
documentation
6-8
DS1 Alarms
4-4
DS3 Alarms
4-4
gigabit Ethernet
autonegotiation
GBIC type
viii
document conventions
4-6, 4-7
viii
4-7
4-6
gigabit Ethernet interface converter
4-6
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
IN-1
Index
PRE
L
3-1
alphanumeric display
LEDs
console connection
alarm
3-3
critical
faceplate
1-5
line cards
major
1-5
minor
1-5
PRE LEDs
4-2, 4-4, 4-6
line card alarms
1-5
3-1 to 3-4
description
5-1 to 5-8
fail LED
line cards
3-2
3-3
PRE status LED
channelized T3
4-4
gigabit Ethernet
LEDs
6-8
PRE display
1-5
unlit
3-3
3-3
PRE card
PEM fault
transmit
faults
3-3
3-2
fault indications
4-2, 4-4, 4-6
1-5
3-1
3-3
4-6
4-2, 4-4, 4-6
OC-12 POS
R
4-1
troubleshooting
4-1 to 4-7
related documentation
viii
link LEDs
gigabit Ethernet line card
4-7
S
loopback
connector
6-8
show commands
show environment
1-3
show facility-alarm status
M
show hardware
management Ethernet
3-3
mode conditioning patch cord
multimode
show version
6-2
6-2
single mode
1-1
1-2 to 1-3
6-2
slot problems
startup
3-3
3-3
1-5
system startup
O
behavior
overheating
1-5
2-5
T
P
terminal settings
PEM faults
2-1
3-5
troubleshooting
AC
2-3
basic
DC
2-1
fiber optic cables
power cables
2-4
1-1
flowchart
6-1
1-4
Cisco 10000 Series Router Troubleshooting Guide
IN-2
OL-0439-02
Index
hardware
1-4
line cards
4-1 to 4-7
serial lines
6-1
troubleshooting line cards
4-1 to 4-7
Cisco 10000 Series Router Troubleshooting Guide
OL-0439-02
IN-3
Index
Cisco 10000 Series Router Troubleshooting Guide
IN-4
OL-0439-02