Trouble Shooting
CERNET NOC
Pengfei Li
Contents
Trouble Shooting
Trouble Shooting
Trouble Shooting
Trouble Shooting
Trouble Shooting
Trouble Shooting
Case Studies
Tools
BGP
OSPF
IS-IS
CEF
IP Multicast
2
Trouble Shooting Tools
Key of trouble shooting –
Excellent NMS
Route Monitoring
4
Routing (BGP summary)
5
Routing Mornitoring
6
BGP Statistics (current status)
7
Looking-glass
8
BGP Monitoring
(TEIN2-NORTH)
9
BGP Monitoring
(TEIN2-SOUTH)
10
BGP Monitoring
(TEIN2-JP)
11
AS Path Entries
12
Community Entries
13
IPv4 Prefix
14
IPv6 Prefix
15
Basic tools
Ping
Traceroute
telnet
Show commands
etc
16
Trouble Shooting BGP
BGP in Large Scale Networks
18
Avoid the Problem in the
First Place
Use simple configurations
maintain a consistent policy throughout
the AS
Promote stable networks
nail-down your routes use loopback
interfaces
Grow into your network
use peer-groups and RRs for scalability
19
Agenda
Basic Tools
Peer Establishment
UPDATE Exchange
Selection Algorithm
Route Reflectors
Route Flap Damping
20
BGP Troubleshooting Tools
show commands
debug output
Log messages
21
show Commands
router#show ip bgp ?
A.B.C.D
IP prefix <network>/<length>, e.g., 35.0.0.0/8
A.B.C.D
Network in the BGP routing table to display
cidr-only
Display only routes with non-natural netmasks
community
Display routes matching the communities
community-list Display routes matching the community-list
dampened-paths Display paths suppressed due to dampening
filter-list
Display routes conforming to the filter-list
flap-statistics
Display flap statistics of routes
inconsistent-as Display only routes with inconsistent origin ASs
neighbors
Detailed information on TCP and BGP neighbor connections
paths
Path information
peer-group
Display information on peer-groups
quote-regexp
Display routes matching the AS path "regular expression"
regexp
Display routes matching the AS path regular expression
summary
Summary of BGP neighbor status
|
Output modifiers
22
<cr>
show Commands (Cont.)
router#show ip bgp neighbors x.x.x.x ?
advertised-routes Display the routes advertised to a BGP neighbor
dampened-routes Display the dampened routes received from neighbor
flap-statistics
Display flap statistics of the routes learned from
neighbor
paths
Display AS paths learned from neighbor
received
Display information received from a BGP neighbor
received-routes
Display the received routes from neighbor
routes
Display routes learned from neighbor
|
Output modifiers
<cr>
23
The BGP Table
router#show ip bgp
BGP table version is 9, local router ID is 7.72.6.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
Network
Next Hop
Metric LocPrf Weight Path
*> 3.0.0.0
0.0.0.0
0
32768 i
*> 5.0.0.0
0.0.0.0
0
32768 i
*> 6.0.0.0
6.72.6.2
4294967294
02i
*i
6.72.6.2
4294967294 100
02i
*> 7.0.0.0
0.0.0.0
0
32768 i
*> 8.0.0.0/5
0.0.0.0
0
32768 i
*> 17.0.0.0
6.72.6.2
4294967294
02i
*i
6.72.6.2
4294967294 100
02i
*> 23.0.0.0
6.72.6.2
4294967294
02i
*i
6.72.6.2
4294967294 100
02i
*> 35.0.0.0
6.72.6.2
4294967294
02i
*i
6.72.6.2
4294967294 100
02i
24
The BGP Table (Cont.)
router#show ip bgp 6.0.0.0
BGP routing table entry for 6.0.0.0/8, version 2
Paths: (2 available, best #1)
Advertised to non peer-group peers:
7.25.14.4 7.72.6.3 7.75.7.1
2
6.72.6.2 from 6.72.6.2 (7.72.6.2)
Origin IGP, metric 4294967294, localpref 100, valid,
2
6.72.6.2 from 7.75.7.1 (7.75.7.1)
Origin IGP, metric 4294967294, localpref 100, valid,
external, best
internal
25
show ip bgp Summary
router#show ip bgp summary
BGP router identifier 7.72.6.1, local AS number 1
BGP table version is 9, main routing table version 9
8 network entries and 12 paths using 1176 bytes of memory
3 BGP path attribute entries using 144 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
BGP activity 8/0 prefixes, 12/0 paths
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
6.72.6.2 4 2 6885 6882
9 0 0 4d18h
4
7.25.14.4 4 3 6882 6883
9 0 0 4d18h
0
7.72.6.3 4 1 6880 6886
9 0 0 4d18h
0
7.75.7.1 4 1 6884 6885
9 0 0 4d18h
4
26
show ip bgp neighbors
router#show ip bgp neighbors 6.72.6.2
BGP neighbor is 6.72.6.2, remote AS 2, external link
Index 1, Offset 0, Mask 0x2
BGP version 4, remote router ID 7.72.6.2
BGP state = Established, table version = 9, up for 4d21h
Last read 00:00:56, last send 00:00:48
Hold time 180, keepalive interval 60 seconds
Neighbor NLRI negotiation:
Configured for unicast routes only
Peer negotiated unicast and multicast routes
Exchanging unicast routes only
Received route refresh capability from peer
Minimum time between advertisement runs is 30 seconds
Received 7044 messages, 0 notifications, 0 in queue
Sent 7041 messages, 0 notifications, 0 in queue
Prefix advertised 4, suppressed 0, withdrawn 0
Route refresh request: received 0, sent 0
Inbound path policy configured
Route map for incoming advertisements is k
Connections established 1; dropped 0
Last reset never
Number of unicast/multicast prefixes received 4/0
External BGP neighbor may be up to 255 hops away.
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 3.72.6.1, Local port: 179
Foreign host: 6.72.6.2, Foreign port: 11014
27
debug ip bgp
router#debug ip bgp ?
A.B.C.D
BGP neighbor address
dampening BGP dampening
events
BGP events
keepalives BGP keepalives
updates
BGP updates
<cr>
Remember—can be dangerous!
Use only in the lab or If advised by the TAC
To make a little safer:
logging buffered <size>
no logging console
28
Session Establishment
(debug ip bgp )
16:06:30: BGP: 7.72.6.1 sending OPEN, version 4
16:06:31: BGP: 7.72.6.1 OPEN rcvd, version 4
16:06:31: BGP: 7.72.6.1 rcv OPEN w/ OPTION parameter len: 12
16:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 6
16:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 1, length 4
16:06:31: BGP: 7.72.6.1 OPEN has MP_EXT CAP for afi/safi: 1/1
16:06:31: BGP: 7.72.6.1 rcv OPEN w/ option parameter type 2 (Capability) len 2
16:06:31: BGP: 7.72.6.1 OPEN has CAPABILITY code: 128, length 0
16:06:31: BGP: 7.75.7.1 passive open
16:06:31: BGP: 7.75.7.1 OPEN rcvd, version 4
16:06:31: BGP: 7.75.7.1 sending OPEN, version 4
16:06:31: BGP: 7.75.7.1 rcv OPEN w/ OPTION parameter len: 12
16:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 6
16:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 1, length 4
16:06:31: BGP: 7.75.7.1 OPEN has MP_EXT CAP for afi/safi: 1/1
16:06:31: BGP: 7.75.7.1 rcv OPEN w/ option parameter type 2 (Capability) len 2
29
16:06:31: BGP: 7.75.7.1 OPEN has CAPABILITY code: 128, length 0
Session Establishment
(debug ip bgp events)
17:31:39: BGP: 7.72.6.1 went from Idle to Active
17:32:00: BGP: 7.72.6.1 went from Active to OpenSent
17:32:00: BGP: 7.72.6.1 went from OpenSent to OpenConfirm
17:32:00: BGP: 7.72.6.1 went from OpenConfirm to Established
17:31:59: BGP: 7.75.7.1 went from Idle to Active
17:32:00: BGP: 7.75.7.1 went from Active to Idle
17:32:00: BGP: 7.75.7.1 went from Idle to Connect
17:32:00: BGP: 7.75.7.1 went from Connect to OpenSent
17:32:00: BGP: 7.75.7.1 went from OpenSent to OpenConfirm
17:32:00: BGP: 7.75.7.1 went from OpenConfirm to Established
30
Looking at the Updates
router#debug ip bgp updates?
<1-199>
Access list
<1300-2699>
Access list (expanded range)
<cr>
router#debug ip bgp x.x.x.x updates?
<1-199>
Access list
<1300-2699>
Access list (expanded range)
<cr>
Use an access-list to limit the output!
31
debug ip bgp Updates
Peer Address
Prefix Being Advertised
BGP: 6.72.6.2 computing updates, neighbor version 0, table version
13, starting at 0.0.0.0
BGP: 6.72.6.2 send UPDATE 3.0.0.0/8, next 3.72.6.1
BGP: , metric 0, path 1
BGP: 6.72.6.2 send UPDATE 5.0.0.0/8 (chgflags: 0x0), next 3.72.6.1
BGP: 6.72.6.2 send UPDATE 7.0.0.0/8 (chgflags: 0x0), next 3.72.6.1
BGP: 6.72.6.2 1 updates enqueued (average=56, maximum=56)
BGP: 6.72.6.2 update run completed, ran for 0ms, neighbor version 0,
version 13, throttled to 13, check point net 0.0.0.0
NEXT_HOP
start
32
debug ip bgp Updates (Cont.)
BGP: 6.72.6.2 rcv UPDATE
494, path 2
BGP: 6.72.6.2 rcv UPDATE
BGP: 6.72.6.2 rcv UPDATE
BGP: 6.72.6.2 rcv UPDATE
BGP: 6.72.6.2 rcv UPDATE
Peer Address
w/ attr: nexthop 6.72.6.2, origin i, metric
about 6.0.0.0/8
about 17.0.0.0/8
about 23.0.0.0/8
about 35.0.0.0/8
Prefixes in the Same
UPDATE
Attributes Apply
to
All Prefixes
BGP: 6.72.6.2 rcv UPDATE w/ attr: nexthop 6.72.6.2, origin i, metric
294, path 2 1
BGP: 6.72.6.2 rcv UPDATE about 3.0.0.0/8 -- DENIED due to: as-path
contains our own AS;
BGP: 6.72.6.2 rcv UPDATE about 7.0.0.0/8 -- DENIED due to: as-path
contains our own AS;
33
Logging Neighbor Changes
Generate a log message whenever a BGP
neighbor changes state, also indicate reason
for reset
Syntax (router subcommand):
[no] bgp log-neighbor-changes
Typical log messages:
%BGP-5-ADJCHANGE: neighbor x.x.x.x Up
%BGP-5-ADJCHANGE: neighbor x.x.x.x Down-Remote AS
changed
34
show ip bgp neighbors x.x.x.x
router#show ip bgp neighbors 7.75.7.1
BGP neighbor is 7.75.7.1, remote AS 2, external link
...
Received 194 messages, 1 notifications, 0 in queue
Sent 194 messages, 0 notifications, 0 in queue
Prefix advertised 0, suppressed 0, withdrawn 0
Route refresh request: received 0, sent 0
Connections established 7; dropped 7
Last reset 00:04:11, due to BGP Notification received, hold time expired
Number of unicast/multicast prefixes received 0/0
External BGP neighbor may be up to 255 hops away.
No active TCP connection
35
show ip bgp neighbors x.x.x.x
cont.
"BGP protocol initialization"
"No memory"
"BGP Notification received"
"BGP Notification sent"
"User reset"
"Peer timeout”
"Password change”
"Error during connection collision"
"Peer closed the session"
"Peer over prefix limit"
"Interface flap"
"Router ID changed”
"Neighbor deleted"
"Member added to peergroup"
"Admin. shutdown"
"Remote AS changed"
"Capability changed"
"RR client config change”
"Soft reconfig change"
"Local AS change"
36
Common Problems
Sessions are not established
No IP reachability
Incorrect configuration
Peering addresses
OPEN parameters
37
Can’t Establish Session Symptoms
routerA#show ip bgp summary
BGP router identifier 7.72.6.1, local AS number 1
BGP table version is 4, main routing table version 4
6 network entries and 6 paths using 774 bytes of memory
2 BGP path attribute entries using 96 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
BGP activity 6/0 prefixes, 6/0 paths
Neighbor
6.72.6.2
7.25.14.4
7.72.6.3
7.75.7.1
V AS
4 2
4 3
4 1
4 1
MsgRcvd
0
0
4
5
0
0
7
5
MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
0 0 0 never Idle
4 0 0 00:01:43
0
0 0 0 never Active
4 0 0 00:01:55
3
The peering session is not established!
State may change between active,
idle and connect
38
The BGP Finite State Machine
has 5 states:
Idle: no resources are allocated
Connect: waiting for TCP session to be established;
no connection is established actively, but one may
be accepted
Active: actively trying to establish a TCP
connection
OpenSent: waiting for OPEN message from peer
OpenConfirm: waiting for confirmation of the
OPEN message (KEEPALIVE or NOTIFICATION)
Established: UPDATES may now be exchanged…this
is the normal operational state of BGP.
39
Can’t Establish Session—
Troubleshooting I
Is the remote-as assigned correctly?
Local AS
router bgp 1
neighbor 6.72.6.2 remote-as 2
neighbor 7.72.6.3 remote-as 1
eBGP Peer
iBGP Peer
40
Can’t Establish Session—
Troubleshooting I (Cont.)
Verify IP connectivity
check the routing table
use ping/trace to verify two way reachability
inspect for ACLs in the path to the neighbor
routerA#show ip route 7.72.6.3
Routing entry for 7.72.6.3/32
Known via "ospf 123”, distance 110, metric 87, type intra area
Last update from 27.27.27.254 on POS5/0, 00:09:33 ago
Routing Descriptor Blocks:
* 27.27.27.254, from 7.72.6.3, 00:09:33 ago, via POS5/0
Route metric is 87, traffic share count is 1
routerA#ping 7.72.6.3
Sending 5, 100-byte ICMP Echos to 7.72.6.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/30/32 ms
41
Can’t Establish Session—
Troubleshooting I (Cont.)
routerA#debug ip bgp
BGP debugging is on
10:51:02: BGP: 7.72.6.3 open active, delay 6864ms
10:51:09: BGP: 7.72.6.3 open active, local address 27.27.27.253
10:51:09: BGP: 7.72.6.3 open failed: Connection refused by remote host
• Is the remote router configured for BGP?
What IP address is the remote router configured
to receive?
router bgp 1
no synchronization
bgp log-neighbor-changes
neighbor 7.72.6.1 remote-as 1
42
Can’t Establish Session—
Troubleshooting I (Cont.)
The TCP session is always
sourced from the closest IP
address to the destination!
A
C
27.27.27.254
Configuration:
Router A
router bgp 1
neighbor 27.27.27.254 remote-as 1
Router C
router bgp 1
neighbor 27.27.27.253 remote-as 1
27.27.27.253
If redundant paths exist,
use loopback interfaces to
establish the session.
43
Can’t Establish Session—
Troubleshooting I (Cont.)
router bgp 1
neighbor 7.72.6.3 remote-as 1
neighbor 7.72.6.3 update-source Loopback0
Information sourced
from the IP address in
interface Loopback0
routerA#debug ip tcp transactions
11:19:48: BGP: 7.72.6.3 open active, delay 9916ms
11:19:53: TCP: sending RST, seq 0, ack 3098129121
11:19:53: TCP: sent RST to 7.7.7.6:11719 from 7.72.6.1:179
Solution:
make sure both routers source the information
from the appropriate interface
44
Can’t Establish Session—
Symptoms
routerA#show ip bgp summary
BGP router identifier 7.72.6.1, local AS number 1
BGP table version is 4, main routing table version 4
6 network entries and 6 paths using 774 bytes of memory
2 BGP path attribute entries using 96 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
BGP activity 6/0 prefixes, 6/0 paths
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
6.72.6.2 4 2
0
0
0 0 0 never Idle
7.25.14.4 4 3 385 385
4 0 0 06:22:17
0
7.72.6.3 4 1
42
49
4 0 0 00:00:15
0
7.75.7.1 4 1 388 385
4 0 0 06:22:30
3
The eBGP session is still having trouble!
45
Can’t Establish Session Troubleshooting II
Verify IP connectivity
check the routing table
use ping/trace to verify two way reachability
routerA#show ip route 6.72.6.2
%Network not in table
routerA#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
routerA(config)#ip route 6.72.6.2 255.255.255.255 1.1.1.5
routerA#ping 6.72.6.2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 6.72.6.2, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
46
Can’t Establish Session—
Troubleshooting II (Cont.)
Peering with a loopback interface
Advantages
Interface is always up
Multiple physical paths may
exist to reach it
Disadvantages
Physical link failure may take
longer to detect
47
Can’t Establish Session—
Troubleshooting II (Cont.)
routerA#debug ip bgp
routerA#debug ip tcp transactions
13:25:30: TCP: sending RST, seq 0, ack 2030100669
13:25:30: TCP: sent RST to 6.72.6.2:11041 from 3.72.6.1:179
router bgp 1
neighbor 6.72.6.2 remote-as 2
neighbor 6.72.6.2 update-source Loopback1
Neighbor is trying
to peer with this IP
address
The debug output indicates the neighbor’s
configured peering address
48
Can’t Establish Session—
Troubleshooting II (Cont.)
13:33:30: TCP: sending RST, seq 0, ack 2510129645
13:33:30: TCP: sent RST to 6.72.6.2:11045 from 3.72.6.1:179
Hint: by default, eBGP peers should be
directly connected
in this case, the peering address
doesn’t match a connected interface
in the local router
49
Can’t Establish Session—
Troubleshooting II (Cont.)
routerA#show ip bgp neighbors 6.72.6.2
BGP neighbor is 6.72.6.2, remote AS 2, external link
Index 1, Offset 0, Mask 0x2
BGP version 4, remote router ID 0.0.0.0
BGP state = Idle, table version = 0
Last read 00:00:06, last send never
Hold time 180, keepalive interval 60 seconds
Neighbor NLRI negotiation:
Configured for unicast routes only
Minimum time between advertisement runs is 30 seconds
Received 0 messages, 0 notifications, 0 in queue
Sent 0 messages, 0 notifications, 0 in queue
Prefix advertised 0, suppressed 0, withdrawn 0
Route refresh request: received 0, sent 0
Connections established 0; dropped 0
Last reset never
Number of unicast/multicast prefixes received 0/0
External BGP neighbor not directly connected.
No active TCP connection
50
Can’t Establish Session—
Troubleshooting II (Cont.)
router bgp 1
neighbor 6.72.6.2 remote-as 2
neighbor 6.72.6.2 ebgp-multihop 255
neighbor 6.72.6.2 update-source Loopback1
At this point, the session
should come up
51
Can’t Establish Session—
Symptoms
routerA#show ip bgp summary
BGP router identifier 7.72.6.1, local AS number 1
…
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
6.72.6.2 4 2
10
26
0 0 0 never Active
router bgp 1
neighbor 6.72.6.2 remote-as 2
neighbor 6.72.6.2 ebgp-multihop 255
neighbor 6.72.6.2 update-source Loopback1
Still having trouble!
Connectivity issues have already been checked
and corrected.
52
Can’t Establish Session—
Troubleshooting II (Cont.)
14:06:37: BGP: 6.72.6.2 open active, local address 3.72.6.1
14:06:37: BGP: 6.72.6.2 went from Active to OpenSent
14:06:37: BGP: 6.72.6.2 sending OPEN, version 4
14:06:37: BGP: 6.72.6.2 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 0001
14:06:37: BGP: 6.72.6.2 remote close, state CLOSEWAIT
14:06:37: BGP: service reset requests
14:06:37: BGP: 6.72.6.2 went from OpenSent to Idle
14:06:37: BGP: 6.72.6.2 closing
If an error is detected, a notification
is sent and the session is closed
In this case the remote router had
a bad configuration
53
UPDATE Exchange
Once the session has been established,
UPDATEs are exchanged
all the locally known routes
only the bestpath is advertised
Incremental UPDATE messages are
exchanged afterwards
54
Propagation Decisions
bestpath received from eBGP peer
advertise to all peers
bestpath received from iBGP peer
advertise only to eBGP peers
a full iBGP mesh must exist
55
Common Problems
Missing routes
No iBGP full mesh
Filters: routes are not received/sent
Slow convergence
56
Missing Routes—
Troubleshooting Steps
Determine which filters are applied
to the BGP session
show ip bgp neighbors x.x.x.x
Look at the configuration
Examine the route and pick out the
relevant attributes
show ip bgp x.x.x.x
57
Missing Routes—
Troubleshooting Steps (Cont.)
Compare the route against the filters
If no match is found
Use route-refresh or soft-reconfiguration
Filter the updates through an ACL to
determine where the problem is
58
Missing Routes—Symptoms
Missing 4.0.0.0/8 in 7.75.7.1 (routerA)
not received from 7.72.6.3 (routerB)
routerB#sh ip bgp nei 7.75.7.1 advertised-routes | include 4.0.0.0
*> 4.0.0.0
0.0.0.0
0
32768 i
routerB shows that the route was advertised to routerA!
59
Missing Routes—
Troubleshooting
routerA#show access-lists 10
Standard IP access list 10
permit 4.0.0.0
routerA#debug ip bgp 7.72.6.3 updates 10
BGP updates debugging is on for access list 10 for neighbor 7.72.6.3
routerA#clear ip bgp 7.72.6.3 in
01:22:41: BGP: 7.72.6.3 rcv UPDATE w/ attr: nexthop 7.72.6.3, origin i,
metric 0, path 2
01:22:41: BGP: 7.72.6.3 rcv UPDATE about 4.0.0.0/8 -- DENIED due
to: distribute/prefix-list;
60
Missing Routes—
Troubleshooting (Cont.)
router bgp 1
no synchronization
bgp log-neighbor-changes
neighbor 7.72.6.3 remote-as 2
neighbor 7.72.6.3 ebgp-multihop 255
neighbor 7.72.6.3 update-source Loopback0
neighbor 7.72.6.3 prefix-list filter in
!
ip prefix-list filter seq 5 deny 4.0.0.0/8
ip prefix-list filter seq 10 permit 0.0.0.0/0 le 32
61
Common Problems
Inconsistent decision/policy
MED
External paths
Communities
By default, communities are not
propagated
neighbor x.x.x.x send-community
62
Inconsistent Decision—
Symptom I
The bestpath changes every time the
peering is reset.
routerA#sh ip bgp 160.100.0.0
BGP routing table entry for 160.100.0.0/16, version 40
Paths: (3 available, best #3, advertised over IBGP, EBGP)
1
204.146.33.10 from 204.146.33.10 (204.146.33.1)
Origin IGP, metric 0, localpref 100, valid, internal
3
204.146.33.66 from 204.146.33.66 (204.146.33.2)
Origin IGP, metric 20, localpref 100, valid, internal
3
204.146.33.6 from 204.146.33.6 (10.4.1.1)
Origin IGP, metric 30, valid, external, best
63
Inconsistent Decision—
Symptom I (Cont.)
routerA#sh ip bgp 160.100.0.0
BGP routing table entry for 160.100.0.0/16, version 2
Paths: (3 available, best #3, advertised over EBGP)
1
204.146.33.10 from 204.146.33.10 (204.146.33.1)
Origin IGP, metric 0, localpref 100, valid, internal
3
204.146.33.6 from 204.146.33.6 (10.4.1.1)
Origin IGP, metric 30, valid, external
3
204.146.33.66 from 204.146.33.66 (204.146.33.2)
Origin IGP, metric 20, localpref 100, valid, internal, best
Same paths, but different result!
64
Inconsistent Decision—
Symptom I (Cont.)
routerA#sh ip bgp 160.100.0.0
BGP routing table entry for 160.100.0.0/16, version 12
Paths: (3 available, best #3, advertised over EBGP)
3
204.146.33.6 from 204.146.33.6 (10.4.1.1)
Origin IGP, metric 30, valid, external
3
204.146.33.66 from 204.146.33.66 (204.146.33.2)
Origin IGP, metric 20, localpref 100, valid, internal
1
204.146.33.10 from 204.146.33.10 (204.146.33.1)
Origin IGP, metric 0, localpref 100, valid, internal, best
Different result…again!!
65
Deterministic MED
By default, the prefixes are compared in
order of arrival
it may result in inconsistent decisions
use bgp deterministic-med
the bestpath is recalculated as soon as the
command is entered
enable in all the routers in the AS
66
Deterministic MED—Operation
The paths are ordered by peer-AS
The bestpath for each group
is selected
The overall bestpath results from
comparing the winners in each group
67
Deterministic MED—Result
routerA#sh ip bgp 160.100.0.0
BGP routing table entry for 160.100.0.0/16, version 15
Paths: (3 available, best #1, advertised over EBGP)
1
204.146.33.10 from 204.146.33.10 (204.146.33.1)
Origin IGP, metric 0, localpref 100, valid, internal, best
3
204.146.33.66 from 204.146.33.66 (204.146.33.2)
Origin IGP, metric 20, localpref 100, valid, internal
3
204.146.33.6 from 204.146.33.6 (10.4.1.1)
Origin IGP, metric 30, valid, external
The bestpath will always be the same!
68
Inconsistent Decision—
Symptom II
The bestpath changes every time the
peering is reset
routerA#show ip bgp 7.0.0.0
BGP routing table entry for 7.0.0.0/8, version 15
Paths: (2 available, best #2)
Not advertised to any peer
2
1.1.1.5 from 1.1.1.5 (1.1.1.1)
Origin IGP, metric 0, localpref 100, valid, external
2
21.21.21.254 from 21.21.21.254 (7.75.7.1)
Origin IGP, metric 0, localpref 100, valid, external, best
69
Inconsistent Decision—
Symptom II (Cont.)
routerA#show ip bgp 7.0.0.0
BGP routing table entry for 7.0.0.0/8, version 17
Paths: (2 available, best #2)
Not advertised to any peer
2
21.21.21.254 from 21.21.21.254 (7.75.7.1)
Origin IGP, metric 0, localpref 100, valid, external
2
1.1.1.5 from 1.1.1.5 (1.1.1.1)
Origin IGP, metric 0, localpref 100, valid, external, best
The “oldest” external is the bestpath.
All other attributes are the same
Stability enhancement!
70
Routes Flapping
BGP learned routes change periodically
resulting in high cpu
BGP prefixes bounces between a suboptimal
path through a different provider and a
recursive route
Recursive routing occurs when the next hop
address of a network is an address of that
network itself
71
Routes Flapping (Contd..)
72
Routes Flapping (Contd..)
The DMZ address space between AS1 and
AS3 (161.108.0.0/16) also belongs to AS3.
The address 161.108.0.3 is the next hop
address advertised by AS3-EBGP toward AS1IBGP.
Router AS1-IBGP is NOT advertising the DMZ
address space into AS1 via the IGP. This in
combination with not having next-hop-self
configured
73
Routes Flapping (Contd..)
S2-R3-2500#show ip bgp 192.168.1.0
BGP routing table entry for 192.168.1.0/24, version 0
Paths: (1 available, no best path)
3
161.108.0.3 (inaccessible) from 172.108.10.1
Origin IGP, metric 0, localpref 100, valid, internal
74
Routes Flapping (Contd..)
S2-R3-2500#show ip bgp 161.108.0.0
BGP routing table entry for 161.108.0.0/16, version 0
Paths: (1 available, no best path)
3
161.108.0.3 (inaccessible) from 172.108.10.1
Origin IGP, metric 0, localpref 100, valid, internal
75
Routes Flapping (Contd..)
After EBGP session comes up
S2-R3-2500#show ip bgp 192.168.1.0
BGP routing table entry for 192.168.1.0/24, version 4
Paths: (2 available, best #1, advertised over IBGP)
23
131.108.50.1 from 131.108.50.1 (10.1.1.1)
Origin IGP, localpref 100, valid, external, best
3
161.108.0.3 (inaccessible) from 172.108.10.1
Origin IGP, metric 0, localpref 100, valid, internal
S2-R3-2500#
76
Routes Flapping (Contd..)
The next time that BGP scans the routing table, we have a route to the
next hop 161.108.0.3 via AS2
S2-R3-2500#
BGP: scanning routing tables
RT: del 161.108.0.0 via 131.108.50.1, bgp metric [20/0]
RT: delete network route to 161.108.0.0
RT: add 161.108.0.0/16 via 161.108.0.3, bgp metric [200/0]
RT: del 192.168.1.0 via 131.108.50.1, bgp metric [20/0]
RT: delete network route to 192.168.1.0
RT: add 192.168.1.0/24 via 161.108.0.3, bgp metric [200/0]
77
Routes Flapping (Contd..)
These routes, once installed This inconsistency is discovered the next time that BGP
computes an update for his AS2 neighbor, are recursive.
S2-R3-2500#
BGP: 131.108.50.1 computing updates, neighbor version 2, table version 10,
starting at 0.0.0.0
RT: recursion error routing 161.108.0.3 - probable routing loop
RT: recursion error routing 161.108.0.3 - probable routing loop
BGP: 131.108.50.1 update run completed, ran for 8ms, neighbor version 2, start
version 10, throttled to 10, check
point net 0.0.0.0
78
Routes Flapping (Contd..)
These unreachable routes will be marked appropiately the next time BGP scans the
routing table and will be removed.
S2-R3-2500#show ip bgp 192.168.1.0
BGP routing table entry for 192.168.1.0/24, version 10
Paths: (2 available, best #2, advertised over IBGP, EBGP)
23
131.108.50.1 from 131.108.50.1 (10.1.1.1)
Origin IGP, localpref 100, valid, external
3
161.108.0.3 from 172.108.10.1
Origin IGP, metric 0, localpref 100, valid, internal, best
79
Routes Flapping (Contd..)
We can see that the routes are changed and the routes via the next hop address of
161.108.0.3 are removed the next time that BGP scans:
BGP: scanning routing tables
RT: del 161.108.0.0 via 161.108.0.3, bgp metric [200/0]
RT: delete network route to 161.108.0.0
RT: add 161.108.0.0/16 via 131.108.50.1, bgp metric [20/0]
RT: del 192.168.1.0 via 161.108.0.3, bgp metric [200/0]
RT: delete network route to 192.168.1.0
RT: add 192.168.1.0/24 via 131.108.50.1, bgp metric [20/0]
S2-R3-2500#
80
Routes Flapping (Contd..)
S2-R3-2500#show ip bgp 192.168.1.0
BGP routing table entry for 192.168.1.0/24, version 12
Paths: (2 available, best #1, advertised over IBGP, EBGP)
23
131.108.50.1 from 131.108.50.1 (10.1.1.1)
Origin IGP, localpref 100, valid, external, best
3
161.108.0.3 (inaccessible) from 172.108.10.1
Origin IGP, metric 0, localpref 100, valid, internal
81
BGP Case Studies
82
© 2002, Cisco Systems, Inc. All rights reserved.
82
Case 1：Peer Establishment
Missing Routes
Inconsistent Route Selection
Loops and Convergence Issues
Add CERNET CASE
Peer Establishment
Routers establish a TCP session
Port 179—permit in ACLs
IP connectivity (route from IGP)
OPEN messages are exchanged
Peering addresses must match the
TCP session
Local AS configuration parameters
84
Common Problems
Sessions are not established
No IP reachability
Incorrect configuration
Peers are flapping
Layer 2 problems
85
Peer Establishment—Diagram
1.1.1.1
2.2.2.2
IBGP
?
R1
R2
EBGP
3.3.3.3
AS 1
R3
?
AS 2
R2#sh run | begin bgp
router bgp 1
bgp log-neighbor-changes
Neighbor 1.1.1.1 remote-as 1
Neighbor 3.3.3.3 remote-as 2
86
Peer Establishment—
Symptoms
R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor
V
AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State
1.1.1.1
4
1
0
0
0
0
0 never
Active
3.3.3.3
4
2
0
0
0
0
0 never
Idle
Both peers are having problems
State may change between active,
idle and connect
87
Peer Establishment
Is the remote-as assigned correctly?
Verify with your diagram or other documentation!
Local AS
R2#
router bgp 1
neighbor 1.1.1.1 remote-as 1
neighbor 3.3.3.3 remote-as 2
iBGP Peer
eBGP Peer
88
Peer Establishment—IBGP
Assume that IP connectivity has been checked
Check TCP to find out what connections we are accepting
R2#show tcp brief all
TCB
Local Address
005F2934 *.179
0063F3D4 *.179
Foreign Address
3.3.3.3.*
1.1.1.1.*
(state)
LISTEN
LISTEN
We are listening for TCP connections for port 179 for the configured peering addresses
only!
R2#debug ip tcp transactions
TCP special event debugging is on
R2#
TCP: sending RST, seq 0, ack 2500483296
TCP: sent RST to 4.4.4.4:26385 from 2.2.2.2:179
Remote is trying to open the session from 4.4.4.4 address …
89
Peer Establishment—IBGP
What about us ?
R2#debug ip bgp
BGP debugging is on
R2#
BGP: 1.1.1.1 open active, local address 4.4.4.5
BGP: 1.1.1.1 open failed: Connection refused by remote host
We are trying to open the session from 4.4.4.5 address…
R2#sh ip route 1.1.1.1
Routing entry for 1.1.1.1/32
Known via "static", distance 1, metric 0 (connected)
* directly connected, via Serial1
Route metric is 0, traffic share count is 1
R2#show ip interface brief | include Serial1
Serial1
4.4.4.5
YES manual up
up
90
Peer Establishment—IBGP
Source address is the outgoing interface towards the
destination but peering in this case is using loopback
interfaces!
Force both routers to source from the
correct interface
Use “update-source” to specify the loopback when
loopback peering
R2#
router bgp 1
neighbor 1.1.1.1
neighbor 1.1.1.1
neighbor 3.3.3.3
neighbor 3.3.3.3
remote-as 1
update-source Loopback0
remote-as 2
update-source Loopback0
91
Peer Establishment—
Symptoms
R1 is established now
The EBGP session is still having trouble!
R2# sh ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.1 4
1
7
7
1
0
0 00:00:24
3
3.3.3.3 4
2
0
0
0
0
0 never
Idle
92
Peer Establishment—EBGP
Trying to load-balance over multiple links to the eBGP
peer
Verify IP connectivity
Check the routing table
Use ping/trace to verify two way
reachability
R2#ping 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/8 ms
Routing Towards Destination Correct, but…
93
Peer Establishment—EBGP
R2# ping ip
Target IP address: 3.3.3.3
Repeat count [5]:
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface: 2.2.2.2
Type of service [0]:
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)
No Route Back from our Peer!
94
Peer Establishment—EBGP
Neighbor Added Route but Still Having Problems?…
R2#sh ip bgp neigh 3.3.3.3
BGP neighbor is 3.3.3.3, remote AS 2, external link
BGP version 4, remote router ID 0.0.0.0
BGP state = Idle
Last read 00:00:04, hold time is 180, keepalive interval is 60 seconds
Received 0 messages, 0 notifications, 0 in queue
Sent 0 messages, 0 notifications, 0 in queue
Route refresh request: received 0, sent 0
Default minimum time between advertisement runs is 30 seconds
For address family: IPv4 Unicast
BGP table version 1, neighbor version 0
Index 2, Offset 0, Mask 0x4
0 accepted prefixes consume 0 bytes
Prefix advertised 0, suppressed 0, withdrawn 0
Connections established 0; dropped 0
Last reset never
External BGP neighbor not directly connected.
No active TCP connection
95
Peer Establishment—EBGP
router bgp 1
neighbor 3.3.3.3 remote-as 2
neighbor 3.3.3.3 ebgp-multihop 255
neighbor 3.3.3.3 update-source Loopback0
eBGP peers are normally directly connected
By default, TTL is set to 1 for eBGP peers
If not directly connected, specify ebgp-multihop
At this point, the session should come up
96
Peer Establishment—EBGP
R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 1
Neighbor
3.3.3.3
V
4
AS MsgRcvd MsgSent
2
10
26
TblVer
0
InQ OutQ Up/Down
0
0 never
State/PfxRcd
Active
Still having trouble!
Connectivity issues have already been
checked and corrected
97
Peer Establishment—EBGP
R2#debug ip bgp events
14:06:37: BGP: 3.3.3.3 open active, local address 2.2.2.2
14:06:37: BGP: 3.3.3.3 went from Active to OpenSent
14:06:37: BGP: 3.3.3.3 sending OPEN, version 4
14:06:37: BGP: 3.3.3.3 received NOTIFICATION 2/2
(peer in wrong AS) 2 bytes 0001
14:06:37: BGP: 3.3.3.3 remote close, state CLOSEWAIT
14:06:37: BGP: service reset requests
14:06:37: BGP: 3.3.3.3 went from OpenSent to Idle
14:06:37: BGP: 3.3.3.3 closing
If an error is detected, a notification is sent and the session
is closed
R3 is configured incorrectly
Has “neighbor 2.2.2.2 remote-as 10”
Should have “neighbor 2.2.2.2 remote-as 1”
After R3 makes this correction the session comes up
98
Case 2：Load
Balancing
Parallel Links
eBGP Multipath
Multi-homed
Command
neighbor {ip-address | peer-groupname} ebgp-multihop [ttl]
To accept and attempt BGP connections to external peers
residing on networks that are not directly connected, use
the neighbor ebgp-multihop command in router
configuration mode.
100
Command
neighbor {ip-address | peer-group-
name} update-source interface-type
To have the Cisco IOS software allow Border Gateway
Protocol (BGP) sessions to use a specific operational
interface for TCP connections, use the neighbor updatesource command in router configuration mode.
To restore the interface assignment to the closest
interface, which is called the best local address, use the
no form of this command.
101
Command
maximum-paths maximum-number
To control the maximum number of parallel
routes an IP routing protocol can support,
use the maximum-paths command in
address family or router configuration
mode. To restore the default value (1), use
the no form of this command.
102
Load Balancing
BGP is not designed to load-balance traffic
BGP chooses and installs one “best” route
“Attempting” to balance traffic comes in two parts
Inbound traffic
Outbound traffic
Load balancing is possible in some topologies
A pair of eBGP peers connected via multiple links
Two connections from one router to the same AS
But not others
Multi-homed to more than one provider
103
Single Path
Router A:
interface loopback 0
RA must do a recursive lookup for 2.2.2.2
ip address 1.1.1.1 255.255.255.255
RA has two equal cost paths to 2.2.2.2
!
RA will load balance traffic over these two links
router bgp 100
neighbor 2.2.2.2 remote-as 200
Both inbound and outbound traffic will be balanced
neighbor 2.2.2.2 update-source loopback0
neighbor 2.2.2.2 ebgp-multi-hop
!
ip route 2.2.2.2 255.255.255.255 serial 0
ip route 2.2.2.2 255.255.255.255 serial 1
!
B
Loopback 0
2.2.2.2/32
A
200
100
104
eBGP Multipath Support
100
200
Router peers with multiple routers in the same neighbor AS
Install multiple routes in IP routing table
Use ‘maximum-paths’ command
Routes must be identical in terms of LOCAL_PREF, AS_PATH, MED,
etc…
Outbound traffic will be split over these two links
BGP still advertises one best path
Next-hop is set to self (use loopback interface)
105
Multi-Homed AS
AS 100
AS 300
D
A
B
C
AS 200
Very common topology for many customers
Customer wants to split traffic between AS 100 and AS 300
Misconception: “I’ll make half of my routes preferred via AS
100 and the other half through AS 300. Then I’ll have loadbalancing!!”
106
Multi-Homed AS
This does not provide load balancing, just “prefix splitting”
Traffic may be balanced perfectly until traffic patterns change
Load balancing is now over :(
Some customers use this method but they are forced to change
their policies to accommodate for changes in traffic patterns
For outbound balancing use
Weight
LOCAL_PREF (recommended)
For inbound balancing use
Conditional-advertisement
AS_PATH prepending (may not work)
MEDs (may not work)
Communities and LOCAL_PREF (recommended)
107
Case 3：BGP community
community-list
route-map
109
Demand
Backup
link
The backup path of KOREN
access TEIN2 backbone via
CERNET
110
Demand (Contd..)
The backup path of TEIN2
Beijing POP and TEIN2
Singapore POP via CERNETKOREN connection
Backup
link
111
The detail demands
CERNET advertises all the routers from TEIN2 Peking POP (containing
community 24489:65200）except for the router containing community
24489:65500 to KOREN, advertising such routers to KOREN, CERNET
does the as-path-prepending (adding one “4538”）in order to avoid
KOREN selecting the path from CERNET to TEIN2-NORTH priorityr;
CERNET receives all the routers ( containing community
24489:65200）transmitting from TEIN2 Singapore POP from KOREN,
adds community 4538:9270, and advertises those routers to TEIN2
Peking POP;
CERNET receives all the routers ( containing community 9270:65155）
transmitting from KOREN internal R&E network from KOREN, adds the
community 4538:9270, and advertises those routers to the TEIN2
Peking POP;
When the routers containing the community 4538:9270 are received
from TEIN2 Peking POP, and the routers containing the community
4538:9270 are advertised to TEIN2 Peking POP, any restricting and
adjusting should not be done, including the IPv4 and IPv6 router.
112
Configuration
router bgp 4538
bgp router-id 202.112.60.247
no bgp fast-external-fallover
bgp log-neighbor-changes
bgp deterministic-med
neighbor 202.112.61.86 remote-as 9270
neighbor 202.112.61.86 description KOREN
neighbor 202.112.61.86 activate
neighbor 202.112.61.86 send-community
neighbor 202.112.61.86 soft-reconfiguration inbound
neighbor 202.112.61.86 route-map KORENtoTEIN2 in
neighbor 202.112.61.86 route-map TEIN2toKOREN out
neighbor 202.112.61.86 activate
neighbor 202.112.61.86 soft-reconfiguration inbound
neighbor 202.112.61.86 filter-list 12 out
neighbor 202.179.241.25 remote-as 24489
neighbor 202.179.241.25 description TEIN2
neighbor 202.179.241.25 password 7 02040D550C0A0A774F410615
neighbor 202.179.241.25 activate
neighbor 202.179.241.25 send-community
neighbor 202.179.241.25 soft-reconfiguration inbound
neighbor 202.179.241.25 distribute-list 26 in
neighbor 202.179.241.25 route-map tein2-in in
neighbor 202.179.241.25 route-map CERNETtoTEIN2 out
TEIN2
AS24489
.25
202.179.241/30
2001:254:1:7::2/64
.85
.26
CERNET
AS4538
.86
202.112.61/30
2001:250:0:30::1/64
KOREN
AS9270
113
Configuration (Contd..)
route-map TEIN2toKOREN permit 10
match community TEIN2bjpop
set as-path prepend 4538
!
route-map TEIN2toKOREN permit 20
match as-path 12
match community TEIN2bjpop1
!
route-map TEIN2toKOREN permit 30
match ip address prefix-list CERNET2
!
route-map tein2-in permit 10
set local-preference 330
set community 4538:24489 additive
!
route-map CERNETtoTEIN2 permit 10
match community CERNETtoTEIN2
!
route-map CERNETtoTEIN2 permit 20
match ip address prefix-list CERNET2
set community 4538:65155 additive
route-map KORENtoTEIN2 permit 10
match community KORENtoTEIN2
set local-preference 330
set as-path prepend 4538
set community 4538:9270 additive
!
route-map KORENtoTEIN2 permit 20
match ip address prefix-list KOREN-filter
set local-preference 330
set as-path prepend 4538
set community 4538:9270 additive
TEIN2
AS24489
.25
202.179.241/30
2001:254:1:7::2/64
.85
.26
CERNET
AS4538
.86
202.112.61/30
2001:250:0:30::1/64
KOREN
AS9270
114
Configuration (Contd..)
ip community-list
ip community-list
ip community-list
ip community-list
ip community-list
ip community-list
ip community-list
TEIN2
AS24489
standard TEIN2bjpop deny 24489:65500
standard TEIN2bjpop permit 24489:65200
standard TEIN2bjpop1 deny 24489:65500
standard KORENtoTEIN2 permit 24489:65200
standard KORENtoTEIN2 permit 9270:65155
standard CERNETtoTEIN2 permit 4538:9270 9270:65155
standard CERNETtoTEIN2 permit 4538:9270 24489:65200
.25
202.179.241/30
2001:254:1:7::2/64
.85
.26
CERNET
AS4538
.86
202.112.61/30
2001:250:0:30::1/64
KOREN
AS9270
115
Show commands
bj-bgw-r0a#sh ip bgp sum
BGP router identifier 202.112.60.247, local AS number 4538
BGP table version is 61059, main routing table version 61059
219479 network entries using 22167379 bytes of memory
353680 path entries using 16976640 bytes of memory
53580 BGP path attribute entries using 3000592 bytes of memory
74 BGP rrinfo entries using 1776 bytes of memory
41271 BGP AS-PATH entries using 1169092 bytes of memory
2465 BGP community entries using 173974 bytes of memory
2 BGP extended community entries using 48 bytes of memory
11515 BGP route-map cache entries using 368480 bytes of memory
776 BGP filter-list cache entries using 9312 bytes of memory
BGP using 43867293 total bytes of memory
249339 received paths for inbound soft reconfiguration
BGP activity 228135/1064 prefixes, 768702/399806 paths, scan interval 60 secs
Neighbor
V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
193.62.157.29 4 786
0
0
0 0 0 never Idle
202.112.60.246 4 4538 576 161 61059 0 0 02:37:48 16791
202.112.60.251 4 4538 4252 4407 61059 0 0 02:37:53 47349
202.112.61.14 4 23911 89332 192 61059 0 0 02:38:39
4
202.112.61.46 4 9264 775 344 61059 0 0 02:37:40
132
202.112.61.86 4 9270 2294 913 61055 0 0 02:38:50 10038
202.112.61.94 4 9505 215 186 61059 0 0 02:37:41
366
202.179.241.25 4 24489 3896 420 61055 0 0 02:38:44 9589
203.181.194.125 4 7660 4626 1275 61055 0 0 02:38:19 10036
203.181.194.126 4 7660 9799 1116 61055 0 0 02:37:16 10031
203.222.180.225 4 1239 44795 215 61059 0 0 02:37:30
3
bj-bgw-r0a#
116
Bgp session end
Trouble Shooting OSPF
Agenda
LSA Overview
Troubleshooting Commands
Common Issues
119
LSA OVERVIEW
RST-3301
9721_05_2004_c2
120
© 2004 Cisco Systems, Inc. All rights reserved.
120
LSA Type Review
Type
LSA
1
Router
2
Network
3
Summary Network
4
Summary ASBR
5
External
6
Group Membership
7
NSSA
8
External Attributes
9–11
Opaque
121
Router LSA Details
Router LSA (Type 1)
Describes the state and cost of the router’s links
to the area
All of the router’s links in an area must be
described
in a single LSA
Flooded throughout the particular area and no
more
Router indicates whether it is an ASBR, ABR,
or end point of virtual link
122
Router LSA of R3 for Area 1
R3#show ip ospf database router 3.3.3.3
Router Link States (Area 1)
DR
192.1.1.4
LS age = 0
Options = (E-bit)
LS type = 1
Link State ID = 3.3.3.3
Advertising Router = 3.3.3.3
It is an area border router
# links = 2
Link ID = 192.1.1.4
Link Data = 192.1.1.3
Type = 2
# TOS metrics = 0
metric = 1
Link ID = 192.1.4.0
Link Data = 255.255.255.0
Type = 3
# TOS metrics = 0
metric = 2
Always 0 at origination
1
R4
Area 0
192.1.1.3
This is a router LSA
Router ID of R3
Router ID of R3
bit B = 1
R3
2
192.1.4.0/24
IP address of the DR
Interface address of this router
This is a transit network
Cost to reach the interface
IP network number
Subnet mask of the interface
Stub network
123
Router LSA of R3 for Area 0
(Cont.)
Router Link States (Area 0)
DR
LS age = 0
Options = (E-bit)
LS type = 1
Link State ID = 3.3.3.3
Advertising Router = 3.3.3.3
It is an area border router
# links = 2
Link ID = 6.6.6.6
Link Data = 18.10.0.5
Type = 1
# TOS metrics = 0
metric = 8
Link ID = 18.10.0.4
Link Data = 255.255.255.252
Type = 3
# TOS metrics = 0
metric = 8
192.1.1.4
1
R4
192.1.1.3
R3
bit B = 1
Area 0
18.10.0.5/30
2
8
6.6.6.6
R6
192.1.4.0/24
Router id of the neighbor
IP interface address of the router
This is a point-to-point link
IP subnet address
Subnet mask
This is a stub link
124
Router LSA Details
Type
Description
Link ID
Link Data
1
Point-to-Point
Numbered
Neighbors’
RID
Interface IP
Address
1
Point-to-Point
Unnumbered
Neighbors’
RID
MIB-II Ifindex
Value
2
Transit
IP Address
of the DR
Interface IP
Address
3
Stub
IP Network
Number
Subnet Mask
4
Virtual Link
Neighbors’
RID
Interface IP
Address
125
Network LSA
Network LSA (Type 2)
Generated for every transit broadcast and
NBMA network
Describes all the routers attached to the
network
Only the designated router originates this
LSA
Flooded throughout the area and no more
126
Network LSA for 192.1.1.0
R3#show ip ospf database network 192.1.1.4
Network Link States (Area 1)
LS age = 948
Options = (E-bit)
LS type = 2
Link State ID = 192.1.1.4
IP interface address of DR
Advertising Router = 4.4.4.4
RID of DR
Network Mask = 255.255.255.0
Attached Router = 4.4.4.4
Attached Router = 3.3.3.3
RID of attached routers FULL with the DR
Attached Router = 2.2.2.2
4.4.4.4
DR
Attached Router = 1.1.1.1
1.1.1.1
R1
2.2.2.2
R2
192.1.1.
4
/24
1
R4
Area 0
6.6.6.6
192.1.1.
3
R3
8
2 18.10.0.4/32
R6
192.1.4.0/24
127
Summary LSA
Describes the destination outside the area but
still in the AS
Summary is created for each IP subnets in
one area and is flooded out in all other areas
Originated by an ABR
Only intra-area routes are advertised into the
backbone
Type 4 is the information about the ASBR
128
Type 3 Details
R4#show ip ospf database summary 192.1.2.0
Summary Net Link States (Area 0)
LS age = 1514
Options = (E-bit)
LS type = 3
Link State ID = 192.1.2.0
IP network number
Advertising Router = 4.4.4.4
RID of ABR
Network Mask = 255.255.255.0
ABR
metric = 4
DR
O IA
192.1.2.0/24
metric 4
1.1.1.1
192.1.2.0/24
3
R1
2.2.2.2
R2
192.1.1.
4
/24
1 192.1.1.
3
R4
ABR
R3
8
Area 0
6.6.6.6
8
2 18.10.0.4
R6
192.1.4.0/24
129
Type 4 Details
R4#show ip ospf database asbr-summary 7.7.7.7
Summary ASB Link States (Area 0)
LS age = 1548
Options = (E-bit)
LS type = 4
Link State ID = 7.7.7.7
RID of ASBR
Advertising Router = 4.4.4.4
RID of ABR
Network Mask = 0.0.0.0
Metric = 16
Type 4 Summary
ABR
3
192.1.1.
4
/24
R1
2.2.2.2
R2
140.10.0.0
DR
1.1.1.1
192.1.2.0/24
External Route
(4.4.4.4)
1
192.1.1.
3
8
R4
2
Area 0
RID ASBR
7.7.7.7
6.6.6.6
R3
R3
8
8
18.10.0.4
192.1.4.0/24
R6
R7
130
External LSA
External LSA (Type 5)
Defines routes to destination external
to the AS
Default route is also sent as external
Two types of external LSA:
E1: Consider the total cost up to the
external destination
E2: Considers only the cost of the outgoing
interface to the external destination
131
Type 5 Details
R4#show ip ospf database external 140.10.0.0
LS age = 1156
Options = (E-bit)
LS type = 5
Link State ID = 140.10.0.0
Advertising Router = 7.7.7.7
Network Mask = 255.255.0.0
Metric Type: 2
metric = 20
Forwarding address = 0.0.0.0
IP network number
Router ID of R7
bit E = 1 -> O E2 (Default)
The metric is 20 in all redistributed E2 routes
Traffic should be forwarded to the ASBR
External Route
3
192.1.2.0/24
1.1.1.1
R1
2.2.2.2
R2
140.10.0.0
DR
192.1.1.
4
/24
8
R4
8
Area 0
1 192.1.1.
3
RID ASBR
7.7.7.7
6.6.6.6
R3
2
192.1.4.0/24
18.10.0.4
R6
R7
External Type 5
132
NSSA External LSA
NSSA External LSA (Type 7) RFC1587
NSSA was created to inject external routes
from stub area into OSPF domain
Redistribution in NSSA creates Type 7 LSA
Generated by the NSSA ASBR
Type 7 can only exist in NSSA area
NSSA ABR does the translation from 7–5
133
TROUBLESHOOTING
COMMANDS
RST-3301
9721_05_2004_c2
134
© 2004 Cisco Systems, Inc. All rights reserved.
134
Show IP OSPF
R3#show ip ospf
Routing Process "ospf 1" with ID 3.3.3.3 and Domain ID 0.0.0.1
Supports only single TOS(TOS0) routes
Supports opaque LSA
It is an area border router
SPF schedule delay 5 secs, Hold time between two SPFs 10 secs
Minimum LSA interval 5 secs. Minimum LSA arrival 1 secs
Number of external LSA 1. Checksum Sum 0x3B57
Number of opaque AS LSA 0. Checksum Sum 0x0
Number of DCbitless external and opaque AS LSA 0
Number of DoNotAge external and opaque AS LSA 0
Number of areas in this router is 2. 2 normal 0 stub 0 nssa
External flood list length 0
Area BACKBONE(0)
Number of interfaces in this area is 2
Area has no authentication
SPF algorithm executed 2773 times
Area ranges are
Number of LSA 17. Checksum Sum 0x686B5
Number of opaque link LSA 0. Checksum Sum 0x0
Number of DCbitless LSA 0
Number of indication LSA 0
Number of DoNotAge LSA 9
Flood list length 0
135
Show IP OSPF (Cont.)
...
Area 1
Number of interfaces in this area is 2
Area has no authentication
SPF algorithm executed 22 times
Area ranges are
Number of LSA 19. Checksum Sum 0x8FE73
Number of DCbitless LSA 0
Number of indication LSA 0
Number of DoNotAge LSA 0
Flood list length 0
136
Show IP OSPF Database
R3#show ip ospf database
OSPF Router with ID (3.3.3.3) (Process ID 1)
Router Link States (Area 0)
Link ID
3.3.3.3
...
ADV Router
3.3.3.3
Age
106
Seq#
Checksum Link count
0x80000009 0xC3F1
3
Summary Net Link States (Area 0)
Link ID
ADV Router
Age
Seq#
Checksum
18.10.0.0 7.7.7.7
3 (DNA) 0x80000008 0x3DC2
18.10.0.0 8.8.8.8
1396
0x80000004 0x27D8
...
Router Link States (Area 1)
Link ID
1.1.1.1
...
ADV Router
1.1.1.1
Age
671
Seq#
Checksum Link count
0x80000016 0xE6CD 2
137
Show IP OSPF Database
Database-Summary
R3#show ip ospf database database-summary
OSPF Router with ID (3.3.3.3) (Process ID 1)
Area 0 database summary
LSA Type
Count
Router
6
Network
4
Summary Net
10
Summary ASBR
0
Type-7 Ext
0
Opaque Link
0
Opaque Area
0
Subtotal
20
Area 1 database summary
LSA Type
Count
Router
4
Network
1
Summary Net
10
Summary ASBR
4
...
Delete
0
0
0
0
0
0
0
0
Maxage
0
0
0
0
0
0
0
0
Delete
0
0
0
0
Maxage
0
0
0
0
138
Show IP OSPF Neighbor
R3#show ip ospf neighbor
Neighbor ID
1.1.1.1
2.2.2.2
4.4.4.4
6.6.6.6
R3#
Pri
1
1
1
1
State
Dead Time
FULL/DROTHER 00:00:33
FULL/DROTHER 00:00:32
FULL/DR
00:00:39
FULL/ 00:00:38
R1
2.2.2.2
R2
192.1.1.
4
/24
1
Interface
FastEthernet0/0
FastEthernet0/0
FastEthernet0/0
Serial0/0
4.4.4.4
DR
1.1.1.1
Address
192.1.1.1
192.1.1.2
192.1.1.4
18.10.0.6
R4
Area 0
6.6.6.6
192.1.1.
3
R3
8
2 18.10.0.4
R6
192.1.4.0/24
139
OSPF Log-Adjacency-Changes
Default as of 12.1.3 and 12.0.12S
R3#config terminal
R3(config)#router ospf 1
R3(config-router)#log-adjacency-changes
%OSPF-5-ADJCHG: Process 1, Nbr 1.1.1.1 on FastEthernet0/0 from 2WAY to DOWN,
Neighbor Down: Interface down or detached
%OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from FULL to DOWN,
Neighbor Down: Interface down or detached
%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet0/0 from FULL to DOWN,
Neighbor Down: Interface down or detached
%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on OSPF_VL0 from FULL to DOWN, Neighbor
Down: Interface down or detached
%LINK-3-UPDOWN: Interface FastEthernet0/0, changed state to up
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/0, changed state to up
%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on FastEthernet0/0 from LOADING to FULL,
Loading Done
%OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL,
Loading Done
%OSPF-5-ADJCHG: Process 1, Nbr 4.4.4.4 on OSPF_VL0 from LOADING to FULL,
140
Loading Done
Show IP OSPF Neighbor Detail
R3#show ip ospf neighbor detail
Neighbor 1.1.1.1, interface address 192.1.1.1
In the area 1 via interface FastEthernet0/0
Neighbor priority is 1, State is 2WAY, 2 state changes
DR is 192.1.1.4 BDR is 192.1.1.2
Options is 0x2
Dead timer due in 00:00:39
Neighbor is up for 00:06:30
Index 0/0, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
Neighbor 2.2.2.2, interface address 192.1.1.2
In the area 1 via interface FastEthernet0/0
Neighbor priority is 1, State is FULL, 6 state changes
DR is 192.1.1.4 BDR is 192.1.1.2
Options is 0x42
Dead timer due in 00:00:38
Neighbor is up for 00:06:31
Index 2/2, retransmission queue length 0, number of retransmission 0
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 0, maximum is 0
Last retransmission scan time is 0 msec, maximum is 0 msec
141
Show IP OSPF Interface
R3#show ip ospf interface
FastEthernet0/0 is up, line protocol is up
Internet Address 192.1.1.3/24, Area 1
Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 1
Transmit Delay is 1 sec, State DROTHER, Priority 1
Designated Router (ID) 4.4.4.4, Interface address 192.1.1.4
Backup Designated router (ID) 2.2.2.2, Interface address 192.1.1.2
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:03
Index 1/1, flood queue length 0
Next 0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 5
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 3, Adjacent neighbor count is 2
Adjacent with neighbor 2.2.2.2 (Backup Designated Router)
Adjacent with neighbor 4.4.4.4 (Designated Router)
Suppress hello for 0 neighbor(s)
142
Show IP OSPF Virtual-Links
R3#show ip ospf virtual-links
Virtual Link OSPF_VL0 to router 4.4.4.4 is up
Run as demand circuit
DoNotAge LSA allowed.
Transit area 1, via interface FastEthernet0/0, Cost of using 1
Transmit Delay is 1 sec, State POINT_TO_POINT,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
Hello due in 00:00:09
Adjacency State FULL (Hello suppressed)
Index 1/3, retransmission queue length 0, number of retransmission 1
First 0x0(0)/0x0(0) Next 0x0(0)/0x0(0)
Last retransmission scan length is 1, maximum is 1
Last retransmission scan time is 0 msec, maximum is 0 msec
R3#
143
Show IP OSPF Stat
Requires enable mode
R3#sh ip ospf stat
Area 0: SPF algorithm executed 42 times
Area 1: SPF algorithm executed 38 times
SPF calculation time
Delta T Intra D-Intra Summ D-Summ Ext D-Ext Total
00:22:00 0
0
0
0
0
0
0
00:21:44 0
0
4
0
0
0
4
00:21:34 0
0
4
0
0
0
4
00:21:24 0
0
0
4
0
0
4
00:21:14 0
0
0
0
0
0
0
00:21:04 0
0
0
0
0
0
0
00:20:54 0
0
0
0
0
0
0
00:20:44 0
0
4
0
0
0
4
00:20:34 0
0
0
0
0
0
0
00:00:17 4
0
0
0
0
0
4
...
R=Router LSA; N=NetworkLSA; SN=Summary Network LSA;
ASBR LSA; X=External LSA
Reason
R, N, SN,
R, SN, X
R, SN, X
R, SN, X
R,
R, N, SN,
X
R, SN, X
X
R, N, SN, SA, X
SA=Summary
144
Show IP OSPF Borders
R3#show ip ospf borders-routers
OSPF Process 1 internal Routing Table
Codes: i - Intra-area route, I - Inter-area route
i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 0, SPF 42
i 4.4.4.4 [1] via 192.1.1.4, FastEthernet0/0, ABR, Area 1, SPF 38
i 8.8.8.8 [10] via 18.10.0.6, Serial0/0, ABR/ASBR, Area 0, SPF 42
i 7.7.7.7 [17] via 192.1.1.4, FastEthernet0/0, ABR/ASBR, Area 0, SPF 42
145
Other Show Commands
R3#show ip ospf database self-originate
OSPF Router with ID (3.3.3.3) (Process ID 1)
Router Link States (Area 0)
Link ID
ADV Router
Age
Seq#
Checksum Link count
3.3.3.3
3.3.3.3
1520
0x80000015
0xABFD
2
Summary Net Link States (Area 0)
Link ID
ADV Router
Age
Seq#
Checksum
192.1.1.0
3.3.3.3
1520
0x80000006
0x4E1A
192.1.2.0
3.3.3.3
1521
0x80000006
0x6103
...
Router Link States (Area 1)
Link ID
ADV Router
Age
Seq#
Checksum Link count
3.3.3.3
3.3.3.3
1536
0x80000028
0x612D
2
146
Other Show Commands (Cont.)
R3#show ip ospf database adv-router 7.7.7.7
OSPF Router with ID (3.3.3.3) (Process ID 1)
Router Link States (Area 0)
Link ID ADV Router Age
Seq#
Checksum Link count
7.7.7.7
7.7.7.7
871(DNA) 0x8000000D 0x8FE2 2
Summary Net Link States (Area 0)
Link ID
ADV Router Age
Seq#
Checksum
20.10.0.0 7.7.7.7
871 (DNA) 0x8000000A 0x39C4
Type-5 AS External Link States
Link ID
ADV Router
Age
Seq#
Checksum Tag
140.100.0.0 7.7.7.7
1944
0x80000004 0x3759
0
147
COMMON ISSUES
148
148
Common Issues
Adjacency is not coming up
OSPF neighbor stuck in ? state
Information is in the DB but not in the
RT
SPF running constantly
Neighbor flapping (Frame Relay)
NSSA ABR not translating Type 7 LSA
Demand circuit problems
149
Adjacency Is Not Coming Up
Useful commands for this problem
Show IP OSPF neighbor
Show IP OSPF interface
Debug IP OSPF adjacency
150
Adjacency Is Not Coming Up
Layer 2 is down
R3#show ip ospf neighbor
R3#
R3#show ip ospf interface serial 2
Serial2 is down, line protocol is DOWN
Internet Address 18.10.0.3/30, Area 0
Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT,
Cost: 64
Transmit Delay is 1 sec, State DOWN,
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
151
Adjacency Is Not Coming Up
OSPF not enabled on the interface
R3#show ip ospf neighbor
R3#
R3#show ip ospf interface serial 2
Serial2 is up, line protocol is up
OSPF not enabled on this interface
In 12.0:
R3#show ip ospf interface serial 2
R3#
Tip: Check for the wrong network statement
re-enter the network statement
152
Adjacency Is Not Coming Up
Interface is defined as passive
R3#show ip ospf neighbor
R3#
R3#show ip ospf interface e0
Ethernet0 is up, line protocol is up
Internet Address 192.1.1.3/24, Area 1
Process ID 1, Router ID 3.3.3.3, Network Type BROADCAST, Cost: 10
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 192.1.1.4, Interface address 192.1.1.3
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
No Hellos (Passive interface)
153
Adjacency Is Not Coming Up
Mismatched subnet mask
R3#show ip ospf neighbor
R3#
R3#debug ip ospf adj
OSPF adjacency events debugging is on
R3#
OSPF: Mismatched hello parameters from 192.1.1.4
Dead R 40 C 40, Hello R 10 C 10 Mask R 255.255.255.192 C 255.255.255.0
154
Adjacency Is Not Coming Up
Mismatched hello/dead interval
R3#show ip ospf neighbor
R3#
R3#debug ip ospf adj
OSPF adjacency events debugging is on
R3#
OSPF: Mismatched hello parameters from 192.1.1.4
Dead R 40 C 40, Hello R 15 C 10 Mask R 255.255.255.0 C 255.255.255.0
R4(config-if)#interface ethernet 0
R4(config-if)#no ip ospf hello-interval 15
Tip: Default is 10 second on LAN
155
Adjacency Is Not Coming Up
Mismatched authentication key
R3#show ip ospf neighbor
R3#
R3#debug ip ospf adj
OSPF adjacency events debugging is on
R3#
OSPF: Rcv pkt from 192.1.1.4, Ethernet0 : Mismatch Authentication Key Clear Text
Tip: Watch for the “space” at the end of the Authentication key
156
Adjacency Is Not Coming Up
Mismatched area ID
R4#show ip ospf neighbor
R4#
R4#debug ip ospf adj
OSPF adjacency events debugging is on
OSPF: Rcv pkt from 192.1.1.4, Ethernet0, area 0.0.0.1
mismatch area 0.0.0.2 in the header
Neighbor is in area 2 but we are not:
%OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from
backbone area must be virtual-link but not found from 192.1.1.4,
Ethernet0
157
Adjacency Is Not Coming Up
Mismatched transit/stub/NSSA option
7.7.7.7 18.10.0.1
area 2
R7
18.10.0.2 8.8.8.8
area 2 nssa
R8
R7#show ip ospf neighbor
R7#
R7#debug ip ospf adj
OSPF adjacency events debugging is on
OSPF: Hello from 18.10.0.2 with mismatched Stub/Transit area
option bit
158
Options
Normal area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2
flag 0x3 len 492
E bit is 1, Allow externals, option: 0x2(HEX) = 00000010(Bin)
Stub area: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1866 opt 0x0
flag 0x3 len 372
E bit is 0, no external allowed, options: 0x0 = 00000000
NSSA: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x118 opt 0x8 flag 0x3
len 372
N/P bit is on, options: 0x8 = 00001000
DC: OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0x1A1E opt 0x20 flag 0x3
len 392
DC bit is negotiated, options: 0x20 = 00100000
*
O
DC EA N/P MC
E
*
159
OSPF Neighbor Stuck in ?
State
Useful commands for this problem
Show IP OSPF neighbor
Debug IP OSPF adjacency
160
Stuck in ATTEMPT
3.3.3.3
6.6.6.6
NBMA
R3
R6
Hello
RID =3.3.3.3
Hello
RID =6.6.6.6
N = 3.3.3.3
161
Stuck in ATTEMPT
Reasons:
Our hellos are getting lost in NBMA cloud
Neighbor hellos are getting lost in NBMA
cloud
We received neighbor’s hello but rejects it
for some reason
Misconfigured neighbor statement
Broken Unicast
162
Stuck in INIT
1.1.1.1
2.2.2.2
R1
R2
Hello
RID =1.1.1.1
Hello
RID =2.2.2.2
Hello
RID =1.1.1.1
N =2.2.2.2
Hello
RID =2.2.2.2
163
Stuck in INIT
Reasons:
One side is blocking the hello packet
with access-list
One side is translating (NAT) ospf hello
One side multicast capabilities is broken
(Layer 2)
Dialer map or frame-relay map is missing
keyword ‘broadcast’
164
Stuck in 2-WAY
1.1.1.1
2.2.2.2
R1
R2
Hello
RID =1.1.1.1 P=0
Hello
RID =2.2.2.2 P=0
N =1.1.1.1
Hello
RID =1.1.1.1 P=0
N =2.2.2.2
165
Stuck in 2-WAY
Reasons:
This is normal in broadcast network types
This is to reduce the amount of flooding on the
wire
Problem can happen if all the router are
configured
with priority equal to ‘0’
In a situation where you have high and low end
boxes
on the same segment the configure low end
routers with priority 0 so they don’t participate in
166
DR election
Stuck in EXSTART/EXCHANGE
3.3.3.3
R3
6.6.6.6
Hello
R6
RID =3.3.3.3
Hello
RID =6.6.6.6
N =3.3.3.3
DBD
MTU = 1500 flag = 0x7
Seq = 1E55
DBD
MTU = 1500 flag = 0x7
Seq = 22AB
167
Stuck in EXSTART/EXCHANGE
Useful in debugging, defines I, M and MS bits
OSPF: Send DBD to 141.108.97.1 on Serial0 seq 0xBC4 opt 0x2 flag
0x3
len 492
Flag 0x7--> 111 means I(Initial) = 1, M = 1(More), MS = 1(Master)
Flag 0x6 --> 110 not possible
Flag 0x5 --> 101 not possible
Flag 0x4 --> 100 not possible
Flag 0x3 --> 011 means master has more data to send
Flag 0x2 --> 010 means slave has more data to send
Flag 0x1 --> 001 means master has no more data left to send
Flag 0x0 --> 000 means slave has no more data left to send
0
0
0
0
0
M
MS
168
Stuck in EXSTART/EXCHANGE
Reasons:
MTU mismatch
Note: If Cisco IOS is < 12.0.3 neighbor will show stuck in EXCHANGE
Neighbor RID is same as ours.
Note: If Cisco IOS is > 12.0.7, it displays msg: %OSPF-3-DUP_RTRID &
OSPF neighbor list will be empty
Unicast is broken
a. Wrong VC/DLCi mapping in frame/ATM environment in highly
redundant network
b. MTU problem, can’t ping across with more than certain length packet
c. Access-list blocking unicast; after 2-way OSPF send unicast packet
except p2p links
d. NAT is translating unicast packet
Between PRI and BRI/dialer and network type is p2p
169
Stuck in LOADING
3.3.3.3
6.6.6.6
R3
R6
LS Req
LS Type
Link State ID
Advertising Router
LS Update
# LSAs
LSAs
...
170
Stuck in LOADING
Reasons:
LS request is being made and neighbor is sending
bad packet or mem corrupt
a. Do show IP OSPF bad to see bad lsa
b. Show log will show
OSPF-4-BADLSATYPE msg
LS request is being made and neighbor is ignoring
the request
MTU mismatch problem
(RFC 1583 and 2178 compatibility issue)
171
Stuck in LOADING
3.3.3.3
6.6.6.6
MTU = 2048
IOS 11.3.10T
RFC 1583
MTU = 4470
R3
R6
LS Req
IOS 12.0.7T
RFC 2178
LS Type
Link State ID
Advertising Router
LS Update
# LSAs
LSAs
Too Big!
...
172
Information Is in the DB
but Not in the RT
Useful commands for this problem
Show IP OSPF interface <interface>
Show IP OSPF database <x>
Where ‘x’ can be router, network, summary,
summary-asbr, external, nssa
173
Mismatched Network Types
3.3.3.3
Area 0
6.6.6.6
R3
18.10.0.4/30
R6
R3#show ip ospf neighbor
Neighbor ID
6.6.6.6
Pri State
1
FULL/ -
Dead Time Address
00:00:30
18.0.0.6
Interface
Serial0
R3#show ip ospf interface serial 0
Serial0 is up, line protocol is up
Internet Address 18.0.0.5/30, Area 0
Process ID 1, Router ID 3.3.3.3, Network Type POINT_TO_POINT, Cost: 64
R6#show ip ospf interface serial 0
Serial0 is up, line protocol is up
Internet Address 18.0.0.6/30, Area 0
Process ID 1, Router ID 6.6.6.6, Network Type BROADCAST, Cost: 64
174
Mismatched Network Types
(Cont.)
R3#show ip ospf database router 3.3.3.3
...
Link ID = 6.6.6.6
Router id of the neighbor
Link Data = 18.10.0.5
IP interface address
Type = 1
This is a point-to-point link
# TOS metrics = 0
metric = 8
...
R3#show ip ospf database router 6.6.6.6
...
Link ID = 18.10.0.6
IP address of the DR
Link Data = 18.10.0.6
Interface address
Type = 2
This is a transit link
# TOS metrics = 0
metric = 8
3.3.3.3
Area 0
R3
18.10.0.4/30
6.6.6.6
R6
175
Point-to-Point Numbered and
Unnumbered Links
3.3.3.3
192.1.4.1
R3
R3#show ip ospf neighbor
Neighbor ID
6.6.6.6
Pri State
1
FULL/ -
Area 0
6.6.6.6
18.10.0.4/30
R6
Dead Time Address
00:00:30
18.0.0.6
Interface
Serial0
R3#show interface serial0
Serial0 is up, line protocol is up
Hardware is HD64570
Interface is unnumbered.Using address of Ethernet1 (192.1.4.1)
R6#show interface serial0
Serial0 is up, line protocol is up
Hardware is HD64570
Internet address is 18.10.0.6/30
176
Point-to-Point Numbered and
Unnumbered Links (Cont.)
R3#show ip ospf database router 3.3.3.3
...
Link ID = 6.6.6.6
Router id of the neighbor
Link Data = 0.0.0.5
MIBII IfIndex Value
Type = 1
This is a point-to-point link
# TOS metrics = 0
3.3.3.3
metric = 8
...
192.1.4.1 R3
R3#show ip ospf database router 6.6.6.6
...
Link ID = 3.3.3.3
Router id of the neighbor
Link Data = 18.10.0.6
IP interface address
Type = 1
This is a transit link
# TOS metrics = 0
metric = 8
Area 0
18.10.0.4/30
6.6.6.6
R6
177
Different Mask or IP Subnet
on p2p Links
3.3.3.3
R3
Area 0
19.10.0.5/30
18.10.0.6/30
6.6.6.6
R6
R3#show ip ospf neighbor
Neighbor ID
6.6.6.6
Pri State
1
FULL/ -
Dead Time Address
00:00:30
18.0.0.6
Interface
Serial0
R3#show interface serial 0
Serial0 is up, line protocol is up
Hardware is HD64570
Internet address is 19.10.0.5/24
R6#show interface serial 0
Serial0 is up, line protocol is up
Hardware is HD64570
Internet address is 18.10.0.6/30
178
Different Mask or IP Subnet
on p2p Links (Cont.)
R3#show ip ospf database router 3.3.3.3
...
Link ID = 6.6.6.6
Router id of the neighbor
3.3.3.3
Link Data = 19.10.0.5
Interface address
Type = 1
This is a point-to-point link
# TOS metrics = 0
R3
metric = 8
...
R3#show ip ospf database router 6.6.6.6
...
Link ID = 3.3.3.3
Router id of the neighbor
Link Data = 18.10.0.6
Interface address
Type = 1
This is a point-to-point link
# TOS metrics = 0
metric = 8
Area 0
6.6.6.6
R6
179
Address Flipped on Dual Links
Area 0
3.3.3.3
S0
R3
S1
6.6.6.6
18.10.0.4/30
18.10.0.20/30
R6
R3#show ip ospf neighbor
Neighbor ID
6.6.6.6
6.6.6.6
Pri State
1 FULL/ 1 FULL/ -
Dead Time Address
00:00:30
18.10.0.6
00:00:33
18.10.0.22
Interface
Serial0
Serial1
R3#show interface serial 0
Serial0 is up, line protocol is up
Hardware is HD64570
Internet address is 18.10.0.21/30
180
Forwarding Address Problem
External Route
Area 0
18.10.0.5/30
Area 1
140.10.0.0
RIP
18.10.0.1/30
R1
R4
R5 18.10.0.9/29
R7
R1#show ip ospf database external 140.10.0.0
...
Link State ID = 140.10.0.0
Advertising Router = 5.5.5.5
Network Mask = 255.255.0.0
...
Forwarding address = 18.10.0.10
R1#show ip route 18.10.0.10
Routing entry for 18.10.0.8/29
Known via "ospf 1", distance 110, metric 20, type extern 2,
forward metric 10
...
181
Forwarding Address Problem
External Route
Area 0
18.10.0.5/30
18.10.0.1/30
R1
R4
Area 1
140.10.0.0
RIP
R5 18.10.0.9/29
R7
R5:
router ospf 1
network 18.10.0.0 0.0.0.255 area 1
redistribute rip subnets
redistribute connected subnets
!
router rip
network 10.0.0.0
R4:
router ospf 1
area 1 range 18.10.0.0 255.255.255.240
182
Discontigous
Backbone
Summary
Summary
LSA
LSA
Area 0
Area 0
R1
Area 0
ABR1
Area 1
ABR2
R2
R1 and R2 are not be able to see each other
Summary LSA for Inter-area routes must not
be generated into the backbone
The solution is to create virtual link between
ABR1 and ABR2
183
Distribute-List in Blocking the
Routes
18.10.0.1/30
R4
R5 18.10.0.9/29 R7
R4#show ip route 18.10.0.9
% Subnet not in table
R4:
router ospf 1
network 18.10.0.0 0.0.0.255 area 1
distribute-list 1 in
!
access-list 1 permit 18.10.0.0 0.0.0.3
184
SPF Running Constantly
Useful commands for this problem
Show IP OSPF stat
Show IP OSPF database
Show IP OSPF database database-sum
185
SPF Running Constantly
Reasons:
LSA flaps due to:
Duplicate RID/IP address
Constant link flapping in an area
186
SPF Running Constantly
Requires enable mode
R3#sh ip ospf stat
Area 0: SPF algorithm executed 42 times
Area 1: SPF algorithm executed 38 times
SPF calculation time
Delta T Intra D-Intra Summ D-Summ Ext D-Ext Total
00:22:00 0
0
0
0
0
0
0
00:21:44 0
0
4
0
0
0
4
00:21:34 0
0
4
0
0
0
4
00:21:24 0
0
0
4
0
0
4
00:21:14 0
0
0
0
0
0
0
00:21:04 0
0
0
0
0
0
0
00:20:54 0
0
0
0
0
0
0
00:20:44 0
0
4
0
0
0
4
00:20:34 0
0
0
0
0
0
0
00:00:17 4
0
0
0
0
0
4
...
R=Router LSA; N=NetworkLSA; SN=Summary Network LSA;
ASBR LSA; X=External LSA
Reason
R, N, SN,
R, SN, X
R, SN, X
R, SN, X
R,
R, N, SN,
X
R, SN, X
X
R, N, SN, SA, X
SA=Summary
187
SPF Running Constantly
R3#deb ip ospf mon
OSPF: schedule SPF in area 1
Change in LS ID 1.1.1.1, LSA type R,
OSPF: schedule SPF: spf_time 0ms wait_interval 861421816s
OSPF: begin SPF at 0x33585480ms, process time 752ms
Spf_time 0ms, wait_interval 861421816s
OSPF: end SPF at 0x33585488ms, total elapsed time 8ms
Intra: 4ms, inter: 0ms, external: 0ms
188
SPF Running Constantly
R3#show ip ospf database
OSPF Router with ID (3.3.3.3) (Process ID 1)
Router Link States (Area 0)
Link ID
ADV Router
Age
Seq#
Checksum Link count
3.3.3.3
3.3.3.3
106
0x80000009 0xC3F1
3
...
Summary Net Link States (Area 0)
Link ID
ADV Router
Age
Seq#
18.10.0.0
7.7.7.7
3
(DNA)
18.10.0.0
8.8.8.8
1396
Checksum
0x80000008 0x3DC2
0x80000004 0x27D8
...
Router Link States (Area 1)
Link ID
ADV Router
Age
1.1.1.1
1.1.1.1
2
...
Seq#
Checksum Link count
0x80000016 0xE6CD
2
189
SPF Running Constantly
R3#show ip ospf database database-summary
OSPF Router with ID (3.3.3.3) (Process ID 1)
Area 0 database summary
LSA Type
Count
Router
124
Network
4
Summary Net
10
Summary ASBR
0
Type-7 Ext
0
Opaque Link
0
Opaque Area
0
Subtotal
138
Area 1 database summary
LSA Type
Count
Router
4
Network
1
Summary Net
10
Summary ASBR
4
...
Delete
0
0
0
0
0
0
0
0
Maxage
0
0
0
0
0
0
0
0
Delete
0
0
0
0
Maxage
0
0
0
0
190
Neighbor Flapping (Frame
Relay)
Useful commands for this problem
Debug ip ospf adj
OSPF log-adjacency-change
Show IP OSPF neighbors detail
Show interface
191
NSSA ABR Not Translating
Type 7 LSA
Only NSSA ABR with the highest RID
does the conversion
NSSA
ASBR
Area 1
NSSA
Type 7
router ospf 1
network 10.10.10.10 0.0.0.0 area 0
RID: 7.7.7.7
NSSA
ABR
Area 0
No Type 7/5 Translation
RID: 8.8.8.8
Type 7–5 Conversion
192
NSSA ABR Not Translating
Type 7 LSA
Only NSSA ABR with the highest RID
does the conversion
NSSA
ASBR
Area 1
NSSA
Type 7
router ospf 1
network 10.10.10.10 0.0.0.0 area 0
RID: 7.7.7.7
NSSA
ABR
Area 0
Type 7–5 Conversion
RID: 8.8.8.8
193
Demand Circuit Problems
OSPF (DC)
18.10.0.1/30
18.10.1.0/24
The DC is bringing up the link:
There is a change in OSPF topology
debug IP OSPF monitor is helpful in this case
Network type on DC is defined broadcast
There is a router in the network that is incapable
to understand DC bit
The DC is configured over async interface
(need to configure a dialer interface as a solution)
194
Demand Circuit Problems
OSPF (DC)
RIP
18.10.0.1/30
18.10.1.0/24
RIP—OSPF Redistribution
DC is bringing up the link (cont.)
PPP host route is also owned by RIP, when PPP
host route disappears, the database is change
Solution 1: no peer neighbor-route
Solution 2: Block /32 route getting into OSPF
with
route-map
Solution 3: Use different majornet for RIP
195
Summary
What we learned?
Overview of OSPF LSAs
Different troubleshooting commands and
what to look for in those commands while
troubleshooting?
Common issues in OSPF networks; e.g
adjacency problems, CPU hogs and SPF
problems, NSSA and DC problems etc and
how to correct those problems
196
OSPF case studies
© 2002, Cisco Systems, Inc. All rights reserved.
197
Case 1
Why Does the show ip ospf neighbor
Command Reveal Neighbors Stuck in TwoWay State?
Introduction
This case explains why the show ip
ospf neighbor command shows
neighbors stuck in a two-way state. It
also provides configuration tips.
199
How OSPF Forms Its
Neighbors
In this topology, all routers are running
Open Shortest Path First (OSPF) over
the Ethernet network
200
The sample output of the show
ip ospf neighbor command
R7# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
170.170.3.4 1 2WAY/DROTHER 00:00:34 170.170.3.4 Ethernet0
170.170.3.3 1 2WAY/DROTHER 00:00:34 170.170.3.3 Ethernet0
170.170.3.8 1 FULL/DR 00:00:32 170.170.3.8 Ethernet0
170.170.3.2 1 FULL/BDR 00:00:39 170.170.3.2 Ethernet0
R8# show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
170.170.3.4 1 FULL/DROTHER 00:00:37 170.170.3.4 Ethernet0
170.170.3.3 1 FULL/DROTHER 00:00:37 170.170.3.3 Ethernet0
170.170.3.7 1 FULL/DROTHER 00:00:38 170.170.3.7 Ethernet0
170.170.3.2 1 FULL/BDR 00:00:32 170.170.3.2 Ethernet0
201
• Whenever a router sees itself in a neighbor hello packet,
it confirms bidirectional communication and transitions
the neighbor state to two-way. At this point, the routers
perform DR and BDR election. Once DR and BDR are
elected, a router attempts to form a full adjacency with a
neighbor if one of the two routers is the DR or BDR.
OSPF routers become fully adjacent with routers with
which they have successfully completed the database
synchronization process. This is the process by which
OSPF routers exchange link-state information to
populate their databases with the same information.
Again, this database synchronization process is only
executed between two routers if one of the two routers is
the DR or BDR.
202
Why Do Routers Only Form
Full Adjacencies with the DR
or BDR?
203
Why Do Routers Only Form
Full Adjacencies with the DR
or BDR? (cont.)
Sometimes it is desirable for a router to be configured so that it
is not eligible to become the DR or BDR. You can do this by setting
the OSPF priority to zero with the ip ospf priority priority#
interface subcommand. If two OSPF neighbors both have their OSPF
interface priority set to zero, they establish two-way adjacency
instead of full adjacency.
204
example.(topology)
205
example.
The topology below provides an example.
There are three routers connected via Frame
Relay. The Frame Relay interfaces are defined
as broadcast, but only the router with a
connection back to the main network is
eligible to be the DR. The other two routers
have their interface priorities set to zero, so
they are not eligible to become the DR or
BDR. Although they do become neighbors,
they only reach two-way state.
206
example.
The neighbor table for this topology looks
like this:
DRother1# show ip ospf neighbor
Neighbor
ID Pri State
Dead Time
170.170.9.5 1 FULL/DR
00:00:30
170.170.10.8 0 2WAY/DROTHER 00:00:38
DRother1#
Address
Interface
170.170.9.5 Serial0.5
170.170.9.8 Serial0.5
Notice that, in the figure above, the DRother1
router establishes a two-way adjacency with
the DRother2 router.
207
Case 2
OSPF routers do not form neighbor
relationships due to authentication type
mismatch
OSPF Errors, Warnings,
and Log Messages
Receiving "Mismatch Authentication
type."
209
Core Issue
Before exchanging routing information, routers
running Open Shortest Path First (OSPF) form
neighbor relationships with other OSPF routers on the
same segment. This is done by exchanging hello
packets. The hello packets contain various
parameters, one of which is related to authentication.
This specifies the authentication type and
authentication information for the originating
interface. Authentication is useful for preventing
malicious or incorrect routing information from
getting introduced into the routing table. OSPF
supports two types of authentication: plain text and
Message Digest 5 (MD5).
210
To resolve this issue, perform
these steps:
1. Identify the adjacency state with the
neighbor by issuing the show ip ospf
neighbor command from privileged
EXEC mode.
2. Find the authentication type
configured under an interface by issuing
the show ip ospf interface command
from privileged EXEC mode.
211
To resolve this issue, perform
these steps(cont.):
If the authentication type does not match between
two routers on the same segment, you will see a
message similar to this:
OSPF: Rcv pkt from x.x.x.x, Ethernet1/0 : Mismatch
Authentication type. Input packet specified type 2,
we use type 1
This occurs when you issue the debug ip ospf adj
command from privileged EXEC mode.
This message indicates this information:
Type 0 indicates that no authentication is enabled.
Type 1 is for plain text authentication to be enabled.
Type 2 means that MD5 authentication is enabled.
212
To resolve this issue, perform
these steps:
3. Make sure that all the routers connected to the
same segment use the same authentication type.
This is done by issuing the area authentication
command in router configuration mode or the ip
ospf authentication command in interface
configuration mode.
The area authentication command configures all
the interfaces under a particular area to use the
specified authentication type.
The ip ospf authentication command can be used
to individually configure the authentication type for
an interface or override the type configured under an
interface with the area authentication command.
213
Trouble shooting IS-IS
214
© 2002, Cisco Systems, Inc. All rights reserved.
214
Troubleshooting Agenda
Adjacencies
LSP Flooding and Contents
SPF Computation
Monitoring Performance
Advanced/New Features
215
Basic Configuration
Router-A
-------------interface Loopback0
ip address 192.168.1.5
255.255.255.255
!
interface Serial0
ip address 192.168.120.5
255.255.255.0
ip router isis
!
router isis
passive-interface Loopback0
net 49.0001.1921.6800.1005.00
is-type level-1
Router-B
L1 Router
-------------interface Loopback0
ip address 192.168.1.1
255.255.255.255
!
Interface Serial0
ip address 192.168.120.10
255.255.255.0
ip router isis
!
interface Serial1
ip address 192.168.222.1
255.255.255.0
ip router isis
!
L1 Router
router isis
passive-interface Loopback0
net 49.0001.1921.6800.1001.00
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
216
Basic Configuration
Router-C
-------------interface Loopback0
ip address 192.168.2.2
255.255.255.255
!
interface Serial0
ip address 192.168.111.2
255.255.255.0
ip router isis
isis circuit-type level-1
!
interface Serial1
ip address 192.168.222.2
255.255.255.0
ip router isis
isis circuit-type level-2
!
router isis
passive-interface Loopback0
net 49.0002.1921.6800.2002.00
Router-D
L1 Router
-------------interface Loopback0
ip address 192.168.2.4
255.255.255.255
!
interface Serial1
ip address 192.168.111.4
255.255.255.0
ip router isis
!
router isis
passive-interface Loopback0
net 49.0002.1921.6800.2004.00
is-type level-1
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
217
ADJACENCIES
RST-3302
9803_05_2004_c2
© 2004 Cisco Systems, Inc. All rights reserved.
218
Check that CLNS Is Running
OK?
General CLNS information at a glance:
Lists
Lists
Lists
Lists
number of CLNS-enabled interfaces
NET configured on router
the mode that is running (IP or OSI)
IS-type
show clns
show clns protocol
219
show clns
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
Rtr-B#show clns
Global CLNS Information:
2 Interfaces Enabled for CLNS
NET: 49.0001.1921.6800.1001.00
Configuration Timer: 60, Default Holding Timer: 300, Packet Lifetime 64
ERPDU's requested on locally generated packets
Running IS-IS in IP-only mode (CLNS forwarding not allowed)
220
show clns protocol
L1 Router
Rtr-A
S0
Rtr-B#show clns protocol
IS-IS Router: <Null Tag>
System Id: 1921.6800.1001.00 IS-Type: level-1-2
Manual area address(es):
49.0001
Routing for area address(es):
49.0001
Interfaces supported by IS-IS:
Serial1 - IP
Serial0 - IP
Redistribute:
static (on by default)
Distance for L2 CLNS routes: 110
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
221
Are the Hellos Sent?
Interface up/line protocol up
Are bits exchanged: sh cdp neigh detail
Show clns neighbor
Debug isis adj-packet
222
Check Interface Status
L1L2 Routers
S0
S1 Rtr-B
S1
Rtr-C S0
Area 49.0001
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
Rtr-B#show ip int brief
Interface
IP-Address
Loopback0
192.168.1.1
OK?
YES
Method
NVRAM
Status
up
Protocol
up
Serial0
192.168.222.1
YES
manual
up
up
Serial1
192.168.120.10 YES
manual
up
up
223
Check CDP Neighbor
Connectivity
L1L2 Routers
S0
S1 Rtr-B
S1
Rtr-C S0
Area 49.0001
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
Rtr-B#show cdp neighbor
Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge
S - Switch, H - Host, I - IGMP, r - Repeater
Device ID
Rtr-C
Rtr-A
Local Intrfce
Ser 0
Ser 1
Holdtme
125
148
Capability Platform Port ID
R
2500
Ser 1
R
2500
Ser 0
224
Are the Adjacencies Up?
Entry in “show clns neighbor”?
Protocol is IS-IS (not ES-IS)?
State is up (should not show init)?
IP addresses on both sides match?
MTU size OK?
225
Check CLNS Neighbors
L1L2 Routers
S0
S1 Rtr-B
S1
Rtr-C S0
Area 49.0001
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
Rtr-B#show clns neighbors
System Id
Interface SNPA
Rtr-C
Se0
*HDLC*
1921.6800.1005 Se1
*HDLC*
State Holdtime Type Protocol
Up 23
L2
IS-IS
Up 21
L1
IS-IS
226
Check CLNS Neighbor Detail
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
S1
Rtr-A
Rtr-B#sh clns nei det
System Id
Interface
SNPA
Rtr-C
Se0
*HDLC*
Area Address(es): 49.0002
IP Address(es): 192.168.222.2*
Uptime: 03:09:19
1921.6800.1005 Se1
*HDLC*
Area Address(es): 49.0001
IP Address(es): 192.168.120.5*
Uptime: 03:50:01
Rtr-D
L1 Routers
State Holdtime Type
Up
23
L2
Protocol
IS-IS
Up
IS-IS
27
L1
227
If Protocol Field Shows ES-IS?
Common Causes:
Misconfigured IP interface subnet
Cisco IOS® validates source IP address
of neighbor before bringing up
adjacency
No “ip router isis” command on
interface
Rtr-B#show clns neighbors
System Id
Interface SNPA
Rtr-C
Se0
*HDLC*
1921.6800.1005 Se1
*HDLC*
State Holdtime Type Protocol
Up 23
L2
IS-IS
Up 21
L1
ES-IS
228
Mixing IS-IS IP and IS-IS OSI
Can cause protocol field to show ES-IS!
Mode is per router, not per interface
All routers in an area must agree
Using passive-interface forces IP mode
IP-only, CLNS-only or integrated (dual)
Can only mix areas, not inside areas
L2 routing not guaranteed to work
229
show clns interface
L1 Router
Rtr-B#show clns int serial1
Serial1 is up, line protocol is up
Checksums enabled, MTU 1500, Encapsulation HDLC
ERPDUs enabled, min. interval 10 msec.
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 47 seconds
Routing Protocol: IS-IS
Circuit Type: level-1-2
Interface number 0x2, local circuit ID 0x101
Level-1 Metric: 10, Priority: 64, Circuit ID: Rtr-A.00
Number of active level-1 adjacencies: 1
Level-2 Metric: 10, Priority: 64, Circuit ID: Rtr-B.01
L1 Router
Number of active level-2 adjacencies: 0
Next IS-IS Hello in 6 seconds
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
230
show clns interface
L1 Router
Rtr-B#show clns int serial0
Serial0 is up, line protocol is up
Checksums enabled, MTU 1500, Encapsulation HDLC
ERPDUs enabled, min. interval 10 msec.
CLNS fast switching enabled
CLNS SSE switching disabled
DEC compatibility mode OFF for this interface
Next ESH/ISH in 30 seconds
Routing Protocol: IS-IS
Circuit Type: level-1-2
Interface number 0x1, local circuit ID 0x100
Level-1 Metric: 10, Priority: 64, Circuit ID: Rtr-C.01
Number of active level-1 adjacencies: 0
Level-2 Metric: 10, Priority: 64, Circuit ID: Rtr-B.00
Number of active level-2 adjacencies: 1
L1 Router
Next IS-IS Hello in 6 seconds
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
231
debug isis adj-packets
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S0
Rtr-A
S1
Rtr-C S0
Area 49.0002
S1
Rtr-D
L1 Routers
Rtr-B#debug isis adj-packets
IS-IS Adjacency related packets debugging is on
Rtr-B#
05:45:21: ISIS-Adj: rcvd state UP, old state UP, new state UP
05:45:21: ISIS-Adj: Action = ACCEPT
05:45:24: ISIS-Adj: Sending serial IIH on Serial0, length 1499
05:45:26: ISIS-Adj: Rec serial IIH from *HDLC* (Serial1), cir type L1, cir id 00, length 1499
05:45:26: ISIS-Adj: rcvd state UP, old state UP, new state UP
05:45:26: ISIS-Adj: Action = ACCEPT
05:45:26: ISIS-Adj: Sending serial IIH on Serial1, length 1499
05:45:31: ISIS-Adj: Rec serial IIH from *HDLC* (Serial0), cir type L1L2, cir id 01, length 1499
232
Are the LSPs Flooded?
show isis database
Compare sequence numbers
Compare checksum or content
debug isis update-packets
debug isis snp-packets
show clns traffic
233
LSP FLOODING AND
CONTENTS
show isis database
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
S1
Rtr-A
Rtr-B#show isis database
IS-IS Level-1 Link State Database:
LSPID
LSP Seq Num
Rtr-B.00-00
* 0x00000020
1921.6800.1005.00-00 0x00000023
1921.6800.1005.01-00 0x00000017
IS-IS Level-2 Link State Database:
LSPID
LSP Seq Num
Rtr-B.00-00
* 0x00000024
Rtr-C.00-00
0x00000028
Rtr-D
L1 Routers
LSP Checksum
0x0C24
0x909E
0xC896
LSP Holdtime
674
830
841
ATT/P/OL
1/0/0
0/0/0
0/0/0
LSP Checksum LSP Holdtime
0x7D98
748
0x1E01
1128
ATT/P/OL
0/0/0
0/0/0
235
debug isis update-packets
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
Rtr-B#debug isis update-packets
IS-IS Update related packet debugging is on
Rtr-A
S1
Rtr-D
L1 Routers
Rtr-B#
05:45:21: ISIS-Upd: Rec L2 LSP 1921.6800.2002.00-00, seq 24E, ht 1199,
05:45:21 : ISIS-Upd: from SNPA *HDLC* (Serial0)
05:45:21 : ISIS-Upd: LSP newer than database copy
05:45:21 : ISIS-Upd: No change
05:45:21 : ISIS-Upd: Refreshing L2 1921.6800.1001.00-00
05:45:21 : ISIS-Upd: Sending L2 LSP 1921.6800.1001.00-00, seq 25E, ht 1199 on Serial0
05:45:21 : ISIS-Upd: Rec L2 LSP 1921.6800.2002.00-00, seq 24F, ht 1199,
05:45:21 : ISIS-Upd: from SNPA *HDLC* (Serial0)
05:45:21 : ISIS-Upd: LSP newer than database copy
05:45:21 : ISIS-Upd: No change
05:45:21 : ISIS-Upd: Refreshing L2 1921.6800.1001.00-00
05:45:21 : ISIS-Upd: Sending L2 LSP 1921.6800.1001.00-00, seq 25F, ht 1199 on Serial0
236
debug isis snp-packets
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
Rtr-B#debug isis snp-packets
Rtr-A
IS-IS CSNP/PSNP packets debugging is on
L1 Routers
Rtr-B#
07:51:59: ISIS-Snp: Build L2 PSNP entry for 1921.6800.2002.00-00, seq 35
07:51:59: ISIS-Snp: Sending L2 PSNP on Serial0
07:53:50: ISIS-Snp: Rec L1 PSNP from 1921.6800.1005 (Serial1)
07:53:50: ISIS-Snp: PSNP entry 1921.6800.1001.00-00, seq 31, ht 1197
07:53:50: ISIS-Snp: Same entry 1921.6800.1001.00-00, seq 31
07:54:26: ISIS-Snp: Build L1 PSNP entry for 1921.6800.1005.00-00, seq 2F
07:54:26: ISIS-Snp: Sending L1 PSNP on Serial1
07:55:18: ISIS-Snp: Rec L2 PSNP from 1921.6800.2002 (Serial0)
07:55:18: ISIS-Snp: PSNP entry 1921.6800.1001.00-00, seq 32, ht 1197
07:55:18: ISIS-Snp: Same entry 1921.6800.1001.00-00, seq 32
S1
Rtr-D
237
Does the LSP Contain ALL
Info?
LSPs are identified by:
LSP identifier (8 bytes—sysID, n-sel, frag.)
Sequence number (higher means newer LSP)
Remaining lifetime (expiration purges LSP)
Checksum (if corrupt discard—sender r-tx’s)
show isis database detail!
Correct IP prefixes and metrics present?
debug isis local-updates
238
show isis database detail
L1 Router
Rtr-B#sh isis dat det 1921.6800.1001.00-00
IS-IS Level-1 LSP Rtr-B.00-00
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-B.00-00
* 0x00000001 0x20D5
1015
Area Address: 49.0001
NLPID:
0xCC
Hostname: Rtr-B
IP Address: 192.168.1.1
Metric: 0
IP 192.168.1.1/32
Metric: 10
IP 192.168.222.0/24
Metric: 10
IP 192.168.120.0/24
Metric: 10000 IS-Extended 1921.6800.1005.00
IS-IS Level-2 LSP Rtr-B.00-00
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-B.00-00
* 0x00000263 0xAD32
1184
Area Address: 49.0001
NLPID:
0xCC
Hostname: Rtr-B
IP Address: 192.168.1.1
Metric: 10000 IS-Extended Rtr-C.00
Metric: 10
IP 192.168.120.0/24
Metric: 0
IP 192.168.1.1/32
Metric: 20
IP 192.168.1.5/32
Metric: 10
IP 192.168.222.0/24
Rtr-A
S0
ATT/P/OL
1/0/0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
ATT/P/OL
0/0/0
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
239
debug isis local-updates
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S0
S1
Rtr-C S0
Area 49.0002
S1
tr-B#sh debug
CLNS:
Rtr-A
Rtr-D
IS-IS local updates debugging is on
Rtr-B#conf t
L1 Routers
Enter configuration commands, one per line. End with CNTL/Z.
Rtr-B(config)#int serial 1
Rtr-B(config-if)#shut
Rtr-B(config-if)#
07:59:26: ISIS-Loc: IP route adjust (level-1-2/Serial1/192.168.120.10)
07:59:26 : ISIS-Upd: Building L1 LSP
07:59:26 : ISIS-Upd: Building L2 LSP
07:59:26 : ISIS-Upd: Building L2 LSP
07:59:26 : ISIS-Loc: IP route adjust (level-1-2/Serial1/192.168.120.10)
07:59:26 : ISIS-Upd: Building L1 LSP
07:59:26 : ISIS-Upd: Building L2 LSP
07:59:26 : ISIS-Loc: L2 non-summarized metric change (4294967295->20), 192.168.1.5
07:59:26 : ISIS-Upd: Building L2 LSP
240
show isis lsp-log
L1 Router
Rtr-B#show isis lsp-log
Level 1 LSP log
When
Count
Interface
01:50:44
1
01:50:35
1
Loopback0
01:50:28
1
Serial0
01:50:20
1
Serial1
01:50:20
1
Serial1
01:50:18
1
01:36:49
1
Loopback0
Level 2 LSP log
When
Count
01:50:46
1
01:50:36
1
01:50:30
2
01:50:22
1
01:50:10
1
01:48:21
1
01:48:16
1
01:36:51
1
Interface
Loopback0
Serial0
Serial1
Serial0
Serial0
Loopback0
Rtr-A
S0
Triggers
CONFIG
IPUP
IPUP
IPUP
NEWADJ
ATTACHFLAG
CONFIG
Triggers
CONFIG
IPUP
NEWADJ IPUP
IPUP
IPIA
DELADJ
NEWADJ
CONFIG
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
241
LSP Checksum
Detection of LSP Corruption
During Flooding
Depending on Layer 2 CRC is not
enough, corruption happens in routers
and switches
Compute checksum of received LSP and
check against checksum inside LSP
If corrupt, silently discard LSP
Sender will always retransmit the LSP
242
Does SPF Calculate Correct
Routes?
show isis topology
debug isis spf-events
debug isis spf-statistics
Also:
show ip route isis
debug ip routing
debug clns routing
243
SPF COMPUTATION
show isis topology
L1L2 Routers
S0
S1 Rtr-B
S1
Rtr-C S0
Area 49.0001
Area 49.0002
S0
Rtr-B#show isis topology
S1
Rtr-A
Rtr-D
L1 Routers
IS-IS paths to level-1 routers
System Id
Metric Next-Hop
Interface
Rtr-B
-1921.6800.1005 10
1921.6800.1005 Se1
IS-IS paths to level-2 routers
System Id
Metric Next-Hop
Rtr-B
-Rtr-C
10
Rtr-C
SNPA
*HDLC*
Interface
SNPA
Se0
*HDLC*
245
debug isis spf-events
L1 Router
Rtr-A
S0
Rtr-B#debug isis spf-events
IS-IS SPF events debugging is on
Rtr-B#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Rtr-B(config)#int serial 0
Rtr-B(config-if)#shut
Rtr-B(config-if)#
07:46:22: ISIS-Spf: L2 LSP 1 (1921.6800.2002.00-00) flagged for recalculation
from 35F69EE
07:46:22: ISIS-Spf: L1 LSP 2 (1921.6800.1001.00-00) flagged for recalculation from
360CF3E
07:46:22: %LINK-5-CHANGED: Interface Serial0, changed state to administratively down
07:46:22: ISIS-Spf: Calculating routes for L2 LSP 1 (1921.6800.2002.00-00)
07:46:22: ISIS-Spf: Add 192.168.111.0/255.255.255.0 to IP RIB, metric 20
07:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)
07:46:22: ISIS-Spf: Add 192.168.2.2/255.255.255.255 to IP RIB, metric 10
07:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)
07:46:22: ISIS-Spf: Add 192.168.2.4/255.255.255.255 to IP RIB, metric 20
07:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)
07:46:22: ISIS-Spf: Add 192.168.222.0/255.255.255.0 to IP RIB, metric 20
07:46:22: ISIS-Spf: Next hop 1921.6800.2002/192.168.222.2 (Serial0) (accepted)
L1 Router
07:46:22: ISIS-Spf: Aging L2 LSP 1 (1921.6800.2002.00-00), version 516
07:46:22: ISIS-Spf: Calculating routes for L1 LSP 2 (1921.6800.1001.00-00)
07:46:22: ISIS-Spf: Aging L1 LSP 2 (1921.6800.1001.00-00), version 144
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
246
debug isis spf-statistics
L1 Router
Rtr-A
S0
Rtr-B#debug isis spf-statistics
IS-IS SPF Timing and Statistics Data debugging is on
Rtr-B#conf t
Enter configuration commands, one per line. End with CNTL/Z.
Rtr-B(config)#int serial 0
Rtr-B(config-if)#shut
Rtr-B(config-if)#no shut
Rtr-B(config-if)#
07:49:37: %LINK-3-UPDOWN: Interface Serial0, changed state to up
07:49:37: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-C (Serial0) Up, new adjacency
07:49:37: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0, changed state to
up
07:49:37: ISIS-Spf: Compute L2 SPT
07:49:37: ISIS-Spf: Complete L2 SPT,
07:49:37: ISIS-Spf: Compute time 0.008/0.008, 2/0 nodes, 1/0 links, 0 suspends
Rtr-B(config-if)#
07:49:37: ISIS-Spf: Compute L1 SPT
07:49:37: ISIS-Spf: Complete L1 SPT,
07:49:37: ISIS-Spf: Compute time 0.004/0.004, 2/0 nodes, 1/0 links, 0 suspends
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
247
Is the Change Noticed?
show isis spf-log
debug isis spf-trigger
show isis lsp-log on generating router
show clns traffic to watch for PRCs
248
show isis spf-log
L1 Router
Rtr-B#show isis spf-log
Level 1 SPF log
When Duration Nodes Count
02:16:52
0
1
1
02:16:42
0
1
1
02:16:32
0
1
2
02:16:22
8
3
4
02:02:57
02:01:52
01:46:52
01:31:53
01:16:52
01:01:52
00:46:52
00:31:51
00:16:51
00:01:50
4
8
8
8
8
8
8
8
8
64
3
3
3
3
3
3
3
3
3
3
Rtr-A
S0
1
1
1
1
1
1
1
1
1
1
Area 49.0001
First trigger LSP
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Triggers
NEWLSP
TLVCODE
NEWADJ TLVCONTENT
ATTACHFLAG LSPHEADER
TLVCON TENT
TLVCONTENT
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
L1 Router
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
249
show isis spf-log
L1 Router
S0
Level 2 SPF log
When Duration Nodes Count
02:16:54
0
1
1
02:16:44
0
1
1
02:16:34
8
2
3
02:14:29
8
2
3
02:14:23
4
2
1
02:13:56
8
2
1
02:02:59
4
2
1
02:01:54
4
2
1
01:46:54
4
2
1
01:31:54
4
2
1
01:16:54
4
2
1
01:01:54
4
2
1
00:46:53
4
2
1
00:31:53
4
2
1
00:16:53
4
2
1
00:01:53
60
2
1
Rtr-A
Area 49.0001
First trigger LSP
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Rtr-C.00-00
Rtr-C.00-00
Rtr-B.00-00
S1
Triggers
NEWLSP
Rtr-B
TLVCODE
S0
NEWADJ NEWLSP TLVCONTENT
L1L2 Routers
NEWADJ TLVCONTENT
TLVCODE
S1
TLVCONTENT
TLVCONTENT
PERIODIC
Rtr-C
S0
PERIODIC
PERIODIC
Area 49.0002
PERIODIC
PERIODIC
S1
PERIODIC
PERIODIC
Rtr-D
PERIODIC
L1 Router
PERIODIC
250
debug isis spf-triggers
L1 Router
Rtr-A
S0
Rtr-B#debug isis spf-triggers
IS-IS SPF triggering events debugging is on
Rtr-B#
07:32:10: ISIS-Spf: L1 SPF needed, periodic SPF, from 0x356C8DC
07:32:10: ISIS-Spf: L2 SPF needed, periodic SPF, from 0x356C8DC
Rtr-B#conf t
Rtr-B(config)#int serial0
Rtr-B(config-if)#isis metric 15
Rtr-B(config-if)# ^Z
07:38:27: ISIS-Spf: L1 SPF needed, new metric, from 0x3560762
Rtr-B(config)#int serial0
Rtr-B(config-if)#shut
Rtr-B(config-if)# ^Z
07:39:23: ISIS-Spf: L2, 1921.6800.1001.00-00 TLV contents changed, code 0x2
07:39:28: ISIS-Spf: L1 SPF needed, L2 attach changed, from 0x357CF36
07:39:28: ISIS-Spf: L1, LSP fields changed 1921.6800.1001.00-00
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
251
Monitoring Performance
show proc cpu
IS-IS adjacency process(es)
Send and receives hellos
Manages adjacency database
DIS election
Should be low CPU usage (< 1%)
252
MONITORING PERFORMANCE
Monitoring Performance
IS-IS update process(es)
SPF computation and flooding
CPU usage should fluctuate
Don’t worry unless constant > 20%
Distributed (CEF) switching should help
254
show proc cpu
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
Rtr-A
L1 Routers
Rtr-B# sh proc cpu
CPU utilization for five seconds: 0%/0%; one minute: 0%; five minutes: 0%
PID Runtime(ms) Invoked uSecs 5Sec
1Min 5Min TTY Process
1
15760 89776
175 0.00% 0.00% 0.00%
0 Load Meter
2
108116
6965 15522 0.08% 0.00% 0.00%
0 Exec
3
1205472 83405 14453 0.00% 0.25% 0.25%
0 Check heaps
.
.
.
47
70580 121119
582 0.00% 0.00% 0.00%
0 CLNS Input
48
14184
79119
179 0.00% 0.00% 0.00%
0 ES-IS Routing
49
205748 206607
995 0.00% 0.00% 0.00%
0 ISIS Adj
50
34596
52216
662 0.00% 0.00% 0.00%
0 ISIS Upd
S1
Rtr-D
255
show clns traffic
LSPs sourced indicates stability of IS
LSP retransmissions should stay low
PRCs can not be checked elsewhere
LSP checksum errors are a bad sign
Update queue should not stay full
Update queue should not drop much
256
show clns traffic
L1 Router
Rtr-B#show clns traffic
CLNS: Time since last clear: never
CLNS & ESIS Output: 669, Input: 4773
CLNS Local: 0, Forward: 0
CLNS Discards:
Hdr Syntax: 0, Checksum: 0, Lifetime: 0, Output cngstn: 0
No Route: 0, Discard Route: 0, Dst Unreachable 0, Encaps. Failed: 0
NLP Unknown: 0, Not an IS: 0
CLNS Options: Packets 0, total 0 , bad 0, GQOS 0, cngstn exprncd 0
CLNS Segments: Segmented: 0, Failed: 0
CLNS Broadcasts: sent: 0, rcvd: 0
Echos: Rcvd 0 requests, 0 replies
Sent 0 requests, 0 replies
ESIS(sent/rcvd): ESHs: 0/0, ISHs: 669/660, RDs: 0/0, QCF: 0/0
ISO-IGRP: Querys (sent/rcvd): 0/0 Updates (sent/rcvd): 0/0
ISO-IGRP: Router Hellos: (sent/rcvd): 0/0
ISO-IGRP Syntax Errors: 0
IS-IS: Time since last clear: never
IS-IS: Level-1 Hellos (sent/rcvd): 282/0
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
257
show clns traffic
L1 Router
IS-IS: Level-2 Hellos (sent/rcvd): 285/0
IS-IS: PTP Hellos (sent/rcvd): 420/415
IS-IS: Level-1 LSPs sourced (new/refresh): 8/2
IS-IS: Level-2 LSPs sourced (new/refresh): 9/1
IS-IS: Level-1 LSPs flooded (sent/rcvd): 5/8
IS-IS: Level-2 LSPs flooded (sent/rcvd): 7/8
IS-IS: LSP Retransmissions: 0
IS-IS: Level-1 CSNPs (sent/rcvd): 1/1
IS-IS: Level-2 CSNPs (sent/rcvd): 2/2
IS-IS: Level-1 PSNPs (sent/rcvd): 7/4
IS-IS: Level-2 PSNPs (sent/rcvd): 7/5
IS-IS: Level-1 DR Elections: 1
IS-IS: Level-2 DR Elections: 1
IS-IS: Level-1 SPF Calculations: 7
IS-IS: Level-2 SPF Calculations: 9
IS-IS: Level-1 Partial Route Calculations: 1
IS-IS: Level-2 Partial Route Calculations: 5
IS-IS: LSP checksum errors received: 0
IS-IS: Update process queue depth: 0/200
IS-IS: Update process packets dropped: 0
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
258
show isis spf-log
How often are SPFs run?
Doesn’t really matter how often
Frequent SPFs can indicate a problem
In a stable network—only periodic SPF
runs
Who triggered the SPF?!
Check the LSPID of first trigger LSP
Helps to find the source of the problem
259
show isis spf-log
L1 Router
Rtr-A
S0
Rtr-B#show isis spf-log
Area 49.0001
Level 1 SPF log
When Duration Nodes Count
02:16:52
0
1
1
02:16:42
0
1
1
02:16:32
0
1
2
02:16:22
8
3
4
02:02:57
02:01:52
01:46:52
01:31:53
01:16:52
01:01:52
00:46:52
00:31:51
00:16:51
00:01:50
4
8
8
8
8
8
8
8
8
64
3
3
3
3
3
3
3
3
3
3
1
1
1
1
1
1
1
1
1
1
First trigger LSP
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Rtr-B.00-00
Triggers
NEWLSP
TLVCODE
NEWADJ TLVCONTENT
ATTACHFLAG LSPHEADER
TLVCON TENT
TLVCONTENT
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
L1 Router
PERIODIC
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
260
show isis spf-log
How Much CPU Time Is Used per SPF?
Normally in the region of msecs
400 node network takes around 50msecs
If closer to seconds then maybe an issue
(Could mean LSP re-transmissions over
p2p links)
If so, can change lsp-retransmit-timer
261
show isis spf-log
L1 Router
Rtr-A
S0
Level 2 SPF log
When Duration Nodes Count
02:16:54
0
1
1
02:16:44
0
1
1
02:16:34
8
2
3
02:14:29
8
2
3
02:14:23
4
2
1
02:13:56
8
2
1
02:02:59
4
2
1
02:01:54
4
2
1
01:46:54
4
2
1
01:31:54
4
2
1
01:16:54
4
2
1
01:01:54
4
2
1
00:46:53
4
2
1
00:31:53
4
2
1
00:16:53
4
2
1
00:01:53
60
2
1
Area 49.0001
First trigger LSP Triggers
Rtr-B.00-00
NEWLSP
Rtr-B.00-00
TLVCODE
Rtr-B.00-00
NEWADJ NEWLSP TLVCONTENT
Rtr-B.00-00
NEWADJ TLVCONTENT
Rtr-C.00-00
TLVCODE
Rtr-C.00-00
TLVCONTENT
Rtr-B.00-00
TLVCONTENT
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
PERIODIC
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
262
show isis lsp-log
How often do we generate a new LSP?
One router can influence whole net
Why did it generate a new LSP?
Flapping adjacency is shown by
interface
show clns neighbour detail
Look at the “uptime value”
263
show isis lsp-log
L1 Router
Rtr-B#show isis lsp-log
Level 1 LSP log
When
Count
01:50:44
1
01:50:35
1
01:50:28
1
01:50:20
1
01:50:20
1
01:50:18
1
01:36:49
1
Level 2 LSP log
When
Count
01:50:46
1
01:50:36
1
01:50:30
2
01:50:22
1
01:50:10
1
01:48:21
1
01:48:16
1
01:36:51
1
Rtr-A
S0
Area 49.0001
Interface
Loopback0
Serial0
Serial1
Serial1
Loopback0
Interface
Loopback0
Serial0
Serial1
Serial0
Serial0
Loopback0
Triggers
CONFIG
IPUP
IPUP
IPUP
NEWADJ
ATTACHFLAG
CONFIG
Triggers
CONFIG
IPUP
NEWADJ IPUP
IPUP
IPIA
DELADJ
NEWADJ
CONFIG
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
264
Traffic Engineering
Support for MPLS-TE:
Traffic Engineering with Multi Protocol
Label Switching
IS-IS allows MPLS-TE to flood resource,
policy and reservation information
about links inside LSPs
New information carried in sub-TLVs
IS-IS Metrics were extended for TE
265
ADVANCED/NEW FEATURES
Traffic Engineering: IS-IS
Metrics
The interface metric was increased from
6 bits wide to 24 bits wide
The total path metric was increased
from 10 bits wide to ~32 bits wide
Can configure the old or new metrics
Default is old style metrics
However—must configure wide metrics
to use TE
267
Traffic Engineering: IS-IS
Metric Styles
Current Metric Styles That May Be Present:
Narrow
Use old style of TLVs with narrow metric
Wide
Use new style of TLVs to carry wider metric
Transition
Used when migrating from old to new format
Router will accept both old and new style metrics
Caveat: Should only be used in transitioning
268
Traffic Engineering Support
L1 Router
sh isis database detail
Rtr-C#sh isis dat det
IS-IS Level-1 Link State Database:
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-C.00-00
* 0x00000080 0x2237
958
Area Address: 49.0002
NLPID:
0xCC
Hostname: Rtr-C
IP Address: 192.168.2.2
Metric: 0
IP 192.168.2.2/32
Metric: 10
IP 192.168.111.0/24
Metric: 10
IP 192.168.222.0/24
Metric: 10000 IS-Extended Rtr-D.00
Rtr-D.00-00
0x00000076 0xB426
1137
Area Address: 49.0002
NLPID:
0xCC
Hostname: Rtr-D
IP Address: 192.168.2.4
Metric: 0
IP 192.168.2.4/32
Metric: 10
IP 192.168.111.0/24
Metric: 10000 IS-Extended Rtr-C.00
Rtr-A
S0
Area 49.0001
S1
ATT/P/OL
1/0/0
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
0/0/0
S1
Rtr-D
L1 Router
269
Traffic Engineering Support
L1 Router
sh isis database detail
IS-IS Level-2 Link State Database:
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-B.00-00
0x0000007D 0x0BAD
670
Area Address: 49.0001
NLPID:
0xCC
Hostname: Rtr-B
IP Address: 192.168.1.1
Metric: 6250000 IS-Extended Rtr-C.00
Metric: 10
IP 192.168.120.0/24
Metric: 0
IP 192.168.1.1/32
Metric: 10
IP 192.168.1.5/32
Metric: 10
IP 192.168.222.0/24
Rtr-C.00-00
* 0x00000080 0xA130
1141
Area Address: 49.0002
NLPID:
0xCC
Hostname: Rtr-C
IP Address: 192.168.2.2
Metric: 10000
IS-Extended Rtr-B.00
Metric: 10
IP 192.168.111.0/24
Metric: 0
IP 192.168.2.2/32
Metric: 10
IP 192.168.2.4/32
Metric: 10
IP 192.168.222.0/24
Rtr-A
S0
ATT/P/OL
0/0/0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
0/0/0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
270
Traffic Engineering Support
sh isis database verbose (for TE BW info)
hr3.iad.00-00
0x00001FE3 0x5FEE
34751
0/0/0
Area Address: 39.840f.8011.3824.0000.9000.0820
NLPID:
0xCC
Hostname: hr3.iad
Router ID: 204.152.166.8
IP Address: 204.152.166.8
Metric: 5
IS-Extended hr3.iad.03
Affinity: 0x00000000
Interface IP Address: 209.143.220.163
Physical BW: 100000000 bits/sec
Reservable BW: 100000000 bits/sec
BW Unreserved[0]: 100000000 bits/sec, BW Unreserved[1]: 100000000 bits/sec
BW Unreserved[2]: 100000000 bits/sec, BW Unreserved[3]: 100000000 bits/sec
BW Unreserved[4]: 100000000 bits/sec, BW Unreserved[5]: 100000000 bits/sec
BW Unreserved[6]: 100000000 bits/sec, BW Unreserved[7]: 100000000 bits/sec
Metric: 4
IS-Extended cr2.iad3.00
Affinity: 0x00000000
Interface IP Address: 206.132.253.62
Neighbor IP Address: 206.132.253.61
Physical BW: 155000000 bits/sec
Reservable BW: 155000000 bits/sec
BW Unreserved[0]: 155000000 bits/sec, BW Unreserved[1]: 155000000 bits/sec
BW Unreserved[2]: 155000000 bits/sec, BW Unreserved[3]: 155000000 bits/sec
BW Unreserved[4]: 155000000 bits/sec, BW Unreserved[5]: 155000000 bits/sec
BW Unreserved[6]: 155000000 bits/sec, BW Unreserved[7]: 155000000 bits/sec
271
Traffic Engineering Support
Also to View Bandwidth Information Use:
sh isis mpls traffic-eng advertisements
System ID: wr1.sfo1.00
Router ID: 206.132.110.69
Link Count: 9
Link[1]
Neighbor System ID: wr2.sfo1.00 (P2P link)
Interface IP address: 206.132.110.73
Neighbor IP Address: 206.132.110.74
Admin. Weight: 1
Physical BW: 2488000000 bits/sec
Reservable BW: 2480000000 bits/sec
BW unreserved[0]: 2480000000 bits/sec, BW unreserved[1]: 2420891904 bits/sec
BW unreserved[2]: 2420771840 bits/sec, BW unreserved[3]: 2356474880 bits/sec
BW unreserved[4]: 2313820928 bits/sec, BW unreserved[5]: 2313820928 bits/sec
BW unreserved[6]: 2313820928 bits/sec, BW unreserved[7]: 2313820928 bits/sec
Affinity Bits: 0x00000000
272
Traffic Engineering Support
To View Tunnel Information:
sh isis mpls traffic-eng tunnel
Rtr-B# show isis mpls traffic-eng tunnel
Station Id
Rtr-C.00
Rtr-D.00
Tunnel Name
Tunnel1022
Tunnel1021
Tunnel1031
Tunnel1032
Bandwidth
3333
10000
10000
10000
Nexthop
2.2.2.2
2.2.2.2
3.3.3.3
3.3.3.3
Metric
-3
11
-1
Mode
Relative
Absolute
Relative
273
Traffic Engineering Support
To View Adjacency Information:
sh isis mpls traffic-eng adjacency-log
Rtr-B#show isis mpls traffic-eng adjacency-log
IS-IS RRR log
When
Neighbor ID
04:52:52 0000.0024.0004.02
04:52:50 0000.0026.0001.00
04:52:37 0000.0024.0004.02
IP Address
0.0.0.0
170.1.1.2
0.0.0.0
Interface
Et0/2
PO1/0/0
Et0/2
Status
Up
Up
Up
Level
level-1
level-1
level-1
274
Metric Style Issues
Metric style issues
An IS is configured for WIDE metrics and receives a TLV
with NARROW metrics:
Mismatch TLVs 0x80 (TLV 128) and 0x87 (TLV 135)
TLV 128 is IP Internal Reachability (Narrow)
TLV 135 is Extended IP Reachability TLV (Wide)
Rtr-B#debug isis update-packets
Sep 2 19:03:19.591: ISIS-Update: Rec L1 LSP 3333.3333.3333.00-00, seq B5D, ht 1199,
Sep 2 19:03:19.591: ISIS-Update: from SNPA *HDLC* (Serial2/2)
Sep 2 19:03:19.591: ISIS-Update: LSP newer than database copy
Sep 2 19:03:19.591: ISIS-Update: TLV code mismatch (87, 80)
Sep 2 19:03:19.591: ISIS-Update: TLV contents different, code 87
Sep 2 19:03:19.591: ISIS-Update: TLV code mismatch (16, 2)
275
Sep 2 19:03:19.591: ISIS-Update: Full SPF required
Route Leaking
Feature to enable redistributing Level 2
IP routes into Level 1 areas
Enables Level 1-only routers to pick the
best path to exit the area
Enables shortest-exit and MED for BGP
Redistribution can be controlled via
distribute-lists
IP-only feature (CLNS still uses stub)
276
Verifying Route Leaking
L1 Router
Leaking from Rtr-C to Rtr-D:
Rtr-A
S0
Area 49.0001
Rtr-D#sh ip ro isis
i ia 1.0.0.0/8 [115/74] via 192.168.111.2, Serial1
i ia 2.0.0.0/8 [115/74] via 192.168.111.2, Serial1
i ia 3.0.0.0/8 [115/74] via 192.168.111.2, Serial1
i ia 4.0.0.0/8 [115/74] via 192.168.111.2, Serial1
i ia 5.0.0.0/8 [115/74] via 192.168.111.2, Serial1
i ia 6.0.0.0/8 [115/74] via 192.168.111.2, Serial1
i ia 7.0.0.0/8 [115/74] via 192.168.111.2, Serial1
192.168.2.0/32 is subnetted, 2 subnets
i L1 192.168.2.2 [115/10] via 192.168.111.2, Serial1
i L1 192.168.222.0/24 [115/20] via 192.168.111.2, Serial1
i*L1 0.0.0.0/0 [115/10] via 192.168.111.2, Serial1
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
Routes highlighted are shown as IS-IS inter-area L1 Router
Routes originated on Rtr-B and leaked from Rtr-C
277
Verifying Route Leaking
L1 Router
Leaking from Rtr-C to Rtr-D:
Rtr-A
S0
Area 49.0001
Rtr-D#sh isis dat det
S1
IS-IS Level-1 Link State Database:
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-C.00-00
0x000000FB 0x8E85
620
Area Address: 49.0002
NLPID:
0xCC
Hostname: Rtr-C
IP Address: 192.168.2.2
Metric: 0
IP 192.168.2.2/32
Metric: 10
IP 192.168.111.0/24
Metric: 10
IP 192.168.222.0/24
Metric: 10000 IS-Extended Rtr-D.00
Metric: 74
IP-Interarea 1.0.0.0/8
Metric: 74
IP-Interarea 2.0.0.0/8
Metric: 74
IP-Interarea 3.0.0.0/8
Metric: 74
IP-Interarea 4.0.0.0/8
Metric: 74
IP-Interarea 5.0.0.0/8
Metric: 74
IP-Interarea 6.0.0.0/8
Metric: 74
IP-Interarea 7.0.0.0/8
Rtr-B
ATT/P/OL
1/0/0
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
278
HMAC-MD5 Authentication
Supported platforms:
12000, 10720, 10000, 7500, 7200
Supported across all packet types:
Hellos, LSPs, CSNPs, PSNPs
Can authenticate under the IS-IS process
(for LSP/CSNP/PSNP) and/or on the interface
(for hellos)
Can authenticate at Level 1 and/or Level 2
279
HMAC-MD5 Authentication
L1 Router
Typical Configuration:
!
key chain cisco
key 100
key-string systems
!
interface Serial1/0
ip address 10.1.1.1 255.255.255.252
ip router isis
isis authentication mode md5
isis authentication key-chain cisco
!
router isis
net 49.0000.0101.0101.0101.00
authentication mode md5
authentication key-chain cisco
Rtr-A
S0
Area 49.0001
S1
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
280
HMAC-MD5 Authentication
Use “sh clns neighbor” to verify
adjacency
Rtr-A#sh clns nei
System Id
Rtr-B
Interface
Se2/2
SNPA
*HDLC*
State Holdtime Type Protocol
Up
28
L1L2 IS-IS
Use “sh key chain” to view key chain
information
Rtr-A#sh key chain
Key-chain cisco:
key 100 -- text "systems"
accept lifetime (always valid) - (always valid) [valid now]
send lifetime (always valid) - (always valid) [valid now]
281
HMAC-MD5 Authentication
Use “debug isis authentication information”
Debug below shows normal operation as same
MD5 info is re-used if there is no change in IIH
Rtr-A#debug isis authentication information
Sep 23 15:41:45.040: ISIS-AuthInfo: IIH no change, use the same hmac value
Sep 23 15:41:54.880: ISIS-AuthInfo: IIH no change, use the same hmac value
Sep 23 15:42:04.620: ISIS-AuthInfo: IIH no change, use the same hmac value
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
282
HMAC-MD5 Authentication
Use “debug isis authentication information”
Debug below shows that local router received
a packet that does not contain authentication data
Rtr-B#debug isis authentication information
Sep 23 15:42:28.950: ISIS-AuthInfo: No auth TLV found in received packet
Sep 23 15:42:29.830: ISIS-AuthInfo: No auth TLV found in received packet
Sep 23 15:42:30.106: ISIS-AuthInfo: No auth TLV found in received packet
Sep 23 15:42:30.422: ISIS-AuthInfo: No auth TLV found in received packet
Sep 23 15:42:30.458: ISIS-AuthInfo: No auth TLV found in received packet
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
283
HMAC-MD5 Authentication
Neighbor does not use authentication for the IS-IS
instance
No LSPs, CSNPs or PSNPs sent with authentication
No LSPs, CNPs or PSNPs received by neighbor
Rtr-A#sh isis dat
IS-IS Level-1 Link State Database
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-A.00-00
* 0x00000002 0xDFE8
1195
0/0/0
IS-IS Level-2 Link State Database
LSPID
LSP Seq Num LSP Checksum LSP Holdtime
Rtr-A.00-00
* 0x00000002 0x5E9A
1195
0/0/0
ATT/P/OL
ATT/P/OL
284
HMAC-MD5 Authentication
No neighbor routes installed in local RIB
Rtr-A#sh ip route
C
C
C
C
192.168.60.0/30 is subnetted, 1 subnets
192.168.60.4 is directly connected, FastEthernet0/0
192.168.110.0/32 is subnetted, 1 subnets
192.168.110.1 is directly connected, Loopback0
192.168.20.0/30 is subnetted, 1 subnets
192.168.20.12 is directly connected, Serial2/2
10.0.0.0/22 is subnetted, 1 subnets
10.200.96.0 is directly connected, Ethernet5/0
285
HMAC-MD5 Authentication
Neighbor does not use authentication for the
interface
No hellos sent with authentication
No hellos received by neighbor
Neighborship is taken down—hold time expired
Rtr-A#
Sep 23 17:16:36.191: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-B (Serial2/2)
Down, hold time expired
Rtr-A#sh clns nei
System Id
Interface
SNPA
State Holdtime Type Protocol
286
HMAC-MD5 Authentication
Neighbor does not use authentication
for the interface
Neighborship taken down
Non-authenticating router in Init state
Rtr-B#
Sep 23 17:16:43.102: %CLNS-5-ADJCHANGE: ISIS: Adjacency to Rtr-A (Serial0/2)
Down, neighbor forgot us
Rtr-B#sh clns nei
System Id
Interface SNPA
Rtr-A
Se0/2
*HDLC*
State Holdtime Type Protocol
Init
27
L1L2 IS-IS
287
HMAC-MD5 Authentication
No Routes in RIB on Both Neighbors
Rtr-A#sh ip ro
192.168.60.0/30 is subnetted, 1 subnets
C
192.168.60.4 is directly connected, FastEthernet0/0
192.168.110.0/32 is subnetted, 1 subnets
C
192.168.110.1 is directly connected, Loopback0
192.168.20.0/30 is subnetted, 1 subnets
C
192.168.20.12 is directly connected, Serial2/2
10.0.0.0/22 is subnetted, 1 subnets
C
10.200.96.0 is directly connected, Ethernet5/0
Rtr-B#sh ip ro
192.168.30.0/30 is subnetted, 1 subnets
C
192.168.30.4 is directly connected, Serial0/0
192.168.150.0/30 is subnetted, 1 subnets
C
192.168.150.4 is directly connected, Ethernet3/2
192.168.20.0/30 is subnetted, 1 subnets
C
192.168.20.12 is directly connected, Serial0/2
10.0.0.0/22 is subnetted, 1 subnets
C
10.200.96.0 is directly connected, Ethernet3/0
192.168.220.0/32 is subnetted, 1 subnets
C
192.168.220.1 is directly connected, Loopback0
288
Multi-Topology (MT) Support
for IS-IS
Now IPv4 and IPv6 topologies are
separate
Lifts previous restrictions
Much easier deployment of IPv6
No need for IPv4 and IPv6 to be congruent
Avoids previous possible complications
Supported from 12.0(26)S, 12.2(15)T
Draft-ietf-isis-wg-multi-topology-xx.txt
289
MT for IPv6: Hello Processing
Maintaining MT Adjacencies
Basic hello processing and checking is the same
MT membership is advertised in IIH packets
MT ID #0 for IPv4
MT ID #2 for IPv6
Must have at least one common set of topology types
to form an adjacency
LAN nodes will always establish an adjacency
regardless of MT for reliable flooding and sync
290
MT for IPv6: LSP Generation
and Flooding
The LSP flooding mechanism is
unchanged for multi-topology
integrated IS-IS
For LANs—DIS, CSNP and PSNP
functions are unchanged by MT
extension
Use standard show and debug
commands to troubleshoot
291
MT for IPv6: SPF and RIBs
Each topology (IPv4 and IPv6) runs its own SPF
During IPv6 SPF we examine TLV 222 for MT ID #2
to build the Shortest Path Tree
Two-Way Connectivity Check (TWCC) follows
TWCC ensures bidirectional reachability
For leaf nodes we examine TLV 237 for MT ID #2
and install the IPv6 prefixes in the IPv6 RIB
292
MT for IPv6: Memory and
Performance
Additional memory will be required
However—will not have major impact on
overall system requirements
Performance impact will be additional
SPF runtime to compute L1 and L2 IPv6
topology
However IPv6 SPF not run back-to-back
with IPv4—but interleaved so no CPU
hogging
293
MT for IPv6: Restrictions
Not compatible with previous implementation
Not compatible with single SPF IPv6
However transition mode possible to migrate from
current IPv6 to MT IPv6
Advertises both IPv6 and MT IPv6 TLVs
If IPv4 and IPv6 are configured on the same
interface—they must be running the same level
Must use wide metrics
294
Multi-Topology IS-IS Example
Area C
Area B
Area A
Area D
IPv4-IPv6 Enable Router
The Multi-Topology Software Will Create
Two Topologies Inside Area IPv4 and IPv6
IPv4-Only Enable Router IPv4-Only Routers Will Be Excluded from
the IPv6 Topology
295
MT for IPv6: Basic
Configuration
Area
B
LAN1: 2001:0001::45c/64
Ethernet-1
Router1#
interface ethernet-1
ip address 10.1.1.1 255.255.255.0
ipv6 address 2001:0001::45c/64
ip router isis
ipv6 router isis
isis ipv6 metric 20
Router1
Ethernet-2
LAN2: 2001:0002::45a/64
The optional keyword
TRANSITION may be used for
transitioning existing IS-IS
IPv6 single SPF mode to MT
IS-IS
Wide metric is mandated for
Multi-Topology to work
interface ethernet-2
ip address 10.2.1.1 255.255.255.0
ipv6 address 2001:0002::45a/64
ip router isis
ipv6 router isis
isis ipv6 metric 20
router isis
net 49.0000.0100.0000.0000.0500
metric-style wide
!
address-family ipv6
multi-topology
exit-address-family
296
Cisco IOS Multi-Topology IS-IS
Display
Router# show clns neighbors detail
System Id Interface
SNPA
State
Holdtime
2653
Se0/1
*HDLC*
Up
25
Area Address(es): 49.0000.01
IP Address(es): 192.168.0.6*
IPv6 Address(es): FE80::204:C1FF:FEDB:2FA0
Uptime: 00:01:22
Topology: IPv4, IPv6
Type
L1L2
2652# show isis database detail
IS-IS Level-2 Link State Database:
LSPID
LSP Seq Num
LSP Checksum
Holdtime
ATT/P/OL
2651.00-00 0x0000000F
0x0161
1066
Area Address: 49.0000.01
Topology: IPv4 (0x0) IPv6 (0x2)
NLPID: 0xCC 0x8E
Hostname: 2651
IP Address: 192.168.0.2
IPv6 Address: 3FFF:FFFF:2::1
Metric: 10 IS-Extended 2652.00
Metric: 10 IS-Extended 2653.01
Metric: 10 IS (MT-IPv6) 2653.01
Metric: 10 IP 192.168.0.0/30
Metric: 20 IP 192.168.0.4/30
Metric: 10 IP 192.168.1.0/24
Metric: 20 IPv6 (MT-IPv6) 3FFF:FFFF:1::/64
Metric: 10 IPv6 (MT-IPv6) 3FFF:FFFF:2::/64
Protocol
M-ISIS
LSP
0/0/0
MT IS-IS
297
Local RIB
The Local RIB contains entries for all
possible routes to destinations
From this RIB, the best route is chosen
to be installed in the ‘IP routing table’
Local RIB is also necessary for prefix
prioritization
Local RIB can significantly reduce
convergence time
298
Viewing Local RIB Information
L1 Router
Rtr-A
S0
Rtr-B#sh isis rib ?
A.B.C.D
Network prefix
redistribution ISIS IP redistribution RIB information
|
Output modifiers <cr>
Area 49.0001
S1
Rtr-B
Rtr-B#sh isis rib 192.168.40.14
IPv4 local RIB for IS-IS process
Routes under majornet 192.168.40.0/24:
192.168.40.4/30
[115/L1/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[3/51]
[115/L2/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[4/166]
192.168.40.16/30
[115/L1/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[3/51]
[115/L2/20] via 192.168.40.5(Serial0/0), from 192.168.40.17, tag 0, LSP[4/166]
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
299
Debugging Local RIB
Information
L1 Router
Rtr-B#debug isis rib ?
global
IS-IS IP routes in global RIB
local
IS-IS IP routes in local RIB
redistribution IS-IS IP redistribution RIB
Rtr-A
S0
Area 49.0001
S1
Rtr-B#debug isis rib local
IS-IS IPv4 local RIB debugging is on
Rtr-B#
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.4/30: Looking for RT
*Mar 15 01:18:34.882: ISIS-LR: RT exists
*Mar 15 01:18:34.882: ISIS-LR: path exists: [115/40/20] via 192.168.40.5(Se0/0)
from 192.168.40.17 tg 0 LSP[3/(51->52)]
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: Looking for RT
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: Create new RT
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: create new path: [115/40/20]
via 192.168.40.5(Se0/0) from 192.168.40.5 tag 0 LSP[3/52]
*Mar 15 01:18:34.882: ISIS-LR: Enqueued to updateQ[2] for 192.168.40.12/30
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.16/30: Looking for RT
*Mar 15 01:18:34.882: ISIS-LR: RT exists
*Mar 15 01:18:34.882: ISIS-LR: path exists: [115/40/20]
via 192.168.40.5(Se0/0) from 192.168.40.17 tg 0 LSP[3/(51->52)]
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.12/30: not aged out in LSP ix 3 same ver(52)
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.16/30: not aged out in LSP ix 3 same ver(52
L1 Router
*Mar 15 01:18:34.882: ISIS-LR: 192.168.40.4/30: not aged out in LSP ix 3 same ver(52)
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
300
Debugging Global RIB
Information
L1 Router
Rtr-B#debug isis rib ?
global
IS-IS IP routes in global RIB
local
IS-IS IP routes in local RIB
redistribution IS-IS IP redistribution RIB
Rtr-A
S0
Area 49.0001
S1
Rtr-B#debug isis rib global
IS-IS IPv4 global RIB debugging is on
Rtr-B#
*Mar 15 01:22:06.117: ISIS-GR: ------ Start updateQ, from 60773650
*Mar 15 01:22:06.117: ISIS-GR: 192.168.210.1 255.255.255.255: updateQ[1] entry
*Mar 15 01:22:06.117: ISIS-GR: del rdb
*Mar 15 01:22:06.117: ISIS-GR: 192.168.250.12 255.255.255.252: updateQ[2] entry
*Mar 15 01:22:06.117: ISIS-GR: del rdb
*Mar 15 01:22:06.117: ISIS-GR: 192.168.250.4 255.255.255.252: updateQ[2] entry
*Mar 15 01:22:06.121: ISIS-GR: del rdb
*Mar 15 01:22:06.121: ISIS-GR: 192.168.30.16 255.255.255.252: updateQ[2] entry
*Mar 15 01:22:06.121: ISIS-GR: del rdb
*Mar 15 01:22:06.121: ISIS-GR: 192.168.20.4 255.255.255.252: updateQ[2] entry
*Mar 15 01:22:06.121: ISIS-GR: del rdb
*Mar 15 01:22:06.121: ISIS-GR: ------ End updateQ, 0 suspends
*Mar 15 01:22:06.121: ISIS-GR: ------ Start updateQ, from 60773DE0
*Mar 15 01:22:06.121: ISIS-GR: 192.168.40.12 255.255.255.252: updateQ[2] entry
*Mar 15 01:22:06.125: ISIS-GR: del rdb
L1 Router
*Mar 15 01:22:06.125: ISIS-GR: ------ End updateQ, 0 suspends
Rtr-B
S0
L1L2 Routers
S1
S0
Rtr-C
Area 49.0002
S1
Rtr-D
301
Incremental SPF (i-SPF)
Incremental SPF
Modified Dijkstra algorithm
We keep the unchanged part of the tree
We rebuild only the affected parts of the
tree
Re-attach the affected parts of the tree
to the unchanged part of the tree
Fairly complex algorithm
302
Incremental SPF (i-SPF)
When new LSPs are received, each router will check
what has changed in the LSP
Based on the changed information, the SPT is
“modified” in order to reflect the changes
Start the computation from the node that received
the change
i-SPF is essentially used to prepare lists for running
standard Dijkstra and computing next-hops
No need to run SPF if a link that was not in the SPT
is reported down
303
Incremental SPF (i-SPF)
G
F
Cost: 8, NH: D, B
C
E
Cost: 13, NH: D
Cost: 6, NH: D, B
D
Cost: 3, NH: D
Cost: 11, NH: B
S2
B
Cost: 3, NH: B
S3
S1
S0
A
Cost: 0, NH: --
C-G Link Is Down; C-G Link
Was Not Used in SPT
Anyway, Therefore There Is
No Need to Run SPF
304
Incremental SPF (i-SPF)
G
H
F
Cost: 8, NH: D, B
C
E
F Reports a New Neighbor;
the SPT Need Only To Be
Extended Behind F; There
Is No Need for Router A to
Recompute the Whole SPT
Router A Will Compute SPF
from Node F
Cost: 13, NH: D
Cost: 6, NH: D, B
D
Cost: 3, NH: D
Cost: 11, NH: B
S2
B
Cost: 3, NH: B
S3
S1
S0
A
Cost: 0, NH: --
305
Incremental SPF (i-SPF):
Configuration
When router is powered on—we always run a full
SPF to build the tree
To configure i-SPF under router isis:
ispf [level-1/level-2/level-1-2] [1-600]
Must specify levels
The last parameter (1–600) is optional and
corresponds to the duration (in seconds) of full SPF
runs after reload
By default, full SPF is run for the first 60 seconds
after reload—even if i-SPF is configured
Integrated into 12.0(24)S
306
debug isis spf-statistics
Rtr-A#debug isis spf-statistics
*Mar 15 01:03:44.044: ISIS-Stats:
*Mar 15 01:03:44.044: ISIS-Stats:
*Mar 15 01:03:44.048: ISIS-Stats:
*Mar 15 01:03:44.048: ISIS-Stats:
*Mar 15 01:03:44.048: ISIS-Stats:
*Mar 15 01:03:44.048: ISIS-Stats:
*Mar 15 01:03:44.048: ISIS-Stats:
Compute L1 SPT
Starting incremental SPF for level-1
SPF only compute time 0.000
IPv4 RIB only compute time 0.004
Complete L1 SPT, Compute time 0.004, 2 nodes, 0 links on SPT, 0 suspends
I-SPF NewLSP: 0.000 (3 nodes) - Reattach: 0.000 - WalkTENT: 0.000
I-SPF WalkParents: 0.000 (0 nodes) - Total time: 0.000
L1L2 Routers
S0
S1 Rtr-B
Area 49.0001
S1
Rtr-C S0
Area 49.0002
S0
S1
Rtr-A
Rtr-D
L1 Routers
307
Viewing i-SPF
L1 Router
Rtr-A
S0
Rtr-A#sh isis spf-log detail
Area 49.0001
level 1 SPF log
S1
i-SPF
triggered 2 times, first at 01:03:38.548 Mar 15 1993 by TLVCONTENT
wait enforced 5.500, next wait interval 5.500
SPT node total/processed/time:
4/2/0.000
RIB prefix processed/time:
0/0.000 1/0.000 3/0.004
Rtr-B
S0
L1L2 Routers
S1
S0
Full SPF
triggered 2 times, first at 00:51:17.877 Mar 15 1993 by NEWADJ
wait enforced 5.500, next wait interval 5.500
SPT node total/processed/time:
2/2/0.000
RIB prefix processed/time:
0/0.000 0/0.000 2/0.004
Rtr-C
Area 49.0002
S1
Rtr-D
L1 Router
308
Non-Stop Forwarding
Allows uninterrupted data forwarding during a
switchover to a standby RP
Routing protocol mechanisms are also applicable
to a restart in the RP
Forwarding Information Base (FIB) is maintained and
updated once the routing protocols
reconverge
May be used for planned and unplanned events
Two modes of NSF available:
Cisco and IETF
309
NSF: Routing Protocol
Requirements
Switchover MUST be completed before dead/hold
timer expires
Peers will reset the adjacency and reroute the
traffic after that time
FIB MUST remain unchanged during switchover
Current routes marked as “dirty” during restart;
“cleaned” once convergence is complete
Adjacencies MUST NOT be reset when switchover is
complete
Protocol state is not maintained
Peers of restarting router SHOULD also be
NSF-aware
310
NSF Configuration Commands
To Configure NSF under “router isis”:
nsf [cisco/ietf ]
Enables isis nsf, OFF by default
nsf interval xxx
Minimal time interval between two restart (default
=5min)
nsf holdtime [manual <seconds> | adjacency]
IETF version only
Time NSF will wait for the LSP database to synchronize
before generating and flooding its own LSP with the
overload-bit set
311
NSF Configuration and Debug
Commands
nsf interface wait xxx
Cisco version only (default=10 sec, range [1–60] )
Time an NSF restart will wait for all interfaces with ISIS
adjacencies to
come up before completing the restart
Show commands
show isis nsf
nsf capability and restart
information
show clns neighbor detail peer’s nsf capability
show isis database detail LSP information
Debug commands
debug isis nsf [cisco | detail | ietf]
debug isis adj-packets
312
IS-IS NSF “show commands”
NSF Cisco mode—check that it is enabled
chi-hr1#show isis nsf
NSF is ENABLED, mode 'cisco'
RP is ACTIVE, standby ready, bulk sync complete
NSF interval timer expired (NSF restart enabled)
Checkpointing enabled, no errors
Local state: ACTIVE, Peer state: STANDBY HOT, Mode: SSO
We maintain the IS-IS Neighbor and Database Information
on the STANDBY [G]RP
GRP-Slot9#show clns neighbor
System Id Interface SNPA State Holdtime Type Protocol
chi-ar1 PO2/0 *HDLC* Stby 30
L2 IS-IS
313
IS-IS NSF “show commands”
NSF IETF mode—check that it is enabled
chi-hr1#sh isis nsf
NSF is ENABLED, mode 'ietf'
NSF pdb state: Inactive
NSF L2 active interfaces: 0
NSF L2 active LSPs: 0
NSF interfaces awaiting L2 CSNP: 0
Awaiting L2 LSPs:
NSF T3 remaining: 0 seconds
Interface: Ethernet0
NSF L2 Restart state: Running
NSF L2 Restart retransmissions: 0
Maximum L2 NSF Restart retransmissions: 3
Interface: POS2/0
NSF L2 Restart state: Running
NSF p2p Restart retransmissions: 0
Maximum L2 NSF Restart retransmissions: 3
314
IS-IS NSF “show commands”
chi-hr1#show clns neighbor detail
System Id Interface SNPA State Holdtime Type Protocol
chi-ar1
PO2/0
*HDLC* Up
21
L2
IS-IS
Area Address(es): 10
IP Address(es): 172.1.1.21*
Uptime: 01:52:02
NSF capable
315
IS-IS NSF “debug commands”
Rtr-A#debug isis nsf?
cisco Include only Cisco NSF information
detail Include detailed information
ietf
Include only IETF NSF information
Other debug information also useful
with NSF:
debug isis update-packets
debug isis snp-packets
debug isis adj-packets
316
IS-IS NSF “debug isis nsf ietf”
*Aug 12 17:32:45.507 PDT: ISIS-NSF: Inserting p2p
NSF REQ on POS3/0 IIH
*Aug 12 17:32:45.519 PDT: ISIS-NSF: POS3/0 level-2
state progression: state=Restarting/event=RA
Rcvd/newstate=RA Seen
*Aug 12 17:32:45.519 PDT: ISIS-NSF: ISIS NSF restart
ACK option received on p2p itf from 2002.0020.0001
(POS3/0)
*Aug 12 17:32:45.559 PDT: ISIS-NSF: POS3/0 level-2
state progression: state=RA Seen/event=CSNP
Rcvd/newstate=Running
317
IS-IS NSF: Other Useful Debug
Commands
• debug isis update-packets
• debug isis snp-packets
• debug isis adj-packets
.Aug 23 14:10:18.926 PDT: ISIS-Update: Sending L2 P2P sync CSNP on PO2/0
.Aug 23 14:10:18.926 PDT: ISIS-Update: Sending L2 CSNP on POS2/0
.Aug 23 14:10:18.926 PDT: ISIS-SNP: Rec L2 PSNP from 2002.0020.0001
(POS2/0)
.Aug 23 14:10:18.926 PDT: ISIS-SNP: PSNP entry 2002.0020.0003.FE-FE, seq
0, ht 1199
.Aug 23 14:10:20.962 PDT: ISIS-Update: Sending L2 LSP
2002.0020.0003.00-00, seq F7, ht 1199 on POS2/0
.Aug 23 14:10:21.790 PDT: ISIS-Adj: Rec serial IIH from *HDLC* (POS2/0),
cir type L2, cir id 00, length 4469
.Aug 23 14:10:21.790 PDT: ISIS-Adj: rcvd state UP, old state UP, new state
318
UP
Trouble Shooting CEF
Vocabulary
CEF: Cisco Express Forwarding
FIB: Forwarding Information Base
collection of data used to make switching
decision
•
RIB: Routing Information Base
topology information that the router learns
from routing protocols
320
CEF...What is it?
Advanced layer-3, IP switching
method; scaleable, distributed, high
performance
CEF differs from current fast switching
in the way the router maintains its
forwarding table
CEF switching is required for most of
the new IP QoS features
321
Features
Load balancing
Per destination (the default) and per
packet over equal/unequal cost links for as
many paths as known in the routing
topology
Traffic statistics
Byte and packet counts at a granularity of
per-prefix, per-neighbor etc..
322
Features (cont..)
Media independence
CEF currently supports Packet over Sonet,
ATM/AAL5, Frame Relay, Ethernet, FDDI,
HDLC, PPP and tunnels.
Tunneling: Generic Route Encapsulation
(GRE).
Subinterface support: allowing for the
flexibility of per subinterface configurations
e.g. MTU.
323
Today’s Problems and CEF’s
Solutions
Type of traffic - Efficiency
Today’s traffic is short lived. Cache on demand
(fast switching) requires constant cache
create/invalidate (aging). FIB has a forwarding
entry for every routable prefix.
Dynamic networks - Performance
The FIB table mirrors the routing table exactly thus fast convergence
324
Today’s Problems and CEF’s
Solutions (cont.)
Scalability
Core routers carry high number of prefixes
(mostly recursive) which need to be
resolved each time a packet has to be
process switched (cache on demand). In
CEF, the recursive route is resolved when
the FIB table is built. dCEF is also available.
Better use of memory and CPU.
325
Possible problem
The throughput of the router is less
than expected.
The throughput of an interface is less
than expected.
Packets are getting dropped when
CEF/dCEF is enabled
Packets are not CEF switched but fast
switched instead.
326
Possible problem(cont.)
Packets are not dCEF switched but fast
switched instead
Packets are not dCEF switched but CEF
switched instead.
Load-sharing is not working
The route processor runs out of memory
when dCEF is enabled
CEF disables itself on a linecard due to lack
of memory
327
The throughput of the router
is less than expected.
Possible problem :CEF/dCEF is
not enabled globally on the router.
Solution
Step 1 Check is CEF/dCEF is enabled globally on
router.
Use global command ip cef for central processor
switching or ip cef distributed if you want
distributed CEF switching. When CEF/dCEF is enabled
globally, CEF/dCEF is automatically enabled on the
interfaces.
If show ip summary command gives an output that
means CEF is enabled. If not, you get the following
output for show ip summary:
router#show ip summary
%CEF not running
329
The throughput of an
interface is less than
expected.
Possible problem : CEF/dCEF is
not enabled on this interface.
Solution
Step 1 Check if CEF/dCEF is enabled on the interface.
Use interface command ip route-cache cef for
central processor switching or ip route-cache
distributed if you want distributed CEF switching.
The result of show cef interface x/x and show
ip interface x/x must report one of the following:
IP CEF switching enabled
IP Normal CEF switching turbo vector
IP Distributed CEF switching enabled
IP Distributed CEF switching turbo vector
Below is a example on a router:
331
Example
router#show cef interface FastEthernet0/1
FastEthernet0/1 is up (if_number 3)
Corresponding hwidb fast_if_number 3
Corresponding hwidb firstsw->if_number 3
Internet address is 10.254.0.200/24
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
Hardware idb is FastEthernet0/1
Fast switching type 1, interface type 18
IP CEF switching enabled
IP Feature Fast switching turbo vector
IP Feature CEF switching turbo vector
Input fast flags 0x0, Output fast flags 0x0
ifindex 2(2)
Slot 0 Slot unit 1 VC -1
Transmit limit accumulator 0x0 (0x0)
IP MTU 1500
332
Packets are getting dropped
when CEF/dCEF is enabled
Possible problem : The CEF
entries and the adjacencies table
are not completely built.
Solution
Step 1 Verify whether the router has completely built the CEF
entries and adjacencies table. If it is not the case the router
may switch the packet using an alternative switching method.
This is why the packets are 50% successful. To verify this,
issue the command show ip cache. There should be an IP
cache entry built for this prefix.
router#show ip cache
IP routing cache 0 entries, 0 bytes
3 adds, 3 invalidates, 0 refcounts
The above result of the command shows that the router has
been doing cache switching for a moment: 3 adds, 3 invalidates.
The result of the ping is 50% successful as the number of
packets that are punted is rate limited.
334
Solution(cont.)
router #ping ip
Target IP address: 12.0.0.1
Repeat count [5]: 20
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 20, 100-byte ICMP Echoes to 12.0.0.1, timeout is 2 seconds:
!.!.!.!.!.!.!.!.!.!.
Success rate is 50 percent (10/20), round-trip min/avg/max = 1/1/1 ms
An adjacency on interface POS8/0/0 is still incomplete.
router#show adjacency
Protocol Interface
Address
IP
POS8/0/0
point2point(3)
IP
Serial10/0/0:28
point2point(3)
335
IP
POS8/0/0
point2point(2) (incomplete)
Packets are getting dropped
when CEF/dCEF is enabled
Possible problem : Part of the
adjacencies is “incomplete
adjacency” or “drop adjacency”.
Solution
Step 1: Check the features you have enabled; some
platforms only support dCEF with a limited number of
features. If you enable a feature not supported by
dCEF a drop adjacency is created and the packets are
dropped. Issue the command show cef drop and
look for packets being dropped. Below you will find
the output of the command when a feature is not
supported.
router#show cef drop
CEF Drop Statistics
Slot Encap_fail Unresolved Unsupported
No_route
No_adj
RP
0
0
2
0
0
337
Solution(cont.)
Step 2 If your hardware supports CEF and dCEF, in case of
packet drops you may see only “incomplete adjacency”.
Regardless of the platform, here is the method to follow to
isolate the problem.
Use show ip route to determine which inbound interface the
traffic should be switched through. Issue show ip cef
<prefix> and look for the interface adjacencies:
router#show ip cef 10.254.0.200
10.254.0.200/32, version 18, cached adjacency 10.254.0.200
0 packets, 0 bytes
via 10.254.0.200, FastEthernet0/1, 0 dependencies
next hop 10.254.0.200, FastEthernet0/1
valid cached adjacency
338
Solution(cont.)
router#show adjacency FastEthernet 0/1 detail
Protocol Interface
Address
IP
FastEthernet0/1
10.254.0.200(5)
0 packets, 0 bytes
00055FAF2C06
00055FAC18010800
ARP
03:45:10
IP
FastEthernet0/1
server(5) (10.254.0.1)
0 packets, 0 bytes
00508B121E46
00055FAC18010800
ARP
03:54:27
339
Packets are getting dropped
when CEF/dCEF is enabled
Possible problem : None of the
above solutions are applying – try
to debug the issue.
Solution
Step 1: Check if the packets are dropped or punt for
unexpected reason. The following counters are
reporting the punt and dropped packets:
show cef not-cef-switched (punt reason
counters)
show cef drop (drop reason counters)
The above is particularly useful in a controlled
environment. You send a number of packets. If all
the packets are dropped or punt you it gives you the
reason for this unexpected behavior.
341
Solution(cont.)
Step 2: If the traffic to a destination is
limited, you can try the following debug
commands:
debug ip cef drop <acl>
debug ip cef receive <acl>
access-list <acl> permit
<destination>
The drop and receive debug information will
provide more information on why packets are
being dropped or punted by CEF/dCEF.
342
Packets are getting dropped
when CEF/dCEF is enabled
Possible problem :None of the
above solutions are applying –
Call the TAC
Solution
step 1: As always, for a CEF/dCEF unexpected behavior, collect:
show tech cef
Step 2: If you cannot get a destination and if none of the above solutions
are applying to your case, you will have to collect a list of information
to help the Cisco TAC solving your problem.
show ip route <destination>
show ip cef <destination> internal
show adjacency <output interface >
show ip interface brief
show cef interface
exec all show ip cef <destination> internal
exec all show adjacency <output interface> detail
exec all show cef interface
show ip arp (or analogous link info, e.g. show frame-relay map)
To get to a destination, there has to be a route, a forwarding entry, and a
valid adjacency. The data collected above will show which element is
missing.
344
Packets are not CEF switched
but fast switched instead.
Possible problem: CEF does not
support a configured feature.
Solution
Step 1 While CEF supports a large range of
features, however some features are not
supported. If a feature is not supported
packets are forwarded to the next slower
level of switching - fast switching.
Check the following document to know
exactly what is supported:
IOS features supported by CEF and dCEF
matrix
346
Solution(cont.)
Step 2 Check the configuration of the router to make
certain CEF is activated. Also use the following
commands:
show cef interface x/y,
show ip cef <prefix> ,
show adjacency <interface> detail
Ultimately issue show cef not-switch and look for
packets that are not CEF switched.
If CEF is not able to forward packets to a destination,
the CEF table is built anyway but the adjacency is
called a “punt”. If just a part of the packets cannot be
CEF switched a cache adjacency is created.
347
Solution(cont.)
Router #show ip cef 172.20.1.2
172.20.1.0/30, version 4, attached, connected, per-destination sharing
0 packets, 0 bytes
via Serial4/0, 1 dependency
valid punt adjacency
As the result of punt adjacency the router will report CEF packets passed on to next
switching layer.
router#show cef interface serial4/0
Serial4/0 is up (if_number 13)
Corresponding hwidb fast_if_number 13
Corresponding hwidb firstsw->if_number 13
Internet address is 172.20.1.1/30
ICMP redirects are always sent
Per packet load-sharing is disabled
IP unicast RPF check is disabled
Inbound access list is not set
Outbound access list is not set
IP policy routing is disabled
Interface is marked as point to point interface
Packets switched to this interface are dropped to the next slow path 348
Solution(cont.)
Step3 Look under interface/global configuration for
features that you are not sure to be compatible with
CEF. Log messages from router, then remove and put
back one by one each of these features.
When removing a not CEF switched feature, the log will
report:
%FIB-5-NOPUNTINTF: CEF resuming switching packets
to <interface>
When adding a feature which cannot be CEF switched,
the log will report:
%FIB-4-PUNTINTF: CEF punting packets switched to
<interface> to next slower path
349
Packets are not dCEF switched
but fast switched instead
Possible problem: Neither dCEF
nor CEF support the feature
configured.
Solution
Step 1 While dCEF supports a large range of
features, some features are not supported. In
this case the packets are forwarded to the
next slower level of switching. If dCEF cannot
process the packet, the router tries to use
CEF; if the packet cannot be processed with
CEF switched it will be fast switched.
Check the following document to know
exactly what is supported:
IOS features supported by CEF and dCEF
matrix
351
Solution(cont.)
Step 2 Even if neither dCEF nor CEF
are enabled to forward the packet to a
destination, the CEF table is built
anyway but the adjacency is called a
“punt”. See table 4 for further
troubleshooting steps.
352
Packets are not dCEF switched
but CEF switched instead.
Possible problem: dCEF does
not support the feature configured.
Solution
Step 1 While dCEF supports a large range of
features, However some features or hardware are not
supported. In this case the packets are forwarded to
the next higher level of switching, which is
centralized. In this case the feature configured is
supported by CEF.
Check the following documents to know exactly what
is supported:
-
IOS features supported by CEF and dCEF matrix
IOS hardware support of CEF and dCEF matrix
354
Solution(cont.)
Step 2 Make certain dCEF is configured on the
routing using the following commands:
show cef interface x/y,
show ip cef <prefix>,
show adjacency <interface> detail
Ultimately issue show cef not-switch and look for
packets that are not dCEF switched.
Use the show interface <interface> stats
command to figure out if the packets are dCEF
switched.
355
Example
Router# show interface FastEthernet0/0/0 stats
FastEthernet0/0/0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor
3736
383776
280261
387230212
Route cache
3800
390105
380276
487270282
Distributed cache 0
0
0
0
Total
7536
773881
660537
874500494
In this case none of the packets going through the interface are
dCEF switched. All the counters for “Distributed cache” row are
still zero.
356
Load-sharing is not
working
Possible problem: The traffic
includes only a small number of
flows.
Solution
Step1 Check the load-sharing mode. Load-sharing
during CEF/dCEF switching has two modes:
Per-destination (original, tunnel, universal)
Per-packet.
If you do not want to enable per-packet load sharing,
enable the per-destination tunnel algorithm that is
designed to give a good traffic load-sharing when the
number of flows is small. Issue the global following
command:
ip cef load-sharing algorithm tunnel
358
Load-sharing is not
working
Possible problem: The network
topology is subject to traffic
polarization
Solution
Step1 Per destination load sharing is based upon a hash
function to perform the balancing. This hash function is subject
to polarization, canceling the benefits of load sharing – some
routers, under specific traffic distribution patterns, assign all the
sessions to the same link regardless of other available paths.
The next hop will perform the same hash algorithm and will
assign all the traffic to a single link.
Check the load-sharing mode. Load-sharing during CEF/dCEF
switching has two modes:
Per-destination (old algorithm, new algorithm, tunnel)
Per-packet.
Issue the following global command on all the routers in the
network:
ip cef load-sharing algorithm universal
This modifies the hash algorithm on each router to correct the
effects of polarization. You can also enable per packet load
sharing.
360
Load-sharing is not
working
Possible problem: Only one
next-hop is accessible to go to a
destination.
Solution
Step1 Make certain there are multiple valid next-hops
to the destination. If the data streams are destined to
different hosts, then we should have load sharing. If
not, check to see if all the possible parallel paths are
up and their next-hop are reachable. You can issue
the following commands:
show ip route <prefix> and look for more than 1
next-hop.
show ip cef <prefix> and look for more that 1 nexthop.
show ip cef <prefix> internal
show ip cef exact-route <source address>
<destination address>
show adjacency detail on those next hop interfaces.
362
The route processor runs out of
memory when dCEF is enabled
Possible problem:
CEF messages are queued in the
route processor.
Solution
Step1 Check the memory available on the route
processor. dCEF is downloading CEF tables to all of
the linecards of the router if distributed CEF is turned
on. During this process the route processor may
queue a lot of CEF messages, and run out of memory.
The following commands can be issued on the route
processor to track the problem:
show ip cef summary
show memory summary
show process memory
The depth of the queue of IPC information elements
(xdrs) packed into IPC messages sent from the route
processor to the line card is reported in the output of
the following command:
364
Example
router#show cef linecard detail
CEF linecard slot number 0, status up
Sequence number 46, Maximum sequence number expected 118, Seq
Epoch 1
Send failed 0, Out Of Sequence 0, drops 0
Linecard CEF reset 0, reloaded 1
1208 elements packed in 944 messages(50328 bytes) sent
0 elements cleared
linecard disabled - failed a reload
0/0/0 xdr elements in LowQ/MediumQ/HighQ
24/9/25 peak elements on LowQ/MediumQ/HighQ
Input packets 0, bytes 0
Output packets 0, bytes 0, drops 0
CEF Table statistics:
Table name
Version Prefix-xdr Status
Default-table
106
117 Active, sync
365
Solution(cont.)
Step2 Tuning the windowing mechanism
used for the route processor/linecard
communication will help fixing the problem.
Increasing this may reduce the queuing, and
thus memory used on the route processor.
With the following command you will change
the default value (100 Kbytes) of the window:
ip cef linecard ipc memory
<Kbytes>
366
CEF disables itself on a linecard
due to lack of memory
Possible problem:
The linecard has a memory
shortage.
Solution
Step1 Check the memory available on the linecards.
dCEF downloads CEF tables to all of the linecards of
the router if distributed CEF is turned on, even if you
aren't CEF switching on that interface. Because of
this, all linecards must have the memory necessary to
carry all the routes.
The following commands can be issued on each
linecards to track the problem:
show ip cef summary
show memory summary
show process memory
show cef linecard
On a 7500, enter the VIP card with if-con <x>
console.
368
CEF Case Study
Case Study(topology)
R1
12.0.0.1
FDDI
Dual Ring
12.0.0.3
12.0.0.2
11.0.0.2
11.0.0.1
IGRP-555
BGP-AS-333
R3
R2
13.0.0.3
BGP
BGP
13.0.0.1
AS-222
Local Preference 100
AS-111
Local Preference 2000
Nets
99, 192, 195,197
370
R1’s Output
R1#show ip route
Codes: C - connected, S - static, I - IGRP, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
E1 - OSPF external type 1, E2 - OSPF external type 2, E - EGP
i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, * - candidate default
U - per-user static route
Gateway of last resort is not set
B 2.0.0.0/8 [200/0] via 13.0.0.1, 18:17:13
B 197.12.11.0/24 [200/0] via 11.0.0.1, 18:17:50
B 99.0.0.0/8 [200/0] via 11.0.0.1, 18:17:50
B 197.12.12.0/24 [200/0] via 11.0.0.1, 18:17:51
B 195.31.54.0/24 [200/0] via 11.0.0.1, 18:17:51
I 11.0.0.0/8 [100/1110] via 12.0.0.2, 00:00:49, Fddi1/1/0
B 195.27.191.0/24 [200/0] via 11.0.0.1, 18:17:51
B 197.28.99.0/24 [200/0] via 11.0.0.1, 18:17:51
C 12.0.0.0/8 is directly connected, Fddi1/1/0
B 192.12.1.0/24 [200/0] via 11.0.0.1, 18:17:51
B 192.33.1.0/24 [200/0] via 11.0.0.1, 18:17:51
B 195.181.32.0/24 [200/0] via 11.0.0.1, 18:17:51
I 13.0.0.0/8 [100/252] via 12.0.0.3, 00:00:00, Fddi1/1/0
B 197.21.251.0/24 [200/0] via 11.0.0.1, 18:17:51
B 197.111.82.0/24 [200/0] via 11.0.0.1, 18:17:51
B 192.18.2.0/24 [200/0] via 11.0.0.1, 18:17:51
371
R1’s Output (cont.)
R1#show ip cef
Prefix
Next Hop
0.0.0.0/32
receive
2.0.0.0/8
12.0.0.3
11.0.0.0/8
12.0.0.2
12.0.0.0/8
attached
12.0.0.0/32
receive
12.0.0.1/32
receive
12.0.0.2/32
12.0.0.2
12.0.0.3/32
12.0.0.3
12.255.255.255/32 receive
13.0.0.0/8
12.0.0.3
99.0.0.0/8
12.0.0.2
192.12.1.0/24
12.0.0.2
192.18.2.0/24
12.0.0.2
192.33.1.0/24
12.0.0.2
195.27.191.0/24 12.0.0.2
195.31.54.0/24 12.0.0.2
195.181.32.0/24 12.0.0.2
197.12.11.0/24 12.0.0.2
197.12.12.0/24 12.0.0.2
197.21.251.0/24 12.0.0.2
197.28.99.0/24 12.0.0.2
197.111.82.0/24 12.0.0.2
224.0.0.0/4
receive
255.255.255.255/32 receive
Interface
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
Fddi1/1/0
372
R1’s Output (cont.)
R1#show ip cef 195.181.32.0 d
195.181.32.0/24, version 17, cached adjacency 12.0.0.2
0 packets, 0 bytes
via 11.0.0.1, 0 dependencies, recursive
next hop 12.0.0.2, Fddi1/1/0 via 11.0.0.0/8
valid cached adjacency
R1#show ip cef 11.0.0.1 detail
11.0.0.0/8, version 9, cached adjacency 12.0.0.2
0 packets, 0 bytes
via 12.0.0.2, Fddi1/1/0, 12 dependencies
next hop 12.0.0.2, Fddi1/1/0
valid cached adjacency
R1#show adj detail
Protocol Interface
IP
Fddi1/1/0
IP
Fddi1/1/0
Address
12.0.0.2(31)
0 packets, 0 bytes
5000000C0B
1C8000603E617C28AAAA030000000800
ARP
01:33:09
12.0.0.3(9)
0 packets, 0 bytes
5000603E28
B22800603E617C28AAAA030000000800
ARP
01:32:43
373
CEF Case Study
Troubleshooting
Useful Information
Verify CEF/DCEF entry
Verify adjacency entry
Check interface switching mode
Check all features that CEF/DCEF
supports
375
Useful Information (cont.)
Use “show” and “debug” commands
show cef drop
show cef not-cef-switch
show cef [linecard] [interface]
Verify with other switching modes
376
Show Outputs
center#show cef drop
CEF Drop Statistics
Slot Encap_fail Unresolved
RP
4
0
2
0
0
3
0
0
4
0
0
5
0
0
Unsupported
0
0
0
0
0
No_route
0
0
2452
0
0
center#show cef not-cef-switched
CEF Packets passed on to next switching layer
Slot No_adj No_encap Unsupp'ted Redirect
RP
0
0
41462
2
0
0
0
3
0
0
357
4
0
0
0
5
0
0
216504
No_adj ChksumErr
0
0
0
0
0
0
0
0
0
0
Receive Bad_ttl Options Access
0
7922
0
0
0
0
0
0
0
0
0
1161
0
0
0
0
23
3250
0
0
0
0
0
0
0
377
Show Outputs (cont.)
center#show cef int f9/0/0
Fddi9/0/0 is up (if_number 26)
Internet address is 190.1.0.2/16
ICMP redirects are always sent
Per packet loadbalancing is disabled
Inbound access list is 1
Outbound access list is 3
Hardware idb is Fddi9/0/0
Fast switching type 2, interface type 8
IP Distributed CEF switching enabled
Fast flags 0x5. ifindex 25(25)
Slot 9 Slot unit 0 VC -1
Hardware transmit queue ptr 0x48001B80
(0x48001B80)
Transmit limit accumulator 0x48001B82
(0x48001B82)
IP MTU 4352
center#show cef linecard 9
CEF linecard slot number 9, status up, sync
Linecard CEF version number 246
Sequence number 1364, Maximum sequence number
expected 1388
Send failed 0, Out Of Sequence 0
Linecard CEF reset 1, reloaded 1
190/10761/178 prefix/adjacency/interface elements
queued
11022 elements packed in 1365 messages(235456 bytes)
sent
0/0 xdr elements in LowQ/HighQ
Input packets 7050739, bytes 2838181523
Output packets 7042682, bytes 2799387214, drops 0
378
Show Outputs (cont.)
VIP-Slot5#show ip cef sum
IP Distributed CEF with switching (Table Version 223)
223 routes, 0 unresolved routes (0 old, 0 new)
223 leaves, 168 nodes, 199464 bytes, 724 inserts, 501 invalidations
0 load sharing elements, 0 bytes, 0 references
4 CEF resets, 0 revisions of existing leaves
refcounts: 36926 leaf, 36706 node
VIP-Slot5#show ip cache
IP radix m-way trie cache 0 entries, 0 tables
0 unique encapsulations, 0 total bytes used by cache
7 adds, 7 invalidates, 46024768 switched, 36807 misses
misses: 0 frags, 0 runts, 0 chksm, 0 access
0 badiplen
Prefix/Length
Age
Interface
MAC Header
IP Flow Cache, 0 active, 512 alloced, 0 opened
0 closed: 0 tcp-fin, 0 idle-time, 0 prefix-inval
0 create failures, 0 switched
VIP-Slot5#show VIP memd
MEMD statistics :
0 local-switched
misses: 0 tqls, 0 bhs, 0 txaccs, 0 deferred
379
Debug Commands
• debug ip cef [drop] | [receive] |
[events] | [prefix-ipc] | [table] | [ipc]
| [interface-ipc]
• debug ip [packet] [error]
• debug adjacency
380
Debug Outputs
center#debug ip cef table
Nov 13 12:31:21.075: CEF-IP: Receive address 99.0.0.0/32 already exists
Nov 13 12:31:21.075: CEF-IP: Receive address 99.255.255.255/32 already exists
Nov 13 12:31:21.075: CEF-Table: Event up, 2.0.0.0/8
Nov 13 12:31:21.075: CEF-Table: Event up, 197.12.11.0/24
Nov 13 12:31:21.087: CEF-Table: Event up, 197.21.251.0/24
Nov 13 12:31:21.087: CEF-Table: Event up, 192.18.2.0/24
Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 1 Seq 0
Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 1 Seq 0
Nov 13 12:31:21.111: CEF-IPC: ipc sent. Slot 4 Seq 0
Nov 13 12:31:36.067:
Nov 13 12:31:36.067:
Nov 13 12:31:36.067:
Nov 13 12:31:36.067:
Nov 13 12:31:57.695:
Nov 13 12:31:57.695:
CEF-Table: attempting to resolve 197.12.12.0/24
CEF-IP: resolved 197.12.12.0/24 via 99.0.0.1 to 99.0.0.1 Loopback1
CEF-Table: attempting to resolve 197.12.11.0/24
CEF-IP: resolved 197.12.11.0/24 via 99.0.0.1 to 99.0.0.1 Loopback1
CEF-IP: Checking dependencies of 12.0.0.0/8
CEF-Table: Adjacency-prefix 12.0.0.1/32 add request -- succeeded
Nov 13 12:33:03.079:
Nov 13 12:33:03.079:
Nov 13 12:33:03.079:
Nov 13 12:33:03.079:
CEF-Table:
CEF-Table:
CEF-Table:
CEF-Table:
Flushing
Flushing
Flushing
Flushing
entry for 195.27.191.0/24
entry for 195.31.54.0/24
entry for 195.181.32.0/24
entry for 197.12.11.0/24
381
Debug Output (cont.)
VIP-Slot9#debug ip pack
IP-DCEF: Try to DCEF switch 196.2.3.1 from Fddi0/0
IP-DCEF: DCEF switched 196.2.3.1 to FastEthernet8/1/0
IP-DCEF: Try to DCEF switch 196.2.3.1 from Fddi0/0
IP-DCEF: DCEF switched 196.2.3.1 to FastEthernet8/1/0
border2#debug adjacency
ADJ: add 16.2.0.2 (Ethernet0/0) via ARP for 01:21:15
ADJ: add 16.2.0.1 (Ethernet0/0) via ARP for 01:20:37
ADJ: add 62.0.0.1 (Ethernet0/1) via ARP for 01:20:32
ADJ: add 66.0.0.1 (Ethernet0/2) via ARP for 01:22:13
ADJ: add 161.2.1.50 (Fddi1/0) via ARP for 03:45:59
ADJ: add 180.2.0.2 (FastEthernet8/0/0) via ARP for 01:22:13
ADJ: add 200.2.0.2 (Fddi10/0/0) via ARP for 01:23:07
ADJ: add 16.2.146.97 (Ethernet0/0) via ARP for 01:23:07
ADJ: add 0.0.0.0 (ATM 5/0/0.100) via ATM-PVC for 00:00:00
ADJ: add 0.0.0.0 (ATM 5/0/0.103) via ATM-PVC for 00:00:00
ADJ: add 0.0.0.0 (Serial4/0/0) via FIB for 00:02:59
ADJ: add 0.0.0.0 (Serial4/0/1) via FIB for 00:03:00
VIP-Slot9#debug ip cef drop
IP-CEF: No route found for
34.1.1.1
382
Troubleshooting
IP Multicast
0981_03F8_c3
NW98_US_112
Agenda
Troubleshooting Tools
Basic Troubleshooting
Advanced Troubleshooting
Case studies
384
Troubleshooter’s “Hand” Tools
“show
“show
“show
“show
“show
“show
ip igmp group” command
ip igmp interface” command
ip pim neighbor” command
ip pim interface” command
ip rpf ” command
ip mroute” commands
385
Troubleshooter’s “Hand” Tools
Special Sparse Mode Tools
“show ip pim rp” command
“show ip pim rp map” command
386
show ip igmp group
Shows:
Currently joined multicast groups.
Troubleshooting usage:
Verify that a receiver has actually joined
the target group
If not, use “show ip igmp interface” to check for
proper igmp version, querier, timers, etc.
Use “debug ip igmp” to verify that proper igmp
host-router exchange is happening
Watch for IGMP v1-v2 interoperability problems
387
show ip igmp group
R4#show ip igmp groups
IGMP Connected Group Membership
Group Address
Interface
224.1.1.1
Ethernet1
224.0.1.40
Ethernet0
Uptime
3d16h
4d15h
Expires
00:01:59
never
Last Reporter
172.16.7.2
172.16.6.2
388
show ip igmp interface
Shows:
Key IGMP timers, status, etc.
Troubleshooting usage:
Verify that correct IGMP version is running
Verify that timers are set properly
Verify that correct router is IGMP Querier
If not, use “debug ip igmp” to determine
what’s wrong
389
show ip igmp interface
R4#show ip igmp interface
Ethernet1 is up, line protocol is up
Internet address is 172.16.7.1, subnet mask is 255.255.255.0
IGMP is enabled on interface
Current IGMP version is 2
CGMP is disabled on interface
IGMP query interval is 60 seconds
IGMP querier timeout is 120 seconds
IGMP max query response time is 10 seconds
Inbound IGMP access group is not set
Multicast routing is enabled on interface
Multicast TTL threshold is 0
Multicast designated router (DR) is 172.16.7.1 (this system)
IGMP querying router is 172.16.7.1 (this system)
No multicast groups joined
390
show ip pim neighbor
Shows:
PIM Neighbor Adjacencies
Troubleshooting usage:
Verify that all neighbors are up and using
proper mode
If not, check router configs and/or interface status
Use “debug ip pim” to observe PIM Query msg
exchange
391
show ip pim neighbor
R6#show ip pim neighbor
PIM Neighbor Table
Neighbor Address Interface
172.16.10.2
Serial0
172.16.11.2
Serial1
172.16.9.1
Ethernet0
Uptime
4d15h
4d15h
4d15h
Expires
00:01:19
00:01:00
00:01:00
Mode
Dense
Dense
Dense
392
show ip pim interface
Shows:
PIM Interface information.
Mode, Neighbor Count, DR
Troubleshooting usage:
Verify correct PIM mode is configured
on interface(s)
If not, check router configs
Verify Designated Router is correct
If not, check router configs
Especially critical for Sparse Mode!
393
show ip pim interface
R6#show ip pim interface
Address
Interface
Mode
172.16.10.1
172.16.11.1
172.16.9.2
Dense
Dense
Dense
Serial0
Serial1
Ethernet0
Nbr
Count
1
1
1
Query
Intvl
30
30
30
DR
0.0.0.0
0.0.0.0
172.16.9.2
394
show ip rpf
Shows:
RPF interface information for source
Troubleshooting usage:
Verify that RPF information is correct
If not, check unicast routing data for correctness
Ping or Trace “source” to verify unicast route is
working. (Fix any unicast routing problems first!)
May need to use DVMRP routes or Static Mroutes
to fix unicast-multicast incongruency
395
show ip rpf
R4#show ip rpf 172.16.8.1
RPF information for Source1 (172.16.8.1)
RPF interface: Ethernet0
RPF neighbor: R3 (172.16.6.1)
RPF route/mask: 172.16.8.0/255.255.255.0
RPF type: unicast
R4#sh ip rpf 172.16.12.2
RPF information for Source2 (172.16.12.2)
RPF interface: Tunnel0
RPF neighbor: R6 (172.16.11.1)
RPF route/mask: 172.16.12.0/255.255.255.0
RPF type: DVMRP
396
show ip mroute commands
“show
“show
“show
“show
ip mroute sum”
ip mroute count”
ip mroute active”
ip mroute”
397
show ip mroute summary
Shows:
Multicast state at a glance
Active groups
Active senders in the group. (If SPT joined)
Troubleshooting usage:
Verify multicast group(s) are active.
If not, check for group state at RP. (Sparse mode)
Work your way from a known source to a receiver
or the RP to find where things stop
Verify senders are active. (If SPT joined)
If not, check state in 1st-hop router
Verify sender is really sending
398
show ip mroute summary
dallas-gw>show ip mroute summary
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
Timers: Uptime/Expires
Interface state: Interface, Next-Hop, State/Mode
(*, 224.0.255.255), 6d23h/00:02:59, RP 171.69.10.13, flags: SJC
(171.68.37.121/32, 224.0.255.255), 6d23h/00:02:55, flags: CT
(171.69.58.88/32, 224.0.255.255), 6d23h/00:02:58, flags: CT
(171.69.60.189/32, 224.0.255.255), 6d23h/00:02:55, flags: CT
(171.69.128.115/32, 224.0.255.255), 3d01h/00:02:58, flags: CJT
(171.69.199.49/32, 224.0.255.255), 6d23h/00:02:57, flags: CT
(171.70.247.82/32, 224.0.255.255), 01:55:33/00:02:58, flags: CJT
...
399
show ip mroute count
Shows:
Multicast traffic flow rates, drops, etc.
Group traffic summary
Sender rates, packet counts, drops, etc.
Troubleshooting usage:
Verify multicast traffic is being received
If not, work your way from source to receiver to
find where things stop
Verify multicast traffic is being forwarded
If not, why? “oif null”, “rpf-failure”
400
show ip mroute count
dallas-gw>show ip mroute 224.0.255.255 count
IP Multicast Statistics - Group count: 7, Average sources per group: 3.28
Forwarding Counts: Pkt Count/Pkts per second/Avg Pkt Size/Kilobits per second
Other counts: Total received/RPF failed/Other drops(OIF-null, rate-limit etc)
Group: 224.0.255.255, Source count: 6, Group pkt count: 538169
RP-tree: Forwarding: 0/0/0/0, Other: 0/0/0
Source: 171.68.37.121/32, Forwarding: 120484/0/94/0, Other: 120535/0/51
Source: 171.69.58.88/32, Forwarding: 120281/0/84/0, Other: 120283/2/0
Source: 171.69.60.189/32, Forwarding: 120445/0/95/0, Other: 120447/2/0
Source: 171.69.128.115/32, Forwarding: 53018/0/93/0, Other: 53018/0/0
Source: 171.69.199.49/32, Forwarding: 120414/1/92/0, Other: 120415/1/0
Source: 171.70.247.82/32, Forwarding: 3527/1/78/0, Other: 3527/0/0
401
show ip mroute active
Shows:
Sources with traffic rates above threshold
Aggregate RP Tree and (S, G) rates shown
Rates in Kbps (1 sec, 1 min, 5 min avgs.)
Troubleshooting usage:
Determine which sources/groups are active
Determine the traffic rate of each source
Note: Must have switched to Shortest-Path tree
Verify “target” group multicast traffic is
being received
If not, work your way from source to receiver
402
show ip mroute active
barrnet-gw>show ip mroute active
Active IP Multicast Sources - sending >= 4 kbps
Group: 224.2.156.43, *cisco: Bloomington IPTV Beacon
Source: 172.17.67.43 (bloom-iptv.cisco.com)
Rate: 6 pps/63 kbps(1sec), 65 kbps(last 19 secs), 37 kbps(life avg)
Group: 224.2.154.118, Radio Bandit
Source: 192.36.125.68 (falcon.pilsnet.sunet.se)
Rate: 11 pps/30 kbps(1sec), 30 kbps(last 33 secs), 23 kbps(life avg)
Group: 224.2.246.13, UO Presents KWAX Classical Radio
Source: 128.223.83.204 (d83-204.uoregon.edu)
Rate: 24 pps/69 kbps(1sec), 72 kbps(last 2 secs), 70 kbps(life avg)
Group: 224.2.180.115, ANL TelePresence Microscopy Site
Source: 146.139.72.5 (aem005.amc.anl.gov)
Rate: 1 pps/5 kbps(1sec), 9 kbps(last 52 secs), 12 kbps(life avg)
...
403
show ip mroute
Shows:
Detailed multicast state in the router
Troubleshooting usage:
Verify Incoming Interface is correct
If not, check unicast routing table (May need to
use DVMRP routes or Static mroutes)
Verify Outgoing Interface(s) are correct
If interface incorrectly “Pruned”, check state in
downstream router?
May need to “debug ip pim <group>” to
determine problem
404
show ip mroute
barrnet-gw>show ip mroute
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join
SPT
Timers: Uptime/Expires
Interface state: Interface, Next-Hop, State/Mode
(*, 224.2.130.100), 00:18:53/00:02:59, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Fddi1/0, Forward/Dense, 00:09:20/00:02:38
Hssi3/0, Forward/Dense, 00:18:53/00:00:00
(208.197.169.209/32, 224.2.130.100), 00:18:53/00:02:27, flags: T
Incoming interface: Hssi3/0, RPF nbr 131.119.26.9
Outgoing interface list:
Fddi1/0, Forward/Dense, 00:16:16/00:02:38
(*, 239.100.111.224), 05:35:08/00:02:58, RP 171.69.10.13, flags: DP
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list: Null
405
show ip pim rp map
Shows:
RP assignments by multicast group range
Troubleshooting usage:
Verify that configured (static or Auto-RP)
RP’s are correct
If not, check local router config
and/or network Auto-RP configuration
406
show ip pim rp map
dallas-gw>show ip pim rp map
PIM Group-to-RP Mappings
Group(s) 224.0.0.0/4, uptime: 6d21h, expires: 00:02:56
RP 171.69.10.13 (sj-eng-mbone.cisco.com)
Info source: 192.31.7.37 (barrnet-gw.cisco.com)
407
show ip pim rp
Shows:
RP’s by active group
Troubleshooting usage:
Verify that the RP for the target multicast
group is correct
If not, check RP mapping, local router RP
config and/or network Auto-RP
408
show ip pim rp
dallas-gw>sh ip pim rp
Group: 224.2.127.253, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34
Group: 224.1.127.255, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34
Group: 224.2.127.254, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34
Group: 224.0.255.255, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34
Group: 224.2.0.1, RP: 171.69.10.13, uptime 6d21h, expires 00:01:34
409
Troubleshooter’s “Power” Tools
“mtrace” and “mstat” commands
“mrinfo” command
“show ip mpacket” command
410
mtrace and mstat commands
Based on Unix “mtrace” command
Split into two separate commands
Both use the same mechanism
draft-ietf-idmr-traceroute-ipmxx.txt
411
mtrace
Shows:
Multicast path from source to receiver
Similar to unicast “trace” command
Trace path between any two points in network
TTL Thresholds and Delay shown at each node
Troubleshooting usage:
Find where multicast traffic flow stops
Focus on router where flow stops
Verify path multicast traffic is following
Identify sub-optimal paths
412
mstat
Shows:
Multicast path in pseudo graphic format
Trace path between any two points in network
Drops/Duplicates shown at each node
TTLs and Delay shown at each node
Troubleshooting usage:
Locate congestion point in the flow
Focus on router with high drop/duplicate count
Duplicates indicated as “negative” drops
413
mtrace/mstat—How it Works
Mtrace Packet Flow
Adds
mtrace
data
Adds
mtrace
data
Adds
mtrace
data
src
Adds
mtrace
data
Adds
mtrace
data
dest
First-hop
Router
Last-hop
Router
Multicast
Dist. Tree
Mtrace Packet
Note: Mtrace packets use special
IGMP packets with IGMP Type
codes of 0x1E and 0x1F.
Unix Workstation
or
Cisco Router
414
mtrace/mstat—How it Works
Each hop adds data to packet
Query arrival time
Incoming Interface
Outgoing Interface
Prev. Hop Router address
Input packet count
Output packet count
Total packets for this Source/Group
Routing Protocol
TTL Threshold
Fowarding/Error Code
415
mtrace
dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43
Type escape sequence to abort.
Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43
From source (?) to destination (bwilliam-ss5.cisco.com)
Querying full reverse path...
0 bwilliam-ss5 (171.68.37.121)
-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms
-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms
-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms
-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms
-5 bloom-iptv-svr (172.17.67.43)
dallas-gw>
416
mstat
dallas-gw>mstat bloom-iptv-svr bwilliam-ss5 224.2.156.43
Source
Response Dest
Packet Statistics For
Only For Traffic
172.17.67.43
171.68.86.194
All Multicast Traffic
From 172.17.67.43
|
__/ rtt 547 ms Lost/Sent = Pct Rate
To 224.2.156.43
v
/
hop 547 ms
---------------------------------------172.17.67.33
171.68.39.28
bloom-mnlab
|
^
ttl
0
v
|
hop -409 ms
-11/168 = --% 16 pps
0/67 = 0% 6 pps
171.68.39.1
171.68.27.2
bloomington-mn-gw
|
^
ttl
1
v
|
hop 379 ms
-9/170 = --%
17 pps
-3/67 = --% 6 pps
171.68.27.1
171.68.86.193
wan-gw4
|
^
ttl
2
v
|
hop 28
ms
-3/195 = --%
19 pps
0/70 = 0%
7 pps
171.68.86.194
171.68.37.1
dallas-gw
|
\__
ttl
3
v
\ hop 0
ms
196
19 pps
70
7 pps
171.68.37.121
171.68.86.194
Receiver
Query Source
0981_03F8_c3
NW98_US_112
417
35
mstat
dallas-gw>mstat bloom-iptv-svr bwilliam-ss5 224.2.156.43
Source
Response Dest
Packet Statistics For
Only For Traffic
172.17.67.43
171.68.86.194
All Multicast Traffic
From 172.17.67.43
|
__/ rtt 399 ms Lost/Sent = Pct Rate
To 224.2.156.43
v
/
hop 399 ms
---------------------------------------172.17.67.33
171.68.39.28
bloom-mnlab
|
^
ttl
0
v
|
hop 119 ms
77/694 = 11% 69 pps
0/65 = 0%
6 pps
171.68.39.1
171.68.27.2
bloomington-mn-gw
|
^
ttl
1
v
|
hop -150 ms
395/609 = 65% 60 pps
44/65 = 68% 6 pps
171.68.27.1
171.68.86.193
wan-gw4
|
^
ttl
2
v
|
hop 30
ms
-8/39 = --%
3 pps
-1/21 = --% 2 pps
171.68.86.194
171.68.37.1
dallas-gw
|
\__
ttl
3
v
\ hop 0
ms
39
3 pps
22
2 pps
171.68.37.121
171.68.86.194
Receiver
Query Source
0981_03F8_c3
NW98_US_112
418
36
mrinfo
Shows:
Multicast neighbor router information
Indicates router’s capabilities and code version
Multicast interface information
TTL-Thresholds, Metric, Protocol, Status
Troubleshooting usage:
Verify multicast neighbors.
Confirm bi-directional neighbor adjacency exists
Verify Tunnels are up in both directions
419
mrinfo
dallas-gw>mrinfo paloalto-mbone1.bbnplanet.net
Translating " paloalto-mbone1.bbnplanet.net "...domain server (171.68.10.70) [OK]
131.119.0.197 (paloalto-mbone1.bbnplanet.net) [version cisco 11.2] [flags: PMSA]:
131.119.0.197 -> 131.119.0.201 (paloalto-cr1.bbnplanet.net) [1/0/pim]
131.119.244.244 -> 0.0.0.0 [1/32/pim/querier]
131.119.0.197 -> 204.162.119.8 (hydra.precept.com) [1/32/tunnel/querier]
192.42.110.249 -> 192.9.9.71 (mbone.Sun.COM) [1/32/tunnel]
192.42.110.249 -> 204.123.13.69 (chocolate.research.digital.com) [1/32/tunnel]
192.42.110.249 -> 36.253.0.11 (alpo.Stanford.EDU) [1/32/tunnel]
131.119.0.197 -> 0.0.0.0 [1/64/tunnel/pim/querier/leaf]
131.119.0.197 -> 0.0.0.0 [1/32/tunnel/pim/querier/leaf]
192.42.110.249 -> 204.94.211.39 (sgi-too.SGI.COM) [4/64/tunnel/querier]
192.42.110.249 -> 192.216.174.1 [1/32/tunnel/querier/down/leaf]
192.42.110.249 -> 198.94.216.2 [1/32/tunnel/querier/down/leaf]
192.42.110.249 -> 204.161.60.33 (berkeley.faslab.com) [1/32/tunnel/querier]
131.119.0.197 -> 204.154.181.12 [1/32/tunnel/querier/down/leaf]
...
420
show ip mpacket
Used to view multicast packet headers
Command syntax
show ip mpacket <source> <group> [detail]
You can view:
{source, group} traffic pairs
IP ident and ttl
Inter-packet delay
Configure multicast header capture first
“ip multicast cache-headers” config cmd
Captures multicast headers in 1024 entry ring buffer
421
show ip mpacket
dino-cisco-fr#show ip mpacket 224.2.231.173
IP Multicast Header Cache - entry count: 29, next index: 30
Key: id/ttl timestamp (name) source group
D782/117
7302/113
6CB2/114
D786/117
E2E9/123
1CA7/127
1CAA/127
1CAC/127
1CAF/127
1CB0/127
1CB2/127
2BBB/114
3D1D/123
2BC0/114
7303/113
7304/113
2C7E/123
206416.908
206417.172
206417.412
206417.868
206418.488
206418.544
206418.584
206418.624
206418.664
206418.704
206418.744
206418.840
206419.380
206419.672
206419.888
206420.140
206420.360
(all-purpose-gunk.near.net) 199.94.220.184 224.2.231.173
(speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.173
(wayback.uoregon.edu) 128.223.156.117 224.2.231.173
(all-purpose-gunk.near.net) 199.94.220.184 224.2.231.173
(dino-ss20.cisco.com) 171.69.58.81 224.2.231.173
(dino-ss2.cisco.com) 171.69.129.220 224.2.231.173
(dino-ss2.cisco.com) 171.69.129.220 224.2.231.173
(dino-ss2.cisco.com) 171.69.129.220 224.2.231.173
(dino-ss2.cisco.com) 171.69.129.220 224.2.231.173
(dino-ss2.cisco.com) 171.69.129.220 224.2.231.173
(dino-ss2.cisco.com) 171.69.129.220 224.2.231.173
(crevenia.parc.xerox.com) 13.2.116.11 224.2.231.173
(dalvarez-ss20.cisco.com) 171.69.60.189 224.2.231.173
(crevenia.parc.xerox.com) 13.2.116.11 224.2.231.173
(speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.173
(speedy.rrz.uni-koeln.de) 134.95.19.23 224.2.231.173
(lwei-ss20.cisco.com) 171.69.58.88 224.2.231.173
422
Agenda
Troubleshooting Tools
Basic Troubleshooting
Advanced Troubleshooting
Case studies
423
Basic Troubleshooting
Troubleshooting Table
Source
Network
Receivers
State
NA
?
?
Packet Flow
?
?
?
Is each piece working correctly?
424
Check Source Packet Flow
Check interface counters on source
Check source TTL > 1
Verify TTL setting in application
Confirm on upstream router
show ip traffic
Increasing “Invalid hop count”
425
Check Source Packet Flow
Check 1st-Hop router for traffic flow
show ip mroute count
show ip mroute active
show ip mpacket
Don’t forget to turn on:
ip multicast cache-headers
debug ip mpacket
Use with caution!!
“detail” or ACL for granularity
426
Check Network State
Most complex piece
Depends of protocol, mode, etc.
Check initial state creation
Check for pruning and timer expiration
during session
427
Network State
show/debug ip mroute commands
watch oilist for null entries
show/debug ip pim commands
show/debug ip dvmrp commands
show ip rpf
mtrace command
428
PIM SM Troubleshooting
show ip pim rp [<group>]
show ip pim rp mapping
indicates RP for the group
indicates RP for the group
debug ip pim auto-rp
429
Check Network Packet Flow
show ip mroute count
show ip mroute active
show ip mpacket
debug ip mpacket
Turn on “ip multicast cache-headers”
first
Be Careful with this one!
mstat
430
Check Receiver State
show ip igmp interface
show ip igmp group
debug ip igmp
IGMPv1 vs. IGMPv2
431
Check Receiver Packet Flow
Check receiver interface stats
Is the stack installed and configured
properly?
Is the application installed and configured
properly?
Watch for duplicates
Performance implication
432
Agenda
Troubleshooting Tools
Basic Troubleshooting
Advanced Troubleshooting
Case studies
433
Advanced Troubleshooting
Troubleshooting Network State
Troubleshooting PIM-DVMRP
Troubleshooting ATM P2MP VCs
434
Troubleshooting Network State
Use “show ip mroute <group>”
Specify target group to limit output.
DM: Trace state from source to receiver.
SM: Trace state from receiver to RP then from source
to RP
Use “debug ip pim <group>”
Be sure to specify target group to limit debug output!
Use with caution!!
Use “debug ip mroute <group>”
Be sure to specify target group to limit debug output!
Use with caution!!
435
Mroute Flags
“S”—Sparse Mode
“D”—Dense Mode
Appears only on the (*, G) entries
Appears only on the (*, G) entries
“P”—Pruned
Sparse mode: oilist is null
Dense mode: all interfaces in oilist =
Pruned
436
Mroute Flags (Cont.)
“C”—Connected
“L”—Local
A rcvr for this group is directly connected
to this router.
The router itself is a member of this group
and is receiving group traffic.
“T”—Shortest-path Tree (SPT) flag
Set on (S,G) entry when packets are being
successfully received via the SPT
437
Mroute Flags (Sparse Mode Only)
“R”—RP bit
Only appears on (S, G) entries
(S, G) state is associated with Shared Tree
RPF interface points up Shared Tree to RP
Used to prune unwanted (S, G) traffic from
the Shared Tree after SPT switchover
“F”—Register Flag
Appears on (S, G) entries
Set on (*, G) if any (S, G) “F” flags set
Router is directly connected to a source
Register messages must be sent to RP
438
Mroute Flags (Sparse Mode Only)
(*, G) Entry
“J” —Join SPT
Set once/sec when SPT-Threshold exceeded
Switch to SPT for next (S, G) packet rcvd’d
(S, G) Entry
“J” —SPT Joined
SPT Joined due to SPT-Threshold exceeded.
Switch back to Shared Tree if traffic rate falls
below SPT-Threshold. (Checked once/min)
439
Network Mroute State Examples
PIM DM
PIM SM
Joining
Registering
SPT-Switchover
440
PIM DM
S1
S0
1
Multicast Packets
(128.9.160.43, 224.2.127.254)
S3
rtr-a
S0
rtr-b
E1
1
“rtr-a” initially floods (S, G) traffic out all interfaces in “oilist”.
441
PIM DM
S1
S0
Multicast Packets
(128.9.160.43, 224.2.127.254)
Initial “Flooding” State
in “rtr-a”
S3
rtr-a
S0
rtr-b
E1
(*, 224.2.127.254), 00:00:10/00:00:00, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Serial1, Forward/Dense, 00:00:10/00:00:00
Serial3, Forward/Dense, 00:00:10/00:00:00
(128.9.160.43/32, 224.2.127.254), 00:00:10/00:02:49, flags: T
Incoming interface: Serial0, RPF nbr 198.92.1.129
Outgoing interface list:
Serial1, Forward/Dense, 00:00:10/00:00:00
Serial3, Forward/Dense, 00:00:10/00:00:00
442
PIM DM
S1
S0
Multicast Packets
(128.9.160.43, 224.2.127.254)
S3
rtr-a
3
2 Prune
S0
rtr-b
E1
1
“rtr-a” initially floods (S, G) traffic out all interfaces in “oilist”
2
“rtr-b” is a leaf node w/o receivers. Sends Prune for (S,G)
3
“rtr-a” Prunes interface for (S,G)
443
PIM DM
S1
S0
Multicast Packets
(128.9.160.43, 224.2.127.254)
S3
rtr-a
S0
State in “rtr-a”
after Pruning
rtr-b
E1
(*, 224.2.127.254), 00:00:12/00:00:00, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Serial1, Forward/Dense, 00:00:12/00:00:00
Serial3, Forward/Dense, 00:00:12/00:00:00
(128.9.160.43/32, 224.2.127.254), 00:00:12/00:02:48, flags: T
Incoming interface: Serial0, RPF nbr 198.92.1.129
Outgoing interface list:
Serial1, Forward/Dense, 00:00:12/00:00:00
Serial3, Prune/Dense, 00:00:04/00:02:56
444
PIM DM
S1
S0
Multicast Packets
(128.9.160.43, 224.2.127.254)
S3
rtr-a
S0
State in “rtr-b”
after Pruning
rtr-b
E1
(*, 224.2.127.254), 00:00:12/00:00:00, RP 0.0.0.0, flags: D
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
Serial0, Forward/Dense, 00:00:12/00:00:00
(128.9.160.43/32, 224.2.127.254), 00:00:12/00:02:48, flags: PT
Incoming interface: Serial0, RPF nbr 198.92.1.129
Outgoing interface list: Null
445
PIM SM Joining
4 Shared Tree
S1
To RP (10.1.5.1)
3 PIM Join
S0
10.1.4.2
Shared Tree
10.1.2.2
1
IGMP Join
E1
E0
rtr-a
E010.1.2.1
2 PIM Join
rtr-b
Rcvr A
1•
“Rcvr A” wishes to receive group G traffic. Sends IGMP Join for G
2
•
“rtr-b” creates (*,G) state; sends (*,G) PIM Join towards RP
3
•
“rtr-a” creates (*,G) state; sends (*,G) PIM Join towards RP
4
•
Shared tree is built all the way back to the RP
446
PIM SM Joining
S1
To RP (10.1.5.1)
rtr-a
S0
10.1.4.2
E0
10.1.2.1
Shared Tree
10.1.2.2
E1
E0
rtr-b
Rcvr A
rtr-b>sh ip mroute
(*, 224.1.1.1), 00:00:05/00:02:54, RP 10.1.5.1, flags: SC
Incoming interface: Ethernet0, RPF nbr 10.1.2.1
Outgoing interface list:
Ethernet1, Forward/Sparse, 00:00:05/00:02:52
State in “rtr-b” after Joining (*, 224.1.1.1)
447
PIM SM Joining
S1
To RP (10.1.5.1)
S0
10.1.4.2
Shared Tree
10.1.2.2
E1
rtr-a
E010.1.2.1
E0
rtr-b
Rcvr A
rtr-a>sh ip mroute
(*, 224.1.1.1), 00:00:05/00:02:54, RP 10.1.5.1, flags: S
Incoming interface: Serial0, RPF nbr 10.1.4.1
Outgoing interface list:
Ethernet0, Forward/Sparse, 00:00:05/00:02:54
State in “rtr-a” after Joining (*, 224.1.1.1)
448
PIM SM Registering
171.68.28.139
RP
S3
rtr-a
rtr-b
rtr-c
S0
S1
Shared Tree
rtr-c>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:00:03/00:02:56, RP 171.68.28.140, flags: S
Incoming interface: Null, RPF nbr 0.0.0.0,
Outgoing interface list:
Serial0, Forward/Sparse, 00:03:14/00:02:59
Serial1, Forward/Sparse, 00:03:14/00:02:59
State in “RP” before Registering
(with receivers on Shared Tree)
449
PIM SM Registering
RP
S0
E0
rtr-a
rtr-b
rtr-c
Shared Tree
rtr-a>sh ip mroute 224.1.1.1
No such group.
State in “rtr-a” before Registering
(with receivers on Shared Tree)
450
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
2
Register Msgs
RP
1
Source
171.68.37.121
rtr-a
rtr-b
rtr-c
3 (*, 224.1.1.1)
Mcast Traffic
Shared Tree
1•
“Source” begins sending group G traffic
2
•
“rtr-a” encapsulates packets in Registers; unicasts to RP
3
•
“rtr-c” (RP) de-encapsulates packets; forwards down Shared tree
451
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
Register Msgs
RP
S0
Source
171.68.37.121
E0
rtr-a
rtr-b
rtr-c
(*, 224.1.1.1)
Mcast Traffic
Shared Tree
rtr-a>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:00:03/00:02:56, RP 171.68.28.140, flags: SP
Incoming interface: Serial0, RPF nbr 171.68.28.191,
Outgoing interface list: Null
(171.68.37.121/32, 224.1.1.1), 00:00:03/00:02:56, flags: FPT
Incoming interface: Ethernet0, RPF nbr 0.0.0.0, Registering
Outgoing interface list: Null
State in “rtr-a” while Registering
452
70
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
Register Msgs
RP
Source
171.68.37.121
rtr-a
Join
rtr-b
Join
rtr-c
4
(*, 224.1.1.1)
Mcast Traffic
Shared Tree
1•
“Source” begins sending group G traffic
2
•
“rtr-a” encapsulates packets in Registers; unicasts to RP
3
•
RP (“rtr-c”) de-encapsulates packets; forwards down Shared tree
4
•
RP sends (S,G) Join toward Source; builds SPT
453
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
Register Msgs
5
Source
171.68.37.121
RP
rtr-a
rtr-b
6
rtr-c
(*, 224.1.1.1)
Mcast Traffic
Register-Stop
5•
RP begins receiving (S,G) traffic down SPT
6
•
RP sends “Register-Stop” to “rtr-a”
Shared Tree
454
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
8
Source
171.68.37.121
rtr-a
RP
rtr-b
rtr-c
(*, 224.1.1.1)
Mcast Traffic
Shared Tree
5•
RP begins receiving (S,G) traffic down SPT
6
•
RP sends “Register-Stop” to “rtr-a”
7
•
8
•
“rtr-a” stops encapsulating traffic in Register Messages
(S,G) Traffic now flowing down a single path (SPT) to RP
455
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
RP
S0
Source
171.68.37.121
E0
rtr-a
rtr-b
rtr-c
(*, 224.1.1.1)
Mcast Traffic
Shared Tree
rtr-a>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: SP
Incoming interface: Serial0, RPF nbr 171.68.28.191,
Outgoing interface list: Null
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: FT
Incoming interface: Ethernet0, RPF nbr 0.0.0.0
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
State in “rtr-a” after Registering
456
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
RP
Source
171.68.37.121
S0
rtr-a
171.68.28.190
S1
rtr-b
rtr-c
(*, 224.1.1.1)
Mcast Traffic
Shared Tree
rtr-b>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: SP
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list: Null
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T
Incoming interface: Serial0, RPF nbr 171.68.28.190
Outgoing interface list:
Serial1, Forward/Sparse, 00:04:28/00:01:32
State in “rtr-b” after “rtr-a” Registers
(with receivers on Shared Tree)
457
PIM SM Registering
(171.68.37.121, 224.1.1.1)
Mcast Packets
171.68.28.139
RP
Source
171.68.37.121
S3
rtr-a
rtr-b
rtr-c
S0
S1
(*, 224.1.1.1)
Mcast Traffic
Shared Tree
rtr-c>sh ip mroute 224.1.1.1
(*, 224.1.1.1), 00:09:21/00:02:38, RP 171.68.28.140, flags: S
Incoming interface: Null, RPF nbr 0.0.0.0,
Outgoing interface list:
Serial0, Forward/Sparse, 00:09:21/00:02:38
Serial1, Forward/Sparse, 00:03:14/00:02:46
(171.68.37.121, 224.1.1.1, 00:01:15/00:02:46, flags: T
Incoming interface: Serial3, RPF nbr 171.68.28.139,
Outgoing interface list:
Serial0, Forward/Sparse, 00:00:49/00:02:11
Serial1, Forward/Sparse, 00:00:49/00:02:11
State in “RP” after “rtr-a” Registers
(with receivers on Shared Tree)
458
PIM SM SPT-Switchover Review
SPT-Switchover Mechanism
Once each second
Compute new (*, G) traffic rate
If threshold exceeded, set “J” flag in (*, G)
For each (Si , G) packet received:
If “J” flag set in (*, G)
Clear “J” flag in (*,G)
Join SPT for (Si , G)
Mark (Si , G) entry with “J” flag
459
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.4.2
10.1.2.2
E1
To Source “Si”
rtr-a
E010.1.2.1
E0
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S
Incoming interface: Serial0, RPF nbr 10.1.5.1,
Outgoing interface list:
Serial1, Forward/Sparse, 00:01:43/00:02:11
State in “rtr-c” before switch
460
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
S0
10.1.4.2
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.2.2
E1
To Source “Si”
rtr-a
E010.1.2.1
E0
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S
Incoming interface: Serial0, RPF nbr 10.1.4.1,
Outgoing interface list:
Ethernet0, Forward/Sparse, 00:01:43/00:02:11
State in “rtr-a” before switch
461
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
rtr-a
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.4.2
To Source “Si”
E0
10.1.2.1
10.1.2.2
E1
E0
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC
Incoming interface: Ethernet0, RPF nbr 10.1.4.2,
Outgoing interface list:
Ethernet1, Forward/Sparse, 00:01:43/00:02:11
State in “rtr-b” before switch
462
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.4.2
10.1.2.2
E1
E0
To Source “Si”
rtr-a
E010.1.2.1
1 Group “G” rate > Threshold
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC
Incoming interface: Ethernet0, RPF nbr 10.1.4.2,
Outgoing interface list:
Ethernet1, Forward/Sparse, 00:01:43/00:02:11
1
2
Group “G” rate exceeds SPT Threshold at “rtr-b”;
Set J Flag in (*, G) and wait for next (Si,G) packet
2
J
463
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
rtr-a
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
To Source “Si”
10.1.4.2
E0
10.1.2.1
10.1.2.2
E1
E0
3
5 (Si,G) Join
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SCJ
Incoming interface: Ethernet0, RPF nbr 10.1.4.2,
Outgoing interface list:
Ethernet1, Forward/Sparse, 00:01:43/00:02:11
1
2
3
4
4
SC
Group “G” rate exceeds SPT Threshold at “rtr-b”;
Set J Flag in (*, G) and wait for next (Si,G) packet.
(Si,G) packet arrives down Shared tree.
5 Send (S ,G) Join towards S .
Clear J Flag in (*,G) and
i
i
0981_03F8_c3
NW98_US_112
464
82
PIM SM SPT-Switchover
10.1.4.1
S0
(Si,G) Traffic
S1
To RP (10.1.5.1)
S1
rtr-c
(Si, G) Traffic Flow
Shared (RPT) Tree
(Si,G)RP-bit
Prune
SPT Tree
9
8
S0
10.1.4.2
10.1.2.2
E1
E0
rtr-a
To Source “Si”
7 (Si,G) Join
E010.1.2.1
6 (Si,G)RP-bit Prune
rtr-b
Rcvr A
6
Send (Si,G)RP-bit Prune toward RP to prune traffic from RPT
7
SPT & RPT diverge; “rtr-a” forwards (Si,G) Join toward Si
8
“rtr-a” forwards (Si,G)RP-bit Prune toward RP
9
(Si, G) traffic begins flowing down SPT tree
465
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
rtr-c
S1
10
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
rtr-a
S0
10.1.4.2
To Source “Si”
E0
10.1.2.1
10.1.2.2
E1
E0
rtr-b
Rcvr A
6
Send (Si,G)RP-bit Prune toward RP to prune traffic from RPT
7
SPT & RPT diverge; “rtr-a” forwards (Si,G) Join toward Si
8
“rtr-a” forwards (Si,G)RP-bit Prune toward RP
9
(Si, G) traffic begins flowing down SPT tree
10
•
(Si, G) traffic ceases flowing down Shared tree
466
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
rtr-a
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.4.2
To Source “Si”
E0
10.1.2.1
10.1.2.2
E1
E0
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S
Incoming interface: Serial0, RPF nbr 10.1.4.1,
Outgoing interface list:
Ethernet0, Forward/Sparse, 00:01:43/00:02:11
(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: T
Incoming interface: Serial1, RPF nbr 10.1.9.2
Outgoing interface list:
Ethernet0, Forward/Sparse, 00:13:25/00:02:30
State in “rtr-a” after switch
467
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
rtr-a
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.4.2
To Source “Si”
E0
10.1.2.1
10.1.2.2
E1
E0
rtr-b
Rcvr A
J Flag indicates
(S, G) created by
exceeding the
SPT-threshold
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: SC
Incoming interface: Ethernet0, RPF nbr 10.1.2.1,
Outgoing interface list:
Ethernet1, Forward/Sparse, 00:01:43/00:02:11
(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: SCJT
SCJT
Incoming interface: Ethernet0, RPF nbr 10.1.2.1
Outgoing interface list:
Ethernet1, Forward/Sparse, 00:13:28/00:02:53
State in “rtr-b” after switch
468
PIM SM SPT-Switchover
10.1.4.1
S0
S1
To RP (10.1.5.1)
S1
rtr-c
rtr-a
S0
(Si, G) Traffic Flow
Shared (RPT) Tree
SPT Tree
10.1.4.2
To Source “Si”
E0
10.1.2.1
10.1.2.2
E1
E0
rtr-b
Rcvr A
(*, 224.1.1.1), 00:01:43/00:02:13, RP 10.1.5.1, flags: S
Incoming interface: Serial0, RPF nbr 10.1.5.1,
Outgoing interface list:
Serial1, Forward/Sparse, 00:01:43/00:02:11
(171.68.37.121/32, 224.1.1.1), 00:13:28/00:02:53, flags: PR
Incoming interface: Serial0, RPF nbr 10.1.5.1
Outgoing interface list: Null
State in “rtr-c” after switch
469
Advanced Troubleshooting
Troubleshooting Network State
Troubleshooting PIM-DVMRP
Troubleshooting ATM P2MP VCs
470
PIM-DVMRP Troubleshooting
Example Network
pim-dvmrp-gw:
ISP
mrouted
Tunnel0
Site
pim-dvmrp-gw
Ethernet0
Ethernet1
135.1.2.100
interface tunnel0
ip unnumbered ethernet0
ip pim dense-mode
tunnel mode dvmrp
tunnel source ethernet0
tunnel destination 135.1.22.98
interface ethernet0
ip addr 135.1.3.102 255.255.255.0
ip pim dense-mode
interface ethernet1
ip addr 135.1.2.102 255.255.255.0
ip pim dense-mode
471
PIM-DVMRP Troubleshooting
Verifying the DVMRP tunnel
Verifying DVMRP route exchange
472
Verifying the DVMRP Tunnel
Using the “show interface” Command
pim-dvmrp-gw> show int tunnel 0
Tunnel0 is up, line protocol is up
Hardware is Tunnel
Interface is unnumbered. Using address of Ethernet0 (135.1.3.102)
MTU 1500 bytes, BW 9 Kbit, DLY 500000 usec, rely 255/255, load 1/255
Encapsulation TUNNEL, loopback not set, keepalive set (10 sec)
Tunnel source 135.1.3.102 (Ethernet0), destination 135.1.22.98
Tunnel protocol/transport IP/IP (DVMRP), key disabled, sequencing disabled
Checksumming of packets disabled, fast tunneling enabled
Last input 00:00:05, output 00:00:08, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0 (size/max/drops); Total output drops: 0
.
.
.
473
Verifying the DVMRP Tunnel
Using the “mrinfo” Command
pim-dvmrp-gw>mrinfo
135.1.3.102 [version cisco 11.2] [flags: PMA]:
135.1.3.102 -> 0.0.0.0 [1/0/pim/querier/leaf]
135.1.2.102 -> 135.1.2.2 [1/0/pim/querier]
135.1.2.102 -> 135.1.2.3 [1/0/pim/querier]
135.1.3.102 -> 135.1.22.98 [1/0/tunnel/querier]
pim-dvmrp-gw>mrinfo 135.1.22.98
135.1.22.98 [version mrouted 3.8] [flags: GPM]:
172.21.32.98 -> 172.21.32.191 [1/1]
172.21.32.98 -> 172.21.32.1 [1/1]
135.1.22.98 -> 135.1.22.102 [1/1/querier]
135.1.22.98 -> 135.1.3.102 [1/1/tunnel]
Both Ends See
Each Other
474
PIM-DVMRP Troubleshooting
Verifying the DVMRP tunnel
Verifying DVMRP route exchange
475
Verifying DVMRP Route Exchange
Using “show ip dvmrp route”
pim-dvmrp-gw# show ip dvmrp route
DVMRP Routing Table - 8 entries
130.1.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
135.1.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
135.1.22.0/24 [0/2] uptime 00:19:03, expires 00:02:13
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
171.69.0.0/16 [0/3] uptime 00:19:03, expires 00:02:13
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.27.0/24 [0/3] uptime 00:19:04, expires 00:02:12
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.32.0/24 [0/2] uptime 00:19:04, expires 00:02:12
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.33.0/24 [0/3] uptime 00:19:04, expires 00:02:12
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
172.21.120.0/24 [0/3] uptime 00:19:04, expires 00:02:12
via 135.1.22.98, Tunnel0, [version mrouted 3.8] [flags: GPM]
476
Verifying DVMRP Route
Exchange
Using “debug ip dvmrp”
pim-dvmrp-gw# debug ip dvmrp
DVMRP debugging is on
pim-dvmrp-gw#
Mar 20 11:39:36.335: DVMRP: Aging routes, 0 entries expired
Mar 20 11:39:41.271: DVMRP: Received Probe on Tunnel0 from 135.1.22.98
Mar 20 11:39:45.335: DVMRP: Building Report for Tunnel0 224.0.0.4
Mar 20 11:39:45.335: DVMRP: Send Report on Tunnel0 to 135.1.22.98
Mar 20 11:39:45.335: DVMRP: 2 unicast, 8 DVMRP routes advertised
Mar 20 11:39:47.335: DVMRP: Aging routes, 0 entries expired
Mar 20 11:39:51.371: DVMRP: Received Probe on Tunnel0 from 135.1.22.98
Mar 20 11:39:52.379: DVMRP: Received Report on Tunnel0 from 135.1.22.98
477
Verifying DVMRP Route Exchange
Checking DVMRP Routes
Being Advertised
pim-dvmrp-gw# debug ip dvmrp detail
DVMRP debugging is on
Mar 20 11:42:45.337: DVMRP: Building Report for Tunnel0 224.0.0.4
Mar 20 11:42:45.337: DVMRP: Report 130.1.0.0/16, metric 35, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 135.1.0.0/16, metric 35, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 135.1.22.0/24, metric 34, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 171.69.0.0/16, metric 35, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 172.21.27.0/24, metric 35, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 172.21.32.0/24, metric 34, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 172.21.33.0/24, metric 35, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 172.21.120.0/24, metric 35, from DVMRP table
Mar 20 11:42:45.337: DVMRP: Report 135.1.2.0/24, metric 1
Mar 20 11:42:45.337: DVMRP: Report 135.1.3.0/24, metric 1
Mar 20 11:42:45.337: DVMRP: Send Report on Tunnel0 to 135.1.22.98
Mar 20 11:42:45.337: DVMRP: 2 unicast, 8 DVMRP routes advertised
478
Verifying DVMRP Route
Exchange
Checking DVMRP Routes
Being Received
pim-dvmrp-gw# debug ip dvmrp detail
DVMRP debugging is on
... : DVMRP: Received Report on Tunnel0 from 135.1.22.98
... : DVMRP: Origin 130.1.0.0/16, metric 2, metric-offset 1, distance 0
... : DVMRP: Origin 135.1.0.0/16, metric 2, metric-offset 1, distance 0
... : DVMRP: Origin 171.69.0.0/16, metric 2, metric-offset 1, distance 0
... : DVMRP: Origin 135.1.2.0/24, metric 34, metric-offset 1, infinity
... : DVMRP: Origin 135.1.3.0/24, metric 34, metric-offset 1, infinity
... : DVMRP: Origin 135.1.22.0/24, metric 1, metric-offset 1, distance 0
... : DVMRP: Origin 172.21.27.0/24, metric 2, metric-offset 1, distance 0
... : DVMRP: Origin 172.21.32.0/24, metric 1, metric-offset 1, distance 0
... : DVMRP: Origin 172.21.33.0/24, metric 2, metric-offset 1, distance 0
... : DVMRP: Origin 172.21.120.0/24, metric 2, metric-offset 1, distance 0
479
Advanced Troubleshooting
Troubleshooting Network State
Troubleshooting PIM-DVMRP
Troubleshooting ATM P2MP VCs
480
Multicast over ATM P2MP VCs
ATM
NBMA
Cloud
A
C
B
D
One p2mp VC/group
performs multicast
replication instead of the
router
B’cast p2mp VC used
when # Groups > max
p2mp VC count
Use PIM Sparse mode
p2mp VCs map group
membership
Fast Switched!!
Note: Only p2mp m’cast VCs for
Router A shown for clarity.
481
M’cast P2MP VC Troubleshooting
rtr-a> show ip pim vc
IP Multicast ATM VC Status
ATM0/0 VC count is 5, max is 5
Group
VCD
Interface
224.0.1.40
21
ATM0/0
224.2.2.2
26
ATM0/0
224.1.1.1
28
ATM0/0
224.4.4.4
32
ATM0/0
224.5.5.5
35
ATM0/0
Leaf Count
2
1
1
2
1
Rate
0 pps
0 pps
0 pps
0 pps
0 pps
482
M’cast P2MP VC Troubleshooting
Root P2MP VC with 3 Leaf Routers
rtr-a> show atm vc
Interface
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
ATM0/0
VCD
1
2
3
4
5
6
9
10
11
12
13
14
15
VPI
0
0
0
0
0
0
0
0
0
0
0
0
0
VCI
5
16
124
125
126
127
130
131
132
133
134
135
136
Type
PVC
PVC
MSVC-3
MSVC
MSVC
MSVC
SVC
SVC
MSVC-3
MSVC-1
SVC
MSVC-2
MSVC-2
P2MP VC for which we are a Leaf
AAL /
Encapsulation
AAL5-SAAL
AAL5-ILMI
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
AAL5-SNAP
Peak
Kbps
155000
155000
155000
155000
155000
155000
155000
155000
155000
155000
155000
155000
155000
Avg. Burst
Kbps Cells Status
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
155000
96 ACT
483
M’cast P2MP VC Troubleshooting
show ip mroute 224.1.1.1
IP Multicast Routing Table
Flags: D - Dense, S - Sparse, C - Connected, L - Local, P - Pruned
R - RP-bit set, F - Register flag, T - SPT-bit set, J - Join SPT
Timers: Uptime/Expires
Interface state: Interface, Next-Hop or VCD, State/Mode
(*, 224.1.1.1), 00:03:57/00:02:54, RP 130.4.101.1, flags: SJ
Incoming interface: Null, RPF nbr 0.0.0.0
Outgoing interface list:
ATM0/0, VCD 3, Forward/Sparse, 00:03:57/00:02:53
ATM P2MP VC information for Group
484
M’cast P2MP VC Troubleshooting
P2MP VC Opened by Group 224.1.1.1
rtr-a> show atm vc 3
ATM0/0: VCD: 3, VPI: 0, VCI: 124, etype:0x0, AAL5 - LLC/SNAP, Flags: 0x650
PeakRate: 155000, Average Rate: 155000, Burst Cells: 96, VCmode: 0xE000
OAM DISABLED, InARP DISABLED
InPkts: 0, OutPkts: 12, InBytes: 0, OutBytes: 496
InPRoc: 0, OutPRoc: 0, Broadcasts: 12
InFast: 0, OutFast: 0, InAS: 0, OutAS: 0
OAM F5 cells sent: 0, OAM cells received: 0
Status: ACTIVE, TTL: 2, VC owner: IP Multicast (224.1.1.1)
interface = ATM0/0, call locally initiated, call reference = 2
vcnum = 11, vpi = 0, vci = 132, state = Active
aal5snap vc, multipoint call
Retry count: Current = 0, Max = 10
timer currently inactive, timer value = 00:00:00
Leaf Atm Nsap address: 47.0091810000000002BA08E101.444444444444.02
Leaf Atm Nsap address: 47.0091810000000002BA08E101.333333333333.02
Leaf Atm Nsap address: 47.0091810000000002BA08E101.222222222222.02
NSAP Addresses of Leaf Nodes
485
Agenda
Troubleshooting Tools
Basic Troubleshooting
Advanced Troubleshooting
Case studies
486
Troubleshooting Cheat Sheet
Check IGMP membership on PIM DR on Last Hop LAN
Check RP address in (*,G) entry on the DR
Check RPF interface to RP in (*,G) entry
Repeat above check for (*,G) state on routers along
the shared tree, up to RP.
Repeat above check for (S,G) state on routers along
the source tree, up to Source DR.
487
PIM Timers
The secret to understanding PIM is to watch the
timers.
3 minutes (3:30) is the “magic” number.
Interface expiration timers are updated every minute
so if the expire timer goes below 2:00 the route is
not being used.
Entry expiration timers are updated when data is
forwarded so if the timer drops below 2:59, the
source has stopped sending.
488
Source Tree
In IOS a (*,G) entry is always created
whenever a (S,G) entry is created.
The Source-tree may overlap the
Shared-tree in which case the (*,G)
entry will be non-NULL.
The Source-tree may be independent of
the Shared-tree in which case the (*,G)
entry will be NULL.
489
PIM SM Source Tree
•(S,G) forwarding entry
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T
Incoming interface: Serial0, RPF nbr 171.68.28.190
Outgoing interface list:
Serial1, Forward/Sparse, 00:04:28/00:01:32
NOTE: These uptimes indicate the receiver has always
been present
490
PIM SM Source Tree
•(S,G) forwarding entry
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T
Incoming interface: Serial0, RPF nbr 171.68.28.190
Outgoing interface list:
Serial1, Forward/Sparse, 00:04:28/00:01:32
Receivers have stopped joining
491
PIM SM Source Tree
•(S,G) forwarding entry
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: T
Incoming interface: Serial0, RPF nbr 171.68.28.190
Outgoing interface list:
Serial1, Forward/Sparse, 00:04:28/00:01:32
Data is not flowing
492
PIM SM Shared Tree
•(*,G) forwarding entry
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
All Sources for this group will be forwarded out the olist
493
PIM SM Shared Tree
•(*,G) forwarding entry
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
This always points to the RP
494
PIM SM Shared Tree
•(*,G) forwarding entry
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
This is the next-hop to the RP from “sh ip RPF”
495
PIM SM Shared tree
•(*,G) forwarding entry
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
The entry has been up for this long.
Note the uptime of the olist
496
PIM SM Shared tree
•(*,G) forwarding entry
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
All receivers for the entry may have left
497
PIM SM Shared tree
•(*,G) forwarding entry
(*, 224.1.1.1), 00:04:28/00:01:32, RP 171.68.28.140, flags: S
Incoming interface: Serial1, RPF nbr 171.68.28.140,
Outgoing interface list:
Serial0, Forward/Sparse, 00:04:28/00:01:32
A sparse-mode group must have an RP
498
PIM Shared Tree
•(S,G,R) forwarding entry
(171.68.37.121/32, 224.1.1.1), 00:04:28/00:01:32, flags: RP
Incoming interface: Serial0, RPF nbr 171.68.28.190
Outgoing interface list:
NULL
Points toward the RP!!!!
Last-hop router is sending (s,g,r) prunes
Note R-flag
499
mtrace
• Shows:
– Multicast path from source to receiver.
» Similar to unicast “trace” command
» Trace path between any two points in network
» TTL Thresholds & Delay shown at each node
• Troubleshooting Usage:
– Find where multicast traffic flow stops.
» Focus on router where flow stops
– Verify path multicast traffic is following.
» Identify sub-optimal paths.
mtrace/mstat—How it
works
Mtrace Packet Flow
Adds
mtrace
data
Adds
mtrace
data
Adds
mtrace
data
src
Adds
mtrace
data
Adds
mtrace
data
dest
First-hop
Router
Last-hop
Router
Multicast
Dist. Tree
Mtrace Packet
Note: Mtrace packets use special
IGMP packets with IGMP Type
codes of 0x1E and 0x1F.
Unix Workstation
or
Cisco Router
501
mtrace
dallas-gw>mtrace bloom-iptv-svr bwilliam-ss5 224.2.156.43
Type escape sequence to abort.
Mtrace from 172.17.67.43 to 171.68.37.121 via group 224.2.156.43
From source (?) to destination (bwilliam-ss5.cisco.com)
Querying full reverse path...
0 bwilliam-ss5 (171.68.37.121)
-1 dallas-gw (171.68.37.1) PIM thresh^ 0 3 ms
-2 wan-gw4 (171.68.86.193) PIM thresh^ 0 32 ms
-3 bloomington-mn-gw (171.68.27.2) PIM thresh^ 0 717 ms
-4 bloom-mnlab (171.68.39.28) PIM thresh^ 0 730 ms
-5 bloom-iptv-svr (172.17.67.43)
dallas-gw>
mstat
• Shows:
– Multicast path in pseudo graphic format.
» Trace path between any two points in network
» Drops/Duplicates shown at each node
» TTLs & Delay shown at each node
• Troubleshooting Usage:
– Locate congestion point in the flow.
» Focus on router with high drop/duplicate count
» Duplicates indicated as “negative” drops
mstat
dallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43
Source
Response Dest
Packet Statistics For
172.17.67.43
171.68.86.194
All Multicast Traffic
|
__/ rtt 547 ms
Lost/Sent = Pct Rate
v
/
hop 547 ms
--------------------172.17.67.33
171.68.39.28
bloom-mnlab
|
^
ttl
0
v
|
hop -409 ms
-11/168 = --% 16 pps
171.68.39.1
171.68.27.2
bloomington-mn-gw
|
^
ttl
1
v
|
hop 379 ms
-9/170 = --%
17 pps
171.68.27.1
171.68.86.193
wan-gw4
|
^
ttl
2
v
|
hop 28
ms
-3/195 = --%
19 pps
171.68.86.194
171.68.37.1
dallas-gw
|
\__
ttl
3
v
\ hop 0
ms
196
19 pps
171.68.37.121
171.68.86.194
Receiver
Query Source
Only For Traffic
From 172.17.67.43
To 224.2.156.43
--------------------
0/67 = 0%
6 pps
-3/67 = --%
6 pps
0/70 = 0%
7 pps
70
7 pps
mstat
dallas-gw>mstat 172.17.67.43 bwilliam-ss5 224.2.156.43
Source
Response Dest
Packet Statistics For
172.17.67.43
171.68.86.194
All Multicast Traffic
|
__/ rtt 399 ms
Lost/Sent = Pct Rate
v
/
hop 399 ms
--------------------172.17.67.33
171.68.39.28
bloom-mnlab
|
^
ttl
0
v
|
hop 119 ms
77/694 = 11% 69 pps
171.68.39.1
171.68.27.2
bloomington-mn-gw
|
^
ttl
1
v
|
hop -150 ms
395/609 = 65% 60 pps
171.68.27.1
171.68.86.193
wan-gw4
|
^
ttl
2
v
|
hop 30
ms
-8/39 = --%
3 pps
171.68.86.194
171.68.37.1
dallas-gw
|
\__
ttl
3
v
\ hop 0
ms
39
3 pps
171.68.37.121
171.68.86.194
Receiver
Query Source
Only For Traffic
From 172.17.67.43
To 224.2.156.43
--------------------
0/65 = 0%
6 pps
44/65 = 68%
6 pps
-1/21 = --%
2 pps
22
2 pps
Debugging Auto-RP Operation
Understand the Auto-RP mechanisms
This is the fundamental debugging tool for problems with Auto-RP!!!
Verify Group-to-RP Mapping Caches
First on the Mapping Agents
Other routers will learn Group-to-RP mapping info from these
routers
If not correct, use debug commands to see what’s wrong
Make sure all MA’s have consistent Group-to-RP information
If not, watch for TTL Scoping problems
Then on other routers
If info doesn’t match MA, there is a problem distributing the
information
Use show and debug commands to find where the break is
506
Debugging Auto-RP Operation
Insure Auto-RP group state is correct
Should normally be in Dense mode
Watch out for mixed DM and SM conditions
Can occur when Static RP’s are also defined
Always ‘deny’ Auto-RP groups on Static RP configurations
Use ‘Accept-RP’ filters on all routers as insurance
Watch out for DM problems in NBMA networks
(See Module 7 for details)
507
Debugging BSR Operation
Understand the BSR mechanisms
This is the fundamental debugging tool for problems with BSR!!!
Verify Group-to-RP Mapping Caches
First on the BSR
Other routers will learn Group-to-RP mapping info from this
router
If not correct, use debug commands to see what’s wrong
Then on other routers
If info doesn’t match BSR, there is a problem distributing the
information
Use show and debug commands to find where the break is
508
Case Studies
THANKS
510
