BMP(BGP Monitoring Protocol) Testing by JANOGers
BMP Test Results
1
© BIGLOBE Inc. 2014
16 Sep 2014
Joint Test Members and their Motivation
BIGLOBE
An ISP in Japan which has about 3 million subscribers.
We have several POPs in JP, HK, SG and the US, and we connect with 150+
ASs globally.
Primary goal is to be able to detect BGP trouble and investigate quickly.
GREE
One of the biggest social game providers in the mobile phone market.
They hope more oriented towards routing optimization to enhance the user
experience. e.g. Troubleshoot convergence time and latency.
Cisco
2
The biggest network device vendor.
They want to know the operators needs and get feedback for
future development.
© BIGLOBE Inc. 2014
16 Sep 2014
Current BMP implementation(Router)
Juniper
JUNOS 9.5 and later
BMP version1
http://www.juniper.net/techpubs/en_US/junos13.2/topics/topic-map/bgp-monitor
ing-protocol.html
JUNOS 13.3 and later
BMP version3
http://www.juniper.net/techpubs/en_US/junos13.3/topics/task/configuration/bgp-monitoring-protocol-v3.html
Cisco
IOS-XE 3.11S and later
BMP version3
http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/x
e-3s/irg-xe-3s-book/bgp-monitor-protocol.html
IOS
15.4(2)T and later
BMP version3
http://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/15-mt/irg-15-mt-book/bgp-monitor-protocol.html
IOS-XR will supports 5.2.2 (Mid of 2014)
3
© BIGLOBE Inc. 2014
16 Sep 2014
Current BMP implementation(Software)
BMP Receiver
https://code.google.com/p/bmpreceiver/source/list
Developed by a Google engineer involved in writing the Internet draft for BMP.
Supported BMP version 1 only.
We requested info on updates for version 3, but apparently, it crashes after
receiving static reports.
Forhadahmed/bmp
https://github.com/forhadahmed/bmp
Developed by a former Cisco engineer
Did not work properly in current state
Existing open source software did not support BMP version3.
Luckily, Cisco was able to provide us with internal tool for testing.
4
© BIGLOBE Inc. 2014
16 Sep 2014
Test network topology
AS64705
AS64600
ASR9000
EBGP
MX960
(JUNOS13.3R1.8)
IBGP
ASR1000
IBGP
IBGP
(IOS XE 3.11S)
Full
Route
IBGP
ASR1000
Route Reflector
(IOS XE 3.11S)
BMP Server
AS2518
(Cisco Internal tool)
the Internet
5
© BIGLOBE Inc. 2014
16 Sep 2014
Testing…
Test Period ：
16th to 27th of July 2014
Testing Place：
Cisco Tokyo Office
So nice ice cream
6
© BIGLOBE Inc. 2014
16 Sep 2014
Test result： Inbound filtering
MX960 (JUNOS13.3R1.8)
Loc-RIB-In
deny AS109$
Inbound
Policy
AS7521
AS55349
Number of Route
5
EBGP
Adj-RIB-In
AS109
AS7521
AS55349
Number of Route
80
BGP
UPDATE
Route
Reflector
AS109
AS7521
AS55349
Number of route
80
BMP Server
Number of routes in Adj-RIBs-In: 80
Number of routes in Loc-RIB: 5
7
© BIGLOBE Inc. 2014
16 Sep 2014
Test result ： Inbound filtering
In
this(JUNOS13.3R1.8)
situation,
80 routes
Route
deny the
AS109$MX960 is receiving
MX960
EBGP
Reflector
from
the
remote
router
and
denying
routes
Inbound
Adj-RIB-In
Loc-RIB-In
Policy
AS109
AS109
by
inbound
filtering.
AS7521
AS7521
BGP
UPDATE
AS55349
Number of Route
5
AS7521
AS55349
AS55349
Number of Route
80
Number of route
80
According to the BMP server log, we could find
differences
in
the
number
of
routes
between
BMP Server
Adj-RIBs-in
andinLocal-RIB.
Number of routes
Adj-RIBs-In: 80
Number of routes in Loc-RIB: 5
8
© BIGLOBE Inc. 2014
16 Sep 2014
Test result: downed BGP session by remote router
ASR1000
(IOS XE 3.11S)
EBGP
clear bgp
MX960
(JUNOS13.3R1.8)
BMP Server
BMP msg count: Route-monitor=2 Stat-report=115 Peer-down=1 Peer-up=2
4byte-ASN
Last Peer-down: Thu Jun 26 14:03:20 2014
Remote system closed the session
NOTIFY received: cease/admin reset
9
© BIGLOBE Inc. 2014
16 Sep 2014
Test result: downed BGP session by remote router
The remote
the “clear
bgp”MX960
command,
ASR1000router ranEBGP
clear ip bgp
(IOS XE 3.11S)
(JUNOS13.3R1.8)
and we
saw these behaviors.
We could find the peer-down message along with
BMP Server
the message of the down reason.
BMP msg count: Route-monitor=2 Stat-report=115 Peer-down=1 Peer-up=2
4byte-ASN
Last Peer-down: Thu Jun 26 14:03:20 2014
Remote system closed the session
NOTIFY received: cease/admin reset
10
© BIGLOBE Inc. 2014
16 Sep 2014
Test result ：downed BGP session by local router
ASR1000
(IOS XE 3.11S)
Interface
shutdown
IBGP
ASR1000
(IOS XE 3.11S)
BMP Server
BMP msg count: Route-monitor=3 Stat-report=20 Peer-down=1 Peer-up=1
Last Peer-down: Thu Jun 26 14:58:36 2014
Local system closed the session
NOTIFY sent: hold timer expired
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00 15 03 04 00
11
© BIGLOBE Inc. 2014
16 Sep 2014
Test result ：downed BGP session by local router
The ASR1000
local router ran the "Interface
shutdown"
ASR1000
IBGP
(IOS XE 3.11S)
command,
andInterface
we saw these behaviors.(IOS XE 3.11S)
shutdown
We
could
find
that
the
local
system
closed
the
BMP Server
session by hold timer.
BMP msg count: Route-monitor=3 Stat-report=20 Peer-down=1 Peer-up=1
Last Peer-down: Thu Jun 26 14:58:36 2014
Local system closed the session
NOTIFY sent: hold timer expired
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00 15 03 04 00
12
© BIGLOBE Inc. 2014
16 Sep 2014
Test result： max prefix down on MX960
MX960
(JUNOS13.3R1.8)
max prefix
down
iBGP
Route
Reflector
BGP
UPDATE
max prefix: 50
Number of
routes：
80
BMP Server
BMP msg count: Route-monitor=40 Stat-report=44 Peer-down=6 Peer-up=6
Last Peer-down: Thu Jun 26 16:28:50 2014
Local system closed the session
NOTIFY sent: cease/max prefixes reached
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00 1c 03 06 01 00 01 01 00 00 00 32
13
© BIGLOBE Inc. 2014
16 Sep 2014
Test result： max prefix down on MX960
max prefix
Here,MX960
the MX960
received
80
routes
from
remote
Route
down
iBGP
(JUNOS13.3R1.8)
Reflector
router. But MX max prefix is 50, so the BGP
Number of
max prefix:
50
routes：
session was
shut.
BGP
UPDATE
80
BMP Server
We could find downed status and the reason
BMP msg count: Route-monitor=40 Stat-report=44 Peer-down=6 Peer-up=6
caused.
Last Peer-down: Thu Jun 26 16:28:50 2014
Local system closed the session
NOTIFY sent: cease/max prefixes reached
ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
00 1c 03 06 01 00 01 01 00 00 00 32
14
© BIGLOBE Inc. 2014
16 Sep 2014
Test result： max prefix down on ASR1000
ASR1000
(IOS XE 3.11S)
max prefix
down
MX960
EBGP
BGP
UPDATE
max prefix: 50
(JUNOS13.3R1.8)
Number of
routes：80
BMP Server
BMP msg count: Route-monitor=21 Stat-report=11 Peer-down=1 Peer-up=2
4byte-ASN
ASR 1000 IOS-XE 3.11S does not
Last Peer-down: Thu Jun 26 16:10:37 2014
support RFC4486
(BGP cease notification Messages)
Local system closed the session
sub-code1
FSM event code: 0
（Maximum Number of Prefixes Reached）
*It supported from IOS-XR3.13
No NOTIFY sent
} ???
15
http://www.cisco.com/c/en/us/td/docs/iosxml/ios/iproute_bgp/configuration/xe-3s/irg-xe-3s-book/irg-maxprefix.html
© BIGLOBE Inc. 2014
16 Sep 2014
Test result： max prefix down on ASR1000
prefix
ThisASR1000
situation ismax
completely
the
same
as
previous
MX960
down
EBGP
(IOS XE 3.11S)
scenario. But the BMP server log reason(JUNOS13.3R1.8)
was
Number of
routes：80
unclear. max prefix: 50
In our survey, this case was caused by no support
BMP Server
of RFC4486 (BGP cease notification messages) on IOS-XE
BMP msg count: Route-monitor=21 Stat-report=11 Peer-down=1 Peer-up=2
3.11S.
4byte-ASN
BGP
UPDATE
ASR 1000
doesIOS-XE3.13.
not support
Last Peer-down:
Thuthat
Jun 26ASR1000
16:10:37 2014 supported
Cisco
said
from
RFC4486
Local system closed the session
FSM event code: 0
No NOTIFY sent
If you want to use BMP function,sub-code1
you should pay
}
???
attention for supported status of*ItRFC4486,
not
only
will support from 3.13
BMP.
16
(BGP cease notification Messages)
（Maximum Number of Prefixes
Reached）
© BIGLOBE Inc. 2014
16 Sep 2014
The test result summary
We were able to successfully implement BMP
and see how we could benefit from its use.
It allows operators to obtain valuable information that
previously required debug commands and router login.
Most Japanese ISPs forbid the use of debug command,
because of the high CPU usage.
It helps us troubleshoot complex outages.
The data gathered is based on RFC-defined standards,
so the output is consistent among various vendors.
17
© BIGLOBE Inc. 2014
16 Sep 2014
The test result summary
Current Implementation of BMP
Supported Routers
In recent implementations, Juniper and Cisco routers
support BMP version3.
We are hoping more vendors will support BMP
Available Software
It lacked a server-side software that was compatible with
BMP version 3.
We need Superman
18
© BIGLOBE Inc. 2014
16 Sep 2014
Last on the Agenda
What is BMP(BGP Monitoring Protocol)?
BIGLOBE/GREE/CISCO BMP test result summary
After JANOG34…
19
© BIGLOBE Inc. 2014
16 Sep 2014
JANOG34 in Takamatsu
Day
July 16-18 2014
Host
STNet, Incorporated
Attendee
Takamatsu
Tutorial and BoF: 93(remote 118)
Meeting 547
We presented these BMP test .
Some operators take interest in it
to troubleshoot without debug command.
20
© BIGLOBE Inc. 2014
16 Sep 2014
Opinion to IETF
We will report to IETF IDR(Inter-Domain routing)
and GROW working group about the gap between
operators expectations and vendor implementation
Timestamp
Timestamp is not mandatory requirement but operator's
expectation is pretty high.
Security
Draft is not explicitly request security for BMP it's self.
We thought this would be implementation's issue between
server-side software and router and multi vendors.
21
© BIGLOBE Inc. 2014
16 Sep 2014
Implementation update
After JANOG34, some software engineers have been
working on developing BMP servers.
NTT R&D Wataru Ishida published BMPv3 software
Ryu BMP Sever
https://github.com/osrg/ryu
http://osrg.github.io/bmp/
22
© BIGLOBE Inc. 2014
16 Sep 2014
Future Plans
BMP
Further testing for installation of our network.
Developing an automation tool triggered by BMP
messages for our daily operation.
Find other new useful technologies
We want to find and test new technologies with JANOG
operators.
23
© BIGLOBE Inc. 2014
16 Sep 2014