GINS
The GARR Network Monitoring System
Giovanni Cesaroni, GARR
EUMEDCONNECT2 Training – Rome, 22-25 June 2009
Agenda
PART 1
PART 2
GINS description
• NOC Tools Motivation
Let’s code the Network
Monitoring!
• Required Functionality
• SNMP in action
• Monitoring Environment
• BGP, OSPF, MPLS, IPv6
• Statistics Examples
• Visualization
PART 3
RRD World
• Reports
• RRD in action
• Slicing
• Traffic Flows Analysis
• How to avoid loosing data
• Work in progress
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
2
GARR Network
•
•
43 POPs (University and Research Centre)
PEERING: 76 Gbps
• 52.5Gbps vs GEANT2
•
•
•
•
10G + 2.5G IP Access
3*10GE E2E links
9*1GE E2E links
3x2.5Gbps IP Transit
•
2 Milan + 1 Rome
•
•
•
7x1Gbps+10Gbps National PEERING
BackBone Capacity ~110Gbps
7 TLC Operators
•
•
•
•
•
•
•
•
3 International IP Carrier
•
•
•
•
Telecom Italia
Infracom (ex Autostrade TLC)
Fastweb
Interoute (ex Eurostrada)
WIND
BT-Italia (ex Albacom)
COLT-Telecom
Global Crossing
Telia
Level3
Access Capacity: ~60Gbps
•
Starting from 2M  10G
•
N.Access Links: 500
•
N.Backbone Links: 62
•
E2E Capacity: ~40Gbps
• from 1G  10G
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
3
GOALS
•
•
•
Provide the NOC, Operations and Planning staff with all the tools needed to
do their work as well as possible
Monitor users site connectivity
Check the status of the services at each level of the network
• service oriented approach (not metric oriented)
•
•
•
•
Integrate monitoring services
Automate tools configuration
Give easy access to the information
Automatic generation of fault and performance reports
The goal is not to manage the control
plane, but to have full control of the
network
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
4
Measurements Storage
(MySQL & RRD)
Consistency Tools Robots
GINS Architecture
GINS Monitoring Tools
GINS Visualization Tools
GARR-DB: Network Database
(Network Structure MySql)
GARR
Network
GARR NOC
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
5
GARR-DB: the Information System
administrative
and technical
information!!!
Aggregate
Logical “circuit” (IP link,MPLS LSP, lambda service, etc)
physical object
segments
User Site
physical circuit
physical objects
physical circuit
eq
GARR
Backbone
GARR Domain
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
6
SW tools used by GINS
Scheduler:
Cron
Data visualization:
PHP, HTML, Javascript, Ajax, SVG
Data management:
AWK, Bash, PHP, RRDtools
Reports:
PHP, Jpgraph, HTMLDOC
Data storage:
MySQL, File, RRD
~5500 RRD files
Data acquisition:
MRTG, SNMP polls, ping
Network
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
7
NOC in action
Alarms
Trouble
Ticket
TLC NOC
APM
End Site
GARR NOC
GARR
Backbone
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
8
GINS at a glance
Main functionalities

• Network monitoring
• Statistics acquisition
• Trouble Ticket System
• Fault and Performance Reports

Monitoring Services
 Lambda
 SDH/SONET
 MPLS
 IPv4, IPv6
 OSPF, BGP
 E2E
 Multicast Beacons
 Equipment
Statistics Services
 IPv4, IPv6, Multicast traffic
 Physical interface errors
 Routers CPU
 Premium IP
 SDH/SONET errors
 Backbone weathermap
 Uncompressed Statistics
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
9
Monitoring services
•
GINS detects/defines the status of different services, on the basis of the information
gathered through the network.
Monitoring is supported on the following service classes:
• IPv4 and IPv6: [service status, input errors and output drops on physical interfaces]
• end-user site
• backbone interface
• IP Multicast Beacons [service status]
• Routing protocols:
• OSPF [link costs]
• BGP [peering status, adv/rec routes]
• SDH/Sonet [SDH/Sonet errors]
• router interface on leased-lines
• Lambda [service status, optical equipment port status]
• MPLS [MPLS LSP status]
• E2E: [E2E service status]
• defined as the stitching of multiple intra-domain and inter-domain links
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
10
Statistics services
•
GINS stores performance measurements data and provides:
•
Traffic Statistics
• IPv4 and IPv6, Multicast for end user sites and backbone
• Aggregate
• Peering
• Premium IP
• Uncompressed Statistics
• Sonet/SDH errors on leased lines
• Router CPU load and temperature
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
11
Other services
•
GINS includes a Trouble Ticket System which is highly customized for the
GARR operations procedures. In particular, it manages user services, leased
lines and PoP ticket.
•
Fault and performance reports:
• User monthly and yearly reports (HTML and PDF)
• User fault report and circuit availability
• Uncompressed traffic statistics (IP BW usage, 95th percentile, etc.)
• Carrier fault report and circuit availability (HTML and PDF)
•
Monitored physical devices:
• Juniper J6350, M7i, M10, M20, M320
• Cisco: 12xxx, 17xx, 18xx, 2xxx, 3750, 72xx, 75xx
• ADVA FSP3000
• Metrobility R4000, R5000
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
12
• Who is the target user of monitoring UIs?
The NOC & the Operation Staff, private access
Monitoring
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
13
•Control Panel and IP Monitoring
•BGP Alarms & Monitoring
•E2E Monitoring, Lambda & MPLS
•Other Services
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
14
Monitor Control Panel
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
15
NOC Interface (1/2) : links status
Last action
Trouble ticket
Telnet
Traffic in/out
End Site Info Giovanni Cesaroni,
EUMEDCONNECT2 Training, Rome 22-25 June 2009
16
NOC Interface (2/2): other services and quick ticket management
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
17
End Site Info
Trouble Tickets
Traffic
Interface Errors
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
18
Physical Interface Input Errors and Output Drops
2Mbps
The link is going to be upgraded to a Gbps link in the next days!
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
19
E2E Monitoring
Status of the “domain segment”
Status of the Interdomain Link
Aggregate status of the “domain link”
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
20
E2E Stitching Monitoring
IP
MPLS LSP
10GE
Lambda
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
21
GINS vs Gn2 E2E CU
GINS
E2Emon
data aggregation
E2Emon XML schema
GARR archive
GARR NOC
GN2:JRA4
Switch & DFN
GN2 E2E CU
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
22
MPLS Monitoring
 MUPBED: one e2e connection
Informations on:
1- LSP1
2- L2 connection
GINS MPLS Service
TLAB
GN2
IT
TO
GARR
SNMP Polls
LSP1
MI2
GN2
GN2
DFN
LSP2
MI1
LSP3
FF
DF
N
TSystem
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
23
MPLS Monitoring: MUPBED case
LSP Status
E2E L2
inter-domain status
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
24
BGP monitoring
• Peer status & prefixes information
...
• Alarms
...
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
25
SONET Alarms (rfc2558)
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
26
Statistics
•
Common statistics sets, different type of
representation
•
Online Network Status
•
Other Services
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
27
Traffic, Input errors & output drops
Long Term Analysis
CPU load
& temperature
Router
aggregate traffic
& peaks
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
28
Example of temperature statistics
In such cases I’d like to be alerted by email, SMS, phone and voice!!!
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
29
The backbone weathermap
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
30
Ticket info
25
20
615M
OSPF cost
Traffic load
Router CPU temperature
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
31
Traffic load
Ticket info
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
32
How it works
Weathermap
Merge
HTML dynamic map
SVG image
Generate
Convert
Network
PNG image
Measurements Storage
Network Database
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
33
• Who is the target user for network reports?
• What kind of reports are provided?
1- Network users, end sites
• fault and availability reports of the services
• historical Fault
traffic data
& Performance Reports
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
34
Fault & Performance Reports: UI
monthly report
95th percentile
GARR User
Uncompressed
statistics
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
35
User monthly and yearly PDF Reports
Introduction
Faults and availability
~1,000 report pages per month
~50MB disk space per month
Monthly and yearly traffic statistics
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
36
Uncompressed Traffic Statistics, monthly view
5 minutes
95th percentile
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
37
Uncompressed Traffic Statistics, yearly view
Monthly values
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
38
Historical data
2005!!
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
39
Fault & Performance Reports
• Who is the target user for network reports?
• What kind of reports are provided?
1- Network users, end sites
• fault and availability reports of the services
• historical traffic data
2- Network planning staff
• to extrapolate the traffic trends for the future network planning
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
40
GARR Traffic Trends
30.67 Gbps
35
Gbps
30
Ave In
Ave Out
Max In
Max Out
Vol In (Pbps)
Vol Out(Pbps)
95th In
95th Out
95th
25
20
15
3.84 Gbps
10
5
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
9
5/
20
0
9
1/
20
0
8
9/
20
0
8
5/
20
0
8
1/
20
0
7
9/
20
0
7
5/
20
0
7
1/
20
0
6
9/
20
0
6
5/
20
0
6
1/
20
0
5
9/
20
0
5
5/
20
0
1/
20
0
5
0
41
Traffic Evolution
GLOBAL INTERNET
r ~ 1.4/y
NATIONAL INTERNET r ~ 1.6/y
RESEARCH TRAFFIC
2001
2002
2003
r ~ 2.0/y
2004
2005
E2E
2006
2007
2008
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
2009
42
http://oss.oetiker.ch/smokeping/
Latency Measurements
By Tobias Oetiker
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
43
Latency Measurements
Server
Fping probe
End Site
• Round Trip Time fluctuations
• Packet Loss pecentage
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
44
GARR-DB: Network Database
Description of the
infrastructure
Slices
• Temporary infrastructures
• Network Labs
• Temporary research projects
Homer’s dream
is just:
• Infrastructures requiring monitoring only
• Dedicated monitoring systems (users or projects)
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
45
Slices
Dedicated monitoring systems
User requirements:
Administrator requirements:
• Quick and easy setup
• Traffic statistics
• Weathermaps
• Alarms
• Easy to manage
• Replicable
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
46
Slices
• Slice link, description and status
• Access policy
• MRTG log status
• Slice status (on,off)
• Url
• Status of MRTG CFG generation (red if disabled)
• Cronjob status (red if disabled)
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
47
Slices
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
48
Based on NetFlow protocol
Traffic Flows Analysis
Suite Nfsen/Nfdump by Peter Haag
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
49
Traffic Flows Analysis, architecture overview
www
Nfdump
Nfsen
RRDs
User
Nfdump (CLI)
NetFlow, data export, sampling
Network
Raw data
Nfcapd
Daily numbers:
• ~2000 flows/s export
• sampling 1:1000
• ~40MB-1.6GB each router
(raw data)
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
50
Traffic Flows Analysis, example
Servers vs DHCP
Analysis of 2 subnets traffic on one interface
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
51
MRTG vs NetFlow
GINS (SNMP)
Nfsen (NetFlow)
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
52
Do I trust sampling?
From router counters (GINS by MRTG):
From flows (NetFlow):
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
53
Traffic Flows Analysis with ASTracker
ASTracker Nfsen plugin by Nino Ciurleo @ GARR
=
+
+
+
+
+
+
+
How to get information on the traffic exchanged between ASes?
Example of an IP commodity peering
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
54
Traffic Flows Analysis with ASTracker: Microsoft black hole
 Microsoft AS8075 announce by GEANT
 Output traffic on Geant, input traffic lost
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
55
Traffic Flows Analysis with ASTracker: other examples
Facebook:
From the Microsoft Web Site
“As part of Microsoft's routine, monthly security update cycle,
we released 10 new security updates on June 9, 2009”.
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
56
Work in progress
• Tools that are currently going to be integrated:
• Reports on Traffic Flows Analysis
• Equipment SNMP Traps
• Future plans:
• Packaging: module packaging for distribution
• Optical Network Monitoring
• GINSv2
#@%$!
Operations
Support
System
Tell me guy!
GINS
Optical Network
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
57
Part 2
LET’S CODE THE NETWORK MONITORING!
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
58
SNMP, RFC
• n. 1441
• Introduction to version 2 of the Internet-standard Network
Management Framework
• n. 2578
• Structure of Management Information for version 2 of the Simple
Network Management Protocol (SNMPv2)
• n. 1213 (updates 2011,2013,2013)
• Management Information Base for Network Management of TCP/IPbased internets: MIB-II
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
59
SNMP
• 2 different approaches:
• SNMP POLL
You ask for something
The equipment sends a response
• SNMP TRAP
The equipment advises you about an event
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
60
USING SNMP, POLL
snmpget -v2c -c <community> <router> <Object Identifier OID>
snmpwalk -v2c -c <community> <router> <part of an OID>
Poll response: <OID> = <data type>: <value>
Basic examples:
snmpget -v2c -c <community> <router> IP-MIB::ipAdEntIfIndex.194.116.96.25
IP-MIB::ipAdEntIfIndex.194.116.96.25 = INTEGER: 82
snmpget -v2c -c <community> <router> IF-MIB::ifName.82
IF-MIB::ifName.82 = STRING: ge-1/2/0.4
snmpget -v2c -c <community> <router> IF-MIB::ifHighSpeed.82
IF-MIB::ifHighSpeed.82 = Gauge32: 1000
snmpget -v2c -c <community> <router> IF-MIB::ifHCInOctets.82
IF-MIB::ifHCInOctets.82 = Counter64: 262925908632166
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
61
SNMP in action: BGP Monitoring
Status of the Peer BGP: 1.3.6.1.2.1.15.3.1.2 (RFC 1269)
snmpwalk -v2c -c <community> <router>
1.3.6.1.2.1.15.3.1.2 |
awk -F 'SNMPv2-SMI::mib-2.15.3.1.2.' '{print $2}' |
awk -F ' = INTEGER: ' '{
if($2=="1"){status=sprintf("Idle");};
if($2=="2"){status=sprintf("Connect");};
if($2=="3"){status=sprintf("Active");};
if($2=="4"){status=sprintf("Opensent");};
if($2=="5"){status=sprintf("Openconfirm");};
if($2=="6"){status=sprintf("Established");};
print $1,status;}'
Returns a list of:
<IP address of the Peer> <Status of the Peer>
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
62
SNMP in action: BGP Monitoring
AS of the Peer BGP: 1.3.6.1.2.1.15.3.1.9 (RFC 1269)
snmpwalk -v2c -c <community> <router> 1.3.6.1.2.1.15.3.1.9 |
awk -F 'SNMPv2-SMI::mib-2.15.3.1.9.' '{print $2}' |
awk -F ' = INTEGER: ' '{print $1,$2;}'
Returns a list of:
<IP address of the Peer> <AS of the Peer>
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
63
SNMP in action: BGP Monitoring
A rude but simple BGP Monitor
Content of /<some path>/BGPmon.sh
#!/bin/bash
snmpwalk -v2c -c <community> <router> 1.3.6.1.2.1.15.3.1.2 |
awk -F 'SNMPv2-SMI::mib-2.15.3.1.2.' '{print $2}' |
awk -F ' = INTEGER: ' '{
if($2!="6"){alarm=sprintf(“The Peer has a problem: ");};
print alarm,$1;}'
In the crontab
MAILTO="giovanni.cesaroni@garr.it"
0-55/5 * * * * /<path>/BGPmon.sh
Why rude?
0- If a peering goes down for 24 hours, I get 288 emails, please change the email
address!!!
1- A better way of coding is to use the libraries of an higher language (php, perl, java,
etc.), allowing you to manage errors, performances and historical data
Why simple? Just a lovely command line
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
64
SNMP in action: BGP Monitoring
Monitoring BGP Prefixes
no more standard MIBs available
CISCO-BGP4-MIB
Accepted prefixes from Peer
1.3.6.1.4.1.9.9.187.1.2.4.1.1.<IP>.1.1 (.1.1 = IPv4 Unicast)
Advertised prefixes to Peer
1.3.6.1.4.1.9.9.187.1.2.4.1.6.<IP>.1.1
BGP4-V2-MIB-JUNIPER
Received prefixes from Peer
1.3.6.1.4.1.2636.5.1.1.2.6.2.1.7.<Peer Index>.1.1
Advertised prefixes to Peer
1.3.6.1.4.1.2636.5.1.1.2.6.2.1.10.<Peer Index>.1.1
Accepted prefixes from Peer
1.3.6.1.4.1.2636.5.1.1.2.6.2.1.8.<Peer Index>.1.1
Peer Index from:
1.3.6.1.4.1.2636.5.1.1.2.1.1.1.14
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
65
SNMP in action: OSPF Monitoring
OSPF cost of a link:
1.3.6.1.2.1.14.8.1.4.<IP Address>.0.0
(RFC 1850)
snmpwalk -v2c -c <community> <router> 1.3.6.1.2.1.14.8.1.4 |
grep '.0.0 =' |
awk -F '.0.0 = INTEGER: ' '{print $1,$2}' |
awk -F 'SNMPv2-SMI::mib-2.14.8.1.4.' '{print $2}'
Returns a list of:
<IP address> <OSPF cost>
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
66
SNMP in action: MPLS LSP Monitoring
On Juniper Routers:
To get the information about an LSP,
we have to know the index identifying
the LSP (<LSP index>),
Example:
BO1-MI1-VPN :
.66.79.49.45.77.73.49.45.86.80.78.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
Some information:
mplsLspName 1.3.6.1.4.1.2636.3.2.3.1.1
mplsLspPathChanges 1.3.6.1.4.1.2636.3.2.3.1.10
mplsLspLastPathChange 1.3.6.1.4.1.2636.3.2.3.1.11
mplsLspConfiguredPaths 1.3.6.1.4.1.2636.3.2.3.1.12
mplsLspStandbyPaths 1.3.6.1.4.1.2636.3.2.3.1.13
mplsLspOperationalPaths 1.3.6.1.4.1.2636.3.2.3.1.14
mplsLspFrom 1.3.6.1.4.1.2636.3.2.3.1.15
mplsLspTo 1.3.6.1.4.1.2636.3.2.3.1.16
mplsPathName 1.3.6.1.4.1.2636.3.2.3.1.17
mplsPathType 1.3.6.1.4.1.2636.3.2.3.1.18
mplsPathExplicitRoute 1.3.6.1.4.1.2636.3.2.3.1.19
mplsLspState 1.3.6.1.4.1.2636.3.2.3.1.2
snmpget -v2c -c <comunity> <router>
<mplsLspState>.<LSP index>
1 = unknown
2 = up
3 = down
mplsPathRecordRoute 1.3.6.1.4.1.2636.3.2.3.1.20
mplsPathBandwidth 1.3.6.1.4.1.2636.3.2.3.1.21
mplsPathCOS 1.3.6.1.4.1.2636.3.2.3.1.22
mplsPathInclude 1.3.6.1.4.1.2636.3.2.3.1.23
mplsPathExclude 1.3.6.1.4.1.2636.3.2.3.1.24
mplsPathSetupPriority 1.3.6.1.4.1.2636.3.2.3.1.25
mplsPathHoldPriority 1.3.6.1.4.1.2636.3.2.3.1.26
mplsPathProperties 1.3.6.1.4.1.2636.3.2.3.1.27
mplsLspOctets 1.3.6.1.4.1.2636.3.2.3.1.3
mplsLspPackets 1.3.6.1.4.1.2636.3.2.3.1.4
mplsLspAge 1.3.6.1.4.1.2636.3.2.3.1.5
mplsLspTimeUp 1.3.6.1.4.1.2636.3.2.3.1.6
mplsLspPrimaryTimeUp 1.3.6.1.4.1.2636.3.2.3.1.7
mplsLspTransitions 1.3.6.1.4.1.2636.3.2.3.1.8
mplsLspLastTransition 1.3.6.1.4.1.2636.3.2.3.1.9
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
67
SNMP in action: MPLS LSP Monitoring
How to build the LSP index:
B O 1 - M I .....
CHAR to DEC translation
.66.79.49.45.77.73.49.45.86.......
Build the monster using a translator
(or use an ASCII table on wikipedia):
<?
$name=$argv[1];
$oid=name2oid($name);
print $name.": ".$oid."\n";
function name2oid($string) {
$oid = '';
$len = strlen($string);
for ($i = 0; $i < $len; $i++) {
$oid .= ".".str_pad(ord($string[$i]), 2, 0, STR_PAD_LEFT);
}
$npoints=32-$len;
for ($i=0;$i<$npoints;$i++){
$oid .= ".0";
}
return $oid;
}
?>
$ php name2oid.php BO1-MI1-VPN
BO1-MI1-VPN: .66.79.49.45.77.73.49.45.86.80.78.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
68
SNMP in action: IPv6 traffic
There are no IPv6-MIBs available to measure the IPv6 traffic on the
Interfaces on Cisco and Juniper Routers.
A solution for Juniper Routers is using the firewall,
defining a counter for the IPv6 traffic
1- firewall configuration
2- interface configuration
> show configuration firewall
family inet6 {
filter ipv6-traffic {
interface-specific;
term count {
then {
count ipv6-traffic;
accept;
}
}
}
}
> show configuration interfaces ge-0/2/4.0 family inet6
filter {
input ipv6-traffic;
output ipv6-traffic;
}
3- result
> show firewall | grep ipv6
Filter: ipv6-traffic-ge-0/2/4.0-i
ipv6-traffic-ge-0/2/4.0-i
253874255
Filter: ipv6-traffic-ge-0/2/4.0-o
ipv6-traffic-ge-0/2/4.0-o
278249000
2929972
3005956
And now is time to understand how the OID counter is built
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
69
SNMP in action: IPv6 traffic
From JUNIPER-FIREWALL-MIB:
jnxFWCounterDisplayFilterName: 1.3.6.1.4.1.2636.3.5.2.1.1
jnxFWCounterDisplayName: 1.3.6.1.4.1.2636.3.5.2.1.7
And what we need to measure:
jnxFWCounterByteCount: 1.3.6.1.4.1.2636.3.5.2.1.5
How to build the index of the counter?
After some long reverse engineering….
1.3.6.1.4.1.2636.3.5.2.1.5 +
<length of the filter_name> +
<CHAR to DEC translation of the filter_name> +
<length of the counter_name> +
<CHAR to DEC translation of the counter_name> +
.2
In this case the filter_name and the counter_name are the same
(ipv6-traffic-ge-0/2/4.0-i)
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
70
SNMP in action: IPv6 traffic
Example for the counter on the ae1.0 interface:
ipv6-traffic-ae1.0-i :
.105.112.118.54.45.116.114.97.102.102.105.99.45.97.101.49.46.48.45.105
1.3.6.1.4.1.2636.3.5.2.1.5 +
.20 +
.105.112.118.54.45.116.114.97.102.102.105.99.45.97.101.49.46.48.45.105 +
.20 +
.105.112.118.54.45.116.114.97.102.102.105.99.45.97.101.49.46.48.45.105 +
.2
Finally, you can get the counter value by snmp or you can use the OID in a
MRTG configuration file.
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
71
RRD World
Where to find all the information:
http://oss.oetiker.ch/rrdtool/
thanks to Tobias Oetiker
How to store data in an efficient and systematic manner:
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
72
RRD World
MRTG
Network
CACTI
Handmade
or other
poller
Storage
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
73
RRD World
The RRD file:
Round Robin Archive: RRA
a possible and typical temporal structure:
600 values
600 values
600 values
600 values
Average
on
5 minutes
Average
on
30 minutes
Average
on
2 hours
Average
on
1 day
50 hours
12.5 days
50 days
AVERAGE
600
600
600
600 days
MAX
600
600
600
600
600
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
74
RRD World
RRD in action:
New value
600 values
600 values
600 values
600 values
Average
on
5 minutes
Average
on
30 minutes
Average
on
2 hours
Average
on
1 day
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
75
RRD World: how to avoid loosing data, method 1
First thing to do:
Change the size of the yearly RRA, for example to 10 years
3650 values
600 values
600 values
600 values
600 values
Average
on
5 minutes
Average
on
30 minutes
Average
on
2 hours
Average
on
1 day
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
76
RRD World: how to avoid loosing data , method 1
RRD API:
Info, create, update, fetch, tune, graph, dump, restore, etc.
rrdtool info <file.rrd>
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].pdp_per_row = 1
rra[1].cf = "AVERAGE"
rra[1].rows = 600
rra[1].pdp_per_row = 6
rra[2].cf = "AVERAGE"
rra[2].rows = 600
rra[2].pdp_per_row = 24
rra[3].cf = "AVERAGE"
rra[3].rows = 600
rra[3].pdp_per_row = 288
rrdtool resize <file.rrd> 3 GROW 3050
5m
30 m
2h
1d
rra[0].cf = "AVERAGE"
rra[0].rows = 600
rra[0].pdp_per_row = 1
rra[1].cf = "AVERAGE"
rra[1].rows = 600
rra[1].pdp_per_row = 6
rra[2].cf = "AVERAGE"
rra[2].rows = 600
rra[2].pdp_per_row = 24
rra[3].cf = "AVERAGE"
rra[3].rows = 3650
rra[3].pdp_per_row = 288
RRA number
10 years RRD
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
77
RRD World : how to avoid loosing data , method 2
Building RRD without compression:
12 values
Script
(every hour)
600 values
600 values
600 values
600 values
Average
on
5 minutes
Average
on
30 minutes
Average
on
2 hours
Average
on
1 day
Average
on
5 minutes
Yearly RRD without compression
Single RRA with 105408 values
366 days
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
78
RRD World : how to avoid loosing data , method 2
Building RRD without compression:
how to do it
1- Uncompressed RRD creation (once for year):
rrdtool create <destination.rrd>
>
--start <some year ago> --step 300
>
DS:in:GAUGE:600:U:U DS:out:GAUGE:600:U:U
>
RRA:LAST:0.5:1:105408
2- Data extraction and insertion (once for hour):
rrdtool fetch <source.rrd> --end now-600s --start now-4200s AVERAGE |
awk -F ' ' 'BEGIN {x=0;}{x++; if (x>2){ print $1 $2":"$3 } }' |
xargs rrdtool update <destination.rrd>
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
79
Reference
• URL: www.gins.garr.it
• Email: sw.dev@garr.it
• Giovanni.Cesaroni@garr.it
Giovanni Cesaroni, EUMEDCONNECT2 Training, Rome 22-25 June 2009
80