perfSONAR-PS Installation Report (v0.9) Takatoshi Ikeda APAN-JP John Hicks TransPAC2 1 This document shows the procedures to install perfSONAR-PS based on APAN Tokyo XP case. It aims to help NOC people to install perfSONAR-PS and begin perfSONAR service smoothly. Conventions This is conventions used in this manual This box means Command Line Interface (CLI) of the OS. User execute commands in this box at your server “#” appeared at first string on the line in above box indicates command prompt of root user This means just press ENTER key at the point Italics, BIG LETTER bounded by < > is arbitrary character strings. Bold indicates command the user must enter Italics, Bold and Pink indicates the text user must write in the file. Bold, Green indicates the answer of the dialogical question Bold, Blue indicates is the comment 2 Contents 1 2 3 4 5 perfSONAR-PS .......................................................................................................... 4 1.1 perfSONAR services ........................................................................................... 4 1.2 perfSONAR-PS software .................................................................................... 5 Specification of APAN Tokyo XP................................................................................ 6 2.1 Deployment services ........................................................................................... 6 2.2 Server Allocation ................................................................................................ 6 2.3 Server Specification ............................................................................................ 6 2.4 Network Configuration....................................................................................... 7 OS and Basic Settings ............................................................................................... 8 3.1 OS ....................................................................................................................... 8 3.2 Packages ............................................................................................................. 8 3.3 Basic Setting....................................................................................................... 9 3.4 Package update .................................................................................................. 9 3.5 Unnecessary Linux Services .............................................................................10 3.6 Run level ............................................................................................................10 3.7 TCP Wrapper .....................................................................................................10 3.8 CPAN ................................................................................................................. 11 3.9 Reboot ................................................................................................................12 Lookup Service (perfSONAR-PS Lookup Service) ...................................................13 4.1 dbxml .................................................................................................................13 4.2 perl-rrdtool ........................................................................................................13 4.3 perfSONAR_PS::Services::LS::LS .....................................................................14 4.4 Firewall (iptables) .............................................................................................16 4.5 Check .................................................................................................................16 Interface Usage (perfSONAR-PS SNMP MA) ..........................................................17 5.1 RRDtool .............................................................................................................17 5.2 Net-SNMP .........................................................................................................17 5.3 MySQL ...............................................................................................................18 5.4 PHP Module.......................................................................................................18 5.5 Cacti...................................................................................................................18 5.6 perfSONAR_PS::Services::MA::SNMP ..............................................................24 5.7 Check .................................................................................................................27 6 Throughput (perfSONAR-PS perfSONAR-BUOY) ..................................................28 7 Delay (perfSONAR-PS perfSONAR-BUOY) ................ Error! Bookmark not defined. 3 1 perfSONAR-PS perfSONAR is an infrastructure for network performance monitoring, making it easier to solve end-to-end performance problems on paths crossing several networks. It contains a set of services delivering performance measurements in a federated environment. These services act as an intermediate layer, between the performance measurement tools and the diagnostic or visualization applications. This layer is aimed at making and exchanging performance measurements between networks, using well-defined protocols. (http://www.perfsonar.net/) perfSONAR-PS is a set of independent software services that implement the perfSONAR protocols. This software development is the result of a collaboration between ESnet, Fermi National Accelerator Laboratory, The Georgia Institute of Technology, Indiana University, Stanford Linear Accelerator Center, University of Delaware, and Internet2. (http://www.internet2.edu/performance/pS-PS/) 1.1 perfSONAR services An overview of the perfSONAR services is following Services Description Lookup Service(LS) Service discovery Topology Service(TopS) Publication of network topology Authentication Service(AS) Authentication Functionality Transformation Service(TS) Pipeline and modify data between other services Resource Protector Service(RPS) Arbitration of the consumption of limited resources Measurement Point Service(MP) Creating and providing the measurement data Measurement Archive Service(MA) Publication of historical monitoring data * http://www.perfsonar.net/services.html 4 1.2 perfSONAR-PS software The v0.9 perfSONAR-PS software package contains following services that add value to the perfSONAR framework. perfSSONAR-PS PerfSONAR product Services Lookup Service LS Providing perfSONAR LS sercice Topology Service TopS Providing perfSONAR TopS sercice SNMP MA Exposing data collected via variables from Measurement Archive Description the Simple Network Management Protocol (SNMP) devices protocol and found stored in on networked Round Robin Databases (RRD) archives. Circuit Status MA Providing networks status information to Measurement Archive E2EMon. This service uses the Status Measurement Archive and can use the Topology Service to provide the complete domain status required by the E2EMon service Status Measurement MA Providing layer2 link status information. Archive This information can be obtained via a polling service that can use SNMP, or scripts to collect the link status information, and stores it into a SQL database. PerfSONAR-BUOY MP&MA Exposing active measurement data obtained by the BWCTL tool (throughput measurements) and currently consist of regularly-scheduled Iperf tests to a configurable list of source and destination hosts. PingER Services MP&MA Conducting and storing ping measurements and making available such data. This service is an evolution of the PingER project * http://www.internet2.edu/performance/pS-PS/ 5 2 Specification of APAN Tokyo XP This chapter provides the specification of APAN Tokyo XP where the perfSONAR-PS services installed. 2.1 Deployment services Deployment services at APAN Tokyo XP are followings. No. Service Description 1 Lookup Service Providing Lookup Service 2 Interface Usage Exposing interface usage data of network equipment 3 Throughput Exposing throughput measurement data 4 Delay Exposing one-way delay measurement data 2.2 Server Allocation The server allocation of perfSONAR-PS service at APAN Tokyo XP is following. A server machine is allocated to each service. Server Service hostname 1 Lookup Service ps1.jp.apan.net 2 Interface Usage nms2.jp.apan.net 3 Throughput nms1.jp.apan.net 4 Delay nms4.jp.apan.net 2.3 Server Specification Server specification of servers is following Hardware Host Server Server 1 ps1 BareBone SuperMicro SuperServer 6015X-TV CPU Intel Xeon QuadCore 2.66GHz *2 Memory DDR2-667 1GB * 2 Disk SATA 500GB * 2 (RAID1) NIC On-board 10/100/1000 * 2 6 nms1 BareBone nms2 nms4 SuperMicro SuperServer 6013P CPU Intel Xeon 3.06GHz *2 Memory DDR266 1GB * 2 Disk SAS 73GB * 2 (RAID1) NIC On-board 10/100/1000 * 2 OS&Kernel Host OS Kernel ps1 CentOS 5.1 x86_64 (Linux) 2.6.18-53.1.21.el5 nms1 CentOS 5.1 i386 (Linux) 2.6.24.7 + web100-2.5.19 nms2 2.6.18-53.1.21.el5 nms4 Installed packages Host perfSONAR-PS feature Measurement tools(version) ps1 Lookup Service Oracle Sleepycat XML Database(2.3.10) nms1 PerfSONAR-BUOY Iperf (2.0.2) BWCTL (1.2a) nms2 SNMP Measurement Archive RRDtool (1.2.23) SNAPP (2.0) or Cacti (0.8.7b-2) nms4 PerfSONAR-BUOY OWAMP (3.0c) 2.4 Network Configuration 7 3 OS and Basic Settings This chapter describes how to install OS and do basic setting for perfSONAR-PS. This is common procedure for all perfSONAR-PS servers. Target Lookup Service ps1.jp.apan.net Server Interface Usage nms2.jp.apan.net Throughput nms1.jp.apan.net Delay nms4.jp.apan.net 3.1 OS This section describes the procedure of OS installation. This procedure apply to all servers. 1 Download the disk image from CentOS Web page below and make an install disk. CentOS Download - http://www.centos.org/modules/tinycontent/index.php?id=15 2 Insert the install disk you created to CD or DVD drive then boot the server. 3 Set parameters depends on your environment within install wizard [Items of install wizard] Language – arbitrary setting Keyboard – arbitrary setting Disk Partition – arbitrary setting Network – setting depend on your environment Time zone – arbitrary setting Root Password – arbitrary setting Package – see section 3.2 3.2 Packages This section describes the package selection for perfSONAR-PS servers. To install perfSONAR-PS, following package set and additional packages are necessary. 1 Select [ Server-GUI ] and check [ Customize now ] . Click [Next] 2 Select following packages as additional packages. Then click [Next] Group Package 8 Application Development Libraries Development Tools Legacy Software Development 3.3 Basic Setting This section describes the basic setting of the OS. You must set up after package install and rebooting. Firewall – default setting SElinux – Select [disabled] (To avoid unexpected access denied) Time – arbitrary setting User account – arbitrary setting Sound card – arbitrary setting Reboot the server 3.4 Package update This section descries the how to update the package. A lot of initial packages are needed to update due to security and functional reason. We manage RPM packages by using yum1. Following procedure is how to update the package by yum. . 3.4.1 Setting up for yum yum chooses the repository from mirror list to download the packages. But, sometimes optimal mirror site is not selected. So, install yum plugin which chooses the fastest repository from a mirror list. # yum -y install yum-fastestmirror <snip> Complete! Add new repository, DAG, to get necessary packages. Import GPG-KEY and make a new configuration file for DAG. # rpm --import http://dag.wieers.com/rpm/packages/RPM-GPG-KEY.dag.txt # vi /etc/yum.repos.d/dag.repo [dag] 1 yum - yum is an automatic updater and package installer/remover for rpm systems http://linux.duke.edu/projects/yum/ 9 name=RPMForge: Dag baseurl=http://apt.sw.be/redhat/el$releasever/en/$basearch/dag gpgcheck=1 gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY.dag.txt enabled=1 3.4.2 yum update Update the packages # yum -y update Loading "installonlyn" plugin Setting up Update Process <snip> Complete! *Take much time to complete the update 3.5 Unnecessary Linux Services To improve security level, save system resource and avoid the performance problems, turn off unnecessary Linux services for perfSONAR-PS. # /sbin/service <SERVICE> stop & /sbin/chkconfig <SERVICE> off <SERVICE> --- Put service name you want disable. Unnecessary service is depends on your environment but possible unnecessary services for perfSONAR are following. avahi-daemon, bluetooth, cups, hidd, ipmi, irqbalance, lvm2-monitor, nfslock, pcscd, xfs, isdn, setroublesboot *Some services cannot be disabled. But it will be disabled when system is rebooting. 3.6 Run level To save the system resource, change the run level to turn off X window. # vi /etc/inittab id:3:initdefault: 3.7 TCP Wrapper To control the access to services by hostname and IP address, set up TCP Wrapper. # vi /etc/hosts.allow ALL : < Allowed host> # vi /etc/hosts.deny 10 ALL : ALL <Allowed host> --- put hostname or IP addresses you want to allow the access. ex) ALL : 192.168.0.1 , ALL : ns.jp.apan.net If you want to know how to configure in detail, please refer to following web page. http://www.redhat.com/docs/manuals/linux/RHL-9-Manual/ref-guide/s1-tcpwrappersaccess.html 3.8 CPAN You can install perfSONAR-PS packages from CPAN. So, set up CPAN. # cpan We have to reconfigure CPAN.pm due to following uninitialized parameters: * following messages are only question. Explanation messages are snipped. Are you ready for manual configuration? [yes] CPAN build and cache directory? [/root/.cpan] Cache size for build directory (in MB)? [10] Perform cache scanning (atstart or never)? [atstart] Cache metadata (yes/no)? [yes] Your terminal expects ISO-8859-1 (yes/no)? [yes] File to save your history? [/root/.cpan/histfile] Number of lines to save? [100] Policy on building prerequisites (follow, ask or ignore)? [ask] Where is your gzip program? [/bin/gzip] Where is your tar program? [/bin/tar] Where is your gzip program? [/bin/gzip] Where is your tar program? [/bin/tar] Where is your unzip program? [/usr/bin/unzip] Where is your make program? [/usr/bin/make] Where is your links program? [/usr/bin/links] Where is your wget program? [/usr/bin/wget] Warning: ncftpget not found in PATH Where is your ncftpget program? [] Warning: ncftp not found in PATH Where is your ncftp program? [] Where is your ftp program? [/usr/kerberos/bin/ftp] Where is your gpg program? [/usr/bin/gpg] What is your favorite pager program? [/usr/bin/less] What is your favorite shell? [/bin/bash] PREFIX=~/perl Your choice: -j3 non-root users (please see manual for more hints) [] dual processor system 11 Your choice: [] UNINST=1 Your choice: to always uninstall potentially conflicting files [] Timeout for inactivity during Makefile.PL? [0] Your ftp_proxy? Your http_proxy? Your no_proxy? Shall I use the local database in /root/.cpan/sources/MIRRORED.BY? [y] (1) Africa (2) Asia (3) Central America (4) Europe (5) North America (6) Oceania (7) South America Select your continent (or several nearby continents) [] * choose the number to select prefer mirror site. Enter another URL or RETURN to quit: [] cpan> quit # 3.9 Reboot Reboot the system # /sbin/shutdown –r now 12 4 Lookup Service (perfSONAR-PS Lookup Service) This chapter describes how to install and set up perfSONAR-PS Lookup Service Target Server Target Server Lookup Service ps1.jp.apan.net Install software Software Version 2.3.10 dbxml perfSONAR_PS Lookup Service 0.9 Description an embedded XML database with XQuery-based access. This is used to store the service registration information. . perfSONAR_PS Lookup Service 4.1 dbxml perfSONAR-PS stores the service registration information to Oracle Sleepycat XML Database. Installation procedure is following. # wget –c http://download.oracle.com/berkeley-db/dbxml-2.3.10.tar.gz # gzip -dc dbxml-2.3.10.tar.gz | tar xf - # cd dbxml-2.3.10 # ./buildall.sh --enable-perl --prefix=/usr/local/dbxml-2.3.10 <snip> # echo "/usr/local/dbxml-2.3.10/lib" > /etc/ld.so.conf.d/dbxml.conf # /sbin/ldconfig 4.2 perl-rrdtool perl-rrdtool is perl module for rrdtool. This module is necessary for perfSONAR configuration tool (psConfigureDaemon) # yum -y install perl-rrdtool <snip> Complete! 13 4.3 perfSONAR_PS::Services::LS::LS 4.3.1 Install The software packages for perfSONAR_PS Lookup Service are available at CPAN distribution system. Following is installation procedure from CPAN. # cpan Terminal does not support AddHistory. cpan shell -- CPAN exploration and modules installation (v1.7602) ReadLine support available (try 'install Bundle::CPAN') cpan> install perfSONAR_PS::Services::LS::LS CPAN: Storable loaded ok <snip> Shall I follow them and prepend them to the queue of modules we are processing right now? [yes] * Answer “ yes “ for same question above. <snip> /usr/bin/make install -- OK cpan> quit # 4.3.2 Set up Configuration utility is prepared for PerfSONAR Daemon. This utility tool is an interactive mode. Following is a dialogue result. # psConfigureDaemon <snip> What file should I write the configuration to? [/etc/perfsonar/daemon.conf]: 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 4) Save configuration 5) Exit ? []: 3 Enter the maximum number of children processes (0 means infinite) [0]: Enter number of seconds a child can process before it is stopped (0 means infinite) [0]: Disable echo by default (0 for yes, 1 for now) [0]: The LS for MAs to register with [http://packrat.internet2.edu:8005/perfSONAR_PS/services/LS]: Interval between when LS registrations occur [in minutes] [60]: Interval between when children are repeaed [in seconds] [20]: Enter pid dir location [/var/run]: Enter pid filename [ps.pid]: 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 14 yes 4) Save configuration 5) Exit ? []:1 Enter endpoint in form 'port/endpoint_path' (e.g. 8080/perfSONAR_PS/services/SERVICE_NAME) or select from a number from the above []: 8080/perfSONAR_PS/services/LS Enter endpoint module [snmp,ls,perfsonarbuoy,pingerma,pingermp,status,circuitstatus,topology] []: ls Enter the external host or IP for this machine [localhost.localdomain]: <HOSTNAME> Enter the accesspoint for this service [http://<HOSTNAME>:8080/perfSONAR_PS/services/LS]: Enter default TTL for registered data [in minutes] [60]: Enter the directory of the XML database [/etc/perfsonar/xmldb]: Enter the name of the container inside of the XML database [store.dbxml]: Should the LS periodically remove old registration information (0 for no, 1 for yes) [0]: Enter a name for this service [Lookup Service]: Enter the service type [MA]: Enter a service description [Lookup Service]: <SERVICE DESCRIPTION> Enter the service's URI [http://< HOSTNAME>:8080/perfSONAR_PS/services/LS]: 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 4) Save configuration 5) Exit ? []: 4 Saved config to /etc/perfsonar/daemon.conf 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 4) Save configuration 5) Exit ? []: 5 <HOSTNAME> --- hostname or IP address of this server machine. <SERVICE DESCRIPTION> --- Description of the service 4.3.3 run Run the perfSONAR-PS daemon for Lookup Service # mkdir /etc/perfsonar/xmldb # /sbin/service perfsonar-daemon start /usr/bin/perfsonar --config /etc/perfsonar/daemon.conf --piddir=/var/run --pidfile=perfsonar-daemon.pid --logger=/etc/perfsonar/logger.conf 2008/07/11 00:06:15 (2421) WARN> LS.pm:148 perfSONAR_PS::Services::LS::LS::init - Setting 'reaper_interval' to '0'. /etc/init.d/perfsonar-daemon start: perfsonar-daemon started 15 4.3.4 Check daemon Check perfSONAR-PS daemon is running successfully. # ps -ef | grep perfsonar root 27936 root 27937 27936 0 Apr23 ? 1 0 Apr23 ? 00:00:26 perfsonar.pl (27936) - Listener (8080) 00:00:00 perfsonar.pl (27936) root 27938 27936 0 Apr23 ? 00:05:18 perfsonar.pl (27936) - LS Reaper You can see processes below if perfSONAR-PS is running. 4.3.5 start-up configuration Set up start-up configuration # /sbin/chkconfig --add perfsonar-daemon # /sbin/chkconfig perfsonar-daemon on 4.4 Firewall (iptables) Change the rule of firewall for perfSONAR-PS daemon. TCP 8080 is used for perfSONAR-PS daemon as you configured at section 4.3.2. # vi /etc/sysconfig/iptables <snip> -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 8080 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # /sbin/service iptables restart 4.5 Check 4.5.1 psping psping is check tool whether perfSONAR services is available or not. psping can be installed by CPAN # cpan cpan> install perfSONAR_PS::Client::Echo <snip> /usr/bin/make install -- OK cpan>quit 4.5.2 check service Check whether perfSONAR-PS Lookup Service is available or not by psping. # psping http://<HOSTNAME>:8080/perfSONAR_PS/services/LS Service http://localhost.localdomain:8080/perfSONAR_PS/services/LS is up -Time to make request: 0.26802 * if Lookup Service is available, you can see above message. If not, you can see “down” instead of “up” on above message. 16 5 Interface Usage (perfSONAR-PS SNMP MA) This chapter describes how to provide Interface Usage data by perfSONAR-PS. Target Server Target Server Interface Usage nms2.jp.apan.net Install software Software Version 1.2.23 RRDTool NetSNMP MySQL Cacti perfSONAR_PS SNMP MA 5.3.1-24 5.0.45-7 0.8.7b-2 0.9 Description the OpenSource high performance data logging and graphing system. Interface usage data is stored to this tool’s format database. Free SNMP tool. This tool is used to collect MIB information of interface usage from network equipments. Open source database. The configuration for Cacti is stored to MySQL RRDTool frontend tool which has RRDTool's data storage and graphing functionality perfSONAR_PS SNMP Measurement Archive *Sections 5.1 to 5.6 are the installation procedure of the software to gather the interface usage data and store it in RRD format file. If you already stored the interface usage in RRD format by some tools, please skip to section 5.6. 5.1 RRDtool To store the interface usage data, install the RRDtool and related libraries # yum -y install rrdtool.i386 rrdtool-devel.i386 perl-rrdool <snmip> Complete! # 5.2 Net-SNMP Install Net-SNMP utilities. Cacti gathers the data by SNMP 17 # yum -y install net-snmp-utils <snip> Complete! 5.3 MySQL MySQL is used by Cacti. Configuration and some stuffs are stored. 5.3.1 Install Install MySQL and related packages. # yum -y install mysql-server mysql-devel <snip> Complete! 5.3.2 Run Run MySQL server # /sbin/service mysqld start <snip> Starting MySQL: [ OK ] 5.3.3 Startup Configuration Set up start-up config # /sbin/chkconfig mysqld on 5.4 PHP Module Install PHP module for MySQL # yum -y install php-mysql <snip> Complete! 5.5 Cacti Cacti gathers the interface usage data by SNMP and stored it to RRDtool format database. This data in RRDTool database is published by perfSONAR-PS SNMP MA. 5.5.1 Install Install Cacti by yum. # yum -y install cacti cacti-docs <snip> Complete! 5.5.2 DB setting Set up mySQL DB for Cacti. # mysqladmin --user=root create cacti 18 # mysql cacti < /var/www/cacti/cacti.sql # mysql --user=root mysql <snip> mysql> grant all on cacti.* to cacti@localhost identified by ‘<PASSWORD>’ ; Query OK, 0 rows affected (0.05 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) mysql> exit <PASSWORD> --- Put the password for user cacti on mySQL. This password is used to access to the DB from Cacti. Also, this password has to be set at configuration file of Cacti. See next section. 5.5.3 Cacti setting for DB Modify configuration file for Cacti to in accordance with Section 5.5.2. # vi /var/www/cacti/include/config.php $database_username = "cacti"; $database_password = "<PASSWORD>"; <PASSWORD> --- Put the password you configured at section 5.5.2 . 5.5.4 Poller.php Check whether the script (poller.php) which gather the data works or not. # php /var/www/cacti/poller.php sh: -q: command not found sh: -: command not found ^C # *If above messages appeared and no response after executing command, it’s working. Please exit by press Ctrl-C 5.5.5 Cron Set up cron for gathering the data periodically. # crontab -u cacti -e */5 * * * * /usr/bin/php /var/www/cacti/poller.php > /dev/null 2>&1 5.5.6 HTTPD Cacti can be configured by WEB interface. Also, Cacti publish the traffic graphs on the WEB. So, turn up HTTP server. # vi /etc/httpd/conf.d/cacti.conf # /sbin/service httpd start # /sbin/chkconfig httpd on allow from <ALLOW_HOST> <ALLOW_HOST> --- This is the configuration for access restriction to Cacti’s WEB 19 interface. Put IP addresses or hostname you want to allow. ex) allow from 192.168.0.0 , allow from ns.jp.apan.net , allow from all Detail information is on following link. http://httpd.apache.org/docs/2.2/en/mod/core.html#directory 5.5.7 Firewall (iptables) To open the port for HTTP server, change the rule of firewall. # vi /etc/sysconfig/iptables -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 80 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # /sbin/service iptables restart Flushing firewall rules: [ OK Setting chains to policy ACCEPT: filter [ OK Unloading iptables modules: ] ] [ OK ] Applying iptables firewall rules: [ OK Loading additional iptables modules: ip_conntrack_netbios_n[ OK ] ] # 5.5.8 Basic setting for Cacti Basic setting for Cacti can be set up by WEB Interface. Access to HTTP server from client you allow at 5.5.6. http://<HOSTNAME>/cacti/ Cacti Installation Guide is launched. All setting is as default. Click [Next ] Click [Next] 20 click [Finish] Main page appeared Enter new password Login - User Name: admin Password: admin 5.5.9 Register Network Equipments Register the network equipments you want to get data. Please refer to the manual for Cacti below. http://www.cacti.net/downloads/docs/html/ 1. Click [Devices] on left menu. Then, Click [Add] red circle) 2. Fill in the form. Then, click [create] Description --- description of the device Hostname --- hostname or IP address of the device Host Template --- [Cisco Router]. Most router is OK for this template Downed Device Detection --- [SNMP]. SNMP version --- [Version 1] SNMP community --- As your router setting 21 3. You can create the graphs for interface usage of registered device. Click [New Graphs] on left menu. Select Host which has interface you want to see.. Then, select the interface you want to see the usage, and select [In/Out Bits (64-bit Counters)] Click [Create] 4. Publish the graph under graph tree. Click [Graph Trees] on left menu. Click [Default Tree] on Graph Trees 22 Click [Add] at the red circle below. Select the information you want add. Parent Item --- select [root] Tree Item Type --- select [Host] Host --- select the router you want add Graph Grouping Style --- Graph Template Then, Click [Create] 5. Cacti gather interface usage and you can see graphs. Click [Graphs] tab and select [Host] you want to see. 23 5.6 perfSONAR_PS::Services::MA::SNMP 5.6.1 Install The software packages for perfSONAR_PS SNMP MA are available at CPAN distribution system. Following is installation procedure from CPAN. # cpan <snip> cpan> install perfSONAR_PS::Services::MA::SNMP CPAN: Storable loaded ok <snip> Shall I follow them and prepend them to the queue of modules we are processing right now? [yes] yes * Answer “ yes “ for same question above. <snip> /usr/bin/make install -- OK cpan> quit # 5.6.2 Set up Configuration utility is prepared for PerfSONAR Daemon. This utility tool is an interactive mode. Following is a dialogue result. # psConfigureDaemon <snip> What file should I write the configuration to? [/etc/perfsonar/daemon.conf]: 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 24 4) Save configuration 5) Exit ? []: 3 Enter the maximum number of children processes (0 means infinite) [0]: Enter number of seconds a child can process before it is stopped (0 means infinite) [0]: Disable echo by default (0 for yes, 1 for now) [0]: The LS for MAs to register with [http://packrat.internet2.edu:8005/perfSONAR_PS/services/LS]: Interval between when LS registrations occur [in minutes] [60]: Interval between when children are repeaed [in seconds] [20]: Enter pid dir location [/var/run]: Enter pid filename [ps.pid]: 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 4) Save configuration 5) Exit ? []:1 Enter endpoint in form 'port/endpoint_path' (e.g. 8080/perfSONAR_PS/services/SERVICE_NAME) or select from a number from the above []: 8080/perfSONAR_PS/services/snmpMA Enter endpoint module [snmp,ls,perfsonarbuoy,pingerma,pingermp,status,circuitstatus,topology] []: snmp Enter the external host or IP for this machine [localhost.localdomain]: <HOSTNAME> Enter the accesspoint for this service [http://<HOSTNAME>:8080/perfSONAR_PS/services/snmpMA]: Enter the location of the RRD binary [/usr/bin/rrdtool]: Enter the default resolution of RRD queries [300]: Enter the database type to read from (file or xmldb) [file]: Automatically generate a 'test' metadata database (0 for no, 1 for yes) [0]: Enter the filename of the XML file [/etc/perfsonar/store.xml]: Will this service register with an LS (0 for no, 1 for yes) [0]: 1 Interval between when LS registrations occur [in minutes] [60]: URL of an LS to register with [http://packrat.internet2.edu:8005/perfSONAR_PS/services/LS]: http://<YOUR_LOOKUP_SERVER>:8080/perfSONAR_PS/services/LS Enter a name for this service [SNMP MA]: Enter the service type [MA]: Enter a service description [SNMP MA]: <SERVICE DESCRIPTION> Enter the service's URI [http://localhost.localdomain:8080/perfSONAR_PS/services/snmpMA]: 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 4) Save configuration 5) Exit ? []: 4 Saved config to /etc/perfsonar/daemon.conf 1) Add/Edit endpoint 2) Enable/Disable port/endpoint 3) Set global values 25 4) Save configuration 5) Exit ? []: 5 <HOSTNAME> --- hostname or IP address of this server machine. <YOUR_LOOKUP_SERVER> --- hostname or IP address of the server which serve lookup service. The server you set up at chapter 4 is. <SERVICE DESCRIPTION> --- Description of the service 5.6.3 Setting for SNMP MA Set up the SNMP MA. Sample configuration is at Appendix-A # vi /etc/perfsonar/store.xml <Configuration> * Please refer to Appendix-A 5.6.4 run run the perfSONAR-PS daemon for SNMP MA # /sbin/service perfsonar-daemon start /usr/bin/perfsonar --config /etc/perfsonar/daemon.conf --piddir=/var/run --pidfile=perfsonar-daemon.pid --logger=/etc/perfsonar/logger.conf /etc/init.d/perfsonar-daemon start: perfsonar-daemon started 5.6.5 Check daemon Check perfSONAR-PS daemon is running successfully. # ps -ef | grep perfsonar root 27936 root 27937 27936 0 Apr23 ? 1 0 Apr23 ? 00:00:26 perfsonar.pl (27936) - Listener (8080) root 27938 27936 00:05:18 perfsonar.pl (27936) - LS Reaper 0 Apr23 ? 00:00:00 perfsonar.pl (27936) You can see processes above if perfSONAR-PS is running. 5.6.6 Firewall (iptables) Change the rule of firewall for perfSONAR-PS daemon. TCP 8080 is used for perfSONAR-PS daemon as you configured at section 5.6.2. # vi /etc/sysconfig/iptables <snip> -A RH-Firewall-1-INPUT -p tcp -m tcp --dport 8080 -j ACCEPT -A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited COMMIT # /sbin/service iptables restart 26 5.7 Check 5.7.1 psping psping is check tool whether perfSONAR services is available or not. psping can be installed by CPAN # cpan cpan> install perfSONAR_PS::Client::Echo <snip> /usr/bin/make install -- OK cpan>quit 5.7.2 check service Check whether perfSONAR-PS Lookup Service is available or not by psping. # psping http://<HOSTNAME>:8080/perfSONAR_PS/services/snmpMA Service http:// <HOSTNAME>:8080/perfSONAR_PS/services/snmpMA is up -Time to make request: 0.26802 * if SNMP MA is available, you can see above message. If not, you can see “down” instead of “up” on above message. 27 6 Throughput & Delay . (perfSONAR-PS perfSONAR-BUOY) The first step to installing these measurement tools is to install the Network Time Protocol (NTP). NTP get ntp-4.2 or later ./configure make make install Check version installed with OS and update with yum or up2date Edit /etc/ntp.conf (Must use at least 4 time servers in configuration - suggested time servers) server ntp-1.cso.uiuc.edu server ntp-2.cso.uiuc.edu server ntp-e.abilene.ucaid.edu server ntp-w.abilene.ucaid.edu chkconfig --add ntpd chkconfig --level 35 ntpd on sudo crontab -e # ntp time sync 0 2 * * * /usr/sbin/ntpdate -s -b -p 8 -u ntp-1.cso.uiuc.edu The current perfSONAR-BUOY implementation requires the Abilene Measurement Infrastructure (AMI). Create user and group ami /usr/sbin/useradd ami Create data directories (may need root or sudo access) 28 mkdir /home/ami/data/bwctl mkdir /home/ami/data/owamp Change owner for data directories chown –R ami /home/ami/data 6.1 Throughput (perfSONAR-PS perfSONAR-BUOY) This chapter describes how to provide throughput data by perfSONAR-PS. Target Server Hostname nms1.jp.apan.net BWCTL Get latest version of 1.3 RC of BWCTL from http://packrat.internet2.edu/~aaron/ gzip -cd bwctl-$VERS.tar.gz | tar xf cd bwctl-$VERS ./configure # --prefix defaults to /usr/local on most systems make make install cd conf cp bwctld.sh /etc/init.d/bwctld vi /etc/init.d/bwctld add PREFIX=/usr/local cp bwctld.conf /usr/local/etc cp bwctld.limits /usr/local/etc /sbin/chkconfig --add bwctld Edit /usr/local/etc/bwctld.conf Change user and group to ami Change vardir to /var/run (bwctld.pid location) Change datadir to /usr/local/data/bwctl Change peer_ports to range, example 35000-35005 29 Edit /usr/local/etc/bwctld.limits (See http://e2epi.internet2.edu/npw/binder-docs/bwctl-cookbook.pdf for more details) Open firewall ports for bwctl -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8423 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 35000:35005 -j ACCEPT Open firewall port for iperf -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 5001 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport ACCEPT Start bwctld /etc/init.d/bwctld start 6.2 Delay (perfSONAR-PS perfSONAR-BUOY) This chapter describes how to provide delay data by perfSONAR-PS. Target Server Hostname nms4.jp.apan.net OWAMP Get latest version of OWAMP from http://e2epi.internet2.edu/owamp/download.html gzip -cd owamp-$VERS.tar.gz | tar xf cd owamp-$VERS ./configure # --prefix is only needed if you don't like the default # (/usr/local on most systems) make make install cd conf 30 5001 -j cp owampd.conf /usr/local/etc cp owampd.limits /usr/local/etc Edit /usr/local/etc/bwctld.conf Change user and group to ami Change vardir to /var/run (bwctld.pid location) Change datadir to /home/ami/data/owamp Change testports to range, example 6000-6010 Edit /usr/local/etc/owampd.limits (See http://e2epi.internet2.edu/npw/binder-docs/owamp-cookbook.pdf for more details) Open firewall ports -A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 861 -j ACCEPT -A RH-Firewall-1-INPUT -m state --state NEW -m udp -p udp --dport 6000:6010 -j ACCEPT Start owampd /usr/local/bin/owampd –c /usr/local/etc/ Install the Abilene Measurement Infrastructure (AMI) There are two types of machines involved in the AMI: Beacon – Machines running OWAMP, BWCTL, bwmaster, and powmaster. Collector – Machines running MySQL, bwcollector, and powcollector. The collector may also run a web server for data presentation. Download the AMI binary http://dc211.internet2.edu/perfSONAR-PS/AMI-binary-x86.tar.gz gzip -cd AMI-binary-x86.tar.gz | tar xf - 31 from cd AMI-binary-x86 Edit the owmesh.conf file. There are 'XXX's in this file at locations that should be examined. The most important things to note are the names of the nodes, the members of the meshes, locations of the data, and database nuances. Example: OWPBinDir /usr/local/bin OwampdVarDir /var/run/ OWPDataDir /home/ami/owamp BWBinDir /usr/local/bin BWDataDir /home/ami/bwctl <HOST=nms1.transpac2.net> NODE NODE1 </HOST> <HOST=nms1.jp.apan.net> NODE NODE2 </HOST> Copy owmesh.conf to /usr/local/etc/ on all machines (beacon and collectors). Copy bwmaster and powmaster to /usr/local/bin on each beacon. Copy create_bwdb and create_owampdb and bwcollector and powcollector to /usr/local/bin on the collector. Start the beacons. /usr/bin/powmaster -c /usr/local/etc /usr/bin/bwmaster -c /usr/local/etc Check for data in /home/ami/owamp/ and /home/ami/bwctl/ 32 Run create_bwdb and create_owampdb on the collector to create the databases: /usr/local/create_bwdb -c /usr/local/etc/owmesh.conf -i ADMIN_USER /usr/bin/create_owampdb -c /usr/local/etc/owmesh.conf -i ADMIN_USER Install the following on the collector: perl -MCPAN -e 'install DBI' perl -MCPAN -e 'install DBD::mysql' Start the collector scripts: /usr/bin/powcollector -c /usr/local/etc/owmesh.conf /usr/bin/bwcollector -c /usr/local/etc/owmesh.conf To see if the collector has data check the AMI archive folder: find /home/ami/bwctl/archive Check the database: mysql> select * from BW_BWTCP4_NODE1_NODE2 Download the perfSONAR-BUOY package from http://dc211.internet2.edu/perfSONAR-PS/perfSONAR-PS_perfSONAR-BUOY_MA-bin ary-x86.tar.gz Under this directory in /usr/local. Edit the 'daemon.conf' # If you want to change the port, change it here <port 9099> <endpoint /perfSONAR_PS/services/pSB> module disabled perfSONAR_PS::Services::MA::perfSONARBUOY 0 <perfsonarbuoy> legacy 1 enable_registration metadata_db_file 0 ./store.xml metadata_db_type file 33 # XXX # Location (directory only) of the owmesh.conf file owmesh . # XXX # Change this to your public port/hostname service_accesspoint http://yourhostname:9099/perfSONAR_PS/services/pSB service_description service_name service_type Internet2 perfSONARBUOY MA perfSONARBUOY MA MA </perfsonarbuoy> </endpoint> </port> disable_echo 0 #ls_instance http://yourLS:8005/perfSONAR_PS/services/LS ls_registration_interval max_worker_lifetime max_worker_processes pid_dir . pid_file ps.pid reaper_interval 60 300 30 20 Open firewall port 9099 Run it for the first time to check things: ./pSBMA --config=daemon.conf --logger=logger.conf --verbose You may see this sql errors but this is ok. Check that the 'store.xml' file is full of the pairs of your mesh. Ctrl-c to stop Edit the following in logger.conf. # XXX 34 # Comment this line after testing, and uncomment the next log4perl.appender.A1=Log::Dispatch::Screen # XXX # uncomment after testing #log4perl.appender.A1=Log::Dispatch::FileRotate Start as a daemon: ./pSBMA --config=daemon.conf --logger=logger.conf 35 Appendix-A Sample Configuration of SNMP MA This is sample configuration(store.xml) of SNMP MA. <?xml version="1.0" encoding="UTF-8"?> <nmwg:store xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/" xmlns:neterr="http://ggf.org/ns/nmwg/characteristic/errors/2.0/" xmlns:netdisc="http://ggf.org/ns/nmwg/characteristic/discards/2.0/" xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/" xmlns:snmp="http://ggf.org/ns/nmwg/tools/snmp/2.0/" xmlns:nmtm="http://ggf.org/ns/nmwg/time/2.0/"> *1 *2 <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="m-in-netutil-1"> <netutil:subject xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/" id="s-in-netutil-1"> <nmwgt:interface xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/"> <nmwgt:ifAddress type="ipv4">127.0.0.1</nmwgt:ifAddress> *3 <nmwgt:hostName>tpr5.jp.apan.net</nmwgt:hostName> <nmwgt:ifName>so-1/1/0.0</nmwgt:ifName> *4 <nmwgt:ifIndex>2</nmwgt:ifIndex> <nmwgt:direction>in</nmwgt:direction> *5 <nmwgt:capacity>1000000000</nmwgt:capacity> </nmwgt:interface> *6 *7 </netutil:subject> <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType> </nmwg:metadata> *8 <nmwg:data xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="d-in-netutil-1" metadataIdRef="m-in-netutil-1"> *1 <nmwg:key id="k-in-netutil-1"> *9 <nmwg:parameters id="pk-in-netutil-1"> *10 <nmwg:parameter name="eventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter> <nmwg:parameter name="eventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> <nmwg:parameter name="type">rrd</nmwg:parameter> <nmwg:parameter name="file">/var/www/cacti/rra/router_traffic_in_10.rrd</nmwg:parameter> <nmwg:parameter name="valueUnits">Bps</nmwg:parameter> *11 <nmwg:parameter name="dataSource">traffic_in</nmwg:parameter> </nmwg:parameters> *12 </nmwg:key> </nmwg:data> <nmwg:metadata xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="m-out-netutil-1"> <netutil:subject xmlns:netutil="http://ggf.org/ns/nmwg/characteristic/utilization/2.0/" id="s-out-netutil-1"> <nmwgt:interface xmlns:nmwgt="http://ggf.org/ns/nmwg/topology/2.0/"> <nmwgt:ifAddress type="ipv4">127.0.0.1</nmwgt:ifAddress> <nmwgt:hostName>tpr5.jp.apan.net</nmwgt:hostName> <nmwgt:ifName>so-1/1/0.0</nmwgt:ifName> <nmwgt:ifIndex>2</nmwgt:ifIndex> <nmwgt:direction>out</nmwgt:direction> <nmwgt:capacity>1000000000</nmwgt:capacity> </nmwgt:interface> </netutil:subject> <nmwg:eventType>http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:eventType> <nmwg:eventType>http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:eventType> </nmwg:metadata> 36 <nmwg:data xmlns:nmwg="http://ggf.org/ns/nmwg/base/2.0/" id="d-out-netutil-1" metadataIdRef="m-out-netutil-1"> <nmwg:key id="k-out-netutil-1"> <nmwg:parameters id="pk-out-netutil-1"> <nmwg:parameter name="eventType">http://ggf.org/ns/nmwg/tools/snmp/2.0</nmwg:parameter> <nmwg:parameter name="eventType">http://ggf.org/ns/nmwg/characteristic/utilization/2.0</nmwg:parameter> <nmwg:parameter name="type">rrd</nmwg:parameter> <nmwg:parameter name="file">/var/www/cacti/rra/router_traffic_in_10.rrd</nmwg:parameter> <nmwg:parameter name="valueUnits">Bps</nmwg:parameter> <nmwg:parameter name="dataSource">traffic_out</nmwg:parameter> </nmwg:parameters> </nmwg:key> </nmwg:data> </nmwg:store> Yellow box is a set of configuration for interface usage data of input. There are 2 parts. The part quoted by <nmwg:metadata> tag is a metadata information of interface usage data. The second part quoted by <nmwg:data> tag is data information. Second part of configuration, part without yellow box, is for output of same interface. Red box means the parameter that would be modified depend on your environment. The explanatory note is there below. Same parameters of configuration for output also would be modified. *1 --- unique id for metadata. Example id is “m-in-netutil-1”. Increment the number when other interfaces are configured. *2 --- unique id for subject of metadata. Example id is “s-in-netutil-1”. Increment the number when other interfaces are configured. *3 --- IP address of interface. If no IP address is allocated, put blank. *4 --- hostname or IP address of the router *5 --- interface name of target interface. *6 --- interface index of target interface. *7 --- interface capacity, interface speed of target interface *8 --- unique id for data. Example id is “d-in-netutil-1”. Increment the number when other interfaces are configured. *9 --- unique id for key of data. Example id is “k-in-netutil-1”. Increment the number when other interfaces are configured. 37 *10 --- unique id for parameter key of data. Example id is “pk-in-netutil-1”. Increment the number when other interfaces are configured. *11 --- rrd file stored interface usage data. *12 --- data source name of rrd file. Cacti’s data source name is “traffic_in” for input, “traffic_out” for output. You can check *3,5,6,7 information by SNMP from agent host (Cacti). # snmpwalk -c mrtg -v 2c tpr5 IP-MIB::ipAdEntIfIndex IP-MIB::ipAdEntIfIndex.10.0.0.4 = INTEGER: 14 IP-MIB::ipAdEntIfIndex.117.103.111.133 = INTEGER: 70 *6 interface index IP-MIB::ipAdEntIfIndex.127.0.0.1 = INTEGER: 16 IP-MIB::ipAdEntIfIndex.133.69.37.1 = INTEGER: 75 IP-MIB::ipAdEntIfIndex.192.203.116.146 = INTEGER: 29 IP-MIB::ipAdEntIfIndex.202.179.241.37 = INTEGER: 34 IP-MIB::ipAdEntIfIndex.203.178.133.141 = INTEGER: 53 IP-MIB::ipAdEntIfIndex.203.181.194.89 = INTEGER: 74 IP-MIB::ipAdEntIfIndex.203.181.194.125 = INTEGER: 35 IP-MIB::ipAdEntIfIndex.203.181.194.158 = INTEGER: 63 *3 IP address IP-MIB::ipAdEntIfIndex.203.181.194.198 = INTEGER: 26 IP-MIB::ipAdEntIfIndex.203.181.194.246 = INTEGER: 54 IP-MIB::ipAdEntIfIndex.203.181.194.254 = INTEGER: 50 # snmpwalk -c mrtg -v 2c tpr5 IF-MIB::ifDescr IF-MIB::ifDescr.1 = STRING: fxp0 *6 interface index IF-MIB::ifDescr.2 = STRING: fxp1 IF-MIB::ifDescr.4 = STRING: lsi IF-MIB::ifDescr.5 = STRING: dsc IF-MIB::ifDescr.6 = STRING: lo0 <snip> IF-MIB::ifDescr.21 = STRING: lo0.16385 *5 interface name IF-MIB::ifDescr.22 = STRING: ge-0/0/0 IF-MIB::ifDescr.23 = STRING: ge-0/1/0 IF-MIB::ifDescr.24 = STRING: ge-1/0/0 IF-MIB::ifDescr.25 = STRING: so-1/1/0 IF-MIB::ifDescr.26 = STRING: ge-0/0/0.15 # snmpwalk -c mrtg -v 2c tpr5 IF-MIB::ifSpeed IF-MIB::ifSpeed.1 = Gauge32: 100000000 *6 interface index IF-MIB::ifSpeed.2 = Gauge32: 100000000 <snip> IF-MIB::ifSpeed.18 = Gauge32: 1000000000 *7 capacity IF-MIB::ifSpeed.21 = Gauge32: 0 (* 4294967295 means 10000000000) IF-MIB::ifSpeed.22 = Gauge32: 4294967295 IF-MIB::ifSpeed.23 = Gauge32: 4294967295 IF-MIB::ifSpeed.24 = Gauge32: 4294967295 38 You can check *11 information on Cacti management page. Select [Data Sources] and click the router interface you want know the rrd file name. See at Data Source Path. rrd file name for the interface usage data is written. Defalt <path_rra> is “/var/www/cacti/rra”. You can check *12 information by RRDtool fetch command. rrdtool fetch /var/www/cacti/rra/router_traffic_in_10.rrd AVERAGE | head traffic_in traffic_out 1217595000: nan nan 1217595300: nan nan 1217595600: nan nan 1217595900: nan nan 1217596200: nan nan 1217596500: nan nan 1217596800: nan nan 1217597100: nan nan 39