CERN openlab Summer Student Report 2015

CERN openlab Report 2015
This report presents the work performed in the context of developing a flow visualisatoin framework to enable real-time monitoring of multiple flows while testing network devices. For that purpose the project is divided in different stages of execution. The first one consists of specifications of the traffic generator and collecting statistics. The script is written in Python and iperf is used as a tool for active measurements of the maximum achievable bandwidth on IP networks. On the second stage the collected statistics from iperf has to be stored in the database with PostgreSQL 9.4 and the Psycopg2 DB API 2.0 compliant PostgreSQL driver. The final phase includes the visualization of the stored data using REST and JavaScript as a technology.
The results from this report indicate a successful implementation of the first two stages, and outline the future development required for the final stage.

CERN openlab Report 2015
The CERN Data Centre is the heart of CERN’s entire scientific, administrative, and computing infrastructure. All IT services use equipment located in the data centre. Because of its significant role, the performance of the devices deployed in the network is essential for providing high quality network services.
In order to provide a reliable and high-performance network the performance and limitations of each device have to be checked. The Flow Visualizer project consists in developing a flow visualization framework, which enables real-time monitoring of the flows generated by a distributed traffic generator and allows visualizing the monitored information at different levels of detail.

CERN openlab Report 2015
Abstract ........................................................................................................................... 3
1 Introduction .............................................................................................................. 5
2 Current test environment architecture ...................................................................... 6
3 Flow Visualizer architecture...................................................................................... 7
3.1
Stages of development ................................................................................................ 8
3.1.1
Traffic generator specifications and collecting statistics .................................. 8
3.1.2
Storing collected statistics in the database .................................................... 10
3.1.3
Visualization (future development) ................................................................ 10
4 Summary................................................................................................................ 11

CERN openlab Report 2015
The CERN Data Centre is a significant part of CERN’s entire scientific, administrative, and computing infrastructure. All the IT services, including email, scientific data management and videoconferencing use state of the art equipment based in the Data
Centre. Considering the importance of the Data Centre the performance of the devices deployed in the network is essential for providing high quality network services: running the computing facilities and services of the laboratory as well as transferring the colossal amount of LHC data to and from the Data Centre.
In consideration of providing high quality and reliability of the network the capabilities and the limitations of the network devices have to be tested. That is the reason why network performance tests are executed, which emulate real-life Data Centre traffic patterns. These tests evaluate the throughput and the fairness with which a network device (denoted as DUT – Device Under Test). The typical test setup is depicted in
Figure 1. All the nodes in cluster A communicate with all the nodes in cluster B
establishing multiple connections through the DUT. A set of custom scripts is used to inject traffic consisting of a large number of TCP flows (TCP is the transport protocol used by the vast majority of applications running in CERN’s Data Centre).
Figure 1 Device Under Test (DUT) test setup.
The traffic generation tool relies on the ‘iperf’ program and allows generating traffic with
a configurable distribution (see Figure 2):
 non-meshed (pairs) – any node from pool A connects to only one node from pool
B;
5 | P a g e F l o w V i s u a l i z e r

CERN openlab Report 2015
 partially meshed - any node from pool A connects to all of the nodes from pool B; and
 fully meshed - any node communicates with every other node in the test environment regardless of their pools.
Figure 2 Traffic distribution
Additional configurable parameters are the connection count (the number of connections established between two nodes) and the traffic orientation (can be adjusted to be unidirectional or bidirectional).
The current test environment architecture is shown in Figure 3.
6 | P a g e
Figure 3 Traffic generator current architecture.
F l o w V i s u a l i z e r

CERN openlab Report 2015
The test environment enables to provision two pools of twenty servers each, which are called nodes. An AGENT module is deployed on every node and it handles traffic generation and simple reporting tasks. The whole test is orchestrated by the CENTRAL module, which handles the test logic. Both components are written in Python.
The agent’s main purpose is to enable execution of remote commands (i.e. running traffic). The communication with the agent is established using XMLRPC (Python's standard xmlrpclib module).
The CENTRAL module is the test environment’s core. It coordinates the whole test scenario (i.e. calculating the pairs of nodes to run traffic between) and pushes the remote command execution to the agents. The CENTRAL module provides a convenient CLI which can be used to run the test.
In terms of monitoring, the current test environment can perform only the display of global interface bitrates. A more detailed look into individual traffic flows performance is needed in order to fully understand the performance and the limitations of the DUT, notably the fairness with which multiple TCP flows are treated.
In order to provide an improved monitoring solution we need to develop a flow visualization framework, which enables real-time monitoring of flows (e.g. throughput) and allows visualizing the monitored information at different level of detail. The general
concept of the Flow Visualizer framework is described in Figure 4.
Figure 4 Flow Visualizer architecture.
Figure 4 depicts one AGENT component running the network traffic generator program
based on ‘iperf’. The AGENT collects detailed flow statistics from the iperf traffic generator (source IP, destination IP, flow ID, bandwidth and timestamp) and pushes those
7 | P a g e F l o w V i s u a l i z e r

CERN openlab Report 2015 into a database table using PostgreSQL. Finally using REST technology and JavaScript the data get visualized.
The project was divided into the following stages, which will be detailed further in this section:
1.
Traffic generator specifications and collecting statistics – (specifying the desired output from iperf)
2.
Storing collected statistics in the database
3.
Visualization (future development)
3.1.1 Traffic generator specifications and collecting statistics
To test the network performance we have decided to use iperf as an open source tool developed by NLANR/DAST for measuring maximum TCP and UDP bandwidth. It is user friendly, platform independent and supports binding to a specific interface (both sender and receiver).
Iperf uses a client/server model, where traffic is initiated from the client and traverses the network to the server. When testing bandwidth in both directions we will need to run the test twice, once in each direction. By default the server will listen for the client on TCP port 5001 but this can be changed.
3.1.1.1 Setting up an iperf client and iperf server
1) Iperf server (setting up on lxbsu0504)
The -s command designates this workstation as the “server” and the -i 1 command sets the console output interval to one second.
2) Iperf client root#iperf - c lxbsu0505 -p 5001 -t 10 -P -i 1
The parameters, which we specified for our tests are the following:

-c – hostname

-p – port

-t – timestamp

-P – connection count
8 | P a g e F l o w V i s u a l i z e r

CERN openlab Report 2015

-i – interval

-y - csv output format
The -c lxbsu0505 command designates this workstation as the “client” and the IP address of the server and -p 5001 selects the port to be 5001. The -t 10 option sets the test duration to 10 seconds. The -P specifies the number of parallel client threads to run
(connection count). From the image below it’s clear that the test was run (from client to server) displaying the transfer rate (MB) and bandwidth (Mb) performance every second.
3.1.1.2 Collecting statistics
After adding the -y parameter and executing this command line: root#iperf - c lxbsu0505 -p 5001 -t 10 –P 2 -i 1 -y C
The statistics are outputted in csv (Comma-Separated Values) format include all the information required for precisely monitoring individual flows (bandwidth, the timestamp, the flow id, the interval, the receiver ip, the receiver port, the sender ip and the sender port).
9 | P a g e F l o w V i s u a l i z e r

CERN openlab Report 2015
3.1.2 Storing collected statistics in the database
At this stage of the project the AGENT has already the collected statistics from IPERF and has to push those to the database. For that purpose we have decided to use
PostgreSQL 9.4 as a technology.
After installing PostgreSQL we have created a database called ‘devdb’. Firstly we have used the graphical frontend tool pgAdminIII and the PostgreSQL interactive terminal program psql to create and manipulate the database. Considering the fact that we need to use Python to communicate with the PostgreSQL database we also need the Psycopg2 driver.
3.1.2.1 The specific use of Psycopg2
# import the adapter import psycopg2
#connect to the existing database conn = psycopg2.connect("dbname=devdb host=paolina-playground user=dev")
#open a cursor to perform database operations cur = conn.cursor()
# create a new table with the already specified parameters (3.1.1.1)
cur.execute("""CREATE TABLE stats ( bandwidth bigint, timest bigint, id bigint, interval varchar, receiver_ip varchar, receiver_port bigint, sender_ip varchar, sender_port bigint);""")
# make the changes to the database persistent conn.commit()
# insert the collected statistics into the created database table
# Note: “dictionary” contains the values to be inserted statement = """INSERT INTO stats (bandwidth, timest, id, interval, receiver_ip, receiver_port, sender_ip, sender_port) VALUES
(%(bandwidth)s, %(timest)s, %(id)s, %(interval)s, %(receiver_ip)s,
%(receiver_port)s, %(sender_ip)s, %(sender_port)s);""" log.debug('SQL statement: %s', statement % dictionary) cur.execute(statement, dictionary)
After executing the Python script the collected detailed flow statistics are stored into the database table. The next step is to get those statistics visualized.
3.1.3 Visualization (future development)
The visualization part of the project is still under development. The chosen technology for the development of the visualization module is REST (Representational State Transfer) and
JavaScript To ease the interpretation of the test results, the GUI (Graphical User Interface) should enable the display of statistics at different levels of granularity :

Total per node

Aggregated statistics (average and standard deviation) between each pair of nodes

Detailed time series for all the flows between each pair of nodes.
10 | P a g e F l o w V i s u a l i z e r

CERN openlab Report 2015
The purpose of the Flow Visualizer project was to develop a flow visualization framework, which enables real-time monitoring of flows and allows visualizing the monitored information at different level of detail. The project was divided into three stages.
In the first stage the task was to get precise flow statistics from the traffic generation. To test the network performance we have decided to use IPERF as an open source tool for measuring TCP bandwidth. We have set up an iperf server and an iperf client on the
AGENT module and specified the needed parameters for the output from iperf in order to get information about the bandwidth performance, source ip, destination ip, flow id and timestamp. The statistics were collected in Comma-Separated Values format.
Afterwards in the second stage we had to store the collected statistics from iperf in a database. For that purpose we have chosen the PostgreSQL 9.4 technology as an opensource object-relational database management system. Firstly the database was created and manipulated through the graphical frontend tool pgAdminIII and the PostgreSQL interactive terminal program psql. In order to use Python to communicate with the database we had to include also the Psycopg2 DB API 2.0 compliant PostgreSQL driver.
This enabled pushing detailed flow statistics to the database, directly from Python code.
Finally the Visualization component has to be developed. Its task is to visualize the collected flow statistics from the database at different levels of detail. The technology for that purpose is REST and JavaScript.
11 | P a g e F l o w V i s u a l i z e r