BANDWIDTH TEST CONTROLLER:
ANALYZING THE NEES CONSORTIUM NETWORK BANDWIDTH
Author:
Home Institution:
REU Host Institution:
REU Faculty Advisor:
Alek Harounian*
University of California, Los Angeles
NEES Cyberinfrastructure Center (NEESit)
San Diego Supercomputer Center
University of California, San Diego
Dr. Lelli Van Den Einde*
Abstract
NEESgrid is a network and a communications web that uses collaborative facilities and
telepresence technologies to connect earthquake engineering researchers together using
Internet2. In order to allow research facilities to collaborate together on experiments and
simulations, understanding the bandwidth between all of the facilities is crucial. This
project uses Bandwidth Test Controller (bwctl) and Bandwidth Test Controller daemon
(bwctld) to run 3-way network testing. These tests will help improve NEESit and its user
community’s understanding of their network’s behavior when running bandwidth
intensive programs.
This project initially involved learning the Linux command line, installing and
configuring bwctld, and performing 3-party testing on three NEESit Linux servers.
Ultimately, a test will be conducted from NEESit between the University of California,
Berkeley and the University of Buffalo. The results will provide information about the
amount of available bandwidth between these two facilities. With the correct
configuration for bwctld, informative data can be obtained that will help analyze the
NEESgrid network bandwidth to enable effective use of available computing resources.
Introduction
In the last several decades, computer networks have played an essential role in the world
of computing and telecommunications. The internet spread across the globe to create the
World Wide Web and has linked millions of computers through an electronic network.
Today, computer communication networks are being used to provide cutting-edge
network capabilities to support and improve both educational and research missions. As
a result, the importance of monitoring and analyzing these networks has also increased
significantly.
Internet2, the second generation of the Internet, is a high speed network developed
primarily for research. “It enables the research and education community to unlock new
frontiers in the development of true next-generation internet technologies by providing
reliable and cutting-edge network infrastructure. It exists to advance the mission of its
members and the broader research and education community.” (Dr. Larry Faulkner)
The Network for Earthquake Engineering Simulation (NEES) is a nationwide network of
15 experimental facilities, connected by means of Internet2 for the purpose of
communication and collaboration between these equipment sites. Funded by the National
Science Foundation, NEES endeavors to advance earthquake engineering research and
education by allowing earthquake engineers to share research data and, “develop better
and more cost-effective ways of mitigating earthquake damage through the innovative
use of improved designs, materials, construction techniques, and monitoring tools.”
(http://nees.org/About_NEES/) Thus, managing and analyzing this network, which is
used to run bandwidth intensive applications to transfer full-motion video and real time
data, is high priority.
Bandwidth Test Controller is an application developed by the Internet2 group to measure
the network bandwidth and throughput. The NEES Cyberinfrastructure Center (NEESit),
located at the University of California, San Diego, came to a decision to use this software
to monitor its network activity and control the bandwidth between the 15 facilities.
Research
Linux is a very popular open source operating system that supports different kinds of
hardware platforms. It is a powerful and secure operating system for servers that run the
networks. All server systems at NEESit are Linux based, thus the first step in this
research was to build an understanding of Linux, bash, and command line scripting.
Scripts were also used to automate some specific tasks and will be discussed later in this
paper.
The second step was to learn and utilize Iperf. Iperf is a command line tool that measures
the maximum TCP (Transmission Control Protocol) bandwidth. TCP is a transport
protocol within the internet protocol suite. Iperf is open source software written in C++
and copyrighted by the University of Illinois. The goal was to use Iperf to measure the
available network bandwidth between any two Linux servers at NEESit. This test
required both Linux servers to use Iperf; thus, Iperf version 2.0.2 was installed on two
different systems. Iperf then needed to be initialized as a server and a client respectively.
A server is a computer that provides services to other systems or clients in a network.
Figure 1 is a simple schematic describing the network setup for the Iperf test. During a
test, one server runs Iperf in server mode and the other runs Iperf in the client mode.
After doing the final run on the client the results are sent to the client side for further
study (Figure 2).
Figure 1: Iperf Test Schematic
Figure 2: Iperf Test Results
-----------------------------------------------------------Client connecting to neesdev4.sdsc.edu, TCP port 5001
TCP window size: 63.0 KByte (default)
-----------------------------------------------------------[1904] local 137.110.117.217 port 1799 connected with 198.202.90.49 port 5001
[ ID] Interval
Transfer
Bandwidth
[1904] 0.0- 5.0 sec 166 MBytes 279 Mbits/sec
As illustrated in the Figure 2, the client is connected to the TCP port 5001 on the server
and sends traffic to that specific TCP port. The amount of bandwidth from this specific
Iperf test is 279 Mega bits per seconds (Mbits/sec). “It is often useful to test to multiple
points along a network path to determine the network characteristics along that path.
Typically, users who want to do this path decomposition have to contact the
network/system administrators that control the hosts along the path directly. The
administrator needs to either run half of the test for the user or provide a user account on
the host. Also, network paths of interest usually are controlled by multiple administrators.
These difficulties have made this kind of testing difficult in practice. BWCTL is designed
to help with this problem. It allows an administrator to configure a given host as an Iperf
server that can be shared by multiple users without concern that those users will interfere
with each other. It is also possible to divide incoming server connections based upon a
user name and a hexadecimal key combination or, alternatively, based upon the client’s
network address.” (http://internet2.edu)
BWCTL must be installed on all nodes that are involved in the testing process. It can be
run in two modes, server or daemon, and client. In the simplest scenario (Figure 1), it is
possible to have one end run the daemon and the other run the client. The two end points
test is explained in an example below.
The goal of this example is to measure the network bandwidth in Figure 1. Consequently,
both downloading and uploading bandwidths need to be measured in this test. To do that,
system one is set as the daemon and machine two is configured as the client. Two
different tests are needed to calculate two download bandwidths and two upload
bandwidths. To clarify, from system one’s perspective there is one upload and one
download and from system two’s point of view there is one upload and one download. In
the first test, system one acts as a traffic catcher and system two acts as a sender. In the
second test these two machines switch their roles. Tests can be initiated from either
system one or two. Since most of the networking environments include more than two
machines, this testing method is not used frequently.
Figure 3 illustrates another testing scheme that includes more than two end points, i.e. all
nodes need to have the daemon. In Figure 3, system one can initiate a bandwidth test
between machines two and three and receive the end results (Figure 4). Also, all other
systems in the network can initiate similar tests between any two nodes inside the
network. The collected data can be used to analyze the network throughput and to prepare
a network weathermap for the entire network that will show bandwidth utilization by
drawing a graphical map of the network (Figure 5).
Figure 3: Test Schematic for Three Machines
Figure 4: Bandwidth Test Results
[alek@neesdev bin]$ ./bwctl -c neesesque4.sdsc.edu A AESKEY neesmonitor -s neesesque3.sdsc.edu
A AESKEY neesmonitor -t 4
Enter passphrase for host 'neesesque4.sdsc.edu', identity 'neesmonitor':
Enter passphrase for host 'neesesque3.sdsc.edu', identity 'neesmonitor':
bwctl: 11 seconds until test results available
RECEIVER START
3363701290.420807: iperf -B 198.202.90.56 -P 1 -s -f b -m -p 5001 -t 4 -i 1
-----------------------------------------------------------Server listening on TCP port 5001
Binding to local address 198.202.90.56
TCP window size: 87380 Byte (default)
-----------------------------------------------------------[ 14] local 198.202.90.56 port 5001 connected with 198.202.90.55 port 5001
[ 14] 0.0- 4.0 sec 117576568 Bytes 940612544 bits/sec
[ 14] MSS size 1448 bytes (MTU 1500 bytes, ethernet)
RECEIVER END
Figure 5: Network Weathermap for the Entire Network illustrating Bandwidth Utilization
Current Status and Future Plans
As all fifteen NEES equipment sites use this network to share their facilities, they also
employ telepresence technologies to connect earthquake engineering researchers together
using Internet2. So investigating the network bandwidth between all of the facilities is
important. To analyze NEESgrid’s bandwidth, a method very similar to the one used in
Figure 3 has been employed. Table 1 lists the 15 nodes inside the NEES communication
web.
Table 1: Fifteen Nodes inside NEES Communication Web
NEES Facility Site
State
University of California, Los Angeles
California
University of California, San Diego
California
University of California, Berkeley
California
University of California, Davis
California
University of California, Santa Barbara
California
Oregon State University
Oregon
University of Nevada, Reno
Nevada
University of Colorado at Boulder
Colorado
University of Texas at Austin
Texas
University of Minnesota
Minnesota
University of Illinois at Urbana Champaign
Illinois
University at Buffalo, SUNY
New York
Cornell University
New York
Rensselaer Polytechnic Institute
New York
Lehigh University
Pennsylvania
The tests will be executed and monitored by the NEESit team. To run the bandwidth test
all sites (Figure 5) need to install Iperf, BWCTL, and run the daemon. A software
package that includes Iperf, BWCTL, and necessary instructions was sent to all sites.
After installing the software, all 15 nodes will be ready for the bandwidth test. The test
between any two sites will be started at NEESit and the IT team will receive the results
for further analysis.
After installing BWCTL, any configuration changes should be done inside a
configuration file called bwctld.conf. “The bwctld.conf file is the configuration file for
the bwcltd daemon. It is used to configure basic operation of the server such as server
listening port, the path for Iperf, and error logging.” We can use any text editor to change
the content of the configuration file. Another text file which is used to configure the
authentication and authorization limits for the daemon is called bwctld.limits or the limits
file. Once the configuration file and the limits file are set we can start the test. In our test
we sent preconfigured limits file and configuration file to the equipment sites. In other
words, they only need to copy these two files in the specific directory on the server which
is running the daemon.
BWCTL uses Iperf to do the actual bandwidth testing and its output is very similar to
Iperf. In other words, it just encapsulates Iperf (Figure 4). (Additional information about
configuration file and the limits file can be found in the installation section.) The data that
comes out of the bandwidth test (Figure 4) are used to generate a weather map (Figure 5).
This is done by a Perl script and a small shell script that automates the process. The Perl
script is developed by Panagiotis Christias and its being used in Europe. It has been
modified at NEESit to address our needs for NEESgrid. This Perl script reads the
network’s data from an html file and represents them graphically on a PNG (Portable
Network Graphics) file (Figure 5).
The shell script runs bwctl between any two specified nodes and separates the bandwidth
value from the output using AWK (AWK and SED are traditional UNIX system tools
used for text manipulation) and sends it to a text file called bwctl-1.txt. Then it uses an
html file named rep.html as a model readable by the Perl script and enters the data from
bwctl-1.txt into the appropriate field inside the html file using SED and saves the new
html file as link-initial.html. Then it waits for a specific period of time (e.g. 5 minutes)
and runs the second test. Again it separates the bandwidth value from the output and
forwards it to bwctl-2.txt. It uses link1.html and enters the second value into that html file
using SED and saves it as link-final.html. This html file is saved in a directory that is
used by the Perl script. Perl script reads from link-final.html whenever is trying to update
the map. This process can be done for every single test between two facilities.
Once the bandwidth test controller is successfully installed at each site, NEESit will not
only be able to remotely initiate bandwidth testing between any two equipment sites but
also identify the best usage of network resources. Moreover, having a visual way of
presenting the NEESgrid and the available bandwidth between the sites is another step to
advance and ease the monitoring process of the NEES’ communication web.
Installation Guidelines
Hardware Requirements:
1. No strict requirements for CPU, memory, bus speed, NIC
2. Your hardware will dictate the possible intensity of tests you can perform
Network Requirements:
If you are working with firewalls, you will need to open the appropriate ports for
communication and testing:
1. TCP/4823 (Control communication – client to server)
2. TCP/ephemeral (Control communication – server to server: Specific range
settable using peer ports in bwctld.conf)
3. TCP/5001 (Iperf testing port – Settable to a range using test ports in bwctld.conf)
4. UDP/5001 (Iperf testing port – Settable to a range using test ports in bwctld.conf)
General Requirements:
1. Iperf version 2.0
2. NTP (ntpd) synchronized clock on the local system
3. NTP system calls
Note: “To get good results, the end hosts will almost certainly need to be well-tuned. The
bwctld daemon prefers an NTP synchronized system clock to ensure the two endpoints of
the test are actually agreeing to the same scheduled time window for test execution.
Therefore, BWCTL requires that NTP be running to synchronize the system clock. For
the NTP algorithms to work correctly, NTP must be configured with no fewer than four
(4) clocks. TCP ports need to be opened for control communication. Both from the client
to the server and for peer connections between the servers.”
(http://e2epi.internet2.edu/bwctl)
Authenticated communication is required for all tests (AES Keys).
Follow the instructions below to install and run the bwctld:
1. Download Iperf 2.0.2 source code from:
http://lss.rutgers.edu/index.php?page=tool_iperf
Install it in /home/user/bin directory (we are going to install bwctl in the same
directory later)
To install Iperf do the following:
a. wget http://lss.rutgers.edu/downloads/iperf-2.0.2.tar.gz
b. Decompress the file
c. Change directory to iperf-2.0.2 and run the command ./configure -prefix=/home/user
d. From the same path run make and then make install
2. Download bwctl-1.2a from:
http://e2epi.internet2.edu/bwctl/download/bwctl-1.2a.tar.gz
and install it in the same directory /home/user/bin
To install bwctl do the followings:
a. wget http://e2epi.internet2.edu/bwctl/download/bwctl-1.2a.tar.gz
b. Decompress the file
c. Change directory to bwctl-1.2a and run the command
d. ./configure --prefix=/home/user
e. From the same path run make and then make install
3. Go to /home/user/bwctl-1.2a/conf directory and copy the bwctld.conf file to
/home/user/bin directory. Then change directory to /home/user/bin.
“The bwctld.conf file is the configuration file for the bwcltd daemon. It is used to
configure basic operation of the server such as server listening port, the path for
Iperf, and error logging.”
4. Use a text editor to open the bwctld.conf file and do the following changes:
#user
user

user
enter the user name
Specifies the uid the bwctld process will run as.
#group
group

group
Specifies the gid the bwctld process will run as.
enter the group name
#var_dir
/var/run

var_dir
Directory where bwctld.pid file is stored.
/home/user/bin
#allow_unsync

allow_unsync
This will cause bwctld to blindly accept tests assuming the clock is synchronized
to within the sync_fuzz value that is also defined in the bwctld.conf file.
Note: These changes are for illustration purpose only.
5. Copy the bwctld.limits file from /home/user/bwctl-1.2a/conf directory to
/home/user/bin. Change Directory to /home/user/bin and use a text editor to open
this file. The bwctld.limits file is used to configure the authentication and
authorization limits for the daemon. An example of limits file could be:
#setup the root node with max bandwidth as open as possible
#this will set root as a parent limit class
#set bandwidth to 900 mbps
#Maximum duration of a single Iperf test for this user-class (0=unlimited)
#this will not allow UDP Iperf tests for this user-class
#this will allow TCP Iperf tests for this user class
#this will allow open authentication
limit root with \
bandwidth=900m, \
duration=0, \
allow_udp=off, \
allow_tcp=on, \
allow_open_mode=on
#minimal everything
#it gives all permissions of root to jail
#but it limits jail by setting specific limit types
#set bandwidth to 1 bps
#set duration of an Iperf test to 1 second
#this will not allow UDP Iperf tests for this user-class
#this will not allow TCP Iperf tests for this user-class
#this will not allow open authentication
limit jail with parent=root, \
bandwidth=1, \
duration=1, \
allow_udp=off, \
allow_tcp=off, \
allow_open_mode=off
#this set of assign lines specifically denies access from any open mode connection
from #the badguys subnet (IP/nbits). It specifically allows access to authenticated
or encrypted #mode
#transactions that can authenticate as the identities joe, jim or bob (even from the
#badguys #subnet).
#badguys subnet
assign net 192.168.1.0/24 jail
#network admins
assign user joe root
assign user jim root
assign user bob root
Note: Users are grouped into hierarchical limit classes the first limit line cannot
have a parent since none have been defined yet. As such, the first line defines the
root of your class hierarchy. All remaining limit lines MUST assign a parent. It is
hierarchical, after all. Lines that start with # mark will not be processed. (This is
true for both bwctld.conf and bwctld.limits files)
6. Do the necessary changes.
7. Go to the /home/user/bin directory and run this command:
./aespasswd –f /home/user/bin/bwctld.keys -n user
& enter the provided pass phrase.
8. Repeat all these steps on every machine that will be involved in the testing
process.
9. Activate the daemon on all systems using the command below:
./bwctld –c path of the configuration file
Note: for more help and information about the arguments run ./bwctld –h.
10. Run the bandwidth test by running this command:
./bwctl –c the catcher address A AESKEY username –s the sender address
A AESKEY username
Note: for more help and information about the arguments run ./bwctl –h.
Acknowledgements
Special thanks to Wei Deng who directed me in the learning process of Linux, shell
scripting, and Iperf.
I would also like to thank Dr. Lelli Van Den Einde and Tess Lacuesta who helped me by
giving feedback on this report paper and my presentation as well as Jason Gerk who
helped me to initiate some real bandwidth testing between the equipment sites.
References
http://www.internet2.edu/about/
http://nees.org/About_NEES/
http://e2epi.internet2.edu/bwctl/