TBI: End-to-End Network
Performance Testbed for
Empirical Bottleneck Detection
Prasad Calyam,
OARnet, A Division of The Ohio Supercomputer Center,
The Ohio State University
TRIDENTCOM, Trento, Italy, February 2005
Dima Krymskiy, Mukundan Sridharan, Paul Schopis
1
Topics of Discussion
Third Frontier Network Measurement Project
Basics of Network Measurement
Infrastructures (NMIs)
“ActiveMon” NMI Software
TFN Beacon Infrastructure (TBI) Testbed
End-to-End Network Performance Bottleneck
Detection Case Studies!
Work in progress…
Conclusion
2
Third Frontier Network
The Third Frontier Network (TFN)
funded by the Ohio Board of Regents
A dedicated high-speed fiber-optic network
linking Ohio colleges and universities with
research facilities to promote research and
economic development
Over 1,600 miles of fiber has been
purchased to create a network backbone in
Ohio to connect colleges and universities,
K-12 schools, and communities together
3
Third Frontier Network (2)
4
TFN Measurement Project
Started in early 2004
Project funding from the Ohio Board of Regents
To ensure that University campuses can effectively
use the advanced networking services the new
network provides
Project Partners
OARnet (Project Lead and Co-ordination)
University of Cincinnati, Cincinnati State, The Ohio
State University, Kent State University, Southern
State Community College, University of Toledo,
Wright State University
5
Project Goals
Identify end-to-end performance bottlenecks in the TFN on
an ongoing fashion by building a comprehensive Network
Measurement Infrastructure (NMI)
Test new and advanced technologies and equipment
before wide-scale adoption in the TFN Higher Education
communities
Technologies: H.323/SIP based Voice and
Videoconferencing, MPEG3, HDTV, Multicast, Bulk FTP
Equipment: Video streaming Caches, Firewalls, Intrusion
Detection Systems, Traffic shapers
Bring awareness and train campus-networking
professionals to make optimum use of the capabilities of
TFN so that their campus network infrastructures can be
upgraded suitably
6
Project Goals
Identify end-to-end performance bottlenecks in the TFN on
an ongoing fashion by building a comprehensive Network
Measurement Infrastructure (NMI)
Test new and advanced technologies and equipment
before wide-scale adoption in the TFN Higher Education
communities
Technologies: H.323/SIP based Voice and
Videoconferencing, MPEG3, HDTV, Multicast, Bulk FTP
Equipment: Video streaming Caches, Firewalls, Intrusion
Detection Systems, Traffic shapers
Bring awareness and train campus-networking
professionals to make optimum use of the capabilities of
TFN so that their campus network infrastructures can be
upgraded suitably
7
What constitutes the
“infrastructure”in an NMI?
Measurement Toolkit hosted on “Beacon”
servers
Scheduler for measurement orchestration
between multiple measurement servers
Large database for storage and archiving
measurement data
Analysis Engine and web service for
visualization of network performance data
AAA procedures for handling Security/Privacy
Issues
8
A Typical NMI
9
Examples of NMI Software and
Deployments…
NIMI (Developed by Vern Paxson, currently
maintained by NLANR)
Surveyor (Developed by Advanced, currently
maintained by Univ. Of Wisconsin)
AMP (NLANR)
IEPM-BW (SLAC)
Scriptroute (Univ. of Washington)
E2E piPES (Internet2)
Many Many More… (ETOMIC, D-ITG, …)
ActiveMon (Being developed by OARnet)
10
“ActiveMon” NMI Software
An “application-specific” active measurement toolkit
To identify bottlenecks pertaining to specific applications;
e.g., Voice and Video over IP (Delay, Jitter, Loss,
Reordering), Bulk FTP flows (Throughput,
Available/Bottleneck Bandwidth, Web server/ Oracle server
response times, etc…
An advanced active measurements scheduler”OnTimeMeasure”
To orchestrate and regulate network-wide active
measurements
An analysis engine with web interface
To correlate performance along multiple paths
To generates appropriate alerts for concerned personnel!
Support for federated measurements…
To support end-to-end performance debugging along
multiple ISP domains
11
ActiveMon Measurement Toolkit
Tools used for end-to-end network performance data
collection










Ping (Delay) Open Source
Traceroute (Topology) Open Source
Iperf (TCP/UDP Bandwidth Performance) Open Source
Pathchar (Hop-by-Hop performance) Open Source
Pathload (Available Bandwidth) Open Source
Pathrate (Bottleneck Bandwidth) Open Source
OWAMP (µs Precision Delay – CDMA+NTP) Open Source
H.323 Beacon (Voice/Video Performance) Open Source
appareNet (Hop-by-Hop Performance) Commercial
NetQoS (Application Response Times) Commercial
12
“OnTimeMeasure” Scheduler
Active measurements can encroach network
bandwidth required for actual application traffic
Active measurements need to be regulated
Running multiple simultaneous measurements on
monitoring probes could result in misleading reports
of network performance
Active measurements require dedicated system and network
resources
Network Interface Card (NIC), CPU processing,
Application ports, Multimedia codecs, Bandwidth, …
Active measurements between measurement beacons need
to be orchestrated
13
Results in a LAN with WAN Emulation!
14
OnTimeMeasure Framework
15
OnTimeMeasure Framework
16
ActiveMon Analysis Engine
To retrieve relevant “summary-views” of the large
network-wide active measurement datasets
“Weather-map” functionality, Query-able XML schemas, …
To effectively identify anomalies and alert relevant
support and operations personnel
Target for a low probability of false-alarms
Cover anomalies that indicate better/poor/marginal changes
To perform multi-path data correlation to isolate
performance problems involving multiple-links
Observed end-to-end performance is a function of
performance of individual intermediate links
17
Example: Alarm check to notify
network health status
Watermarks for “Good”, “Acceptable” and
“Poor” grade of audiovisual quality as
experienced by end-user
Delay: (0-150)ms, (150-300)ms, > 300ms
Jitter: (0-20)ms, (20-50)ms, > 50ms
Loss: (0-0.5)%, (0.5-1.5)%, >1.5%
Prasad Calyam, Mukundan Sridharan, Weiping Mandrawa,
Paul Schopis, “Performance Measurement and Analysis
of H.323 Traffic”, Proceedings of Passive and Active
Measurement Workshop (PAM), 2004
18
TFN Beacon Infrastructure (TBI) Testbed
19
TFN Beacon Infrastructure (TBI) Testbed
20
Paths being Measured
21
Purpose of TBI Testbed
To understand network end-to-end performance
characteristics
Via partial path and intermediate bottleneck hop analysis
To understand network performance measurement
data reported by various tools
How good are they to empirically correlate network events in
a routine monitoring infrastructure?
To compare performance at campus, regional,
national-academic and national-commerical
backbone network levels
To quantify end-to-end network performance stability in the
Internet
22
End-to-End Network Performance
Bottleneck Detection Case Studies!
Anomalies in measured paths…
month period)
(2-
Effects of route changes
Network device misconfiguration
Misrepresentation of network health by
measurement tool misconfiguration
Performance comparison of campus,
regional, national-academic and nationalcommercial backbone networks using H.323
Beacon
23
Effects of route changes
RTT OSUB to OSUL
Jitter OSUB to OSUL
(both directions)
RTT OSUL to OSUB
24
Effects of route changes (2)
RTT OSUB to UOCB
RTT UOCB to OSUB
25
Network device misconfiguration
Available Bandwidth OSUB to UOCB
26
“Ocean Wave” Anomaly of OWAMP
RTT UOCB to OSUB
27
H.323 Beacon*
An application-specific measurement tool
To monitor and qualify the performance of H.323
Videoconferencing sessions at the host and in the network
(end-to-end)
Useful to an end-user/conference operator/network
engineer
Addresses problems due to H.323 protocol-specific
idiosyncrasies
Can be generalized to RTP packets performance over the
network
Many in-built tools that generate various kinds of
measurement data for pre/during/post Videoconference
troubleshooting!
An “easy to install and use” tool that is open source
* Project supported by Internet2, The Ohio Board of Regents, OARnet
28
A few H.323 Beacon screenshots…
http://www.itecohio.org/beacon
29
Performance comparison of academic and commercial
network backbones using H.323 Beacon
Most Stable
Least Stable
Academic networks most suitable for Voice
and Video over IP applications
Bottlenecks of multi-domain/last-mile links
mainly impact end-to-end performance
30
ActiveMon work in progress…
Extend testbed into a production-level NMI spanning
TFN and major Ohio-based University Campuses
Better multi-link data correlations in Analysis Engine
DDoS anomaly detection using active measurement
data signatures
OnTimeMeasure extensions using real-time
scheduling principles for various active measurement
specifications
Better visualization module…
Planning on ActiveMon 1.0 release on sourceforge
(Summer 2005)
http://sourceforge.net/projects/activemon/
31
Thanks!
Scripts Development and Data Analysis
Mukundan Sridharan, Dima Krymskiy, Phani Kumar Arava
Project Management
Steve Gordon, Paul Schopis
OSU Border and Lab Deployment
Prof. David Lee, Dave Kneisly, Arif Khan, Weiping Mandrawa
UC Border and Lab Deployment
Prof. Jerry Paul, Prof. Fred Annexstein, Bruce Burton, Bill Bohmer,
Tom Ridgeway, Michal Kouril, Diana Noelcke
NCSU Deployment
John Moore, Chintan Desai
Tools Deployment
Loki Jorgenson, Chris Norris (appareNet)
Jeff Boote (OWAMP)
Leandro Lustoza (H.323 Beacon E-Model implementation)
32
Questions?
TFN Measurement Project Reference:
http://tfn.oar.net/measurement
33