Campus Measurement
Matt Zekauskas, matt@internet2.edu
Internet2 Campus Workshop
Atlanta, GA
30 May 2001
Outline
Existing measurement tools & projects
A sample performance problem
Vision for infrastructure to solve problems
Steps campuses can take today
Campus Measurement
30-May-2001
2
My Bias
I once ran a (corporate) campus network
My recent focus: measurements
Most recently: end-to-end performance
I have been helping solve some widearea performance problems
Caveat: I haven’t been doing the fixing
(My view is from the center)
Campus Measurement
30-May-2001
3
We Need Your Help
I have probably missed something
Give us the benefit of your experience
• fixing problems
• operating a network
…also join a working group!
Campus Measurement
30-May-2001
4
Measurement Goals
Solving performance problems
Network operations
Network engineering
Network research
Operational data
• performance, flows, anomalies
Network characterization
• how used? load response? SLS?
Campus Measurement
30-May-2001
5
Measurements from the Center
Active
• Measurement within Abilene
• Measurement using entire Internet2 infra.
Passive
• SNMP stats (esp. core Abilene links)
• “IOS” stats (for QoS)
• Characterization of traffic (on the way)
–Netflow; OCxMON
Campus Measurement
30-May-2001
6
Measurement Projects
Surveyor (one-way delay, loss, routing)
• www.advanced.org/surveyor
• On many Internet2 campuses (70 sites)
• Abilene presence
AMP (round-trip delay, loss, routing)
• moat.nlanr.net/AMP
• At even more Internet2 campuses (120 sites)
PMA (passive, packet traces)
• moat.nlanr.net/PMA
• 1 min, 8 times a day, ~13 sites
Campus Measurement
30-May-2001
7
Measurement Projects
PingER (round-trip delay, routing)
• http://www-iepm.slac.stanford.edu/pinger/
• Long term data from a few locations to many
• High-energy physics focus
NIMI
• http://www.ncne.nlanr.net/nimi/
• Designed to be platform for experiments
• Undergoing some redesign/revitalization
• ~ 60 sites?
Campus Measurement
30-May-2001
8
Usefulness
AMP, Surveyor, Pinger
• If at your campus, a view from your campus
• If at destination, a view of destination
• Look for campus connected to same gigaPoP if
not at local or destination
• [“Phase 0” measurement points for e2eperf]
• Routing, congestion problems
Campus Measurement
30-May-2001
9
Usefulness
PMA
• If at your campus, can look at traces for
anomalies
• Not as useful for on-demand debugging
(but don’t ignore ability to take traces)
Campus Measurement
30-May-2001
10
Surveyor on One Slide
Continuous measurement
One-way delay and loss
1/sec on Poisson Schedule
12 Byte UDP packets
Traceroutes at 1/600 sec
72 Machines
• http://hartman.advanced.org/IPPMApplet/report/
Report.html -- Java, close to real-time
• http://ippm-db.advanced.org/plots/ -- static
Campus Measurement
30-May-2001
11
Campus Measurement
30-May-2001
12
Campus Measurement
30-May-2001
13
AMP
Like Surveyor, but
• Round-trip latency instead of one-way
–Easier to deploy
• Working on more comprehensive set of
“alarms”
• Potentially more available
Campus Measurement
30-May-2001
14
Campus Measurement
30-May-2001
15
An “Application-Level” Example
Pioneer
• http://pelle.internet2.edu:8080/pioneer/
• Synthesis of existing infrastructure
• Focus: video conferencing tests
• Goal: use this to tell if video likely to work
Campus Measurement
30-May-2001
16
Abilene
Abilene goal to be an exemplar
• Measurements open
• Tests possible to router nodes
• Web-mediated on-demand measurements
• Throughput tests routinely through
backbone
• …as well as existing utilization, etc.
Campus Measurement
30-May-2001
17
Active within Abilene
Each Router Node has a PC
Now 10 of 11 are OC3-ATM attached
• missing: Houston
No GPS
• working towards GPS within CDMA
solution
Campus Measurement
30-May-2001
18
Ad-hoc Active on Abilene
With OC-3, can do moderate throughput
testing (e.g., iperf UDP & TCP). ~90
Mbps
Adding on-demand tests in support of
performance debugging
Contact me (matt@internet2.edu) if you
want to perform an ad-hoc test
Campus Measurement
30-May-2001
19
Passive - Utilization
The Abilene NOC takes
• Packets in,out
• Bytes in,out
• Drops/Errors
• ..for all interfaces, publishes internal links
& peering points (at 5 min intervals)
• ..via SNMP polling – every 3 sec
http://hydra.uits.iu.edu/~abilene/traffic/
Campus Measurement
30-May-2001
20
Campus Measurement
30-May-2001
21
Campus Measurement
30-May-2001
22
Passive – Characterization
Some sparse via NLANR/MOAT
• http://moat.nlanr.net/PMA/
Starting some NetFlow measurements
• QoS
• AS-AS information for K-20 & ITN
• Intend to do some characterization
Campus Measurement
30-May-2001
23
Others via Abilene NOC
BGP Peering
MSDP (multicast source discovery)
logging
See: http://www.abilene.iu.edu/
-> Operational Status
Campus Measurement
30-May-2001
24
Multicast-specific
Multicast measurements
• Not fully understood
• Debugging is an art
Tools
• Mtrace
• ‘sdr’ announcements in backbone
• Mhealth, Mantra via UCSB
http://www.nmsl.cs.ucsb.edu/
http://www.cs.ucsb.edu/~almeroth/
Campus Measurement
30-May-2001
25
JPL/Caltech – GSFC
The situation
• Using Abilene
• Tuned hosts
• Things work locally
Therefore it MUST be Abilene
• Tests show good flows router-router
• Intermediate tests point towards CA
Bad fiber connection!
Campus Measurement
30-May-2001
26
Vision I
Ongoing monitoring to test major
elements, and (some, important) end-toend paths.
• Elements: gigaPoP links, peering, …
• Utilization
• Delay
• Loss
• Occasional throughput
• Multicast connectivity
Campus Measurement
30-May-2001
27
Vision II
There are many more paths end to end
than can be monitored.
Diagnostic tools available on-demand
(with authorization)
• Show routes
• Perform flow tests (perhaps app tests)
• Parse/debug flows (a-la tcpdump or
OCXmon with heuristic tools)
Campus Measurement
30-May-2001
28
For TCP (and Streaming)
Eliminating loss is the goal
Focus on noncongestive losses
TCP: 100 Mbit Ethernet coast-to-coast:
• Full size packets… need 10-6 Ploss [Mathis]
• Less than 1 loss every 83 seconds
http://www.psc.edu/~mathis/papers/JTechs200105/
GigE/655: 10-8, 1 loss every 497 seconds
Campus Measurement
30-May-2001
29
Enabling Divide & Conquer
and Ongoing Monitoring
Wall Jack
GigaPoP A
Backbone 1
Campus
P
P
Campus
Backbone 2
GigaPoP B
Wall Jack
Campus Measurement
30-May-2001
30
Some Commercial Tools
Caveat: only a partial list, give me more!
Spirent (nee Netcom/Adtech):
• working on a box for ‘end-to-end’ measurements
• SmartBits: test at low & high rates, QoS; test
components or end-to-end path
NetIQ: Chariot/Pegasus
Ixia (like SmartBits/Spirent); Agilent
Brix Networks (like Surveyor, for ‘QoS’)
Campus Measurement
30-May-2001
31
Some Noncommercial Tools
Iperf: dast.nlanr.net/Projects/iperf
• See also http://www-itg.lbl.gov/nettest/
Flowscan:
• http://www.caida.org/tools/utilities/flowscan/
• http://net.doit.wisc.edu/~plonka/FlowScan/
SLAC’s traceroute perl script:
• http://www.slac.stanford.edu/comp/net/wan-mon/traceroutesrv.html
One large list:
• http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html
Campus Measurement
30-May-2001
32
What You Can Do
Export SNMP data
• I can keep an “internet2 list”, would like it to be
public [Current Measurement WG project]
• Monitor loss as well as throughput
Performance test point at campus edge
• Netperf or iperf, so can be from anywhere
• Traceroute “looking glass”
• Commercial (e.g., NetIQ) complements
• I’m willing to keep a master list [MWG project]
Portable performance test point
Campus Measurement
30-May-2001
33
For TCP Tuning
Keep an eye out for Web100:
• http://www.web100.org/
NCNE Tuning Page:
• http://www.psc.edu/networking/perf_tune.html
• http://www.ncne.nlanr.net/research/tcp/
Campus Measurement
30-May-2001
34
What You Can Do
If you have a Cisco router at your edge,
use NetFlow and cflowd + FlowScan to
see your traffic characteristics
RTFM / RMON probes
See also Joe St.Sauver’s presentation
from the last “Joint Techs” meeting:
http://www.ncne.nlanr.net/training/techs/2001/0514/presentations
/200105-sauver1.html
Campus Measurement
30-May-2001
35
A Summer Project
Measurement box at edge
Spend month or two with mobile box,
checking throughput/loss/.. from every
point.
Eliminate noncongestive losses
Develop a baseline to get a complete
picture of the campus: map the campus
networks
Campus Measurement
30-May-2001
36
NTP everywhere!
If GPS, get good NTP distribution
Allow correlation among campuses
Campus Measurement
30-May-2001
37
Plug: Internet2 Measurement
Working Group Activities
Measurement architecture
Encourage common
• Measurements, tools
• Parameters
• Reporting
Work with (at least) management, QoS,
multicast; End-to-end Performance
Initiative
Campus Measurement
30-May-2001
38
Contact Information
Matt Zekauskas, matt@internet2.edu
Measurements Working Group
• http://www.internet2.edu/measurement/
End-to-end interest list
• listserv@internet2.edu
• subscribe e2e-interest
Campus Measurement
30-May-2001
39
(Some) URLs
Http://www.internet2.edu/measurement/
http://www.advanced.org/surveyor/
http://moat.nlanr.net/ http://dast.nlanr.net/
http://www.ncne.nlanr.net/ http://www.ncne.org/
http://www.caida.org/ http://www.web100.org/
http://www.auckland.ac.nz/net/Internet/rtfm/
http://www.slac.stanford.edu/xorg/icfa/ntf/home.html
http://www.merit.edu/ipma/
Campus Measurement
30-May-2001
40
www.internet2.edu