Netlab, Caltech
Lachlan Andrew, George Lee,
Steven Low(PI), John Doyle, Harvey Newman

Outline





What and why is WAN-in-Lab?
What can I do with WiL?
Why would I use WiL?
How do I use WiL?
Future plans

What is WAN-in-Lab?

“Wide Area” Network in a laboratory
• Real fibre delays
• Carrier-class routers, switches, …

Why -- Spectrum of tools cost
UltraLight
PlanetLab
Abilene
NLR
LHCNet
CENIC etc
?
DummyNet
EmuLab
ModelNet
WAIL
NS2
SSFNet
QualNet
JavaSim
Mathis formula
Optimization
Control theory
Nonlinear model
Stocahstic model live netwk WANinLab emulation simulation maths abstraction
All scales are important– WAN-in-Lab fills a gap

What can I do with WAN-in-Lab?

Other groups’ interests




Protocol development
• FAST, delay-based
• MaxNet, explicit signalling
• ADPM, single-bit explicit signalling
Impact of small buffers (U. Pittsburgh)
Test automatic configuration of routers
(MonALISA, Ultralight)
Test distributed file-system (MojaveFS)

TCP Benchmarking


Our current main direction
Evaluating others’ protocols, not ours

Web interface
• Submit kernel patch
• Standard tests automatically performed
• Results mailed back

Explicit or implicit signalling protocols

Physical topology eth2
10.1.13.2
Svr11
Svr2
Eth2
10.2.12.2
Svr10
Eth2
10.2.21.2
Eth2
10.1.12.2
Svr9
Eth2
10.1.12.2
Svr1
Eth2
10.2.11.2
POS9/1
10.0.24.2
Eth2
10.2.24.2
POS1/1
10.0.12.2
RB
POS2/1
Gi2/2
10.0.23.2
10.2.22.1
RA
Eth??
10.1.51.2
Svr6
POS1/1
10.0.12.1
POS9/1
10.0.13.1
Disk2
POS1/1
10.0.23.3
Eth2
10.3.11.2
POS9/1
10.0.13.3
RC
Eth??
Disk1
Eth2
10.3.12.18
Svr7
Eth2
10.3.12.17
SA:gi1/0/12
Eth1
10.3.12.2
Disk3
Svr13
Svr14
POS1/1
10.0.24.4
POS1/1
RD
10.0.24.4
Disk4
Sonet (movable by MEMS)
Link via copper GbE switch
Direct fibre link
Direct copper GbE link
Wired in, but not yet configurable (network booted)

Capabilities: Delay



24 spools of 100km fibre, many loopbacks
• Set delay by MEMS switching loops in/out
130ms physical delay
• more with IP loopback
2 Dummynets: long delay for cross-traffic
125 ms, 1.8ms steps

External connections


Linked to Ultralight, 10Gbps Physics WAN
Smooth migration testing -> deployment


Delay
• longer
• jitter
Cross traffic

Monitor data routed through WiL

Why use WAN-in-Lab?

Why use WiL?

Complement other levels of abstraction, not replace them

Different ways to use it: reasons for each

Standard platform for TCP benchmarking
• Easier to compare with others’ results
• No need to write your own test suite

Artifacts of software delays


Packets sent on 1ms “ticks”
1Gbps = 83,333 pk/s
83 packets
1ms

How can I use WAN-in-Lab?

Management structure
Data plane
Wil-ns.cs.caltech.edu
Start script, configure, compile
Servers
Network boot
Read-only FS
virtual /etc
scratch disk

Time sharing




Coarse switching between projects
• Servers rebooted, routers reconfigured
Switchover takes ~5 minutes
Book in advance
• For longer bookings, book further in advance
• Also “ad hoc” bookings for individual hosts
Can log in while others have booked

Future plans

Future plans

Benchmarking infrastructure
• Standardise tests
• Use it ourselves
• Develop “indices” of TCP performance




Better control over capacities and buffers
Better cross-traffic generation
• Currently Harpoon
Investigate differences from DummyNet
Integrate DAG cards

Conclusion



WAN-in-Lab fills the gap between emulation and live network experiments
Seeks to be as realistic as possible
• Long links, simple topology
Focus will be on TCP benchmarking

We welcome people to use it
<http://wil.cs.caltech.edu>

Spare Slides

Case Study: MaxNet

Aim: Wind Tunnel of Networking






WAN in Lab
• Capacity: 2.5 – 10 Gbps
• Delay: 0 – 120 ms round trip
Breakable
• Won’t take down live network
Flexible, active debugging
• Passive monitoring, AQM
Configurable & evolvable
• Topology, rate, delays, route
• Modular design stays up to date
Integral part of R&A networks
• Transition from theory, implementation, demonstration, deployment
• Transition from lab to marketplace
Global resource
• Part of global infrastructure UltraLight led by Harvey Newman

Equipment






4 Cisco 7609 routers with OC48 line cards
6 Cisco ONS 15454 switches
A few dozen high speed servers
1G switch to routers/servers
Calient switch for OC48
2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules

63ms aggregate RTT delay, in two hops
• 120ms using IP loopbacks

Accounts


Mail wil at cs.caltech.edu
Sudo access to “network” commands
• Ifconfig/…/
• Custom commands to set topologies

Login to routers if required

Separate accounts for “benchmark only”

Configuration -- Delays



Want maximum delay from limited fibre
• Signals traverse fibre 16 times
4 WDM wavelengths
4 OC48 (2.5G) MUXed onto OC192 (10G)

Lots of transponders
• WDM amplifier joins 100km spools  200km

Configuration – delays
16x200km
OC48 slot
-------WDM Wavelength--------
Bidirectional 100km
Bidirectional 100km
Amp

Configuration – delays



Delay varied by adjusting the number of
OC48 hops traversed
Calient optical switch selects required hops
Hop lengths 200km up to 1600km
• Maximise granularity given limited switch ports
Switch

Projects



TCP benchmarking
FAST
• Delay-based congestion control
MaxNet
• Explicit signalling congestion control



MojaveFS
• New distributed file system
University of Pittsburgh
• TCP with small buffers
University of Melbourne
• Single-bit congestion marking

WAN-in-Lab testbed




Dummynet and simulation introduce artifacts
Also need to test on real equipment
WAN with real delays, located in a single room
• Connected to an external WAN (Ultralight)
Open for the community to use for benchmarking
OC-48
OC-48

WAN-in-Lab capabilities
Current
Two 2.5G bottlenecks
Multiple 1G bottlenecks
Two “real” delays
(Emulate cross traffic delay)
Planned
Six 2.5G bottlenecks
Up to six “real” delays
End-to-end RTT, drop Per-router delay, drop
(movable DAG cards)

Configuration -- delays
OC48 slot
-------WDM Wavelength--------
Bidirectional 100km
Bidirectional 100km
Amp

Using WAN-in-Lab

Contact me – lachlan at caltech . Edu

Coarse timesharing
• Some users set up experiments while others run experiments

Software setup still being developed
• Your chance to influence our directions to tailor it to your needs

Sample MaxNet results

Achieves realistic delay at 1Gbit/s