In VINI Veritas
Realistic and Controlled
Network Experimentation
Andy Bavier Nick Feamster* Mark Huang
Larry Peterson Jennifer Rexford
Princeton University
*Georgia Tech
How to Validate an Idea?
Emulation
Simulation
VINI
Small-scale
experiment
Live
deployment
Fixed, shared among many experiments
 Runs real routing software
 Exposes realistic network conditions
 Gives control over network events
 Carries traffic on behalf of real users

Scientific Value
The most exciting phrase to hear in science, the
one that heralds new discoveries, is not ‘Eureka!’
(I found it!) but ‘That’s funny …’ -- Isaac Asimov

Move off the emulator, into the wild
 Opportunity

for more ‘that’s funny’ moments
Avoid “Fallacy of Misplaced Concreteness”
 Simulation and emulation are important tools
 Modeling abstracts general properties from reality

Philosophy: the devil may be in the details…
 But
insights and soundness are found there too
“Controlled Realism”
Arbitrary,
emulated
Actual
network

Topology
Synthetic
or traces
Real
clients,
servers
Traffic
Inject faults,
anomalies


Start with a controlled
experiment
Relax constraints,
study effects
Result: an operational
virtual network that’s
 Feasible
Observed in
operational
network
Network Events
 Valuable
 Robust
 Scalable,
etc.
Overview

VINI requirements
 Fixed,
shared infrastructure
 Flexible network topology
 Expose/inject network events
 External connectivity and routing adjacencies




Strategy for building VINI
PL-VINI: prototype on PlanetLab
Experimental results
Timeline
Fixed Infrastructure
Deploying VINI nodes in National LambdaRail,
Abilene with Gigabit links
Shared Infrastructure
Experiments given illusion of dedicated h/w
Flexible Topology
VINI supports arbitrary virtual topologies
Network Events
VINI exposes, can inject network failures
External Connectivity
c
s
Experiments can carry traffic for real end-users
External Routing Adjacencies
BGP
BGP
c
s
BGP
BGP
Experiments can participate in Internet routing
PlanetLab  VINI

Build VINI from PlanetLab, a global
testbed for distributed services
 Begun
in 2002
 700 nodes at 336 sites in 35 countries
 600 projects and 2500 researchers
 Serves 3-4 TB/day to ~1M clients

MyPLC: PlanetLab software distribution
 Anyone
can run their own private PlanetLab
PlanetLab Experiments

Simultaneous experiments in separate VMs
 Each

has “root” in its own VM, can customize
Reserve CPU, network capacity per experiment
Node
Mgr
Local
Admin
VM1
VM2
…
VMn
Virtual Machine Monitor (VMM)
(Linux++)
PlanetLab node
PL-VINI: Prototype on PlanetLab


Feasible?  prototype on public PlanetLab
Enable experiment: Internet In A Slice
open-source routing protocol suite (NSDI ’05)
 Click modular router (TOCS ’00, SOSP ’99)
 XORP

Clarify issues that a VINI must address
 Unmodified
routing software on a virtual topology
 Forwarding packets at line speed
 Illusion of dedicated hardware
 Injection of faults and other events
XORP: Control Plane
XORP
(routing protocols)



PlanetLab VM
Goal: real routing
protocols on virtual
network topologies
BGP, OSPF, RIP,
PIM-SM, IGMP/MLD
XORP can run in a
PlanetLab VM
User-Mode Linux: Environment
UML
XORP

(routing protocols)

eth0
eth1
eth2
eth3
 Experiments
cannot
create new interfaces


PlanetLab VM
Interface ≈ network
PlanetLab limitation:
Run routing software
in UML environment
Create virtual network
interfaces in UML
Click: Data Plane
UML
XORP

(routing protocols)
eth0
eth1
eth2
 Avoid
UML overhead
 Move to kernel, FPGA
eth3
Control

Data
Packet
Forward
Engine
Click
PlanetLab VM
Performance
 Click
UDP tunnels
correspond to UML
network interfaces
UmlSwitch
element
Tunnel table
Filters
Interfaces  tunnels

Filters
 “Fail
a link” by blocking
packets at tunnel
Resource Isolation

Issue: Forwarding packets in user space
 PlanetLab
sees heavy use
 CPU load affects virtual network performance
Property
Depends On
Solution
Throughput
CPU% received
PlanetLab provides CPU
reservations
Latency
CPU scheduling
delay
PL-VINI: boost priority of
packet forward process
Intra-domain Route Changes
s
856
2095
700
260
1295
c
639
366
233
548
587
846
902
1893
1176
Watch OSPF route convergence on Abilene
Ping During Link Failure
Link down
120
Routes converging
110
Ping RTT (ms)
Link up
100
90
80
Abilene RTT: 73ms
70
0
10
20
30
Seconds
40
50
TCP Throughput
Link down
12
Link up
Megabytes transferred
Packet receiv ed
10
8
6
4
Zoom in
2
0
0
10
20
30
Seconds
40
50
Arriving TCP Packets
2.45
Megabytes in stream
Packet receiv ed
2.4
2.35
PL-VINI
user-space virtual network
2.3enables
Slowa
start
to behave
like
a
real
network
on
PlanetLab
2.25
2.2
Retransmit
lost packet
2.15
2.1
17.5
18
18.5
19
Seconds
19.5
20
Attracting Real Users
Could have run experiments on Emulab
 Goal: Operate our own virtual network

 Carrying
traffic for actual users
 We can tinker with routing protocols

We expect that:
 PlanetLab
services will subscribe to VINI
network architectures to access Gb/s
 Experiments will advertise routes via BGP
Timeline
You are
here
PL-VINI
• PlanetLab
• Resource resv
• CPU priority
Fall 2006
NLR-VINI
Abilene-VINI
• PCs
• PlanetLab OS
• MyPLC
• Gigabit layer 2
• eBGP uplinks
to friendly ISPs
2007
NLR-VINI
Abilene-VINI
Japan-VINI
• PCs
• VINI OS
• MyVINI
• Xen
• Exchange traffic
with ISPs
Other features?
2008
NLR-VINI
Abilene-VINI
Japan-VINI
???-VINI
• Other GREN
• PC + FPGAs, NPs
• Create layer 2
“on the fly”
The End

Questions?
The End
URL: http://www.vini-veritas.net
 Questions?

Backup slides
Conclusion
VINI = evolution of PlanetLab
 Installing VINI nodes in NLR, Abilene
 Download and run Internet In A Slice
 MyPLC  MyVINI as code diverges

 Build,
run, modify your own VINI
 We expect there to be many VINIs
http://www.vini-veritas.net
Timeline
Conclude with a timeline instead? Like the
one for Gibson.
 Experiments on the top, infrastructure on
the bottom, “You are here.”
 Today: IIAS, PL-VINI
 Next: RCP, VINI-NLR
 What other experiments?

Ongoing Work

Improving realism
 Exposing
network failures and changes in the
underlying topology
 Participating in routing with neighboring
networks

Improving control
 Better
isolation
 Experiment specification
Performance is bad
User-space Click: ~200Mb/s forwarding
 Can do a lot with 200Mb/s

 20
experiments can have dedicated 10Mb/s
nationwide networks

Improving performance is ongoing work
 Allow
experiments to load custom Click
modules into the VINI kernel
PL-VINI Summary
Flexible Network Topology
Virtual point-to-point connectivity
Tunnels in Click
Unique interfaces per experiment
Virtual network devices in UML
Exposure of topology changes
Upcalls of layer-3 alarms
Flexible Routing and Forwarding
Per-node forwarding table
Separate Click per virtual node
Per-node routing process
Separate XORP per virtual node
Connectivity to External Hosts
End-hosts can direct traffic through VINI
Connect to OpenVPN server
Return traffic flows through VINI
NAT in Click on egress node
Support for Simultaneous Experiments
Isolation between experiments
PlanetLab VMs and network isolation
CPU reservations and priorities
Distinct external routing adjacencies
BGP multiplexer for external sessions
PL-VINI / IIAS Router
UML
XORP

(routing protocols)

eth0
eth1
eth2
eth3
 Virtual
Control
Data
Packet
Forward
Engine
Click
XORP: control plane
UML: environment
UmlSwitch
element
Tunnel table

interfaces
Click: data plane
 Performance


Avoid UML overhead
Move to kernel, FPGA
 tunnels
 “Fail a link”
 Interfaces
What’s New with VINI?
Integration of routing w/Internet
 Better isolation
 Real topologies
 Inject events

“Controlled Realism”
Arbitrary,
emulated
Actual
network

Real
clients,
servers

 Reproduce
results
 Methodically change or
relax constraints
Topology
Synthetic
or traces
Traffic
Inject faults,
anomalies
Observed in
operational
network
Network Events
Control:
Realism:
 Long-running
services
attract real “customers”
 Forward high traffic
volumes (Gb/s)
 Robustly handle
unexpected events