Remote Network Labs: An On-Demand
Network Cloud for Configuration Testing
Huan Liu, Dan Orban
Accenture Technology Labs
Configuration is hard
• A few reasons
– Many SW images for each box
• IOS: 5+ trains, 8 packages for routers, 5 packages for switches
– Primitive CLI interface
– Commands may behave differently  a design may work on paper, but not in practice
– Local configuration at a switch/router could have global impact
• Sample manifestations
– More outages are caused by operator errors rather than equipment failures 1,2
– 3 in 4 new BGP prefix advertisements are results of misconfiguration (200-1200prefixes/day)
– Average 7.17 to 9.63 errors per firewall config 4
1.
2.
3.
4.
“Configuration management delivers business resiliency,” The Yankee Group, Nov. 2002
D. Oppenheimer, A. Ganapathi, and D. Patterson, “Why internet services fail and what can be done
about these,” in Proc. USENIX USITS, Oct. 2003.
R. Mahajan, D. Wetherall and T. Anderson, “Understanding BGP Misconfiguration”, SIGCOMM 2002
A. Wool, “A Quantitative Study of Firewall Configuration Errors”, Computer, 2004
2
3
Current solutions and their problems
• Solution 1: Simulator (e.g., RouterSim, OPNET)
– Cannot capture all aspects of real hardware (e.g., many IOS images)
– Support limited # of commands
– Have simulation models for a limited number of routers
• Solution 2: Emulator (Dynamips)
– Limited set of interfaces are simulated
– Support a limited set of Cisco routers
• Solution 3: Build a smaller scale network in Lab to mimic the production
network, and test in lab before roll out
– Costly investment (routers are not cheap)
– Poorly utilized (only during testing), yet necessary in case configuration changes
again
– May not be available when you need it (spare parts problem)
– Time consuming to wire up
– Hard to make sure the test network is the same as designed on paper.
3
What your lab looks like?
Our solution: Remote Network Labs
• Setup a virtual lab, then test
– Equipment could be anywhere (no longer needs to be co-located)
– Design from anywhere (no longer needs to be physically on site)
– Quickly and easily reconfigurable
• Advantages
– Enable efficient sharing of equipment between projects
• No more procurement delay
• Essential zero cost to each project
• No need for physical lab space
– Enable design from a Web browser
• Everything digitized, including the wiring
– Save/restore legacy environment
– Fully automate lab set up
• No physical work to mount equipment and wire
• Reduce travel (go Green)
5
Distributed architecture
http://netlabs.accenture.com
Internet
Web browsers
Client site
San Jose
Chicago
6
How it works
When press Deploy
button, send instruction
to build tunnel
Interface
SW
Interface
SW
Internet
Front end
Back end
7
Making a router/firewall/server part of
the lab inventory
Internet
Interface
SW
8
User defines mapping
Copyright © 2007 Accenture All Rights Reserved.
9
Key differences from prior work
Real router
Distributed
Wire
Emulab
N
N
L2 tunnel
PlanetLab
N
Y
L3
ONL
N
N
L2 tunnel
VINI
N
Y
L3 tunnel
WAIL
Y
N
Real wire
RNL
Y
Y
L2 + L3 tunnel
Copyright © 2007 Accenture All Rights Reserved.
10
Use case 1 (layer 2 config)
• Configuring failover using
Cisco Catalyst 6500
switches with a Firewall
Services Module (FWSM)
• Can also capture
transient behavior
Copyright © 2007 Accenture All Rights Reserved.
11
Use case 2 (test automation)
• Full automation (from setup to teardown)
– Implementing web services API
– Implementing library for traffic generation/capture, CLI parsing.
• Add delay/jitter
• Inject/observe anywhere
x
x
Add delay/jitter
Inject
x
Observe
x
12
Copyright © 2007 Accenture All Rights Reserved.
Challenges/drawbacks
• Additional delay
• WAN bandwidth is not free
• L2 interface diversity
• Performance testing (see paper for more details)
Copyright © 2007 Accenture All Rights Reserved.
13