The Cutting EDGE of Router
Configuration
D. Caldwell, A. Gilbert, J. Gottlieb,
A. Greenberg, G. Hjalmtysson, and J. Rexford
AT&T Labs—Research; Florham Park, NJ
Manual configuration: Type first, and ask questions later
Manual Configuration
• Dangerous
– Typo in routing policy: black hole
– Wrong OSPF area: no traffic on link
– Missing packet filter: DoS vulnerability
• Expensive
– Delays in deploying equipment
– Hiring & training skilled engineers
– Lock-in to the router vendor
• Disruptive
– Half of network outages (Yankee Group)
– BGP routing anomalies (SIGCOMM’02)
– Failures of Internet services (USITS’03)
Why is the Situation So Bad?
• Networking field: emphasis on speed & features
– Not manageability and simplicity
– Constant change, without revisiting the design
– Oodles of complex protocols and tunable parameters
• Router vendors: lack of abstraction
– Assembly language commands
– Element-level configuration
– Low-level mechanisms, not intent
• Network administrators: melting under complexity
– Learning on the job
– Struggling just to keep up
– The path to automation is overwhelming
What’s a Networking Researcher to Do?
• Myth: Only large service providers care.
– Large enterprise networks care, big time
– Other network’s problems affect the user
• Myth: No research lives here.
– The state-of-the-art is really pitiful (please help!)
– Config languages, protocol modeling, data mining, …
• Myth: I can’t play without special access.
– Router testbeds, protocol modeling, simplicity, etc.
– Get data from your campus/regional administrator
• Myth: Campus/regional networks are too easy.
– Maybe it is harder than we think…
– If easy, please scale this simplicity to larger networks!
Our Goal: Automated Configuration
Technical Questions (TQ)
What is your AS
number?
What export policy do
you want?
Do you want a dynamic
default?
What are your address
blocks?
Do you need to receive
communities?
DB
interface <name>
description <cust name>
ip address <addr> <mask>
ip access-group <acl> in
!
router bgp 7018
neighbor <ip> remote-as <asn>
neighbor <ip> route-map CUST-FACE in
neighbor <ip> route-map <outmap> out
neighbor <ip> distribute-list <racl> in
neighbor <ip> soft-reconfiguration-inbound
[neighbor <ip> send-community]
!
query
R
U
L
E
S
interface Serial10/1/0/12:0
description CBB Customer
ip address 12.7.35.1 255.255.255.252
ip access-group 666 in
!
router bgp 7018
neighbor 12.7.35.2 remote-as 18585
neighbor 12.7.35.2 route-map CUST-FACE in
neighbor 12.7.35.2 route-map FULL-ROUTES out
neighbor 12.7.35.2 distribute-list 13 in
neighbor 12.7.35.2 soft-reconfiguration-inbound
!
router
configlet
template
• How to transition an existing network?
• How to get value as you move from here to there?
• Our approach: detailed analysis of configuration data
Lowering Barrier-to-Automation for Enterprises
• Large enterprise networks
–
–
–
–
Large stand-alone data networks
Retail, financial, health, business, etc.
Heterogeneity due to mergers and acquisitions
Stringent reliability and performance demands
Today
EDGE
Inventory
database
Existing
network
The network is
the database
Mistakes and
inconsistencies
Extracted from the
configuration files
Discord reports from
analyzing config files
Local
config
rules
Poor (or no)
Reverse-engineered by
documentation, data mining on the
policy violations configuration files
Enablement and Debugging of Growing Enterprises
Web
reports
Automation
BOTTOM-UP APPROACH
1. Inventory database
queries
Abstract
network
database


Extract summary information
Bootstrap the database


Report errors & warnings
Allows immediate fixes


Reverse-engineer policies
Aids in fixing inconsistencies


Explicitly enforce the rules
Avoid future config mistakes
2. Fixing config mistakes
3. Codifying local policies
Low level
standard
Discords
form (tables)
fix
4. Automated configuration
polled
Router configuration
Runs on many thousands of configs a day.
Bootstrapping the Inventory Database
• Extracting summaries
–
–
–
–
Routers & interfaces
Routing protocols
IP address usage
Configuration commands
• Network visualization
– Routers and links
– Attributes (capacity,
routing parameters)
– Layout (spring model)
– Graph compression
– Geographical info
Dual hub-and-spoke
Pinpointing Configuration Mistakes
• Referential integrity
– Multiple interfaces with same IP address
– Items used but not defined, and vice versa
• Packet/route filters, route-maps, QoS policy,…
• Routing protocol (e.g., OSPF)
– Local integrity checks (warnings)
• Network statement matches no interfaces
• Interface enabled in OSPF has no cost metric
• Interface not enabled in OSPF has a cost metric
– Correct adjacencies (errors)
• OSPF interface is not part of a complete link
• Remote end of link does not participate in OSPF
• Interfaces on same link have different OSPF areas
Example: Joining Parts of OSPF Config Together
hostname MyRouter
Remote end is in 12.123.36.72/30
!
interface POS7/0
ip address 12.123.36.74 255.255.255.252
ip ospf cost 50
!
Interface participates in OSPF
router ospf 2
network 12.123.36.0 255.255.255.0 area 9
passive-interface Serial2/1/0/3.1
!
Example: Remote End in Different OSPF Area
Extracted
tables
interface
link
OSPF network
OSPF passive interface
OSPF interface
Intermediate
tables
active OSPF interface
Simple SQL queries
OSPF link with area mismatch
Presentation
query result
Using Data Mining to Infer Local Policies
• Pattern matching across routers
– Equivalence: same configuration, same names
– Synonym: same configuration, different names
– Homonym: same name, different configuration
• Infer rules and report exceptions
–
–
–
–
“Finger daemon disabled”
“Edge interfaces have inbound packet filters”
“OSPF costs of 100 to hub1 and 200 to hub2”
“IPX interfaces have SAP encapsulation”
• Identifying “use cases”
– Initial base configuration of each router
– “Diffs” over time and across related routers
– Common cases (e.g., adding link, new BGP session)
Conclusion
• Moving beyond manual configuration
– Manual configuration is bad
• Error-prone, expensive, and disruptive
– Migrating to automation is hard
• No inventory DB, buggy network, and poor docs
– EDGE supports migration to automation
• Bootstrap DB, report discords, and infer policies
• More innovation is needed
–
–
–
–
Better router configuration languages
Protocol modeling and best common practices
Data mining on existing data networks
Autoconfigurable protocols and mechanisms