The Cutting EDGE of Router
Configuration
D. Caldwell, A. Gilbert, J. Gottlieb,
A. Greenberg, G. Hjalmtysson, and J. Rexford
AT&T Labs—Research; Florham Park, NJ
Manual configuration: Type first, and ask questions later
Outline
• Motivation for automation
• EDGE “bottom up” approach
• Example functionality of the EDGE tool
– Inventory and network visualization
– Configuration errors and warnings
– Reverse engineering of local policies
• Software architecture
– Data feeds and configuration parsing
– Extracting tables and joining them
– Presentation queries and GUI
• Conclusions
Manual Configuration
• Dangerous
– Typo in routing policy: black hole
– Wrong OSPF area: no traffic on link
– Missing packet filter: DoS vulnerability
• Expensive
– Delays in deploying equipment
– Hiring and training skilled engineers
– Lock-in to the router vendor
• Disruptive
– Half of network outages (Yankee Group)
– BGP routing anomalies (SIGCOMM’02)
– Failures of Internet services (USITS’03)
Why is the Situation So Bad?
• Networking field: emphasis on speed & features
– Not manageability and simplicity
– Constant change, without revisiting the design
– Oodles of complex protocols and tunable parameters
• Router vendors: lack of abstraction
– Assembly language commands
– Element-level configuration
– Low-level mechanisms, not intent
• Network administrators: melting under complexity
– Learning on the job
– Struggling just to keep up
– The path to automation is overwhelming
Our Goal: Automated Configuration
Technical Questions (TQ)
What is your AS
number?
What export policy do
you want?
Do you want a dynamic
default?
What are your address
blocks?
Do you need to receive
communities?
DB
interface <name>
description <cust name>
ip address <addr> <mask>
ip access-group <acl> in
!
router bgp 7018
neighbor <ip> remote-as <asn>
neighbor <ip> route-map CUST-FACE in
neighbor <ip> route-map <outmap> out
neighbor <ip> distribute-list <racl> in
neighbor <ip> soft-reconfiguration-inbound
[neighbor <ip> send-community]
!
query
R
U
L
E
S
interface Serial10/1/0/12:0
description CBB Customer
ip address 12.7.35.1 255.255.255.252
ip access-group 666 in
!
router bgp 7018
neighbor 12.7.35.2 remote-as 18585
neighbor 12.7.35.2 route-map CUST-FACE in
neighbor 12.7.35.2 route-map FULL-ROUTES out
neighbor 12.7.35.2 distribute-list 13 in
neighbor 12.7.35.2 soft-reconfiguration-inbound
!
router
configlet
template
• How to transition an existing network?
• How to get value as you move from here to there?
• Our approach: detailed analysis of configuration data
Lowering Barrier-to-Automation for Enterprises
• Large enterprise networks
–
–
–
–
Large stand-alone data networks
Retail, financial, health, business, etc.
Heterogeneity due to mergers and acquisitions
Stringent reliability and performance demands
Today
EDGE
Inventory
database
Existing
network
The network is
the database
Mistakes and
inconsistencies
Extracted from the
configuration files
Discord reports from
analyzing config files
Local
config
rules
Poor (or no)
Reverse-engineered by
documentation, data mining on the
policy violations configuration files
Enablement and Debugging of Growing Enterprises
Web
reports
Automation
BOTTOM-UP APPROACH
1. Inventory database
queries
Abstract
network
database


Extract summary information
Bootstrap the database


Report errors & warnings
Allows immediate fixes


Reverse-engineer policies
Aids in fixing inconsistencies


Explicitly enforce the rules
Avoid future config mistakes
2. Fixing config mistakes
3. Codifying local policies
Low level
standard
Discords
form (tables)
fix
4. Automated configuration
polled
Router configuration
Runs on many thousands of configs a day.
Network Inventory
• Equipment
– Routers
• Model, OS version, available slots, protocol usage, …
– Interfaces
• Media, speed, edge/core, protocol usage, …
• Address space
– Network equipment and announced supernets
– Network services (e.g., NTP and SNMP)
– Packet/route filters and eBGP end-points
• Configuration commands
– Histogram of configuration commands
– Frequency of formats and options
Identifying the Topology
• Interfaces to links
12.7.35.0/30
12.7.35.1/30
12.7.35.2/30
• BGP end-points to sessions
12.123.37.250
neighbor 12.7.108.3
12.7.108.3
neighbor 12.123.37.250
Network Visualization
• Goal: automatic network design diagram
– Input: router configuration files
– Output: picture a network designer would draw
• Key ingredients from configuration
– Layer-3 topology (e.g., routers and links)
– Routing protocols and their attributes
– Route injection, filtering, & aggregation policies
• Visualization software
– Placement based on node degree, routing, …
– GUI with browsing, zooming, attributes,…
Automatic classification
– Red: high degree (dual hubs)
– Blue: degree two (spokes)
– Yellow: degree one (stubs)
Visualization clearly shows
disconnected network, or
missing configuration files
OSPF Topology
2
2
4
Area 1
3
4
3
5
3
3
4
2
3
4
3
Area 0
2
1
4
3
1
Area 2
OSPF Example
hostname MyRouter
Remote end is in 12.7.35.2/30
!
interface POS7/0
ip address 12.7.35.1 255.255.255.252
ip ospf cost 50
Interface participates in OSPF
!
router ospf 2
network 12.7.35.0 255.255.255.0 area 9
passive-interface Serial2/1/0/3.1
!
Routing Protocol Consistency
• Warnings
– OSPF interface has no cost metric
– Non-OSPF interface has OSPF attributes
– OSPF network matches no interfaces
• Errors
– Remote end is a non-OSPF interface
– Remote end has a different OSPF area
• Inventory
– OSPF links and edge interfaces
– Routers and links by OSPF area
– Interfaces by OSPF cost
Referential Integrity Checks
• Duplicate IP addresses
– Multiple interfaces with the same address
• Items used but not defined, and vice versa
interface Serial10/1/0/12:0
ip address 12.7.35.1 255.255.255.252
ip access-group 666 in
!
access-list 666 permit 12.34.158.0 0.0.1.255
access-list 666 permit 12.7.35.0 0.0.0.3
– Packet filters, route filters, QoS policies,
routing policies, AS path lists, etc.
Using Data Mining to Infer Local Policies
• Pattern matching across routers
– Equivalence: same config, same names
– Synonym: same config, different names
– Homonym: same name, different config
• Infer rules and report exceptions
–
–
–
–
“Finger daemon disabled”
“Edge interfaces have inbound packet filters”
“OSPF costs of 100 to hub1 and 200 to hub2”
“IPX interfaces have SAP encapsulation”
• Identifying “use cases”
– Initial base configuration of each router
– “Diffs” over time and across related routers
– Common cases (e.g., adding link, new BGP session)
Data Mining Approaches
• Configuration files
– Start with raw or lightly-parsed data
– Look for patterns in the uninterpreted strings
– Discover “templates” for flat parts of configuration
• Database tables
– Start with a model of part of the data
– Look for correlations between fields
– Find features that tend to appear together
• Network graph
– Start with a graph with edge/node attributes
– Look for patterns in topology and attributes
– Classify the graph and routing parameters
EDGE Software Design Principles
• Scalability
– CVS repository for daily feed of configuration data
– Automated data processing for hundreds of networks
– Precomputed tables underling interactive Web GUI
• Extensibility
– Extensible configuration parser
– Incremental additions to data models
– Ease of adding new queries on existing data models
• Ease of development
– Single low-level configuration parser
– SQL database for running queries
– XML specification of GUI functionality
EDGE Software Architecture
Web-based GUI
Presentation queries
developer
smokestacks
OSPF,
EIGRP,
RIP
Utility queries
Referential
integrity
Extractors
Parsed configuration data
Raw router configuration data
Router Configuration Data
• Getting the configuration data
– From a backup server (ssh, wget, ftp, etc.)
– Directly from the router (poller, crawler)
• Mapping files to networks
– Regular expressions on file names
– E.g., “feed-foo/hp*” maps to Hewlett Packard
• Storing the raw files
– Committing the data by network into CVS
• Checking data integrity
– Detecting duplicate configuration files
– Identifying decommissioned routers
Parsing the Configuration Files
• Practical constraints
– No public grammar exists for Cisco IOS
– Too many commands to parse everything
• Solution: light, extensible parsing
– Identify section boundaries
• Router, interface, router ospf, router bgp, etc.
– Parse key commands
• Interface name, address, description, bandwidth
– Retain unparsed commands
• Leave unparsed commands as strings
– Store results in Perl hash table
• Support indexing and sequencing of data
Extractors
• Practical constraints
– Huge breadth and depth of the data
– Unstructured raw configuration data
– Absence of predefined data models
• Solution: incremental data modeling
– Extract low-level tables from the data
• Simple scripts using the Perl hash table or “grep”
– Small number of flat tables for data mining
• Router, interface, BGP end-points, ACLs, names, etc.
– Insert tables into an SQL database
• Allow other queries to build on this data
Example Extractor: Referential Integrity
• Extract relevant commands
– Definitions and references, by type and name
Router
nyc72
nyc72
sfo35
chi19
sea42
sea42
R/D
def
ref
ref
def
def
ref
Command
access-list
access-group
neighbor
ip prefix-list
class-map
class
Type
ACL
ACL
route-map
prefix-list
class-map
class-map
Name
666
666
BLOCK
PEER
Silver
Silver
Utility Queries
• Practical constraints
– Some joined data needed by multiple queries
– Table joins are too slow to do on-demand
• Solution: precomputed utility tables
– SQL queries built on extracted tables
• Run in advance, with results stored in database
– Building blocks for the “smoke-stacks”
• Key constructs like link, OSPF interface, etc.
– Example: constructing the “links” table
• Self-join on the “interface” table
• Interface1 address in same network as interface2
Presentation Queries
• Practical constraints
– Many different reports on the same data
– Customizing column names and order of rows
• Solution: XML specification of output
– SQL query on the database
• Lightweight sub-selecting, summing, max, etc.
• “ORDER BY” to control the order of rows
– Presentation niceties
• Query title, description, and export to Excel
• Renaming of columns to user-friendly terms
• Click on table entry to run additional query
Example: Remote End in Different OSPF Area
Extracted
tables
interface
link
OSPF network
OSPF passive interface
OSPF interface
Utility
tables
active OSPF interface
Simple SQL queries
OSPF link
OSPF link with area mismatch
Presentation
query result
Software Status
• Runs daily for hundreds of networks
–
–
–
–
AT&T’s core backbone networks
Managed enterprise networks
New eVPN customers
Stand-alone enterprise customers
• Reports in the Web portal sites
–
–
–
–
Inventory and visualization
Routing protocols and policies
Referential integrity checks
ACL optimization and synonyms
Conclusion
• Moving beyond manual configuration
– Manual configuration is bad
• Error-prone, expensive, and disruptive
– Migrating to automation is hard
• No inventory DB, buggy network, and poor docs
– EDGE supports migration to automation
• Bootstrap DB, report discords, and infer policies
• More innovation is needed
–
–
–
–
Data mining on existing data networks
Protocol models and best common practices
Better router configuration languages
Self-configuring protocols and mechanisms