Stable Internet Routing Without
Global Coordination
Jennifer Rexford
Princeton University
http://www.cs.princeton.edu/~jrex
Joint work with Lixin Gao, Michael Schapira, and Yi Wang
What is an Internet?
A
“network of networks”
– Networks run by different institutions
 Autonomous
System (AS)
– Collection of routers run by a single institution
 ASes
have different goals
– Different views of which paths are good
 Interdomain
routing is what reconciles those views
– To compute end-to-end paths through the Internet
Wonderful problem setting for game theory and mechanism design
An Open Question
Evolvable Protocols
(under-specified, programmable)
?
Autonomy
Global Properties
(autonomous parties, with
different economic objectives)
(stability, scalability, reliability,
security, managability, …)
Can we have all three? Under what conditions?
Autonomous Systems (ASes)
Path: 6, 5, 4, 3, 2, 1
4
3
5
2
7
1
6
Web server
Client
Interdomain Routing: Border Gateway Protocol
 ASes
exchange info about who they can reach
– Destination: block of IP addresses (an “IP prefix”)
– AS path: sequence of ASes along the path
 Policies
configured by the AS’s network operator
– Path selection: which of the paths to use?
– Path export: which neighbors to tell?
“I can reach d
via AS 1”
“I can reach d”
2
1
data traffic
d
3
data traffic
Interdomain Routing Convergence Challenges
 Must
scale
– Address blocks: 300,000 and growing
– Autonomous Systems: around 35,000
 Must
support flexible policy
– Path selection: which path your AS wants to use
– Path export: who can send packets through your AS
 Must
converge, and quickly
– Routing convergence can take several minutes
– … and the system doesn’t necessarily converge at all!
Goal: Guaranteed convergence of the global routing system with purely local control.
Stable Paths Problem (SPP) Model
 Model
of routing policy
– Each AS has a ranking of the permissible paths
 Model
of path selection
– Pick the highest-ranked path consistent with neighbors
2 23d
2d
12d
1d
 Flexibility
is not free
1
d
31d
3 3d
– Global system converges slowly, or not at all
– Depending on the way the ASes rank their paths
Conflicting Policies Cause Convergence Problems
1
Better choice!
120
10
Only choice!
0
Top choice!
310
30
Only choice!
Better choice!
3
2
230
20
Only choice!
Pick the highest-ranked path consistent with your neighbors’ choices.
Global Control is Not Workable
 Create
a global Internet routing registry
– Keeping the registry up-to-date would be difficult
 Require
each AS to publish its routing policies
– ASes may be unwilling to reveal BGP policies
 Check
for conflicting policies, and resolve conflicts
– Checking for convergence problems is NP-complete
– Link/router failure may result in an unstable system
Need a solution that does not require global coordination.
Think Globally, Act Locally
 Key
features of a good solution
– Flexibility: allow diverse local policies for each AS
– Privacy: do not force ASes to divulge their policies
– Backwards-compatibility: no changes to BGP
– Guarantees: convergence even when system changes
 Restrictions
based on AS relationships
– Path selection rules: which route you prefer
– Export policies: who you tell about your route
– AS graph structure: who is connected to who
Customer-Provider Relationship
 Customer
pays provider for access to the Internet
– Provider exports its customer’s routes to everybody
– Customer exports provider’s routes only to downstream customers
Traffic to the customer
Traffic from the customer
d
provider
advertisements
provider
traffic
customer
d
customer
Peer-Peer Relationship
 Peers
exchange traffic between their customers
– AS exports only customer routes to a peer
– AS exports a peer’s routes only to its customers
Traffic to/from the peer and its customers
advertisements
peer
d
traffic
peer
Hierarchical AS Relationships
 Provider-customer
graph is a directed, acyclic graph
– If u is a customer of v and v is a customer of w
– … then w is not a customer of u
w
v
u
Valid and Invalid Paths
Valid paths: “6
“1 4
23
d”d”
and
and
“7“8
d”5 d”
Invalid
Invalidpaths:
path: “5
“6 85 d”
d” and “1 4 3 d”
1
d
5
Provider-Customer
Peer-Peer
4
3
2
7
8
6
Act Locally, Prove Globally
 Route
export
– Do not export routes learned from a peer or provider
– … to another peer or provider
 Global
topology
– Provider-customer relationship graph is acyclic
– E.g., my customer’s customer is not my provider
 Route
selection
– Prefer routes through customers
– … over routes through peers and providers
 Guaranteed
to converge to unique, stable solution
Our Local Path Selection Rules
 Classify
routes based on next-hop AS
– Customer routes, peer routes, and provider routes
 Rank
routes based on classification
– Prefer customer routes over peer and provider routes
 Allow
any ranking of routes within a class
– E.g., can rank one customer route higher than another
– Gives network operators the flexibility they need
 Consistent
with traffic engineering practices
– Customers pay for service, and providers are paid
– Peer relationship contingent on balanced traffic load
Solving the Convergence Problem
 Result
– Safety: guaranteed convergence to unique stable solution
– Inherent safety: holds under failures and policy changes
 Definitions
– System state: current best route at each AS
– Activating AS: re-do decision based on neighbor choices
 Sketch
of (constructive) proof
– Find an activation sequence that leads to a stable state
– Any “fair” sequence (eventually) includes this sequence
Rough Sketch of the Proof
 Two
phases
– Walking up the customer-provider hierarchy
– Walking down the provider-customer hierarchy
1
d
5
Provider-Customer
Peer-Peer
4
3
2
7
8
6
Economic Incentives Affect Protocol Behavior
 ASes
already follow our rules, so system is stable
– High-level argument
» Export and topology assumptions are reasonable
» Path selection rule matches with financial incentives
– Empirical results
» BGP routes for popular destinations are stable for ~10 days
» Most instability from failure/recovery of a few destinations
 ASes
should follow our rules to make system stable
– Need to encourage operators to obey these guidelines
– … and provide ways to verify the network configuration
– Need to consider more complex relationships and graphs
Playing One Condition Off Against Another
 All
three conditions are important
– Path ranking, export policy, and graph structure
 Allowing
more flexibility in ranking routes
– Allow same preference for peer and customer routes
– Never choose a peer route over a shorter customer route
…
at the expense of stricter AS graph assumptions
– Hierarchical provider-customer relationship (as before)
– No private peering with (direct or indirect) providers
Peer-peer
Extension to Backup Relationships
 Backups:
more liberal export policies, and different ranking
– The motivation is increased reliability
– …but ironically it may cause routing instability!
 Generalize
rule: prefer routes with fewest backup links
– Need to maintain a count of the # of backup links in the path
Backup Provider
Peer-Peer Backup [RFC 1998]
provider
primary
provider
failure
backup path
failure
backup path
backup
provider
peer
Results Hold Under More Complex Scenarios
 Complex
AS relationships
– AS pair with different relationship for different prefixes
– AS pair with both a backup and a peer relationships
– AS providing transit service between two peer ASes
 Stability
under changing AS relationships
– Customer-provider to/from peer-peer
– Customer-provider to/from provider-customer
Extensions of the Work
 Influence
of AS relationships on BGP convergence
– Algebraic framework and design principles for policy languages
– Fundamental limits on relaxing the assumptions
 Application
of the idea to internal BGP inside an AS
– Sufficient conditions for iBGP convergence inside an AS
– “What-if” tool for traffic engineering inside an AS
 AS-level
analysis of the Internet topology
– Inference of AS relationships and policies from routing data
– Characterization of AS-level topology and growth
 Practical
applications of knowing AS relationships
– Analyzing your competitors’ business relationships
– Identifying BGP routes that violate export conditions
A Case For Customized Route Selection
 ISPs
usually have multiple paths to the destination
 Different paths have different properties
 Different neighbors may prefer different routes
Bank
Most secure
VoIP
provider
School
Lowest cost
Shortest latency
Neighbor-Specific Route Selection
A
node has a ranking function per neighbor

i
j
is node i’s ranking function for neighbor node j.
Stability Conditions for NS-BGP
 Surprisingly,
NS-BGP improves stability!
– Neighbor-specific selection is more flexible
– Yet, requires less restrictive stability conditions
 “Prefer
customer” assumption is not needed
– Choose any “permissible” route per neighbor
 That
is, need just two assumptions
– No cycle of provider-customer relationships
– Do not export routes learned from one peer/provider to
other peers/providers
Why Do Weaker Conditions Work?
1
120
10
0
310
30
 An
3
2
230
20
AS always tells its neighbor a route
– If it has any route that is permissible for that neighbor
Deploying NS-BGP
 An
AS can deploy NS-BGP alone
– Without upgrading their routers
– Without coordinating with all their neighbors
 Three
aspects to the solution
– Disseminating extra BGP routes
– Customized route selection
– Directing traffic from ingress to egress
 Can
be done exploiting existing mechanisms
– Designed for Virtual Private Networks (VPNs)
Disseminating Extra BGP Routes
 Advertising
more than one BGP route
– Route distinguisher feature for VPNs
– Multiple internal BGP sessions
– ADD-PATHs extensions to internal BGP
Customized Route Selection
 Multiple
virtual routing and forwarding tables
– Cisco: Virtual Routing and Forwarding (VRF)
– Juniper: Virtual Router
R3’s forwarding table (FIB) entries
D: (red path): R6
D: (blue path): R7
Directing Traffic from Ingress to Egress
 Tunnels
from ingress to egress
– IP-in-IP tunneling
– MPLS
?
Customized Route Selection
 Customized
route selection as a service
– Select a different best route for different neighbors
 Different
menu options
– Cheapest route (e.g., “prefer customer”)
– Best performing routes, or most secure routes
– Routes that avoid undesirable ASes (e.g., censorship)
 Nice
practical features of NS-BGP
– An individual AS can deploy NS-BGP alone
– … without compromising global stability!
Conclusions
 Avoiding
convergence problems
– Hierarchical of provider-customer relationships
– Export policies based on commercial relationships
– (Path ranking based on AS relationships)
 Salient
features
– No global coordination (locally implementable)
– No changes to BGP protocol or decision process
– Guaranteed convergence, even under failures
– Guidelines consistent with financial incentives
References Related to This Talk

“The stable paths problem and interdomain routing”
– Tim Griffin, Bruce Shepherd, and Gordon Wilfong
– http://portal.acm.org/citation.cfm?id=508332

“Stable Internet routing without global coordination”
– Lixin Gao and Jennifer Rexford
– http://www.cs.princeton.edu/~jrex/papers/sigmetrics00.long.pdf

“Inherently Safe Backup Routing with BGP”
– Lixin Gao, Tim Griffin, and Jennifer Rexford
– http://www.cs.princeton.edu/~jrex/papers/infocom01.pdf

“Neighbor-Specific BGP: More flexible routing policies while
improving global stability”
– Yi Wang, Michael Schapira, and Jennifer Rexford
– http://www.cs.princeton.edu/~jrex/papers/nsbgp_sigmetrics09.pdf
Other Related Research Papers

Inherently Safe Backup Routing with BGP
– http://www.cs.princeton.edu/~jrex/papers/infocom01.pdf

Design Principles of Policy Languages for Path Vector Protocols
– http://conferences.sigcomm.org/sigcomm/2003/papers/p61-griffin.pdf

Implications of Autonomy for the Expressiveness of Policy Routing
– http://conferences.sigcomm.org/sigcomm/2005/paper-FeaBal.pdf

Meta-routing
– http://conferences.sigcomm.org/sigcomm/2005/paper-GriSob.pdf

An Algebraic Theory of Interdomain Routing
– http://portal.acm.org/citation.cfm?id=1103561

Searching for Stability In Interdomain Routing
– http://www.cs.yale.edu/homes/schapira/PID808559.pdf
Related Papers With Game Theory

Interdomain Routing and Games
– http://www.cs.huji.ac.il/~mikesch/routing_games-full.pdf

Rationality and Traffic Attraction: Incentives for Honest Path Announcements in BGP
– http://ccr.sigcomm.org/online/?q=node/395

Incentive-Compatible Interdomain Routing
– http://cs-www.cs.yale.edu/homes/jf/FRS.pdf

Mechanism Design for Policy Routing
– http://cs-www.cs.yale.edu/homes/jf/FSS.pdf

The Complexity of Game Dynamics: BGP Oscillations, Sink Equlibria, and Beyond
– http://www.cs.berkeley.edu/~alexf/papers/fp08.pdf

Specification Faithfulness in Networks with Rational Nodes
– http://www.eecs.harvard.edu/econcs/pubs/podc04.pdf

Distributed Algorithmic Mechanism Design
– http://cs-www.cs.yale.edu/homes/jf/AGTchapter14.pdf

Partially Optimal Routing
– http://www.stanford.edu/~rjohari/pubs/por.pdf
36
Background on Interdomain Economics






http://drpeering.net/a/Home.html
http://www.fcc.gov/Bureaus/OPP/working_papers/oppwp32.pdf
http://www.potaroo.net/papers/1999-6-peer/peering.pdf
http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac201/about_
cisco_ipj_archive_article09186a00800c83a5.html
http://www.cisco.com/en/US/about/ac123/ac147/ac174/ac200/about_
cisco_ipj_archive_article09186a00800c8900.html
http://www.vjolt.net/vol3/issue/vol3_art8.html
37