BGP Instability
Jennifer Rexford
Fall 2014 (TTh 3:00-4:20 in CS 105)
COS 561: Advanced Computer Networks
http://www.cs.princeton.edu/courses/archive/fall14/cos561/
Holding the Internet Together
• Distributed cooperation for resource allocation
– BGP: what end-to-end paths to take (for ~50K ASes)
– TCP: what rate to send over each path (for ~3B hosts)
AS 2
AS 1
AS 3
AS 4
2
What Problem is BGP Solving?
The Stable Paths Problem
http://www.cs.princeton.edu/courses/archive/f
all14/cos561/papers/SPP02.pdf
3
What Problem Does Routing Solve?
• Most do shortest-path routing
–Shortest hop count
 Distance vector routing (e.g., RIP)
–Shortest path as sum of link weights
 Link-state routing (e.g., OSPF and IS-IS)
• Policy makes BGP is more complicated
–An AS might not tell a neighbor about a path
 E.g., Sprint can’t reach Verizon through AT&T
–An AS might prefer one path over a shorter one
 E.g., ISP prefers to send traffic through a customer
4
Could Use A Simulation Model
• Simulate the message passing
–Advertisements and withdrawals
–Message format
–Timers
• Simulate the routing policy on each session
–Filter certain route advertisements
–Manipulate the attributes of others
• Simulate the decision process
–Each router applying all the steps per prefix
Feasible, but tedious and ill-suited for formal arguments
Stable Paths Problem Instance
• Node
– BGP-speaking router
– Node 0 is destination
2 21 0
20
– BGP adjacency
– Set of routes to 0 at each
node 1
– Ranking of the paths
5210
2
• Edge
• Permitted paths
5
4
0
1
3
130
10
most preferred
…
least preferred
420
430
30
Solution to a Stable Paths Problem
2
210
20
• Solution
– Path assignment per node
– Can be the “null” path
• Each node is assigned
1
– The highest ranked path
– … consistent with the
assignment of its neighbors
5210
2
• If node u has path uwP
– {u,w} is an edge in the graph
– Node w is assigned path wP
5
4
0
1
130
10
3
420
430
30
A solution need not represent
a shortest path tree, or
a spanning tree.
Translating a Real Configuration into SPP
• Permitted paths at a node
– Composition of “export” policies at other nodes
Node 0 exports
route to node 2
0
210
20
Node 2 exports
5210
“2 1 0” but not “2 0”
2
5
Node 1 exports
“1 0” to node 2
• Ranking of paths at a node
– “Import” policies at the node
– Rank in terms of BGP decision process (i.e., local
preference, AS path length, origin type, MED, …)
An SPP May Have Multiple Solutions
120
10
120
10
1
120
10
1
0
0
2
210
20
1
0
2
210
20
First solution
2
210
20
Second solution
An SPP May Have No Solution
2
210
20
4
0
130
10
1
320
30
3
3
Stable System Unstable After Failure
210
20
BGP is not robust :
it is not guaranteed
to recover from
network failures.
1
130
10
2
Becomes a BAD GADGET if link
(4, 0) goes down.
4
40
420
430
0
3
3420
30
Strawman Solution Doesn’t Work
• Create a global Internet routing registry
– Store the AS-level graph and all routing policies
– Store all routing policies
– But, ASes may be unwilling to divulge
• Check for conflicting policies
– Analyze the global system and identify conflicts
– Contact the affected ASes to resolve them
– But, checking is an NP-complete problem
– … and, a safe system may be unsafe after failure
Goal: sufficient condition for convergence with local control
BGP Protocol
13
Two Kinds of Routing Protocols
Link State
Vectoring
• Topology information is flooded
within the routing domain
• Each router knows little about
network topology
• Best end-to-end paths are
computed locally at each router.
• Only best next-hops are chosen by
each router for each destination.
• Best end-to-end paths
determine next-hops.
• Best end-to-end paths result from
composition of all next-hop choices
• Based on minimizing some notion • Does not require any notion of
of distance
distance
• Works only if policy is shared and • Does not require uniform policies at
uniform
all routers
• Examples: OSPF, IS-IS
• Examples: RIP, BGP
AS Numbers (ASNs)
ASes represent units of routing policy
•
•
•
•
•
•
•
•
•
Level 3: 1
MIT: 3
Harvard: 11
Yale: 29
Princeton: 88
AT&T: 7018, 6341, 5074, …
Verizon: 701, 702, 284, 12199, …
Sprint: 1239, 1240, 6211, 6242, …
…
Currently around 50,000 in use.
Interdomain Routing
Path: 6, 5, 4, 3, 2, 1
4
3
5
2
7
1
6
Web server
Client
Border Gateway Protocol
• ASes exchange info about who they can reach
– IP prefix: block of destination IP addresses
– AS path: sequence of ASes along the path
• Policies configured by the AS’s operator
– Path selection: which of the paths to use?
– Path export: which neighbors to tell?
“12.34.158.0/24:
path (2,1)”
3
“12.34.158.0/24:
path (1)”
1
2
data traffic
data traffic
12.34.158.5
BGP Session Operation
Establish session on
TCP port 179
AS1
BGP session
Exchange all
active routes
AS2
Exchange incremental
updates
While connection
is ALIVE exchange
route UPDATE messages
Incremental Protocol
• A node learns multiple paths to destination
–Stores all of the routes in a routing table
–Applies policy to select a single active route
–… and may advertise the route to its neighbors
• Incremental updates
–Announcement
 Upon selecting a new active route, add node id to path
 … and (optionally) advertise to each neighbor
–Withdrawal
 If the active route is no longer available
 … send a withdrawal message to the neighbors
19
BGP Route
• Destination prefix (e.g., 128.112.0.0/16)
• Route attributes, including
– AS path (e.g., “7018 88”)
– Next-hop IP address (e.g., 12.127.0.121)
192.0.2.1
AS 7018
12.127.0.121
AT&T
AS 88
AS 11
Yale
Princeton
128.112.0.0/16
AS path = 88
Next Hop = 192.0.2.1
128.112.0.0/16
AS path = 7018 88
Next Hop = 12.127.0.121
20
ASPATH Attribute
128.112.0.0/16
AS Path = 1755 1239 7018 88
128.112.0.0/16
AS Path = 1239 7018 88
AS 1239
Sprint
AS 1755
AS 88
Princeton
Global Access
128.112.0.0/16
AS Path = 1129 1755 1239 7018 88
Ebone
AS 12654
128.112.0.0/16
AS Path = 7018 88
AS7018
128.112.0.0/16
AS Path = 88
AS 1129
RIPE NCC
RIS project
128.112.0.0/16
AS Path = 3549 7018 88
AT&T
128.112.0.0/16
AS Path = 7018 88
AS 3549
Global Crossing
128.112.0.0/16
Prefix Originated
21
BGP Path Selection
• Simplest case
AS 1129
–Shortest AS path
–Arbitrary tie break
• Example
Global Access
128.112.0.0/16
AS Path = 1129 1755 1239 7018 88
–Three-hop AS path preferred
over a five-hop AS path
–AS 12654 prefers path
through Global Crossing
• But, BGP is not limited to
shortest-path routing
–Policy-based routing
AS 12654
RIPE NCC
RIS project
128.112.0.0/16
AS Path = 3549 7018 88
AS 3549
Global Crossing
22
BGP Routing Changes
23
Causes of BGP Routing Changes
• Topology changes
– Equipment going up or down
– Deployment of new routers or sessions
• BGP session failures
– Due to equipment failures, maintenance, etc.
– Or, due to congestion on the physical path
• Changes in routing policy
– Reconfiguration of preferences
– Reconfiguration of route filters
• Persistent protocol oscillation
– More on this next class!
BGP Session Failure
• BGP runs over TCP
– BGP only sends updates
when changes occur
– TCP doesn’t detect lost
connectivity on its own
AS1
• Detecting a failure
– Keep-alive: 60 seconds
– Hold timer: 180 seconds
• Reacting to a failure
– Discard all routes learned from neighbor
– Send updates for any routes that change
AS2
Routing Change: Before and After
0
0
(2,0)
(2,0)
(1,0)
(1,2,0)
1
2
1
2
(3,2,0)
(3,1,0)
3
3
Routing Change: Path Exploration
• AS 1
– Delete the route (1,0)
– Switch to next route (1,2,0)
– Send route (1,2,0) to AS 3
0
(2,0)
• AS 3
– Sees (1,2,0) replace (1,0)
– Compares to route (2,0)
– Switches to using AS 2
(1,2,0)
1
2
(3,2,0)
3
Routing Change: Path Exploration
• Initial situation
– Destination 0 is alive
– All ASes use direct path
• When destination dies
– All ASes lose direct path
– All switch to longer paths
– Eventually withdrawn
(2,0)
(2,1,0)
(2,3,0)
(2,1,3,0)
(1,0)
(1,2,0)
(1,3,0)
1
2
0
• E.g., AS 2
– (2,0)  (2,1,0)
– (2,1,0)  (2,3,0)
– (2,3,0)  (2,1,3,0)
– (2,1,3,0)  null
3
(3,0)
(3,1,0)
(3,2,0)
Discussion of
BGP Instability Paper
29