BGP: Introduction and Issues
Advanced Networks
1
What Is BGP?
Border Gateway Protocol BGP-4
The de-facto interdomain routing protocol
BGP includes specifications:
• Which information gets advertised and how
BGP includes a routing protocol:
• Establishes and uses a routing table
Internal Gateway Protocol (I-BGP in the
book)
Advanced Networks
2
Why Is There Such Fuss about BGP?
BGP dictates routing at the AS level
• Absence of understanding: poor performance
BGP is complicated
• Designed to be flexible
• Involves multiple fields
Understanding BGP behavior is not intuitive
• Implementation and business policies
The routing of the Internet relies on BGP
Advanced Networks
3
Some Open Questions
How well does BGP work now?
How can I manage a BGP network?
How secure and robust is BGP?
• Cyber-terrorism
How would we re-design BGP now?
How well will BGP scale for our future
needs?
Advanced Networks
4
Roadmap
Introduction to BGP
Highlights of BGP issues
Goal: instigate interest in BGP
Advanced Networks
5
Some Basic Numbers
• 14,000 Autonomous Systems approx.
• Corporate Networks
• ISP Internal Networks
• National Service Providers
• Identified by ASN a 16 bit value
• Assigned by IANA
• Superlinear growth (Huston, Siganos et al.)
Advanced Networks
6
How A BGP graph Looks Like
AS 2
AS 5
AS 4
AS 3
Each AS has
designated BGP
routers
BGP routers of an
AS communicate
internally with
another protocol
(IGP)
AS 1
Advanced Networks
7
IP Addresses and Prefixes
IP addresses have 32 bits: 4 octets of bits
(IPv4)
A prefix is a group of IP addresses
128.32.101.5 is an IP address (32 bits)
128.32.0.0/16 is a prefix of the 16 first bits:
• 128.32.0.0 – 128.32.255.255
(2^16 addresses)
128.32.4.0/24 is a prefix of the 24 first bits longer
Advanced Networks
8
Routing is Based on Prefixes
A BGP Routing table has prefixes for entries
For a IP address of a packet, find longest
match
Example: packet IP 128.32.101.1
128.1.1.4 matches the first 8 bits – no match!
128.32.0.0/16 match for 16 bits
128.32.101.0/24 is a longer match
Advanced Networks
9
Prefix Matching in More Detail
For a IP address of a packet, find longest match
Example: Compare
•
•
•
•
•
•
•
packet IP 128.32.101.1
With 128.32.0.0/16
IP
: 01000000. 001000000. 01100101 .00000001
Mask : 11111111. 111111111. 00000000 .00000000
AND : 01000000. 001000000. 00000000 .00000000
Prefix : 01000000. 001000000. 00000000. 00000000
Equal? Yes
Advanced Networks
10
Advertising Routing Information
Each AS advertises what it can reach from
each BGP router
Policies I: filter what you advertise
Policies II: filter from what you hear
advertised
Build up a BGP routing table
• Remember which prefix you hear from which link
Advanced Networks
11
What Does a Routing Table Look Like?
Prefix
Origin
AS
Path
128.32.0.0/16
123
14 56 123
123
34 101 203 123
128.32.101.0/24 15
50 15 15
Origin AS “owns” the address
Routing tables can have peculiarities
Advanced Networks
12
Route Advertising
Distance Vector style protocol
Hear advertisements: IP prefix, AS-path
Filter if desired (i.e. ignore)
Append yourself: IP prefix, myAS+AS-path
Forward to appropriate ASs
Advanced Networks
13
Basic AS relationships
Customer – Provider
• Customer pays Provider for service
• The Customer is always right
Peer to Peer: mutual cooperation
• Ex. MCI and AT&T
Sibling-Sibling
• Ex. AT&T research and AT&T wireless
Advanced Networks
14
The Internet as a Directed Graph
Every edge is
bidirectional
Business relationships
are represented
Provider
Peer
Advanced Networks
Customer
Peer
15
The Initial Idea
Data flows between customers-providers
Top level providers are peers
• They exchange information to ensure connectivity
What can possibly go wrong?
Advanced Networks
16
And then came the rain…
Thousands of ASs
Complicated relationships
Multiple providers for one AS!!
• Multihoming
Traffic engineering
• I want to use multiple paths and load balance
Advanced Networks
17
Example: The Intended Use
Provider
Peer
Advanced Networks
Customer
Peer
18
BGP Graph and Routing Policies
200
100
10
11
1
2
12
3
13
4
Up then down: 1, 10, 100, 200, 13, 4
No valleys, no up-down-up, no more than 1 peer-peer
Advanced Networks
19
Examples of Illegal Paths
1, 10, 100, 11, 200, 13,4
1, 10, 100, 11, 12, 3 (Why?)
1, 10, 11, 12, 3
1, 10, 100, 20, 11, 2 (is this illegal?)
• Providers prefer to send traffic through customers
Advanced Networks
20
The Rules of BGP Routing
Transit traffic: traffic that does not go to my
customers (or their customers)
A provider carries any traffic to, from
customer
Peers exchange traffic only if between their
customers
Advanced Networks
21
Implementing BGP Rules
A customer advertises whatever she wants
A provider forwards everything from customer
• So that world now where customer IPs are
A provider advertises whatever customer wants to
customer
A peer hears but does not advertise further prefixes
from a peer
A peer advertises only its customers to a peer
Advanced Networks
22
Some Simple Policies: Transitivity
ISP 1
ISP 2
Not allowed
AS X
• A customer should not be transit for its Providers
• For this: ASX should not advertise ISP1 advertisements
Advanced Networks
23
Basic AS Relationships
200
100
10
11
1
2
12
13
3
4
Provider
Peer
Customer
Peer
Not if: 11- 12 peers
Yes if: 11-12 siblings
Customer – Provider: customer pays and is always right
Peer to Peer: Exchange traffic only between their customers
Sibling-Sibling: Exchange traffic at will
Advanced Networks
24
How Policy Restricts Routing
Provider
100
200
Peer
Customer
Peer
10
11
1
2
12
13
3
4
Routing rules:
Path Properties:
• Provider accept everything
• Peer only if it is for its
customers
• Up then down
• No up-down-up, at most 1
peer-peer steps
Advanced Networks
25
How Policies Affect Routing
Customer 1
ISP1
A Provider will get rid of
traffic as soon as
possible,
But a Provider will carry
the traffic for its
customer
Did anyone say traffic is
asymmetric?
ISP2
Customer 2
Advanced Networks
26
BGP Path-Length Asymmetry
Consider number of AS traversed by a path
Asymmetry: 46% of pairs differ by at least one AS hop
[Siganos 01]
Advanced Networks
27
Determining The Logical Graph
The business relationships are critical
How can I find the relationships?
1. Infer relationships from routing tables
2. IRR database: manually maintained – error prone
Advanced Networks
28
Two Inference Algorithms
Inference algorithm [Gao 00]
• Using 1 routing table
• Exploit the up-down path property
in a routing path, assume highest degree node as peak
Inference using multiple tables [Subramanian02]
• Use multiple points of observation to improve results
Advanced Networks
29
Things Become Interesting:
Traffic Engineering
AS 2
LP 80
LP 100
slow
208.1.1.0/24
AS 1
How can I pick a route?
Local Preference: path
attribute
AS2 wants to prefer fast
thick link
Advertisement from
right router of AS2 has
higher Local Preference
Any BGP router in AS2
will prefer the
Advanced Networks
30
Load Balancing - Appetizer
I want to share traffic between my two
providers
How can I do this?
Advanced Networks
31
Load Balancing: Long Prefix Match
Takes All!
138.39/16
ISP 3
138.39.1/24
ISP 2
138.39.1/24
ISP 1
138.39/16
138.39.1/24
Customer 138.39.1/24
Advanced Networks
32
So How Can I Balance the Load?
Ask my provider to not aggregate my prefix
• Will this work?
Split my prefix in two
• 138.39.1.0/24
• A: 138.39.1.31/28
• B: 138.39.1.32/28
Advertise only one part to ISP2
ISP2 traffic destined for prefixes in A
ISP1 traffic destined for prefixes in B
Advanced Networks
33
Summary Up to Now
BGP-4 is the de facto protocol for
interdomain routing
BGP was developed to achieve:
• Flexible policy implementation
• Scalability via route aggregation given CIDR
There are many open issues
• BGP is a hot research topic
Advanced Networks
34
The Growth of BGP Table
Advanced Networks
35
The BGP Growth: The Truth
Growth flattened
out
Why?
• Better
management
• Dot-com crash?
Advanced Networks
36
Routing Table Variation
Larger ASes have significantly larger tables
Advanced Networks
37
Update Activity Per Prefix
Measure rate of announcements +
withdrawals + path updates
Compare relative update rate per prefix
length to the relative number of prefixes
of that length
>1 implies higher than average update rate
(less stable)
 <1 implies lower than average update
rate(more stable)

Advanced Networks
38
Measured Update Rate
Bursty!
Advanced Networks
39
BGP robustness
Measuring the BGP updates
Advanced Networks
40
Analyzing Messages By Content
Aggregated per 30 seconds
Advanced Networks
41
Initial Observations
Updates show daily and weekly periodicity
There is no evidence BGP disturbance:
• The Baltimore tunnel train 18 July that destroyed
Internet lines
• Sept 11 attack
There are some spikes at:
• 19 July
• 18-22 September
Advanced Networks
42
BGP Updates Correlations
Advanced Networks
43
BGP Under Attack
Advanced Networks
44
Router CPU Activity Correlates…
Advanced Networks
45
The Attack of The Worm
Advanced Networks
46
Classification of Instabilities
Advanced Networks
47
Isolating Instability: 1 unstable peer
Advanced Networks
48
Code Red Worm
Advanced Networks
49
The Worm Activity
Advanced Networks
50
The Worm Correlates Again…
Advanced Networks
51
Advanced Networks
52
Advanced Networks
53
Advanced Networks
54
Advanced Networks
55
Advanced Networks
56
Advanced Networks
57
Summary of BGP Instability
Globally correlated BGP instability is not
uncommon
Some causes are well understood
(misconfiguration, bad path announcements)
Some others are less well understood, and
more worrisome:
• worms
Advanced Networks
58
BGP: Up Close and Personal
Establishing a connection
Messages
Path Attributes
Advanced Networks
59
Establishing A BGP Session
BGP uses TCP connections
• For reliability
A BGP session is between two routers
• Typically directly connected (Ethernet, FDDI)
Routers establish a BGP session
• Authentication and set-up
• Update and withdrawals
• If disconnected, all paths are invalidated
Advanced Networks
60
Messages
First open TCP connection
• Identification and authentication
BGP messages
•
•
•
•
OPEN: set-up, negotiate timer for keep-alive
UPDATE: routing changes
NOTIFICATION: termination, and error messages
KEEPALIVE: confirm that connection is active
Advanced Networks
61
UPDATE Message
Advertise reachability information
Withdraw paths to prefixes
Update information prefixes
Introduce new prefixes
Modify important path attributes for new
prefixes and the related paths
Advanced Networks
62
Standard Path Attributes
Attribute: Type, Length, Value
Origin: where did I hear this from
• BGP (external) or IGP (internal)
AS Path: sequence of ASs
Flexible handling of loops
• Recovering from disconnected ASs!
Next Hop: set explicitly who the next router
should be (possibly a non BGP speaker)
Advanced Networks
63
Multi-Exit Discriminator (MED)
MED 10
AS 2
AS 1
MED 50
•Indication to external peers of the
preferred path into the AS
•Lowest Med Preferred
Advanced Networks
64
Note for MED
One AS sets the values
Another AS interpets and uses them
Thus:
• Cooperative ASes
• Only between two ASes (1 hop scope)
MED is meaningless in the next hop
MED can be used only if both routes are
advertised from the same AS
Advanced Networks
65
Local Preference
AS 2
AS 1
L1
Choosing paths
internally
Within an AS:
• Set Local
Preference to pick
the path you want
to send data to
L2
AS3
Here MED can not be used since we
have different AS
Advanced Networks
The higher Local
Preference is
preferred
66
Atomic Aggregate
Sometimes we aggregate paths that are
different (ie different AS sequence)
Atomic aggregate shows that some of the
destinations in this prefix are not necessarily
following the same path
This does not allow other routers to deaggregate the path, thus creating entries that
should not appear
Advanced Networks
67
Atomic Aggregate
138.39/16
:14
AS 3
138.39/16
:314
AS 1
138.39/16
138.39.1/24
AS 4
138.39/16
AS 2
138.39.1/24
AS 5
AS 2 thinks that all 138.39/16 follow the 3 1
4 path
68
Advanced Networks
BGP Route Selection Process
1.
2.
3.
4.
5.
6.
Maximum prefix length match
Highest Local Priority
Shortest AS Path
Lowest MED (if routes through same AS)
Min Cost Next hop router (consulting IGP)
Prefer external to internal routes
1. Pick lowest BGP identifier among many E-BGP
2. Pick lowest BGP identifier among many I-BGP
Advanced Networks
69
Advanced Networks
70
Internal BGP (I-BGP)
Communication between routers of an AS
I-BGP very similar to E-BGP except:
• Different advertising rules
Do not re-advertise a path internally
• I-BGP 1 -> I-BGP 2 -X- I-BGP3
But readvertise:
• I-BGP 1 -> I-BGP 2 -> E-BGP1
• E-BGP 1 -> I-BGP 2 -> I-BGP1
Advanced Networks
71
I-BGP Re-Advertising
I-BGP
E-BGP
A
B
D
C
B will not re-ad. to C
what it hears from A
But it will re-ad to D
Why?
Paths are identified
by AS, and internally
you have the same
AS
• To avoid routing loops
Advanced Networks
72
I-BGP Mesh
I-BGP routers form a fully connected mesh
(clique)
• Scalability becomes an issue
The full mesh is independent of physical
connectivity
Advanced Networks
73
A Subtle Difference I-BGP vs E-BGP
E-BGP: routers communicate using the IP of
the physical interface (link)
• Link based reliability
I-BGP: routers have “virtual or loopback”
interface
• Even when link fails, routers may be reachable
• Node based reliabiity
Advanced Networks
74
Static vs Dynamic Configuration
In practice, many customers do not speak
BGP
Do not have an AS number
They are configured statically
Advanced Networks
75
One vs Many Providers
Single homed vs multihomed
Customers for reliability and performance
connect to many providers
Difference:
• Single homed: easy to manage
• Multihomed: tricky
 Route aggregation
 Load balancing
Advanced Networks
76
Multihoming
How can I use my multiple connections
efficiently?
Multihoming is quite widespread
Users take it up to themselves [Huston]
Consequence: non-aggregatable state
Advanced Networks
77
Two routers two links
Multiplex traffic at link or IP layer
Use Next Hop to point to virtual router IP
• Second look up: how can I reach virtual IP?
• Pick one of the two links randomly or statically
• Furthermore, when one link is down the other one is
chosen
Advanced Networks
78
Multihoming with One Routers 1 ISP
ISP 1
B
C
P1
A
P2
Customer can advertise
different prefixes on each
link
• Reliability?
Use Multi-Exit-Discriminator
Use Local Preference
• ISP to pick link
• Customer’s IBGP to pick router
Advanced Networks
79
Issues
The mechanisms are there, but they depend
on many factors
• How much traffic each prefix generates?
Balancing incoming and outgoing traffic
Dynamically adapting to changing conditions
Technical issues:
• Some ISPs do not accept very long prefixes
 Longer than they would be in classful routing
 Longer than 19 for new prefixes
Advanced Networks
80
Multihoming with Two Routers 1 ISP
Internet
Similar case for Provider to
Customer
For Customer to Provider:
ISP 1
C
B
ISP 1
Customers
A
P1
P2
• A could alternate paths
 Reordering of packets
• ISP 1 could advertise different
addresses on each link
 I.e. 1: ISP customers

2: Default (everybody
else)
Advanced Networks
81
Multihoming With Two Providers
138.39/16
ISP 3
138.39.1/24
ISP 2
138.39.1/24
ISP 1
138.39/16
138.39.1/24
Customer 138.39.1/24
Advanced Networks
82
Two Providers Multihoming: Getting
Address Space
Given two ISP
Get space from one (we saw before)
Get space from both
• Advertise only related prefix to ISP
 Aggregation but not reliability
• Advertise both prefixes to ISPs
 No aggregation but reliability
Get space independently of both
• Max flexibility, not reliability
• If too narrow of prefix, may not propagate, no connectivity
Advanced Networks
83
I-BGP Scalability
Full mesh is not scalable: O(N^2) IBGP sessions
Approaches to scalable I-BGP
• Hierarchical structure: Route reflectors
• Divide and conquer: Confederations
Advanced Networks
84
Route Reflection
D
E
B
C
A
Explicitly allow some IBGP routers to
readvertise
Route reflectors:
represent other routers
Hierarchical structure
avoids loops and
problems
Advanced Networks
85
Confederations
D
E
B
C
A
Decompose an AS to
sub-AS
Externally: one AS
Internally: like E-BGP
(E-I-BGP)
Loop avoidance:
• AS-CONFED-SET
• AS-CONFED-SEQUENCE
Advanced Networks
86
Confederation BGP Rules
Differences of BGP between sub-ASes
Local-Preference is transitive
• Recall in BGP it is not
NEXT-HOP attribute is forwarded
Path within AS is monitored with sub-AS no.
• AS-CONFED-SEQ
• AS-CONFED-SEQUENCE
Advanced Networks
87
Handling the Dynamic Nature
Advanced Networks
88
To Refresh or not to Refresh?
BGP uses hard state:
BGP routers consider a path is “usable”
• until explicitly withdrawn
• the session fails
How do I detect if a connection failed?
• Keep-alive messages
Advanced Networks
89
Path Updates Frequency
Send updates of a path no sooner 30 sec
Why?
• Stability
• Overhead reduction
Side-effects
• Convergence can take longer
What is the right interval?
• Recent studies say that 30s is too long
Advanced Networks
90
Route Flapping and Dampening
Flapping: constant alternating updates
It can happen!
Route update dampening:
• Consider stability of path before using it
How store “penalty” value for each path
Issue: router needs to remember withdrawn
paths
Advanced Networks
91
Overview up to Now
BGP is Distance Vector
BGP uses TCP and hard-state
Routing updates are “delayed” and batched
Route dampening to alleviate instabilities
Advanced Networks
92
End
Advanced Networks
93
Current Research: the AS Topology
Characterize the topology using power-laws
(Faloutsos 99, Siganos et al 01)
Modeling the evolution of the topology
(Barabasi, Siganos01)
Advanced Networks
94
Ongoing Research: AS paths
107 Gb of AS paths over 3 years
Exponential growth, but distances remain the
same
Inflation due to policy:
• 20% of paths are larger than they could
Significant Routing Asymmetry:
• 40% paths by at least one hop
Advanced Networks
95
Near Future Research Plans
Mine the collected paths for patterns
How stable were the paths?
Can we identify “illegal” paths?
Identify pathologies (ie. loops)
Advanced Networks
96
Overview of Plans for Research
How well does BGP work now?
How secure and robust is BGP?
• Cyber-terrorism: how much damage can be done?
How would we design BGP now?
• People are asking this question
How well will BGP scale?
How can we manage BGP (avoid human errors)?
Approach:
• Analytical and simulations with SSFNET
Advanced Networks
97
Conclusions
BGP is hot
The community knows very little
Big ticket items:
• Robustness, security
• Network Management
• Scalability
Practitioners are interested in this
• DARPA, NANOG (Network Operations)
Advanced Networks
98
Multihoming With Two Providers
138.39/16
ISP 3
138.39.1/24
ISP 2
138.39.1/24
ISP 1
138.39/16
138.39.1/24
Customer 138.39.1/24
Advanced Networks
99
Policies on Transitivity
Isp 1
Isp 2
Nontransit
A customer should not
Be transit for its Providers
Not allowed
AS X
Transit
AS 1
AS 3
AS 2
Advanced Networks
AS 4
100
More BGP attributes
•Communities
•Confederations
•Route Reflectors
Advanced Networks
101
Convergence
Advanced Networks
102
Multi-Exit Discriminator(MED)
•Indication to external peers of the
preferred path into the AS
•Lowest Med Preferred
Advanced Networks
103
Note in MED
One AS sets the values
Another AS interpets and uses them
Thus:
• Cooperative ASes
• Only between two ASes (1 hop scope)
MED is meaningless in the next hop
MED can be used only if both routes are
advertised from the same AS
Advanced Networks
104
Local Preference
Choosing paths internally
Within an AS:
• Set Local Preference to pick the path you want to
send data to
The higher Local Preference is preferred
Advanced Networks
105
Atomic Aggregate
Sometimes we aggregate paths that are
different (ie different AS sequence)
Atomic aggregate shows that some of the
destinations in this prefix are not necessarily
following the same path
Advanced Networks
106
Aggregator
Notify that an aggregation took place
• Which AS
• Which router
For management and traceback purposes
Advanced Networks
107