Internet Routing (COS 598A)
Today: Interdomain Topology
Jennifer Rexford
http://www.cs.princeton.edu/~jrex/teaching/spring2005
Tuesdays/Thursdays 11:00am-12:20pm
Outline
• Interdomain topology
– AS graph
– Inferring the topology from routing data
– Structure of the AS graph
• AS relationships
– Common pair-wise relationships
– Inferring the relationships from routing data
– Characteristics of the relationship graph
• Peering policies
– Example of AOL’s peering agreement
What is the AS Graph?
• Node: Autonomous System
• Edge: Two ASes that speak BGP to each other
4
3
5
2
1
7
6
What is an Edge, Really?
• Edge in the AS graph
– At least one BGP session between two ASes
– Some destinations reached from one AS via the other
d
d
AS 1
AS 1
Exchange Point
AS 2
AS 2
AS 3
How Do You Know a Node or Edge Exists?
• Consult the Whois database?
– Tells which ASes have been allocated
– But, might be out-of-date on who owns it
– … and often doesn’t say who the neighbors are
• See a path that uses the node/edge
– Collect measurements of AS paths
– Extract all of the nodes and edges
– E.g., AS path “7018 1 88” implies
• Nodes: 7018, 1, and 88
• Edges: (7018, 1) and (1, 88)
Interdomain Routing: Border Gateway Protocol
• ASes exchange info about who they can reach
– IP prefix: block of destination IP addresses
– AS path: sequence of ASes along the path
• Example: a BGP route in AS 7018 shows
– Prefix 12.34.158.0/24 has path “7018, 1, 88”
“12.34.158.0/24: path (88)”
“12.34.158.0/24: path (7018,1,88)”
88
1
7018
data traffic
data traffic
12.34.158.5
Where to Get BGP Routes: Public Servers
4
7018
1221
701
7
80
3786
9.184.112.0/20
3.0.0.0/8
BGP sessions
Example: BGP Table (“show ip bgp” at RouteViews)
Network
* 3.0.0.0/8
*
*
*
*
*>
*
* 9.184.112.0/20
*
*>
*
*
*
Next Hop
Metric LocPrf Weight Path
205.215.45.50
0 4006 701 80 i
167.142.3.6
0 5056 701 80 i
157.22.9.7
0 715 1 701 80 i
195.219.96.239
0 8297 6453 701 80 i
195.211.29.254
0 5409 6667 6427 3356 701 80 i
12.127.0.249
0 7018 701 80 i
213.200.87.254
0 3257 701 80 i
205.215.45.50
0 4006 6461 3786 i
195.66.225.254
0 5459 6461 3786 i
203.62.248.4
0 1221 3786 i
167.142.3.6
0 5056 6461 6461 3786 i
195.219.96.239
0 8297 6461 3786 i
195.211.29.254
0 5409 6461 3786 i
AS 80 is General Electric, AS 701 is UUNET, AS 7018 is AT&T
AS 3786 is DACOM (Korea), AS 1221 is Telstra
Characteristics of the AS Graph
• AS graph structure
– High variability in node degree (“power law”)
– A few very highly-connected ASes
– Many ASes have only a few connections
CCDF
1
All ASes have degree >= 1
0.1
0.01
Very few have degree >= 100
0.001
1
10
100
1000
AS degree
Characteristics of AS Paths
• AS path may be longer than shortest AS path
• Router path may be longer than shortest path
2 AS hops,
8 router hops
d
s
3 AS hops, 7 router hops
Problem of Missing Edges
• Limited collection of paths
– Some edges might never be traversed
– Especially low in the AS hierarchy
– … and backup links
• Example: paths from two tier-1 ISPs miss an edge
Sprint
AT&T
d1
Harvard
???
Harvard
B-school
d2
Problem of Missing Nodes
• Route aggregation
– AS advertises a larger address block
– Smaller address block not seen everywhere
– Can cause an AS not to appear in BGP table
C: 12.0.0.0/8
A
B
D D: 12.1.0.0/16
C
E E: 12.2.0.0/16
– C’s table: has paths “C”, “C D”, and “C E”
– B’s table: has only path “C” for 12.0.0.0/8
Research Questions
• Incomplete data
– How to get more data?
– How much does missing data affect answers?
– What kinds of questions can be answered safely?
AS Relationships
Interdomain Routing Policies
• Two main decisions
– Path selection: which of the paths to use?
– Path export: which neighbors to tell?
• Both driven by business relationships, e.g.,
– Customer pays provider for Internet access
– Peers find it mutually advantageous to cooperate
“12.34.158.0/24: path (2,1)”
3
“12.34.158.0/24: path (1)”
1
2
data traffic
data traffic
12.34.158.5
Customer-Provider Relationship
• Customer needs to be reachable from everyone
– Provider exports routes learned from customer to everyone
• Customer does not want to provide transit service
– Customer does not export from one provider to another
Traffic to the customer
Traffic from the customer
d
provider
advertisements
provider
traffic
customer
d
customer
Peer-Peer Relationship
• Peers exchange traffic between customers
– AS exports only customer routes to a peer
– AS exports a peer’s routes only to its customers
Traffic to/from the peer and its customers
advertisements
peer
d
traffic
peer
Paths You Should Never See (“Invalid”)
Customer-provider
Peer-peer
two peer edges
transit through a customer
Other Kinds of Relationships
• Siblings
–
–
–
–
Same company
Mutual transit service
Like one bigger AS
Mergers, acquisitions, …
• Backup
– Used only when failure
– Second provider
– Backup peering
• Geography-specific
– Customer in U.S.
– Peer in Europe
E
A
B
H
C
D
F
G
primary
backup
AS Relationships Matter
• Scientific understanding
– Understanding Internet structure and evolution
– Understanding why certain paths are used for traffic
• Placement of Web servers
– Want to be close to most customer networks
• Business decisions
– Selecting new peer or provider, or renegotiating relations
• Security policies
– Knowing which BGP routes look suspicious
• Analyzing BGP convergence
– Relationships have a big impact here (more later!)
Inferring AS Relationships
• Top down: how routes are selected
– AS relationships define routing policy
– Routing policy determines the routes you see
• Bottom up: how policies can be inferred
– Routing data are available from public sources
– The chosen routes tell you about the policy
• Example: seeing path “A B C” tells you…
– B permits A to transit through B to reach C
– (A,B) and (B,C) cannot both be peering links
– A and C are not both upstream providers of B
Type-of-Relationship Problem
• Given the inputs
– AS graph G(V,E) with vertices V and edges E
– Set of paths P on the graph G
• Find a solution that
– Labels each edge with an AS relationship
– Minimizes the number of “invalid” paths in P
• Rich area of research work
– http://www-unix.ecs.umass.edu/~lgao/ton.ps
– http://www.cs.princeton.edu/~jrex/papers/infocom02.pdf
– http://www.cs.berkeley.edu/~sagarwal/research/BGP-hierarchy/
– … lots of scope for algorithms-oriented research project
AS Relationship Graph (2002)
• Lowest level: Stubs
– Leaf nodes: no peers or downstream customers
– 8898 of the 10915 ASes (82.5% of ASes)
– Ex: UC Berkeley (25), Princeton (88), AT&T Labs
(6431), and INRIA (1300)
• Next lowest level: Regional ISPs
– Leaf nodes after successive pruning of leaf nodes
– 971 ASes of the 10915 ASes (8.9% of ASes)
– Ex: PacBell (5676), US West (6223), and UUNET
Canada (815)
• Remaining 1046 ASes: Core
AS Relationship Graph (2002), Continued
• Dense core: Tier-1 providers
– (Nearly) fully-connected nodes with no providers
– Around 15-20 ASes in a near-clique
– Ex: Sprint, UUNET, AT&T, Verio, Level3, C&W,…
• Transit core:
– ASes that peer with the dense core and each other
– 129 ASes, including top providers in Europe and Asia
– Ex: UUNET Europe, KDDI, and Singapore Telecom
• Outer core
– All of the remaining ASes in the core
– 897 ASes, including large regional and national ISPs
– Ex: Turkish Telecom and Minnesota Regional Network
Node Degree is Not Enough
• Node degree ignores relationships
– A stub AS may have many upstream providers
– A core AS may have a small number of peers
– Some ASes have customers that don’t have AS numbers
Ideas for Class Projects
• AS relationship inference
– New algorithms for inferring AS relationships
– Longitudinal study of AS relationship graph
– Influence of policies on measuring AS graph
• AS peering policies
– Analysis of incentives to peer or not
– Study of how one AS can game another
– Analysis of whether regulation is necessary to
keep large ASes from locking out smaller ones
• Alternate settlement models
– Associating prices with routes?
– Source-based routing with third-party control?
Peering Policies
• Contracts that outline:
– Operational requirements on peer network
– Backbone and peering capacity requirements
– Number and geographic diversity of peering points
– Volume and ratio of traffic between two peers
– Routing-policy requirements
• AOL’s Settlement-Free Interconnection Policy
– http://www.atdn.net/settlement_free_int.shtml
AOL’s Settlement-Free Interconnection Policy
• Operational requirements on a peer network
– Handle a single-node outage w/o traffic impact
– Single AS number
– Network Operations Center staffed at all times
• Backbone capacity
– At least 10 gigabits/sec between 8 or more cities
– Minimum peering link speed of 622 megabits/sec
• Peering locations (in U.S.)
– At least four locations
– Must include D.C. area, middle of country, Bay
area, and NYC or Atlanta
Efficient Early-Exit Routing
• Diverse peering
locations
Customer B
– Both costs, and middle
• Comparable capacity at
all peering points
Provider B
– Can handle even load
multiple
peering
points
• Consistent routes
Early-exit
routing
Provider A
Customer A
– Same destinations
advertised at all points
– Same AS path length for
a destination at all points
AOL Routing Requirements
• Consistent advertisements
– All customer routes
– At all peering points
– With the same AS path length
• Address blocks
– Routes aggregated as much as possible
– No address blocks smaller than /24
– Address blocks are registered (e.g., with ARIN)
• No default routing
– Only send traffic to destinations AOL advertises
For Next Tuesday’s Class
• Adapting routing inside an AS to the traffic
– “The Revised ARPANET Routing Metric” (1989)
– “Traffic Engineering With Traditional IP Routing
Protocols” (survey paper, 2002)
• Written one-page review of first paper
– Brief summary of the paper
– Reasons to accept the paper
– Reasons to reject the paper
– Suggestions for future research directions
• Just to skim…
– RFC 3272: “Overview and Principles of Internet
Traffic Engineering”