Internet Topology
COS 461: Computer Networks
Spring 2007 (MW 1:30-2:50 in Friend 004)
Jennifer Rexford
Teaching Assistant: Ioannis Avramopoulos
http://www.cs.princeton.edu/courses/archive/spring07/cos461/
1
Returning the Midterm Exam
• Exam scoring break down
–Average: 90
–High: 99
–Median: low 90s
• See the course Web site
–Exam
–Answer key
2
Goals of Today’s Lecture
• IP routers
– Interface cards
– Switching fabric
– Route processor
• Router-level topology with a single network
– Points of Presence (PoPs)
– Backbone and enterprise network topologies
• AS-level topology of the Internet
– Autonomous System (AS) numbers
– Tier-1 ISPs, regional providers, and stub ASes
– Business relationships between ASes
3
IP Routers
4
Inside a High-End Router
Processor
Line card
Line card
Line card
Line card
Switching
Fabric
Line card
Line card
5
Router Physical Layout
Juniper T series
Crossbar
Linecards
Cisco 12000
6
Line Cards (Interface Cards, Adaptors)
• Interfacing
– Physical link
– Switching fabric
to/from link
lookup
Transmit
– Packet forwarding
– Decrement time-to-live
– Buffer management
– Link scheduling
– Packet filtering
– Rate limiting
– Packet marking
– Measurement
Receive
• Packet handling
to/from switch
7
Switching Fabric
• Deliver packet inside the router
– From incoming interface to outgoing interface
– A small network in and of itself
• Must operate very quickly
– Multiple packets going to same outgoing interface
– Switch scheduling to match inputs to outputs
• Implementation techniques
– Bus, crossbar, interconnection network, …
– Running at a faster speed (e.g., 2X) than links
– Dividing variable-length packets into cells
8
Packet Switching
“4” Link 1, ingress
Choose
Egress
Link 1, egress
Link 2, ingress
Choose
Egress
Link 2, egress
Link 3, ingress
Choose
Egress
Link 3, egress
Link 4, ingress
Choose
Egress
Link 4, egress
Link 2
Link 1
R1“4”
Link 3
Link 4
9
Router Processor
• So-called “Loopback” interface
–IP address of the CPU on the router
• Control-plane software
–Implementation of the routing protocols
–Creation of forwarding table for the line cards
• Interface to network administrators
–Command-line interface for configuration
–Transmission of measurement statistics
• Handling of special data packets
–Packets with IP options enabled
–Packets with expired Time-To-Live field
10
Error Reporting
• Examples of errors a router may see
– Router doesn’t know where to forward a packet
– Packet’s time-to-live field expires
• Router doesn’t really need to respond
– Best effort means never having to say you’re sorry
– So, IP could conceivably just silently drop packets
• But, silent failures are really hard to diagnose
– IP includes basic feedback about network problems
– Internet Control Message Protocol (ICMP)
11
Internet Control Message Protocol
• ICMP runs on top of IP
– In parallel to TCP and UDP
– Though still viewed as an integral part of IP
• Diagnostics
– Triggered when an IP packet encounters a problem
 E.g., time exceeded or destination unreachable
– ICMP packet sent back to the source IP address
 Includes the error information (e.g., type and code)
 … and an excerpt of the original data packet for identification
– Source host receives the ICMP packet
 And inspects the except of the packet (e.g., protocol and ports)
 … to identify which socket should receive the error
12
Example: Time Exceeded
• Host sends an IP packet
– Each router decrements the time-to-live field
• If time-to-live field reaches 0
– Router generates an ICMP message
– Sends a “time exceeded” message back to the source
5.6.7.156
1.2.3.7
host
host ...
DNS
host ...
host
DNS
8.9.10.11
Time exceeded
router
router
router
13
Ping: Echo and Reply
• ICMP includes a simple “echo” function
– Sending node sends an ICMP “echo” message
– Receiving node sends an ICMP “echo reply”
• Ping tool
– Tests the connectivity with a remote host
– … by sending regularly spaced echo commands
– … and measuring the delay until receiving the reply
• Pinging a host
– “ping www.cs.princeton.edu” or “ping 12.157.34.212”
– Used to test if a machine is reachable and alive
– (However, some nodes have ICMP disabled… )
14
Router-level topology of a network
15
Intra-AS Topology
• Node: router
• Edge: link
16
Hub-and-Spoke Topology
• Single hub node
–Common in enterprise networks
–Main location and satellite sites
–Simple design and trivial routing
• Problems
–Single point of failure
–Bandwidth limitations
–High delay between sites
–Costs to backhaul to hub
17
Princeton Example
• Hub-and-spoke
–Four hub routers and many spokes
• Hub routers
–Outside world (e.g., AT&T, USLEC, …)
–Dorms
–Academic and administrative buildings
–Servers
18
Simple Alternatives to Hub-and-Spoke
• Dual hub-and-spoke
– Higher reliability
– Higher cost
– Good building block
• Levels of hierarchy
– Reduce backhaul cost
– Aggregate the bandwidth
– Shorter site-to-site delay
…
19
Backbone Networks
• Backbone networks
–Multiple Points-of-Presence (PoPs)
–Lots of communication between PoPs
–Accommodate traffic demands and limit delay
20
Abilene Internet2 Backbone
21
Points-of-Presence (PoPs)
• Inter-PoP links
–Long distances
–High bandwidth
Inter-PoP
Intra-PoP
• Intra-PoP links
–Short cables between
racks or floors
–Aggregated bandwidth
• Links to other networks
Other networks
–Wide range of media and
bandwidth
22
Where to Locate Nodes and Links
• Placing Points-of-Presence (PoPs)
–Large population of potential customers
–Other providers or exchange points
–Cost and availability of real-estate
–Mostly in major metropolitan areas (“NFL cities”)
• Placing links between PoPs
–Already fiber in the ground
–Needed to limit propagation delay
–Needed to handle the traffic load
23
AS-level topology of the Internet
24
Internet Routing Architecture
• Divided into Autonomous Systems
– Distinct regions of administrative control
– Routers/links managed by a single “institution”
– Service provider, company, university, …
• Hierarchy of Autonomous Systems
– Large, tier-1 provider with a nationwide backbone
– Medium-sized regional provider with smaller backbone
– Small network run by a single company or university
• Interaction between Autonomous Systems
– Internal topology is not shared between ASes
– … but, neighboring ASes interact to coordinate routing
25
Autonomous System Numbers
AS Numbers are 16 bit values.
Currently just over 20,000 in use.
•
•
•
•
•
•
•
•
•
Level 3: 1
MIT: 3
Harvard: 11
Yale: 29
Princeton: 88
AT&T: 7018, 6341, 5074, …
UUNET: 701, 702, 284, 12199, …
Sprint: 1239, 1240, 6211, 6242, …
…
26
AS Topology
• Node: Autonomous System
• Edge: Two ASes that connect to each other
4
3
5
2
7
6
1
27
What is an Edge, Really?
• Edge in the AS graph
– At least one connection between two ASes
– Some destinations reached from one AS via the other
d
d
AS 1
AS 1
Exchange Point
AS 2
AS 2
AS 3
28
Interdomain Paths
Path: 6, 5, 4, 3, 2, 1
4
3
5
2
7
1
6
Web server
Client
29
Business Relationships
• Neighboring ASes have business contracts
–How much traffic to carry
–Which destinations to reach
–How much money to pay
• Common business relationships
–Customer-provider
 E.g., Princeton is a customer of AT&T
 E.g., MIT is a customer of Level 3
–Peer-peer
 E.g., Princeton is a peer of Patriot Media
 E.g., AT&T is a peer of Sprint
30
Customer-Provider Relationship
• Customer needs to be reachable from everyone
– Provider tells all neighbors how to reach the customer
• Customer does not want to provide transit service
– Customer does not let its providers route through it
Traffic to the customer
Traffic from the customer
d
provider
advertisements
provider
traffic
customer
d
customer
31
Customer Connecting to a Provider
Provider
1 access link
Provider
2 access routers
Provider
2 access links
Provider
2 access PoPs
32
Multi-Homing: Two or More Providers
• Motivations for multi-homing
–Extra reliability, survive single ISP failure
–Financial leverage through competition
–Better performance by selecting better path
–Gaming the 95th-percentile billing model
Provider 1
Provider 2
33
Princeton Example
• Internet: customer of AT&T and USLEC
• Research universities/labs: customer of Internet2
• Local non-profits: provider for several non-profits
AT&T
USLEC
Internet2
34
Peer-Peer Relationship
• Peers exchange traffic between customers
– AS exports only customer routes to a peer
– AS exports a peer’s routes only to its customers
– Often the relationship is settlement-free (i.e., no $$$)
Traffic to/from the peer and its customers
advertisements
peer
d
traffic
peer
35
AS Structure: Tier-1 Providers
• Tier-1 provider
– Has no upstream provider of its own
– Typically has a national or international backbone
– UUNET, Sprint, AT&T, Level 3, …
• Top of the Internet hierarchy of 12-20 ASes
– Full peer-peer connections between tier-1 providers
36
Efficient Early-Exit Routing
• Diverse peering locations
Customer B
– Both costs, and middle
• Comparable capacity at all
peering points
Provider B
– Can handle even load
• Consistent routes
multiple
peering
points
Early-exit
routing
– Same destinations advertised
at all points
– Same AS path length for a
destination at all points
Provider A
Customer A
37
AS Structure: Other ASes
• Tier-2 providers
– Provide transit service to downstream customers
– … but, need at least one provider of their own
– Typically have national or regional scope
– E.g., Minnesota Regional Network
– Includes a few thousand of the ASes
• Stub ASes
– Do not provide transit service to others
– Connect to one or more upstream providers
– Includes vast majority (e.g., 85-90%) of the ASes
38
Characteristics of the AS Graph
• AS graph structure
– High variability in node degree (“power law”)
– A few very highly-connected ASes
– Many ASes have only a few connections
CCDF
1
All ASes have 1 or more neighbors
0.1
0.01
Very few have degree >= 100
0.001
1
10
100
1000
AS degree
39
Characteristics of AS Paths
• AS path may be longer than shortest AS path
• Router path may be longer than shortest path
2 AS hops,
8 router hops
d
s
3 AS hops, 7 router hops
40
Shared Risks
• Co-location facilities (“co-lo hotels”)
– Places ISPs meet to connect to each other
– … and co-locate their routers, and share space & power
– E.g., 32 Avenue of the Americas in NYC
• Shared links
– Fiber is sometimes leased by one institution to another
– Multiple fibers run through the same conduits
– … and run through the same tunnels, bridges, etc.
• Difficult to identify and accounts for these risks
– Not visible in network-layer measurements
– E.g., traceroute does not tell you links in the same ditch
41
Conclusions
• Internet topology
–Network inside a router
–Network inside an AS
–Network connecting ASes
• Coming up: two-tiered routing system
–Intradomain routing
–Interdomain routing
• Reading
– 4.2, 4.3.3, 4.3.4
42