Internet Routing (COS 598A)
Today: Multi-Homing
Jennifer Rexford
http://www.cs.princeton.edu/~jrex/teaching/spring2005
Tuesdays/Thursdays 11:00am-12:20pm
Outline
• Multi-homing
– Motivations: reliability, performance, and financial
– Do you really need to use a routing protocol?
• Controlling outbound traffic
– Shortest-path routing
– Primary and backup providers
– Load balancing over multiple links
• Controlling inbound traffic
– Primary and backup providers
– Selective advertising
– BGP communities
• State-of-the-art today
Why Connect to Multiple Providers?
• Reliability
– Reduced fate sharing
– Survive ISP failure
• Performance
– Multiple paths
– Select the best
• Financial
– Leverage through
competition
– Game 95th-percentile
billing model
Provider 1
Provider 2
The Stub AS Doesn’t Need to Speak BGP…
• Sending traffic
– Assume both providers can reach everyone
– Split traffic however you want (e.g., 50%/50%)
– But… what if a provider can’t reach someone?
– But… what if one provider has a better path?
Provider 1
L1
Provider 2
One static route
L2
0.0.0.0/0  L1, L2
The Stub AS Doesn’t Need to Speak BGP…
• Receiving traffic
– Both providers can announce the prefix into BGP
– Ensures that everyone else can reach you
– But… what if traffic load is very uneven?
Advertise 12.34.158.0/24
Provider 1
traffic
Provider 2
traffic
12.34.158.0/24
Controlling Outbound Traffic
Outbound Traffic: Pick a BGP Route
• Easier to control than inbound traffic
– IP routing is destination based
– Sender determines where the packets go
• Control only by selecting the next hop
– Border router can pick the next-hop AS
– Cannot control selection of the entire path
Provider 1
“(1, 3, 4)”
Provider 2
“(2, 7, 8, 4)”
Outbound Traffic: Shortest AS Path
• No import policy on border router
– Pick route with shortest AS path
– Arbitrary tie break (e.g., smallest router-id)
• Performance?
– Shortest AS path is not necessarily best
– Could have long propagation delay or congestion
• Load balancing?
– Could lead to uneven split in traffic
– E.g., one provider with shorter paths
– E.g., too many ties with skewed tie-break
Outbound Traffic: Primary and Backup
• Single policy for all prefixes
– High local-pref for session to primary provider
– Low load-pref for session to backup provider
• Outcome of BGP decision process
– Choose the primary provider whenever possible
– Use the backup provider when necessary
• But…
– What if you want to balance traffic load?
– What if you want to select better paths?
Outbound Traffic: Load Balancing
• Selectively use each provider
– Assign local-pref across destination prefixes
– Change the local-pref assignments over time
• Useful inputs to load balancing
– End-to-end path performance data
• E.g., active measurements along each path
– Outbound traffic statistics per destination prefix
• E.g., packet monitors or router-level support
– Link capacity to each provider
– Billing model of each provider
Outbound Traffic: Load, Performance, and Cost
• Balance traffic based on link capacity
– Measure outbound traffic per prefix
– Select provider per prefix for even load splitting
– But, might lead to poor performance and high bill
• Balance traffic based on performance
– Select provider with best performance per prefix
– But, might lead to congestion and a high bill
• Balance traffic based on financial cost
– Select provider per prefix over time to minimize
the total financial cost
– But, might lead to bad performance
Outbound Traffic: What Kind of Probing?
• Lots of options
– HTTP transfer
– UDP traffic
– TCP traffic
– Traceroute
– Ping
• Pros and cons for each
– Accuracy
– Overhead
– Dropped by routers
– Sets off intrusion detection systems
Outbound Traffic: Getting Probes on Paths
• Problem
– Router selects one path per prefix
– How to measure the alternate paths?
• Solution #1: special sources (source routing)
– Special IP addresses for probe traffic
– Router configured to forward probe traffic
• Solution #2: special destinations
– Special destination servers in various locations
– At least one destination per provider AS
– Probe traffic sent to each destination
Outbound Traffic: How Much Probing?
• How often?
– Continuously, at some rate
– In response to a perceived problem
• How diverse of destinations?
– Per destination prefix
– Just for popular/important prefixes
– Select servers throughout the Internet
Outbound Traffic: How Often to Change Routes?
• ASes with downstream customers
– Each change leads to BGP updates
– If not, then no new BGP updates occur
• TCP flows that switch paths
– Out-of-order packets during transition
– Change in round-trip-time (RTT)
• Impact on the providers
– Uncertainty in the offered load
– Interaction with their own traffic engineering?
• Impact on other end users
– Good: move traffic off of congested paths
– Bad: potential oscillation as stub ASes adapt?
Controlling Inbound Traffic
Inbound Traffic: Influencing What Others Do
• Harder to control than outbound traffic
– IP routing is destination based
– Sender determines where the packets go
• Control only by influencing others’ decisions
– Static configuration of the providers
– BGP route attributes sent by the stub
– Selective advertising of destination prefixes
Provider 1
Provider 2
Inbound Traffic: Primary and Backup Providers
• Ask your provider to be a backup
– Provider violates “prefer customer” policy
– … by assigning lower local-pref to customer
– Backup link is only used if the primary link fails
local-pref=100
Provider 1
Provider 2
local-pref=90
traffic
12.34.158.0/24
Inbound Traffic: AS Prepending
• Make one path look longer
– Advertise short path one way
– … and longer path another
– In the hope of influencing choices
– But, how much prepending to do?
Provider 1
“12.34.158.024: (3)”
Provider 2
“12.34.158.024: (3, 3, 3)”
Inbound Traffic: Prepending and Prefer-Cust
• Example where prepending doesn’t work
– Customer does prepending of AS path
– Provider has a “prefer customer” policy
– Provider 2 prefers the longer path
“12.34.158.024: (1, 3)”
Provider 1
“12.34.158.024: (3)”
Provider 2
“12.34.158.024: (3, 3, 3)”
Inbound Traffic: Programming Your Provider
• Better to have selective control over provider
– Tell the provider whether to prefer your route
– … on a per-prefix basis, with changes over time
– Enables adaptive load balancing
– … without asking provider to reconfigure policy
Provider 1
Provider 2
12.34.158.0/24
Inbound Traffic: RFC 1998 on BGP Communities
• Provider and customer agree on a “tag”
– One tag mean “primary” and the other “backup”
– Customer includes tags in BGP advertisements
– Provider sets local preference based on tags
• BGP community attribute
– Opaque attribute with no real meaning
• Two numbers: usually AS number and arbitrary number
– Sprint example (http://www.sprint.net/policy/bgp.html)
• 1239:70 means “assign local pref of 70”
•…
• 1239:110 means “assign local pref of 110”
Example: Tier-1 ISP Setting Local-Preference
• Customers
– 110: Primary path
– 100: Secondary path
– 80: Primary backup path
– 70: Secondary backup path
Peer
• Peers
– 81-99: In between
– Range for traffic engineering
Customer
Inbound Traffic: Not Enough Prefixes
• Stub ASes usually have only a few prefixes
– E.g., one prefix, or at most a handful
– Not enough granularity to control traffic
• Solutions: advertise smaller subnets of prefix
– Essentially, create a bunch of smaller prefixes
– And apply the load-balancing techniques
• Advertise selectively
• AS prepending
• Communities to set local-pref
•…
Inbound Traffic: Selective Advertising
• Divide the destination prefix
– Advertise one subnet to each provider
– Advertise the supernet to both providers
– Traffic splits due to the longest-prefix match
– Supernet ensures backup connectivity after failure
Provider 1
12.34.0.0/16
12.34.0.0/17
Provider 2
12.34.0.0/16
12.34.128.0/17
Inbound Traffic: Small Subnets, Big Debate
• The players
– Stub ASes want more control
• Advertise smaller subnets
– ISPs want to limit table size
• Filter BGP advertisements for small blocks
– ARIN/RIPE/APNIC
• Publish guidelines for acceptable block sizes
• Problems
– ISPs not getting paid for their routing tables
– Risk of network crashes when memory is full
– Risk of black-holing a small subnet you filter
Project Ideas
• Intelligent route-control techniques
– Survey approaches to measuring performance
– Evaluation of different measurement approaches
• Techniques for controlling inbound traffic
– Negotiation scheme between ASes
– Economic approaches for balancing the tension
between fine-grain control and table size
• Source routing
– Scalable techniques for stub AS to pick the end-toend route (not just the next-hop AS)
Next Class: Convergence Delay
• Two papers, intradomain and interdomain
– “Analysis of Link Failures in an IP Backbone”
– “Delayed Internet Routing Convergence”
• Reviews of both papers
– Summary
– Why accept?
– Why reject?
– New research directions
• Optional NANOG video
– “Toward Millisecond IGP Convergence”