Stub Traffic Engineering
Jennifer Rexford
Fall 2010 (TTh 1:30-2:50 in COS 302)
COS 561: Advanced Computer Networks
http://www.cs.princeton.edu/courses/archive/fall10/cos561/
Why Connect to Multiple Providers?
• Reliability
– Reduced fate sharing
– Survive ISP failure
• Performance
Provider 1
Provider 2
– Multiple paths
– Select the best
• Financial
– Leverage through competition
– Game 95th-percentile billing model
Stub: an Autonomous System that does not provide transit
Outline
• Outbound traffic
– Shortest-path routing
– Primary and backup providers
– Load balancing over multiple links
• Inbound traffic
– Primary and backup providers
– Selective advertising
– BGP communities
• Looking forward
– Control by routers vs. end hosts
– Exposing greater path diversity
– Stability of selfish routing
Controlling Outbound Traffic
Outbound Traffic: Pick a BGP Route
• Easier to control than inbound traffic
– IP routing is destination based
– Sender determines where the packets go
• Control only by selecting the next hop
– Border router can pick the next-hop AS
– Cannot control selection of the entire path
Provider 1
“(1, 3, 4)”
Provider 2
“(2, 7, 8, 4)”
Outbound Traffic: Shortest AS Path
• No import policy on border router
– Pick route with shortest AS path
– Arbitrary tie break (e.g., router-id)
d
• Performance?
– Shortest path is not necessarily best
– Propagation delay or congestion
• Load balancing?
– Could lead to uneven split in traffic
– E.g., one provider with shorter paths
– E.g., too many ties with a skewed tie-break
s
Outbound Traffic: Primary and Backup
• Single policy for all prefixes
– High local-pref for session to primary provider
– Low load-pref for session to backup provider
• Outcome of BGP decision process
– Choose the primary provider whenever possible
– Use the backup provider when necessary
• But…
– What if you want to balance traffic load?
– What if you want to select better paths?
Outbound Traffic: Load Balancing
• Selectively use each provider
– Assign local-pref across destination prefixes
– Change the local-pref assignments over time
• Useful inputs to load balancing
– End-to-end path performance data
 E.g., active measurements along each path
– Outbound traffic statistics per destination prefix
 E.g., packet monitors or router-level support
– Link capacity to each provider
– Billing model of each provider
Outbound Traffic: What Kind of Probing?
• Lots of options
– HTTP transfer
– UDP traffic
– TCP traffic
– Traceroute
– Ping
• Pros and cons for each
– Accuracy
– Overhead
– Dropped by routers
– Sets off intrusion detection systems
• How to monitor the “paths not taken”?
Outbound Traffic: How Often to Change?
• Stub ASes have no BGP customers
– So, routing changes do not trigger BGP updates
• TCP flows that switch paths
– Out-of-order packets during transition
– Change in round-trip-time (RTT)
• Impact on the providers
– Uncertainty in the offered load
– Interaction with their own traffic engineering?
• Impact on other end users
– Good: move traffic off of congested paths
– Bad: potential oscillation as other stub ASes adapt?
Controlling Inbound Traffic
Inbound Traffic: Influencing Others
• Harder to control than outbound traffic
– IP routing is destination based
– Sender determines where the packets go
• Control only by influencing others’ decisions
– Static configuration of the providers
– BGP route attributes sent by the stub
– Selective advertising of destination prefixes
Provider 1
Provider 2
Inbound Traffic: Primary and Backup
• Ask your provider to be a backup
– Provider violates “prefer customer” policy
– … by assigning lower local-pref to customer
– Backup link is only used if the primary link fails
local-pref=100
Provider 1
Provider 2
local-pref=90
traffic
12.34.158.0/24
Inbound Traffic: AS Prepending
• Make one path look longer
– Advertise short path one way
– … and longer path another
– In the hope of influencing choices
– But, how much prepending to do?
Provider 1
“12.34.158.024: (3)”
Provider 2
“12.34.158.024: (3, 3, 3)”
Inbound Traffic: Prepending Fail
• Example where prepending doesn’t work
– Customer does prepending of AS path
– Provider has a “prefer customer” policy
– Provider 2 prefers the longer path
“12.34.158.024: (1, 3)”
Provider 1
“12.34.158.024: (3)”
Provider 2
“12.34.158.024: (3, 3, 3)”
Inbound Traffic: Programming Your Provider
• Better to have selective control over provider
– Tell the provider whether to prefer your route
– … on a per-prefix basis, with changes over time
• Enables adaptive load balancing
– Without asking provider to reconfigure policy
Provider 1
Provider 2
12.34.158.0/24
Inbound Traffic: BGP Communities
• Provider and customer agree on a “tag”
– One tag mean “primary” and the other “backup”
– Customer includes tags in BGP advertisements
– Provider sets local preference based on tags
• BGP community attribute
– Opaque attribute with no real meaning
 Two numbers: usually AS number and arbitrary number
– Sprint example (http://www.sprint.net/policy/bgp.html)
 1239:70 means “assign local pref of 70”
 …
 1239:110 means “assign local pref of 110”
See RFC 1998
Example: ISP Setting Local-Pref
• Customers
– 110: Primary path
– 100: Secondary path
– 80: Primary backup path
– 70: Secondary backup path
Peer
• Peers
– 81-99: In between
– Range for traffic engineering
Customer
Inbound Traffic: Selective Advertising
• When you don’t have enough prefixes…
– Advertise one subnet to each provider
– Advertise the supernet to both providers
– Traffic splits due to the longest-prefix match
– Supernet ensures backup connectivity after failure
Provider 1
12.34.0.0/16
12.34.0.0/17
Provider 2
12.34.0.0/16
12.34.128.0/17
Causes unwanted increase in global routing tables
Inbound Traffic: Multiple Addresses
• Multiple external addresses for a service
– One IP address for each entry point
• Use DNS to adjust mapping of name to address
– Give different answers to different clients
– Adjust over time to performance, cost, traffic, etc.
Provider 1
Provider 2
12.34.1.0/24
5.6.7.0/24
12.34.1.2
5.6.7.8
20
Inbound Traffic: Tunneling
• Cooperation with sender
– Sending network picks the
ingress point
– Encapsulates the packet
with the interface address
Encapsulate with
12.34.1.1 or 5.6.7.8
• Advantages
– Sender may
know best
Provider 1
Provider 2
• Disadvantages
– Requires cooperation
– Failure recovery
12.34.1.1
12.34.1.2
5.6.7.8
21
Looking Forward
22
Greater Path Diversity
• Performance depends on the end-to-end path
– Need greater path diversity
• How to get more paths?
– Add more providers? (expensive)
– Support multipath routing?
• Ways to support multipath routing
– Source routing
– Provider offering multipath as a service
– Overlays (e.g., RON)
– Deflecting packets through intermediate Ases
– Etc.
23
Overlay Example: RON
Premise: by building application overlay network,
can increase performance and reliability of routing
Princeton
application-layer
router
Yale
Two-hop (application-level)
Berkeley-to-Princeton route
Berkeley
http://nms.csail.mit.edu/ron/
Overlay Example: RON
• Exchange the results of the probes
– Each host shares results with every other host
– Essentially running a link-state protocol!
– So, every host knows the performance properties
• Forward through intermediate host when needed
BB
A
C
Discussion: Who’s in Charge?
• Routers
– Reduces complexity and overhead the end hosts
– Avoids placing (unwarranted?) trust in end hosts
– Managed based on an organization’s goals
– Lower performance measurement overhead
• End hosts
– Best equipped to judge if performance is good enough
– Decisions customized to the specific application
– Finer-granularity of load-balancing decisions
– Monitor performance by observing their own traffic
26
Other Discussion
• At what layer or component (e.g., DNS, BGP,
overlays, application, etc.)?
• How to get extra path diversity?
• Who pays for extra resources?
• How to prevent global inefficiency or instability?
• What is a good clean-slate solution?
27