Internet Routing (COS 598A) Today: Interdomain Traffic Engineering Jennifer Rexford http://www.cs.princeton.edu/~jrex/teaching/spring2005 Tuesdays/Thursdays 11:00am-12:20pm Outline • Border Gateway Protocol – BGP protocol – BGP policies – BGP decision process • BGP traffic engineering for outbound traffic – Predicting effects of policy changes – Identifying good policy changes • What about inbound traffic? – AS prepending and MEDs – Inter-AS negotiation Interdomain Routing: Border Gateway Protocol • ASes exchange info about who they can reach – IP prefix: block of destination IP addresses – AS path: sequence of ASes along the path • Policies configured by the AS’s operator – Path selection: which of the paths to use? – Path export: which neighbors to tell? “12.34.158.0/24: path (2,1)” 3 “12.34.158.0/24: path (1)” 1 2 data traffic data traffic 12.34.158.5 Components of BGP • BGP protocol – Definition of how two BGP neighbors communicate – Message formats, state machine, route attributes, etc. – Standardized by the IETF • Policy specification – Flexible language for filtering and manipulating routes – Indirectly affects the selection of the best route – Varies across vendors, though constructs are similar • BGP decision process – Complex sequence of rules for selecting the best route – De facto standard applied by router vendors – Being codified in a new RFC for BGP coming soon BGP Protocol: BGP Sessions Establish session on TCP port 179 AS1 BGP session Exchange all active routes AS2 Exchange incremental updates While connection is ALIVE exchange route UPDATE messages BGP Protocol: Update Messages • Update messages – Advertisement • New route for the prefix (e.g., 12.34.158.0/24) • Attributes such as the AS path (e.g., “2 1”) – Withdrawal • Announcing that the route is no longer available • Numerous BGP attributes – AS path – Next-hop IP address – Local preference – Multiple-Exit Discriminator –… BGP Policy: Applying Policy to Routes • Import policy – Filter unwanted routes from neighbor • E.g. prefix that your customer doesn’t own – Manipulate attributes to influence path selection • E.g., assign local preference to favored routes • Export policy – Filter routes you don’t want to tell your neighbor • E.g., don’t tell a peer a route learned from other peer – Manipulate attributes to control what they see • E.g., make a path look artificially longer than it is BGP Policy: Influencing Decisions Open ended programming. Constrained only by vendor configuration language Receive Apply Policy = filter routes & BGP Updates tweak attributes Apply Import Policies Based on Attribute Values Best Routes Best Route Selection Best Route Table Apply Policy = filter routes & tweak attributes Apply Export Policies Install forwarding Entries for best Routes. IP Forwarding Table Transmit BGP Updates BGP Decision Process: Path Selection on a Router • Routing Information Base – Store all BGP routes for each destination prefix – Withdrawal message: remove the route entry – Advertisement message: update the route entry • Selecting the best route – Consider all BGP routes for the prefix – Apply rules for comparing the routes – Select the one best route • Use this route in the forwarding table • Send this route to neighbors BGP Decision Process: Multiple Steps • Highest local preference – Set by import policies upon receiving advertisement • Shortest AS path – Included in the route advertisement • Lowest origin type – Included in advertisement or reset by import policy • Smallest multiple exit discriminator – Included in the advertisement or reset by import policy • Smallest internal path cost to the next hop – Based on intradomain routing protocol (e.g., OSPF) • Smallest next-hop router id – Final tie-break BGP Decision Process in Action “(2, 1)” “(3, 4, 1)” “(2, 1)” But, what if the path “(3,4,1)” would be better? Manipulating Policy to Move the Traffic • Assign local preference to… – Prefer one neighbor over another for a prefix – Prefer certain AS paths over others • Router configuration languages – Specifying rules for setting local-pref attribute – “if path(3, *, 1), then local-pref=110” – “else, local-pref=100” • Allow policy to over-ride shortest AS path – Indirect way of making one path look better or worse than another – Main way to do BGP traffic engineering today BGP Traffic Engineering BGP Traffic Engineering • Limitations of intradomain traffic engineering – Alleviating congestion on edge links – Making use of new or upgraded edge links – Influencing choice of end-to-end path • Extra flexibility by allowing changes to BGP policies – Direct traffic toward/from certain edge links – Change the set of egress links for a destination 2 4 1 3 BGP Modeling for Traffic Engineering • Predict effects of changes to import policies – Inputs: routing, traffic, and configuration data – Outputs: flow of traffic through the network Topology eBGP routes BGP policy configuration BGP routing model Offered traffic Flow of traffic through the network Steps in the Model • Effects of import policy on the routes – Inputs: BGP routes, and import policies – Output: BGP routes modified by policies • Set of best routes for the network – Inputs: BGP routes modified by policies – Output: set of best routes (max local-pref, min AS path,…) • Best route per router – Inputs: set of BGP routes, and intradomain costs – Outputs: best route (closest egress) per prefix per router • Link-level paths – Inputs: best route per router, and intradomain costs – Outputs: link-level shortest path(s) from entry to egress But, Hard to Select Good Import Policy Changes • Can’t enumerate all choices – Many eBGP sessions – Around 100K prefixes – Highly programmable policies • Risk of creating side-effects – Overhead of BGP routing changes – Unpredictable behavior of others • Vulnerability to changes – New eBGP routes from neighbors – Shifts in the traffic volume per prefix Traffic Engineering Guidelines • Predictability – Ensure the BGP decision process is deterministic – Assume that BGP updates are (relatively) stable • Outbound traffic (import policy, local preference) – Easier to control how traffic leaves the network – Cooperate with neighbor ASes for inbound traffic • Limit overhead introduced by routing changes – Minimize frequency of changes to routing policies – Limit number of prefixes affected by changes • Limit impact on how traffic enters the network – Avoid new routes that might change neighbor’s mind – Select route with same attributes, or at least path length Measurement Data • Analysis of peering links – Links connecting AT&T to other large providers – Relatively small # of high-end routers • BGP routing tables – Log daily output of “show ip bgp” command – Extract BGP routes for each prefix • Cisco Netflow – Collect continuous flow-level measurement – Aggregate the traffic by prefix – Focus on outbound traffic to peers Controlling the Scale • Destination prefixes – More than 90,000 destination prefixes • Don’t want to have per-prefix routing policies – Small fraction of prefixes contribute most of the traffic • Focus on the small number of heavy hitters – Define routing policies for selected prefixes • Routing choices – About 27,000 unique “routing choices” • Help in reducing the scale of the problem – Small fraction of “routing choices” contribute most traffic • Focus on the very small number of “routing choices” – Define routing policies on common attributes Avoid Impacting Downstream Neighbors Will traffic volume change??? Predictable Routing Change • Predictability – Do not change the route sent to downstream neighbor – Focus on prefixes where all “best” routes are identical – Neighbors do not even receive a new BGP advertisement! • Example application – Multiple links to same peer, with one congested link – Assign lower local-pref at that link for some prefixes • Empirical results – 83.5% of prefixes have shortest AS paths that are all identical (same next-hop AS, same AS path, etc.) – These prefixes are responsible for 45% of the traffic – Plenty of scope to move traffic in a predictable fashion Semi-Predictable Routing Change • Semi-predictability – Do not change the length of the AS path sent to neighbor – Neighbors receive new advertisement with same length – (Hopefully) they still make the same routing decision • Example application – Need to move some traffic from one peer to another – Find prefixes with “best” paths via both neighbors – Assign lower local-pref at some links for some prefixes • Empirical results – 10-15% of prefixes have shortest paths with 2 next hops – These prefixes contribute 35-40% of the traffic – Potential to move traffic in a semi-predictable fashion Influence of AS Path Length • AS path length – Plays a significant role in the BGP decision process – All “best” routes must have the same AS path length – 10% of prefixes have choices with different path lengths • An idea: a more flexible approach – Possible to disable consideration of AS path length, and incorporate AS path length in local-pref assignment – E.g., treat paths of length 3 and 4 as equally good • AS prepending by other ASes – Inflating AS path length by adding fake hops – E.g., “701 80 80 80” instead of “701 80” – 18% of routes had some form of AS prepending Inbound Traffic Why Inbound Traffic is Hard to Manage • Other ASes decide how to send to you – Destination-based routing – Other ASes decide which path to take – Based on their own policies 2 4 1 p 3 AS 2 doesn’t know how AS 1 will send traffic toward p AS Prepending • Artificial inflate AS path length – Prepend your own AS in the path – E.g., turn “3 4 5” into “3 3 3 4 5” – Hope to make the path less attractive 1 “3 4 5” 3 “3 3 3 4 5” Multiple Exit Discriminator (MED) • Tell your neighbor what you want – MED attribute to indicate receiver preference – Decision process picks route with smallest MED – Can use MED for “cold potato” routing – But, have to get your neighbor to accept MEDs 1 “3 4 5” with MED=1 3 “3 4 5” with MED=2 Inter-AS Negotiation • Better to cooperate? – Negotiate where to send – Inbound and outbound – Mutual benefits Customer B Provider B multiple peering points • But, how to do it? Early-exit routing – What info to exchange? – How to prioritize the many choices? – How prevent cheating? • Open research territory Provider A Customer A Next Time: Multi-Homed Customers • Two papers – “A Measurement-Based Analysis of Multi-homing” (SIGCOMM’03) – “Optimizing Cost and Performance for Multihoming” (SIGCOMM’04) • Reviews of both papers – Brief summary of the paper – Reasons to accept the paper – Reasons to reject the paper – Suggestions for future research directions