New Interdomain Routing Architectures
Jennifer Rexford
Well, BGP Has Some Problems
• Instability
– Not guaranteed to converge to a unique, stable
solution (e.g., oscillation and bi-stable solutions)
• Slow convergence
– Path exploration can take a very long time
• Non-linearity
– Small changes can have large effects (e.g.,
intradomain changes leading to BGP changes)
• Feature interaction
– Unexpected side effects (e.g., the interaction of
route-flap damping and path exploration)
Well, BGP Has Some More Problems
• Scalability challenges
– Large memory, processing, and message-passing
overhead, dependent on behavior in other ASes
• Security vulnerabilities
– BGP update messages are relatively easy to forge
with bogus prefix, AS path, or other attributes
• Difficult to configure
– Operators must express their (many) goals in an
indirect and contorted manner
• Difficult to troubleshoot
– Hard to infer the root cause of a routing change,
or even the AS(es) responsible
But Wait, There’s More!
• No performance guarantees
– BGP announcements do not include any
information about the performance of the path
• Limited flexibility for path selection
– Only single-path routing on destination prefix
• Limited control over path selection
– Chosen path is a byproduct of the composition of
local routing policies in many different ASes
• Forwarding path may not match AS path
– Routers may deflect packets to other paths (e.g.,
route aggregation, misconfiguration, and malice)
And, Perhaps the Biggest Problem of All…
• Hard to change
– BGP is the glue that holds the disparate parts of
the Internet together
– So, hard to change BGP in a fundamental ways
– … or is it?
Why is BGP Such a Mess?
• Absence of underlying models
– Protocol designed without a design for the
decision process or policy language
– BGP models (e.g., SPP) came much later
• Incremental evolution of the protocol
– Many additions to BGP over the years
– New route attributes and decision-process steps
• Rapid growth of the Internet
– Many ASes, and much more complex topology
• Commercialization of the Internet
– More complex routing policies and competition
– Heightened importance of security concerns
What’s a Networking Researcher to Do?
• Characterize the existing routing system
– Measurement, modeling, simulation
• Improved operational practices
– Best common practices for configuration
• Timer settings, route filtering, features to use, …
– Methods and tools for complicated tasks
• Traffic engineering, config checking, root-cause analysis
• Incremental changes to BGP
– Better implementations of the protocol
– New route attributes for greater flexibility
– Graceful handling of failures, config changes, …
What’s a Networking Researcher to Do?
• Build on top of the existing routing system
– Overlay networks
• Propose new architectures anyway
– Identify and evaluate new and better solutions
– … even if we can’t readily deploy them today
• Explore incremental-deployment approaches
– Meta solutions that enable new architectures
– With the goal of leading to a better architecture
• Create models of the fundamental limits
– Maybe the problem BGP is solving is really hard…
Key Ingredients of Architectural Proposals
• More control at the edge
– Source routing and multi-path routing
• Negotiation between ASes
– Explicit coordination for path selection
• Design for the common case
– Handle common routing policies well (e.g., HLP)
• Servers computing the routes
– Greater visibility and control than any one router
• Multiple simultaneous architectures
– Virtualization to support customized solutions
Source Routing
• The ultimate in flexibility
– Sender determines path for each packet
– At the router level, or at the AS level
• At the cost of…
– Overhead of propagating topology information
– Need for fast path changes after a failure
– Lost control for intermediate routers/ASes
B
A
ADECF
C
ABEF
F
D
E
Source Routing Deployment Challenges
• ASes have little incentive to cooperate
– No business relationship with ASes far away
– Don’t want to relinquish control over routing
– So, source routing is typically disabled
• Policy-compliant source routing
– Limiting what sources routes can be used
• Progress on limited forms of source routing
– Overlay networks
• Source routing on an overlay topology
– Multi-homing route control
• “One hop source routing”
Multipath Routing
• Expose more paths to ASes
– Multiple paths through same next-hop AS
– Giving ASes greater control over path selection
• Extreme case: export all paths
– High protocol overhead
– Lost control for intermediate ASes
• Compromise: export selective paths
– Under control of intermediate ASes
– Based on their routing policies, pricing, etc.
Exposing Extra Paths
BEF*
BCF
B
ABEF*
ADEF
A
C
CF*
CEF
CBEF
BCF
F F*
D
DEF*
DABEF
E
EF *
ECF
• Example: AS A doesn’t like AS E
– Default routes both go through E
– Good to learn alternate route that avoids E
Encapsulation to Use Non-Default Paths
e
d
B
e
d
C
A
F
D
E
• Direct packet along alternate route
– Destination-based forwarding not enough
– Encapsulate the packet to egress point
Inter-AS Negotiation
• Directives
– Tell other AS what to do
– E.g., which link to use
Customer B
• Suggestions
Provider B
multiple
peering
points
– Tell other AS your wishes
– Which link you prefer used
• Cooperation
Early-exit
routing
Provider A
Customer A
–
–
–
–
Negotiate where to send
Across flows and time
Inbound and outbound
Trade small losses for
larger gain (“win win”)
Inter-AS Negotiation
• But, how to do it?
– What info to exchange?
– How to prioritize the
many choices?
– How prevent cheating?
Customer B
Provider B
• Open research territory
multiple
peering
points
Early-exit
routing
Provider A
Customer A
– Some initial work on
exchanging preferences
– E.g., ASes exchange
preferences, and iterates
Revisiting the Division of Labor
• Routers
– Forward packets: yes
– Compute paths: maybe not
• End users or ASes
– Dictate path properties: yes
– Control path selection: maybe not
• Intermediate ASes (e.g., ISPs)
– Forward packets: yes
– Business relationships: yes
– Compute end-to-end paths: maybe not
Removing Routing From Routers
• Should routers forward packets: yes
– Must be done at high speed
– … in line-card hardware on fast routers
– So, needs to be done on the routers
• Should routers compute routes: no
– Hard to do in a distribution fashion
– Difficult to make load-sensitive routing stable
– Lacking complete information for good decisions
– Not flexible enough for end users
– Difficult to extend over time
Routing As a Service (RAS)
• Goal: third parties pick end-to-end paths for
clients to satisfy diverse user objectives
• Forwarding infrastructure
– Basic routing (e.g., default routing)
– Primitives for inserting forwarding-table entries
• Routing Service Provider (RSP)
– Contracts ISPs for service (e.g., virtual links)
– Selects end-to-end routes on behalf of clients
– Competes with other selectors for customers
• End host
– Queries RSP to pick path & install forwarding state
Routing As a Service (RAS)
RSP
ISP
ISP
user
ISP
Routing Control Platform (RCP)
• Goal: Move beyond today’s artifacts, while
remaining compatible with the legacy routers
• Incentive compatibility: phased evolution
– Intelligent route reflector in a single AS
– Learning BGP routes directly from neighbor ASes
– Interdomain routing between RCPs
• Backwards compatibility: BGP to the routers
eBGP
Inter-AS
Protocol
–eBGP
Using
answers to the routers
RCPBGP to “push” RCP
RCP
iBGP
– No need
RCPat all
RCPto change the legacy routers
iBGP
iBGP
– Keep
message
format
and
router
implementation
AS
1
AS
2
AS 3
Physical
peering
How Does This Differ From Overlays
• Overlays: circumventing the underlay
– Host nodes throughout the network
– Logical links between the host nodes
– Active probes to observe the performance
– Direct packets through good intermediate nodes
• Routing services: controlling the underlay
– Servers collect data directly from the routers
– Servers compute forwarding tables for the routers
– Data packets do not go through the servers
– Like an overlay for managing the underlay
Maybe some combination of the two makes sense?
Concurrent Architectures Better than One
• Multiple customized architectures in parallel
– Multiple virtual routers on a single platform
– Resource isolation in CPU, forwarding table,
bandwidth
– Programmability for different protocols and
mechanisms (routing, forwarding, addressing, …)
Cabo: Applications Within an Single ISP
• Customized virtual networks
– Security for online banking
– Fast-convergence for VoIP and gaming
– Specialized handling of suspicious traffic
• Testing and deploying new protocols
– Evaluate on a separate virtual network
– Rather than in a dedicated test lab
– Large scale and early-adopter traffic
• Leasing virtual components to others
– ISPs have unused node and link capacity
– Can allow others to construct services on top
Cabo: Enabling Economic Refactoring
Infrastructure Providers
Service Providers
• Infrastructure providers: Maintain routers,
links, data centers, other physical infrastructure
• Service providers: Offer end-to-end services
(e.g., layer 3 VPNs, SLAs, etc.) to users
Today: ISPs try to play both roles, and cannot offer end-to-end services
Key Ingredients of Incremental Deployability
• Backwards compatibility
– Technically possible to deploy the solution
– E.g., change anything but the message format
• Incentive compatibility
– Offer concrete benefits to early adopters
– Generate incentives for others to follow
– Do not rely on complete participation
• QoS, multicast, secure routing, IPv6, …
• Need creativity on incremental deployment
Lessons on Incremental Deployment
• Adding on top: tunneling in the data plane
– Circumvent destination-based forwarded
– Traverse routers
• Adding on top: servers in the control plan
– Tricking the routers into doing the server’s bidding
– Implementing new functionality on the servers
• Sneaking in on the side: virtualization
– Running additional virtual networks in parallel
– Offering new functionality for special applications
– … while continuing to support “legacy” Internet
Discussion
• Tussles between stake-holders
– Users and ISPs
• Division of function
– Data, control, and management planes
– End host, edge routers, and core routers
• How much better could BGP be?
– While still being an interdomain protocol with
control distributed across Autonomous Systems?
– With an even cleaner slate than that?
– Where should the economics be???