MIRO:
Multi-path
Interdomain
ROuting
Slide 1
Wen Xu
July 30, 2009
Internet Routing : current status
Slide 2
AS 1
The
Internet
AS 701
AS
7018
AS 29
AS 88
AS = Autonomous System
31,311 ASes in 2009 Internet
Interdomain Routing
Slide 3

Interdomain routing is important



31,311 ASes in April 2009
Most traffic traverses multiple ASes
Interdomain routing is challenging




A federation of multiple independent ASes
Economic factors drive routing policies
Not necessarily using shortest path
People are not willing to reveal internals
Our Contribution: MIRO
Slide 4

Motivation


MIRO: a new Interdomain routing protocol

Offers substantial flexibility
Avoid state explosion in disseminating info

Give ASes control over the traffic in their networks


Current protocol lacks flexibility and control
Backward compatible and incrementally deployable
Slide 5
Outline of The Talk
Motivation and Related Work
 Design and Implementation
 Measurement Based Evaluation
 Guarantee the Convergence

Current Interdomain Routing Protocol:
BGP (Border Gateway Protocol)
Slide 6

Path-Vector Protocol:

Only a single route is chosen
Each node calculates its AS paths to each reachable destination
and advertises that to adjacent neighbors
ABEF*
ADEF
BEF*
BCF
B
CF*
CEF
CBEF
C
F*
A
F
D
DEF*
DABEF
E
EF *
ECF
Slide 7
When BGP is not enough:
Avoiding an AS
ABEF*
ADEF
A




BEF*
BCF
B
C
CF*
CEF
CBEF
F*
F
D
E
DEF*
EF *
DABEF ECF
AS A does not trust AS E to carry its traffic
AS A wants the route BCF in AS B instead of BEF
Even if B is willing to satisfy A’s request, B can not do that
because all of B’s neighbors would also switch to BCF
BCF is available, it is just not advertised or used
When BGP is not enough: Incoming
Traffic Control
Slide 8
ABEF*
ADEF
BEF*
BCF
B
C
F*
A
F
D
DEF*
DABEF



CF*
CEF
CBEF
E
EF *
ECF
Too much traffic uses EF, CF barely used
Receiving AS has little say over which path to use
People do want that in today’s Internet


12,468 stubs out of 31,311 ASes multi-homed
More than 4,900 ASes use AS prepending
Slide 9

Flexibility: single path routing limits flexibility



Different people have different needs
They care about properties of the paths
Efficiency: avoid disclosing states unnecessarily


The High Level Problem
Most ASes are satisfied with BGP routes
Control: give intermediate ASes more control

Get more alternative paths

Current business relationship limited to adjacent ASes
Slide 10
Option 1: BGP

Flexibility


Efficiency


No, single path limits flexibility
Yes
Control



Yes, but not perfect
ASes use local policies to control its traffic
But very hard to control non-adjacent ASes
Option 2: Source Routing
Slide 11

Flexibility


Efficiency


Yes, ultimate flexibility
No, all internal states exposed
Control

No, intermediate ASes lose control
B
ABCF
A
ADECF
C
ABEF
F
D
E
Slide 12



At Another Level: Overlays
Virtual networks above physical networks
Additional cost in setting up overlay boxes
They have no control over physical paths
B
C
A
Overlay nodes
F
D
E
Our Proposal: MIRO
Slide 13

Flexibility


Efficiency



Expose the alternative routes already there
Use BGP for default routes
Explore alternatives only when necessary
Control



Intermediate ASes still have routing policies
They can negotiate with an arbitrary AS
They can control what routes they expose
Slide 14
Outline of The Talk
Motivation and Related Work
 Design and Implementation
 Measurement Based Evaluation
 Guarantee the Convergence

Slide 15

AS-level routing




MIRO Design:
AS-level Path-Vector Protocol
How do you divide the boundaries
AS as a natural entity of trust and policy specifications
Each AS decides how traffic should flow
Path-vector protocol



Path-vector protocol fits “federated” world
Build AS paths along the way
Downstream AS should be able to control who may
send traffic through its network
MIRO Design: Route Negotiation
Slide 16
Any route to F
avoiding
BCF is OK
E?
ABEF*
ADEF
A
BEF*
BCF
B
BCF
F F*
D
DEF*
DABEF

E
EF *
ECF
Pull-based route retrieval


C
CF*
CEF
CBEF
Solicit routes only when necessary
Bilateral negotiations

Business relationships usually bilateral anyway
MIRO Design: Negotiation with Anyone
Slide 17
ABCF*
ABEF*
ADEF
BEF
BEF*
BCF
BCF*
B
C
A
F
D
DEF*
DABEF
DABCF


CF*
CEF
CBEF
E
EF *
ECF
F*
Negotiate with adjacent/non-adjacent ASes
Negotiate with ASes in either direction


Sender-side negotiation
Receiver-side negotiation
MIRO Design: Routing Tunnels
New IP
header
Slide 18
New IP
header
Old
packet
Old
packet
B
C
A
F
D




E
Destination based routing is no longer sufficient
Tunnels transmit traffic between two endpoints
Normally implemented by IP-in-IP encapsulation
Assign unique tunnel id upon successful negotiation
Slide 19
MIRO Implementation:
Revisiting Intra-AS Structure
AS F
AS C
AS E
R2(12.34.56.2)
R3(12.34.56.3)
AS B
(12.34.56.0/24)
Use IBGP to
distribute routes
R1(12.34.56.1)
Ingress routers
AS A
Egress routers
Slide 20



What IP Do We Use in Encapsulation?
IP of exit links
IP of egress
routers
One IP for all
tunnels

AS C
AS E
R2(12.34.56.2)
R3(12.34.56.3)
AS B
Rewrite to IP of
egress router at
ingress routers
(12.34.56.0/24)
R1(12.34.56.1)
AS A
Egress routers
Ingress routers
Slide 21
When to Destroy Tunnels

Terminating the business contract


BGP route is withdrawn or changed



Either party may tear it down actively
The route from upstream to downstream
The route selected at the responding AS
Failure of the ASes
Slide 22
Outline of The Talk
Motivation and Related Work
 Design and Implementation
 Measurement Based Evaluation
 Guarantee the Convergence

Slide 23
Evaluation Outline

Flexibility
MIRO does provide more flexibility

Efficiency
Not exploring too much state in negotiations

Control
ASes can choose strict or flexible policy, strict
policy already gives much benefit
Evaluation Methodologies
Slide 24




How do we evaluate a new routing
protocol without deploying it
Something resembling the Internet
The behavior of routing protocols
governed by local policies
Where do we find local policies
Local Policies in Today’s Internet
Slide 25
Export rules:
Advertise all paths to customers or siblings
Advertise only customer routes to peers or providers
Preference rules:
Prefer customer routes over peer or provider routes
sibling
sibling
AT&T
AT&T 2
provider
$$
customer
Princeton
UUNET
peer
peer
Slide 26
Evaluation Methodologies

Local policies not available


Business relationships not available


Use business relationships to approximate
Use relationship inferring algorithm which
takes real BGP dumps as input
They are not perfect, but they are the
closest we can get toward Internet
Slide 27



The Data
2000/2003/2005 RouteView BGP dump
Gao’s AS relationship inference algorithm
Each AS is one node in the topology
Date
AS #
Link #
P/C links
10/1/2000
8829
17793
Peering
links
16531
1031
Sibling
Links
231
10/8/2003
16130
34231
30649
3062
520
10/8/2005
20930
44998
40558
3753
687
Node Percentage
Slide 28
Node Degree Distribution
1
0.8
0.6
0.4
0.2
0
2000
2003
1/3 of nodes
with only 1 edge
40% of nodes
with 2 edges
1
10
2005
10/48/80
nodes with
degree > 200
100
1000
Number of Edges (Log Scale)
Slide 29

Evaluation Methodologies
Evaluated applications



Evaluated policies





Avoid an AS for security or performance
Controlling incoming traffic
Strict policy (/s)
Respect export policy (/e)
Most flexible policy (/a)
Only build one tunnel
Negotiate with adjacent neighbors and the ASes
on the default BGP path only (avoiding AS)
Slide 30




Avoiding AS: success rate
BGP only gets around 30%
Strict policy can already get around 65%
More flexible policy can get around 76%
Even source routing can only push 10% more
Date
BGP
MIRO/s
MIRO/e
MIRO/a
Source
Routing
2000 27.8%
65.4%
72.9% 75.3% 89.5%
2003 31.2%
67.0%
74.6% 76.6% 90.4%
2005 29.5%
67.8%
73.7% 76.0% 91.1%
Avoiding AS: state explosions
Slide 31

With more flexible policies



Number of ASes we negotiated with decrease
Number of paths increase
Results on 2000 and 2003 data


Number of ASes roughly the same
Number of paths increase with time
Policy
Strict
2005 Export
Flexible
Success rate
AS #/tuple
Path #/tuple
67.8%
73.7%
76.0%
2.80
2.53
2.38
36.6
58.9
139.0
Percentage of total gain
Slide 32
Avoiding AS: Incremental
of total gain
Deployment 99.9%
if 25% of nodes
100
80
53% of total gain
if 0.2% of nodes
(40 nodes)
adopted MIRO
adopted MIRO
60
40
44% of total gain
if 0.2% of nodes
(40 nodes)
adopted MIRO
20
0
0.01
0.1
1
82% of total gain
if 25% of nodes
adopted MIRO
10
Percentage of ASes Adopted MIRO (log scale)
2005/s
2005/e
2005/a
100
Slide 33

Assume that a multi-homed stub AS wants to
balance incoming traffic


Another Real Application:
Incoming Traffic Control
How much traffic can it move by just negotiating
with one upstream AS
10,383 stubs out of 20,930 ASes (2005 data)



90% can move more than 10% of the traffic
50% can move more than 40% (flexible policy)
50% can move more than 25% (strict policy)
Slide 34


Another Real Application:
Incoming Traffic Control (cont)
In the MIRO paper, we assume stub AS
blindly obey selection made by power node,
here, stub AS re-run BGP selection process
10,383 stubs out of 20,930 ASes (2005 data)



77% can move more than 10% of the traffic
28% can move more than 40% (flexible policy)
50% can move more than 18% (strict policy)
Slide 35
Outline of The Talk
Motivation and Related Work
 Design and Implementation
 Measurement Based Evaluation
 Guarantee the Convergence

The Convergence Problem
Slide 36




Oscillation is bad, it leads to lost packets
BGP does not guarantee convergence
MIRO introduces more flexibility, so it
opens up new problems for convergence
Certain policies guarantees convergence


One tunnel + do not advertise to others
Advertise to stub ASes only
Slide 37

Convergence Problem:
BGP May Not Converge
A counter-example that does not converge
AX
ABX
ABX
AX
A
A prefers ABX over AX
B prefers BCX over BX
C prefers CAX over CX
X
B
BX
BCX
BCX
BX
C
CAX
CX
CAX
CX
Slide 38

Topology + Policy -> Convergence
Many scenarios do not happen in the real world


A few prevalent relationships
Provider/Customer, Peer/Peer, Sibling/Sibling
Topology


Policy:



Provider/customer relationships implies a partial order
Export policy -> some paths never occur
Preference policy -> some paths always favored
Conclusion:
BGP converges without global coordination
Slide 39



The Convergence of MIRO
Counter-examples show that MIRO does not
converge without restrictions
Proved that MIRO would converge if certain
policy guidelines are obeyed
Guideline A is the original Guideline that
guarantees the convergence of BGP:
customer routes are always preferred over
any provider routes or peer routes
Slide 40
Guideline B: Tunnels as a Higher Level

Tunnels constructed using only BGP and
not advertised back to BGP
BCEF
BEF*
BCF
ABCF
ABEF*
ADEF
B
CF*
CEF
CBEF
C
A
F
D
E
Slide 41

Guideline C: Advertise Tunnels
to Stub ASes Only
Advertise tunnels as BGP paths to stubs only
BCEF
BEF*
BCF
ABCEF
ABEF*
ADEF
B
CF*
CEF
CBEF
C
A
Stub node
F
D
E
Strict Policy Does Not
Slide 42 Guarantee Convergence
Strict Policy with Restriction
Slide 43 Guarantees Convergence


Guideline D:
Require that each AS enforces a strict
order such that
first_downstream(r) ≺ a(r.prefix)
Guideline E:
Only allow a tunnel if the route from
current node to first_downstream(r)
does not contain a tunnel
Mixing and Matching the Guidelines
Slide 44



Guideline A can be replaced with the
other Guidelines in the original paper
Each AS can choose between A+C and
A+D, or they can choose between A+C
and A+E
Guideline B-tunnel can be built on top of
Guideline C-tunnel, Guideline D-tunnel,
or Guideline E-tunnel
These Guidelines Are Practical
Slide 45

Most end-to-end paths contain only 1 tunnel





Many ASes are stub ASes
Average AS path is 4
Negotiations allowed between non-adjacent ASes
Most stub ASes can be satisfied by Guideline C
Economic incentives motivate for strict policy,
and the restrictions are easy to implement
Slide 46






Summary
MIRO: Multi-path Interdomain ROuting
Use BGP for default routes, negotiate only
when necessary
Give more power to individual ASes
Incrementally deployable
Much flexibility achieved using one tunnel
Can adopt flexible policies
Slide 47
Thanks