CSCI-1680
Network Layer:
Inter-domain Routing – Policy and
Security
Theophilus Benson
Based partly on lecture notes by Jennifer Rexford, Rob Sherwood, David Mazières, Phil Levis, John Jannotti
Today
• BGP Recap
• BGP + IGP
– iBGP, Scaling iBGP
• Using BGP to take down the internet
• BGP Security
– Hijacking prefixes  making money
– Solution: S-BGP
• BPG Issues
– ISP issues versus end-user issues
– Solution: Overlays, CDNs
Recall BGP
Tier 1 ISP
Tier 1 ISP
Default free,
Has information on
every prefix
Default: provider
Tier 2
Regional
$$
$$
$$
Tier 2
$$
Tier 2
$$
$$
Tier 3
(local)
Tier 3
(local)
Recall BGP
Tier 1 ISP
Tier 1 ISP
Default free,
Has information on
every prefix
$$
$$
“Best Route” is not The shortest route
Default: provider
Tier 2
Regional
$$
$$
Tier 2
Tier 2
$$
$$
Tier 3
(local)
Tier 3
(local)
Recall BGP: Realistic Example
Tier 1 ISP
Tier 2
Tier 2
$10
Tier 2
Regional
$$
$$
Tier 2
$20
$$
Tier 2
$$
$$
Tier 3
(local)
Tier 3
(local)
Zooming
ISPD
ISPC
D is provider for B
Peering
ISPB
Regional
ISPA
10.10/16
10.10/16 ISPA
ISPZ
Zooming-in!
Who should ISPB send routes to?
Who should ISPB use to get to Alice?
ISPD
ISPC
Everyone? No one? Friends? Enemies?
D is provider for B
Peering
ISPB
Regional
Export Policies:
(Who to send routes to?)
10.10/16 ISPA
•
Provider  Customer
– All routes so as to
provide transit
service
ISPA
10.10/16
•
Customer  Provider
– Only customer routes
•
Peer  Peer
– Only customer routes
Import Policies:
(Who to use for transit?)
ISPZ
.10/16 ISPB
Zooming-in!
ISPA
Who should ISPB send routes to?
Who should ISPB use to get to Alice?
ISPD
ISPC
D is provider for B
10.10/16 ISPB
ISPA
Everyone? No one? Friends? Enemies?
Peering
ISPB
Regional
Export Policies:
(Who to send routes to?)
•
Provider  Customer
– All routes so as to
provide transit
service
ISPA
10.10/16
•
Customer  Provider
– Only customer routes
•
Peer  Peer
– Only customer routes
Import Policies:
(Who to use for transit?)
ISPZ
Zooming-in!
10.10/16 ISPZ
ISPD
Who should ISPB send routes to?
Who should ISPB use to get to Alice?
ISPC
Everyone? No one? Friends? Enemies?
D is provider for B
Peering
ISPB
Regional
Export Policies:
(Who to send routes to?)
•
Provider  Customer
– All routes so as to
provide transit
service
ISPA
10.10/16
•
customer > peer >
provider
•
Customer route:
charge $$ 
•
Peer route: free
•
Provider route: pay $$

Customer  Provider
– Only customer routes
•
Import Policies:
(Who to use for transit?)
Peer  Peer
– Only customer routes
ISPZ
Zooming-in!
Who should ISPB send routes to?
Who should ISPB use to get to Alice?
ISPD
10.10/16 ISPD
ISPZ
D is provider for B
ISPC
10.10/16 ISPC
ISPZ
Everyone? No one? Friends? Enemies?
Peering
ISPB
Regional
Export Policies:
(Who to send routes to?)
•
Provider  Customer
– All routes so as to
provide transit
service
ISPA
10.10/16
•
customer > peer >
provider
•
Customer route:
charge $$ 
•
Peer route: free
•
Provider route: pay $$

Customer  Provider
– Only customer routes
•
Import Policies:
(Who to use for transit?)
Peer  Peer
– Only customer routes
Zooming-in!
ISPZ
Who should ISPB send routes to?
Who should ISPB use to get to Alice?
ISPD
ISPC
Everyone? No one? Friends? Enemies?
D is provider for B
Peering
Export Policies:
(Who to send routes to?)
ISPB
Regional
10.10/16
ISPB
ISPA
10.10/16
•
ISPC
Provider  Customer
– All routes so as to
provide transit
service
ISPZ
•
customer > peer >
provider
•
Customer route:
charge $$ 
•
Peer route: free
•
Provider route: pay $$

Customer  Provider
– Only customer routes
•
Import Policies:
(Who to use for transit?)
Peer  Peer
– Only customer routes
Z is provider for D
Valley Free Routing
ISPZ
Z is provider for C
ISPD
ISPC
D is provider for B
C is provider for B
ISPB
Regional
B is provider for A
ISPA
10.10/16
DBCZ
-1
+1
+1
D Z
+1
ABCZ
+1
+1
+1
Z is provider for D
How to get Peering
ISPZ
Z is provider for C
ISPD
ISPC
DZC
+1 -1
D is provider for B
C is provider for B
ISPB
Regional
All users in network D want to
go to something in network C:
• Network C could be Google
• Network C could be Netflix
ISP D can try and Peer with C
B is provider for A
ISPA
10.10/16
Z is provider for D
How to get Peering
ISPZ
Z is provider for C
ISPD
ISPC
Valley free:
DZC
+1 -1
D is provider for B
C is provider for B
ISPB
Regional
B is provider for A
ISPA
10.10/16
All users in network D want to
go to something in network C:
• Network C could be Google
• Network C could be Netflix
ISP D can try and Peer with C
• Path:
DC
• Valley free: 0
Why is this good for D? or C?
Neither has to pay Z anymore
BGP State
• BGP speaker conceptually maintains 3 sets of state
• Adj-RIB-In
– “Adjacent Routing Information Base, Incoming”
– Unprocessed routes learned from other BGP speakers
• Loc-RIB
– Contains routes from Adj-RIB-In selected by policy
– First hop of route must be reachable by IGP or static route
• Adj-RIB-Out
– Subset of Loc-RIB to be advertised to peer speakers
Today
• BGP Recap
• BGP + IGP
– iBGP, Scaling iBGP
• Using BGP to take down the internet
• BGP Security
– Hijacking prefixes  making money
– Solution: S-BGP
• BPG Issues
– ISP issues versus end-user issues
– Solution: Overlays, CDNs
How does router-X learns to the route
to 10.20/8 or 10.20/16??
If don’t have routes
Send to M
ISPD
•
M
Stub Ass (e.g. ISP A, D)
– Border router clear choice for
default route
– Inject into IGP: “any unknown route
to border router”
Who to send
unknown to?
Y or W?
•
Y
– E.g., Provider injects routes to
customer prefix
Z
X
•
W
ISP-B
ISPA
Inject specific prefixes in IGP
For Large networks
– Too many prefixes for IGP
– Run internal version of BGP, iBGP
– All routers learn mappings: Prefix ->
Border Router
– Use IGP to learn how to get to
Border Router
Two types of BGP sessions
128.112.0.0/16
Next Hop = 192.0.2.1
128.112.0.0/16
iBGP
AS23
eBGP
192.0.2.1
Forwarding Table
destination
next hop
192.0.2.0/30
AT&T
Sprint
AS23
10.10.10.10
+
BGP (iBGP)
destination
next hop
128.112.0.0/16
192.0.2.1
Forwarding Table
destination
next hop
128.112.0.0/16
192.0.2.0/30
10.10.10.10
10.10.10.10
Two types of BGP sessions
iBGP
eBGP
AT&T
Sprint
• eBGP session is a BGP session between two
routers in different ASes
• iBGP session is a BGP session between
internal routers of an AS.
Scaling iBGP
• Every Router runs iBGP
• All-to-All iBGP peering
• Doesn’t scale
• N*(N-1) connections
Scaling iBGP
Route reflectors
• Every Router runs iBGP
• Selective peering
• Scales
• N*K connections
Today
• BGP Recap
• BGP + IGP
– iBGP, Scaling iBGP
• Using BGP to take down the internet
• BGP Security
– Hijacking prefixes  making money
– Solution: S-BGP
• BPG Issues
– ISP issues versus end-user issues
– Solution: Overlays, CDNs
“Shutting off” the Internet
• Starting from Jan 27th, 2011, Egypt was
disconnected from the Internet
– 2769/2903 networks withdrawn from BGP (95%)!
Source: RIPEStat - http://stat.ripe.net/egypt/
Egypt Incident
Source: BGPMon (http://bgpmon.net/blog/?p=480)
Today
• BGP Recap
• BGP + IGP
– iBGP, Scaling iBGP
• Using BGP to take down the internet
• BGP Security
– Hijacking prefixes  making money
– Solution: S-BGP
• BPG Issues
– ISP issues versus end-user issues
– Solution: Overlays, CDNs
BGP Security Goals
• Confidential message exchange between
neighbors
• Validity of routing information
– Origin, Path, Policy
• Correspondence to the data path
Origin: IP Address Ownership and
Hijacking
• IP address block assignment
– Regional Internet Registries (ARIN, RIPE, APNIC)
• Who can advertise a prefix with BGP?
– By the AS who owns the prefix
– … or, by its upstream provider(s) in its behalf
• However, what’s to stop someone else?
– Prefix hijacking: another AS originates the prefix
– BGP does not verify that the AS is authorized
– Registries of prefix ownership are inaccurate
28
Prefix Hijacking: full or partial control
4
3
5
2
7
1
6
12.34.0.0/16
12.34.0.0/16
• Consequences for the affected ASes
29
– Blackhole: data traffic is discarded
– Snooping: data traffic is inspected, and then redirected
– Impersonation: data traffic is sent to bogus destinations
Hijacking is Hard to Debug
• Real origin AS doesn’t see the problem
– Picks its own route
– Might not even learn the bogus route
• May not cause loss of connectivity
– E.g., if the bogus AS snoops and redirects
– … may only cause performance degradation
• Or, loss of connectivity is isolated
– E.g., only for sources in parts of the Internet
• Diagnosing prefix hijacking
– Analyzing updates from many vantage points
– Launching traceroute from many vantage points
30
Sub-Prefix Hijacking
Full control over sub-prefix
4
3
5
2
6
7
1
12.34.158.0/24
• Originating a more-specific prefix
31
12.34.0.0/16
– Every AS picks the bogus route for that prefix
– Traffic follows the longest matching prefix
How to Hijack a Prefix
• The hijacking AS has
– Router with eBGP session(s)
– Configured to originate the prefix
• Getting access to the router
– Network operator makes configuration mistake
– Disgruntled operator launches an attack
– Outsider breaks in to the router and reconfigures
• Getting other ASes to believe bogus route
– Neighbor ASes not filtering the routes
– … e.g., by allowing only expected prefixes
– But, specifying filters on peering links is hard
32
Pakistan Youtube incident
• Youtube’s has prefix 208.65.152.0/22
• Pakistan’s government order Youtube blocked
• Pakistan Telecom (AS 17557) announces
208.65.153.0/24 in the wrong direction (outwards!)
• Longest prefix match caused worldwide outage
• http://www.youtube.com/watch?v=IzLPKuAOe50
Cool Bit Coin attack
using Prefix-Hijacking
Bit Coin Incident
• Bit Coin Primer
– You donate resources on your computer to ‘mine’ (create)
bit coins
– Your computer connects to a server
– Servers tells it how to mine
– Server rewards you for mining
• Transaction fees for using coins
• subsidies for new coins
• Hacker steal Bit Coins
– Hacker Hijacks a prefix
– Pretends to be the bit coin-server
– Collects bit coins you mine
• Doesn’t give miners any rewards
4
3
5
2
7
1
12.34.158.0/24
hacker
6
12.34.0.0/16
Bit-coin
Miners
Legitimate
Bit-Coin
Server
Avoiding Spam Detection with Prefix
Hijacking
• People create a whitelist of acceptable
addresses for Mail servers
– Only accept mail from address in that whitelist
• Spammers steal unused IP space to hide
–
–
–
–
–
Announce very short prefixes (e.g., /8). Why?
For a short amount of time
Hijack route == announce a route you don’t own
Send lots of spam!!
Stop Hijack == Withdraw Route
• Interesting talk:
https://www.usenix.org/conference/lisa07/homeless-vikings-bgp-prefix-hijacking-andspam-wars
Attacks on BGP Paths
• Remove an AS from the path
– E.g., 701 3715 88 -> 701 88
• Why?
–
–
–
–
Attract sources that would normally avoid AS 3715
Make path through you look more attractive
Make AS 88 look like it is closer to the core
Can fool loop detection!
• May be hard to tell whether this is a lie
– 88 could indeed connect directly to 701!
Attacks on BGP Paths
• Adding ASes to the path
– E.g., 701 88 -> 701 3715 88
• Why?
– Trigger loop detection in AS 3715
• This would block unwanted traffic from AS 3715!
– Make your AS look more connected
• Who can tell this is a lie?
– AS 3715 could, if it could see the route
– AS 88 could, but would it really care?
Attacks on BGP Paths
• Adding ASes at the end of the path
– E.g., 701 88 into 701 88 3
• Why?
– Evade detection for a bogus route (if added AS is
legitimate owner of a prefix)
• Hard to tell that the path is bogus!
701
18.0.0.0/8 88
3
18.0.0.0/8
Data Plane Attacks
(Forwarding Attacks)
• Routers/ASes can advertise one route, but not
necessarily follow it!
• May drop packets
– Or a fraction of packets
– What if you just slow down some traffic?
• Can send packets in a different direction
– Impersonation attack
– Snooping attack
• How to detect?
– Congestion or an attack?
– Can let ping/traceroute packets go through
– End-to-end checks?
• Harder to pull off, as you need control of a
router
Today
• BGP Recap
• BGP + IGP
– iBGP, Scaling iBGP
• Using BGP to take down the internet
• BGP Security
– Hijacking prefixes  making money
– Solution: S-BGP
• BPG Issues
– ISP issues versus end-user issues
– Solution: Overlays, CDNs
Proposed Solution: S-BGP
• Based on a public key infrastructure
• Address attestations
– Claims the right to originate a prefix
– Signed and distributed out of band
– Checked through delegation chain from ICANN
• Route attestations
– Attribute in BGP update message
– Signed by each AS as route along path
• S-BGP can avoid
– Prefix hijacking
– Addition, removal, or reordering of intermediate ASes
Today
• BGP Recap
• BGP + IGP
– iBGP, Scaling iBGP
• Using BGP to take down the internet
• BGP Security
– Hijacking prefixes  making money
– Solution: S-BGP
• BPG Issues
– ISP issues versus end-user issues
– Solution: Overlays, CDNs
BGP Issues
BGP issues from ISP’s
perspective
BGP issues user’s
perspective
• Prefix Hijacking
• Network Outages
– Internet outage
• Route table overflow
– Internet outage
• Convergence issues
– Temporary outage
• Large latency
• Low Bandwidth
Alice  Eve: 50 milliseconds
Alice  Bob: 10 milliseconds
Bob  Eve: 20 milliseconds
Eve
Bob
Alice
Alice  Eve: 50 milliseconds
Alice  Bob: 10 milliseconds
Bob  Eve: 20 milliseconds
Why not send traffic through Bob?
Eve
Bob
Alice
Alice  Eve: 50 milliseconds
Alice  Bob: 10 milliseconds
Bob  Eve: 20 milliseconds
Why not send traffic through Bob?
• Internet uses destination based routing ..
For Alice  Eve to go through Bob
• Packets must use Bob as the destination
Eve
Bob
Alice
IP tunnels: IP-in-IP Encapsulation
20.0.0.1
Alice-> Eve
Alice-> Eve
Bob->Eve
Eve
Bob->Eve
Bob
Alice-> Eve
Alice
Alive->Bob
• Alice/Bob/Eve runs special software that
perform
– IP Encapsulate/decapsulation
Content Delivery Networks
20.0.0.1
Alice
Content Delivery Networks
20.0.0.1
Alice
Content Delivery Networks
20.0.0.1
Alice
BGP Recap
• Key protocol that holds Internet routing together
– Path Vector Protocol between ASs
• Valley Free routing
– Import policies: customer > peering > provider
– Export policies:
• BGP+IGP: iBGP v eBGP
• BGP issues
– Scalability  CIDR, Route Reflectors
– Convergence  manual intervention
– Traffic-Engineering  MEDs,Prefix-pretending
• Hot-Potato Routing  MEDs
– Security  Manual detection, S-BGP
– Poor network paths  CDNs, Overlay Networks
Assignment 2
UDP connections
node
node
node
Localhost:7000
Localhost:7001 10.116.89.157 10.10.168.73
Localhost:7002
Localhost:7001 14.230.5.36 10.42.3.125
Localhost:7001
Localhost:7000 10.10.168.73 10.116.89.157
Localhost:7002 10.42.3.125 14.230.5.36
Protocol == 200
RIP packet
Hdr
len
vers
TOS
Total Length
Fragment Offset
Identification
TTL
Protocol
Hdr Checksum
Command = 1  RIP request
Command = 2 RIP response
Source IP Address
Destination IP Address
Options
Command = 0  send test data
Padding
Command
Number of entries
cost
Payload
address
Protocol == 200
RIP packet
Hdr
len
vers
TOS
Total Length
Fragment Offset
Identification
TTL
Protocol
Hdr Checksum
Source IP Address
Command = 0  send test data
Destination IP Address
Options
Padding
Number of entries
Command
cost
address
Command = 1  RIP request
Command = 2 RIP response
Hdr
len
vers
TOS
Fragment Offset
Identification
TTL
Total Length
Protocol
Hdr Checksum
Source IP Address
Destination IP Address
Options
Padding
node
node
node
Localhost:7000
Localhost:7001 10.116.89.157 10.10.168.73
Localhost:7002
Localhost:7001 14.230.5.36 10.42.3.125
Localhost:7001
Localhost:7000 10.10.168.73 10.116.89.157
Localhost:7002 10.42.3.125 14.230.5.36
Assignment 2
UDP connections
10.10.168.73
10.116.89.157
10.42.3.125
14.230.5.36
node
node
node
Localhost:7000
Localhost:7001 10.116.89.157 10.10.168.73
Localhost:7002
Localhost:7001 14.230.5.36 10.42.3.125
Localhost:7001
Localhost:7000 10.10.168.73 10.116.89.157
Localhost:7002 10.42.3.125 14.230.5.36
Assignment 2
UDP connections
node
node
node
Localhost:7000
Localhost:7001 10.116.89.157 10.10.168.73
Localhost:7002
Localhost:7001 14.230.5.36 10.42.3.125
Localhost:7001
Localhost:7000 10.10.168.73 10.116.89.157
Localhost:7002 10.42.3.125 14.230.5.36
Forwarding: Send Test data
node
node
node
Localhost:7002
Localhost:7001 14.230.5.36 10.42.3.125
Node> send 14.230.5.36 hi there buddy
what’s up
Node>
Node> hi there buddy what’s
up
Node>
Protocol == 0
Test data
Hdr
len
vers
TOS
Total Length
Fragment Offset
Identification
TTL
Protocol
Hdr Checksum
Source IP Address
Destination IP Address
Options
H
0
E
Padding
L
L
Hdr
len
vers
TOS
Fragment Offset
Identification
TTL
Total Length
Protocol
Hdr Checksum
Source IP Address
Destination IP Address
Options
Padding
node
Localhost:7000
Localhost:7001 10.116.89.157 10.10.168.73
Node> send 14.230.5.36 hi there buddy
what’s up
Node>
node
node
Localhost:7002
Localhost:7001 14.230.5.36 10.42.3.125
Node> hi there buddy what’s
up
Node>