Sofía Silva Berenguer
sofia @ lacnic.net
Internet Exchange Points
Workshop
Paramaribo - Surinam
AGENDA
•
•
•
•
•
•
How the Internet Works
Intro to BGP
IPv4 Exhaustion and IPv6 Deployment
Internet Exchange Points
How to request Internet Resources
Advanced topics
–
–
–
–
–
Route Hijacking
Leaks
Attacks against the path
Well known incidents
Securing the Routing System
HOW THE INTERNET WORKS
Interconnection of Networks
http://prezi.com/agzxt0exlfpk/network-interconnection/
Internet Routing
ASN 6057
announces
200.40.0.0/16
ASN 8158
receives
200.40.0.0/16
The prefix 200.40.0.0/16 is
propagated with BGP to the
Internet
Atributos:
200.40.0.0/16 AS_PATH ASN1 ASN3 ASN6057
Transit and Peering
• Transit
– Traffic and prefixes originating from one AS are
carried across an intermediate AS to reach their
destination AS
– Usually for a fee
• Peering
– Private interconnect between two ASNs
– Usually for no fee
Transit and Peering
ASN 65538
ASN 64511
Transit
ASN 65536
Peering
ASN 65537
Peering in an Internet Exchange Point
(IXP)
• Internet Exchange Point
– Common interconnect location where several
ASNs exchange routing information and traffic
ASN 65536
ASN 65537
ASN 65538
ASN 65539
IP address, where they come from?
Standards
Central Registry
Distribution
Regional Internet Registries
(RIRs) distribute IPv4, IPv6
and Autonomous System
Numbers
*
*
Distribution
Sometimes the distribution is done through
National Internet Registries (NIRs)
Allocations and
Assignments
End
user
Regional Internet Registries
INTRO TO BGP
Border Gateway Protocol
• A Routing Protocol used to exchange routing
information between different networks
• Exterior gateway protocol
• Described in RFC4271
– RFC4276 gives an implementation report on BGP
– RFC4277 describes operational experiences using
BGP
• Works on TCP port 179
More about BGP
• Learns multiple paths via internal and external
BGP speakers – Initial exchange of entire table
• Incremental Updates
– Picks THE bestpath and installs it in the IP forwarding
table – Policies applied by influencing the bestpath
selection
•
•
•
•
Keepalive messages exchanged
Many options for policy enforcement
Classless Inter Domain Routing (CIDR)
Widely used for Internet backbone
Neighbors
• BGP speakers
– Internal (iBGP) if they are in the same ASN
– External (eBGP) if they are in different ASN
eBGP
iBGP
ASN 65538
ASN 65536
Where to use BGP: Stub Network
ASN 65536,
Transit Provider
• Only one exit for
customer
• Not really need
to add BGP
ASN 65538, Customer
Multihomed Network
Transit Providers
ASN 65538
ASN 65539
• Different situations possible
•
•
•
ASN 65536
•
ASN 65537
•
Peering in IXP
Multiple links to same ISP
Secondary for only backup
Load share between
primary and secondary
Selectively use different
ISPs
Peering at IXP
INTERNET EXCHANGE POINTS
Why peer?
• Consider a region with one ISP
– It provides internet connectivity to it’s customers
– It has one or two international connections
• Internet grows, another ISP sets up in competition
– They provide internet connectivity to their
customers
– They have one or two international connections
• How does traffic from customer of one ISP get to
customer of the other ISP?
– Via the international connections
Why peer? (Cont.)
Internet
ASN 65536
ASN 65538
Why peer? (Cont.)
• Yes, international connections…
– If satellite, RTT is around 550ms per hop
– So local traffic takes over 1s round trip
• International bandwidth…
– Is much more expensive than domestic bandwidth
– Becomes congested with local traffic
• Wastes money, harms performance
Why peer? (Cont.)
• Solution:
– Two competing ISPs peer with each other
• Result:
– Both save money
– Local traffic stays local
– Better network performance
– More international bandwidth for international
traffic
Why peer? (Cont.)
Internet
ASN 65536
ASN 65538
Why peer? (Cont.)
• A third ISP enters the equation
– Becomes a significant player in the region
– Local and international traffic goes over their
international connections
• They agree to peer with the two other ISPs
– To save money
– To keep local traffic local
– To improve network performance
Why peer? (Cont.)
• Peering means that the three ISPs have to buy
circuits between each other
– Works for three ISPs, but adding a fourth or a fifth
means this does not scale
• Solution:
– Internet Exchange Point
Why peer? – Non-financial Motivations
•
•
•
•
Low latency
Control over routing
Redundancy
Aggregation benefits w/peering and Transit at
IXP
• ISP relationships – be one of the cool kids
• Marketing benefits
• Network reliability
Internet Exchange Point
• Every participant has to buy just one whole
circuit
– From their premises to the IXP
• Rather than N-1 half circuits to connect to the
N-1 other ISPs
– 5 ISPs have to buy 4 half circuits = 2 whole circuits
-> already twice the cost of the IXP connection
Simple Topology
• Layer 2 fabric
• N^N BGP relations
IXP Design
• Each ISP participating in the IXP brings a
router to the IXP location
• Router needs:
– One Ethernet port to connect to IXP switch
– One WAN port to connect to the WAN media
leading back to the ISP backbone
– To be able to run BGP
IXP Design (Cont.)
• IXP switch located in one equipment rack
dedicated to IXP
– Also includes other IXP operational equipment
(Management network, TLD DNS, Routing
Registry, Looking Glass, etc.)
– Optional: Second switch for redundancy
• Routers from participant ISPs located in
neighbouring/adjacent rack(s)
• Copper (UTP) connections made for 10Mbps,
100Mbps or 1Gbps connections
• Fibre used for 10Gbs and higher speeds
Peering at an IXP
• Each participant need to run BGP
– They need their own AS number
– Public ASN, NOT private ASN
• Each participant configures external BGP with
the other participants in the IXP
– Peering with all participants
Or
– Peering with a subset of participants
IXP - Routing
• ISP border routers at the IXP generally should
NOT be configured with a default route or
carry the full Internet routing table
– Carrying default or full table means that this
router and the ISP network is open to abuse by
non-peering IXP members
• ISP border routers at the IXP should not be
configured to carry the IXP LAN network
within the IGP or iBGP
– Set BGP next-hop to local router (Cisco IOS nexthop-self)
IP Address Space
• Some IXPs use private addresses for the IXP
LAN
– Public address space means the IXP network can
be leaked to the Internet, which could be
undesirable
– Filtering RFC1918 address space by ISPs is Best
Practice; this avoids leakage
• Some IXPs use public addresses for the IXP
LAN
– Address space is available from LACNIC
– IXP terms of participation usually forbid carrying the
IXP LAN addressing in the ISP backbone
Hardware
• The IXP Core is an Ethernet Switch
(Mandatory)
– Therefore invest in the best and most expandable
equipment that its financial circumstances allow
– Having 2 switches is good for redundancy if the
funds can allow
• Route Server (Optional)
– Provides ease of configuration for new members
– Direct peering between the IXP members can be
implemented in the absence of the Route Server
Hardware (Cont.)
• Other optional equipment
– Web Server (website, monitoring, etc.)
– Mail Server (email, mailing list, etc.)
– Transit Router (to provide Internet access to the
IXP website, email and staff Internet access)
– Route Collector (Looking glass which assists IXP
members with troubleshooting. It can also be
used to collect routes for statistics measurements)
Hardware - Suggestions
• Try not to mix port speeds
– If 10Mbps and 100Mbps connections available,
terminate on different switches
• Insist that IXP participants bring their own
router
– Moves buffering problem off the IXP
– Ensures integrity of the IXP
– Security is responsibility of the ISP, not the IXP
Location
• The location of the IXP is very important.
• The IXP location should be neutral and low cost.
• In considering the IXP location the following
factors should be considered:
–
–
–
–
–
–
–
Space
Environment Control
Security
Power
Access to terrestrial Infrastructure
Cabling
Support
Recommendations and Best Practices
• Only announce your aggregates and your
customer aggregates at IXPs
• Only accept the aggregates which your peer is
entitled to originate
• Never carry a default route on an IXP (or
private) peering router
• Failing to do so leads to route-hijacks and
leaks
General Info about IXPs
.
.
.
Source:https://prefix.pch.net/applications/ixpdir/summary/
.
.
.
General Info about IXPs
.
.
.
Source: https://prefix.pch.net/applications/ixpdir/?show_active_only=0&sort=traffic&order=desc
General Info about IXPs
Source: https://prefix.pch.net/applications/ixpdir/summary/ipv6/
HOW TO REQUEST INTERNET RESOURCES
• Go to https://solicitudes.lacnic.net/
• Or fill out a form and send it in the body of a
message to hostmaster@lacnic.net
– You can find templates at:
http://lacnic.net/templates/
• Once the online request or the form has been
processed by the system, the requestor will receive a
confirmation email with a ticket number.
• After that the hostmasters will analyze the request.
• If the request is approved, it may be necessary to pay
a fee and to sign the Registration Service Agreement.
Who can request resources?
• The person allowed to request resources for an
organization is the Administrative POC.
• To request resources through the new Requests
System you will have to log in using the
Administrative POC handle.
Requesting an ASN
• In order to qualify for an ASN allocation the organization
should have:
– A unique routing policy, meaning a policy that differs from that
applied by the upstream provider.
– Or, a network with more than one independent connection to the
Internet. (Multi-homed site)
• From January 1, 2007 to December 31, 2010 Lacnic assigned
ASN of 16 and 32bits upon request. However, since January 1,
2011 Lacnic stopped making distinctions between the
assignment of 16- and 32-bit Autonomous Systems Numbers
(ASNs) and will only assign ASNs from a general 32-bit pool.
This change will be introduced to comply with the Global
Policy "Internet Assigned Numbers Authority (IANA) Policy for
Allocation of ASN Blocks to Regional Internet Registries"
adopted in September 2010.
Micro-assignments to Critical Infrastructure
• Micro-assignment -> prefixes between /24 and /20.
• For projects and network infrastructure that are key or critical for the region,
such as IXPs (Internet Exchange Points), NAPs (Network Access Points), RIRs,
ccTLDs, among others.
• IXPs or NAPs must meet the following requirements:
– Duly document the following aspects:
• Prove by means of their bylaws their IXP or NAP capacity. The organization shall have at least
three members and an open policy for the association of new members.
• Submit a diagram of the organization's network structure.
• Document the numbering plan to be implemented.
– Provide a utilization plan for the following three and six months.
– If the applicant does not already have an IPv6 block assigned by LACNIC,
simultaneously request an IPv6 block in accordance with the corresponding
applicable policy.
• The rest of the applications shall be studied based on the analysis of the
documentation justifying the critical and/or key aspects of the project.
• Organizations receiving micro-assignments shall not sub-assign these IPv4
addresses.
Requesting an IPv4 block for ISPs
• To qualify for the allocation of a /22 block the org
must:
– Prove usage or immediate necessity of a /24
– Submit a detailed one-year usage plan for a /23
– Agree to renumber from previously allocated space and
return those IP addresses to their ISPs within 12 months
– If the applicant does not already have an IPv6 block
assigned by LACNIC, simultaneously request an IPv6
block in accordance with the corresponding applicable
policy.
• For a larger block additional requirements apply
Requesting an IPv6 block for ISPs
• To qualify for an initial allocation of a /32 block the
organization should:
– Be a LIR (Local Internet Registry), which means being an
organization that assigns address spaces for its network
services customers
– Not be an end site (end user)
– Document a detailed plan for the services and IPv6
connectivity to be offered to other organizations (clients)
– Announce the allocated block in the Internet inter-domain
routing system, with the minimum possible level of
disaggregation to the one that is publishing the IP blocks,
within a period no longer than 12 months.
– Offer IPv6 services to clients physically located within the
region covered by LACNIC within a period not longer than 24
months
More info
• Policy Manual
– http://www.lacnic.net/web/lacnic/manual
• Registration Services
– http://www.lacnic.net/web/lacnic/servicios-registro
ADVANCED TOPICS
Route Hijacking
• This occurs when a participant in the Internet
Routing announces a prefix for which it has no
authority
• Malicious or by operational errors
• More know cases:
– Pakistan Telecom vs. You Tube (2008)
– China Telecom (2010)
– Google in Eastern Europe (various AS, 2010)
– Latin American cases (beginning 2011)
Route Hijacking
AS 6057
announces
200.40/16
ASN 8158
ASN 8158
receives
receives y
200.40.0.0/16
200.40.0.0/16
200.40.235.0/24
AS 15358
announces
200.40.235.0/24
200.40.0.0/16 AS_PATH ASN1 ASN3 ASN6057
200.40.235.0/24 AS_PATH ASN1 ASN15358
Leaks
• There is not a standard definition of leaks
• But it happens when an ASN “leaks” noncustomer or self-originated routes to other
peers.
• The effects is to give transit to those networks
for the peers of the ASN
Route Leaks
2001:db8::/40
65536
2001:db8:100::/40 65537
How this
should work
without leaks
2001:db8::/40
ASN 64511
Traffic to whole
2001:db8::/40
goes this way
2001:db8:100:/40
Transit
2001:db8::/40
2001:db8:1/48
ASN 65536
2001:db8::/40
Peering
ASN 65537
2001:db8:100:/40
Route Leaks
2001:db8::/40
65536
2001:db8:100::/40 65537
2001:db8:1::/48
65536 65537
Now a Route
Leak
ASN 64511
2001:db8::/40
Traffic to
2001:db8:1::/48
goes this way
ASN 65536
2001:db8::/40
2001:db8:1/48
2001:db8::/40
Peering
2001:db8:100:/40
2001:db8::/40
2001:db8:1::/48
ASN 65537
2001:db8:100:/40
Transit
Attacks against the path
• AS Insertion:
– A router might insert one or more ASNs, other
than its own ASN, into an update message
• False (Route) Origination with valid ASN:
– A router might originate a route for a prefix using
an ASN not authorized to originate routes for that
prefix.
Attacks against the path
AS 6057
announces
200.40/16
ASN 6057
ASN 8158
ASN 8158
receives
receives y
200.40.0.0/16
200.40.0.0/16
200.40.235.0/24
Fake AS 6057
announces
200.40.235.0/24
200.40.0.0/16 AS_PATH ASN1 ASN3 ASN6057
200.40.235.0/24 AS_PATH ASN1 ASN15358
WELL KNOW INCIDENTS
Pakistan Telecom vs. Youtube
• On Sunday, 24 February 2008, Pakistan Telecom
(AS17557) started an unauthorized
announcement of the prefix 208.65.153.0/24
(Youtube)
• One of Pakistan Telecom's upstream providers,
PCCW Global (AS3491) forwarded this
announcement to the rest of the Internet, which
resulted in the hijacking of YouTube traffic on a
global scale.
• Reason: “Fat fingers”
• Video de RIPE NCC
– http://www.youtube.com/watch?v=IzLPKuAOe50
Moratel vs Google
• Reported by Cloudflare on November 06, 2012
• Google's services experienced a limited
outage for about 27 minutes over some
portions of the Internet.
• Moratel (23947) was “leaking” Google one
route and packets were going through
Indonesia
• Reason: “Fat fingers”
ASN and Prefix Hijack
• On 2011 one large European ISP complain that
one of their prefixes was being announced by
a Mexican ISP
• The Mexican ISP review their network but
could not found the problem
• Later it appears that a Brazilian ISP was using
the Mexican ISP’s ASN to announce the
European prefix
• Reason: Poor BGP knowledge
SECURING THE ROUTING SYSTEM
Recall how Internet Resources are managed
Recall how Internet Resources are managed
IANA
ARIN
ISP
LACNIC
NIC.br
NIC.mx
APNIC
ISP #1
LIRs/ISPs
RIPE NCC
LIRs/ISPs
End users
ISP mx
•Each RIR is an authoritative source of information about the
relation “user” <-> “resource”
•Each RIR operates its registration data base
•Members and RIRs sign Service Agreements between them
AfriNIC
Who has the "right" to use resources?
• When an ISP obtains resources from its RIR
(IPv6/IPv4/ASN):
– The ISP has to notify its upstream ASs which prefixes are going
to be announced via BGP
– This is usually done via e-mail, web forms or by updating an IRR
(Internet Routing Registry)
• Upstreams verify (or at least they should) the right of use
for the announced resources
– RIR WHOIS Text-based and not really suitable for automatic
usage
– IRR WHOIS Non-signed information, little additional tools
provided for verification of usage rights except for names,
phone numbers and email POCs
• This verification process is sometimes not as thorough as it
should be
What is RPKI?
• RPKI (Resource Public Key Infrastructure) allows
the validation of an organization right to use of a
certain resource (IPv4, IPv6, ASN)
• RPKI combines the hierarchy of the Internet
resource assignment model through RIRs with the
use of digital certificates based on standard X.509
• RPKI is standardized in the IETF through the SIDR
WG. It has produced RFCs 6480 – 6492
RPKI
• All RPKI signed objects are listed in public repositories
• After verification, these objects can be used to
configure filtering in routers
• Validation Process
– Signed objects have references to the certificate used to
sign them
– Each certificate has a pointer to an upper level certificate
– The resources listed in a certificate MUST be valid subsets
of the resources listed in its parent's certificate
– In this way a trust chain can be traced to a "trust anchor"
both cryptographically as well as in CIDR terms
X.509 v3 certificates with RFC 3779
extensions
Version
• X.509 Digital Certificates
– Subject, validity period, public
key and other fields
• With extensions:
– RFC 3779 defines extensions that
allow the representation of
Internet resources as certificate
fields
• List of IPv4, IPv6 and ASNs
assigned to an organization
• Implemented in OpenSSL 1.0c
onwards
Serial Number
Signature Algorithm
Issuer
Subject
Subject Public Key
Extensions
Subject Information
Authority (SIA)
Authority Information
Access (AIA)
Addr: 10.10.10.0
Asid: 65535
ROAs
• Using Certificates we can create objects
describing the origin of a prefix
• ROAs: Routing Origin Authorization
– ROAs contain data on the allowed origin-as for a
set of prefixes
– ROAs are signed using the certificates generated
by the RPKI
– Signed ROAs are copied to the repository
ROAs (ii)
• A simplified ROA contains the following information:
Prefix
MaxLen
Origin AS
Valid Since
Valid Until
200.40.0.0/17 20
6057
2013-01-02
2013-12-31
200.3.12.0/22 24
28000
2013-01-02
2014-12-31
• These ROAs states that:
– "The prefix 200.40.0.0/17 will be originated by ASN 6057
and could be de-aggregated up to /20" "This statement is
valid starting on Jan 2, 2013 until Dec 31, 2013"
• Other ROA content
– ROAs contain cryptographic material that allows validation
of the ROAs content
ROAs (iii) - Validation
• ROAs validation process includes:
– Criptographic validation of End Entity certificates
(EE) that are included in each ROA
– CIDR validation of resources listed in the EE
against the resources listed in the issuing
certificate
– Verification that prefixes listed in the route origin
attestations are included in the prefixes listed in
the EE certificates of each ROA
Creation of ROAs
Creation of ROAs
Origin Validation
• Routers build a database with the information
they receive from the caches
• This table contains
– Prefix, Min length, Max length, Origin-AS
• By applying a set of rules a validity status is
assigned to each UPDATE prefix
• Network operators can use “validity” attribute
to construct routing policies
Origin Validation
VALID
UPDATE 200.0.0.0/9
ORIGIN-AS 20
IP prefix/[min_len – max_len]
Origin AS
172.16.0.0 / [16-20]
10
200.0.0.0/[8-21]
20
• If the "UPDATE pfx" is not covered by any entry in
the DB -> "not found"
• If the "UPDATE pfx" is covered by at least one entry
in the DB, and the origin-AS matches the ASNs in
the DB -> "valid"
• If the origin-AS does NOT match -> "invalid"
Origin Validation
INVALID
UPDATE 200.0.0.0/22
ORIGIN-AS 20
IP prefix/[min_len – max_len]
Origin AS
172.16.0.0 / [16-20]
10
200.0.0.0/[8-21]
20
• If the "UPDATE pfx" is not covered by any entry in
the DB -> "not found"
• If the "UPDATE pfx" is covered by at least one entry
in the DB, and the origin-AS matches the ASNs in
the DB -> "valid"
• If the origin-AS does NOT match -> "invalid"
Origin Validation
INVALID
UPDATE 200.0.0.0/9
ORIGIN-AS 66
IP prefix/[min_len – max_len]
Origin AS
172.16.0.0 / [16-20]
10
200.0.0.0/[8-21]
20
• If the "UPDATE pfx" is not covered by any entry in
the DB -> "not found"
• If the "UPDATE pfx" is covered by at least one entry
in the DB, and the origin-AS matches the ASNs in
the DB -> "valid"
• If the origin-AS does NOT match -> "invalid"
Origin Validation
NOT FOUND
UPDATE 189.0.0.0/9
ORIGIN-AS 66
IP prefix/[min_len – max_len]
Origin AS
172.16.0.0 / [16-20]
10
200.0.0.0/[8-21]
20
• If the "UPDATE pfx" is not covered by any entry in
the DB -> "not found"
• If the "UPDATE pfx" is covered by at least one entry
in the DB, and the origin-AS matches the ASNs in
the DB -> "valid"
• If the origin-AS does NOT match -> "invalid"
Routing Policies with Origin-validation
• Using the BGP validation attribute network
operators can construct routing policies
• For example:
– Assign higher preference to routes with status
“valid” than routes with status “unknown”
– Drop routes with status of “invalid”
• Very important, RPKI is a source of data,
operators are free to use it as it suits them
better
RPKI in action
Routers validate updates
from other BGP peers
RPKI Management
System
Caches feeds routers
using RTR protocol with
ROA information
Cache
Repository
Caches retrieve and cryptographically validates
certificates and ROAs from repositories
RPKI Operation Modes
• “Hosted” mode
– LACNIC issues certificates and keeps public and private
keys in its systems
• Certificates are issued at the request of member
organizations
– LACNIC provides a web interface for management
• “Delegated” mode
– Each organization has it’s own certificate, signed by
LACNIC’s CA
– The organization sends signing requests to LACNIC,
who returns them signed
• “Up-down” protocol
RPKI today
• 5 CAs and repositories working (RIRs) in
hosted and delegated mode (just APNIC)
• Validation, CA and origin validation working
software and devices
• Some other tools built around (bgpmon,
LACNIC Labs, RIPE Labs)
• ~ 11,700 routes signed (~ 8.9% 1,226 invalid or
hijacks)
• Origin Validation available in Cisco, Juniper,
Quagga, BIRD
LACNIC’s RPKI Structure
Self-signed RTA
LACNIC RTA
LACNIC’s Resources
Signing chain
LACNIC
Production
<<INHERITED>>
ISP #2
ISP #1
Resouces of ISP #2
ROA
End Entity cert.
Resources of ISP #1
ROA
End Entity cert.
End User CA
#1
(Resources of EU#1)
ROA
End Entity cert.
ROA
End Entity cert.
LACNIC’s RPKI Structure (ii)
• CAs
– Entity that issues certificates (bit CA=1)
• ISPs can use the certificate to sign it’s clients’
certificates
• Certificates Repository
– Repository of certificates, CRLs and manifests
– Accesible through “rsync”
• Management Interface
– User web interface for those who prefer “hosted”
mode
RPKI CA Services
• Children certificates issuance when the
registration database is updated or on user
demand
• Children certificates revocation in a
centralized manner or on user demand
• Periodic issuance of CRL for CA certificate
• CA Certificate and children certificates
publication in a public repository (rsync)
Conclusions
• The routing system is one of the core
operations of the Internet
• Still is vulnerable to attacks and bad configs
• Some work has been done (RPKI, Origin
Validation)
• We need to continue our work
– Protocol specification
– Deployment (Filtering, RPKI, Origin validation)
Links / References
• LACNIC’s RPKI System
– http://rpki.lacnic.net
• LACNIC’s RPKI Repository
– rsync://repository.lacnic.net/rpki/
• To see the repository
– rsync --list-only
rsync://repository.lacnic.net/rpki/lacnic/
• RPKI Statistics
– http://www.labs.lacnic.net/~rpki
Questions?
Comments?
CASA DE INTERNET DE LATINOAMÉRICA Y EL CARIBE
twitter.com/LACNIC
facebook.com/ LACNIC
youtube.com/user/lacnicstaff
gplusme.at/LACNIC
Sofía Silva Berenguer
sofia @ lacnic.net
Thank you!