Interconnection, Peering, and
Settlements
Geoff Huston
Overview
•
•
•
•
•
•
•
•
Introduction
Interconnection: Peer or client
Interconnection architecture
Interconnection financials
Settlement models
Settlement structures
QOS & financial settlement
Conclusions
Introduction
• Tens of thousands of ISPs operating in
deregulated business space
• Complex environment
– Transaction between two ISPs involves multiple
providers
• Basic financial cost distribution is based on
bilateral relationships of customer/provider and
mutual peering
• Internet industry use small set of physical
interconnection mechanisms (LAN switches)
Interconnection: retailing, reselling,
wholesale
• Internet is an outcome of business and
technology interaction
• ISPs do not have clear precise roles:
–
–
–
–
Retail, resel, and wholesale
Retail ISP can easily become a wholesale provider
Many ISPs operate as client and provider
Hard to support stable segmentation into wholesale
and retail market sectors
Intrernet enviroment:
•There is no well ordered hierarchical model of a set of wholesale ISPs and
retail ISPs
•Diverse ISPs operating as retailers and wholesale providers to other retailers
Peer or Client
• Leads to a question of who is making subjective
decision and on what basis.
• Traditional public solution: pay a fee to a
regulator that gives ISP a peer license
• Two problems:
– Regulators are artificial in defining the market entities
(client vs. peers)
– Discourages large-scale private investment, thus
putting funding burden on public sector
Things have changed
• Regulatory environment is changing to
shift burden of comm. infrastructure
– Public sector to private investment
– Deregulated environment
– No one can say whether two ISPs are client/provider or peers
• Who can say so in the industry?
–
–
–
–
–
–
Commercial Internet eXchange (CIX)
Based on description of infrastructure of each party
Peer: if you have a national transit infrastructure capability
Modified later: pay a fee to CIX assicaition
Not bilateral but multilateral relationship with other peers !!
Zero financial settlement (based on a fee)
• Other models use functional peer specification:
– If ISP attaches to physical exchange entity, it is up to the ISP to open
bilateral negotiation with other ISPs attached to the exchange
• True peer relationship is based on the assumption that either
party can terminate the interconnection
– If one ISP relies on the interconnection more than the other 
provider/client relationship
– If there is a balance of mutual requirement between two ISPs  peer
interconnection relationship
– Problem: no metrics to quantify requirements (based on perception)
• There are various levels of peering in today’s internet due to
business pressures
– Local ISPs see competing local ISPs as peers
– Local ISPs are clients of trunk ISPs
Interconnection Architecture
• Strict hierarchical structure
– Worst case: traffic between two ISPs may traverse transit ISPs
– Extended paths are inefficient and costly
– To reduce costs, ISPs at different levels construct bilateral
interconnections
How to connect with ISPs?
• Connect with all ISPs (full mesh, not scalable:N2 connections)
• Local exchange model
– ISP connects to a single local exchange (scalable: N connections)
– Exchange router is active component in peering policy
• Each ISP must have multilateral peering with all other ISPs
– Router must execute its own routing policy
• When two ISPs advertise a route to same destination, router makes
decision on behalf of all other connected ISPs
– Router may not be completely neutral to all ISPs
What do ISPs expect?
• Flexibilty of policy determination from exchange structure
• Bilateral interconnection at the exchange structure
• Make policy decisions when same destination is
adverised by multiple providers
• Exchange must be neutral with respect to individual
routing policies
• HOW?
– Modify exchange model to use LAN switch as exchange element
• Each ISP:
– has a dedicated router at the LAN exchange
– has a bilateral peering agreement with another ISP by initiating
router peering session with the other’s router
– If multiple peers advertise a path to same dest, ISP can use its
own policy-based preference to choose route
• Exchange environment must offer high degree of
resilience and security (costs a lot)
Distributed exchange model
• Exchange comes to ISP location
• Must have uniform access technology between every
exchange participant
• Issues: switching speed (contention can be a problem)
Network Access Points NAPs
• Roles of NAP:
– Exchange provider between ISPs
– Transit purchase site to make agreements between
ISPs and trunk ISPs
Exchange Business Models
• In the ISP industry, a common business model require the
internet exchange to be
–
–
–
–
–
–
Operated by neutral party that is not an ISP
Constructed in robust and secure fashion
Located in high density areas of internet market
Scalable
Operated in sound and stable fasion
Others (Performance of the exchange, QOS)
• Common business models use flat-fee structure
– Based on the number of rack units used by an ISP at the exchange
• Other models are strcutured as cooperative entity between a
number of ISPs
– Problem: no ISP wants to financially take responsibilty for ensuring
quality of the exchange
Today’s Internet
• Increasing ISPs will lead to increasing complexity of
interconnection structure
• Inability to reach stable cost distribution model makes each
ISP optimize itself by making direct connections with peer
ISPs (thru exchanges or direct 1:1 links)
• Curdity of inter-AS routing policy tools makes internet
structure a source of considerable concern especially with
– Absence of coherent policy (or even commonly accepted set of practices)
– Lack of administration of the AS space
Interaction Financials
• Cost distribution:
– Compensation of all ISPs who participate in the delivery of a
service to a customer of a single ISP
• Users want comprehensive end-to-end service with clients
being parts of different ISPs
• How do all ISPs involve in a transaction?
• Who incur the cost in supporting the transaction?
• Who receive compensation?
• What is the cost distribtion model?
What is the currency of interaction?
• Routing advertisements
– ISPs exchange routing entries to allow traffic flows
– Traffic flows in opposite direction of route advertisement
Types of Routes
• Clients routes
– Passed to ISP routing by contract with client, static configuration at edge of ISP,
learned by BGP, or part of DHCP addresses
• Internal ISP routes
– DNS, SMTP, SNMP, POP..etc
• Upstream routes
– Learned from making a transit service contract with upstream ISP
• Peer routes
– Learned from exchange or private interconnection
• What is the route export policy ?
Internet settlement models
• Packet cost accounting (strawman model)
– Everytime a packet crosses network boundry, it is sold to next ISP
– Ultimately, the packet is sold to the receiver client
• Pros:
– Revenue gains from packets deliverd on egress from network
– Economic incentives not to drop packets in transit ISPs
• Cons:
– Packet drop is inevitable
– Mechanism is open to abuse
TCP session accounting
• Network boundry can
– Detect initial TCP handshake
– Count all subsequent packets with same TCP session
– Session initiator pays for entire traffic flow
• Such accounting allows for settlement based on
dutration (TCP packets) or volume (TCP sessions)
• Problem: very hard to do
– Router at the network boundry must do all work !!
• Real problem with any settlement models
– Today’s internet have many retail pricing structures
– Based on received volume, sent volume, mix, access capacity
– There is no uniform retail pricing
Internet settlement structures
• Two structures of interactions between two ISPs
– Customer/provider and peering with no form of financial settlement
• Sender Keep All (SKA)
– Usually applies to domestic traffic between two ISPs
– Stable when both parties perceive equal benefit from interconnection
– Ex. Among local ISPs, regional ISPs, national ISPs
• How does it work? On each interconnection
– each ISP ONLY presents/accepts to/from other ISP routes associated with
its customers
– Clients make contract with an ISP to present their routes to all other
customers of the same ISP, to the upstream providers of the ISP, and all
peer ISPs
How does all that look like ?
• Internet into two domains: transit ISPs, local ISPs
• Transit ISPs: high capacity carriage infrastructure
• Local ISPs: retail services
– Participate at exchanges with SKA peer interconnection with other ISPs
– Exchange does not have full connectivity  ISPs purchase transit services
Negotianted financial settlement
• Alternative to customer/provider and peer structures
• Based on both parties selling services to each other
across the interconnection
• Simple model:
– Measure volume of traffic in each direction
– Use single accounting rate for all traffic
– At the end of accounting period, two ISPs settle based on the agreed
rate applied to the traffic
• Which way should the money flow in relation to traffic flow?
– One model: Originating ISP should pay terminating ISP to deliver traffic
– Another model: when traffic is generated because of an action of a
receiver (webpage, downloads), terminating ISP should pay
Settlement Debate
• Despite great ISP attention, today’s internet does not have
sound models of financial settlements
• Why has the internet managed to pose hard challenge to
the ISP industry?
– Caused by adopted retail models of ISP services
– The internet as retailed to clients is not a comprehensive end-to-end service
– Internet works as a result of partial path paired services
• Sender funds initial path component and receiver funds terminating path
component
• Natural outcome of today’s internet settlement environment
is one of aggregation of ISPs
QOS and financial settlements
• The shift towards end-to-end service model and support
of QOS are strong factors to change current ISP service
model
• Meaningful inter-provider financial settlements depend
highly on introducing end-to-end service retail models
• That in turn depends on universally shifting from besteffort regime to layered end-to-end service regimes
Conclusions
• $0 peering and customer/provider relationships
are the only stable models within the internet
• As a consequence
– deployment of end-to-end QOS is highly
unlikely in such an environment
– Inability to support highly diverse ISP env
– Aggregation within ISP industry