APRICOT 2007 Peering
Tutorial
Instructor: William B. Norton
Co-Founder & Chief Technical Liaison
Equinix, Inc.
wbn@equinix.com
Orchid Room
Bali, Indonesia
Agenda
• 9:15 – 10:30 Introduction to Peering
– Definitions, Applying the definitions
– Motivations to peer or not
• 10:30-11:00 Break
• 11:00-12:30 Peering Simulation Game
Internet Researcher
•
•
•
•
•
EQIX: Carrier Neutral Colocation
90% Externally Focuses
Observe: documentation on HW&Protocols
Lack of Operations documentation
Research How does Peering work?
White paper process..
Community Operations
Research
• Ground Truth w/dozens of experts
• Write White Paper v0.1
• Walk community through WP for
comments
• Revise White Paper into new version
• Present White Paper at conferences
• Solicit comments over lunches and
dinners
White papers so far…
Internet Operations White Papers
“Interconnection Strategies for ISPs”
“Internet Service Providers and Peering”
“A Business Case for Peering”
“The Art of Peering: The Peering Playbook”
“The Peering Simulation Game”
“Do ATM-based Internet Exchanges Make Sense
Anymore?”
7) “Evolution of the U.S. Peering Ecosystem”
8) “The Asia Pacific Internet Peering Guidebook”
9) “The Folly of Peering Traffic Ratios?”
10) “Internet Video: The Next Wave of Disruption”
1)
2)
3)
4)
5)
6)
Freely available. See Web site or send e-mail to wbn@equinix.com
Start at the most basic definitions
Definitions of Peering
Take Notes
The Internet is a Network of Networks.
Def:
Transit
An ISP sells access
to the
Internet, so…
…must itself get attached to someone
who is already attached to the Internet.
1) Transit Provider sells metered access to the
Global Internet
Transit Provider A
$ meter
Transit Provider B
2) Peering is a business relationship whereby
two companies RECIPRICALLY
exchange
Def:
Peering
access to each others customers.
Why Peer?
1. Reduce Transit Costs
2. Lower Latency
3. More Control
over Traffic
Transit Provider A
Peering
Transit Provider B
What is this Internet Peering Ecosystem anyway?
Global Internet Peering Ecosystem
JP Internet Region
AU Internet Region
US Internet Region
Tier 1 ISPs
Tier 2 ISPs
Content Providers
Characteristics of these
Ecosystem Organisms?
Ecosystem Member: Tier 1 ISP
P
Tier 1 ISP
$ … $
T
T
P
Def: A Tier 1 ISP is an ISP
that has access to the
ENTIRE Internet Region Routing Table
Solely via Peering Relationships
(Doesn’t buy transit from anyone
to reach any destination in the
Internet Region.)
Motivation: Is NOT motivated to
Peer in region to reduce transit fees,
Is NOT motivated to peer with
anybody else.
Behavior: “Restrictive” Peering
Policy
$
$
T … T
Ecosystem Member: Tier 2 ISP
P
Tier 2 ISP
P
Def: A Tier 2 ISP is an ISP
that has to purchase Transit to access
some part of the Internet Region.
$ … $
T
T
Motivation: Is motivated to
Peer in region to reduce transit fees.
Behavior: “Open” Peering or
“Selective” Peering Policy
Active in Peering Forums
$
$
T … T
Content Providers
Def: A Content Provider focuses on
Content
Provider
content development and does not
Sell access to the Internet.
Motivation: SLAs w/well known ISP
Behavior: “No Peering” Policy
Generic Peering Ecosystem
Tier 1 ISPs
$ flow up
T
T
T
T
T T T T T
Tier 2 ISPs
TTT
Active Peering Groups
Peering Forums
IX Meetings
Content Providers
Quiz
• Draw:
Tier 1 ISP X
– Piece of paper
1) Definition of Transit:
________________
2) Definition of Peering:
________________
P
Tier 1 ISP Y
$
T
T
$
Tier 2 ISP B
Tier 2 ISP A
T
$
Content
Provider C
3) Definition of an “Open” Peering Policy: _____________________
4) Definition of a “Selective” Peering Policy: _____________________
5) Definition of a “Restrictive” Peering Policy: ____________________
Apply Defs: Peering Dynamics & Motivations
Tier 1 ISP X
$
P
$
P?
T
Tier 1 ISP Y
P?
No, Tlike $.
No, like $,
Tier 2 ISP
Tier 2 ISP
T
No, I already hear your routes
for FREE!
$
P?
& I Like
customer
B
Content
Provider
Synch Point:
You have all the defs needed to predict behavior in the Peering Ecosystem.
You should be able to answer the question at hand.
Why Telstra Won’t
Peer in Australia?
Why Telstra Won’t
Peer in Australia?
That’s right.
They don’t have to.
Why Telstra Won’t
Peer in Australia?
They don’t have to. And,
No different from other
Tier 1’s in the world
U.S. Evolution #1
EvolutionCable
#1Companies Peer
T1 ISPs
Significant Evolution…
T2 ISPs
Content
1) Volume of traffic is huge
2) Cable Cos Open Peering
3) “Kazaa Effect” amplifies
peering benefits
U.S. Evolution #2
T1 ISPs
Large Scale
Content Players Peer
Significant Evolution…
T2T2ISPs
ISPs
Content
Content
1) Volume of traffic is huge
2) Content is Open Peering
3) Improves End-User
Experience
4) Leading Players are
paving the way
…need to move out of
Bankrupt colo anyway…
U.S. Evolution #3
Cable Cos Peer w/Large
Scale Content Players
T1 ISPs
Significant Evolution…
T2T2
ISPs
ISPs
T2
ISPs
Content
Content
Content
1) Volume of traffic pulled away
from T1s is huge
2) Reduces perceived need for
T1s (for local delivery anyway)
3) T1s still needed for distance
 Content Literally right on the
Cable Company Network
The Process of Peering
Peering Decision Tree:
3 Phases of Peering
Agreed on Terminology
Then the Interviews with Peering
Coordinators and the 3 Phases of Peering:
1) Identification or Potential Peer – the who
2) Initial Contact and Qualification – the why
3) Implementation Discussions – the how
What does Internet Transit Cost?
Commit
Commit
Commit
Commit
to
to
to
to
1
5
15
25
mbps
mbps
mbps
mbps
Transit Price
Min/month
$100 per mbps
$100
$80 per mbps
$400
$70 per mbps
$1,050
$50 per mbps
$1,250
Transit Cost Model
$120
$/mbps
$100
$80
$60
TransitPrice
$40
$20
$0
1 5 9 13 17 21 25 29 33 37 41 45
Mbps Exchanged
Motivations
•
Why Peer? Motivations for
Peering
Financial: Reduce load on expensive Transit service
• Traffic src/dest
• Measure vs Intuit
• Usage-based Billing
• Engineering: Lower latency
Transit
$$$
ISP A
Seek transport
Interconnection
$
•
1st
Stage of Peering:
• Top 10 destination ISP list
x
Transit ISP
ISP B
Transit
$$$
Top 10 list
Sample Top 10 Destination List
Internet Service Provider A
AS Number Mbps Destination ISP
6172 24.35 HOME-NET-1
701
Contact
[HOME-NOC-ARIN]
8.90 ALTERNET-AS
[IE8-ARIN]
1668
8.14 AOL-PRIMEHOST
[AOL-NOC-ARIN]
4766
7.08 APNIC-AS-BLOCK
[SA90-ARIN]
3320
5.12 RIPE-ASNBLOCK4
[RIPE-NCC-ARIN]
4.24 BACOM
[EQ-ARIN]
6327
3.90 SHAWFIBER
[IAS-ARIN]
1
3.89 BBNPLANET
[CS15-ARIN]
7018
3.66 ATT-INTERNET4
[JB3310-ARIN]
9318
3.13 APNIC-AS-3-BLOCK
[SA90-ARIN]
5769
2.67 VIDEOTRON
[NAV1-ARIN]
6830
2.30 HCSNET-ASNBLK
[MD205-ARIN]
9277
2.22 APNIC-AS-3-BLOCK
[SA90-ARIN]
2.08 TAMPA2-TWC-5
[JD6-ARIN]
2.05 SprintLink
[SPRINT-NOC-ARIN]
577
10994
1239
Make W e
Def: Peering
I. Phase 1: Identification of Peer:
Traffic Engineering Data Collection
and Analysis
• Reduce load on expensive Transit
service
Transit
$$
• Traffic src/dest
ISP A
• Measure or Intuit
Seek
interconnection
Transit ISP
ISP B
• ResultTop 10 list
• 2nd Goal: Lower latency
Transit
$$
Part of Broad
Business
Relationship?
Dominant Traffic
Flow?
Large new customer
impact?
Yes
Phase 1:
Identification
of Potential
Peer
Traversing
Expensive Transit
Circuit?
Yes
Yes
Will Peering have a
positive affect on my
network?
Yes
Proceed
to Phase
2:
Contact
Peer
Yes
II. Phase 2: Contact &
Qualification, Initial Peering
Discussion
A) Part of broader business transaction?
a) E-mail person or peering@<ispdomain>.net
b) Exchange point participant list
c) Tech-c/admin-c from DNS/ASN registries
d) Engineering Forums NANOG, IETF, RIPE, etc.
e) Trade shows: speakers and booth staff
f) Target ISP sales force
g) Target ISP NOC
II. Phase 2: Contact &
Qualification, Initial Peering
Discussion
• Mutual NDA
• BiLateral Peering Agreement (BLPA)
• Traffic Value Data shared
– One basis: PeeringCost<TransitSavings?
• Requirements Exchange
– (e.g. Must be at n Public Peering Points)
• Either Party may walk away…..
Finding the Right Contact
Larger
Business
Transaction
peering
@ or
personal
contact
Exchange
Point
Contact list
Phase 2:
Contact and
Qualificatio
n
Initial
Contact
Sign
NDA,
see
policies
Share
traffic
data,
BLPA
Do both parties find
motivation to continue
peering discussion?
No
Close discussion
Yes
Proceed to
Phase 3:
Implementati
on
Discussion
tech-c or
admin-c
in DNS/
ASN
Registry
Operations
Forum
Trade
Shows
Sales
Force
III. Phase 3: Implementation
Discussions
How to interconnect?
Direct Circuit-based Interconnection
Vs
Exchange-Based Interconnection
White Paper available:
“Interconnection Strategies for ISPs”
Email to: <wbn@equinix.com>
Exchange-based vs. Direct
Circuit Interconnection
Cost Comparison of Interconnection Strategies
$400,000
Direct Circuits Model
Monthly Cost of Interconnection
$350,000
$300,000
MUX Big Pipes Model
$250,000
$200,000
Dark Fiber Model
$150,000
$100,000
$50,000
64
61
58
55
52
49
46
43
40
37
34
31
28
25
22
19
16
13
10
7
4
1
$# of participants
See http://www.nanog.org/mtg-9905/norton.html for slides
Exchange Selection Criteria
Criteria Includes:
•
•
•
•
•
•
•
•
•
Telecommunications Access Issues
Deployment Issues (getting in & up)
ISP Current Presences (there yet?)
Operations Issues (restrictions?)
Business Issues (neutrality/alignment)
Cost Issues ($$)
Credibility Issue (backing,attraction)
Exchange Population (side effect)
Existing Exchange vs. New Exchange?
Facilities Allocation
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
Facility fills
Facility becomes widely known
Critical mass is achieved
Bulk selling of rack 10-packs
Cross connect revenue grows rapidly
First Carrier
First ISP(s)
First CP(s)
“The Startup Hump”
Large Facility
Scaling
Secondary Carrier(s)
Multiple ISPs provide CP redundancy
Time
Exchange Environment
Evaluation
Exchange Environment Selection Criteria
Telecom
Access
Issues
Deployment
Issues
ISP Current
Presences
Operations
Issues
Business
Issues
Cost
Issues
Peering Implementation
and Operation
Credibility
Issues
Exchange
Population
Existing vs.
Emerging
IV. Summary
• Findings from Interviews:
Several phases lead to peering:
1) Identification of Potential Peers
2) Contact & Qualification
3) Implementation Discussions
(Vary in rigor and policy)
• 9 Selection Criteria for Exchanges
(Vary in relative weighting of criteria)
Peering Decision Tree
Traffic
Engineering
Data
Collection
and Analysis
Where is traffic destined?
Where is it handed off in a peering or transit
relationship?
How will traffic on my backbone change as
a result of regional peering?
If beneficial to peer, where to peer?
Yes
Mostly e-mail/phone, conversations
Sign mutual NDAs
Peering Criteria met? If not, reestablish
contact when criteria met.
Discuss benefits to each of peering based
on traffic data (MRTG Weekly MBps Avg)
Does each side benefit equally?
Peering Poker.
Peering
Negotiation
Yes
"Interconnection Strageties for ISPs" http://www.nanog.org/mtg-9905/norton.html
demonstrate economic and technical dynamics of these two methods of
interconnection
Peering
Methodology
Discussion
Yes
Yes
ExchangeBased
Peering
Direct-Circuit
Peering
Telecommunications
Access Issues
Deployment
Issues
Current
Presence
Exchange
Population
Who can sell me Circuits
Easiest to peer where one
and when?
of us already has a
Dark Fiber?
presence
How do I and how fast can
Is there carrier diversity?
I get equipment into the
How fast can I get
exchange?
connectivity into the
Can I get in?
Side benefits of exchange
exchange?
Do I have to physically
point?
show up, flight, schedule,
Who else can I peer with
time costs, etc. or remote
there?
hands install?
Are there transit sales
How fast can contracts get
possibilities?
done?
Operations
Issues
Cost Issues
Is it easy to evolve my
presence?
More Racks?
More bandwidth?
Operations restrictions?
"Must use" 1 carrier?
"Must buy transit from a
particular ISP?"
Do I have to connect to a
switch?
Is the switch congested?
Credibility
Is exchange well backed?
Will it be around next
year?
WIll it attract the "right"
ISPs?
When will critical mass be
reached?
What are the fees?
Cost of alternative
exchanges?
What are the parameters
surrounding fees?
Do I have to connect to
switch?
When does Peering Make
Sense Financially?
Definition of Transport
Def: Transport refers to the physical/data
link layer media (e.g. circuits, gigE
switching fabric, gigE over fiber cross
connects).
“I’m using Level 3 for transport into the
Exchange Point” --or-“I’m using Telseon gigE for transport into
the Exchange”
10M
R
100M
IX Switch
R
10M
100M
Peering versus
Transit
TransportIntoIX IXFees
TotalPeeringCost
$250
$200
$450
$500
$500
$1,000
Peering vs Transit
$1,200
$/Mbps
$1,000
$800
TransitPrice
$600
10MPeer
$400
100MPeer
$200
$0
1
4
7 10 13 16 19 22 25 28 31 34 37 40 43
# Mbps Exchanged
Generalization:
ISP Peering Breakeven Analysis
Graphs
$/Mbps
Exchanged
Breakeven Point
(ISPs Indifferent between
Peering and Transit
traffic exchange)
Peering
Prefer Peering
Risk
Cost of Transit
Cost of Traffic Exchange in Peering Relationship
Number of Mbps exchanged
Why NOT to Peer…
Already get Traffic for free (through existing
peering relationships)
• Personality Clashes
• Traffic Inequities (“I don’t want to haul your
traffic for free around the world!”)
• Lack of Technical Expertise “We don’t believe
you have the clue”
• Transit Sales Preferred “Let me introduce you
to our salesguys”
• BGP is TOUGH
Questions?
Peering Simulation Game is next