Asia Pacific Peering
Guidebook (v1.6)
William B. Norton
Co-Founder & Chief Technical Liaison
Internet Researcher
•
•
•
•
90% externally focused
Many documents on Protocols
Lack of Operations documents
Research: Peering
– How does Peering work?
– What are the definitions?
– What are the “Tricks of the Trade?”
White paper process..
Community Operations
Research
1. “Ground Truth” w/dozens of experts
2. Write White Paper v0.1
3. Walk community through WP for
comments
4. Revise White Paper into new version
5. Present White Paper at conferences
6. Solicit comments over lunches and
dinners
White papers so far…
Internet Operations White
Papers
“Interconnection Strategies for ISPs”
“Internet Service Providers and Peering”
“A Business Case for Peering”
“The Art of Peering: The Peering Playbook”
“The Peering Simulation Game”
“Do ATM-based Internet Exchanges Make
Sense Anymore?”
7) “Evolution of the U.S. Peering Ecosystem”
8) “Asia Pacific Peering Guidebook”
1)
2)
3)
4)
5)
6)
Freely available. See Web site or send e-mail to wbn@equinix.com
Or Google for “William B. Norton”
Research Topic: Peering in Asia
• Goals of this 12 month research
– Document how the Internet Peering Ecosystems in
Asia are different from the rest of the world
– What did Peering Coordinators find counter-intuitive?
– What surprises did they run into as they expanded
their networks into and within Asia?
• Result: “The Asia Pacific Internet Peering
Ecosystem” (v1.6)
– Value of IX, Peering Policies, Biz cases for Peering in
AP Ecosystems, etc.
What is this “Peering
Ecosystem?”
• Global Internet Peering Ecosystem: A
system of autonomous but interconnected
Internet Regions, each with players that
provide connectivity and content to the
Internet.
The Global Internet Peering Ecosystem
Global Internet Peering Ecosystem
Internet Peering Ecosystem
Tier 1 ISPs
Tier 2 ISPs
Content Player
Ecosystem Players
• Tier 1 ISPs (ISPs that have access to all
the Internet Peering Ecosystem routes
solely through free peering relationships),
• Tier 2 ISPs (that must buy transit from
someone to reach routes within the
Internet Peering Ecosystem), and
• Content Providers who don’t sell access
to the Internet but offer content.
Motivations: Peering Policy
• Def: A Restrictive Peering Policy is an
articulation of an inclination not to peer.
• Def: A Selective Peering Policy is an
articulation of an inclination to peer, but
with some conditions
• Def: An Open Peering Policy is an
articulation of an inclination to peer with
anyone.
Japan Peering Ecosystem
In Japan, the set of Tier 1 ISPs include
• Japan Telecom (JT Open Data Network
(ODN)),
• NTT (and Internet Initiative Japan (IIJ)),
• KDDI, and
• POWEREDCOM,
Japanese Peering Ecosystem
Japan
Internet Region
NTT & IIJ
KDDI
Tier 1 ISPs
Japan Telecom
POWEREDCOM
@nifty
Yahoo!Broadband
BIGLOBE
So-net
Tier 2 ISPs
Content / Enterprise Companies
Japanese Peering Ecosystem
•
•
•
•
•
•
80% JP Traffic stays in JP
Tier 1 ISPs own infrastructure
Major Disruption: Yahoo!BB (48Gbps)
40+Mbps DSL for <$50 USD/mo
FTTH for $100/mo
1Mbps streaming TV
Interconnect Region
Traffic in Tokyo
Distributed soon
Y!BB especially
NTT OCN
P
KDDI
P
P
P
P
Japan
Telecom
IIJ
Tokyo
Domestic Peering in Japan
BLPA
peering@ doesn’t work – need legs o the ground
Japanese surprises: 3 IXes. Which one to use?
Japan Domestic Exchange Points
Name
JPIX
JPNAP
NSPIXP2
#Mbps Peered # Peers
34000
109
30000
38
18000
100
Cost
$ 11,500
$ 13,000
$ 5,000
IX Value
$ 4,096
$
761
$ 3,257
Monthly Peering cost=$2500 for local loop, $2500 for rack, $4500 FastE port
Transit~$110 for 100Mbps commit
Aggregate traffic volume 34Gbps
Value of JPIX to participant=(34000Mbps*$110/Mbps)/109 - $11,500=$4096/mo
Business Case for JP Peering
Transit Pricing in Japan
Commit
(in Mbps) $ per Mbps (USD)
1
$490
10
$150
100
$110
1000
$50
JP Peering w/FastE
The Cost of Peering in Japan (100M peering model)
Assume NTT Local Loop STM-1 or 100M
ethernet into Otemachi colo
Colocation costs for a rack
FastE Port
Total Cost of Participation at IX
Effective Peering Bandwidth
75% * 94Mbps
Minimum peering cost of traffic exchange
------compared to -----Transit Price (at 100Mbps commit)
Peering Break Even Point
$2,500
$2,500
$4,500
$9,500
per month
per month
per month
per month
70.5 Mbps
$134.75 per Mbps
$110 per Mbps
86 Mbps
JP Peering w/FastE
JP Peering w/gigE
The Cost of Peering in Japan (gigE peering model)
Assume NTT Local Loop STM-12 into
Otemachi colo
Colocation costs for a rack
gigE Port
$9,000
$2,500
$9,000
$20,500
Total Cost of Participation at IX
Effective Peering Bandwidth
Minimum peering cost of traffic exchange
------compared to -----Transit Price (at 1000Mbps commit)
Peering Break Even Point
75% * 940Mbps
per month
per month
per month
per month
705 Mbps
$29.08 per Mbps
$50 per Mbps
410 Mbps
JP Peering w/gigE
Singapore Peering Ecosystem
SingTel
PI
StarHub
20% stays
in SG
Singaporean
Internet Region
SingTel
Pacific Internet
Tier 1 ISPs
ST Teleport
Tier 2 ISPs
StarHub
XA/TMI
Transit is
Expensive
Gov’t
Content / Enterprise Companies
Business Case for SG Peering
Transit Pricing in Singapore
Commit
(in Mbps) $ per Mbps
1
10
100
1000
2nd highest transit prices
$625
$350
$210
$115
SG Peering with FastE
The Cost of Peering in Singapore (100M peering model)
Assume SingTel Local Loop STM-1 or
100M ethernet into Equinix Singapore
Colocation costs for a 1/2 rack
FastE Port
Total Cost of Participation at IX
Effective Peering Bandwidth
75% * 94Mbps
Minimum peering cost of traffic exchange
------compared to -----Transit Price (at 100Mbps commit)
Peering Break Even Point
$9,000
$450
$750
$10,200
per month
per month
per month
per month
70.5 Mbps
$144.68 per Mbps
$210 per Mbps
49 Mbps
SG Peering w/FastE
SG Peering w/gigE
The Cost of Peering in Singapore (gigE peering model)
Assume SingTel Local Loop STM-12 into
Equinix Singapore
Colocation costs for a rack
gigE Port
$18,000
$1,000
$2,000
$21,000
Total Cost of Participation at IX
Effective Peering Bandwidth
Minimum peering cost of traffic exchange
------compared to -----Transit Price (at 1000Mbps commit)
Peering Break Even Point
75% * 940Mbps
per month
per month
per month
per month
705 Mbps
$29.79 per Mbps
$115 per Mbps
183 Mbps
SG Peering w/gigE
Australian Peering Ecosystem
Telstra
Optus
Tier 1 ISPs
Australian
Internet Region
OzeMail
(owned by SingTel)
(now MCI)
Connect.com /AAPT
(Owned by Telecom New Zealand)
Comindico
PowerTel
Pacific Internet
TPG
Tier 2 ISPs
iPrimus
UeComm
Internode
Content / Enterprise Companies
Australian Peering Ecosystem
• Only country to Regulate Peering
• Restrictive Peering – Comindico Story
• Local Loops Expensive in AU
– Retail STM-1 (155M) in Hong Kong $3000/mo
– Retail 4M Ethernet in Australia $3000/mo ternet
• Relatively low traffic volume
– 200Mbps traffic between T1 and T2
• “Content that transcends the language barrier”
disallowed
AU Interconnect Regions
SEA-ME-WE
Cable System
Australian-Japan
Cable System
APCN
Southern
Cross
Cable
Telstra
P
Optus
P
P
(SingTel)
P
P
Brisbane
Sydney
Melbourne
OziMail
(WorldCom)
Connect.com
(TelecomNZ)
Telstra
Optus
P
P
P
(SingTel)
P
Telstra
Optus
P
P
P
(SingTel)
Connect.com
(TelecomNZ)
P
P
OziMail
(WorldCom)
Telstra
Connect.com
(TelecomNZ)
Optus
P
P
P
(SingTel)
P
P
OziMail
(WorldCom)
Connect.com
(TelecomNZ)
Perth
Local Loops Expensive
Volume Billing
Grandma Story
OziMail
(WorldCom)
P
Business Case for AU Peering
Transit Pricing in Austraila
Most expensive
“End of the World”
Commit (in
Mbps)
$ per Mbps (USD)
1
$720
10
$410
100
$325
1000
$305
AU Peering with FastE
The Cost of Peering in Austraila (100M peering model)
Assume Local Loop 100M ethernet into
Equinix colo
Colocation costs for a 1/2 rack
FastE Port
Total Cost of Participation at IX
Effective Peering Bandwidth
75% * 94Mbps
Minimum peering cost of traffic exchange
------compared to -----Transit Price (at 100Mbps commit)
Peering Break Even Point
$2,500
$450
$750
$3,700
per month
per month
per month
per month
70.5 Mbps
$52.48 per Mbps
$325 per Mbps
11 Mbps
AU Peering with FastE
AU Peering with gigE
The Cost of Peering in Austraila (gigE peering model)
Assume Local Loop gigabit into Equinix
colo
Colocation costs for a rack
gigE Port
$7,500
$720
$2,500
$10,720
Total Cost of Participation at IX
Effective Peering Bandwidth
Minimum peering cost of traffic exchange
------compared to -----Transit Price (at 1000Mbps commit)
Peering Break Even Point
75% * 940Mbps
per month
per month
per month
per month
705 Mbps
$15.21 per Mbps
$305 per Mbps
35 Mbps
AU Peering with gigE
5 Reasons to expand into and
within Asia
1. For Incumbent Tier 1 ISPs to peer their
routes outside their home market.
2. To meet U.S. Tier 1 Peering Prerequisites.
3. Customers want them in Asia.
4. Global Marketing Benefits
5. Sell Transit into a high cost transit market.
Costs to expand to Asia…
Hong Kong
Colo w/port
$1,000
Commit Transit
Hong
Int'l
1Mbps
$185
Kong
Cap
10Mbps
$100
100Mbps
$80
Local Loop
1000Mbps
$50
$4,000
100M
HKG-SG
STM-1
STM-4
Tokyo
Commit Transit
1Mbps
$490
10Mbps
$150
100Mbps
$100
1000Mbps
$50
HKG-US
STM-1
STM-4
HKG-TYO
STM-1
STM-4
$16,000
$40,000
Tokyo
$10,000 per month
$25,000 per month
Colo w/port
Int'l
Cap
Int'l
Cap
Local Loop
$1,000
100M
Colo w/port
$5,000
TYO-US
STM-1
STM-4
TYO-SG
STM-1
STM-4
$10,000 per month
$25,000 per month
SG-US
STM-1
STM-4
Singapore
$25,000 per month
$62,000 per month
Int'l
Cap
Sydney
Source: Median seen by from Andy Lee (Equinix)
United States
Commit Transit
1Mbps
$100
10Mbps
$80
100Mbps
$55
1000Mbps
$45
Source: Dave Wales (Secured S
Mar-04
Southern Cross
Local Loop
?
100M
U.S.A.
per month
Colo w/port $1,000
Source: Estimates courtesy Ryan Sher (FLAG)
(Note: Not actual quotes but to be used for illustration purposes)
Sydney
Commit Transit
1Mbps
$250
10Mbps
$140
100Mbps
$100
$3,000
Local Loop
$1,000
100M
$25,000 per month
$40,000 per month
Int'l
Cap
Local Loop
$8,000
100M
$16,000 per month
$40,000 per month
SYDNEY-LA
STM-1
$85,000 per month
STM-4
?
per month
Source: Anonymous
Lesson #1 - Tier 1 ISPs Do Not
Want to Peer in their Internet
Region
• As described in the Foreign Tier 1 ISP
Dynamic
• Peering in Adjacent Internet Regions OK
• Peering in U.S.
– Also Get Cheap U.S. Transit
– Also Get Across U.S. to Europe
Lesson #2 – There are Several
Challenges Peering in Asia
1. Many Language Zones.
– Language
– Internet traffic
2.
3.
4.
5.
Asia is spread across timezones
Asia is spread across oceans
Local Loop Costs
Transit costs are highly variable and in some
cases highly discriminatory across Asia:
Commit Hong Kong Tokyo
Singapore Sydney
US
1Mbps
$185
$490
$625
$1,030
$100
10Mbps
$100
$150
$350
$500
$80
100Mbps
$80
$100
$200
$400
$90
1000Mbps
$50
$50
$50
Lesson #3 – Some Creative Peering
Deals
• “…Peering iff Transport provided to HK
where we will peer out-of-country…”
• Peering w/transit purchase common
• Peered traffic can not be announced back
in to country
• Can not peer with my customers
Lesson #4 - International Peering
Gotcha: “Tromboning” Traffic
through the U.S.
1 AS Hop
Across Ocean
Beats
2 AS Hops
Across Town
K0
K0
T
$
KR-ISP
Pacific
Ocean
P
JP-ISP
JP-ISP
Lesson #5 - Local Presence
Required
•
•
•
•
•
Right Person
Right Time
Manage Time Zone Diff
Manage Peering Socializing
Like old England “Intermediary”
Source: Nigel Titley (FLAG)
And Erasmus Ng (T-Systems)
Lesson #6 - Separation of
International and Domestic Peering
• New Zealand – Separate pipes for Transit
& Domestic Traffic
• Transpacific VERY expensive
• 80% traffic to/from U.S.
• In Japan & Australia as well?
Source: Joe Abley (ISC)
Lesson #7 – “Content that
Transcends the Language Barrier”
• Hosted content not allowed in many parts
of Asia
• Hosted overseas
• Large volume of traffic
• Affects peering and Int’l BW planning
Lesson #8 – No True Regional
Content in Asia
• Like South America
• Few Asian countries host regional content
• Contiguous language zones:
– Hong Kong, Taiwan, China
• Mostly, local eyeballs want local content
• Japan: 80% traffic stays in Japan
• Singapore: 80% traffic leaves Singapore
Lesson #9 – Content Peering in
Asia Works
• Microsoft – 100M XP Updates
– Only delivered over Peering links
– Otherwise, overseas transit
– Increases your 95th percentile billables
• Yahoo!
– Motivated first by best customer experience
– Deployed content locally
– Peering broadly
Summary
•
•
•
•
•
•
Early Research
International Peering Ecosystem
Internet Regions
Foreign Tier 1 ISP Peering Dynamic
Capture Peering Coordinator Data
Asia Vibrant and Leap Frogging U.S. in some
ways
• White Paper Available: Send e-mail to
wbn@equinix.com