William B. Norton
Co-Founder & Chief Technical Liaison
Equinix, Inc.

•
Focused studies on ISP Peering
1. Identify Relevant Internet Operations Topic
2. Speak with prominent Peering Coordinators
3. Write/Evolve Draft White Paper
4. Walk Peering Coordinators through paper
5. Goto Step 3
^Internet Service Providers and Peering
Summary Findings of Peering Research….

Many ISPs pulling out of MAE-East
Why? “..Financial Reasons..”
Apply “A Business Case for Peering” WP
Result: “Do ATM-Based Internet Exchange
Points Make Sense Anymore” white paper v1.5
~200 ISPs have seen this research paper, ~50 Walk-throughs
Data Points…

• Variable based on usage ($/Mbps)
• Range: $350/Mbps down to $35/Mbps
• Greater volume = Lower unit price
Observation: Over the last 4 years, Transit Prices have dropped like a rock.
$1200/Mbps (AboveNet, 1998) now costs $120/Mbps (AT&T, 2002)

Peering _ Costs
 t
 p
 r

R where : t

Monthly _ Transport _ Costs _ Into _ IX p

Monthly _ IX _ Port _ Fees r

Monthly _ Rack _ Fees
R

Router _( Equipment _ Costs )
Observation: Transport Prices have dropped like a rock.
Observation: New Router prices have dropped like a rock.
Observation: Used Router Market is also very healthy (cheap).
Apply these to peering at ATM IX…

Ethernet-based Peering Costs…

When do each of these Peering Infrastructures make sense?

Peering Price per Mbps
Peering Breakeven Point
(Peering=Transit)
Transit Price per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)

Peering Price per Mbps
Definition: The Effective Peering Bandwidth is the maximum bandwidth available for peering, defined as the minimum of the available transport peering fabric.
Transit Price per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)

Peering Price per Mbps
Definition : The Minimum Cost of Traffic Exchange is the unit cost of traffic exchange when the
Transit Price per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)

Definition: The Effective Peering Range (EPR) is the range in which peering at an Internet Exchange makes sense (financially), measured as the range between the
Peering Breakeven Point
(Peering=Transit)
Effective Peering Bandwidth.
Transit Price per Mbps
Transit Cheaper-Peering Cheaper
Effective Peering Range
Min Cost of
Traffic Exchange
(in $/Mbps)
Amount of Traffic Exchanged (in Mbps)
Effective
Peering
Bandwidth (in Mbps)

• Scenario 1: Medium Scale Peering
– ATM-IX
– Ethernet-based IX
– Transit
• Scenario 2: Large Scale Peering
– ATM-IX
– Ethernet-based IX
– Transit
Medium Scale Peering

• Assume: OC-3 into IX for Peering
• Assume: Transit Commit of 500Mbps
• Data Point: $125/Mbps Transit Price
• Assume: Used Cisco 7200 w/OC-3+FastE
– Price=$9K*2, 36 month term=$500/month
– Used Juniper M20 on eBay $14
• Upgrade Peering when at 75% Avg. Util.
Price Survey for ATM Peering

WorldCom Local Loop OC-3 DC-MAE
WorldCom ATM OC-3 Port
Total Cost of Participation at MAE-East
Effective Peering Bandwidth
(Includes 20% ATM Cell Tax/75% Util)
Minimum cost of traffic exchange
Assumption: Transit Price
Peering Breakeven Point
75% * 120 Mbps
$3,000 per month
$8,000 per month
$11,000 per month
90 Mbps Max
$122.22 per Mbps
$125 per Mbps
88 Mbps
Price Survey for FastE Peering

Equipment: Cisco 7200 OC-3 / FastE
Local Loop OC-3 DC-EQ-ASH
1/2 rack + Fast Ethernet at EQ ASH
Total Cost of Participation at EQ-ASH
Effective Peering Bandwidth
(4% HDLC Framing irrelevent/75% Util)
Minimum cost of traffic exchange
Assumption: Transit Price
Peering Breakeven Point
75% * 94 Mbps
$500 per month
$2,500 per month
$2,500 per month
$5,500 per month
70 Mbps Max
$78 per Mbps
$125 per Mbps
44 Mbps
Ethernet-based IX
OC-3 (155 Mbps)
Router
Fast Ethernet (100 Mbps)
Eth

ATM/OC-3
FastE/OC-3
Effective Peering Range
Peering
Breakeven Point
Effective
Peering
Bandwidth
88 Mbps 90 Mbps
44 Mbps 70 Mbps
Shown Graphically…

ATM/
OC-3
ATM/OC-3 Peering Effective
88Mbps-90Mbps
Min. Peering Cost: $122/Mbps
Transit cheaper than Peering
Upgrade
Needed
Effective Peering Range
FastE/
OC-3
Transit cheaper than Peering
FastE/OC-3 Effective
Peering from
44Mbps-70Mbps
Min Peering Cost:$71/Mbps
Range:$125/Mbps  $71/Mbps
Upgrade
Needed
10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95
Effective Peering Range in Mbps
Summary…

ATM /OC-3 and FastE/OC-3 Peering vs. Transit
$2,500
$2,000
$1,500
$1,000
$500
$0
5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90
# Mbps
MAE-OC3
Transit
EQFastEOC3
Large Scale Peering…

• Assume: OC-12 into IX for Peering
• Assume: Transit Commit of ~900Mbps
• Data Point: $100/Mbps Transit Price
• Assume: New Juniper M5 w/OC-12+GigE
– Price=$54K, 36 month term=~$1500/month
– Used Juniper M40 on eBay $35
• Upgrade Peering when at 75% Avg. Util.
Price Survey for ATM Peering

Price WorldCom OC-12 DC-MAE-East
Price WorldCom OC-12 Port
$10,000 per month
$16,000 per month
Total Cost of Participation at MAE-East
Effective Peering Bandwidth
(Includes 20% ATM Cell Tax/75% Util)
Minimum price for traffic exchange
Assumption: Transit Price
Peering Breakeven Point
75% * 500 Mbps
$26,000 per month
375 Mbps Max
$69 per Mbps
$100 per Mbps
260 Mbps
Ethernet-Based Peering

Equipment: Juniper M5 OC-12 / GigE $1,500 per month
Local Loop OC-12 DC-EQ-ASH
1/2 rack + Gigabit Ethernet at EQ ASH
$5,000 per month
$5,000 per month
Total Cost of Participation at EQ-ASH
Effective Peering Bandwidth
(Includes 4% HDLC Framing/75% Util)
Minimum cost of traffic exchange
Assumption: Transit Price
Peering Breakeven Point
75% * 597 Mbps
$11,500 per month
448 Mbps Max
$26 per Mbps
$100 per Mbps
115 Mbps
OC-12 (622 Mbps)
Router
Cisco 7300 gigE Ethernet (1000 Mbps)
Eth

$600
$500
$400
$300
$200
$100
$0
ATM/OC-12 and GigE/OC-12 Peering vs. Transit
MAE-OC12
Transit
EQGigEOC12
# Mbps
Effective Peering Range…

ATM/OC-12
GigE/OC-12
Effective Peering Range
Peering
Breakeven Point
Effective
Peering
Bandwidth
260 Mbps
115 Mbps
375 Mbps
448 Mbps
Graphically …

ATM/
OC-3
FastE/OC-3
ATM/OC-12
GigE/OC-12
ATM/OC-3 Peering effective 88Mbps-90Mbps,
$125/Mbps-$122/Mbps
FastE/OC-3 Peering effective from 44Mbps-70Mbps,
$125/Mbps-$78/Mbps
ATM/OC-12 Peering effective from 260Mbps-375Mbps at
$100/Mbps-$69/Mbps
GigE/OC12 Peering effective from 115Mbps-448Mbps,
$100/Mbps-$26/Mbps
100 200 300 400 500 600 700 800 900
Effective Peering Range in Mbps
Effective Peering Range Gap…

ATM/
OC-3
EPR Gap
FastE/OC-3
ATM/OC-12
GigE/OC-12
100 200 300 400 500 600 700 800 900
Effective Peering Range in Mbps
Do ATM-based IXes make sense?

“Do ATM-based Internet Exchange Points
Make Sense Anymore?”
• Today’s Economy (Inexpensive Equipment,
Transit, Transport, Ethernet IXes) have conspired to make ATM-based IXes a low value proposition
– ATM-based IXes make sense for a very narrow range
– Ethernet-based Peering are always less expensive than
ATM-based Peering
• Transit plays a key role in cost-effective traffic exchange (Ethernet Peering+Transit more cost effective than ATM Peering)
Author’s Notes

•
Top 5 Reasons ISPs are Peering at ATM-IX
1.
Haven’t done the math
2.
Must peer with ATM Participant x not at
Ethernet-based IX – don’t care what it costs
3.
Contract Term yet to expire
4.
“To be a player…”
5.
Let sleeping dogs lie – changes in peering might trigger re-qualification request from peer.
Acknowledgements…

For this white paper I’d like to thank a few folks in particular for their review, insights, and comments on this paper:
Dorian Kim (NTT/Verio), Ingrid Erkman (ICG), Dave McGaugh (ELI),
Eric T. Bell (Time Warner Telecom), Chris Parker (StarNet),
Lane Patterson (Equinix), Jay Adelson (Equinix), Morgan Snyder (Equinix),
John Hardie (Equinix), David Diaz (BellSouth), Joe Wood (Accretive Networks),
Robert Seastrom (inter.net), Kevin Epperson (Level3), Petri Helenius (FICIX),
Scott Sheppard (BellSouth), Ralph Doncaster (iStop.com), Leo Bicknell (ufp.org),
Paul Vixie (vix.com), Ian Somerton and Dave Wodelet (Shaw/BigPipe),
Tony Hain (Cisco), Jeff S. Wheeler (five-elements.com), Cliff Hafen, Dory Liefer,
Shannon Lake (Omnivergent), Nenad Trifunovic (WorldCom), Andre Gironda (eBay),
Jeb Linton (EarthLink), Daniel Golding (SockEye), Peter Moyer (Juniper), and others that preferred no recognition for their contributions to this paper.
Questions…

“Do ATM-based Internet Exchange Points
Make Sense Anymore?”
Copies of this research paper available via e-mail to wbn@equinix.com
Other Resources Available…

• Gigabit Peering Forums
• Other White Papers document Peering
Practices
• Peering Contact Database

“Interconnection Strategies for ISPs”
“Internet Service Providers and Peering”
“A Business Case for Peering”
“The Art of Peering: The Peering Playbook”
“Do ATM-based Internet Exchange Points make sense anymore?”
“The Peering Simulation Game”
Freely available from the author: wbn@equinix.com

For Peering Coordinators Only
Toss in your Business Card &
Receive a copy of everyone’s Business Cards
Every 6 weeks (or so)
Managed as a community service.
E-mail to wbnorton@wbnorton.org
(Or give me your business card)