Presentation for the International Telecommunication Union (ITU)
Overview of recent changes in the international
IP interconnection ecosystem
21 May 2012
Michael Kende
ITU Regional Seminar on Costs and Tariffs for Member Countries of the
Regional Group for Asia and Oceania (SG3RG-AO)
5086-202 | Commercial in confidence
Bali, Indonesia, 28-30 May 2012
2
Introduction
 Last year marked the 15th anniversary of the commercialization of the Internet
backbone. This paper focuses on two underlying trends since then:
 the Internet has globalized, first from the US to the rest of the developed world, and
then beyond to emerging markets
 Internet traffic has increased by many orders of magnitude, based both on the
increased numbers of users as well as the increasing amount of premium content
including video
 The paper describes how interconnection arrangements have evolved in response to
these trends:
 Internet exchange points (IXPs) have helped to localize traffic and increase the
efficiency of the Internet
 countries with successful IXPs have transitioned from accessing Internet traffic as a
‘spoke’ to becoming a ‘hub’ for traffic in other countries
 we highlight successful case studies and policies that countries can adopt to become
a hub, and the impact for countries without those policies
5086-202 | Commercial in confidence
3
The commercial Internet is relatively young
 The National Science Foundation Network (NSFNET) backbone service was decommissioned in favor of the
commercial Internet on April 15, 1995:
 the NSFNET was used by regional networks to exchange traffic
 in its place, four network access points (NAPs) across the country were designated for traffic exchange
 interconnection was not regulated – in its place commercial arrangements known as peering and transit were
negotiated between different providers
ISP 1
Backbone
Network A
Backbone
Network B
Settlementfree transit,
paid transit,
or peering
 The Internet was much different on that day:
 Netscape had just been introduced
 the NSFNET backbone was just 45Mbit/s
 the Internet was very US-centric
5086-202 | Commercial in confidence
ISP 2
4
In 1995 the Internet was very US-centric for a number of reasons
 The US was the historical home of the Internet:
 70% of Internet users were in the US in 1995, along with much of the content
 up to 60% of European traffic routed through the US
 Much of the European traffic was tromboning through the US back to Europe:
 all European ISPs had to connect to the US to access users and content anyway
 the lack of liberalization in Europe made national and inter-European links very expensive for
direct connections. For instance, in 1998 leased lines prices for a 2 Mbit/s link were

London – Paris:
$38,000 / month

London – Virginia: $30,000 / month
 In addition to tromboning, the European traffic to the US had two sources:
 accessing content (including European content that was hosted in the US)
 natural traffic flows between the US and Europe (e.g. email)
5086-202 | Commercial in confidence
5
The early Internet was a hierarchy with Internet backbones at the top
Representation of relationships between Internet players in 1995
All traffic passes through the
backbone providers, which carry
the traffic between the ISPs
Backbone
1
Content provider
and aggregator
End user
ISP 2
ISP 1
NAP
Backbone
3
ISP 4
Backbone
2
ISP 3
Internet access
Transit
Peering
5086-202 | Commercial in confidence
Source: Analysys Mason
6
The Internet began to evolve quickly [1/2]
Internet traffic trends by service
 Usage has changed significantly over the
past ten years:
60,000
 traffic used to be relatively static – text,
email, file-sharing etc.
 Two trends are particularly relevant with
respect to international traffic flows:
 applications such as voice over Internet
Protocol (VoIP) are very sensitive to
latency, but relatively low bandwidth
 applications such as video streaming are
high bandwidth, but less sensitive to latency
(with buffering)
5086-202 | Commercial in confidence
Source: Cisco
PB per month
 dynamic multimedia traffic now dominates
with a significant amount of video and
gaming
50,000
40,000
30,000
20,000
10,000
0
2008 2009 2010 2011 2012 2013 2014 2015
File sharing
Web, email, and data
Internet video
Video calling
Online gaming
Voice over IP (VoIP)
7
The Internet began to evolve quickly [2/2]
Evolution of Internet users by region
100%
 in 2000 Internet usage was heavily
concentrated in developed countries, with a
far higher percentage coming from the US
and Canada then reflected in the population
split
90%
 this reflected the greater availability of PCs
and fixed access in developed countries
50%
Population split
40%
30%
20%
10%
5086-202 | Commercial in confidence
Source: ITU
Asia
2010
2009
2008
2007
2006
2005
Europe
Africa
2010 (pop.)
US & Canada
Latin America
2004
0%
2003
 this results from mobile access, as well as
greater availability of shared access
60%
2002
 usage in Asia and Africa now more closely
reflects the population split
70%
2000
 The usage trend has shifted significantly in
the past ten years:
80%
2001
 Usage began to better reflect population
distribution:
8
The historical architecture was not sustainable
 The US-centric approach imposed increasing costs:
 as developed countries liberalized, the cost of national and regional links fell relative to the cost
of links to the US
 the latency of access through the US became increasingly noticeable
 The Internet began to outgrow the NAPs:
 traffic was exchanged via public peering, involving a shared switch that soon congested as traffic
volumes began to multiply
 the owners of the NAPs included operators who were not perceived as neutral in selling access
 There were three changes in the architecture:
 interconnection evolved from NAPs to IXPs
 the IXPs began to develop outside the US
 some countries sought a policy response to pricing (e.g. ICAIS)
5086-202 | Commercial in confidence
9
Interconnection migrated from NAPs to IXPs
 IXPs emerged in place of NAPs:
 a wide variety of activities are possible, including direct peering and the sale of transit
 these are large data centers open to all parties including ISPs and content providers
 IXPs address the issues raised by NAPs:
 IXPs are neutral, and many are non-profit associations owned and operated by their members,
housed within larger commercial data centers (such as owned by Equinix)
 within the data center, large providers use direct cross-connects to engage in private peering to
avoid congestion
 IXPs also act to flatten the hierarchy of the Internet:
 the data center hosts content providers and ISPs who can use the IXP to peer directly with one
another
 the members of the IXP can also purchase domestic or international transit services within the
data center
5086-202 | Commercial in confidence
10
Content also began to migrate to IXPs
 High-bandwidth content tends to be ‘portable’:
Traffic shares by application
 according to Cisco, over 96% of content consists
of video, file sharing, and web pages
 unlike voice, such content can be moved from the
origination point
 New technology such as caches allow content to be
distributed:
 entire websites or popular pages can be stored
closer to end users
 this lowers the bandwidth costs while also
improving quality of service
 content distribution networks (CDNs) such as
Akamai will manage content on behalf of providers
 IXPs provide a natural location to store content
where it can be efficiently accessed by multiple ISPs
5086-202 | Commercial in confidence
Source: Cisco
17%
34%
45%
File sharing
Internet video
Web, email, and data
Video calling
Online gaming
Voice over IP (VoIP)
Other
11
As a result of IXPs, the backbone is far less hierarchical today
Representation of relationships between Internet players today
Backbone
1
End-user
ISP 2
Content provider
and aggregator
Backbone
3
Backbone
2
IXP
IXP
ISP
ISP 4
ISP 3
5086-202 | Commercial in confidence
Internet access
Transit
Peering
Secondary peering
Partial transit
Paid peering
Source: Analysys Mason
12
There have been three phases of globalization
Number of IXPs by region
 US-Centric phase, for historical reasons
starting with the commercialization of the
Internet
400
350
 OECD-Centric, focused on developed
countries in Europe and Asia:
5086-202 | Commercial in confidence
Source: Packet Clearing House, Analysys Mason
150
100
50
North America
Europe
Latin America
Africa
Asia
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
0
2001
 in Africa only two countries had IXPs before
2002, which has risen to 20 by the end of
2010
200
2000
 Rest of World (ROW) centric, focused on
emerging markets:
250
1999
 founded in 1994, it now connects more than
390 members, who own and operate the
exchange as a non-profit
Total number of IXPs
 the Amsterdam Internet Exchange (AMSIX) is a good example of a successful IXP.
300
13
The OECD-Centric phase has drastically reduced the reliance on the US in
Europe and Asia
U.S. & Canada
Europe
Asia
Latin America
Africa
U.S. & Canada
Europe
Asia
Latin America
2011
2010
2009
0%
2008
0%
2007
10%
2006
10%
2005
20%
2004
20%
2003
30%
2002
30%
1999
40%
2011
40%
2010
50%
2009
50%
2008
60%
2007
60%
2006
70%
2005
70%
2004
80%
2003
80%
2002
90%
2001
90%
2000
100%
1999
100%
2001
International Internet bandwidth
from Asian countries, by region
2000
International Internet bandwidth from
European countries, by region
Africa
 The early IXPs in Europe were setup before
1999, significantly reducing tromboning
before this dataset starts
 Asia had a later start in localizing traffic, but
has now reduced reliance on the US for
Internet bandwidth by half
 Nonetheless, the reliance on the US has now
fallen to about 15% of bandwidth, while Asia
is increasingly a destination
 Intra-Asian traffic has increased as more
traffic localizes, while Europe is increasingly a
destination for traffic exchange
5086-202 | Commercial in confidence
Source: TeleGeography Global Internet Bandwidth
14
The ROW phase has not had as great an impact on international
bandwidth to date
U.S. & Canada
Europe
Asia
Africa
 African reliance on the US for Internet
bandwidth has reduced drastically
 However, the reliance has shifted to Europe,
which demonstrates both the adaptability of
the Internet structure but also the need for
continued localization in Africa
5086-202 | Commercial in confidence
Source: TeleGeography Global Internet Bandwidth
U.S. & Canada
Europe
Asia
Latin America
2011
2010
2009
0%
2008
0%
2007
10%
2006
10%
2005
20%
2004
20%
2003
30%
2002
30%
1999
40%
2011
40%
2010
50%
2009
50%
2008
60%
2007
60%
2006
70%
2005
70%
2004
80%
2003
80%
2002
90%
2001
90%
2000
100%
1999
100%
2001
International Internet bandwidth
from Latin American countries, by region
2000
International Internet bandwidth
from African countries, by region
Africa
 Latin American reliance on the US has begun
to fall, but is still above 80%
 Unlike the African situation, intra-regional
traffic is growing, but still demonstrates the
need for continued localization
15
A successful IXP can evolve from acting as a ‘spoke’ to a ‘hub’
IXP evolution
No IXP
IXP ‘spoke’
Content
provider
Content
provider
ISP 4
ISP 4
IXP ‘hub’
ISP 4
IXP
IXP
IXP
International
International
International
Country A
Country A
ISP 2
ISP 1
ISP 2
ISP 1
Country A
ISP 1
Content
provider
IXP
ISP 3
5086-202 | Commercial in confidence
ISP 3
An IXP ‘spoke’ can quickly
eliminate tromboning, but still
accesses content and
international transit abroad
IXP
ISP 2
ISP 3
An IXP ‘hub’ acts as a magnet
for content and international
backbones to sell services in the
country and greater region
16
The evolution to IXP ‘hub’ may require broad policy changes, with
corresponding benefits
International Internet bandwidth per capita
 Creation of an IXP:
 there are many examples of ISPs acting together
to create an IXP, with little or no regulatory
intervention
600
 the ability of that IXP to attract FDI and become a
hub largely lies outside of the control of the IXP
and likely even the regulator
400
 first, foreign providers will look at the general
business environment
 second, providers look at sector policies including
international liberalization, and licensing
requirements and obligations
 The impact of AMS-IX in the Netherlands is
noticeable in terms of bandwidth per capita (see
right)
5086-202 | Commercial in confidence
Source: TeleGeography Global Internet Bandwidth
kbit/s
300
200
100
0
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
 Broadly speaking, two sets of factors impact the
evolution of the IXP into a hub:
500
Australia
Canada
France
UK
USA
Germany
Singapore
Netherlands
17
A recent study for the Internet Society shows the benefits of an IXP in
developing countries
 An association of ISPs in Kenya (TESPOK)
setup the Kenya Internet Exchange Point
(KIXP) in Nairobi in early 2000:
 the immediate benefit was to eliminate
reliance on satellite for tromboning,
reducing latency and cost significantly
 the incumbent challenged the IXP with the
regulator, but was ultimately denied
 The growth in the IXP has been significant:
 there are now 28 members peering at KIXP,
including all major operators, a government
network, and several DNS servers
 KIXP is one of the fastest growing IXPs in
the world, peaking at up to 1Gbit/s traffic
recently
5086-202 | Commercial in confidence
 The IXP of Nigeria (IXPN) was established in
2006 in Lagos:
 the stated objective was reducing reliance
on international transit for exchanging local
traffic between members
 The growth has been significant:
 there are now more then 30 members
exchanging traffic in IXPN
 traffic peaks at over 300Mbit/s
 Growth has been slower than in Kenya:
 not all major operators are members and
traffic levels are lower
 the key reason appears to be that the high
cost of national backhaul restricts the
benefits of avoiding international
tromboning
Source: Assessment of the impact of Internet Exchange Points – empirical study of Kenya and
Nigeria, April 2012
18
The impact of having local content is particularly significant
Traffic exchanged at KIXP
 Google installed a Google Global Cache (GGC) in
Kenya:
1200
 the cache was initially provided to one operator in
Nairobi, under the condition that the contents
would be made available to all members of KIXP
1000
 the cache retains static content after it has been
downloaded in Kenya, such as YouTube videos
 as shown on the right, traffic through KIXP spiked
immediately
 this increase mostly reflects users’ increased
usage of Google content, notably an increased
willingness to stream YouTube videos based on
lower latency of access
 mobile operators who charged by the MB
experienced increased revenues along with lower
costs
5086-202 | Commercial in confidence
Source: KIXP, Analysys Mason
Mbit/s
 The cache was installed in April 2011:
800
600
400
200
0
Mar-03
Dec-05
Sep-08
Jun-11
19
The benefits of the IXPs are broadly felt in Kenya and Nigeria
Benefit
KIXP
IXPN
Summary
Latency
Reduced from 200–600ms
to 2–10ms
Reduced from 200–400ms to
2–10ms
Noticeable increase in
performance for end users
Local traffic
exchange
1Gbit/s peak
300Mbit/s peak
Savings on international
transit of over $1 million per
annum in each country
Content
Google network present
locally, along with rehoming
of domestic content
Same as in Kenya
Increase in usage and
corresponding revenues for
mobile data traffic
e-government
Kenya Revenue Authority
gathers taxes online
Usage by education and
research networks
Social benefits from egovernment access to IXPs
Other benefits
An increasing amount of
regional traffic exchanged at
KIXP
Financial platforms hosted
locally
Further economic benefits
resulting from IXPs
5086-202 | Commercial in confidence
20
The impact of Internet trends only highlights the need for Internet hubs in
emerging markets
 The trend is towards an increasing reliance
on Internet access and content:
 increasingly personal and business content
and applications are moving into the cloud
 increasingly these are designed for access
by mobile devices
 Policy solutions should focus on creating local
hubs, rather than simply lowering the cost of
acting as a spoke:
 as the bandwidth of access and content
increases, international access costs will
continue to rise
 in addition, access to local or regional hubs
will reduce latency and improve the
resiliency of access
5086-202 | Commercial in confidence
Source: Analysys Mason
Summary of Internet trends
21
Contact details
Michael Kende
michael.kende@analysysmason.com
Cambridge
Tel: +44 (0)845 600 5244
Fax: +44 (0)1223 460866
cambridge@analysysmason.com
Manchester
Tel: +44 (0)845 600 5244
Fax: +44 (0)161 877 7810
manchester@analysysmason.com
Analysys Mason Limited
Via Pietro Verri 6, First Floor
20121 Milan, Italy
Dubai
Tel: +971 (0)4 446 7473
Fax: +971 (0)4 446 9827
dubai@analysysmason.com
Milan
Tel: +39 02 76 31 88 34
Fax: +39 02 36 50 45 50
milan@analysysmason.com
Dublin
Tel: +353 (0)1 602 4755
Fax: +353 (0)1 602 4777
dublin@analysysmason.com
New Delhi
Tel: +91 11 4700 3100
Fax: +91 11 4700 3102
newdelhi@analysysmason.com
Edinburgh
Tel: +44 (0)845 600 5244
Fax: +44 (0)131 443 9944
edinburgh@analysysmason.com
Paris
Tel: +33 (0)1 72 71 96 96
Fax: +33 (0)1 72 71 96 97
paris@analysysmason.com
London
Tel: +44 (0)845 600 5244
Fax: +44 (0)20 7395 9001
london@analysysmason.com
Singapore
Tel: +65 6493 6038
Fax: +65 6720 6038
singapore@analysysmason.com
Madrid
Tel: +34 91 399 5016
Fax: +34 91 451 8071
madrid@analysysmason.com
Washington DC
Tel: +1 202 331 3080
Fax: +1 202 331 3083
washingtondc@analysysmason.com
Partner
5086-202 | Commercial in confidence