..
..
..
..
..
Kristofer J Carlson
PSC 813 Box 121
FPO AE 09620-0121
kristofercarlson@hotmail.com
INSS 510
.
.
Why are some
downloads fast and
others
.
. slow?
.
.
.
.
The elimination of the Last Mile Problem
reveals significant infrastructure
problems
.
.
I. Abstract
The transition from dialup to broadband makes a dramatic difference in the way users
experience the Internet, but this experience is not equal across all sites and all times. This
paper describes the four main Internet bottlenecks---First Mile, Peering, Backbone and Last
Mile---and the ways in which these interact to create the varied user experience. This paper
then extrapolates from current trends and anticipated services to suggest the main problem of
the Internet is one of under-capacity, and that the move to broadband services exposes and
exacerbates the problem.
II. Table of Contents
I.
II.
III.
IV.
V.
VI.
Abstract
Table of Contents
Introduction: From Dial-up to DSL
Bottlenecks
Conclusion
Works Cited & Annotated
Bibliography
2
2
3
3
5
6
III. From Dial-up to DSL
“The Internet is broken. It works because there’s a lot of it, not because it’s
good.” Adam Joffe, Director of Technical Operations, Sony Online Entertainment. (Scalet)
Several months ago I made the switch from my dial-up connection to DSL. It changed
everything for me. The Last Mile Problem1 was gone, and the Internet became a richer and
more engrossing place. I soon discovered the rich assortment of content available to someone
with a fast connection. My wife and I probably doubled the time we spent browsing. We also
began to play online games for the first time.
With ADSL some sites loaded quickly while others didn’t. This wildly variable Internet
performance intrigued me and I began to search for answers. The intuitive response was that
some sites had fast servers while others didn’t. Intuition was partly correct, but mostly
wrong.
The ADSL connection’s rated speed is 192/832 (upload/download). The DSL Reports Speed
Test (http://www.dslreports.com/stest/0) reported the actual connection speed is 62/593 Kbps.
Users should expect no more than 85% of rated connection, but 32%/71% was too low.
Perhaps this was another case of a poor performing site? Speed tests with other sites
generated even lower speeds.
1
The Last Mile Problem is the limited bandwidth of the dial-up Internet connection. Since most users
connect through a dial-up modem, their bandwidth is limited to approximately 47.6 Kbps.
2
Why the different rates in different speed tests? Perhaps the problem was not with the web
sites themselves, but with the networks that delivered their content. As the Internet
congestion following the 9/11 attacks showed, the networks can be unreliable. Overall, this
suggested the Internet has numerous bottlenecks besides the one attached to the end user’s
phone line.
IV. Bottlenecks
A bottleneck is a stage in a process that causes the entire process to slow down. (Bottleneck)
Bottlenecks are limiting factors. Just as a chain is only as strong as the weakest link, a
process will move only as fast as it’s slowest stage. The Internet is full of bottlenecks, and
alleviating one bottleneck does not necessarily speed up the entire process.
The Internet has four main bottlenecks, each of which contributes to the overall problem.
Which bottleneck is the main constraint depends on a number of factors. For the user, the
limiting factor tends to be the speed of his/her connection to the Internet, which is most often--at the time this is written---the 56K modem.
We’ll get to the user’s problem shortly. First we must look at the Internet from the point of
view of the content provider. The First Mile problem involves the bandwidth capacity of the
content provider’s connection to the Internet. (Internet Bottlenecks, 4) This bandwidth
capacity involves the speed of the website server, (which should include any backend
processing,) and the size of the pipe feeding data to into the Internet.
A content provider must balance the desire for increased bandwidth with the cost of content
delivery. Too little bandwidth and the website performance suffers; too much bandwidth and
the cost of content delivery outweigh the benefits. It appears that content providers purchase
additional capacity only when their existing connections are saturated. (Internet
Measurements, 4) This makes sense in a commodity business, but because the First Mile
connections are saturated with normal traffic, they are unable to respond to sudden peaks in
demand. (Internet Bottlenecks, 4)
The Internet is a network of networks. (Internet Infrastructure, 2) The content provider
delivers content on demand into whatever network their server is connected to. That network
is connected to other networks. These networks agree to share each other’s capacity. This
sharing is called peering, and is the second main bottleneck. “Peering occurs when two
[networks] exchange traffic bound for each other’s networks over a direct link, without fees.”
(Impact, 1) Peering is generally through public points set up by the industry, or through
private arrangements between roughly comparable networks. (Impact, 1) Peering can also
occur through the vendor-neutral exchange facility; members share their networks equally,
bandwidth is controlled, and packets are routed around network congestion. (Impact, 4)
Another arrangement, called “Transit,” involves a lesser network buying access into a larger
one. (Impact, 1) Because transit traffic helps pay for the network, it is often given priority
over non-paying, peered traffic. (Impact, 6)
Public peering points are free, and are therefore highly congested. (Impact, 1) Private peering
points are necessary to meet Service Level Agreements with clients, but are costly to set up
and generate no revenue. (Internet Bottlenecks, 5) An Akamai white paper (written to sell
their Content Delivery Model) makes the claim that peering links are saturated, because
overcapacity is costly. (Internet Bottlenecks, 5) A more independent analysis in Network
Magazine says that peering points are added when the capacity is between 40% and 50%.
2
(Impact, 3) Using VisualRoute on a variety of sites has shown latency and packet loss occurs
more often at nodes inside a given network than at the peering points between networks2.
Each Tier 1 network, or ISP3, operates a web of connections commonly called a backbone.
The capacity of these backbones is the third main bottleneck. (Internet Bottlenecks, 6) As the
demand for content has grown, and as the average bandwidth of the users has climbed, so too
has the capacity of the Internet backbones. (Internet Measurements, 1,3; Internet
Measurement, 2) This capacity is not measured by the amount of fiber in the ground, but by
the capacity of the routers forming the network nodes. (Internet Bottlenecks, 6)
Routers receive packets, determine their address, look up the route to that address based on
the map of it’s network connections, and forward the packet to the next destination. If a
router is slow or if a node is saturated, the packet must wait its turn in a queue. (Internet
Measurements, 6) This is one source of latency. A packet can lose its turn in line to higher
priority packets. This is one source of lost packets. While networks may have the ability to
route around underperforming nodes, the saturated node limits the overall bandwidth of the
backbone.
A larger problem is that the growth of optical technologies is not keeping pace with the
growth of the Internet. The Internet is doubling every 6 months, but the capacity carried by
an optical fiber is doubling only every 36 months. (Challenge, 14) Although providers may
be able to lay new fiber as necessary, (Internet Bottlenecks, 6) and upgrade existing lowspeed circuits, the crossover point between the physical capacity and the required capacity
occurred way back in 1997. These trends are illustrated and extrapolated in Figure1.
Figure 1: Internet Trends (Challenge, 14)
While the quality of the backbones have increased, they are still not good enough to provide
the Quality of Service (QOS) required by telephony services, to say nothing of video. Packet
2
This is admittedly unscientific and anecdotal, but does lend credence to the information in the Network
Magazine article.
3 “Tier 1 ISPs are those that have access to the global Internet Routing Table and don’t buy network
capacity from other providers… [Tier 2 ISPs have] a smaller national presence than a Tier 1 ISP and
may lease part or all of its network from a Tier 1…A Tier 3 [ISP] is typically a regional provider with
no national backbone.” (Impact, 1)
3
loss has been cut in half, but must still drop another two orders of magnitude before being
considered adequate for most audio codecs4. (Internet Measurements, 5) In addition, video
will not tolerate a network delay variation of more than 1-2 milliseconds, (Challenge, 7,) a
figure that is beyond the capability of current equipment. Network providers typically double
up on their equipment just to maintain acceptable levels of service in the event of an outage.
(Challenge, 9-10) Despite this, the network cannot provide the standard “five nines”
reliability of circuit switched networks. (Challenge, 10) This suggests the capacity of the
backbones must increase dramatically if new audio and video on demand services are to
succeed5.
Let’s compute demand for long-haul Internet capacity. Consider an example
application of video-on-demand on the Internet, the personalized Cooking
channel. Instead of watching the broadcast version of the generic Cooking
Channel program on TV, each user will be able to put together his own “menu”
of recipes to learn, catering to his or her own specific taste or entertainment
plans. How much Internet capacity will the Cooking Channel consume? Viewer
monitoring performed by Nielsen Media Research shows that cooking shows rat
about 1/10 of a Nielsen rating point, or 100,000 simultaneous viewers. At a
conservatively low encoding rate of 300Kbps and 100,000 unique simultaneous
streams, the personalized Cooking Channel will consume 30 Gbps.
Now consider WorldCom’s UUNET backbone which carries approximately half
of all Internet transit traffic. Its network is comprised of multiple hubs of
varying capacity, ranging from T1 (1.544 Mbps) to OC48 (@,450 Mbps or 2.4
Gbps). UUNET’s Canadian and U.S. networks are linked with more than 3,600
Mbps (or 3.6 Gbps) of aggregate bandwidth between multiple metropolitan
areas. However, the equivalent of 32 cooking channels alone could overload
the capacity of the world’s largest backbone to transfer content from the U.S. to
Canada! (Internet Backbone, 6)
The Last Mile problem is the one most familiar to home users. It is the speed of his/her
connection to the Internet. (Internet Bottlenecks, 6) While the average (or bottleneck)
bandwidth of the Internet has increased from 40 kbps in 1998 to 70kbps in 2000, the
predominant Internet connection remains the 56K modem. (Internet Measurements, 3) “The
Internet is only as fast as its slowest link,” a fact that hides the first three bottlenecks from the
average user. (Internet Bottleneck, 6)
What happens when people dump the dialup and move to some form of broadband
connection? Figures derived from the @Home ISP suggest that broadband users don’t just
access the Internet more quickly; they also demand more of it. In 2000 @Home had 1 million
subscribers, just 0.3% of the total, (377 million,) but generated 5% of the Internet traffic.
(Internet Bottleneck, 7) To put it another way, broadband users require 16 times the network
capacity of dial-up users. Obviously, this capacity cannot be added overnight; from what we
have seen before, the router technologies have barely managed to keep pace with current
requirements.
The transition from dial-up to broadband will demand 16 times the capacity of the current
Internet. To provide acceptable QOS for telephony will require an additional 1.5 times the
capacity. Multiplied together, the Internet capacity needs to grow by forty times just to
provide acceptable levels of service to existing users as broadband and telephony services
4
Audio codecs require packet loss to be less than 2%; in January 2000 average packet loss was 5%.
(Internet Measurements, 5)
5 These services are described as “real time transport [to support] audio and video streams;” “pervasive
networking;” “Internet telephone and, slightly further out, Internet television.” (Brief, 18)
4
expand. Now consider that Internet traffic is growing by a factor of four each year and you
begin to understand the scope of the problem. (Challenge, 4)
The current system of providing capacity seems to be stretched to the limit. The Internet is a
commodity business now, (Brief History, 17) with low profit margins. (Internet Measurement,
2) The ability of any network to increase its capacity by several orders of magnitude is
limited both by technology (Challenge, 8) and by the required capital investment. In some
cases “service providers [are] having to replace their whole infrastructures every year” just to
keep up with existing demand. (Challenge, 5) The increased traffic increases revenue, but
not profitability. (Challenge, 7)
V. Conclusion
Moving from dial-up to broadband exposed a number of weaknesses of the current Internet
structure and suggest the architecture is not as robust as it should be. The current bandwidth
per user has risen as the end user’s Last Mile connection speed has increased, but the
bandwidth required by broadband users outstrips the ability of the ISPs and content providers
to add capacity.
Perhaps the slow rollout of broadband in the U.S. has been a blessing in disguise. Perhaps the
dial-up connection, by effectively rate-limiting the speed of the Internet, has kept the entire
system from breaking down. This suggests that the Adam Joffe’s opening quote is wrong;
that yes, the Internet is broken; but no, the Internet is not large enough. Not by a long shot.
5
VI. Works Cited & Annotated Bibliography
A Brief History of the Internet. Barry M. Leiner, Vinton G. Cerf, David D. Clark, Robert E.
Kahn, Leonard Kleinrock, Daniel C. Lynch, Jon Postel, Larry G. Roberts, Stephen Wolff.
Internet Society. 21 April 2002, http://www.isoc.org/internet/history/brief.shtml
A brief history of the Internet, (infrastructure, protocol development and implementation,
and community,) written by a few of the people involved from the beginning.
Bottleneck. WhatIs?com. 27 April 2002
<http://searchnetworking.techtarget.com/sDefinition/0,,sid7_gci540507,00.html>
Definitions for thousands of the most current IT-related words.
Burd, Stephen D. Systems Architecture. 3rd ed. Canada: Course Technology, Thomson
Learning, 2001.
An introduction to computer hardware, software, and networking concepts.
The Challenge of Building Efficient Networks: Issues, Opportunities, and New Concepts.
Caspian Networks. © 2000 Caspian Networks., 12 September 2001
<http://www.caspiannetworks.com/tech/whitepapers/EfficientNetworks.pdf>
This white paper describes the growth of the Internet and the problems increasing
capacity creates for the ISPs. It suggest that the current router design is wrong, both in
hardware and software.
Computer Networking: Global Infrastructure for the 21st Century. Vinton G. Cerf. University
of Washington, Department of Computer Science and Engineering. 12 April 2002,
http://www.cs.Washington.edu/homes/lazowska/cra/networks.html
A history of the Internet from its early subsidized origins to today’s commercialized
commodity infrastructure.
DRDoS, Distributed Reflection Denial of Service: Description and analysis of a potent,
increasingly prevalent, and worrisome Internet attack. Steve Gibson, Gibson Research
Corporation. 26 April 2002, <http://grc.com/dos/drdos.htm>.
Focused on a new type of Denial of Service attack, it goes into detail about IP routing and
the TCP “three-way handshake” used to establish a virtual connection.
The Impact of Peering on ISP Performance: What’s Best for You?. Doug Allen, Network
Magazine, 5 November, 2001. 21 April 2002,
<http://www.networkmagazine.com/article/printableArticle?doc_id=NMG20011102S0006>
Describes what peering is, how it differs from transit relationships, the costs involved in
transit relationships, and the questions business should have answered before purchasing
access to an Internet backbone.
How Internet Infrastructure Works. Jeff Tyson. Marshall Brain’s How Stuff Works. 12 April
2002 <http://www.hotstuffworks.com/internet-infrastructure.htm/printable>
Basic information involving Internet infrastructure, plus links to more detailed
information and organizations.
6
Internet Bottlenecks: the Case for Edge Delivery Services. Akamai White Paper. ©2000
Akamai Technologies, Inc., 12 April 2002
<http://www.akamai.com/en/resources/pdf/BottlenecksWhitepaper.pdf>
Describes the four main internet bottlenecks and how edge delivery networks can
alleviate the first three problems, providing a scalable solution to content delivery.
Internet measurement and data analysis: topology, workload, performance and routing
statistics. kc claffy. University of California, San Diego, Cooperative Association for Internet
Data Analysis (CAIDA) U.S.A. (NAE ’99 workshop), 21 April 2002,
<http://www.daida.org/outreach/papers/1999/Nae/Nae.html>
Describes the collection and analysis of internet traffic data. Mentions problems with
bandwidth, peering, packet sizes, and changing patterns of traffic flows. Describes
difficulties with measurements to support ISP Service Level Agreements. Measures
latency, packet loss, and traceroutes.
The Internet’s Physical Layer. Russ Haynal. 12 April 2002,
<http://navigators.com/sessphys.html>
Basic information on the internet hardware layer.
Macroscopic Internet topology and performance measurements from the DNS root name
servers. Marina Fomenkov, kc claffy, Bradley Huffaker and David Moore.
CAIDA/SDSC/UCSD. 21 April 2002,
<http://www.caida.org/outreach/papers/2001/Rssac2001a./rssac.html>
Describes the topology and latency of the DNS system. Describes connectivity issues
and refers to problems with the flat structure of the domain name space and its impact on
latency.
Runn Haynal’s ISP Page. Russ Haynal. 12 April 2002, <http://navigators.com/isp.html>
Lists the major Internet backbone providers by market share and connectivity.
Scalet, Sarah D. “The Perfect Path.” CIO 1 Feb: 108
Sidebar to an article on Content Delivery Networks, it briefly describes the need for and
development of software to route packets through an optimal path to their destination.
7