Amogh Dhamdhere
Committee:
Dr. Constantine Dovrolis (advisor)
Dr. Mostafa Ammar
Dr. Nick Feamster
Dr. Ellen Zegura
Dr. Walter Willinger (AT&T Labs-Research)






27,000 autonomous networks independently operated and managed
The “ Internet Ecosystem ”


Different types of networks
Interact with each other and with “environment”
Network interactions


Localized, in the form of interdomain links
Competitive (customer-provider), symbiotic (peering)
Distributed optimizations by each network

Select providers and peers to optimize utility function
The Internet ecosystem evolves
4/17/2020 2





How does the Internet ecosystem evolve?
What is the Internet heading towards?



Topology
Economics
Performance
How do the strategies used by networks affect their utility (profits/costs/performance)?
How do these individual strategies affect the global
Internet?
4/17/2020 3





Static graph properties

No focus on how the graph evolves
“Descriptive” modeling

Match graph properties e.g. degree distribution

Homogeneity
Nodes and links all the same
Game theoretic, computational

Restrictive assumptions



Dynamics of the evolving graph


Birth/death
Rewiring
“Bottom-up”

Model the actions of individual networks
Heterogeneity

Networks with different incentives

Semantics of interdomain links
4/17/2020 4

Measure the evolution of the Internet
Ecosystem
Topological, but focus on business types and rewiring
Model strategies and incentives of different network types
Predict the effects of provider and peer selection strategies
4/17/2020 5


Measuring the Evolution of the Internet Ecosystem
[ IMC ’08 ]

The Core of the Internet: Provider and Peer
Selection for Transit Providers
[ to be submitted ]

The Edge of the Internet: ISP and Egress Path
Selection for Stub Networks
[ Infocom ‘06 ]

ISP Profitability and Network Neutrality
[ Netecon ’08 ]
4/17/2020 6






How did the Internet ecosystem evolve during the last decade?
Is growth more important than rewiring?
Is the population of transit providers increasing or decreasing?

Diversification or consolidation of transit market?
Given that the Internet grows in size, does the average path length also increase?
Where is the Internet heading?
4/17/2020 7



Focus on Autonomous Systems (ASes)

As opposed to networks without AS numbers
Start from BGP routes from RouteViews and RIPE monitors during 1997-2007

Focus on primary links

Classify ASes based on their business function

Enterprise ASes, small transit providers, large transit providers, access providers, content providers

Classify inter-AS relations as “ transit ” and “ peering ”


Transit link – Customer pays provider
Peering link – No money exchanged
Visibility Issue
4/17/2020 8




Number of CP links and ASes showed initial exponential growth until mid-2001
Followed by linear growth until today
Change in trajectory followed stock market crash in North
America in mid-2001
4/17/2020 9


Number of ASes has grown from 5000 in 1998 to 27000 in 2007

Average path length remains almost constant at 4 hops
4/17/2020 10


Most new links due to internal rewiring and not birth (75%)

Most dead links are due to internal rewiring and not death
(almost 90%)
4/17/2020 11




Four AS types:


Enterprise customers (EC)
Small Transit Providers
(STP)


Large Transit Providers
(LTP)
Content, Access and Hosting
Providers (CAHP)
Based on customer and peer degrees
Classification based on decision-trees

80-85% accurate
4/17/2020 12




LTPs: constant population (top-30 ASes in terms of customers)
Slow growth of STPs (30% increase since 2001)
EC and CAHP populations produce most growth

Since 2001: EC growth factor 2.5, CAHP growth factor 1.5
4/17/2020 13




CAHPs have increased their multihoming degree significantly

On the average, 8 providers for CAHPs today
Multihoming degree of ECs almost constant (average < 2)
Densification of the Internet occurs at the core
4/17/2020 14


Peering by CAHPs has increased significantly

CAHPs try to get close to sources/destinations of content
4/17/2020 15








Initial exponential growth up to mid-2001, followed by linear growth phase
Average path length practically constant
Rewiring more important than growth
Need to classify ASes according to business type
ECs contribute most of the overall growth
Increasing multihoming degree for STPs, LTPs and
CAHPs

Densification at the core
CAHPs are most active in terms of rewiring, while ECs are least active
4/17/2020 16


Measuring the Evolution of the Internet Ecosystem

The Core of the Internet: Provider and Peer
Selection for Transit Providers

The Edge of the Internet: ISP and Egress Path
Selection for Stub Networks

ISP Profitability and Network Neutrality
4/17/2020 17




From measurements: Significant rewiring activity

Especially by transit providers
Networks rewire connectivity to optimize a certain objective function



Distributed
Localized spatially and temporally
Rewiring by changing the set of providers and peers
What are the global, long-term effects of these distributed optimizations?



Topology and traffic flow
Economics
Performance (path lengths)
4/17/2020 18

Interdomain
TM
Routing Cost/price parameters
4/17/2020
Interdomain topology
Traffic flow
Provider selection
Per-AS profit
Peer selection

When does it converge?

When no network has the incentive to change its connectivity – “steady-state”
19

4/17/2020
C
C
P1
P1 open to peering with
CPs
P2
C S C
P3
S
S
S
20





What is the outcome when networks use certain provider and peer selection strategies?
Model the feedback loop in the Internet ecosystem



Real-world economics of transit, peering, operational costs
Realistic routing policies
Geographical constraints

Provider and peer selection strategies
Computationally find a “steady-state”

No network has further incentive to change connectivity
Measure properties of the steady-state

Topology, traffic flow, economics
4/17/2020 21





Enterprise Customers (EC)


Stub networks at the edge (e.g. Georgia Tech)
Either sources or sinks
Small Transit Providers (STP)


Provide Internet transit
Mostly regional in presence (e.g. France Telecom)
Large Transit Providers (LTP)

Transit providers with global presence (e.g. AT&T)
Content Providers (CP)
4/17/2020 22



The traffic matrix consists of mostly P2P traffic?
P2P traffic benefits STPs, can make LTPs unprofitable


LTPs peer with content providers?
LTPs could harm STP profitability, at the expense of longer end-to-end paths


Edge networks choose providers using path lengths?
LTPs would be profitable and end-to-end paths shorter
4/17/2020 23


Provider selection strategies



Minimize monetary cost (PR)
Minimize AS path lengths weighted by traffic (PF)
Avoid selecting competitors as providers (SEL)

Peer selection strategies



Peer only if necessary to maintain reachability (NC)
Peer if traffic ratios are balanced (TR)
Peer by cost-benefit analysis (CB)

Peer and provider selection are related
4/17/2020 24

4/17/2020
?X
A
C

Restrictive peering
B
A B
C U


Peering by necessity
Level3-Cogent peering dispute
C
U
25


Realistic transit, peering and operational costs


Transit prices based on data from Norton
Economies of scale

BGP-like routing policies

No-valley, prefer customer, prefer peer routing policy

Traffic matrix


Heavy-tailed content popularity and consumption by sinks
Predominantly client-server : Traffic from CPs to ECs

Predominantly peer-to-peer: Traffic between ECs
4/17/2020 26





Networks perform their actions sequentially
Can observe the actions of previous networks

And the effects of those actions
Network actions in each move




Pick set of preferred providers
Attempt to convert provider links to peering links “due to necessity”
Evaluate each existing peering link
Evaluate new peering links
Networks make at most one change to their set of peers in a single move
4/17/2020 27




Determine the outcome as each network selects providers and peers according to its strategy
Too complex to solve analytically: Solve computationally
Typical computation



Proceeds iteratively, networks act in a predefined sequence
Pick next node n to “play” its possible moves
Compute routing, traffic flow, AS fitness


Repeat until no player has incentive to move
“steady-state” or equilibrium
4/17/2020 28


Is steady-state always reached?

Yes, in most cases

Is steady-state unique?


No, can depend on playing sequence
Different steady-states have qualitatively similar properties

Multiple runs with different playing sequence


Average over different runs
Confidence intervals are narrow
4/17/2020 29








Parameterization of the model that resembles real world
Traffic matrix is predominantly client-server (80%)

Impact of streaming video, centralized file sharing services
20% of ECs are content sources, 80% sinks
Heavy tailed popularity of traffic sources
Edge networks choose providers based on price
5 geographical regions
STPs cheaper than LTPs
4/17/2020 30



Reproduces almost constant average path length
Activity frequency: How often do networks change their connectivity?

ECs less active than providers – Qualitatively similar to measurement results
4/17/2020 31

LTP
S1
S2
EC
CP
EC
CP
CP
4/17/2020

Hierarchy of STPs

Traffic can bypass LTPs –
LTPs unprofitable

STPs should not peer with CPs

Resist the temptation!
32

CP
CP
LTP
S1
S2
EC EC
4/17/2020
CP

What-if: LTPs peer with
CPs

Generate revenue from downstream traffic

Can harm STP fitness

Long paths
33

CP
CP
CP 
P2P traffic helps STPs

Smaller traffic volume from CPs to Ecs
EC
4/17/2020
S2
S2
EC
EC
EC
EC
S3
EC

More EC-EC traffic => balanced traffic ratios

More opportunities for
STPs to peer

Peering by “traffic ratios” makes sense
34


A model that captures the feedback loop between topology, traffic and fitness in the Internet

Considers effects of



Economics
Geography
Heterogeneity in network types

Predict the effects of provider and peer selection strategies

Topology, traffic flow, economics, and performance
4/17/2020 35


Measuring the Evolution of the Internet Ecosystem

The Core of the Internet: Peer and Provider
Selection for Transit Providers

The Edge of the Internet: ISP and Egress Path
Selection for Stub Networks

ISP Profitability and Network Neutrality
4/17/2020 36





Sources and sinks of content


Content Providers (CP): sources
Enterprise Customers (EC): sinks
From measurements:


ECs connect increasingly to STPs

Cost conscious ?
CPs connect increasingly to LTPs

Performance ?
Increasing multihoming (about 60% of stubs)

Redundancy, load balancing, cost effectiveness
How should stub networks choose their providers?
4/17/2020 37



How to select the set of upstream ISPs ?



Low monetary cost
Short AS paths to major destinations
Path diversity to major traffic destinations – robustness to network failures
How to allocate egress traffic to the set of selected ISPs ?


Objective: Avoid congestion on the upstream paths
Also maintain low cost
4/17/2020 38


Select k ISPs out of N

Let C be a subset of k ISPs out of N

Total cost of a selection of ISPs C: Weighted sum of monetary, path length and path diversity costs

Select combination C with minimum total cost

Feasible to enumerate all combinations
4/17/2020 39





For set of ISPs C, what is the monetary and path length cost of routing egress flows?
Find the minimum cost mapping G* of flows to ISPs
( Bin Packing )



Flows = items
ISPs = bins
NP hard !
Use First Fit Decreasing (FFD) heuristic

Mapping G* very close to optimal
Monetary and path length costs of C are calculated using the mapping G*
4/17/2020 40





selection C gives K paths to each destination d
K-shared link to d: link shared by all K paths to d

If a K-shared link fails, destination d is unreachable
Minimize the number of Kshared links

Path diversity metric: The number of k-shared links to destination d averaged over all destinations
Gives the best resiliency to single-link failures
4/17/2020 41


Algorithms for ISP selection


Choosing best set of upstream ISPs
Objectives are minimum monetary cost, short AS paths and high path diversity

ISP selection for monetary and performance constraints


Formulated as a bin-packing problem
Heuristic gives solution very close to optimal

ISP selection for path diversity

Returns set of ISPs with best path diversity to the set of major destinations
4/17/2020 42


Measuring the Evolution of the Internet Ecosystem

The Edge of the Internet: ISP and Egress Path
Selection for Stub Networks

The Core of the Internet: Peer and Provider
Selection for Transit Providers

ISP Profitability and Network Neutrality
4/17/2020 43





Recent evolution trend: Large amounts of video and peer-to-peer traffic
Content providers (CP) generate the content

Provide content and services “over the top” of the basic connectivity provided by ISPs

Profitable (think Google)
Access Providers (AP) deliver content to users



Recent trend: Not profitable
Commoditization of basic Internet access
Want a share of the pie
Tension between AP and CPs: “Network neutrality”
4/17/2020 44




Previous work



Mostly non-technical
Highly emotional debates in the press
Legislation/policy aspects: Do we need network neutrality legislation?
But what about the underlying problem: Nonprofitability of Access Providers?
Our approach: A quantitative look at AP profitability

Investigate reasons for non-profitability

Evaluate strategies for remaining profitable
4/17/2020 45

4/17/2020



Three AS types: AP, CP and transit provider (TP)

Focus on the AP
AS links

 customer-provider
(customer pays provider) peering (no payments)
AP and CPs can transfer traffic either through customer-provider or peering links
46




Reasons why APs can be unprofitable


AP users
The impact of video traffic
AP strategies: Pricing


Heavy hitter charging
Heavy hitter blocking

Non-neutral charging
AP strategies: Connection


Caching CP content
Peering selectively with CPs
4/17/2020 47



Variability in AP users can cause large variability in costs
Video traffic: Increases costs for AP

AP strategies based on differential/non-neutral pricing may not succeed

Have to account for user departure due to competition

AP strategies based on connection are promising


Caching content from CPs
Peering selectively with large CPs
4/17/2020 48


A measurement study of the evolution of the
Internet ecosystem

Modeling the evolution of the Internet ecosystem

“what-if” questions about possible evolution paths

Optimizations at the edge of the Internet

Algorithms for provider selection and egress routing

A technical view of the network neutrality debate

Strategies for ISP profitability
4/17/2020 49


Measurements: Investigate the evolution of the connectivity for monitor ASes


We can observe all links for such ASes
Focus on transitions between peering and customer-provider links

Measurements: What does the interdomain traffic matrix really look like?

Can we use measurements from a large Tier-1 provider?

Can we augment that data with information about the interdomain topology?
4/17/2020 50


What is the best strategy for different types of providers?


Strategies for classes of providers
Strategies for individual providers

Do the distributed optimizations by networks solve a centralized problem?

E.g., minimizing path lengths
4/17/2020 51





Router buffer sizing


“Buffer Sizing For Congested Internet Links”
“Open Issues in Router Buffer Sizing”
[ Infocom ‘05 ]
[ CCR ‘06 ]
Network troubleshooting

“NetDiagnoser: Troubleshooting network disruptions using end-to-end probes and routing data” [ CoNext ‘07 ]
Network monitoring

“Route monitoring from passive data plane measurements”
Measurement [ In progress ]


“Poisson vs. Periodic Path Probing”
“Bootstrapping in Gnutella”
[ IMC ‘05 ]
[ PAM ‘04 ]
4/17/2020 52

4/17/2020
53



Each snapshot of the Internet topology captures 3 months

40 snapshots – 10 years
Perform “majority filtering” to remove backup and transient links from topology


For each snapshot, collect several “topology samples” interspersed over a period of 3 weeks
Consider an AS-path only if it appears in the majority of the topology samples

Otherwise, the AS-path includes links that were active for less than 11 days (probably backup or transient links)
Samples
Snapshot
4/17/2020
4/17/2020
54



Some observed link births may be links revealed due to increased monitor set

Similarly for observed link deaths
We calculated error bounds for link births and deaths


Relative error < 10% for CP links
See paper for details
4/17/2020 55

Issue-3: visibility of ASes, Customer-Provider (CP) and
Peering (PP) links


Number of ASes and CP links is robust to number of monitors
But we cannot reliably estimate the number of PP links
4/17/2020 56

4/17/2020
57



The fraction of peering links has been increasing steadily

But remember: this is just a lower bound
At least 20% of inter-AS links are of PP type today
4/17/2020 58



Average path length remains almost constant at 4 hops
Average multihoming degree of providers increases faster than that of stubs

Densification at core much more important than at edges
4/17/2020 59

4/17/2020 60




Europe is catching up with North America w.r.t the population of
ECs and LTPs
CAHPs have always been more in Europe
More STPS in Europe since 2002
4/17/2020 61

4/17/2020
62



Initially most active customers were in North America
After 2004-05, customers in Europe have been more active

Due to increased availability of providers?

More competitive market?
4/17/2020 63




CAHPs have increased their multihoming degree significantly

On the average, 8 providers for CAHPs today
Multihoming degree of ECs has been almost constant (average < 2)
Densification of the Internet occurs at the core
4/17/2020 64



After 2004, ECs prefer STPs than LTPs

Mainly driven by lower prices or regional constraints?
CAHPs connect to LTPs and STPs with same probability
4/17/2020 65



Initially most active customers were in North America
After 2004-05, customers in Europe have been more active

Due to increased availability of providers?

More competitive market?
4/17/2020 66

4/17/2020
67




Attractiveness of provider X: fraction of new CP links that connect to X

Repulsiveness, defined similarly
Both metrics some positive correlation with customer degree
Preferential attachment and preferential detachment of rewired links
4/17/2020 68



A few providers (50-60) account for 50% of total attractiveness (attractors)
The total number of attractors and repellers increases

The Internet is NOT heading towards oligopoly of few large players


LTPs dominate set of attractors and repellers
CAHPs are increasingly present however
4/17/2020 69




Timeseries of attractiveness and repulsiveness for each provider
Calculate cross-correlation at different lags
Most significant correlation values at lags 1,2 and 3

Attractiveness precedes repulsiveness by 3-9 months
4/17/2020 70

4/17/2020
71



ECs and STPs have low peering frequency
Aggressive peering by CAHPs after 2003

Open peering policies to reduce transit costs
4/17/2020 72



Peering by CAHPs has increased significantly

CAHPs try to get close to sources/destinations of content
Peering by LTPs has remained almost constant (or declined)

“Restrictive” peering by LTPs
4/17/2020 73








Initial exponential growth up to mid-2001, followed by linear growth phase
Average path length practically constant
Rewiring more important than growth
Need to classify ASes according to business type
ECs contribute most of the overall growth
Increasing multihoming degree for STPs, LTPs and
CAHPs

Densification at core
CAHPs are most active in terms of rewiring, while ECs are least active
4/17/2020
4/17/2020
74






Positive correlations between attractiveness & repulsiveness of provider and its customer degree
Strong attractiveness precedes strong repulsiveness by period of 3-9 months
Number of attractors and repellers between shows increasing trend
The Internet market will soon be larger in Europe than in North America

In terms of number of transit providers and CAHPs
Providers from Europe increasingly feature in the set of attractors and repellers
4/17/2020 75






Multihoming: Connection of a stub network to multiple ISPs
 x% of stub networks are multihomed
Redundancy
 primary/backup relationships
Load Balancing

Distribute outgoing traffic among
ISPs
Cost Effectiveness

Lower cost ISP for bulk traffic, higher cost ISP for performancesensitive traffic
Performance

Intelligent Route Control
4/17/2020 76





Should consider both monetary cost and performance
Minimum monetary cost

Estimate the cost that “would be” incurred if a set of
ISPs was selected
Minimum AS path lengths


Longer paths: delays, interdomain routing failures
Measure AS path length offline using Looking Glass
Servers
Maximum Path diversity

AS-level paths to destinations should be as “different” as possible
4/17/2020 77




Two phases
Phase I – ISP Selection:




Select K upstream ISPs
K depends on monetary and performance constraints
“Static” operation
Change only when major changes in the traffic destinations or
ISP pricing
Phase II – Egress Path Selection

Allocate egress traffic to selected ISPs


Avoid long term congestion and minimize cost
“Semi-static” operation , performed every few hours or days
4/17/2020 78





AS-level paths and traffic rates are input to simulator

9 ISPs, 250 destinations
Given K, find the selection
C* with the minimum path diversity cost
For each selection C, find u(C) = total traffic lost due to the failure of each link in topology
Calculate Δu(C) = u(C) – u(C*) for each selection C

Single link failures:
C* is the optimal selection
4/17/2020 79






After Phase-I, S has K upstream ISPs
Problem: How to map outgoing traffic to the ISPs
M flows: K M mappings of flows to ISPs

Some mappings may cause congestion to flows !

Flows can be congested at access links or further upstream
Objective: Find the loss-free mapping with the minimum cost
Challenges:


Upstream topology and capacities are unknown
Iterative routing approaches required

Propose an iterative routing based on simulated annealing
4/17/2020 80





AS-level paths and traffic rates are input to simulator

9 ISPs, 250 destinations
Given K, find the selection
C* with the minimum path diversity cost
For each selection C, find u(C) = total traffic lost due to the failure of each link in topology
Calculate Δu(C) = u(C) – u(C*) for each selection C
Single link failures:
C* is the optimal optimal selection
81 4/17/2020





Detailed model for provider and peer selection

Complex real-world decisions
Provider selection objectives


Monetary cost
AS path lengths
Peer selection


Minimize transit costs
Maintain reachability
Constraints

Only local knowledge

Geographical constraints
4/17/2020 82

A B C




Traditional peering links: Not transitive
Peering federation of A, B, C: Allows mutual transit

Longer chain of “free” traffic
Incentives to join peering federation?
What happens to tier-1 providers if smaller providers form federations?
4/17/2020 83




Variability of users => high variability in the costs incurred by AP
Variability increases with the access speed
Increase in video traffic: higher transit payment by AP
4/17/2020 84





AP and CP connect to the TP as customers
N users of AP, charged a flat rate R ($/month)
Transit pricing: 95 th percentile of traffic volume, concave transit pricing functions


95th / mean = 2:1 for normal traffic, 4:1 for video 1
More video means higher transit payment by AP
AP users: Heavy tailed distribution of content downloaded per month

High variability in AP costs
1 Norton’06: Internet Video: The Next Wave of Massive Disruption to the U.S. Peering Ecosystem
4/17/2020 85



Charging strategies


AP charges “heavy hitters” according to volume downloaded
AP caps heavy hitters

AP charges CP (non-network neutral)
Charging strategies are disruptive

AP cannot control customer departure probability
4/17/2020 86



Charging heavy hitters


 download amount D, threshold T, flat rate R c(D) = D*R/T
AP’s profit is sensitive to customer departure prob
Capping heavy hitters and non-neutral charging would not work for the same reason
4/17/2020 87





Connection Strategies


AP caches content from CPs
AP peers with CPs
Non-disruptive
Caching can reduce transit costs of AP

But depends on the amount of content cacheable
Selective peering with CPs can improve profitability


Peering cost depends on CP
Cost/benefit analysis for each CP

CP with large network: low cost of peering
4/17/2020 88





Connection Strategies


AP caches content from CPs
AP peers with CPs
Non-disruptive
Cost-benefit analysis for peering


Peering cost depends on CP
(easy/medium/hard) r = saving/cost (both estimated)

Peer if r > R
AP controls the factor R
4/17/2020 89



Charging heavy hitters


 download amount D, threshold T, flat rate R c(D) = D*R/T
AP’s profit is sensitive to customer departure prob
Non-neutral charging



Customer departure prob
“How discriminatory is my
AP?”
AP’s profit is sensitive to customer departure prob
4/17/2020 90


“Bottom-up” models (more later)

Understand how local actions lead to emerging properties

Performance of protocols over time

“How would BGP perform 10 years from now?”

Clean slate vs. evolutionary design


After initial design, both must evolve !
Understanding evolution of the current Internet can help design
4/17/2020 91