”Operational stability
of the Internet”
SANOG1 - Kathmandu, Nepal
2003-01-25
Kurtis Lindqvist
<kurtis@netnod.se>
Slides available at
http://www.kurtis.pp.se/presentations
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 1
Who is Netnod?
• Continuation of D-GIX
• Established in 1997
• Operates four exchanges in Sweden
– Stockholm, Gothenburg, Malmö, Sundsvall
• Have been the first (only) exchange
point to use DPT/SRP/RPR/802.17
– Switching to Gigabit Ethernet
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 2
The Internet has changed
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 3
Different needs
• Medical applications on STM-1 based IPVPNs
• On-line banking applications
• Production dependent IP-VPNs
• VoIP
• Down-time is starting to have an effect
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 4
The Internet of today
won’t do it
IGP convergence
• Will impact your service offering
– IP-VPNs
– VoIP services
– Banking?
Not really new…
…and relatively easy to fix
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 5
The Internet of today
won’t do it
IGP convergence
– Can be improved with IGP timers for
both ISIS and OSPF
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 6
The Internet of today
won’t do it
IGP convergence
• Suggestions to fix the protocol also
exists
– See Cengiz Alaettinoglu presentation
at the ISIS-WG in San Diego Dec
2000
• But also be done at Layer 2…
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 7
The Internet of today
won’t do it
BGP Convergence
• Convergence is poor as
documented by Abha Ahuja and
Craig Labovitz
• Will also impact your service
offering
– How do I do intra-provider VoIP?
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 8
The Internet of today
won’t do it
BGP convergence
• Harder to fix
– Problem with the protocol
• A number of suggestions are out there
– Dampening might harm you, see Randy
Bush et al at RIPE in Rhodes and last Nanog
• For a long term fix we need to find a
better way of doing this…
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 9
..but we still need to do
Interconnects…
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 10
Interconnections
• Transit connections
• Direct peerings
• Exchange points
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 11
Transit connections
• ”Where the rest goes”
– Your way to the most of the Internet
• Often protected so that you won’t
get isolated in case of failures
• Critical as otherwise you will be an
isolated island of the Internet
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 12
Direct peerings
• Direct links between two ISPs
• Normally redundant
– For economical ”fairness”
• Not really crucial as you can send
traffic through the upstream
– But might be crucial depending on
your traffic ratio
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 13
Exchange points
• Many ISPs connected to common
and shared media
• Normally limited protection
• Limited effect if lost
– But can still force large volume of
traffic to be shifted and leading to
higher cost
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 14
What if we loose an
exchange point?
Tested in full scale at Netnod
Migration of mountain cave
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 15
Netnod is running a
number of exchangepoint
medias
• Cisco DPT / SRP
– 622 Mbps
– 2.5Gbps
• FDDI
– Legacy
• Gigabit Ethernet
– Bridged to FDDI
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 16
Netnod setup - GigE 1
Customer router
A
B
Extreme switches
FDDI Switches
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 17
Netnod setup - GigE 2
Customer routers
A
B
Extreme switches
FDDI Switches
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 18
Netnod setup - DPT
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 19
…and then we migrated ”A”…
…so what happened?
Nothing really…
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 20
Government project
• Study of the overall stability of the
Internet in Sweden
– DNS
– Exchange points
• Gathered a number of data
– BGP logs of 10 ISPs…almost
– BGP listening on US East/west coast
– Load graphs
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 21
Traffic shifted as planned
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 22
No real BGP activity withdrawals
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 23
No real BGP activity Advertisement
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 24
BGP Activity - details
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 25
What conclusions can we
make?
• Exchange points are not really
crucial
– Some ISPs actually shifted the traffic
beforehand
• Interconnects can be made stable
– Layer 2 convergence will help
– But GigE can also be made to work…
– …but you need to be careful
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 26
So what do we think
happened?
• Well, the only thing that really
changed was the NEXT_HOP
• Backup paths was known
• So BGP recalculation was very
limited, if at all
• But this needs planning and luck!
– And not necessarily in that order…
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 27
Operational stability
• All this is examples of how to
improve operational stability
• But why do I needed it?
– The answer is obvious
• Or is it?
– How dependent is your country on
the Internet?
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 28
Operational stability
• Netnod was to some extent created out
of the needs for stable Internet in
Sweden
– The physical exchange is located in caves
owned by the government
– But they have no control over it
• Gov and we where concerned about
physical security
– As far as I know unique - which have
proved to be useful
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 29
Operational stability
• Physical security is the easy part
– Todays attacks are logical
• The Swedish government is moving into
CERT like functions and is trying to find
their role
– The idea is good
– But they add little value to the solution of
the real problem
– Still they are needed and can be useful
– Early start is goood
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 30
Operational stability
• The challenge in the future lies in
moving forward on two fronts
– Physical
– Logical
• Both are well understood…
• …but poorly implemented
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 31
Conclusions
• Operational stability is becoming more
and more crucial
– For revenue and infrastructure
• Work with your goverment before they
work with you
• There is a lot you can do (should have
done) to increase the stability of your
network
– And the rest of the Internet
• Conferences like these are good!
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 32
Go do it!
Before you wake up and is forced to!
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 33
?
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 34
Contact
Netnod Internet Exchange i Sverige AB
Bellmansgatan 30I
SE-118 47 Stockholm
Sweden
Office address: Bellmansgatan 30I
Telephone: +46-8-615 85 70
Telefax: +46-8-442 09 67
E-mail: kurtis@netnod.se
URL: http//www.netnod.se/
© 2003 - Netnod AB
http://www.netnod.se/
2003-01-25
Sida 35