The Internet & SLAC
Les Cottrell1, SLAC
 http://www.slac.stanford.edu
 /grp/scs/net/talk/internet-connectivity-97/index.htm
 Outline
–
–
–
–
–
–
of Talk
I. SLAC’s connectivity
II. How is it Working?
III. Why is it like it is?
IV. What’s going on out there on the Internet?
V. What are we (DOE/Esnet, HEP, SLAC) doing?
VII. Summary & Future
3/16/2016 Talk presented at SLAC, July 1997
1
Some Acronyms
ARA - Appletalk Remote Access, protocol to connect up remote Macs
 ATM - Autonomous Transfer Mode, a high high speed network
mechanism
 DSL - Digital Subscriber Loop, a proposed medium speed (100s kbps
- Mbps) leased line service (phone company answer to cable modems)
 ESnet - Energy Sciences network (DOE’s research network, SLAC’s
main connection to Internet
 ISDN - Integrated Switched Digital Network, new <= 128 kbps digital
switched phone service)
 POP - Point of Presence, a place where one or more networks have
facilities
 SLIP/PPP protocols to provide Internet access over a serial line
 VPN - Virtual Private Network, a way of tunneling private data over
the public Internet

WAN - Wide Area Network
3/16/2016
2

Outline
 I.
SLAC’s Connectivity
 II. How is it Working?
 III. Why is it like it is?
 IV. What’s going on out there on the Internet?
 V. What are we (DOE/Esnet, HEP, SLAC)
doing?
 VII. Summary & Future
3/16/2016
3
Dial in Access
http://www.slac.stanford.edu/comp/net/residential.html







Terminal/emulator dial in
– 7 ports, 14.4 kbps
ARA
– 16 ports, <=33.6kbps, ~340 accounts
(85 active/mo)
SLIP/PPP thru campus
– 14.4kbps, need campus account
Netcom, $15/mo, nationwide
– 28.8 kbps
Wireless via Ricochet
ISDN Direct & via ISP
– 9 ports, <=128 kbps, in pilot mode
~25 users, production service late
summer
Following VPN developments
3/16/2016
4
SLAC’s WAN Connectivity
43Mbps to ESnet ATM
cloud (Sprint Oakland
POP)
 1.5Mps to Caltech/ESnet
 1.5Mbps to LBNL/ESnet
 10Mbps to Stanford

3/16/2016
5
Outline
 I.
SLAC’s Connectivity
 II. How is it Working?
 III. Why is it like it is?
 IV. What’s going on out there on the Internet?
 V. What are we (DOE/Esnet, HEP, SLAC)
doing?
 VII. Summary & Future
3/16/2016
6
What is Important to User
 We
have to optimize the scarcest & therefore most
valuable commodity - Time
 How long does it take after I hit the button?
3/16/2016
7
Value of Rapid Response Time
 Studies
in late 70’s early 80s by Walt Doherty of
IBM & others showed the economic value of rapid
response time:
–
–
–
–
–
0-0.4s
0.4-2s
2-12s
12s-600s
>600s
= High productivity interactive response
= Fully interactive regime
=Sporadically Interactive regime
=Break in contact regime
=Batch regime
 There
is a threshold around 4-5s where complaints
increase rapidly
3/16/2016
8
Ping Response for Groups of Hosts
Average Ping Response for Various
Groups of Hosts Seen from SLAC Jan-95
thru Dec-96
ESNET
700
Internatl
Ping Response (ms)
600
NAmericaE
NAmericaW
500
400
300
200
100
International little change
N. America E improving 210 ms -> 150ms
N. America W improving 140 ms -> 80ms
3/16/2016
ESnet improving 100ms -> 50 ms
6
-9
ov
N
-9
6
6
Se
p
ay
M
Ju
l-9
-9
6
6
-9
ar
M
Ja
n
-9
6
5
-9
ov
N
Se
p
-9
5
5
Ju
l-9
-9
5
M
ay
-9
ar
M
Ja
n
-9
5
5
0
9
European/Japan Packet Loss to SLAC
 Packet
loss much more important
– loss of packet typically causes 4-5s timeout
Packet Loss to Major HEP International
Sites seen from SLAC Jan-95 thru Apr-97
% 100 Byte Ping Packet Loss
45
40
CERN.CH
Increase UK-US
bandwidth
DESY.DE
IN2P3.FR
RL.AC.UK
KEK.JP
ROMA1.INFN.IT
35
30
Improve Esnet Internet connect
25
20
15
10
5
0
Sep-94
Jan-95
Apr-95
Jul-95
Oct-95
Feb-96
May-96
Aug-96
Dec-96
3/16/2016
RAL: poor to unnacceptable, most others acceptable
Mar-97
Jun-97
10
Quality by Host Group
Ping Loss Quality
Distributions for Host Groups
<= 1% Loss (==Good)
Percentile
70%
(150, 0.79)
(76, 5.46)
(183, 7.18)
>1% & <=5% Loss (==Acceptable)
>5% & <=12% Loss (==Poor)
> 12% & <=25% Loss (==Bad)
>25% Loss (==Unusable)
60%
50%
(188, 6.21)
(199, 6.3)
40%
30%
20%
10%
ic
N
Am
er
er
N
Am
rn
te
In
ic
aW
na
at
io
ca
l
IS
P
Lo
t
Es
ne
(host-months, median loss)
aE
l
0%
0.0-1% Good, 1-5% Acceptable, 5-12% Poor
12-25% Bad, > 25% Unusable
Similar to Internet Weather Report (<6%, <12%, > 12%)
3/16/2016
11
Outline
 I.
SLAC’s Connectivity
 II. How is it Working?
 III. Why is it like it is?
 IV. What’s going on out there on the
Internet?
 V. What are we (DOE/Esnet, HEP, SLAC)
doing?
 VII. Summary & Future
3/16/2016
12
Driving Forces - Hosts
3/16/2016
13
Driving Forces - New Apps
 WWW,
multimedia, Internet voice, video
conferencing
– > 60% graphics
– <20% HTML
3/16/2016
14
Driving Forces - Penetration
US Domains
Countries with Internet
access
3/16/2016
15
Current Internet Hosts
3/16/2016
16
Challenge - Diversity of Traffic
other domains
13%
it
2%
Traffic out of
FNAL
ch
2%
br
3%
edu
43%
gov 4%
com
14%
com
14%
net
19%
3/16/2016
17
Challenge - No single Mgmt for Links
1 RTR-CGB4.SLAC.Stanford.EDU
2 RTR-DMZ.SLAC.Stanford.EDU
3 ESNET-A-GATEWAY.SLAC.Stanford.EDU
4 pppl-atms.es.net
5 nynap-pppl-atms.es.net
6 192.157.69.11 [Sprint NAP]
7 core3-hssi3-0.WestOrange.mci.net
8 core1.WestOrange.mci.net
9 border2-fddi-0.WestOrange.mci.net
10 border2-hssi1-0-gw.WestOrange.mci.net
11 192.204.183.3 [PREPnet]
12 DEFAULT1-GW.UPENN.EDU
13 NISC8.UPENN.EDU
3/16/2016
18
Challenge
 Commercial
Internet focussed on staying
alive as opposed to research or promoting
advanced requirements
3/16/2016
19
More Acronymns







CalREN2 - a California initiative to provide better educational &
research networking
CHEP97 - Computing in High Energy Physics meeting in Berlin,
April 1997
ESSC - ESnet’s Steering Committee
ICFA - International Committee on Future Accelerators
Internet 2 - University initiative to provide improved networking
between universities
NGI - Next Generation Internet, Presidential initiative
vBNS - very high-speed Backbone Network System, a high speed
NSF funded backbone network
3/16/2016
20
Outline
 I.
SLAC’s Connectivity
 II. How is it Working?
 III. Why is it like it is?
 IV. What’s going on out there on the
Internet?
 V. What are we (DOE/Esnet, HEP,
SLAC) doing?
 VII. Summary & Future
3/16/2016
21
New Initiatives - California
 CalREN2
–
–
–
–
3/16/2016
joint proposal NSF, UC, Stanford, Caltech …
includes Pac Bell & Cisco
Distributed GigaPOPs in SF & LA, also SD & Sac
Hi speed (622Mbps) ring around state envisioned
22
Bay Area
 Bay Area
CalREN2 GigaPOP nodes:
– UCSF
– UCB (links to Esnet, Sprint, MCI/vBNS, UC
Davis (state wide)
– UCOP
– Stanford (links to NASA/NSI, BBN, MCI &
statewide ring)
3/16/2016
23
New U.S. Initiatives: vBNS
NSF initiative for
interconnecting
supercomputer centers
for “meritorious
applications”
 Extended to promote
University
interconnectivity
 622 Mbps backbone

3/16/2016
24
New US Initiatives: Internet 2

Started out (Oct-96) as
consortium of ~ 34 major
universities
– Now there are over 100

covers 80% of US university
sites we monitor
– ~$500K / university over
several years, 25% seed
– Will use vBNS as backbone
– GigaPOPs in major areas
3/16/2016
25
Next Generation Internet (NGI)
 Presidential
Initiative
– $100M/yr for 3 years
– 100 sites at 100 times bandwidth (1.5Mbps => 155Mbps
backbone)
– 10 sites at 1000 times bandwidth
– DARPA, DOE, NSF, NASA…
 Internet2/NGI/ESnet
relationship unclear
– can Universities connect to Internet 2 & ESnet?
3/16/2016
26
U.S. International Connections
 Only
list those of interest to HEP
 Moving to colocate US end points at DC POP to
improve peering
 Discussing CERN<=>Esnet<=>KEK link
 STAR-TAP = proposed Int’l GigaPOP at Chicago
Country
Brazil
Canada
CERN
France
Germany (DFN)
Italy
Japan
Today (Jan-Mar '97)
128kbps to FNAL
2*45Mbps
2Mbps
IN2P3 via CERN
1.54 Mbps to DC POP
1.54Mbps to PPPL
KEK to FixW 512kbps
UK
(9 + 8.5)Mbps (ANS+Sprint) 45Mbps via Teleglobe (DC POP?)
3/16/2016
Plans
Move to DC POP
Move to DC POP May-97
?
Add part of 2*45Mbps
Look to move to DC POP
KEK 1.54Mbps to LBNL
27
Europe: TEN-34

W. European and some E.
European countries
interconnect at 4 - 34Mbps
– de, it, ch, uk, gr, nl, pt, at, lu,
es, fr, be, hu, sw+dk+no+fi
– Several links in production,
more by Jul-97
Intra country links generally
good
 Intra Europe links improving
with TEN-34
 Next step TEN-155

3/16/2016
28
Asia & FSU
 Most
connections thru Japan, in general
good to acceptable for KEK
– US/Esnet/KEK 522kbps => 1.5Mbps
– China 64kbps => 128kbps (via KEK)
– => 128kbps BINP/Russia Jun-97
 2Mbps
3/16/2016
satellite DESY <=> MSU (Moscow)
29
Outline
 I.
SLAC’s Connectivity
 II. How is it Working?
 III. Why is it like it is?
 IV. What’s going on out there on the
Internet?
 V. What are we (DOE/ESnet, HEP,
SLAC) doing?
 VII. Summary & Future
3/16/2016
30
ESSC End-user Connectivity WG
 intra-ESnet
connectivity good
 ESnet <=> University connectivity often bad
 ESSC set up WG to look at problem
–
–
–
–
Ranked top 20 university sites by ER funding
Monitored from SLAC & FNAL
Identified worst (bad (6) to poor (8) performance)
Recommended ESnet look at 6 BAD sites to understand
costs of improving, DOE/ESnet will provide money
– Typical Frame Relay 1.5Mbps connections $13K/month
3/16/2016
31

ESnet Peering
Improved peering (63=>110
NSPs), examples:
– MCI & Sprint to avoid public
interconnect swamps
– University of California (avoid
Sprint)
– THEnet at UT Austin
– vBNS
East Coast in place since Feb-97
(avoid W Orange MCI)
 West Coast May 1997
 Chicago to come

– Hubs at DC, Oakland, San
Diego, Chicago
– Now carry > 45K routes
3/16/2016
32
Improved ESnet Internet connection
Cumulative % Packet Loss
Weekday 7am - 7pm Packet Loss
Washington.edu
seen from SLAC '95 - '96
UOregon.edu
UCSC.edu
UCDavis.edu
Colostate.edu
Colorado.edu
100
90
80
70
60
50
40
30
20
10
S
ep
-9
6
N
ov
-9
6
6
Ju
l-9
S
ep
-9
5
N
ov
-9
5
Ja
n96
M
ar
-9
6
M
ay
-9
6
5
Ju
l-9
Ja
n9
5
M
ar
-9
5
M
ay
-9
5
0
ESnet Peers with Sprint/MCI
to avoid MAE-West
3/16/2016
33
UC-Esnet Improved Peering & UCSC
25ms
3/16/2016
16ms
34
vBNS/Esnet Peering & U. Colorado
Improved peering
between Esnet &
vBNS
3/16/2016
35
ICFA Internet Working Group
 Mini-workshop
CHEP97
 Working groups on: monitoring, remote
regions, present status, requirements
analysis, and the proposal
 End
1998 come up with proposal on what to
do & why
 Next
3/16/2016
meeting Santa Fe, Sep-97
36
Monitoring - Why

“You can’t manage what you can’t measure”
 Monitor
to set “user” expectations, help
with problem detection, get long term trends
End-to-end monitoring mainly using ping
 Provides response time, packet loss,
reachability, unpredictability
 Short (trouble shooting) & long term
(planning)
 Most important metric is packet loss
3/16/2016
37
Monitoring - Who
 Many
major HEP sites are monitoring endto-end Internet performance to collaborators
– several hundred remote sites monitored
 Collaborative
effort to provide HEP-wide
and ESnet wide reports, requested by ICFA,
ESnet
– Partially funded by DOE FWP involving SLAC, LBL,
HEPNRC
– Based on SLAC early work (ping based) will
complement LBL NIMI work
– SLAC, HEPNRC/FNAL, LBL collaboration
3/16/2016
38
Monitoring - How
 Plan
to coordinate effort, centered on SLAC/HEPNRC code
–
–
–
–
install common software
distributed architecture
SLAC, HEPNRC Analysis Sites
Umd, RAL, INFN, KEK, ARM, CMU, RMKI, IN2P3,
CERN, DESY, TRIUMF, MSU signed up to be
Collection Sites
– 247 Remote Sites as of 7/7/97
– Reduces network impact of full mesh monitoring
3/16/2016
39
Data Collection & Distribution
Architecture

HTTP
WWW
E.g. HEPNRC
E.g. SLAC
Analysis
Analysis
E.g. RAL
Collecting
Collecting
Ping Data
(via HTTP)
Collecting
Collecting
Pings
Remote
Remote
Remote
Remote
Remote
3/16/2016
40
Results from ~70 Sites in 10
Countries Being Monitored from
SLAC
FNAL
UMd
SLAC
ORNL
Monitoring Site
ESnet Site
N. American Site
International Site
3/16/2016
41
Putting it all Together
3/16/2016
42
Outline
 I.
SLAC’s Connectivity
 II. How is it Working?
 III. Why is it like it is?
 IV. What’s going on out there on the
Internet?
 V. What are we (DOE/Esnet, HEP, SLAC)
doing?
 VII. Summary & Future
3/16/2016
43
Summary
 Driving
forces:
– Internet user growth 8.4M => 28M US users (15 mos)
– Computer power doubling every 12-18 months
– new applications, WWW, Internet phone, VR, Video ...
 Since Apr-95,
no single management for planning,
trouble reporting etc.
 ESnet performance good to acceptable, N. America
poor (~6% packet loss avg), International poor (~7%
packet loss avg)
 Bottlenecks at interchanges
3/16/2016
44
Future
 Many
separate initiatives:
– critical to make sure they interplay well
– identify and avoid bottlenecks
– understand and guide impact for HEP
 Criticality
of Internet to HEP collaborations means
HEP should increase efforts in this area:
– keep tuned in, understand issues
– monitor end-to-end performance
– work with other research and higher education users
3/16/2016
45
 SLAC

http://www.slac.stanford.edu/comp/net/net.html
 SLAC

Networking
WAN Monitoring Page, lots of pointers
http://www.slac.stanford.edu/comp/net/wan-mon.html
 ESnet:
http://www.es.net/
 vBNS:
http://www.vbns.net/
 Internet2: http://www.internet2.edu/
 NGI:
http://www.hpcc.gov/ngi-concept-08Apr97/
 TEN-34:
http://www.scimitar.terena.nl/projects/ten-34/
 ICFA Workshop

3/16/2016
on HEP & the Internet:
http://www.slac.stanford.edu/xorg/icfa/chep97/wg.html
46