Network Services
LHC OPN Networking at BNL
Summer 2006 Internet 2 Joint Techs
John Bigrow
July 18, 2006
Brookhaven Science Associates
U.S. Department of Energy
1
Network Services

LHC Overview (very simple overview, I’m not a physicist)
• LHC / Atlas Experiments Overview (The What)
• The Physics Architecture (The Why)
• Preliminary Network and Security Architecture (The How)
Brookhaven Science Associates
U.S. Department of Energy
2
Network Services
CERN Accelerator Ring Aerial View
Brookhaven Science Associates
U.S. Department of Energy
3
Network Services
Brookhaven Science Associates
U.S. Department of Energy
4
Network Services
CERN:Outside Resource Ratio ~1:2
Tier0:( Tier1):( Tier2)
~1:1:1
~PByte/sec
Online System
ATLAS
Experiment
< GBytes/sec
Tier 0 +1
~10 Gbits/sec
Tier 1
IN2P3 Center
CERN ~5M SI2K
>1 PB Disk
Tape Robot
INFN Center
RAL Center
BNL: ~2M
SI2K; 2PB
Tape Robot
2.5 Gbps
~2.5 Gbps
Tier 2
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Tier2 Center
Tier 3
InstituteInstitute Institute
Physics data cache
Workstations
Brookhaven Science Associates
U.S. Department of Energy
Tier 0: DAQ, reconstruction, archive
Tier 1: Reconstruction, simulation,
archive, mining and (large scale) analysis
Tier 2+: Analysis, simulation
Tier 3+: Interactive analysis
Institute
100 - 1000
Mbits/sec
Tier 4
5
Network Services
The same host name for dual NIC
dCache door is resolved to different IP
addresses depending on which DNS is
inquired.
130.199.185.0
130.199.48.0
…
…
Brookhaven Science Associates
U.S. Department of Energy
130.199.48.0
6
Network Services
US ATLAS Tier 1 WAN Bandwidth Requirement Estimate
Year
2004
(Mbits/sec)
2005
2006
2007
2008
2009
2010
Remote Site(s)
Tier 0 (CERN)
Tier 1's (~2 Peer sites)
Tier 2's (5 USA satellite sites)
Tier 3-4 (150 Individual users)
Total
52
37
64
95
105
75
128
190
349
250
428
632
874
624
1,069
1,581
1,747
1,248
2,139
3,161
1,747
1,248
2,139
3,161
3,494
2,496
4,278
6,322
249
498
1,659
4,148
8,295
8,295
16,590
BNL HEP/NP WAN Bandwidth Requirement Estimate
Year
US ATLAS Tier 1 Req.
RHIC Computing Facility Req.
TOTAL
BNL HEP/NP Requirement
Brookhaven Science Associates
U.S. Department of Energy
2004
249
200
449
OC12
(Mbits/sec)
2005
2006
498
500
998
1,244
1,023
2,267
OC48
OC48
7
2007
4,148
1,286
5,433
OC192
2008
2009
8,295
1,847
10,142
2Xλ
2010
9,954
2,422
12,377
2
x
λ
16,590
3,381
19,971
3
x
λ
Network Services
Brookhaven Science Associates
U.S. Department of Energy
8
MAN LAN
CERN (?)
NLR
ESnet
GEANT, etc.
Network Services
111 8th
32
AoA
Hauppauge
Adva
FSP3000
DWDM
Adva
FSP3000
DWDM
10GbE
Adva
FSP3000
DWDM
Hicksville
BNL internal
Other
connections
Adva
FSP2000
DWDM
Diverse Route
Protected DWDM
Core Ring
Adva
FSP2000
DWDM
Diverse Route
Protected DWDM or
CWDM Access
Ring
Diverse Route
Protected DWDM Core Ring
Adva
FSP3000
DWDM
10GbE
Adva
FSP2000
DWDM
Diverse Route
Protected DWDM
Core Ring
Adva
FSP3000
DWD
60 Hudson
Garden City
Adva
FSP2000
DWDM
Adva
FSP2000
DWDM
CWDM
Brentwood
Other
connections
Brookhaven Science Associates
U.S. Department of Energy
BNL
Adva
FSP2000
DWDM
CWDM
9
10GbE
ADVA
FSP2000
DWDM
CWDM
10GbE
Network Services
BNL LHC OPN Conceptual Block Diagram
LHC OPN Private
Core Intranetwork
CIDR Restricted
Distribute List
ES Net Only
ACL
BNL Internet /
Tier 2 Lambda
LHC OPN T0-T1
Lambda Layer 2
Tunnel ACL
20 Gb/Sec
Other Tier 1 Sites
ES Net / General
Internet / Tier 2
ACL
NYSERNET /
Broadwing
ACL
BNL Border Router
Optional Dedicated
BNL Campus Network LHC OPN FWSMs
20 Gb/Sec
BNL LHC OPN
Primary Distribution
Switchs
ES Net Provisioned
CIDR IP Space
Future 10 Gb/Sec
Upgrades
1 Gb/Sec
d
i
g
i
t
a
l
d
1 Gb/Sec
1 Gb/Sec
i
g
i
t
a
l
d
i
g
i
t
a
l
d
i
g
BNL LHC OPN Disk Cache / Storage / Analysis Facilities
Multi-homed
Brookhaven Science Associates
U.S. Department of Energy
10
i
t
a
l
Network Services

Network Security Limitations
• Current firewall Architecture
– 6 virtual 1 Gb/Sec EtherChannel to Catalyst backplane
– Rated total throughput of 5 Gb/Sec
– EtherChannel Overhead Loss
– Single 1 Gb/Sec flow / interface
• New Cisco ACE blade might address these limitations
Brookhaven Science Associates
U.S. Department of Energy
11
Network Services

Network Security Limitations (Continued)
• Current Router Architecture
– Single Access Control List (ACL) / interface
- 1 inbound and 1 outbound per interface
- Default behavior Implicit deny
- Policy route map for traffic flow
– A single ACL can become unwieldy in a complex WAN environment (what
are the network prefixes, DHCP, NAT)
– Manual changes to the route map for additional access
Brookhaven Science Associates
U.S. Department of Energy
12
Network Services

BNL LHC Overview cont.
• Networking resources
– IP Address space allocations / access
– 10Gig interfaces / 20Gig Etherchannels
– Performance Monitoring
Brookhaven Science Associates
U.S. Department of Energy
13
Network Services

IP Address Allocation Tier 0 to Tier 1 (BNL - CERN)
• Requires routable IP Address space
• Direct dedicated access with CERN to / from BNL
• Limited route advertisements between T0 and T1
– For the LHC OPN Circuit BNL will use 192.12.15.0/24
– No direct T1 to T1 access through CERN at this time
Brookhaven Science Associates
U.S. Department of Energy
14
Network Services

BNL OPN to Tier 2 and others
• Tier 2 and other traffic dependant on Internet connectivity
– Path to BNL via all service providers (ES Net now, NYSERNET,
Broadwing in the future ?)
– Dedicated paths to other institutions welcome (you buy)
Brookhaven Science Associates
U.S. Department of Energy
15
Network Services
1 x 10G
Preliminary
BNL 10 /20 Gig-E LHC OPN
Initial Architecture
1 x 10G
3 Peerings
Internet Peer with ES Net
Direct Layer 2 Interface to CERN
T0 - T1
Gateway ACL
Amon
Mutt
SW9
Core
Shu
Tefnut
BNL LHC OPN
Anubis
Isis
Nephthys
Osiris
Brookhaven Science Associates
U.S. Department of Energy
16
SW7
Network Services

Future BNL LHC OPN Enhancements
• Dedicated Cisco Firewall Service Modules (ACE) when
available
– Eliminate router ACL Functionality / Maintenance
– Connection Logging
– Each FWSM circuit will not impede the 10 Gb/Sec.
– Stateful FWSM redundancy
• IDS / IPS when available
Brookhaven Science Associates
U.S. Department of Energy
17
Network Services
BNL Campus Network
Including Near-Term Upgrades
1 x 10G
1 x 10G
Internet Peer with ES Net
Direct Layer 2 Interface to CERN
T0 - T1
FE
Stateful
Link
Amon
NYSERNET
Broadwing
Mutt
SW9
Core
Shu
Tefnut
BNL LHC OPN
Building Access
Layer Switch (Typical
Deployment)
DL2
DL1
Failover
Anubis
Failover
Isis
FWSM
Nephthys
Brookhaven Science Associates
U.S. Department of Energy
Osiris
18
SW7
Network Services

Mon
• browser-based IP service monitor
• Internet-centric WAN based monitor application
• Interrogates essential BNL network services
Brookhaven Science Associates
U.S. Department of Energy
19
Brookhaven Science Associates
U.S. Department of Energy
20
Network Services

MonaLisa
• Java based SNMP monitoring tool

External WAN based monitor

Tracks BNL 10G/Sec. Interfaces
Firewall Service Module
20 Gb/Sec. Uplinks to the BNL core


Brookhaven Science Associates
U.S. Department of Energy
21
Network Services
Brookhaven Science Associates
U.S. Department of Energy
22
Network Services
Brookhaven Science Associates
U.S. Department of Energy
23
Network Services

Cacti
•
•
•
•
SNMP monitoring tool
Replacement for MRTG
Tracks most BNL core network interfaces
Firewall Service Module EtherChannel interfaces also
Brookhaven Science Associates
U.S. Department of Energy
24
Network Services
Brookhaven Science Associates
U.S. Department of Energy
25
Network Services
Brookhaven Science Associates
U.S. Department of Energy
26
Network Services
Brookhaven Science Associates
U.S. Department of Energy
27
Network Services
Brookhaven Science Associates
U.S. Department of Energy
28
Network Services
Brookhaven Science Associates
U.S. Department of Energy
29
Network Services

Thanks (a few kind words to so many)
• Thanks to the many individuals and groups who have
donated their time, code, and talents to make the Internet
what it is today. Without their efforts, this infrastructure we
take for granted would not exist. We owe many our
gratitude.
Brookhaven Science Associates
U.S. Department of Energy
30
Network Services
Questions/Comments
???
Brookhaven Science Associates
U.S. Department of Energy
31
Network Services
BNL Points of Contact

Scott Bradley, Manager of Network Services
• 631.344.5745, bradley@bnl.gov

John Bigrow, Senior Network Architect
• 631.344.2648, big@bnl.gov
Brookhaven Science Associates
U.S. Department of Energy
32