Glownet and Bossnet Gigabit
Network Infrastructure for e-VLBI
Steven Bernstein, Lorraine Prior,
James Calvin, Vineet Mehta
M.I.T. Lincoln Laboratory
e-VLBI Workshop
8-9 April 2002
eVLBI-Workshop_Glownet&Bossnet
1
This work is sponsored by the Defense Advanced Research Projects Agency
under Air Force Contract #F19628-00-C-0002. Opinions, interpretations,
recommendations and conclusions are those of the authors and are not
necessarily endorsed by the Department of Defense
Glownet and Bossnet Multi-Gigabit/sec
Optical Fiber Networks
•
Glownet
–
–
–
–
•
Laboratory
infrastructure
Leased "dark fiber"
COTS Gigabit
Ethernet and
SONET networking
Wavelength
Division
Multiplexed (WDM)
Glownet: Gigabit Lincoln Optical WDM Network
Millstone/
Haystack
Lincoln
Laboratory
MIT
230 Congress St.
Bossnet
–
–
–
–
DARPA-sponsored
testbed
Leased "dark fiber"
Laboratory-built
WDM transmission
equipment
Experimental
eVLBI-Workshop_Glownet&Bossnet
2
Bossnet (Boston South Optical Network)
to Washington D.C.
MIT Lincoln Laboratory
Outline
•
•
•
•
•
eVLBI-Workshop_Glownet&Bossnet
3
WAN (Bossnet)
MAN (Glownet)
LAN
Processing and Protocols
Protocols
MIT Lincoln Laboratory
Bossnet Overview
NGI Onramp
Access Network
Millstone
Phase 2
Supernet
Connectivity to
Other High
Speed Research
Networks
•
Boston, MA
Phase 1
New York, NY
ISI-East
Washington, D.C.
~850 km optically amplified fiber paths
– fiber leased from Qwest
•
Transmission testbed for Gbps++ efforts (WDM, short pulse, new
formats, mixed usage)
•
•
Connectivity to NGI (Next Generation Internet) and other projects
Application demonstration medium for Gbps+ uses
eVLBI-Workshop_Glownet&Bossnet
4
MIT Lincoln Laboratory
Supernet
http://www.ngi-supernet.org/snet_maps/snet_overview1_raster.htm
eVLBI-Workshop_Glownet&Bossnet
5
MIT Lincoln Laboratory
Bossnet Amplifier Locations
Boston
Stamford
New York City
Framingham
Providence
Newark
Greenhill
Trenton
Philadelphia
Washington DC
Wilmington
New London
New Haven
Dark fiber by
Perryville
Baltimore
eVLBI-Workshop_Glownet&Bossnet
6
MIT Lincoln Laboratory
Obstacles to Overcome to Increase Data
Capacity
1
0.8
•
Noise - number one
issue regardless of
transmission format
•
•
Distortion
Intensity independent
– polarization mode dispersion (PMD)
– intersymbol interference (ISI)
0.6
0.4
•
0.2
200
400
600
800
– self-phase modulation (SPM)
– cross phase modulation (XPM)
1000
0.35
0.3
0.25
•
0.2
0.15
0.1
0.05
0
0
100
eVLBI-Workshop_Glownet&Bossnet
7
200
300
400
500
Intensity dependent
Timing jitter
– Gordon-Haus effect causes a
nonlinear mixing of signal and
noise
MIT Lincoln Laboratory
Bossnet Hut Transmission Hardware
OC-3
3R
1510
Drop Port
DCF
ASE-FG
Monitor &
Control Points
DCF
1510
Drop Port
OC-3
3R
ASE-FG
Monitor &
Control
Computer
Modem
Telco
to Lincoln
Lab
•
Each hut has an erbium doped fiber amplifier (EDFA) pair, dispersion
compensating fiber (DCF), and amplified spontaneous emission
rejection fiber grating (ASE-FG) per direction
•
Some huts have 2x2 switches providing loop-back capability to allow
experiments to be performed from a single location
•
OC-3 at 1510 nm is a fully regenerated private channel
eVLBI-Workshop_Glownet&Bossnet
8
MIT Lincoln Laboratory
Gain Flattening for BOSSNET
-20
•
-30
Before
Optical Power (dB)
-10
-40
-50
-60
1540
1545
1550
1555
Wavelength (nm)
1560
•
1565
•
-10
-20
After
Optical Power (dB)
0
-30
-40
•
•
-50
1540
•
1545
1550
1555
1560
Measurements made
over 570 km
Erbium fiber amplifier
gain profile causes
non-uniform signal to
noise
Gain flattening filters
installed at most huts
Negligible change in
maximum signal to
noise
Uniform signal to noise
over band
Allows WDM
1565
Wavelength (nm)
eVLBI-Workshop_Glownet&Bossnet
9
MIT Lincoln Laboratory
Outline
•
•
•
•
•
eVLBI-Workshop_Glownet&Bossnet
10
WAN (Bossnet)
MAN (Glownet)
LAN
Processing and Protocols
Lessons Learned & Summary
MIT Lincoln Laboratory
MIT-Lincoln Boston Area (Glownet)
Fiber Optic Infrastructure
•
'Dark' fiber infrastructure in
place with terabit/sec capacity
– Multi-gigabit/sec network
equipment deployed
•
Shared resource with Lincoln,
MIT campus, Haystack
– Operational and new
applications
Voice, data, video
New applications
Low loss, low dispersion fiber
Operational
& New
Applications
Research in
Networking &
Transmission
Commercial:
1 - 40 Gbps per
wavelength
Research:
> 100 Gbps per wavelength
at longer distances
radar, VLBI, etc.
– Research
Networking
Transmission
ONRAMP, Bossnet
Wavelength Division Multiplexing
Total bandwidth > 1 THz
eVLBI-Workshop_Glownet&Bossnet
11
MIT Lincoln Laboratory
Glownet Dark Fiber
Millstone/
Haystack
NeesCom
(6)
(2)
NEON (2)
230
Congress
St. Boston
MIT
Campus
Nstar (2)
Lincoln
Laboratory
Bossnet
Qwest (4)
NStar (6)
(#)= # of fibers
eVLBI-Workshop_Glownet&Bossnet
12
MIT Lincoln Laboratory
GLOWNET Phase I
Networking Equipment
Lincoln Lab Lexington
Millstone (RPE Building)
Cisco 1500
WDM & Transponders
Cisco 1500
WDM & Transponders
Wavelength
Transponders
MUX
Wavelength
Boston
MUX
2 GigE
(Jumbo
Frames)
Transponders
2 GigE
(Jumbo
Frames)
2 GigE
(Jumbo
Frames)
Optical
Amplifiers
OC-48
12
10/100
Ethernet
OC-48
(Ditech)
Growth
12
10/100
Ethernet
Growth
Data Mux
Data Mux
Cisco 15454
Cisco 15454
eVLBI-Workshop_Glownet&Bossnet
13
MIT Lincoln Laboratory
Outline
•
•
•
•
•
eVLBI-Workshop_Glownet&Bossnet
14
WAN (Bossnet)
MAN (Glownet)
LAN
Processing and Protocols
Lessons Learned & Summary
MIT Lincoln Laboratory
Millstone Hill Fiber Infrastructure
Haystack Radar (HAY)
Haystack Auxiliary Radar (HAX)
Millstone Hill Radar (MHR)
Fiber
Infrastructure
RPE
To
Building Westford
To Boston
eVLBI-Workshop_Glownet&Bossnet
15
MIT Lincoln Laboratory
The Whole Path from Haystack to GGAO
eVLBI-Workshop_Glownet&Bossnet
16
MIT Lincoln Laboratory
Outline
•
•
•
•
•
eVLBI-Workshop_Glownet&Bossnet
17
WAN (Bossnet)
MAN (Glownet)
LAN
Processing and Protocols
Lessons Learned & Summary
MIT Lincoln Laboratory
~1 Gbps TCP/IP Demonstration
•
•
TCP/IP data transmitted from
Boston to Washington at 990
Mbps between two DEC Alpha
workstations
GBE
Transponder
850 nm ⇒1558 nm
Transponders convert the optical
network interface card output to a
transparent channel within
Bossnet
Albany
BOSTON
Providence, RI
New York, NY
Philadelphia, PA
•
•
Washington, D.C.
Bit rate-length product was 1.0
Tb-km/s (Initial results summer 2000)
TCP flow switching:
BOSSNET
Transponder
GBE
850 nm ⇒1558 nm
– Loopback Boston – New London
eVLBI-Workshop_Glownet&Bossnet
18
MIT Lincoln Laboratory
MTU & Receiver Window Tuning
Delay Path
[BOSSNET]
GbE
GbE
Data
Source
Host 1
Data
Sink
Transponder 1
Transponder 2
Host 2
Typical
eVLBI-Workshop_Glownet&Bossnet
19
MIT Lincoln Laboratory
Onramp Overview
Features:
• WDM Feeder Ring
• Covers 100 - 1000 sq. mi
• 0.1 – 1 Tbps capacity
• Configurable Optics
• Electronic Routing
• IP over WDM
• High speed distribution
networks (1-10 Gbps)
• Optical switching
• Dynamic wavelength
provisioning
• Flow switching
• Unidirectional or
Asymmetric flows
• Protection from link
failures
Objective: Aggregation and distribution of data traffic from high rate
users, sensors, or LANs to other users, LANs or Backbone Networks
eVLBI-Workshop_Glownet&Bossnet
20
MIT Lincoln Laboratory
TCP Flow Switch Experiment
to Access Node #3
Access Node #1
Access Node #2
Control / IP
Control / IP
OADM 1
OADM 2
GbE
GbE
Router 1
Router 2
Fixed λ
Xponder
Fixed λ
Xponder
Delay Path
[BOSSNET]
Fixed λ
Xponder
Tunable λ
Xponder
Tunable λ
Xponder
Fixed λ
Xponder
Splitter
GbE
GbE
Workstation #1
eVLBI-Workshop_Glownet&Bossnet
21
Control / IP Path
Flow Switched Path
GbE
GbE
Workstation #2
MIT Lincoln Laboratory
TCP Flow Switching
Flow switch with standard TCP
implementation.
Bytes
Flow Switch
• Packets arrive out of order at receiver
• False congestion indication generated
• Congestion avoidance limits sender
rate – Degrades Throughput
Time [hh:mm:ss]
Bytes
Flow switch with modified TCP
implementation.
Flow Switch
• Time stamp option used to
recover from false congestion
indications
• Throughput degradation avoided
Time [hh:mm:ss]
eVLBI-Workshop_Glownet&Bossnet
22
MIT Lincoln Laboratory
Outline
•
•
•
•
•
eVLBI-Workshop_Glownet&Bossnet
23
WAN (Bossnet)
MAN (Glownet)
LAN
Processing and Protocols
Lessons Learned & Summary
MIT Lincoln Laboratory
Lessons Learned (so far)
•
Be patient
– Months are needed to implement a new point-point fiber link
with compatible network equipment
– Numerous "air gaps" must be closed
– Make sure someone maintains a detailed link and network
equipment block diagram (e.g., Alan Whitney)
– Test links and network equipment early
•
Use as much commercial equipment as possible
– Network equipment is almost standardized
– But jumbo Ethernet frames are not
•
Be prepared to rely on the generosity of others
– Loaned equipment will work as well as your own and usually
arrives sooner
•
Be generous sharing your own equipment
– You will ultimately be rewarded!
eVLBI-Workshop_Glownet&Bossnet
24
MIT Lincoln Laboratory
Looking Ahead
•
How to implement gigabit/sec services "on demand"?
– Global
– Metropolitan, e.g., Onramp Program
– Local
•
How to assure compatible networking and transport for all
intermediate links?
– jumbo frames
– flow switching
•
Plan now to get "last kilometer" and "last meter" fiber in
place
•
Plan now to be sure user equipment can maintain
gigabit/sec transfers end-to-end
– hardware
– protocols
eVLBI-Workshop_Glownet&Bossnet
25
MIT Lincoln Laboratory
Summary
•
Networking and transport technology is available for global
eVLBI
– but not always where you want it when you want it
•
Glownet and Bossnet enable early real-time eVLBI
experiments
•
Lessons have been learned
eVLBI-Workshop_Glownet&Bossnet
26
MIT Lincoln Laboratory
Backup
eVLBI-Workshop_Glownet&Bossnet
27
MIT Lincoln Laboratory
Bossnet Objectives
•
•
Provide an infrastructure for wide area optical link and network
research
Transmission
–
–
–
–
•
Networking
–
–
–
•
Demonstrate increased capacity through the use of novel waveforms and
advanced optical node technology (TDM and WDM)
Study propagation effects and mitigation techniques for very high rates
over long distances
Investigate performance of analog transmission
Maintain uninterrupted OC-48 transmission on one wavelength
Demonstrate wide area all-optical network management and control
techniques
Interconnect other Gbps testbeds over transparent link
Study impact of large Bandwidth-Delay products on transport protocols
Applications
–
Provide an infrastructure for Gbps research applications
eVLBI-Workshop_Glownet&Bossnet
28
MIT Lincoln Laboratory
Framingham Hut
eVLBI-Workshop_Glownet&Bossnet
29
MIT Lincoln Laboratory
Hut Hardware
Laboratory
eVLBI-Workshop_Glownet&Bossnet
30
230 Congress Street, Boston
MIT Lincoln Laboratory
Desirable Gbps LAN Properties
•
Network switches that support jumbo frames
– Maximum Transmission Unit (MTU) = 9216 bytes
•
Network equipment that supports out of band management
– Remove management traffic from data links
•
Network switches with diverse media interfaces
– GBICs allow SX and LX connections
– Copper RJ-45 for workstation connections
eVLBI-Workshop_Glownet&Bossnet
31
MIT Lincoln Laboratory
Glownet Infrastructure
Westford
HAX
GigE
VLBI
GigE
VLAN GigE Switch
LX
≈ 1 km
Utility & Display
10/100
Millstone (radar control)
Processing
Center
Computers
Transponders
MUX
2 GigE
(Jumbo frames)
Bossnet
to DC
230 Congress St.
Boston
10/100
Ethernet
Growth
OC-48 Data Mux
OC-48
Router
OC-48
Bossnet
Transponder
WDM
eVLBI-Workshop_Glownet&Bossnet
32
Wavelength
Lincoln VLAN GigE
Switched Infrastructure
MIT
Lincoln Laboratory
Millstone (RPE Building)
VLAN GigE Switch
VLAN GigE Switch
VLAN GigE Switch
HAY
MIT Lincoln Laboratory