e-VLBI Developments at JIVE
Arpad Szomoru
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Outline
•
•
•
•
•
Current status, recent results
Network upgrades
EXPReS: first results
Connectivity improvements
The future..
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
1 Gbps
10 Gbps
155 Mbps
EVN Symposium 2004, A. Szomoru, JIVE
2.5 Gbps
Recent developments
• Regular science/test sessions throughout the year
• First open calls for e-VLBI science proposals
• First science run completely lost, but, first ever real-time
fringes to Mc (128 Mbps)
• Second and third science runs: many hours of smooth
sailing at 128 Mbps
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Current status
• Technical tests:
•
•
•
•
6-station fringes at 256 Mbps
first European 512 Mbps fringes (Jb and Wb, May 18)
3-station 512 Mbps fringes (Cm, Wb, On, August 21)
first fringes using new 5 GHz receiver at Mc
• Current connectivity:
• Ar: 64 Mbps in the past, but <32 Mbps this year
• European telescopes: 128 Mbps always, 256 Mbps
often, 512 Mbps to Wb, Jb and On
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Tr connectivity bottleneck - solved
Black Diamond 6808 switches:
New interfaces (10GE) system in old architecture (1GE)
• Originally 8x1GE interface per card
• 10GE NIC served by 8 x 1GE queues
• Queuing regime – RR (packet based) and flow-based
Flow based:
Max. flow capacity – 1Gbit/s – backround traffic.
There is no known reordering workaround to solve this problem.
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
e-VLBI to South America? SMART-1
SMART-1 factsheet
Testing solar-electric propulsion and other deep-space technologies
Name SMART stands for Small Missions for Advanced Research in
Technology.
Description SMART-1 is the first of ESA’s Small Missions for Advanced
Research in Technology. It travelled to the Moon using solar-electric
propulsion and carrying a battery of miniaturised instruments.
As well as testing new technology, SMART-1 is making the first
comprehensive inventory of key chemical elements in the lunar surface. It
is also investigating the theory that the Moon was formed following the
violent collision of a smaller planet with Earth, four and a half thousand
million years ago.
Launched 27 September 2003
Status Arrived in lunar orbit, 15 November 2004. Conducting lunar orbit
science operations.
Notes SMART-1 is the first European spacecraft to travel to and orbit
around the Moon.
This is only the second time that ion propulsion has been used as a
mission's primary propulsion system (the first was NASA's Deep Space 1
probe launched in October 1998).
SMART-1 is looking for water (in the form of ice) on the Moon.
To save precious xenon fuel, SMART-1 uses 'celestial mechanics', that is,
techniques such as making use of 'lunar resonances' and fly-bys.
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
e-VLBI to Sourth America (2)
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
And other continents..
Australia:
• Telescopes connected
• PCEVN-Mk5 interface needed
China:
• Shanghai Observatory connected at 2.5Gbps
• Connection via TEIN (622Mbps), ORIENT?
Issues with CERNET, CSTNet
• Direct lightpath Hong Kong-Netherlight?
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Hybrid networks in the Netherlands..
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Switch from Cisco to
Nortel/Avici equipment has
been completed: for now, 7
* 1 Gbps, ultimately 16 * 1
Gbps.
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
..and across Europe: GÉANT2 network upgrade
Outline
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
EXPReS: now what?
SA1: new hires at JIVE (two software engineers, one network
engineer, one e-VLBI postdoc)
Inclusion of e-MERLIN telescopes in e-EVN
Operational improvements (deliverable driven):
• Robustness
• Reliability
• Speed
• Ease of operation
• Station feedback
And still, pushing data rates, protocols, UDP, Circuit TCP?
Get rid of fairness… Better usage of available bandwidth.
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
e-VLBI control interface
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Interface to station Mk5s
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Runtime control
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Integrating fringe display
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Data status monitor
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Streamlining of post processing
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Web-based Post-processing
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Ongoing
New control computers (Solaris AMD servers)
• Cut down on (re-)start time
• Powerful code development platform
• Tightening up of existing code
• Operational within next few weeks
Other hardware upgrades:
• SX optics (fibres + NICs), managed switches at JIVE and
SARA
• Mark5A→B: motherboards, memory, power supplies, serial
links, CIBs
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
FABRIC:
• Distributed software correlation
• High bandwidth data transport (On part of e-MERLIN @ 4Gbps)
• Two new hires at JIVE
And coming..
SCARIe:
• Collaboration with SARA and UvA
• Distributed software correlation using Dutch grid
• Lambda switching, dynamical allocation of
lightpaths
• JIVE postdoc hired, still looking for UvA postdoc
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
FABRIC components
GRID
resources data
observing schedule
in VEX format
DBBC
VSI
PC-EVN
#2
field system
controls antenna
and acquisition
user correlator
parameters
earth orientation
parameters
resource allocation
and routing
VSIe??
on??
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
correlator control
including model
calculation
FABRIC
=
The GRID
output
data
Martin Swany
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
•
•
•
•
•
2 Heavy duty gamer PCs
Tyan Thunder K8WE Motherboards
Dual AMD Opteron 2.4GHz processors
4GB RAM
2 1Gb PCI-Expres Nics
• First one at Torun, back-to-back to Mark5
• Second one located at Poznan Supercomputing Centre
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Protocols (1)
Mix of High Speed and Westwood TCP (Sansa)
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Protocols (2)
Circuit TCP (Mudambi,
Zheng and Veeraraghavan)
Meant for Dedicated End-toEnd Circuits, fixed
congestion window
No slow start, no backoff:
finally, a TCP rude enough
for e-VLBI?
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Protocols (3)
Home-grown version of CTCP using pluggable TCP congestion
avoidance algorithms in newer Linux kernels (Mark Kettenis)
Rock-steady 560 Mbps transfer using iperf from Mc to JIVE
Serious problem with new version of Mk5A software under
newer kernels
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
e-EVN: the future
• Aim: 16 * 1 Gbps production e-EVN network
• IP: not possible/affordable
• 10 Gbps lightpath across Europe: currently ~20k€/year
• Lightpaths across GÉANT terminating at JIVE
• If possible, all the way from telescopes. If not,
overprovisioned IP connections from telescopes to
GÉANT, lightpaths from there on.
• Guaranteed bandwidth, possibility to use ethernet frames,
no more need to worry about congestion..
• A true connected-element interferometer
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
Proposed connection Surfnet-JIVE
External LPs
GEs, STM-64 direct
Switch
N GE
OME
network
Mk5
OME
network
Dynamic capabilities
through DRAC
10 G IP
5th e-VLBI Workshop, Haystack Observatory, September 2006, A. Szomoru, JIVE
N GE
Switch
Mk5
Mk5