Matt Strong, Ralph Spencer, Richard Hughes-
Jones, Simon Casey, Paul Burgess
June 2006

• Introduction to Radio Astronomy and VLBI
• Protocols – an important choice
• UDP Tests
• TCP Tests
• The Shortcomings of the Mark5 machines
• First Open Call e-VLBI Science Run
• Future Work and Development

An Introduction to Radio
Astronomy and Interferometry
• The study of celestial objects at <1 mm to >1m wavelength.
• Resolution:  

D
• High resolution achieved by interferometers.

The Basics of Radio Interferometry
– Aperture Synthesis
• Large synthesised telescope

• Resolution:
 
B
L
• Data from each telescope multiplied (correlated with every other telescope)
• As Earth rotates, the synthesised aperture is filled
• Need~ 12 hours for full synthesis, not necessarily collecting data for all that time

Very Long Baseline Interferometry
• 18 individual antennae all over
Europe
• Large distances involved mean direct transport of data has been prohibitive in the past
• Data recorded to tape and now disk at rates of 512 MBit/s

• Electronic transfer of data through computer networks to the correlator
• Advantages
– Cheaper (in the long run)
– More responsive
– More reliable
– More sensitive
• Disadvantages
– It isn’t easy to move Terrabytes of data around
Europe (and even the world)

EVN-NREN
MERLIN
Cambridge
UK
Jodrell Bank
UK
Gbit link
Chalmers
University of
Technolog y,
Gothenbur g
Gbit link
Onsala
Sweden
Torun
Poland
Medicina
Italy
Dwingeloo
DWDM link
Dedicate d
Gbit link
Westerbork
Netherlands

The Choice of Transmission
Protocol?
• The choice of transmission protocol is very important in the development of e-VLBI
• There are many choices
– TCP (Currently used in e-VLBI system)
– UDP
– DCCP
– SCTP
• Perhaps the use of Application Protocols could help
– Tsunami
– UDT
– VSI-E
– etc

UDP Tests
Tests on the UKLight dedicated light-path: Manchester to Dwingeloo
• Throughput as a function of interpacket spacing (2.4 GHz dual
Xeon machines)
• Packet loss for small packet size
• Maximum size packets can reach full line rates with no loss, and there was no re-ordering (plot not shown).
1000
900
800
700
600
500
400
300
200
100
0
0 gig03-jiveg1_UKL_25Jun05
10 20
Spacing between frames us
30 40
100
10
1
0.1
0.01
0.001
0.0001
0 gig03-jiveg1_UKL_25Jun05
10 20
Spacing between frames us
30 40
50 bytes
100 bytes
200 bytes
400 bytes
600 bytes
800 bytes
1000 bytes
1200 bytes
1400 bytes
1472 bytes
50 bytes
100 bytes
200 bytes
400 bytes
600 bytes
800 bytes
1000 bytes
1200 bytes
1400 bytes
1472 bytes

Custom Engineered UDP Based
VLBI Software – VLBI_UDP
• Originally developed by Richard Hughes-Jones
– Sent multiple, continuous memory-memory UDP streams
– Multi-threaded, contained ring buffer
– Web interface to manage streams & graph throughput
• Modified to allow disk-disk transfers
• Currently implementing selective packet dropping

GEANT 2 Launch – The debut of
VLBI_UDP

2 streams into JIVE, March 2006

• Current e-VLBI setup uses TCP transmission between
Mark 5 end systems
• Mark 5 end systems are custom built PCs which can record to disk pack or transmit data over the network
• Goal is to achieve 512 Mbit/s from 5 outstations to JIVE
– Jodrell Bank
– Onsala
– Westerbork
– Torun
– Medicina
• Both Onsala and Westerbork achieved 512 Mbit/s, and
Torun and Medicina achieved 256 Mbit/s
• Jodrell could transmit at upto 500 Mbit/s but could not reach 512 Mbit/s threshold
• Why?

UKLight using Mk5 recording terminals

The Shortcomings of the Mark 5
Machines
• All Mark 5 machines are PIII, 1.2 GHz machines
• TCP iperf throughput Onsala Mark 5 to
JIVE ~960 Mbit/s
• TCP iperf throughput JB Mark 5 to
Manchester ~960 Mbit/s
• TCP iperf throughput JB Mark 5 to JIVE
680-800 Mbit/s
• Why?

Investigating Mark 5
Performance
• Performance tests from
Jodrell Mark 5 to
Manchester (gig7) and
JIVE (Jivegig1)
• Tests from Jodrell to
Manchester show better throughput than Jodrell to
JIVE
• JB – Manc: 94.7% kernel mode, 1.5% idle - just OK
• JB – JIVE: 96.3% kernel mode ,0.06 % idle - There is no spare CPU power left.

Investigating Mark 5
Performance II
• What Happens when we run a
CPU intensive task whilst writing to the network
• JBO – Manchester Tests
• TCP throughput falls as priority of task increases
• % Kernel Mode drops as %
Nice and priority of task increases
• CPU mode shares with % Nice
• No losses or timeouts recorded
• Higher Achievable TCP bandwidth required more CPU power in kernel mode
• Mark 5 machine does not have sufficient CPU power to drive the network at line speed

• 1 st Jodrell Mark 5 machine upgraded to
Xeon 2.8 GHz processor and Asus
Motherboard
• Tested over UKLight to JIVE and gives
512Mbit/s transmission
• 2 nd Jodrell Mark 5 machine upgraded and this also gives 512Mbit/s transmission

Investigating Mark 5
Performance III – The Upgraded
Mark 5
• JBO – Manchester Tests with upgraded Mark 5
• TCP throughput constant as priority increases
• %Kernel and %Nice constant
• No losses or time outs
• Can drive network at line speed – 512 Mbit/s e-
VLBI data transfer successful

1 st Successful Open Call eVLBI
Science Run 20/21 April 2006
Jb,Cm,Wb,On,Mc,Tr

• USA tests
• VLBI_UDP development and tests
• Correlation Tests (packet dropping)
• Protocol Investigation/tests
• Continued development of eVLBI

eVLBI Demonstrations - IGRID and
SC 2005
• iGRID 2005 and SC 2005
– Global eVLBI demonstration
– Achieved 1.5 Gbps across Atlantic using UKLight
– 3 VC-3-13c ~700 Mbps SDH links carrying data across the Atlantic from Onsala, JBO and
Westerbork telescopes
– 512 Mps K4 – Mk5data from Japan to USA
– 512 Mbs Mk5 real time interferometry between
Onsala, Westford, Maryland Point antennas correlated at Haystack observatory
– Used VLSR technology from DRAGON project in
US to set up light paths.