Erlang in global radio astronomy
Controlling embedded hardware
Harro Verkouter/Joint Institute for VLBI in Europe
Joint Institute for VLBI in Europe
Very
Long
Baseline
Interferometry
Ever larger telescopes
Electromagnetic radiation
wavelength
diameter
pixelsize
 (m)
D (m)
/D (rad)
smallest optical
10-7
10-1
10-6
biggest radio
10-1
100
10-3
A neat trick
100m
A neat trick
20m

100m
correlator
A different ball game!
 (m)
D (m)
/D (rad)
largest optical (IR)
10-6
10
10-6
VLBI (radio)
10-1
107
10-8
Science
Science
Recording the data
Record up to 4096 Mbps (512 Mbyte/s)
~n2
Number of combinations to be computed as function of n
16 telescopes
at
1024 Mbps
UniBoard
4 frontnode
4 backnode
QuickTime™ and a
decompressor
are needed to see this picture.
>20 Gbps/link
16x 10Gbps
16x 10Gbps
1Gbit/s ethernet
Erlang?!
• typical languages used
– 80% C++
– 15% Python
– PHP, Fortran, perl
• database
– MySQL
• needed to convince management
– pilot projects!
Decoding data files
• VLBI has framed data format(s)
QuickTime™ and a
decompressor
are needed to see this picture.
• Look for ‘syncword’
– different flavours, same syncword, different framesize
• Other format uses different syncword
(not quite) surprising little code
• 212 lines of code
– first dabblings ...
•
•
•
•
about a week’s work
recognizes + decodes 3 formats
decodes the BCD timestamps
supports distributed system
– reading file on one machine
– decoding on other machine
• management duely impressed
– got permission to do a larger pilot!
Soft real time monitoring website
• webcams at stations
– webpage with http links = #FAIL
– unreliable links
• really high latency (China, Chile)
• don’t want broken images
– low bandwidth
• cache in Europe: multiple users eat all bandwidth
• parse telescope logfiles
– immediate antenna calibration info
– generate timeseries plots as observation progresses
• stored in MySQL
Soft real time monitoring website
• webcam URLs + status stored in MySQL
– supervisor starts webcam monitors
• one gen_server for each webcam
– loads info from MySQL
– uses http:request/1
– strict checking on return value
• only HTTP 200 status replies
• content_length header set?
• explode if problem
– use a port to ImageMagick to scale image
– write binary JPEG data into mnesia
Soft real time monitoring website
• monitor all participating telescopes
• telescope log file (ASCII text)
– retrieved via python + ssh
– read into Erlang using a port
– low data rate after start up
• parses lines
– update MySQL
• generate plots using gnuplot
–
–
–
–
for each telescope
query MySQL every so many minutes
using a port
write .png binary data to mnesia
Soft real time monitoring website
• patch it all together using webtool
• cache complete webpage in mnesia
– too old? generate new one
• serve .png and .jpg data from mnesia
– image URL points back to webtool
• extremely simple to cache + serve data
– brilliant idea: mapping URL to erlang fn call!
• in total used ~3000 lines of code
– webcam, logfile monitoring; plotting; serving web pages
– excellent test case for OTP and distributed system
• got the green light for the real thing!
VHDL
Hardware
Software
Nios2
CPU
Nios2
CPU
1Gbit PHY
1Gbit PHY
FPGA
FPGA
Nios2
CPU
Nios2
CPU
1Gbit PHY
1Gbit PHY
FPGA
windows/unix
Control Computer
FPGA
EthernetSwitch
UniBoard
EthernetSwitch
VHDL
Hardware
Software
Nios2
CPU
registers
Polyphase Filterbank
registers
registers
Delay Module
registers
1Gbit PHY
FPGA
VHDL
Hardware
Software
Server
Nios2
CPU
UDP/IPv4 over ethernet
Client
windows/unix
Control Computer
Simple binary protocol
Read N 32bit words starting from START ADDRESS
32 bits
32 bits
32 bits
0x01
N
START ADDRESS
(Over)Write N 32bit words from the packet to START ADDRESS
32 bits
32 bits
32 bits
N*32 bits
0x02
N
START ADDRESS
N 32 bit words of DATA
Read/Modify/Write N 32bit words from START ADDRESS + packet to START ADDRESS
32 bits
32 bits
32 bits
N*32 bits
OPCODE
N
START ADDRESS
N 32bit words of MASK
OPCODE
0x03
0x04
0x05
BITWISE OPERATION
AND
OR
XOR
QuickTime™ and a
decompressor
are needed to see this picture.
FPGA client library
one controller  one fpga
Registermap:
• list of register definitions
• type of the register (details follow)
• name of the register (symbolic)
• address of the register in the memory map
FPGA client library
Available registerdefinition types for the controller
fpga:bit( <name>, <bit #>, <address> )
fpga:bitrange( <name>, <startbit>, <nbit>, <address> )
fpga:word( <name>, <address> )
fpga:range( <name>, <# words>, <address> )
FPGA client library
Available register commands
fpga:read( <name> )
fpga:write( <name>, <value> )
fpga:or_( <name>, <value> )
fpga:and_( <name>, <value> )
fpga:xor_( <name>, <value> )
Example
-module(firfilter). %% this module is called firfilter, driving one of those
-behaviour(personality).
-export([registers/0, start_filter/2]). %% this module implements an FPGA personality
%% Called automatically by the FPGA control framework
registers() ->
{ok, [
fpga:bitrange(num_tap, 5, 3, 16#24),
fpga:bit(pps_enable, 28, 16#24),
fpga:word(control_status, 16#abcc),
].
%% define a high-level command for this personality
start_filter(FPGA, NumTap) ->
%% disable the PPS
fpga:execute(FPGA, fpga:write(pps_enable, 0)),
%% read the control status register
case fpga:execute(FPGA, fpga:read(control_status)) of
1 -> fpga:execute(FPGA, fpga:xor(control_status, 42));
_ -> ok
end,
%% however, you can easily execute a number of commands in one go:
fpga:execute(FPGA, [fpga:write(num_tap, NumTap),
fpga:write(pps_enable, 1)]).
Safety
2> fpga:execute(FPGA, fpga:write(pps_enable, 2)).
OH NOES! error:"Value '2' does not fit in one bit (pps_enable)"
FPGAControl terminating [{{badmatch,
{reply,
{error,
{"Value '2' does not fit in one bit (pps_enable)",
[{fpga,err,2,
[{fil
Full correlator control system
• besides single fpga control
– parallel control of eight fpga’s
• parsing + validating configuration files
– using yecc
• decoding + sending over UDP of binary data files
– data as well as polynomial coefficients
•
•
•
•
capture network data to file
configure and control the hardware
MySQL read/write
two weeks VHDL coding = < 1 day Erlang coding
Full correlator control system
data reader
data reader
UniBoard
UniBoard
data reader
command/control
data writer
The good
• cross platform
– we have Linux, Mac OSX and Solaris
• distributed
– typically many machines involved
• non-variability of ‘variables’
• conciseness / amount of work per LOC
• connects to MySQL
– not easy to connect other languages to mnesia
• hard crash on errors
– about the most valuable property: can’t ignore!
• binary pattern matching
– every language should be jealous of this
• re-use of development/testing code
– it’s so easy to re-use code!
The bad
• our VHDL engineers struggle
– language simple, but
– concepts like recursion + distribution are ‘hard’
• conciseness / amount of work per LOC
– takes effort to read code written by someone who’s better at
it than you
• deploying a developing distributed system
– we haven’t found a simple way yet
– dynamic remote code loading could be awsome
• it works for individual modules
• useless if you have dynamic dependencies ...
– erlang:apply/3
Thank you for your attention!