SiTCP and possibility of collaboration
for
Accelerator Control and Data Acquisition
T. Obina, KEK
16/Jun/2011 EPICS Collaboration Meeting, NSRRC, Hsinchu, Taiwan
Purpose of Discussion
• Short Introduction of “Silicon TCP” (SiTCP)
• Discuss possibility of collaboration/knowledge
sharing
• This slide is only a memo
What is SiTCP?
• TCP on FPGA
– You can connect FPGA directly to the Ethernet
– Without Hard/Soft core CPU As fast as hardware limit
• Developed by Tomohisa Uchida
– KEK, Physics Department
• Paper
– IEEE Transactions on Nuclear Science. vol. 55, no.
3, 2008.6, pp. 1631-1637
Dr. Uchida’s Motivation
• Why he developed it by himself?
• It is very dangerous if you depend on a specific
company’s technology.
• Minimize the effect of “Trend” or “Industry World”.
• For Example,
– IP core supplied by a company  He found a bug, and
report it. But the company did not fix it.
– Original interface of a company  No compatibility
between old and new product.
– Developed by a tool of a company  Discontinued. Most
resources cannot be used anymore.
What is “Next Standard” Bus?
• CAMAC/VME/PCI/cPCI
• PCIe? xTCA ?
• No one knows the future.
• However, TCP may not die.
SiTCP characteristics / specification
• Looking from FPGA Side, it looks like a FIFO
• Looking from Client(PC) side, it looks like a Server
• Small footprint
• Primary development target
– Data Acquisition for Physics (High Energy) Experiment
Example
"Virtual Pipe Network"
of the data can be
achieved
FPGA
Detector
Analog Front-end
Circuit
SiTCP
PMT, etc
S
W
Detector
Analog Front-end
Circuit
PC
SiTCP
PMT, etc


Extensible (Can process many channels)
No data loss, High throughput is important
Interface to SiTCP

SiTCP has Two interfaces


Fast FIFO
Slow Control
: use TCP
: use UDP
like Data Bus
like Control Bus
FIFO for TCP
SiTCP
GMII
Ethernet
PHY
Slow Control
Initial development started for Virtex 2
(100Mbps) and moved to Virtex 4 (1Gbps)
Refer to Paper
Simple Test
My Test setup
Laboratory LAN
Note PC (Fujitsu)
WindowsXP
or
Linux(kernel2.6)
PC
Ethernet Client
PC
Development
192.168.0.10
RS232
1Gbps Ethernet
(crossover cable)
JTAG
192.168.0.16
ML403
Evaluation Board
Desktop PC
Windows XP
ML403 Evaluation Board
FPGA Xilinx Virtex4 (XC4VFX12)
TestData FIFO
Generator
SiTCP
(Server)
GMII
PHY
Marvell
Alaska
88E1111
1000BASE-T
PC
Result : Dependence on recv buffer size
SiTCP Transfer Rate
1000
with Linux, 948Mbps are
achieved without any tuning
900
800
700
Rate [Mbps]
600
Win/with Firewall
Win/No Firewall
Linux
500
400
300
200
100
0
0
500
1000
1500
2000
Buffer Size [Bytes]
2500
3000
3500
Result : Long term stability, distribution, etc
Measure the transfer rate for 20min
(measured as 10 sec average transfer rate)
Distribution
Frequency in 1200 sec (10 sec ave)
Time vs TransferRate (10sec interval)
1000
70
900
60
800
50
600
Frequency [cnt]
Trans fer Rate [Mbps ]
700
500
400
300
40
30
20
200
10
100
0
0
0
200
400
600
800
1000
1200
1400
945
946
947
948
949
Trans fer Rate [Mbps ]
Time [s ]
Measured on Linux
950
951
952
Result : 1 sec average
Frequency (1sec ave)
Transfer Rate (meas in 1sec)
960
30
955
25
950
945
20
Frequency [cnt]
Transfer Rate [Mbps]
940
935
930
925
15
10
920
5
915
910
0
0
20
40
60
80
100
Time [s]
120
140
160
180
200
Not on the realtime OS.
920
930
940
Transfer Rate [Mbps]
950
960
Application
• Readout for Neutron detectors
• Super Kamiokande
• Neutron Imaging Detector,
and many physics experiments …..
• Recently, Synchrotron Radiation user is
contributing. (Dr. Kishimoto, KEK-PF)
Can we use it for accelerator control?
• Academic use
• Open
Question / Request
• Xilinx is supported. How about Altera?
– For now, SiTCP is spplied as netlist for Xilinx
– Maybe possible, but needs time…. Confirm it later.
• Speed : 1G is enough?
– 10G is better, 1G is still quite useful
• Using build-in SERDES, we can interface optical
fiber
• Latency
• Number of Server Session : 1
• SiTCP Client
• Radiation Hardness?
Possibility of Collaboration
• I don’t block. (I’m not an representative of Dr. Uchida)
• I think it might be a good idea to have ‘small community’
for “SiTCP for accelerator control and data acquisition”
• Dr. Uchida is very powerful person, but human.
• Mailing List is better to avoid many people ask same
question to him. Let’s share the knowledge.
• Wiki
– discussion, summary, FAQ, documentation, etc
• License, NDA, Terms of use ? … maybe not. confirm.
• SiTCP on Open Source Hardware
• Send Open Source to Dr. Uchida ?
– Information at first. If he has interest, Hardware
• Client  FPGA data transfer
• Measurement of Latency
• In some detector application, 10G is already
used.
• Possibility to open the source code in the
future.
References
• SiTCP Homepage
– http://e-sys.kek.jp/tech/sitcp/ (Japanese)
• Open Source Consortium of Instrumentation
– http://openit.kek.jp/
• Bee Beans Technology
– http://www.bbtech.co.jp/