Development of xTCA compliant TDAQ elements at IHEP Beijing
Abstract: This talk will be about xTCA compliant (and also ATCA, MTCA and AMC ) and FPGA
based Data Processing Units for trigger and data acquisition applications like in PANDA,
PXD/BelleII and Lumi/BESIII experiments. The ATCA Unit consists of 4 Advanced Mezzanine
Cards (AMC, called xFP card), 1 AMC carrier ATCA board(ACAB) and optionally(for xTCA
compliant) 1 Rear Transition I/O Board(RTM ). The ACAB board features 1 Xilinx Virtex-4 FX60
FPGA chip and 2GBytes DDR2 memory for data buffering and switching and the xFP board
features 1 xilinx Virtex-5 FX70T FPGA chips and 4GBytes DDR2 memory for data processing.
The connection between ACAB board and four xFP boards are by RocketIO port and other LVDS
I/O pairs. 8 optical links by 4 xFP2(with two 6Gbps optical IO) cards provide an input bandwidth
of 48Gbps and 16 optical link by 4 xFP4(with four 4Gbps optical IO) cards provide an highest
input bandwidth of 64Gbps. Optical links can either from panel of AMC card or from RTM card. 5
Gbit Ethernet links are provided for output to higher level trigger or for storage. A single ATCA
shelf can host up to 14 boards interconnected via a full mesh backplane. MTCA type double width
AMC unit for BESIII Luminosity readout will also be described. A prototype system has been
set up and some functions tests have been done and will be reported and discussed.
1.
The structure of the xTCA Compliant and FPGA based Data Processing Unit
This unit has been developed in the frame of PICMG xTCA physics standard activities at IHEP Beijing.
The Aim of this project is The Unit consists of one ATCA carrier board and four Advanced Mezzanine
Cards, and one rear board as shown in Fig.1. The carrier board(Computer Node board) features 1
Xilinx Virtex-4 FX60 FPGA chip and 2GBytes DDR2 memory. The Advanced Mezzanine Card (xFP board)
features 1 xilinx Virtex-5 FX70T FPGA chips and 4GBytes DDR2 memory. And the rear board features 8
SFP connectors, four Gbit Ethernets. The one Computer Node bard, four xFP boards and one rear
board are co-connected by RocketIO port and other LVDS I/O pairs. High bandwidth is designed by 16
optical links and 5 Gbit Ethernt links to receive data from front-end electronics and transmit the data
passed data processing unit and high level trigger to storage.
SFP 1#
ZONE3
RTM Planel
MMC
AMC #1
Isolater
DDR2
MMC
SFP 2#
SFP 3#
SFP 4#
SFP 5#
Clocks
+
Distribution
Isolater
AMC #2
SFP 6#
SFP 7#
FLASH
SFP 8#
CPLD
AMC #3
MMC
Isolater
PHY1
ZONE2
V4 FX60
JTAG
Power
Management
12V
RocketIO
Clock
Gbit
Ethernet
Ethernet
DDR2
PHY4#
DC/DC
12V
DC/DC
PHY2
ZONE1
AMC #4
Isolater
PHY2#
PHY3#
IPMC
IPMB-L
MMC
PHY1#
Gbit
Ethernet1#
Gbit
Ethernet2#
Gbit
Ethernet3#
Gbit
Ethernet4#
UART
JTAG
JTAG
IPMC
5V, 3.3V, 2.5V … 0.9V
Neighbor Link +RIO
Neighbor Link
Fig.1. The structure of the xTCA and FPGA based Data Processing Unit
Fig. 2 The xTCA and FPGA based Data Processing Unit
2.
The Computer Node
The Computer Node consists of Xilinx Virtex4 serial FX60 FPGA chip, one DDR2 memory, four AMC
conncetors, one Gbit Ethernt, IPMC part and power supply part. It offer the co-connecting of four xFP
board and rear board, the power supply for xFP board and rear board, power management and board
station detection by IPMC, FPGA configuration on xFP board .
3.
The xFP board
The xFP(Processing unit based on FPGA and xTCA) board consist of one Xilinx Virtex5 serial FX70T,
two 2GB DDR2 memorys, one platform flash for configuring the FPGA, one Gbit Ethernet, one UART
for board testing, four SFP connector data line rate up to 3Gbitps, one MMC module for power
management, station detection and communication with IPMC, shown in Fig. 3. And the physical
figure of the xFP board is shown in Fig. 4.
2xDDR2
Oscillation
Optical transceiver
Optical transceiver
@2/3.125Gbps
Virtex-5
Optical transceiver
Optical transceiver
Gbit Ethernet
10/100/1000
PHY
Sync. clock
Platform
FLASH
JTAG
FLASH
MMC
RocketIO
Clock
Ethernet
DDR2
JTAG
IPMC
Neighbor
Link +RIO
3.3V, 2.5V … 0.9V
Power
Supplies
UART
Fig. 3 The structure of the xFP board
AMC Edge Connector
UART
Fig. 4 The xFP baord
4.
Some testing result
Some work has been done on testing the carrier board and xFP board. The carrier board now can
be power on and FGPA can download configuration file correctly. Further testing work has been done
on xFP board.
(1) RocketIO and GTP test.
Bit error rate (BER) which shows the reliability of the system is a key parameter in the high speed data
BER 
transmission. It can be calculated using formula 1
Ne
N t , which Nt is the number of the total data
the system transmitting, Ne is the number of error data. If zero error data detected, set Ne to 1.
The sending module sends the data which generated by GTP IP core and saved in a RAM to the
receiver module. It through the parallel to serial conversion in sending part, optical fiber and serial to
parallel conversion and reach the receiver part, and be compared with the data saved in the RAM.
Then the Ne will be given by the test module. Fig. 5 is the test figure that be obtained by Chipscope.
The line rate of the GTP is set to 4Gbitps. Testing time is 24 hours.
-15
calculated that the BER is less than 2.89×10
No error data was obtain. It is
.
发送数据
出错个数
接收数据
Fig. 5 BER testing result
5.
Summary
The carrier board’s power supply works well. And it can be configured correctly. The xFP board’s
GTP work well. Further testing work need to be done. Rear board is under designing. The xTCA new
standard program has been used in BelleII PXD (Pix detector) Trigger and DAQ(Data acquisition)
system, PANDA Trigger and DAQ(Data acquisition) system and BESIII Luminosity project.