Evaluation of the LDC
Computing Platform for Point 2
SuperMicro X7DBE-X
Andrey Shevel
CERN PH-AID
ALICE DAQ (Internal report)
CERN
November 2006
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
1
Purpose
 Background:
 Test the machine Elonex Supermicro based on
motherboard X7DBE-X (2xDual Core Xeon 3.0 GHz) as
LDC with 6 D-RORCs
 Evaluation program:
Linux installation.
DATE V5 installation.
D-RORC throughput with DATE.
Perform a range of runs with internal D-RORC
generator and receiving through DATE.
 Perform a range of long runs (two days each) in
configuration with the test machine as LDC and
another machine as GDC.
 Quick comparison with another machine based on
motherboard X6DHE-XB (Dual Xeon 2.8 MHz).




4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
2
Architecture of motherboard
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
3
Supermicro X7DBE-X
CPU
2 x Dual Core Intel Xeon, cache L2 –
total 4 MB
Chipset
Intel 5000P (BlackFord)
FSB
1333 MHz
Memory
2 GB (max 32GB), 533 MHz,DDR2
3x 64-bit 133MHz PCI-X,
I/O
3x PCI-X (64-bit 133 MHz)
3x PCI-X (64-bit 100 MHz)
Network
2x GbE, Intel 631xESC/632xESB
Disk
120 GB, SATA
Graphics
ATI Technologies Inc ES1000
Periphery
2x USB front, 2x USB back
Serial, PS/2
Chassis
4U (tower)
pcald44.cern.ch
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
4
Software Installation
 Linux
 SLC4.4 with kernel 2.6.9-42.0.3.EL.cernsmp
no specific problems: USB boot successful, BIOS – everything is OK
 To use D-RORC modules the BIOS setting is needed to be changed: all
PCI slots were set to 100 MHz
 DATE 5.25 installation
 MySql
 mysqltcl
 BWidget
 SMI
 DIM
 Drivers: rorc and physmem
have been installed as rpm with additional scripts:
DATE_basic_Install-SLC4x.bash, DATE_User_Install-SLC4x.bash, DATE_DriversInstall.bash
 No specific problems
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
5
Common info about measurement
 Total main memory volume = 2 GB
•
1 GB for Linux (in /etc/grub.conf)
•
1 GB for physmem
 6 cards D-RORC rev 4 (internal data generator).
 All measurement were done with DATE 5.25. No received event
data were recorded to HD. The data from the database were
analyzed with scripts. The pictures were built up with gnuplot.
 In total there were performed 390 measurement runs. Because
the figures (rates, bandwidth) are fluctuating every run was
continued at least 5 minutes.
 Two points: random event size with max=10**4 and max=10**6
Bytes were measured during two days each (LDC on test machine,
GDC on different machine).
 All measurements with randomly distributed event size were
performed with event checking in DATE.
 D-RORC Page Size was 10**5 Bytes.
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
6
Event rate of one card
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
7
Bandwidth of readout of one card
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
8
Example of editDb page
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
9
Example of Memory Banks page
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
10
Event rate for several cards
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
11
Bandwidth for several cards
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
12
Event rate: ConsistencyCheckLevel
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
13
Bandwidth: ConsistencyCheckLevel
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
14
Event rate: NumberOfCards, ConsistencyCheckLevel
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
15
Bandwidth: NumberOfCards, ConsistencyCheckLevel
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
16
Event rate: D-RORC Page Size
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
17
Bandwidth: D-RORC Page Size
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
18
X6DHE-XB vs X7DBE-X
Parameter name
X7DBE-X
2x Xeon 2.8 GHz, 1 MB
2x Xeon 5160 dual core, 3.0
GHz, 4 MB L2 cache
Chipset
E7520
Intel 5000P (BlackFord)
FSB
800 MHz
1333 MHz
Memory
2 GB, DDR 333
2 GB, DDR-2 533 MHz
3x PCI-X (64-bit 100 MHz)
PCI I/O
2x PCI-X (64/133)
4x PCI-X (64/100)
1x PCI-e (x4)
Network
2x GbE, BCM5721
2x GbE, Intel 631xESB/632xESB
CPU type
4 Dec 2006
X6DBE-XB
3x PCI-X (64-bit 133 MHz)
SYSBENCH Memory
1.5 GB/s
bandwidth
3.4 GB/s
RootMarks
1500
614
Testing the machine (X7DBE-X) with 6 D-RORCs
19
X6DHE-XB vs X7DBE-X (continuation)
Parameter name
X6DBE-XB
Event rate for 1 D88.9 KHz
RORC with the
size 1 KB
Event rate for 6
D-RORCs with
size 1 KB
Bandwidth of
readout for 6 DRORCs with size
10**6 Bytes
4 Dec 2006
X7DBE-X
87.2 KHz
6 KHz
12.2 KHz
1.58 GB/sec
1.98 GB/sec
Testing the machine (X7DBE-X) with 6 D-RORCs
20
Recommendation and conclusion
 Recommendation: if you need to allocate D-RORC cards in the machine and
like to have maximum bandwidth for X7DBE-X:
• it is better to avoid to use PCI slot
4 & PCI slot 5 at the same time for high bandwidth
information sources.
 Conclusion
 In the test it was shown bandwidth (1.96 GB/sec) when all 6 D-RORC cards are in
operation and event fragment size is more 10**5 bytes when ConsistencyCheckLevel=0.
 X7DBE-X shows increased event rate and bandwidth with 6 D-RORCs, also it has more
CPU power in comparison to X6DHE-XB.
 The configuration LDC on test machine and GDC on another machine shown about the
speed 118 MB/sec with no error during two days test.
 Randomly distributed event fragment size with maximum 10**6 bytes gives total
bandwidth 1.53 GB/sec during two days run.
 The machine on the base X7DBE-X is excellent to be used as LDC with 6 D-RORC cards.
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
21
Thank you!
Klaus Schossmaier, Sylvain Chapeland, Ulrich Fuchs
and
PH/AID crew
4 Dec 2006
Testing the machine (X7DBE-X) with 6 D-RORCs
22