Status report on DAQ
E. Dénes – Wigner RCF/NFI
ALICE Bp meeting – March 30 2012
1
Visit of ALICE DAQ group in Wigner RCF
• March 6-7, 2012
• CERN participants:
• Paolo Gubellino, ALICE spokesperson
• Pierre Vande Vyvre, head of ALICE DAQ group
• Filippo Costa, member of ALICE DAQ group
• From Wigner RCF:
• Péter Lévai
• Ervin Dénes
• Tivadar Kiss
• György Rubin
• Tamás Tölyhi
• Students:
• Kristóf Blutman, BME
• Gábor Kiss, ELTE
• Hunor Melegh, BME
2
Main talks
• Péter Lévai: About Wigner RCF
• Paolo Gubellino: ALICE upgrade strategy (incl. the
detector upgrades)
• Pierre Vande Vyvre: ALICE online upgrade*
• 2 half-day discussion
• Self introduction of students
• Roadmap*
• Visit to Cerntech
* See following slides
3
Present Online Architecture
Rare/All
BUSY
CTP
L0, L1a, L2
BUSY
LTU
LTU
L0, L1a, L2
TTC
TTC
DDL
H-RORC
H-RORC H-RORC
FEP
FERO FERO
Event
120 DDLs
360 DDLs
Fragment
430 D-RORC
D-RORC D-RORC
LDC
Load Bal.
125 Detector LDC
RCU
FEP
HLT Farm
RCU
10 DDLs
D-RORC
D-RORC
10 D-RORC
D-RORC
D-RORC
LDC
LDC
10 HLT LDC
LDC
LDC
Sub-event
Event Building Network
EDM
Event
GDC
TDSM
75 GDC
30 TDSM
DA
DQM
60 DA/DQM
DSS
18 DSS
File
Storage Network
Archiving on Tape
in the Computing
Centre (Meyrin)
75 TDS
ALICE Mar.2012
4
PDS
P. Vande Vyvre – CERN/PH
ALICE Upgrade Strategy
• Strategy recently approved by ALICE presents a global and coherent
plan to upgrade the experiment for 2018 (Long Shutdown 2 LS2)
• “Upgrade Strategy for the ALICE Central Barrel”
https://aliceinfo.cern.ch/ArtSubmission/node/108
• Key concepts for running the experiment at high rate
• Pipelined electronics
• Triggering the TPC would limit this rate → Continuous readout
• Reduce the data volume by topological trigger and online reconstruction
• Major upgrade of the detector electronics and of the online systems
• Online upgrade design:
• Data buffer and processing off detector
• Fast Trigger Processor (FTP): 2 hw trigger levels to accommodate
various detector latencies and max. readout rate
• HLT: 2 sw trigger levels:
• ITS, TRD, TOF, EMC (reduce the rate before building TPC event)
• Final decision using the data from all detectors
• Common DAQ /HLT farm to minimize cost while preserving the present
flexibility of running modes
ALICE Mar.2012
5
P. VANDE VYVRE CERN-PH
Trigger Rates
•
Key concepts for running ALICE at high rate
• Hardware trigger-less architecture whenever needed and affordable
• Pipelined electronics and continuous readout
• Reduction of the data volume by topological trigger and 2 steps online reconstruction
Trigger with
pp beams
Trigger with
Pb-Pb beams
Trigger
Levels
Detectors
No Trigger
ITS, TPC, TRD,
EMCal, PHOS
Level 0 (hw)
TOF (Pb-Pb)
2000
1.2 s
50
1.2 s
Level 1 (hw)
TOF (p-p), Muon
10-20
10 s
20
10 s
Level 2 (sw)
10-25
1s
Level 3 (sw)
5-25
10 s
ALICE Mar.2012
Frequency
(kHz)
Latency
Frequency
(kHz)
Latency
Continuous read-out at 10 MHz
6
P. VANDE VYVRE CERN-PH
Present TPC Readout
1
216
FEC
TTC
216
160 MB/s
12
CTP
RCU
FEC
216
DDL 2.0 Gb/s
1
FEC
13
216
216
160 MB/s
DDL 2.0 Gb/s
D
R
O
R
C
DAQ
H
R
O
R
C
HLT
FEC
• Present readout :
• Links: DDL at 2 Gb/s. 216 DDLs for the TPC used at 1.6 Gb/s
• PC adapters: D-RORC and H-RORC
• Up to 6 RORCs/PC
ALICE Mar.2012
7
P. VANDE VYVRE CERN-PH
Readout Upgrade
DAQ and HLT systems
FEC2
~7000
10 Gb/s
• Upgrade readout:
FLP
600
10.0 or 40
Gb/s
Network
600
EPN
10.0 or 40
Gb/s
• Non-zero suppressed TPC data 57 Tb/s (570 kchannels x 10 bits x 10 MHz)
• ~7000 links at 10 Gb/s for TPC. ~7800 links for the whole experiment.
• TBD: FEC2 characteristics (GEM readout, very simple FEC vs more links)
• DDL3 and RORC3 for LS2 upgrade (ALICE common solution)
• Address the needs of the new architecture for the period after LS2 (2018)
• DDL3 links at 10 Gb/s
• Exact requirements to be addressed (radiation tolerance/hardness)
• Different physical layers possible (CERN GBT, Eth, FCS, wavelength multiplex.)
• First-Level Processors (FLPs)
• ~650 FLPs needed: 12 detector links 10 Gb/s, 1 network link at 10 or 40 Gb/s
• Data readout and first-level data processing:
• ZS, cluster finder and compression could be performed by the RORC FPGA
• Regional online reconstruction
ALICE Mar.2012
8
P. VANDE VYVRE CERN-PH
Future evolution of DDL & RORC
• New common DAQ/HLT DDL2 and RORC2
• Prototype under design by Heiko Engel (Udo Kebschull’s
team/Frankfurt) and Tivadar Kiss (Budapest) (Ready in 2012)
• It will address the upgrade needs with the present architecture for
the period (2014-16) between Long Shutdown 1 (LS1) and LS2
• Includes 12 DDL2 links at 6 Gb/s. 6 links to DAQ LDC  36 Gb/s.
• PCIe V2 8 lanes (500 MB/s/lane)  32 Gb/s of I/O capacity.
• Data processing in the FPGA (e.g cluster finding)
• Currently: 5 links at 2Gb/s per PC 
10 Gb/s of I/O capacity
• Prototype under development
• 12 links at 6 Gb/s
• 6 links to DAQ LDC  36 Gb/s.
• PCIe Gen2 8 lanes (500 MB/s/lane) 
32 Gb/s of I/O capacity
• Final system
• 12 links at 10 Gb/s per PC
• PCIe Gen3 16 lanes  I/O 128 Gb/s
4
4
4
ALICE Mar.2012
9
P. Vande Vyvre – CERN/PH
Upgrade Online (after LS2)
L2
~7800 DDL3
TPC
10 Gb/s
FLP
ITS
FLP
TRD
FLP
TOF
FLP
PHOS
FLP
Muon
FLP
FTP
FLP
Trigger Detectors
ALICE Mar.2012
10 Gb/s
or
40 Gb/s
FLP
EMC
L0
L1
DAQ and HLT
Network
~ 650 FLPs
10 Gb/s
or
40 Gb/s
EPN
L3
EPN
L3
~1250 EPNs
10
P. VANDE VYVRE CERN-PH
Next Steps
• Major upgrade foreseen for the online systems during LS2
• Improve the Strategy Note (Mar ‘12)
• More detailed and optimized cost estimate (Mar ‘12)
• Prototyping, some using the present online systems (‘12)
• Continue to gain experience with using HLT online results as basis of
offline reconstruction, e.g. the cluster finding
• Essential to reach the necessary data reduction factors after LS2
• R&D program (‘12-’16)
• Fast Trigger Processor
• Readout links
• Event-building and HLT dataflow network
• Online reconstruction: online use of “offline software”
• Efficient use of new computing platforms (many-cores and GPUs)
• Online Conceptual Design Report (TBD)
• Requirements from detectors
• Key results from R&D
ALICE Mar.2012
11
P. Vande Vyvre – CERN/PH
Short and Mid Term SW, FW, HW Tasks (for us)
•
•
During LS1: some detector will upgrade to C-RORC
• C-RORC prototype: June 2012, Tivadar
• C-RORC firmware: September 2012, Filippo
• C-RORC software: Q4 2012, Ervin
• Tailoring of the DDG for the C-RORC: Q4 2012, Filippo, Ervin
Technology Research Project
• Members: Pierre, Filippo, Tivadar, Ervin, Gyuri, Gusty?, students?
• Preparing a list of technological companies, deadline: end of April
• Preparing questionnaires for the meetings with companies: end of May
• Scheduling meetings with industrial companies:
• At CERN: middle of June
• In the US: October
• Preparation of a spread sheet for comparison the different solutions:
• Preliminary: end of September, final: Q1/2013
• Setting up and running demonstrations in Budapest: PCIe-over-fibre, 2012
• Recommended capital investments:
• High-end server motherboard
• PCIe expansion box over-fibre connection
12
Longer time tasks
• Building demonstrations with selected technologies
• Deadline for defining the common interface for the detectors:
June 2014
• Deadline for a Conceptual Design Report: ~ End of 2014
• Deadline for delivering prototypes of "DDL3" for the detectors:
Q4 2015
• Deliver prototype software (Q4 2015)
• Production of DDL3 (Q3 2016 - Q2 2017)
• Release production software (Q2 2017)
• Detector installation and commissioning at Point 2 (Q3 2017)
• A small DAQ with limited performance (but full functionality) has
to be installed by Q3 of 2017
13
Thank you for your attention
14
Reserved slides
15
A quick look to the past:
the DDL and the D-RORC
• Project started in 1995
• Strong push from the management:
George Vesztergombi and now Péter Lévai
• Work of a solid and competent team:
Ervin Dénes, Tivadar Kiss, György Rubin, Csaba Soós,
and many others contributed to this successful project
• The Budapest team delivered to the ALICE experiment
a common and reliable solution for the data transfer
from the detectors to the online systems :
the DDL and the D-RORC
ALICE Mar.2012
16
P. VANDE VYVRE CERN-PH
DDL and D-RORC
• ~ 500 DDLs and
~450 D-RORCs
• Collect all the data of ALICE
(~2 PB/year)
• Configure the electronics of
several detectors, in particular
the biggest one (TPC)
• Goal for the future:
Repeat this success story !
ALICE Mar.2012
17
P. Vande Vyvre – CERN/PH
Continuous Readout
Events
n+2
n+1
n
FLP
Time
t+3x t+2x t+x
t
t-x
t-2x
ITS
Event
Building &
Processing
TRD
EMC
TPC
TOF
ALICE Mar.2012
18
P. Vande Vyvre – CERN/PH
Online reconstruction
• 50 kHz Pb–Pb collisions inspected with the least possible bias using
topological triggers and on-line particle identification
• HI run 2011: online cluster finding data compression factor of 4 for TPC
• Two HLT scenarios for the upgrade:
1. Partial event reconstruction:
Further factor of 5 → overall reduction factor of 20. Close to what could
be achieved and tested now. Rate to tape: 5 kHz.
2. Full event reconstruction:
Overall data reduction by a factor of 100. Rate to tape: 25 kHz.
HI run 2018: min. bias event size ~75 MB  ~4-1 MB after data volume
reduction. Throughput to mass storage: 20 GB/s.
• Target rate after LS2 can only be reached with online reconstruction
•
•
•
ALICE Mar.2012
Build on success of last year with TPC cluster finding
Extremely ambitious: online calibration and data processing
Essential to gain experience with using online results as basis of offline
reconstruction, e.g. the cluster finding. Mandatory to reach the
necessary data reduction factor after LS2.
19
P. Vande Vyvre – CERN/PH
Network Routers and Switches
for the Event Building
• Present DAQ network (Force 10 now DELL)
• Exascale E1200i 3.5 Tb/s
• Capacity of present DAQ router not adequate
• Technology 1: Ethernet (Brocade MLX Series)
• MLX32e up to 15.3 Tb/s, 32 line card slots
Up to 256x10GbE ports, 32x100GbE ports
• Line cards: 2x100 Gb/s or 8x10 Gb/s
• Technology 2: Infiniband (Mellanox)
• MIS6500 up to 51.8 Tb/s, 648 ports at 40 Gb/s
ALICE Mar.2012
20
P. Vande Vyvre – CERN/PH
Processing Power
Current HLT
Currently HLT processing rates with full TPC reconstruction:
•Either ~200 Hz central events (65 MB) or ~800Hz min bias events
•216 front-end nodes (equivalent to the FLPs)
• TPC cluster finder on H-RORC FPGA and coordinates transformation by the CPU
•60 tracking nodes with GPU (equivalent of the EPNs)
HLT
• Current HLT processing nodes:
•Assuming linear scaling of rates and
• Intel Xeon E5520, 2.2 GHz 4 cores
using current HLT technology:
Global track merging + reconstruction,
some trigger algorithms
•Level 2 (reconstruction using ITS, TRD, EMC)
• Nvidia GTX480/580
• 50 kHz min bias events (15 MB, ~4 times less CPU) →
~90% of the tracking
50 kHz/800Hz*60 nodes*1/4= 1250 nodes (CPU+GPU) • Performance gain 33%/yr during 7 years
•Level 3 (full reconstruction incl. TPC)
• 25 kHz central events →
25 kHz/200Hz*60 nodes = 7500 nodes (CPU+GPU)
Factor 7 from 2011 to 2018
• Overall chip performance gain →
Significant R&D on computing and
algorithms concepts
•8750 nodes of 2011 → 1250 nodes of 2018
Event Processing nodes ~1250 EPNs
ALICE Mar.2012
21
P. Vande Vyvre – CERN/PH