Data Transport
in Particle Physics Experiments
Tony Gillman
Particle Physics Department
Rutherford Appleton Laboratory
Tony Gillman – Particle Physics Department
1
RAL – 21st January 2009
Data transport – scope
 A very generic title…
 “Transport” is meaning here “movement of signals and data” –
 How are data transferred all the way from detectors to computers?
 What happens to the signals during this journey – transmission media, formats, …
 I will aim to cover a broad range of topics –
 Analogue signal handling – and some of the pitfalls…
 Analogue to Digital conversion techniques – the good and the bad…
 Data serialisation and deserialisation – why bother…
 Digital data transport media – copper vs silica
 Purpose: give some idea of the problems of getting data from experiments
 The first-level trigger of the ATLAS detector at the CERN LHC neatly illustrates
many of these techniques, so will be used as a general case study
Tony Gillman – Particle Physics Department
2
RAL – 21st January 2009
The data challenge
 Current generation of experiments will generate prodigious data volumes
 ATLAS will produce ~1 Petabyte (1015 bytes) per second
 In addition, the instantaneous data rates can be extremely high
 The LHC collision rate is 40 MHz → bursts of new data arrive every 25 nsec
 How do we transfer these data from the detectors into the data acquisition
electronics → massive communication problem
 Triggering removes the need to transport all of these data –
 Store data for ~2 msec in pipeline memories
 First-level trigger decides from which events to accept and transport the detector data
 40 MHz → 75 kHz max – (higher-level triggers reduce this much further)
Tony Gillman – Particle Physics Department
3
RAL – 21st January 2009
ATLAS trigger system
 Still a challenge even to get trigger
data into trigger electronics
 Transport of data must be almost
error-free, or trigger rate can become
unacceptably high
 Latency (time delay between
collisions and trigger decision) must
be short, to minimise data storage
requirements
(remember 106 Gigabytes per sec!)
 Every part of signal chain must
therefore be as fast as possible
Tony Gillman – Particle Physics Department
4
RAL – 21st January 2009
ATLAS level-1 calorimeter trigger
400Mbit/s
Jet / ET
(JEP)
Analogue
Receivers
Analogue
tower sums
(~7200)
To CTP
DAQ/RoI
PreProcessor
(PPr)
400Mbit/s
DAQ
1Gbit/s
1Gbit/s
e/, /hadron
Clusters
(CP)
To CTP
DAQ/RoI
1Gbit/s
Readout Driver (ROD)
Real-time
signal (data) path
Tony Gillman – Particle Physics Department
To ROS
5
Readout
data path
RAL – 21st January 2009
ATLAS level-1 calorimeter trigger
 Data are transported from detectors (calorimeters) to trigger processing
electronics to generate ACCEPT signals to feed Central Trigger Processor
 Signals undergo transformations at several stages in their journey…
Tony Gillman – Particle Physics Department
6
RAL – 21st January 2009
Analogue signal transmission
 Calorimeter signals are of two types:
 Liquid Argon calorimeter –
 Bipolar, 75 nsec FWHM
 Differential, ±2V max
 Tile calorimeter –
 Unipolar, 50 nsec FWHM
 Differential, ±2V max
Tony Gillman – Particle Physics Department
7
RAL – 21st January 2009
TileCal analogue trigger cable
Transport medium: 16 shielded twisted-pair channels + global shield
Characteristic impedance: 88 Ω  10 %
Cable delay: ≤4.76 nsec/m
Inter-pair delay skew: <2.5 nsec (70m cable)
Attenuation: -0.06 dB/m
Crosstalk: <0.2% (70m cable)
Bandwidth: 13 MHz at -6dB
Tony Gillman – Particle Physics Department
8
RAL – 21st January 2009
Imperfections in transmission lines
 Pre-installation measurements on TileCal analogue cables showed bad timing skew
 Inter-pair skew (tpdn ≠ tpdm) – excessive, up to 20 nsec, but could be calibrated out
 Intra-pair skew (tpdn+ ≠ tpdn-) – excessive, up to 28 nsec
 This effect is totally unacceptable – result is to change the shape and amplitude of resultant
differential signal, because of varying levels of dispersion
Good Pair
Bad Pair
(tower 4, PMT 19)
(tower 4, PMT 19)
Resultant signal
Positive signal
Negative (inverted) signal
Tony Gillman – Particle Physics Department
9
RAL – 21st January 2009
S-parameter measurements
 S-parameters characterise transmission-line performance in the frequency domain
 For the suspect cable, S-parameters were measured for the 4 propagation modes –
1. differential-mode → differential-mode (signal attenuation)
2. common-mode → common-mode (signal attenuation)
3. differential-mode → common-mode (mode conversion)
 Common-mode signal will radiate and couple to adjacent signal pair
4. common-mode → differential-mode (mode conversion)
 Cable susceptible to radiation and resultant differential-mode signal will degrade S/N ratio
 Mode conversion is caused by asymmetries in differential transmission-lines
Tony Gillman – Particle Physics Department
10
RAL – 21st January 2009
S-parameter measurements
 First step was to measure characteristic impedance Z0 of cables in two modes –
common-mode and differential-mode and terminate cables under test in both
ways using a single network
 Measure transfer function of cables over frequency range up to 50 MHz in each
of four modes using sine waves
Test setup for common → common mode and
common → differential mode measurements
Tony Gillman – Particle Physics Department
Test setup for differential → differential mode
and differential → common mode measurements
11
RAL – 21st January 2009
S-parameter measurements
“Bad” Pair
“Good” Pair
 “Bad” pair exhibits severe attenuation at high frequencies → signal dispersion
 Common → differential conversion is extremely large >15 MHz (compare with
differential → differential mode!)
 Conclusion: The entire batch of cables from this manufacturer was rejected
Tony Gillman – Particle Physics Department
12
RAL – 21st January 2009
ATLAS analogue trigger cabling
Tony Gillman – Particle Physics Department
13
RAL – 21st January 2009
ATLAS analogue trigger cabling
Tony Gillman – Particle Physics Department
14
RAL – 21st January 2009
Analogue → Digital conversion
 Digital signals have many advantages over analogue signals (noise immunity,
crosstalk, processing capability, …), so preferable to digitise detector signals as
early as possible in signal chain
 Analogue-to-digital converters (ADCs) are mixed-signal devices
 Digital Output = Input signal / VREF = AIN / VREF x 2N
 AIN = Analogue Input Voltage
 VREF = Vmax - Vmin (Reference Voltage)
 N = No of output bits (resolution)
 Analogue signal resolution = VREF / 2N
 This is the fastest type of converter,
also known as a Flash ADC (FADC)
 The delay between the clock and the
digital output data appearing is latency
 Low latency essential in many applications
(e.g. ATLAS level-1 trigger)
Clock
Tony Gillman – Particle Physics Department
15
RAL – 21st January 2009
ADC performance – some notes
For an n-bit converter…
 Dynamic range in dB – 20 log (2n -1)
 Signal-to-Noise Ratio (SNR) = rms Signal / rms Noise (integrated over 1/2 clock period)
 Several sources of noise –
 Quantisation noise
 Clock jitter
 Electronic circuit noise
 Fundamental limit on ADC performance is quantisation noise – LSB / sqrt 12
 SNR for ideal ADC = (6.02n + 1.76) dB
 Nyquist limit – highest frequency component permitted ≤ ½ sampling frequency
 If f(Ain) > ½ fs aliasing will occur → increased noise
 Avoid aliasing by passing signal through low-pass filter before ADC comparators
Tony Gillman – Particle Physics Department
16
RAL – 21st January 2009
ADC performance – timing jitter
 Clock jitter leads to aperture uncertainty
 For a sine wave signal (V = A sin wt) → dVmax = 2p A f dt
 Aperture uncertainty therefore translates to a noise source,
degrading the ADC resolution for high-frequency signals
 Magnitude scales with the input signal frequency
 The effect only becomes significant if dt > (2n p f)-1
 The demands on clock jitter are very severe…
ADC
resolution
Input frequency
44.1 kHz
192 kHz
1 MHz
10 MHz
100 MHz
8
28.2 ns
6.48 ns
1.24 ns
124 ps
12.4 ps
10
7.05 ns
1.62 ns
311 ps
31.1 ps
3.11 ps
12
1.76 ns
405 ps
77.7 ps
7.77 ps
777 fs
14
441 ps
101 ps
19.4 ps
1.94 ps
194 fs
16
110 ps
25.3 ps
4.86 ps
486 fs
48.6 fs
18
27.5 ps
6.32 ps
1.21 ps
121 fs
12.1 fs
24
430 fs
98.8 fs
19.0 fs
1.9 fs
190 as
Tony Gillman – Particle Physics Department
17
RAL – 21st January 2009
ADC performance – ENOB
 Overall effect of aperture uncertainty is to reduce the Effective Number Of Bits
(ENOB) of the ADC at high frequencies
 N.B. An n-bit ADC will not resolve
to n bits at its full analogue
bandwidth unless clock jitter is
kept below these limits
Tony Gillman – Particle Physics Department
18
RAL – 21st January 2009
Digital signal transmission
 To transfer parallel data between sub-systems, convert to serial bitstreams to reduce
the number of data paths and connector pins – increases reliability (but also latency!)
 Serialising-deserialising (SerDes) chipsets can drive serial bitstreams at ~Gbit/s rate
 Very common technology for serial links is Low-Voltage Differential Signaling (LVDS)
Cable chosen for trigger – shielded Twin-ax
(2 parallel cores – Z0 = 100W)
 Many advantages:
 Low-voltage power supplies
 Good noise immunity
 Low power dissipation
 Small signal swing → high data rates
 “Gigabits at Milliwatts”
Tony Gillman – Particle Physics Department
19
RAL – 21st January 2009
Eye patterns – digital data
Source
Tony Gillman – Particle Physics Department
20
Destination
RAL – 21st January 2009
Pre-compensation techniques
 Adding a passive pre-compensation
network (high-pass filter – CR or LR) to
the LVDS driver outputs boosts the highfrequency components of the signal to
compensate for the cable dispersion
No pre-compensation
LR pre-compensation
N.B. overshoot
Tony Gillman – Particle Physics Department
21
RAL – 21st January 2009
ATLAS PreProcessor Module
ANALOGUE
MCMs
Digital data
outputs
DIGITAL
Processor
ASIC
Flash ADCs
Signal flow
LVDS
Serialisers
Analogue signal
inputs
Signal flow
Tony Gillman – Particle Physics Department
22
RAL – 21st January 2009
Beware!!!
 Installing Cu signal cabling can produce unexpected effects –
Cable Discharge Event (CDE)
 Static electricity on the jacket material of
the cable induces a charge in the cable
wires
 Mechanisms –
 Tribocharging (friction), produced as
cables are pulled across surfaces
 Electromagnetic fields can induce charge
build up on cables, e.g. from electronic
light ballasts
 This may have been an issue for our 8000 LVDS cables installed under-floor between racks
 As a precaution, we “discharged” cables after installation but before connecting any
modules
 N.B. This is another reason why using fibre-optic cabling has advantages
Tony Gillman – Particle Physics Department
23
RAL – 21st January 2009
Optical fibres
 Cylindrical dielectric waveguide transmitting light along its axis by total internal
reflection, consisting of a core covered by a sheath of cladding (ncore > ncladding)
 As an alternative to Cu cabling for digital data transmission, it has many benefits –
 Huge bandwidth
 Immunity from EMI, ground-loops and crosstalk
 Small volume for cable plant
 Two types available – Multi-mode and Single-mode (usual material is silica) –
 Multi-mode fibres – large core diameter (few tens of mm) allows multiple path lengths
→ intermodal dispersion limits Bandwidth x Distance product
 Reduce intermodal dispersion by using graded-index silica – transit time variations → zero
 Single-mode fibres – small core diameter (few mm) forces lowest-order (axial) mode, low
dispersion → high Bandwidth x Distance product
 Propagation delay ~ncore / c (~5 nsec/m – similar to Cu cable)
Tony Gillman – Particle Physics Department
24
RAL – 21st January 2009
Optical fibres – some available types
 Step-index Multi-Mode fibres –




Cheap
Large core diameter → easy to couple light in/out
High intermodal dispersion → low bandwidth
Suitable for short links and low data rates
 Graded-index Multi-Mode fibres –
 Large core diameter → easy to couple light in/out
 Reduced intermodal dispersion → increased bandwidth
 Suitable for medium-range links/low data rates or short links/medium data rates
 Step-index Single-Mode fibres –
 Small core diameter → harder to couple light in/out
 Wide bandwidth
 Suitable for long-range links and high data rates
Tony Gillman – Particle Physics Department
25
RAL – 21st January 2009
Optical fibres – ATLAS level-1 trigger
 Data transmitted to level-2 trigger and DAQ via Readout Driver modules (RODs) –
distance ~10m, total bandwidth >250 Gbyte/s
 Chosen to use Multi-Mode fibres driven by laser diode transmitters (Infineon)
operating at 850 nm, mounted on trigger modules
 Total no of fibres feeding Readout Driver modules (RODs) ~320
56 mm
 Transmitters are driven from Agilent G-link transmitters at 960 Mbaud (800 Mbit/s)
 Receivers are dual Stratos devices mounted on 20 RODs
Tony Gillman – Particle Physics Department
26
RAL – 21st January 2009
ILC Vertex Detector
 International Linear Collider will be an accelerator ~35 km long colliding bunches
of e- and e+ at energies of 500 GeV – physics to complement that from the LHC
 VXD will be based on Si detectors
e.g. CCDs – forming ladders
Tony Gillman – Particle Physics Department
27
RAL – 14th February 2007
ILC Vertex Detector
 5 concentric barrels of ladders, on
radii ranging from 15mm - 60mm
 Thickness <0.1% X0 per barrel (target)
 ~109 pixels – each 20mm  20mm
 ILC will generate many spurious hits from beamstrahlung during bunch crossings
 To minimise these background hits, CCDs must be read out quickly –
 Readout time of 50ms for inner barrel (highest background hit density)
 Readout time of 250ms for each outer barrel (lower background hit density)
Tony Gillman – Particle Physics Department
28
RAL – 14th February 2007
Background hit rates
 Accelerator beam parameters –
 ~1 msec bunch-train
 337 nsec inter-bunch gap
 5 Hz repetition rate (200msec dead-time)
Barrel
Radius
no
CCD
dimensions
CCDs Ladders Readout
Readout
per
per
clock
time
ladder
barrel
rate
Background
hits per
bunch-train
1
15mm
100mm13mm
1
8
50 MHz
50 msec
760K
2
26mm
125mm22mm
2
8
25 MHz
250 msec
370K
3
37mm
125mm22mm
2
12
25 MHz
250 msec
140K
4
48mm
125mm22mm
2
16
25 MHz
250 msec
30K
5
60mm
125mm22mm
2
20
25 MHz
250 msec
30K
Tony Gillman – Particle Physics Department
29
RAL – 14th February 2007
Readout data volumes
 So how much data will the VXD generate?
 Total no of pixels clocked out during each
bunch train ~4.109
CCD
CCD
output
stage
 To read out every pixel (assuming
≤1 byte/pixel) raw data volume ~20 Gbyte/s
Pixel
thresh
to
adjacent
kernel logic
Cluster
thresh
26 bits (h-f addressing)
Bump
bond
Bump
bond
ADC
 Sparsify data in real-time in Readout chips
 Look for 2x2 pixel clusters with signal
>cluster threshold → 6 bytes per cluster
CCD
output
stage
G
 This is unnecessary, most pixels are empty
– only ~0.5% occupancy
 Digitise signals in on-chip ADCs (5 bits OK)
CCD
output
stage
from
col (n-2)
Readout
Chip
CCD
output
stage
G
ADC
ERF
ERF
d/dt
d/dt
Comp
Comp
Pipeline
2x2
kernel
Pipeline
Gate
Comp
Gate
to other
readout
channels
to
adjacent
kernel logic
from
col (n+1)
to other
readout
channels
20 bits for 4 x 5-bit ADC values
Vertical
address
2 spare bits – parity, etc
FIFO
col 1
FIFO
col (n-1)
col n
col 64
64-column Multiplexer
 20 Gbyte/s → 40 Mbyte/s
Tony Gillman – Particle Physics Department
Memory
30
RAL – 14th February 2007
Data acquisition task
 Total sparsified data volume per bunch train ~8 Mbytes (~40Mbyte/s)
 To read this out in real-time requires peak data transfer rate >8 Gbyte/sec
 Readout chips require de-randomising FIFOs → reduce average data transfer rate
 Provide each Readout chip with primary memory to store sparsified data (+ address tags)
 ~1 Mbyte/CPR (Barrel 1) → ~10 Kbyte/CPR (Barrel 5)
 Read data out to DAQ during 200msec dead-time after each bunch train
 Total sparsified data rate from VXD ~40 Mbyte/s (split between ±h)
Tony Gillman – Particle Physics Department
31
RAL – 14th February 2007
Data collection
 Many ways to collect the data from all CPRs – this is only one possibility
 Empty CPR primary memories sequentially at 50 MHz on to byte-wide ring-buses at
ends of each barrel
 Serialise the data from each ring-bus at 400 Mbit/s and drive differential LVDS
signals (or optical links) into 2 DAQ cards (±h)
 DAQ cards de-serialise the LVDS data, combine the 5 data streams, re-format,
assemble and store the data for the entire bunch crossing (taking ~80 msec)
 2 optical fibres/DAQ card export data to main DAQ + import readout control signals
Tony Gillman – Particle Physics Department
32
RAL – 14th February 2007
“Galvanic” links need space…
~10% of digital data links of ATLAS L1Calo trigger in a Birmingham test-rig
Small part of ATLAS L1Calo data
link system installed underground
 Data from ILC Vertex Detector could be transported on a single fibre!
 Upgraded L1Calo for Super-LHC will probably use fibres for all data transport
Tony Gillman – Particle Physics Department
33
RAL – 14th February 2007