High Level Triggering
Fred Wickens
High Level Triggering (HLT)
• Introduction to triggering and HLT systems
– What is Triggering
– What is High Level Triggering
– Why do we need it
• Case study of ATLAS HLT (+ some
comparisons with other experiments)
• Summary
2
Why do we Trigger and why multi-level
• Over the years experiments have focussed on rarer processes
– Need large statistics of these rare events
– DAQ system (and off-line analysis capability) under increasing
strain
• limiting useful event statistics
• Aim of the trigger is to record just the events of interest
– i.e. Trigger selects the events we wish to study
• Originally - only read-out the detector if Trigger satisfied
– Larger detectors and slow serial read-out => large dead-time
– Also increasingly difficult to select the interesting events
• Introduced: Multi-level triggers and parallel read-out
– At each level apply increasingly complex algorithms to obtain better
event selection/background rejection
• These have:
– Led to major reduction in Dead-time – which was the major issue
– Managed growth in data rates – this remains the major issue
3
Summary of ATLAS Data Flow Rates
• From detectors
> 1014 Bytes/sec
• After Level-1 accept
~ 1011 Bytes/sec
• Into event builder
~ 109 Bytes/sec
• Onto permanent storage
~ 108 Bytes/sec

~ 1015 Bytes/year
4
The evolution of DAQ systems
5
TDAQ Comparisons
6
Level 1
• Time:
few microseconds
• Hardware based
– Using fast detectors + fast algorithms
– Reduced granularity and precision
• calorimeter energy sums
• tracking by masks
• During Level-1 decision time event data is
stored in front-end electronics
– at LHC use pipeline - as collision rate shorter than
Level-1 decision time
• For details of Level-1 see Dave Newbold talk
7
High Level Trigger - Levels 2 + 3
• Level-2 : Few milliseconds (10-100)
– Partial events received via high-speed network
– Specialised algorithms
• 3-D, fine grain calorimetry
• tracking, matching
• Topology
• Level-3 : Up to a few seconds
– Full or partial event reconstruction
• after event building (collection of all data from all detectors)
• Level-2 + Level-3
– Processor farm with Linux server PC’s
– Each event allocated to a single processor, large farm of
processors to handle rate
8
Summary of Introduction
• For many physics analyses, aim is to obtain as high
statistics as possible for a given process
– We cannot afford to handle or store all of the data a detector
can produce!
• The Trigger
– selects the most interesting events from the myriad of events
seen
• I.e. Obtain better use of limited output band-width
• Throw away less interesting events
• Keep all of the good events(or as many as possible)
– must get it right
• any good events thrown away are lost for ever!
• High level Trigger allows:
– More complex selection algorithms
– Use of all detectors and full granularity full precision data
9
Case study of the ATLAS HLT system
Concentrate on issues relevant for
ATLAS (CMS very similar issues), but
try to address some more general
points
Starting points for any Trigger system
• physics programme for the experiment
– what are you trying to measure
• accelerator parameters
– what rates and structures
• detector and trigger performance
– what data is available
– what trigger resources do we have to use it
• Particularly network b/w + cpu performance
11
Physics at the LHC
7 TeV
Interesting events are buried in a sea
of soft interactions
B physics
High energy QCD jet
production
top physics
Higgs production
12
The LHC and ATLAS/CMS
• LHC has
– Design luminosity 1034 cm-2s-1
• 2010: 1027 – 2x1032 ; 2011: up to 3.6x1033 ; 2012: up to 6x1033
– Design bunch separation 25 ns (bunch length ~1 ns)
• Currently running with 50 ns
• This results in
– ~ 23 interactions / bunch crossing (Already exceeded!)
• ~ 80 charged particles (mainly soft pions) / interaction
• ~2000 charged particles / bunch crossing
• Total interaction rate
– b-physics
– t-physics
– Higgs
fraction ~ 10-3
fraction ~ 10-8
fraction ~ 10-11
109 sec-1
106 sec-1
10 sec-1
10-2 sec-1
13
Physics programme
• Higgs signal extraction important - but very difficult
• There is lots of other interesting physics
–
–
–
–
–
B physics and CP violation
quarks, gluons and QCD
top quarks
SUSY
‘new’ physics
• Programme evolving with: luminosity and HLT capacity
– i.e. Balance between
• high PT programme (Higgs etc.)
• b-physics programme (CP measurements)
• searches for new physics
14
Trigger strategy at LHC
• To avoid being overwhelmed use signatures with
small backgrounds
– Leptons
– High mass resonances
– Heavy quarks
• The trigger selection looks for events with:
–
–
–
–
Isolated leptons and photons,
-, central- and forward-jets
Events with high ET
Events with missing ET
15
Trigger
40 MHz
ARCHITECTURE
DAQ
Three logical levels
Hierarchical data-flow
LVL1 - Fastest:
Only Calo and Mu
Hardwired
On-detector
electronics:
Pipelines
~40 ms
LVL2 - Local:
LVL1 refinement +
track association
Event fragments
buffered in
parallel
~4 sec.
LVL3 - Full event:
“Offline” analysis
Full event in
processor farm
~2.5 ms
~ 200 Hz
Physics
~1 PB/s
(equivalent)
~ 300 MB/s
16
Example Physics signatures
Objects
Physics signatures
Electron 1e>25, 2e>15 GeV
Higgs (SM, MSSM), new gauge bosons,
extra dimensions, SUSY, W, top
Photon 1γ>60, 2γ>20 GeV
Higgs (SM, MSSM), extra dimensions,
SUSY
Muon 1μ>20, 2μ>10 GeV
Higgs (SM, MSSM), new gauge bosons,
extra dimensions, SUSY, W, top
Jet 1j>360, 3j>150, 4j>100 GeV
SUSY, compositeness, resonances
Jet >60 + ETmiss >60 GeV
SUSY, exotics
Tau >30 + ETmiss >40 GeV
Extended Higgs models, SUSY
17
Selected (inclusive) signatures
Process
Level-1
2 em, ET>20 GeV
H0
H0Z Z* + – + – 2 em, ET>20 GeV
2 µ, pT>6 GeV
1 em, ET>30 GeV
1 µ, pT>20 GeV
2 em, ET>20 GeV
Z+–+X
2 µ, pT>6 GeV
1 em, ET>30 GeV
1 µ, pT>20 GeV
1 em, ET>30 GeV
t t  leptons+jets
1 µ, pT>20 GeV
W', Z'jets
1 jet, ET>150 GeV
SUSYjets
1 jet, ET>150 GeV
E miss
T
Level-2
2 , ET>20 GeV
2 e, ET>20 GeV
2 µ, ET>6 GeV, I
1 e, ET>30 GeV
1 µ, ET>20 GeV, I
2 e, ET>20 GeV
2 µ, ET>6 GeV, I
1 e, ET>30 GeV
1 µ, ET>20 GeV, I
1 e, ET>30 GeV
1 µ, ET>20 GeV, I
1 jet, ET>300 GeV
3 jet, ET>150 GeV
E miss
T
18
Trigger design – Level-1
• Level-1
– sets the context for the HLT
– reduces triggers to ~75 kHz
• Limited detector data
– Calo + Muon only
– Reduced granularity
• Trigger on inclusive
signatures
• muons;
• em/tau/jet calo clusters;
missing and sum ET
• Hardware trigger
– Programmable thresholds
– CTP selection based on
multiplicities and thresholds
19
Level-1 Selection
• The Level-1 trigger
– an “or” of a large number of inclusive signals
– set to match the current physics priorities and beam
conditions
• Precision of cuts at Level-1 is generally limited
• Adjust the overall Level-1 accept rate (and the
relative frequency of different triggers) by
– Adjusting thresholds
– Pre-scaling (e.g. only accept every 10th trigger of a
particular type) higher rate triggers
• Can be used to include a low rate of calibration events
• Menu can be changed at the start of run
– Pre-scale factors may change during the course of a run
20
Trigger design - HLT strategy
• Level 2
– confirm Level 1, some inclusive, some semiinclusive,
some simple topology triggers, vertex
reconstruction
(e.g. two particle mass cuts to select Zs)
• Level 3
– confirm Level 2, more refined topology selection,
near off-line code
21
Trigger design - Level-2
• Level-2 reduce triggers to ~4 kHz (was ~2 kHz)
– Note CMS does not have a physically separate Level-2 trigger, but
the HLT processors include a first stage of Level-2 algorithms
• Level-2 trigger has a short time budget
– ATLAS ~40 milli-sec average
• Note for Level-1 the time budget is a hard limit for every event, for the
High Level Trigger it is the average that matters, so OK for a small
fraction of events to take times much longer than this average
• Full detector data is available, but to minimise resources
needed:
–
–
–
–
–
Limit the data accessed
Only unpack detector data when it is needed
Use information from Level-1 to guide the process
Analysis proceeds in steps - can reject event after each step
Use custom algorithms
22
Regions of Interest
• The Level-1 selection is dominated by local
signatures (I.e. within Region of Interest RoI)
– Based on coarse granularity
data from calo and mu only
• Typically, there are
1-2 RoI/event
• ATLAS uses RoI’s to reduce
network b/w and processing
power required
23
Trigger design - Level-2 - cont’d
• Processing scheme
– extract features from sub-detectors in each RoI
– combine features from one RoI into object
– combine objects to test event topology
• Precision of Level-2 cuts
– Limited (although better than at Level-1)
– Emphasis is on very fast algorithms with
reasonable accuracy
• Do not include many corrections which may be applied
off-line
– Calibrations and alignment available for trigger not
as precise as ones available for off-line
24
ARCHITECTURE
Trigger
Calo
MuTrCh
40 MHz
40 MHz
LVL1
Muon
Trigger
ROD
ROIB
L
ROD
120
RoI’s
LVL2
~ 10 ms
RoI
requests
L2SV
ROB
ROD
GB/s
ROB
ROB
ROS
RoI data = 1-2%
L2P
L2P
L2P
T Event Filter
EFP
EFP
EFP
L2N
~2 GB/s
LVL2 accept
~ 1 sec
~ 1 PB/s
FE Pipelines
2.5 ms
LVL1 accept
75 kHz
~4 kHz
Other detectors
2.5 ms
Calorimeter
Trigger
H
DAQ
Read-Out Drivers
Read-Out Links
Read-Out Buffers
Read-Out Sub-systems
~6 GB/s
Event Builder
EB
~6 GB/s
EFN
~ 600 MB/s
~ 400 Hz
~ 600 MB/s
25
CMS Event Building
• CMS perform Event Building after Level-1
• Simplifies the architecture, but places much
higher demand on technology:
– Network traffic
~100 GB/s
– 1st stage use
Myrinet
– 2nd stage has
8 GbE slices
26
Example for Two electron trigger
LVL1 triggers on two isolated
STEP 4
e/m clusters with pT>20GeV
Signature 
(possible signature: Z–>ee)
HLT Strategy:



Validate step-by-step
Check intermediate signatures
Reject as early as possible
Sequential/modular
approach facilitates
early rejection
STEP 3
Signature 
STEP 2
e30i
+
Iso–
lation
e30
Iso–
lation
+
pt>
30GeV
e
ecand
STEP 1
Cluster
shape
Level1 seed 
EM20i
e30
pt>
30GeV
+
track
finding
Signature 
e30i
e
track
finding
+
time
Signature 
ecand
Cluster
shape
+
EM20i
27
Trigger design - Event Filter / Level-3
• Event Filter reduce triggers to ~400 Hz
– (was ~200 Hz)
• Event Filter budget ~ 4 sec average
• Full event detector data is available, but to
minimise resources needed:
– Only unpack detector data when it is needed
– Use information from Level-2 to guide the process
– Analysis proceeds in steps with – can reject event
after each step
– Use optimised off-line algorithms
28
EM ROI
Execution of a Trigger Chain
Electromagnetic
clusters
Level1:
Region of Interest is found and
position in EM calorimeter is
passed to Level 2
L2 calorim.
cluster?
L2 tracking
track?
Level 2 seeded by Level 1
•Fast reconstruction
algorithms
•Reconstruction within RoI
match?
E.F.calorim.
E.F.tracking
Ev.Filter seeded by Level 2
•Offline reconstruction
algorithms
•Refined alignment and
calibration
track?
e/ reconst.
e/ OK?
29
e/γ Trigger
• pT≈3-20 GeV: b/c/tau decays,
SUSY
• pT≈20-100 GeV: W/Z/top/Higgs
• pT>100 GeV: exotics
• Level 1: local ET maximum in
ΔηxΔφ = 0.2x0.2 with possible
isolation cut
• Level 2: fast tracking and
calorimeter clustering – use
shower shape variables plus
track-cluster matching
L1 EM trigger
pT > 5GeV
• Event Filter: high precision
offline algorithms wrapped for
online running
30
•
Discriminate against hadronic showers
based on shower shape variables
•
Use fine granularity of LAr calorimeter
•
Resolution improved in Event Filter with
respect to Level 2
E 37cells
R 
E 77cells

31
•
•
Low PT: J/Y, Uand B-physics
High PT: H/Z/W/τ➝μ, SUSY, exotics
•
Level 1: look for coincidence hits in
muon trigger chambers
–
–
Muon Trigger
Resistive Plate Chambers (barrel) and
Thin Gap Chambers (endcap)
pT resolved from coincidence hits in look-up
table
•
Level 2: refine Level 1 candidate with
precision hits from Muon Drift Tubes
(MDT) and combine with inner
detector track
•
Event Filter: use offline algorithms and
precision; complementary algorithm
does inside-out tracking and muon
reconstruction
80% acceptance
due to support
structures etc.
32
The Trigger Menu
• Collection of trigger signatures
• In LHC GPD’s menus there can be 100’s of algorithm
chains – defining which objects, thresholds and
algorithms, etc should be used
• Selections set to match the current physics priorities
and beam conditions within the bandwidth and rates
allowed by the TDAQ system
• Includes calibration & monitoring chains
• Principal mechanisms to adjust the accept rate (and
the relative frequency of different triggers)
– Adjusting thresholds
– Pre-scaling higher rate triggers (e.g. only accept every 10th
trigger of a particular type)
• Can be used to include a low rate of calibration events
33
Trigger Menu cont’d
• Basic Menu is defined at the start of a run
– Pre-scale factors can be changed during the course of a run
• Adjust triggers to match current luminosity
• Turn triggers on/off
34
Trigger Evolution in ATLAS
35
36
Matching problem
• Ideally
– off-line algorithms select all
the physics channel and
no background
– trigger algorithms select all
the physics accepted by the
off-line selection
(and no background)
• In practice, neither of these happen
– Need to optimise the combined
selection
Background
Off-line
Physics
channel
On-line
• For this reason many trigger studies quote trigger efficiency wrt
events which pass off-line selection
– BUT remember off-line can change algorithm, re-process and
recalibrate at a later stage
• So, make sure on-line algorithm selection is well known, controlled
and monitored
37
Other issues for the Trigger
• Optimisation of cuts
– Balance background rejection vs efficiency
• Efficiency and Monitoring
– In general need high trigger efficiency
– Also for many analyses need a well known efficiency
• Monitor efficiency by various means
– Overlapping triggers
– Pre-scaled samples of triggers in tagging mode (pass-through)
• Final detector calibration and alignment constants not
available for the trigger
– keep as up-to-date as possible
– allow for the lower precision in the trigger cuts
• Code used in trigger needs to be fast + very robust
– low memory leaks, low crash rate
38
Summary
• High-level triggers allow complex selection procedures to
be applied as the data is taken
– Thus allow large samples of rare events to be recorded
• The trigger stages - in the ATLAS example
– Level 1 uses inclusive signatures (mu’s; em/tau/jet; missing and
sum ET)
– Level 2 refines Level 1 selection, adds simple topology triggers,
vertex reconstruction, etc
– Level 3 refines Level 2 adds more refined topology selection
• Trigger menus need to be defined, taking into account:
– Physics priorities, beam conditions, HLT resources
• Include items for monitoring trigger efficiency and calibration
• Try to match trigger cuts to off-line selection
• Trigger efficiency should be as high as possible and well
monitored
• Must get it right - events thrown away are lost for ever!
• Triggering closely linked to physics analyses – so enjoy!
39
Physics Letters B cover
ATLAS and CMS “Higgs discovery”
papers published side by side in
Phys. Lett. B716 (2012)
40
2e2μ candidate with m2e2μ= 123.9 GeV
pT (e,e,μ,μ)= 18.7, 76, 19.6, 7.9 GeV,
12 reconstructed vertices
m (e+e-)= 87.9 GeV, m(μ+μ-) =19.6 GeV
41
Evolution of the excess with time
Significance
increase from
4th July to now
from including
2012 data for
H WW* search
42
Evolution of the excess with time
Significance
increase from
4th July to now
from including
2012 data for
H WW* search
43
Exotic Physics Search Summary
44
SUSY Searches
45
Additional Foils
46
ATLAS HLT Hardware
Each rack of HLT (XPU) processors contains
- ~30 HLT PC’s (PC’s very similar to Tier-0/1 compute nodes)
- 2 Gigabit Ethernet Switches
- a dedicated Local File Server
Final system will contain ~2300 PC’s
47
SDX1|2nd floor|Rows 3 & 2
48
Price to pay for the high luminosity:
larger-than-expected pile-up
Pile-up = number of interactions
per crossing
Tails up to ~20  comparable to
design luminosity
Period A:
up to end
August
Period B:
Sept-Oct
(50 ns operation; several machine parameters pushed
beyond design)
LHC figures used over the last 20 years:
~ 2 (20) events/crossing at L=1033 (1034)
Event with 20
reconstructed vertices
Z μμ
(ellipses have 20 σ size for
visibility reasons)
Challenging for trigger, computing resources, reconstruction of physics objects
(in particular ETmiss, soft jets, ..)
Precise modeling of both in-time and out-of-time pile-up in simulation is essential
49
Naming Convention
First Level Trigger (LVL1) Signatures in
capitals e.g.
LVL1
HLT
type
e
electron
g
photon
MU
mu
muon
HA
tau
tau
fj
forward
jet
JE
je
jet energy
JT
jt
jet
TM
xe
missing
energy
threshold
EM
MU 20 I
name isolated
HLT in lower case:
threshold
EF in tagging
mode
mu 20 i _ passEF
name isolated
New in 13.0.30:
•
Threshold is cut value applied
•
previously was ~95% effic. point.
•
FJ
More details : see :https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerPhysicsMenu
50
What is a minimum bias event ?
- event accepted with the only requirement being activity in the detector
with minimal pT threshold [100 MeV] (zero bias events have no requirements)
- e.g. Scintillators at L1 + (> 40 SCT S.P. or > 900 Pixel clusters) at L2
- a miminum bias event is most likely to be either:
- a low pT (soft) non-diffractive event
- a soft single-diffractive event
- a soft double diffractive event
(some people do not include the diffractive events in the definition !)
- it is characterised by:
- having no high pT objects : jets; leptons; photons
- being isotropic
- see low pT tracks at all phi in a tracking detector
- see uniform energy deposits in calorimeter as function of rapidity
- these events occur in 99.999% of collisions. So if any given crossing
has two interactions and one of them has been triggered due to a high pT
component then the likelihood is that the accompanying event will
be a dull minimum bias event.
51
Minimum Bias Trigger
•
•
•
Soft QCD studies
Provide control trigger on p-p collisions;
discriminate against beam-related
backgrounds (using signal time)
Minimum Bias Scintillators (MBTS)
installed in each end-cap;
•
•
Phys.Lett.B 688, Issue 1, 2010
Example: MBTS_1 – at least 1 hit in
MBTS
Also check nr. of hits in Inner Detector in
Level-2
Minbias Trigger Scintillator:
32 sectors on LAr cryostat
Main trigger for initial running
 coverage 2.1 to 3.8
LHC collision rate (nb=4)
LHC collision rate (nb=2)
52
Hadronic Tau Trigger
•
W/Z ➝ , SM &MSSM Higgs, SUSY, Exotics
•
Level 1: start from hadronic cluster – local maximum in
ΔηxΔφ = 0.2x0.2 – possible to apply isolation
•
Level 2: track and calorimeter information are
combined – narrow cluster with few matching tracks
•
Event Filter: 3D cluster reconstruction suppresses
noise; offline ID algorithms and calibration used
•
Typical background rejection factor of ≈5-10 from
Level 2+Event Filter
– Right: fake rate for loose tau trigger with pT > 12
GeV – aka tau12_loose
– MC is Pythia with no LHC-specific tuning
53
Jet Trigger
QCD multijet production, top, SUSY,
generic BSM searches
•
Level 1: look for local maximum in ET in
calorimeter towers of ΔηxΔφ = 0.4x0.4 to
0.8x0.8
•
Level 2: simplified cone clustering
algorithm (3 iterations max) on
calorimeter cells
•
Event Filter: anti-kT algorithm on
calorimeter cells; currently running in
transparent mode (no rejection)
Note in preparation
•
54
Jet Trigger in 2012
L2 Single Jets – cone algo. in L1 RoI.
L2 Multi-jets:
•
•
L2FS – fullscan anti-kT jets from L1Calo trigger tower info
L2PS – anti-kT jets in L1 & HLT RoI using cell-level info.
EF (single & multi-jets)- Fullscan
anti-kT jets from topological
clusters of cells
5555
Missing ET Trigger
•
•
•
•
SUSY, Higgs
Level 1: ETmiss and ET calculated from
all calorimeter towers
Level 2: Initially only muon corrections
possible. Later fetch energy sums
from each part of calo ROS
Event Filter: re-calculate from
calorimeter cells and reconstructed
muons
Level 1
5 GeV threshold
Level 1
20 GeV threshold
56
Trigger Menus
• For details of the current ideas on ATLAS Menu evolution see
– https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerPhysicsMenu
• Gives details of menu since Startup and for each year to 2012
• Corresponding information for CMS is at
– https://twiki.cern.ch/twiki/bin/view/CMS/TriggerMenuDevelopment
• The expected performance of ATLAS for different physics
channels (including the effect of the trigger) is documented in
http://arxiv.org/abs/0901.0512 (beware - nearly 2000 pages)
57
ATLAS works!
Top-pair candidate - e-mu + 2b-tag
58
CMS works!
59