High Level Triggering Fred Wickens High Level Triggering (HLT) • Introduction to triggering and HLT systems – What is Triggering – What is High Level Triggering – Why do we need it • Case study of ATLAS HLT (+ some comparisons with other experiments) • Summary 2 Why do we Trigger and why multi-level • Over the years experiments have focussed on rarer processes – Need large statistics of these rare events – DAQ system (and off-line analysis capability) under increasing strain • limiting useful event statistics • Aim of the trigger is to record just the events of interest • i.e. Trigger selects the events we wish to study • Originally - only read-out the detector if Trigger satisfied – Larger detectors and slow serial read-out => large dead-time – Also increasingly difficult to select the interesting events • Introduced: Multi-level triggers and parallel read-out – At each level apply increasingly complex algorithms to obtain better event selection/background rejection • These have: – Led to major reduction in Dead-time – which was the major issue – Managed growth in data rates – this remains the major issue 3 Summary of ATLAS Data Flow Rates • From detectors > 1014 Bytes/sec • After Level-1 accept ~ 1011 Bytes/sec • Into event builder ~ 109 Bytes/sec • Onto permanent storage ~ 108 Bytes/sec ~ 1015 Bytes/year 4 The evolution of DAQ systems 5 TDAQ Comparisons 6 Level 1 • Time: few microseconds • Hardware based – Using fast detectors + fast algorithms – Reduced granularity and precision • calorimeter energy sums • tracking by masks • During Level-1 decision time store event data in front-end electronics – at LHC use pipeline - as collision rate shorter than Level-1 decision time • For details of Level-1 see Dave Newbold talk 7 High Level Trigger - Levels 2 + 3 • Level-2 : Few milliseconds (10-100) – Partial events received via high-speed network – Specialised algorithms • 3-D, fine grain calorimetry • tracking, matching • Topology • Level-3 : Up to a few seconds – Full or partial event reconstruction • after event building (collection of all data from all detectors) • Level-2 + Level-3 – Processor farm with Linux server PC’s – Each event allocated to a single processor, large farm of processors to handle rate 8 Summary of Introduction • For many physics analyses, aim is to obtain as high statistics as possible for a given process – We cannot afford to handle or store all of the data a detector can produce! • The Trigger – selects the most interesting events from the myriad of events seen • I.e. Obtain better use of limited output band-width • Throw away less interesting events • Keep all of the good events(or as many as possible) – must get it right • any good events thrown away are lost for ever! • High level Trigger allows: – More complex selection algorithms – Use of all detectors and full granularity full precision data 9 Case study of the ATLAS HLT system Concentrate on issues relevant for ATLAS (CMS very similar issues), but try to address some more general points Starting points for any Trigger system • physics programme for the experiment – what are you trying to measure • accelerator parameters – what rates and structures • detector and trigger performance – what data is available – what trigger resources do we have to use it • Particularly network b/w + cpu performance 11 Physics at the LHC 7 TeV Interesting events are buried in a sea of soft interactions B physics High energy QCD jet production top physics Higgs production 12 The LHC and ATLAS/CMS • LHC has – Design luminosity 1034 cm-2s-1 • In 2010 from 1027 – 2x1032 ; 2011 up to 2x1033 – Design bunch separation 25 ns (bunch length ~1 ns) • This results in – ~ 23 interactions / bunch crossing • ~ 80 charged particles (mainly soft pions) / interaction • ~2000 charged particles / bunch crossing • Total interaction rate – b-physics – t-physics – Higgs fraction ~ 10-3 fraction ~ 10-8 fraction ~ 10-11 109 sec-1 106 sec-1 10 sec-1 10-2 sec-1 13 Physics programme • Higgs signal extraction important - but very difficult • There is lots of other interesting physics – – – – – B physics and CP violation quarks, gluons and QCD top quarks SUSY ‘new’ physics • Programme will evolve with: luminosity, HLT capacity and understanding of the detector – low luminosity (2010 - 2011) • high PT programme (Higgs etc.) • b-physics programme (CP measurements) – high luminosity (2013 or 2014?) • high PT programme (Higgs etc.) • searches for new physics 14 Trigger strategy at LHC • To avoid being overwhelmed use signatures with small backgrounds – Leptons – High mass resonances – Heavy quarks • The trigger selection looks for events with: – – – – Isolated leptons and photons, -, central- and forward-jets Events with high ET Events with missing ET 15 Example Physics signatures Objects Physics signatures Electron 1e>25, 2e>15 GeV Higgs (SM, MSSM), new gauge bosons, extra dimensions, SUSY, W, top Photon 1γ>60, 2γ>20 GeV Higgs (SM, MSSM), extra dimensions, SUSY Muon 1μ>20, 2μ>10 GeV Higgs (SM, MSSM), new gauge bosons, extra dimensions, SUSY, W, top Jet 1j>360, 3j>150, 4j>100 GeV SUSY, compositeness, resonances Jet >60 + ETmiss >60 GeV SUSY, exotics Tau >30 + ETmiss >40 GeV Extended Higgs models, SUSY 16 Trigger 40 MHz ARCHITECTURE DAQ Three logical levels Hierarchical data-flow LVL1 - Fastest: Only Calo and Mu Hardwired On-detector electronics: Pipelines ~40 ms LVL2 - Local: LVL1 refinement + track association Event fragments buffered in parallel ~4 sec. LVL3 - Full event: “Offline” analysis Full event in processor farm ~2.5 ms ~ 200 Hz Physics ~1 PB/s (equivalent) ~ 300 MB/s 17 Selected (inclusive) signatures Process Level-1 2 em, ET>20 GeV H0 H0Z Z* + – + – 2 em, ET>20 GeV 2 µ, pT>6 GeV 1 em, ET>30 GeV 1 µ, pT>20 GeV 2 em, ET>20 GeV Z+–+X 2 µ, pT>6 GeV 1 em, ET>30 GeV 1 µ, pT>20 GeV 1 em, ET>30 GeV t t leptons+jets 1 µ, pT>20 GeV W', Z'jets 1 jet, ET>150 GeV SUSYjets 1 jet, ET>150 GeV E miss T Level-2 2 , ET>20 GeV 2 e, ET>20 GeV 2 µ, ET>6 GeV, I 1 e, ET>30 GeV 1 µ, ET>20 GeV, I 2 e, ET>20 GeV 2 µ, ET>6 GeV, I 1 e, ET>30 GeV 1 µ, ET>20 GeV, I 1 e, ET>30 GeV 1 µ, ET>20 GeV, I 1 jet, ET>300 GeV 3 jet, ET>150 GeV E miss T 18 Trigger design – Level-1 • Level-1 – sets the context for the HLT – reduces triggers to ~75 kHz • Uses limited detector data – Fast detectors (Calo + Muon) – Reduced granularity • Trigger on inclusive signatures • muons; • em/tau/jet calo clusters; missing and sum ET • Hardware trigger – Programmable thresholds – CTP selection based on multiplicities and thresholds 19 Level-1 Selection • The Level-1 trigger – an “or” of a large number of inclusive signals – set to match the current physics priorities and beam conditions • Precision of cuts at Level-1 is generally limited • Adjust the overall Level-1 accept rate (and the relative frequency of different triggers) by – Adjusting thresholds – Pre-scaling (e.g. only accept every 10th trigger of a particular type) higher rate triggers • Can be used to include a low rate of calibration events • Menu can be changed at the start of run – Pre-scale factors may change during the course of a run 20 Trigger design - HLT strategy • Level 2 – confirm Level 1, some inclusive, some semiinclusive, some simple topology triggers, vertex reconstruction (e.g. two particle mass cuts to select Zs) • Level 3 – confirm Level 2, more refined topology selection, near off-line code 21 Trigger design - Level-2 • Level-2 reduce triggers to ~2 kHz – Note CMS does not have a physically separate Level-2 trigger, but the HLT processors include a first stage of Level-2 algorithms • Level-2 trigger has a short time budget – ATLAS ~40 milli-sec average • Note for Level-1 the time budget is a hard limit for every event, for the High Level Trigger it is the average that matters, so OK for a small fraction of events to take times much longer than this average • Full detector data is available, but to minimise resources needed: – – – – Limit the data accessed Only unpack detector data when it is needed Use information from Level-1 to guide the process Analysis proceeds in steps with possibility to reject event after each step – Use custom algorithms 22 Regions of Interest • The Level-1 selection is dominated by local signatures (I.e. within Region of Interest RoI) – Based on coarse granularity data from calo and mu only • Typically, there are 1-2 RoI/event • ATLAS uses RoI’s to reduce network b/w and processing power required 23 Trigger design - Level-2 - cont’d • Processing scheme – extract features from sub-detectors in each RoI – combine features from one RoI into object – combine objects to test event topology • Precision of Level-2 cuts – Limited (although better than at Level-1) – Emphasis is on very fast algorithms with reasonable accuracy • Do not include many corrections which may be applied off-line – Calibrations and alignment available for trigger not as precise as ones available for off-line 24 ARCHITECTURE Trigger Calo MuTrCh 40 MHz 40 MHz LVL1 Muon Trigger ROD ROIB L ROD 120 RoI’s LVL2 ~ 10 ms RoI requests L2SV ROB ROD GB/s ROB ROB ROS RoI data = 1-2% L2P L2P L2P T Event Filter EFP EFP EFP L2N ~2 GB/s LVL2 accept ~ 1 sec ~ 1 PB/s FE Pipelines 2.5 ms LVL1 accept 75 kHz ~2 kHz Other detectors 2.5 ms Calorimeter Trigger H DAQ Read-Out Drivers Read-Out Links Read-Out Buffers Read-Out Sub-systems ~3 GB/s Event Builder EB ~3 GB/s EFN ~ 300 MB/s ~ 200 Hz ~ 300 MB/s 25 CMS Event Building • CMS perform Event Building after Level-1 • Simplifies the architecture, but places much higher demand on technology: – Network traffic ~100 GB/s – 1st stage use Myrinet – 2nd stage has 8 GbE slices • Time will tell which is better 26 Example for Two electron trigger LVL1 triggers on two isolated STEP 4 e/m clusters with pT>20GeV Signature (possible signature: Z–>ee) HLT Strategy: Validate step-by-step Check intermediate signatures Reject as early as possible Sequential/modular approach facilitates early rejection STEP 3 Signature STEP 2 e30i + Iso– lation e30 Iso– lation + pt> 30GeV e ecand STEP 1 Cluster shape Level1 seed EM20i e30 pt> 30GeV + track finding Signature e30i e track finding + time Signature ecand Cluster shape + EM20i 27 Trigger design - Event Filter / Level-3 • Event Filter reduce triggers to ~200 Hz • Event Filter budget ~ 4 sec average • Full event detector data is available, but to minimise resources needed: – Only unpack detector data when it is needed – Use information from Level-2 to guide the process – Analysis proceeds in steps with possibility to reject event after each step – Use optimised off-line algorithms 28 EM ROI Execution of a Trigger Chain Electromagnetic clusters Level1: Region of Interest is found and position in EM calorimeter is passed to Level 2 L2 calorim. cluster? L2 tracking track? Level 2 seeded by Level 1 •Fast reconstruction algorithms •Reconstruction within RoI match? E.F.calorim. E.F.tracking Ev.Filter seeded by Level 2 •Offline reconstruction algorithms •Refined alignment and calibration track? e/ reconst. e/ OK? 29 Minimum Bias Trigger • • • Soft QCD studies Provide control trigger on p-p collisions; discriminate against beam-related backgrounds (using signal time) Minimum Bias Scintillators (MBTS) installed in each end-cap; • • Phys.Lett.B 688, Issue 1, 2010 Example: MBTS_1 – at least 1 hit in MBTS Also check nr. of hits in Inner Detector in Level-2 Minbias Trigger Scintillator: 32 sectors on LAr cryostat Main trigger for initial running h coverage 2.1 to 3.8 LHC collision rate (nb=4) LHC collision rate (nb=2) 30 e/γ Trigger • pT≈3-20 GeV: b/c/tau decays, SUSY • pT≈20-100 GeV: W/Z/top/Higgs • pT>100 GeV: exotics • Level 1: local ET maximum in ΔηxΔφ = 0.2x0.2 with possible isolation cut • Level 2: fast tracking and calorimeter clustering – use shower shape variables plus track-cluster matching L1 EM trigger pT > 5GeV • Event Filter: high precision offline algorithms wrapped for online running 31 • Discriminate against hadronic showers based on shower shape variables • Use fine granularity of LAr calorimeter • Resolution improved in Event Filter with respect to Level 2 E 37cells Rh E 77cells 32 • • Low PT: J/Y, Uand B-physics High PT: H/Z/W/τ➝μ, SUSY, exotics • Level 1: look for coincidence hits in muon trigger chambers – – Muon Trigger Resistive Plate Chambers (barrel) and Thin Gap Chambers (endcap) pT resolved from coincidence hits in look-up table • Level 2: refine Level 1 candidate with precision hits from Muon Drift Tubes (MDT) and combine with inner detector track • Event Filter: use offline algorithms and precision; complementary algorithm does inside-out tracking and muon reconstruction 80% acceptance due to support structures etc. 33 Hadronic Tau Trigger • W/Z ➝ , SM &MSSM Higgs, SUSY, Exotics • Level 1: start from hadronic cluster – local maximum in ΔηxΔφ = 0.2x0.2 – possible to apply isolation • Level 2: track and calorimeter information are combined – narrow cluster with few matching tracks • Event Filter: 3D cluster reconstruction suppresses noise; offline ID algorithms and calibration used • Typical background rejection factor of ≈5-10 from Level 2+Event Filter – Right: fake rate for loose tau trigger with pT > 12 GeV – aka tau12_loose – MC is Pythia with no LHC-specific tuning 34 Jet Trigger QCD multijet production, top, SUSY, generic BSM searches • Level 1: look for local maximum in ET in calorimeter towers of ΔηxΔφ = 0.4x0.4 to 0.8x0.8 • Level 2: simplified cone clustering algorithm (3 iterations max) on calorimeter cells • Event Filter: anti-kT algorithm on calorimeter cells; currently running in transparent mode (no rejection) Note in preparation • 35 Missing ET Trigger • • • • SUSY, Higgs Level 1: ETmiss and ET calculated from all calorimeter towers Level 2: only muon corrections possible (at present) Event Filter: re-calculate from calorimeter cells and reconstructed muons Level 1 5 GeV threshold Level 1 20 GeV threshold 36 The Trigger Menu • Collection of trigger signatures • In LHC GPD’s menus there can be 100’s of algorithm chains – defining which objects, thresholds and algorithms, etc should be used • Selections set to match the current physics priorities and beam conditions within the bandwidth and rates allowed by the TDAQ system • Includes calibration & monitoring chains • Principal mechanisms to adjust the accept rate (and the relative frequency of different triggers) – Adjusting thresholds – Pre-scaling (e.g. only accept every 10th trigger of a particular type) higher rate triggers • Can be used to include a low rate of calibration events 37 Example use of thresholds/prescales at Level-1 L1 trigger items and estimated rates at 10^31 cm−2 s−1 for jets Jet ET spectrum at 10^31 cm−2 s−1 before (dashed) and after (solid) pre-scaling at L1 38 Trigger Menu cont’d • Basic Menu is defined at the start of a run – Pre-scale factors can be changed during the course of a run • Adjust triggers to match current luminosity • Turn triggers on/off 39 Trigger Commissioning in ATLAS • First Collisions : L1 only • Since June : gradual activation of HLT 40 Matching problem Background Off-line Physics channel On-line 41 Matching problem (cont.) • ideally – off-line algorithms select phase space which shrink-wraps the physics channel – trigger algorithms shrink-wrap the off-line selection • in practice, this doesn’t happen – need to match the off-line algorithm selection • For this reason many trigger studies quote trigger efficiency wrt events which pass off-line selection – BUT off-line can change algorithm, re-process and recalibrate at a later stage • So, make sure on-line algorithm selection is well known, controlled and monitored 42 Selection and rejection • as selection criteria are tightened – background rejection improves – BUT event selection efficiency decreases 1 select / reject fraction 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 0 0.2 0.4 0.6 0.8 1 cut value select reject 43 Other issues for the Trigger • Efficiency and Monitoring – In general need high trigger efficiency – Also for many analyses need a well known efficiency • Monitor efficiency by various means – Overlapping triggers – Pre-scaled samples of triggers in tagging mode (pass-through) • Final detector calibration and alignment constants not available immediately - keep as up-to-date as possible and allow for the lower precision in the trigger cuts when defining trigger menus and in subsequent analyses • Code used in trigger needs to be very robust - low memory leaks, low crash rate, fast 44 Other issues for the Trigger – cont’d • Beam conditions and HLT resources will evolve over several years (for both ATLAS and CMS) – In 2010 luminosity low, but also HLT capacity had < 50% of full system • For details of the current ideas on ATLAS Menu evolution see – https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerPhysicsMenu • Gives details of menu since Startup and for 2011 • Corresponding information for CMS is at – https://twiki.cern.ch/twiki/bin/view/CMS/TriggerMenuDevelopment • The expected performance of ATLAS for different physics channels (including the effect of the trigger) is documented in http://arxiv.org/abs/0901.0512 (beware - nearly 2000 pages) 45 Summary • High-level triggers allow complex selection procedures to be applied as the data is taken – Thus allow large samples of rare events to be recorded • The trigger stages - in the ATLAS example – Level 1 uses inclusive signatures (mu’s; em/tau/jet; missing and sum ET) – Level 2 refines Level 1 selection, adds simple topology triggers, vertex reconstruction, etc – Level 3 refines Level 2 adds more refined topology selection • Trigger menus need to be defined, taking into account: – Physics priorities, beam conditions, HLT resources • Include items for monitoring trigger efficiency and calibration • Try to match trigger cuts to off-line selection • Trigger efficiency should be as high as possible and well monitored • Must get it right - events thrown away are lost for ever! • Triggering closely linked to physics analyses – so enjoy! 46 ATLAS works! Top-pair candidate - e-mu + 2b-tag 47 CMS works! 48 Additional Foils 49 ATLAS HLT Hardware Each rack of HLT (XPU) processors contains - ~30 HLT PC’s (PC’s very similar to Tier-0/1 compute nodes) - 2 Gigabit Ethernet Switches - a dedicated Local File Server Final system will contain ~2300 PC’s 50 SDX1|2nd floor|Rows 3 & 2 51 Naming Convention First Level Trigger (LVL1) Signatures in capitals e.g. LVL1 HLT type e electron g photon MU mu muon HA tau tau fj forward jet JE je jet energy JT jt jet TM xe missing energy threshold EM MU 20 I name isolated HLT in lower case: threshold EF in tagging mode mu 20 i _ passEF name isolated New in 13.0.30: • Threshold is cut value applied • previously was ~95% effic. point. • FJ More details : see :https://twiki.cern.ch/twiki/bin/view/Atlas/TriggerPhysicsMenu 52 What is a minimum bias event ? - event accepted with the only requirement being activity in the detector with minimal pT threshold [100 MeV] (zero bias events have no requirements) - e.g. Scintillators at L1 + (> 40 SCT S.P. or > 900 Pixel clusters) at L2 - a miminum bias event is most likely to be either: - a low pT (soft) non-diffractive event - a soft single-diffractive event - a soft double diffractive event (some people do not include the diffractive events in the definition !) - it is characterised by: - having no high pT objects : jets; leptons; photons - being isotropic - see low pT tracks at all phi in a tracking detector - see uniform energy deposits in calorimeter as function of rapidity - these events occur in 99.999% of collisions. So if any given crossing has two interactions and one of them has been triggered due to a high pT component then the likelihood is that the accompanying event will be a dull minimum bias event. 53 Example Level-1 Menu for 2x10^33 Level-1 signature Output Rate (Hz) EM25i 12000 2EM15i 4000 MU20 800 2MU6 200 J200 200 3J90 200 4J65 200 J60 + XE60 400 TAU25i + XE30 2000 MU10 + EM15i 100 Others (pre-scaled, exclusive, monitor, calibration) Total 5000 ~25000 54 L1 Rates 1031 14.4.0 L1 output rate 1031 (Total Rate 20 kHz) Multi Muon 0% Trigger Group Multi Tau 2% Single EM 28% Multi EM 32% Single Muon 9% XE 0% Jets 0% Multi Object 28% Single Tau 1% Multi EM Multi Object Single EM Single Muon Multi Tau Single Tau Jets Multi Muon XE TOTAL Rate (Hz) 6400 5500 5500 1700 470 150 80 70 50 20000 Removing overlaps between single+multi EM gives 18 kHz Total estimated L1 rate with all overlaps removed is ~ 12 kHz 55 L2 Rates 1031 14.4.0 L2 output rate 1031 (Total Rate 900 Hz) XE+ 10% Jets 3% Trigger Group B Phys 5% Electrons 34% Photons 5% Tau+X 20% Muons 23% Electrons Muons Taus+X XE+ Photons B Phys Jets TOTAL Rate (Hz) 310 210* 180 82 46 43 22 900 X=anything; + includesJE, TE, anything with MET except taus; Bphys includes Bjet * Manually prescaled off pass-through triggers mu4_tile, mu4_mu6 Total estimated L2rate with all overlaps removed is 840 Hz 56 EF Rates 1031 14.4.0 EF output rate 1031 (Total Rate 310 Hz) Trigger Group B Phys 12% XE+ 4% Misc 4% Electrons 22% Jets 8% Photons 6% Muons 26% Tau+X 18% Rate (Hz) Muons Electrons Tau+X B Phys Jets Photons XE+ Misc TOTAL 80 67 56 37 25 18 13 13 310 91 Hz total is in prescaled triggers; 51 Hz of prescaled triggers is unique rate Total estimated EF Rate with overlaps removed is 250 Hz 57 L1 Rates 1032 14.4.0 L1 output rate 1032 (Total Rate 73 kHz) Trigger Group Multi Muon 1% Multi Tau 6% XE 0% Single EM 11% Multi EM 15% Jets 0% Single Muon 24% Multi Object 42% Single Tau 1% Rate (Hz) Multi Object 30000 Single Muon 17000 Multi EM 11000 Single EM 8100 Multi Tau 4300 Single Tau 870 Multi Muon 690 Jets 300 XE 300 TOTAL 73000 Total estimated L1 rate with all overlaps removed is 46 kHz 58 L2 Rates 1032 14.4.0 L2 output rate 1032 (Total Rate 2600 Hz) Trigger Group Other 1% 3 Objects B Phys 10% Electrons 15% 4% Muons 11% XE+ 22% Jets 1% Photons 5% Tau+X 31% Rate (Hz) Tau+X 820 XE+ 590 Electrons 390 Muons 280 3 Objects 270 Photons 120 B Phys 110 Jets 33 Misc 28 TOTAL 2600 Total estimated L2 with all overlaps removed is 1700 (too high!) 59 EF Rates 1032 14.4.0 EF output rate 1032 (Total Rate 510 Hz) Trigger Group Misc 2% Tau+X 3 Objects 9% BPhys 9% Electrons 15% Muons 9% XE+ 8% Jets 2% Photons 9% Tau+X 37% Rate (Hz) 187 Electrons 77 Muons 46 Photons 46 BPhys 45 3 Objects 45 XE+ 42 Jets 11 Misc 11 TOTAL 510 Total estimated EF rates with all overlaps removed is 390 Hz (Fixing L2 will likely come close to fixing EF as well) 60 End of pp trigger operations in 2010 Run 167607 - record peak luminosity 2.1x1032cm-2s-1 Trigger group Trigger chain Rate [Hz] Singlemuon EF_mu13_tight 24 Di-muon EF_2mu6 28 Trigger evolution in 2010 Singleelectron EF_e15_mediu m 38 Di-electron EF_2e10_loos e 2.4 For a given threshold tighten selection Loose->medium->tight Non-isolation->isolation Singlephoton EF_g40_loose 9 Di-photon EF_2g15_loos e L1 output 35kHz, L2 output 5kHz, EF output 400Hz Go higher in pT 2.1 Trigger Report Due to lack of time no physics data collected with 50ns BS 61 61 Example rates for different objects 62