A decomposition of the IceCube
DAQ in a small number of incredibly
information-dense slides
IceCube DAQ Review
2010/11/17
Kael Hanson
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
1
IceCube DAQ
TITLE
The DAQ Design
Hierarchy
• IN
– Performance: yes
– Maintainability: yes, probably, once we finally get to maintenance mode
– Testability: wanted a system that was testable without using SPS. IMO
we’ve fallen a bit short of this goal.
– Developability (OK that’s not a real word): wanted a fast turnaround on
the edit/compile/package/deploy/test cycle – especially in late 2006 /
early 2007 when we were under the gun to get something delivered. We
can typically march through this whole process in 1-2 minutes.
– Portability: there is SPS and SPTS but to achieve the last goal we wanted it
possible to run the DAQ on UNIX / Mac laptops without much
specialization. DAQ code itself runs just about anywhere – I’ve even
brought parts up under WinXP. Python framework is less forgiving.
– Minimal configuration input: yes, but we have some issues as we’ll discuss
• NOT IN
– Ease of use: we figured that world+dog is not interested on their laptops
– Event analysis: it was decided that we would leave the data quality checks
to verification and monitoring. I’m going to admit right now that we don’t
have any framework to speak of designed to analyze our output data.
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
2
IceCube DAQ
TITLE
DAQ Design Goals
ICECUBE DAQ USES MOSTLY TCP STREAMS
OVER GIG-E TO TRANSFER DATA BETWEEN
COMPONENTS. THE COMM LINK BETWEEN
DOMS AND HUBS IS THE CUSTOM DOR
PROTOCOL – 1 MBIT/S MASTER/SLAVE .
LINES INDICATE
1. THIN LINES: UNIDIRECTIONAL DATA
2. THICK LINES: BI-DIRECTIONAL
REQUESTER/SENDER ARCHITECTURE
NOT SHOWN
1. PYTHON CONTROL FRAMEWORK
2. DAQ COMPONENT SYSTEM
3. SECONDARY STREAMS PRODUCED BY
THE STRING HUB COMPONENTS
4. SECONDARY BUILDERS
NOTE: AMANDA SYSTEM IN SHADED RECTANGLE IS DEPRECATED AND HAS
NOT BEEN USED SINCE END OF IC40 DATA RUN MAY 2009
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
3
IceCube DAQ
TITLE
Top Level Data-Flow
Diagram
THE HUB-DOR MESSAGING SYSTEM IS ROLLED
UP IN THE DOMApp CLASS WHICH CURRENTLY
IMPLEMENTS MOST OF THE MESSAGES DEFINED
IN THE DOMAPP MESSAGING API. THE Driver
CLASS HANDLES INTERACTION WITH THE
FUNCTIONALITY EXPOSED IN THE /PROC FS
THE STRINGHUB IS DIVIDED INTO TWO MAJOR FUNCTIONAL SECTIONS: THE FRONT
HALF WHICH CONTAINS ALL OF THE CODE TO TALK TO THE DOMS AND COLLECT
DATA FROM THEM . THE BACK HALF CONTAINS THE SENDER – A MAJOR
COMPONENT THAT CREATES THE TRIGGER SUMMARIES FROM HLC TAGGED HITS
(BUT IS RUN-TIME CONFIGURABLE TO PASS ALL HITS, IN FACT), BUFFERS HIT DATA,
AND FULFILLS EVENTBUILDER-GENERATED READOUT REQUESTS.
1. INITIALIZATION
2. CONFIGURATION
3. RUNNING
1. TAKE TCAL 1 1HZ
2. READ HIT DATA
3. READ MONI DATA
4. READ SN DATA
5. SLEEP, MAYBE
THE MONI/TCAL/SN OUTPUT ENGINES SIMPLY FORWARD HITS TO THE SECONDARY
BUILDERS.
EACH STRINGHUB CONTAINS CODE TO EFFECT A NOISE SIMULATION WHICH CAN
BE ACTIVATED BY RUN-TIME SWITCHES AND CONFIGURATION OPTIONS …
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
4
IceCube DAQ
TITLE
StringHub Internals
SORTING IN ICECUBE DAQ FOLLOWS A CERTAIN PATTERN:
• YOU HAVE N INPUT STREAMS WHICH ARE THEMSELVES ORDERED FOLLOWING SOME
METHODOLOGY BE IT TIMESTAMPS OR BE IT EVENT NUMBERS.
• YOU WANT TO MERGE DOWN INTO A SINGLE OUTPUT STREAM.
• YOU WANT TO DO IT FAST (O(N LOG(N)) IN BIG-OH).
• YOU DON’T HAVE ANY CONTROL OF WHEN THE VALUES ARRIVE
SOMEWHAT UNEXPECTEDLY, SEEM TO BE THE ONLY PEOPLE THAT WANT TO DO THIS
OPERATION IN THE HISTORY OF INFORMATION PROCESSING. SO WE HAD TO DEVELOP
OUR OWN ALGORITHMS. THIS IS THE CORE ALGORITHM AS OF 2006:
CONSTRUCT IN YOUR MIND 3 “NODES” AND CALL THEM L, R, AND SINK. L AND R ARE
PEERS AND SINK IS LIKE A DRAINAGE PIPE FOR THEM. EACH CAN HOLD A LIST OF
OBJECTS. YOU CAN PUSH OBJECTS INTO L OR R FROM THE OUTSIDE. WHEN YOU PUSH
AN OBJECT INTO A NODE IT ADDS THE OBJECT TO ITS LIST AND THEN ASKS ITS PEER IF IT
IS EMPTY. IF THE PEER IS EMPTY IT RETURNS. IF THE PEER IS NOT EMPTY THE TWO
PEERS DECIDE WHO HAS THE LESSER OBJECT AND THEN THEY REMOVE IT FROM THEIR
INTERNAL LISTS AND PUSH THAT OBJECT INTO THE SINK WHICH THEN RECURSIVELY WILL
EXECUTE THE SAME PUSH RULE. IF SINK IS PEERLESS – IT’S A TERMINAL NODE – THEN IT
SIMPLY ACCUMULATES OBJECTS.
A SIMPLE EXAMPLE OF THE SORT IS GIVEN IN THE SEQUENCE OF 5 CARTOONS AT THE
LEFT.
THE 6TH CARTOON SHOWS HOW YOU MAY CONSTRUCT ARBITRARILY LARGE SORTING
TREES BY CONNECTING SINKS AS PEERS OF OTHER SINKS AND SO ON AND SO FORTH.
YOU GET A 2N INPUT TREE. IF YOU WANT FEWER INPUTS DROP THE ∞ SYMBOL INTO AN
UNCONNECTED NODE. IF YOU WANT TO FLUSH AND TERMINATE A SORTING TREE DROP
∞ INTO EACH INPUT. THE SYMBOLS WILL PERCOLATE THROUGH THE TREE
STRINGHUBS USE RAW HNK1 TREE – OTHER COMPONENTS USE TREE
INSIDE SPLICER INTERFACE TO PRESERVE BACKWARDS
COMPATIBILITY WITH OLDER DAQ SORTING.
SPEED : 500,000 OBJECTS/SEC ON NEHALEM ARCHITECTURES
THIS IS THE MAIN BOTTLENECK IN SENDING ALL HITS TO TRIGGERS. TRACK ENGINE
GROUP CLAIMS SIMILAR ALGORITHM GIVES 10X SPEED.
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
5
IceCube DAQ
TITLE
Cascaded Binary Sorting
Tree (aka HKN1)
THE SECONDARY BUILDER DOESN’T REALLY DO THAT MUCH – IT IS ESSENTIALLY
THREE SPLICERS WRAPPED IN A DAQ COMPONENT TO MERGE THE SO-CALLED
SECONDARY STREAMS BEING EMITTED BY THE STRING HUBS.
THIS COULD BE A GOOD PLACE TO DO CERTAIN THINGS AS IT IS THE ONE POINT
IN DAQ WHERE ALL OF THIS DATA IS GLOBALLY VISIBLE IN ONE PLACE, ALL
NICELY TIME-ORDERED:
• MONITORING, IF MONITORING WERE EVER PORTED TO JAVA. RECALL THAT
THIS IS PRETTY MUCH AS CLOSE TO LIVE AS YOU CAN GET WITH LATENCIES
ON THE ORDER OF SECONDS. DURING THE RECENT DROPPED HITS
INVESTIGATIONS I SUGGESTED THAT WE PUT THE OVERFLOW MONITORING
HERE.
• SUPERNOVA TRIGGERING, IF SUPERNOVA TRIGGERING WERE EVER PORTED
TO JAVA. AGAIN, THERE IS NO DELAY DUE TO HAND-OFF OF DATA TO
DOWNSTREAM PROCESSES.
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
6
IceCube DAQ
TITLE
Secondary Builders
TRIGGER ALGORITHMS OPERATE BY EXAMINING HIT PAYLOADS (CHANNEL,
TIME) WHICH COME IN ORDERED BY VIRTUE OF SPLICER IN FRONT WHICH
SORTS (PRE-SORTED) STREAMS FROM THE MANY STRINGHUBS. DAVE G HAS
RECENTLY SPLIT THE SORTING AND I/O THREAD FROM THE TRIGGER
EXECUTION THREAD. WE (OK DAVE) COULD GO FURTHER STILL BY THREADING
EACH TRIGGER ALGORITHM AS THEY ARE IN PRINCIPLE NON-INTERACTING.
FOR NOW THE TRIGGER THREAD STEPS IN SEQUENCE OVER EACH TRIGGER
DEFINED IN THE TRIGGER CONFIGURATION.
THE ICECUBE DAQ TRIGGER IS A MULTI-LEVEL SYSTEM WITH THE IN-ICE AND
ICETOP FORMING THE FRONTLINE AGAINST THE HUBS AND THE GLOBAL
TRIGGER SITTING ABOVE THEM, MERGING OVERLAPPING TRIGGERS AND
SENDING TRIGGER REQUESTS TO THE EVENT BUILDER.
TO DEFINE A NEW TRIGGER ONE NEEDS TO :
• SUBCLASS ABSTRACT TRIGGER
• OVERRIDE THE ADDPARAMETER AND RUNTRIGGER METHODS
• BE SURE TO UPDATE THE “EARLIEST PAYLOAD OF INTEREST” FOR THE
SPLICER (FWIW I THINK EACH TRIGGER SHOULD JUST GET ITS OWN COPY OF
HIT REFERENCES AND NOT WORRY ABOUT THE COMMON /SPLICER/ AREA –
I GUESS THIS MAY BE A HANGOVER FROM THE BYTE BUFFER CACHE DAYS)
IT ACTUALLY TURNS OUT TO BE NOT THAT HARD TO MAKE A TRIGGER IF YOU
ARE JAVA-LITERATE AND THIS SUMMER PAST WE HAD PROOF FROM T
GLUESENKAMP WHO MADE THE SLOWMONOPOLE TRIGGER HAVING NEVER
TOUCHED DAQ CODE BEFORE. SO I THINK WITH PERHAPS SOME DOCS IT
WOULD BE POSSIBLE TO OPEN THE TRIGGER DEVELOPMENT TO ICECUBE.
THERE ARE A FEW LINGERING PROBLEMS IN THE TRIGGER – NAMELY :
1.
FLASHER AFTER BURSTS KILL TRIGGER
2.
CLUSTER TRIGGERS SPONTANEOUSLY DIE WITH ODD NPE
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
7
IceCube DAQ
TITLE
Triggers
Name
Description
Simple Majority
Look for M hits inside time window . M is 8 for IceCube 3 for DeepCore and 6 for IceTop
PhysicsMinBias
Look for 1 hit with prescale and deadtime to remove large-event bias
MinBias
Trigger on every Nth hit
Calibration
Form trigger around certain hit type (Flasher Hits for example)
Cluster
Look for runs of M DOMs inside N adjacent channels on a string and in time window. This is
the current implementation of the string trigger concept but can efficiently run in the central
in-ice trigger unit
Cylinder
Look for local grouping of hits inside cylindrical volume
FixedRateTrigger
Can be programmed to fire every N nanoseconds
Multiplicity String
Original implementation of the string trigger – was used for ULEE – and intended to run inside
Hubs which nominally support a trigger process
SlowMP
Hard to describe – looks for triplets separated by large time windows and actually computes
some physical quantities. Has extremely long readout window – O(1 ms) – which may cause
problems with interpretation of events (many muons in 1 ms get built together and considered
as 1 event). MP = (magnetic) monopoles – this trigger should be sensitive to MM’s down to
beta ~ 10-3
ThroughPut
GT trigger module which simply stitches together triggers and merges readout windows
One/Two/Three-Veto
GT trigger modules which can veto on incoming triggers of certain type
Two/Three-Coincidence
GT trigger modules which require 2/3 incoming triggers of certain type
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
8
IceCube DAQ
TITLE
Current Trigger
Algorithms
EVENT BUILDER IS COMPLEX BUT PRINCIPLE OF
OPERATION IS STRAIGHTFORWARD:
•
READ GLOBAL TRIGGER REQUESTS: EACH REQUEST
IS NUMBERED WITH A UNIQUE ID AND CONTAINS A
TIME WINDOW OF INTEREST TO READOUT
•
EXTRACT INDIVIDUAL REQUESTS (EB COULD BE
DIRECTED TO READOUT JUST ONE HUB OR MAYBE
JUST ICETOP OR MAYBE THE ENTIRE DETECTOR, FOR
EXAMPLE), CREATE READOUT REQUESTS, AND
DIRECT THEM TO THE APPROPRIATE HUB
•
THE SENDER COMPONENT IN THE HUBS READS THE
REQUESTS AND RETURNS A PACKET, POSSIBLY
EMPTY, CONTAINING ALL HITS IN THIS TIME
WINDOW. THE SENDER USES THE TIME MARKERS
TO DELETE BUFFER DATA WHICH IS DETERMINED
NO-LONGER RELEVANT.
•
THE EVENT BUILDER MUST ASSEMBLE THE
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
9
IceCube DAQ
READOUT DATA PAYLOADS RETURNED INTO AN
EVENT. THEY ARE COLLATED BY READOUT ID
WHICH SERVES AS AN EVENT ID PROXY. WHEN
THAT ID CHANGES, AN EVENT IS “TIED OFF” AND
SEND OUT TO THE FILE DISPATCHER.
• THE FILE DISPATCHER JUST WRITES THE DATA
WHICH IS MOSTLY CONTAINED IN BYTE BUFFERS TO
DISK. WITH ADVENT OF V5 AND LATER EVENT
FORMATS THE BYTE BUFFER DUMP GOES THROUGH
SOME PROCESSING TO REDUCE THE SIZE OF THE
EVENT REPRESENTATION ON DISK.
• THE DISK FILES, NOMINALLY ON
/MNT/DATA/PDAQLOCAL
ON THE MACHINE HOSTING THE EVENT BUILDER,
ARE CHUNKED INTO 10 MB FILES. THE EVENT
BUILDER IS NOW DONE WITH THIS DATA AND IT
MUST BE PICKED UP BY P&F.
TITLE
EventBuilder
DAQ COMPONENTS ABSTRACTED BY DAQComponent ABSTRACT CLASS AND
ALL COMPONENTS MUST SUBCLASS THIS. IT CONTAINS CALLBACK STUBS FOR
MANAGING THE LIFE-CYCLE OF COMPONENTS (SEE PICTURE AT LEFT):
STARTING, STOPPING, RESETTING (PARTIAL SUPPORT), CONFIGURING AND
AUTO-DISCOVERY OF COMPONENTS.
ADDITIONALLY THE SUPER CLASS PROVIDES MUCH OF THE SUPPORTING
MONITORING FRAMEWORK TO INSTRUMENT COMMON ACTIVITIES SUCH AS
THE INPUT & OUTPUT ENGINES AND SPLICERS. MORE ON THIS IN
MONITORING.
THE SYSTEM USES XML-RPC AS RPC PROTOCOL. MOST IF NOT ALL OF THE GLUE
IS WRITTEN IN PYTHON WHERE XML-RPC IS A “BATTERIES-INCLUDED” FEATURE.
ON THE JAVA SIDE WE USE THE APACHE XML-RPC 3.X ENGINE.
THIS DIAGRAM IS NOW OBSOLETE – IN THE HOBEES RELEASE OF PDAQ WE GOT
RID OF THE INTERMEDIARY DAQRun AND NOW ICECUBE LIVE AND CnCServer
COMMUNICATE DIRECTLY (DAVE WHIMSICALLY DESCRIBED THE OLD
SITUATION AS DRIVING A CAR BY TURNING A STEERING WHEEL HOOKED TO
ANOTHER STEERING WHEEL HOOKED TO ANOTHER STEERING WHEEL
CONNECTED TO THE STEERING COLUMN …)
THE CnCServer TYPICALLY SITS ON THE EXPERIMENT CONTROL MACHINE BUT IT
IS POSSIBLE TO RUN MULTIPLE INSTANCES OF THE DAQ IN PARALLEL AND THIS
IS ROUTINELY DONE. HANDY FOR MAKING TEST RUNS / CALIBRATIONS /
FLASHERS / &C.
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
10
IceCube DAQ
TITLE
Component Architecture and
the Control Framework
A GENERIC MONITORING BEAN OUTPUT FROM PAYLOAD OUTPUT
ENGINE OPERATING IN THE TRIGGER:
MBEAN MONITORING
• DAQ COMPONENTS ARE INSTRUMENTED WITH MONITORING POINTS SO
THAT EXTERNAL AGENTS CAN PEEK INSIDE TO EXAMINE POTENTIALLY
INTERESTING QUANTITIES SUCH AS STREAM FLOW RATES OR MEMORY
ALLOCATION STATISTICS.
trigger: 2010-11-16 20:51:26:
Depth: [0]
RecordsSent: [6046]
THE DATACOLLECTOR MBEAN: ONE OF THESE FROM EACH CHANNEL IN
THE ARRAY – LOTS OF DATA HERE.
DataCollectorMonitor-00A: 2010-11-16 21:11:28
NumHits: 835245
RunState: RUNNING
NumTcal: 1319
MainboardId: f5d596cfb67c
LBMOverflowCount: 0
AcquisitionLoopCount: 58071
HitRate: 626.6
NumMoni: 1767
NumSupernova: 1278
• WE ARE TALKING JAVA COMPONENTS HERE; WE USE THE JMX
MANAGEMENT BEANS FULLY SUPPORTED FROM JAVA 1.6. THESE ARE RUNTIME DISCOVERABLE VIA SOME JAVA INTROSPECTION MECHANISM: PRETTY
EASY TO ADD NEW MONITOR POINTS. AS MENTIONED, THE
DAQComponent CLASS POSSESSES SOME BOILERPLATE CODE TO MONITOR
AUTOMATICALLY. MOST COMPONENTS ADDITIONALLY EXTEND THIS TO
PROVIDE COMPONENT-SPECIFIC MONITORING.
• THE BEANS ARE QUERIED ULTIMATELY BY A PYTHON DAEMON AND SO
DAVE HAD TO WRITE A THIN RMI->XML-RPC BRIDGE TO EFFECT THIS
CROSSOVER.
• THIS IS A SYNCHRONOUS PROCESS: EVERY 100 SEC WE GET A SNAPSHOT OF
ALL BEANS
DAQ LOGS
ANOTHER TRIGGER BEAN WHICH LOOKS AT THE NUMBER OF TRIGGERS
REPORTED – FROM TIME TO TIME PEOPLE ASK FOR REAL-TIME FEEDS ON
THIS INFORMATION – WOULD BE NICE TO PIPE TO ICECUBE LIVE AND
DISPLACE ON A ROLLING STRIP CHART
manager: 2010-11-16 20:54:47:
TriggerCounts: {
'MinBiasTrigger1': 91, 'SimpleMajorityTrigger3': 8853,
'CalibrationTrigger1': 8808}
• USE LOG4J FRAMEWORK TO MONITOR ASYNCHRONOUS
INFO/WARN/ERROR MESSAGES EMITTED BY THE COMPONENTS.
• SOMEHOW THE CONSOLE MESSAGES TO System.out ARE TRAPPED AND REDIRECTED TO THE LOG FILES TOO – HANDY AS THERE IS NO CONSOLE FOR
REMOTE COMPONENTS
ASCII-FORMATTED LOGS AND MONITOR MESSAGES SENT TO RUN DIRECTORY
UNDER /MNT/DATA/PDAQ/LOGS/RUNXXXXXX
WAS ESSENTIAL DURING HEAVY DEVELOPMENT – STILL NICE TO HAVE AROUND
BUT WE PROBABLY DO EAT UP 100 MB/DAY OVER SCP JUST IN LOG/MONI
TRAFFIC. THERE HAVE BEEN COMPLAINTS …
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
11
IceCube DAQ
TITLE
Monitoring System and
Logging
THE RUN CONFIGURATION SYSTEM (CONTRAST WITH CLUSTER
CONFIGURATION WHICH ESTABLISHES THE SETUP OF THE DAQ PROCESSES
ACROSS A CLUSTER OF DATA TAKING COMPUTERS) SEEKS TO
CONFIGURATION SYSTEM IS A BIT OF A MONSTER EVEN WHEN WE HAVE BEEN
PAYING ATTENTION TO IT, BUT HERE’S HOW IT WORKS:
1.
THE USER VIA ICECUBE LIVE STARTS A RUN WITH A CERTAIN
CONFIGURATION TAG – JUST AN ASCII STRING THAT TYPICALLY DESCRIBES
AN IMPORTANT FEATURE OF THE CONFIGURATION.
2.
“.XML” IS APPENDED TO THIS TAG AND A FILE BY THAT NAME IS SOUGHT
IN A WELL-DEFINED LOCATION. THIS IS CALLED THE “TOP-LEVEL”
CONFIGURATION.
3.
THE TOP LEVEL REFERENCES A TRIGGER CONFIGURATION XML AND ONE
OR MORE DOM CONFIGURATION XMLS THE LATTER WHICH HOLD MOST
OF THE CONFIGURABLE ITEMS. THESE ARE ALSO SOUGHT IN A
FILESYSTEM AT A WELL-DEFINED LOCATION
4.
EACH COMPONENT IS PASSED THIS TAG AND EACH COMPONENT CAN
SCAN ITS FILESYSTEM FOR WHATEVER INFORMATION IT NEEDS TO
CONFIGURE ITSELF OR ITS DEPENDENTS (LIKE THE DOMS IN THE CASE OF
THE STRINGHUB).
5.
THE COMPLETE SET OF ACTIVE XML CONFIGURATION FILES IS PUSHED
OUT ALONG WITH DEPLOYMENT OF THE JAVA BINARIES TO EACH
COMPONENT.
• CAPTURE THE DETECTOR HARDWARE CONFIGURATION AND DISTRIBUTE IT
TO THE 5000 CHANNELS.
• THERE IS ALSO THE CONFIGURATION OF THE TRIGGER SYSTEM.
• THE BUILDERS RUN PRETTY MUCH WITHOUT NEEDING CONFIGURATION.
• THERE ARE A SMALL NUMBER OF PER-RUN CONFIGURATION ITEMS
OPTIONALLY CONFIGURED FOR THE STRING HUBS.
• FINALLY THE RUN CONFIGURATION EXPLICITLY SPELLS OUT WHICH
TRIGGER AND BUILDER COMPONENTS ARE PARTICIPATING IN A RUN
• THE HUBS THEMSELVES ARE AUTO-MAGICALLY DETERMINED BASED ON
THE LIST OF CHANNELS CALLED OUT FOR A PARTICULAR RUN.
NEEDLESS TO SAY, THE CONFIGURATION IS A COMPLEX BEAST: IN A TYPICAL
79-STRING RUN WE CONFIGURE OVER 207K ADJUSTABLE PARAMETERS!
ONE OF THE DESIGN GOALS WAS TO BE ABLE TO GET THE DAQ STARTED WITH
MINIMAL CONFIGURATION (OK, POINT TAKEN, MINIMAL ASIDE FROM THE
OTHER 200,000 CONFIGURATION ITEMS) NECESSARY. I ADMIT THAT THE
DAVE WILL COVER THE CLUSTER CONFIGURATION SYSTEM IN HIS
DISCUSSION OF THE DAQ DEPLOYMENT ARCHITECTURE.
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
12
IceCube DAQ
TITLE
The DAQ Run
Configuration System
THE PAYLOAD SYSTEM
ONE OF THE SPECIFIC POINTS REQUESTED IN THE CHARGE WAS THE SUITABILITY OF THE
PAYLOAD SYSTEM IN THE FUTURE. WITHOUT GETTING INTO A FULL-BLOWN CODE REVIEW
(WHICH WE SHOULD REALLY UNDERTAKE) HERE ARE SOME THOUGHTS ON THE PAYLOAD
SYSTEM:
• DAQ HOLDS MOST OF IT INFORMATION IN JAVA BYTE BUFFERS WHICH GET TRANSMITTED
OVER THE NETWORK OR HELD IN TEMPORARY PENS AWAITING THEIR FATE.
• THE JAVA BYTE BUFFER IS A PLACE TO HOLD, UH, WELL BYTES
• IT DOESN’T ANY STRUCTURE ON THE BYTES IN THE BUFFER AS IT SHOULD NOT – IT’S A
COMPLETELY GENERIC CONTAINER.
• ALL DAQ QUANTITIES HAVE SOME COMMON FEATURES, HOWEVER, LIKE MESSAGE TYPE,
LENGTH, USUALLY A TIMESTAMP (THE 64-BIT # OF 0.1 NS TICKS SINCE JANUARY OF ANY
YEAR), MAYBE A CHANNEL ID.
• SO THE PAYLOAD SYSTEM WAS INVENTED TO DEAL WITH THIS. THINGS LIKE I/O ENGINES
COULD THEN HAVE SOME MORE STRUCTURE WHEN DEALING WITH THESE ENTITIES.
• THE ISSUE WITH THE PAYLOAD SYSTEM IS THAT IT’S IMO A BIT DIFFICULT TO EXTEND – IT’S
NOT THE PAYLOADS THEMSELVES BUT THE CODE THAT DEALS WITH THE PAYLOADS. IT’S
ALSO BEEN JOINED AT THE HIP WITH THE BYTE BUFFER CACHING SYSTEM WHICH IS NOW
DEAD SO IT SHOULD BE EXPUNGED FROM THE PAYLOADS – THAT ALONE WOULD PROBABLY
CLEAR UP SOME COMPLEXITY.
DATA FORMATS
DESCRIBED IN THE WIKI ARTICLE
“PAYLOADS”
Events / sec
Event MB/sec
Readouts
II Hits to
TRIG
II to GT
IT Hits to TRIG
IT to GT
2350
8.1
120 kB /s / hub
28 kB/sec
2394 Hz
9.6 kB/sec
59 Hz
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
13
IceCube DAQ
TITLE
Data Rates, Payloads, &
Formats
• Dave Glowacki – mostly DAQ
– User documentation
• Kael Hanson – 0.2 DAQ
Chef du CCB was KSB < 9/10
Event Builder
– StringHub front half
String Hub Sender component
– Splicer
Trigger structure
– DAQ mid-level data inspection
and debugging
Payloads
– Some configs
Python framework – CnCServer …
Everything else not covered
• Mark Krasberg - ?
– Configuration XMLs for the DOMs
• John Jacobsen
– Now mostly IceCube live
• Also
– Some DOM level support and
– Alberto Marotta – SMT trigger
DOR driver
hang rewrite
– Did DAQ logging work and
– Thorsten Gluesenkamp – Slow
Python framework
MP trigger
–
–
–
–
–
–
–
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
14
IceCube DAQ
TITLE
DAQ Manpower
• Well, this is a problem
• Alex made observation that poor documentation is
an indicator of understaffing
• There is a good user-level document that WOs say is
helpful: it is written and maintained by JEJ
• We don’t have much developer documentation.
• Configuration system – which is a complex system
and would be a good place to begin for us as much
as anyone – is not really documented. I often have
to poke into the code to refresh my memory.
• We’ve been occupied with (A) making it work; (B)
making it work better to now.
• The DAQ would make a good NIM paper.
DRAWN BY
DATE
KH DG JJ
11/8/10
SHEET
SUBJECT
15
IceCube DAQ
TITLE
Documentation