The DØ Control System J. Frederick Bartlett For The DØ Controls Group Outline Fermilab and the D0 Experiment The DØ Control/Monitoring System IOC extensions Process Variable Naming Conventions Database Management Significant Event System Detector Configuration Management Operator Display Support Library The Fermilab Collider Complex CDF DØ DØ Detector Forward Muon Layer C Liquid Argon/Uranium Calorimeter Central Muon Layer A Central Muon Layer B Superconducting Solenoid Scintillating Fibers Silicon Tracker Central Muon Layer C Muon Toroid DØ Run II Control System EPICS-based controls Field buses VME MIL/STD1553B CANBUS (Run IIB) Size ~15 host-level processors ~100 IOCs (Input/Output Controllers) MVME2301, MVME2304, MVME5500 ~6600 high-level devices ~138000 process variables (records) DØ Run II Control System 15 major detector sub-systems Host-Level processes written in Python Source management - CVS Operating systems Host processors - Linux IOC processors – vxWorks Software versions Now - EPICS 3.14.6 Next - EPICS 3.14.7 with patches, using SNS linux build tools Controls staff Core system - 3 FTEs (4 people) Detector-specific components - ~2 FTEs Primarily from other institutions Devices and Records Detector Devices Records Argon/Uranium Calorimeter 844 9219 Scintillating Fiber Tracker 445 35900 Muon Spectrometer 1572 20027 Silicon Microstrip Tracker 2235 45706 Controls 440 3775 Trigger Systems 694 21644 Other Systems 468 2693 Total 6653 138964 EPICS IOC Extensions Ethernet LAN Paused Channel Access End Ramp DB Load Holding Mid Ramp Scanners Database Access Locked Begin Ramp Average IOC Database Tripped On HV Generator State Machine Record Support Offline VME Bus Access With Retries Device Support MIL/STD1553 Bus and CANBUS Driver Off Start VME + MIL/STD1553 Disabled Customizing Elements Gateways to Other Systems Accelerator Gateway link to ACNET system Bidirectional Data access only (no control) Cryogenics and Gas Gateway link to DMACS system Read-only Data access only (no control) ACNET Gateway CLIENTS CLIENTS CLIENTS xmlrpc Server xmlrpc ITC ToFromITC CA ToEPICS EPICS RECORDS CA D0 FromEPICS ToACNET C A C H E xmlrpc FromACNET xmlrpc R E M O T E E X P O R T BD DØ Naming Convention Name elements Detector Sub-det Device Locator Attribute I/O Field Template <det> <sub> <dev> <loc> <attr> <io> <field> CAL N VBD 01 STATUS W SCAN Frequently used in filters <det>[<sub>]_<dev>_<loc>/<attr>[:<io>] [. <field>] CALN_VBD_01/STATUS:W.SCAN Many utility tools use this field for device-specific actions Database Management Basic EPICS uses flat text files Template definition file Template generator file Multiple record definitions with substitution parameters Create a group of records (instance of a template) with values for parameters Record instance file Instances of records read by IOC to create record database Relational Database System EPICS database creation from text files Record structure defined by DBD files Template structure defined by template definition files Template instances defined by template generator files or from Browser GUI IOC record instances (record instance file) are extracted from database Database support programs written in Python Relational Database System Record Field Definitions Template Definitions Template Generators Instance Creation DB Creation Oracle Hardware Database Record Extract List of Records For IOC Template Extract IOC Id Template Generators For IOC Significant Event System The significant event philosophy Alarms are a only a sub-set of the significant events EPICS does not generate all of the significant events of interest Alarm utilities enhance reliability Detect impending failures and fix them before they fail Minimize the time to correct failures Archive all event transitions The archive is a history of the state transitions of the experiment Tools available to search the event archive What was your system doing last month? Alarm/Event Distribution SES is a server-based, event (alarm) system: IOCs and user processes connect to and send alarm transitions to the server Server holds the current experiment (alarm) state Server has one or more filters for each receiving client Pushed by sources not pulled by the server or client Makes use of name structure Rapid display startup of receiving clients User processes may also declare events via API (C, C++, Python) Written in Python Significant Event System F Filter Filter SE Message Filtered Message EPICS Process IOC Process Watcher EPICS IOC Alarm Display Periodic Heartbeat Significant Event Server F F F F seLogger Run Control (COOR) seBrowser Alarm Watcher Run Suspend SES Alarm Display Select Guidance button Events in line selected by filter Select minor alarm button Detector Configuration Management EPICS Save-and-Restore is adequate for accelerators Accelerators are tuned for an operation mode and the configuration is saved for future restoration Some complex devices like HEP detectors do not work this way There are too many distinct modes to be saved Tuning may not be required Settings are often predictable from trigger selection Many detector component settings are taken from separate calibration measurements Comics Server Manages the configuration of the detector Receives load requests from: Run control Expert GUIs Program APIs Load map is a directed graph (tree) Tree node (intermediate) Establishes a layered hierarchy Establishes an execution order Action node (leaf) Issues EPICS CA requests Comics Server Constructed on the server model with multiple clients sending commands Server may be activated independently for configuring detector component Coded in the Python language Configuration Management Tree Root Node CAL Intermediate Tree Node DØ SMT MUO Links determine the order of execution CFT Configuration Data Action Node Uses EPICS Channel Access S0 S1 DEV0 DEV1 S2 T0 DEV2 DEV3 Comics Expert GUI Operator Display Library Standard, process flow (synoptic) displays do not adapt well to the monitoring of HEP detectors Like accelerators, there are many similar devices that make up the typical detector component (cells in a calorimeter) The components are not related in a serial fashion like a cryogenic plant Tabular (spread-sheet designs) are more natural Similar properties for different devices are easily compared Deviations are apparent Operator Display Library DØ has developed a series of Python display classes for building tabular displays that collect information from EPICS process variables Resource Display Basic Frame with Menu Bar, Status Window, and Button Bar Tabbed Pages with CA Connect and Disconnect Device Window Right-Click on Name Field Device Analog Property Device Status (Bit-Encoded) Property