The DØ Control System
J. Frederick Bartlett
For
The DØ Controls Group
Outline








Fermilab and the D0 Experiment
The DØ Control/Monitoring System
IOC extensions
Process Variable Naming Conventions
Database Management
Significant Event System
Detector Configuration Management
Operator Display Support Library
The Fermilab Collider Complex
CDF
DØ
DØ Detector
Forward Muon
Layer C
Liquid
Argon/Uranium
Calorimeter
Central Muon
Layer A
Central Muon
Layer B
Superconducting
Solenoid
Scintillating
Fibers
Silicon Tracker
Central Muon
Layer C
Muon Toroid
DØ Run II Control System
 EPICS-based controls
 Field buses



VME
MIL/STD1553B
CANBUS (Run IIB)
 Size


~15 host-level processors
~100 IOCs (Input/Output Controllers)



MVME2301, MVME2304, MVME5500
~6600 high-level devices
~138000 process variables (records)
DØ Run II Control System




15 major detector sub-systems
Host-Level processes written in Python
Source management - CVS
Operating systems
 Host processors - Linux
 IOC processors – vxWorks
 Software versions
 Now - EPICS 3.14.6
Next - EPICS 3.14.7 with patches, using SNS linux build tools
 Controls staff
 Core system - 3 FTEs (4 people)
 Detector-specific components - ~2 FTEs


Primarily from other institutions
Devices and Records
Detector
Devices
Records
Argon/Uranium Calorimeter
844
9219
Scintillating Fiber Tracker
445
35900
Muon Spectrometer
1572
20027
Silicon Microstrip Tracker
2235
45706
Controls
440
3775
Trigger Systems
694
21644
Other Systems
468
2693
Total
6653
138964
EPICS IOC Extensions
Ethernet LAN
Paused
Channel
Access
End Ramp
DB Load Holding
Mid Ramp
Scanners
Database
Access
Locked
Begin Ramp
Average
IOC Database
Tripped
On
HV Generator
State Machine
Record
Support
Offline
VME Bus Access
With Retries
Device
Support
MIL/STD1553
Bus and
CANBUS
Driver
Off
Start
VME + MIL/STD1553
Disabled
Customizing
Elements
Gateways to Other Systems
 Accelerator
 Gateway link to ACNET system
 Bidirectional
 Data access only (no control)
 Cryogenics and Gas
 Gateway link to DMACS system
 Read-only
 Data access only (no control)
ACNET Gateway
CLIENTS
CLIENTS
CLIENTS
xmlrpc Server
xmlrpc
ITC
ToFromITC
CA
ToEPICS
EPICS
RECORDS
CA
D0
FromEPICS
ToACNET
C
A
C
H
E
xmlrpc
FromACNET xmlrpc
R
E
M
O
T
E
E
X
P
O
R
T
BD
DØ Naming Convention
 Name elements
 Detector
 Sub-det
 Device
 Locator
 Attribute
 I/O
 Field
 Template
<det>
<sub>
<dev>
<loc>
<attr>
<io>
<field>
CAL
N
VBD
01
STATUS
W
SCAN
Frequently used
in filters
<det>[<sub>]_<dev>_<loc>/<attr>[:<io>] [.
<field>]
CALN_VBD_01/STATUS:W.SCAN
Many utility tools use this
field for device-specific
actions
Database Management
 Basic EPICS uses flat text files
 Template definition file


Template generator file


Multiple record definitions with substitution parameters
Create a group of records (instance of a template) with
values for parameters
Record instance file

Instances of records read by IOC to create record
database
Relational Database System
 EPICS database creation from text files
 Record structure defined by DBD files
 Template structure defined by template definition
files
 Template instances defined by template generator
files or from Browser GUI
 IOC record instances (record instance file) are
extracted from database
 Database support programs written in Python
Relational Database System
Record
Field
Definitions
Template
Definitions
Template
Generators
Instance
Creation
DB
Creation
Oracle
Hardware
Database
Record
Extract
List of
Records
For IOC
Template
Extract
IOC Id
Template
Generators
For IOC
Significant Event System
 The significant event philosophy
 Alarms are a only a sub-set of the significant
events
 EPICS does not generate all of the significant
events of interest
 Alarm utilities enhance reliability



Detect impending failures and fix them before they fail
Minimize the time to correct failures
Archive all event transitions



The archive is a history of the state transitions of the
experiment
Tools available to search the event archive
What was your system doing last month?
Alarm/Event Distribution
 SES is a server-based, event (alarm) system:
 IOCs and user processes connect to and send
alarm transitions to the server



Server holds the current experiment (alarm) state
Server has one or more filters for each receiving
client




Pushed by sources not pulled by the server or client
Makes use of name structure
Rapid display startup of receiving clients
User processes may also declare events via API
(C, C++, Python)
Written in Python
Significant Event System
F
Filter
Filter
SE Message
Filtered Message
EPICS
Process
IOC
Process
Watcher
EPICS IOC
Alarm
Display
Periodic
Heartbeat
Significant Event
Server
F
F
F
F
seLogger
Run Control
(COOR)
seBrowser
Alarm
Watcher
Run
Suspend
SES Alarm Display
Select Guidance
button
Events in line
selected by filter
Select minor
alarm button
Detector Configuration Management
 EPICS Save-and-Restore is adequate for accelerators
 Accelerators are tuned for an operation mode and
the configuration is saved for future restoration
 Some complex devices like HEP detectors do not
work this way
 There are too many distinct modes to be saved
 Tuning may not be required


Settings are often predictable from trigger selection
Many detector component settings are taken from
separate calibration measurements
Comics Server
 Manages the configuration of the detector
 Receives load requests from:
 Run control
 Expert GUIs
 Program APIs
 Load map is a directed graph (tree)
 Tree node (intermediate)



Establishes a layered hierarchy
Establishes an execution order
Action node (leaf)

Issues EPICS CA requests
Comics Server
 Constructed on the server model with multiple clients
sending commands
 Server may be activated independently for
configuring detector component
 Coded in the Python language
Configuration Management Tree
Root Node
CAL
Intermediate
Tree Node
DØ
SMT
MUO
Links
determine the
order of
execution
CFT
Configuration
Data
Action Node
Uses EPICS
Channel
Access
S0
S1
DEV0
DEV1
S2
T0
DEV2
DEV3
Comics Expert GUI
Operator Display Library
 Standard, process flow (synoptic) displays do not
adapt well to the monitoring of HEP detectors
 Like accelerators, there are many similar devices
that make up the typical detector component (cells
in a calorimeter)
 The components are not related in a serial fashion
like a cryogenic plant
 Tabular (spread-sheet designs) are more natural
 Similar properties for different devices are easily
compared
 Deviations are apparent
Operator Display Library
 DØ has developed a series of Python display classes
for building tabular displays that collect information
from EPICS process variables
Resource Display
Basic Frame
with Menu Bar,
Status Window,
and Button Bar
Tabbed Pages
with CA
Connect and
Disconnect
Device
Window
Right-Click on
Name Field
Device Analog
Property
Device Status
(Bit-Encoded)
Property