LHCb’s
Experiment Control System
Clara Gaspar, March 2003
Experiment Control
In charge of the Control and Monitoring of:
Detector Operations (ex Slow Controls)
GAS, HV, LV, temperatures...
Data Acquisition and Trigger
FE Electronics, Event building, EFF, etc.
Experimental Infrastructure
Cooling, ventilation, electricity distribution, ...
Interaction with the outside world
Magnet, accelerator system, safety system, etc.
Clara Gaspar, March 2003
2
ECS Scope
Experiment Control System
DCS Devices (HV, LV, GAS, Temperatures, etc.)
Detector Channels
Trigger
Front End Electronics
Readout Network
Processing/Filtering
Storage
DAQ
External Systems (LHC, Technical Services, Safety, etc)
Clara Gaspar, March 2003
3
ECS Context Diagram
Operator
Configuration
Database
Display
Information
Configuration data
Accelerator
Status
Information
Status
Information
Infrastructure
Systems
Commands/
Settings
Experiment
Control
System
Commands/
settings
Safety alarms
& Settings
Event
related
control’s
data
Status
Detector HW
DAQ
DCS Trigger
Clara Gaspar, March 2003
Safety
System
Conditions
Database
Offline
Analysis
4
ECS Requirements
Integrate the different activities
Such that rules can be defined (ex: Stop DAQ when DCS in Error)
Allow stand-alone control of sub-systems
For independent development and concurrent usage.
Automation
Avoids human mistakes and speeds up standard procedures
Easy to operate
Two to three operators (non-experts) should be able to run the
experiment.
Scalable & Flexible
Allow for the integration of new detectors
Maintainable
Experiments run for many years
Clara Gaspar, March 2003
5
Experiment Control
Keywords: Homogeneity & Uniformity
A common approach in the design and
common tools and components for the
implementation of all parts of the system:
Facilitates inter-domain integration
Makes it easier to use:
• Standard features throughout the system
(ex: partitioning rules)
• Uniform Look and Feel
Needs less manpower
Allows an easier upgrade and maintenance
Clara Gaspar, March 2003
6
Design Goals and Principles
Influenced by experience taken from LEP experiments
Integrated approach from design phase
An Architecture
That can handle all aspects of the monitoring and
control of the full experiment
A Framework
An integrated collection of guidelines, tools and
components allowing the development of each subsystem coherently in view of its integration in the
complete system
Clara Gaspar, March 2003
7
Design Goals and Principles
Distributed Hierarchical Control
De-composition in Systems, sub-systems, … , Devices
Local decision capabilities in sub-systems
Scalability
Simplicity
Simple functionalities and protocols
Separation of controls and data path
Reliability
Main-stream technologies where possible
PC based interfaces: PCI cards/OPC (no VME bus)
Less effort in implementation and maintenance
Clara Gaspar, March 2003
8
Generic SW Architecture
LHC
GAS
...
DetDcs1
SubSys
1
DAQ
DCS
DetDcs
N
SubSys
2
Dev
1
Dev
2
DetDaq
1
SubSysN
Dev
3
DSS
...
Commands
T.S
.
Status & Alarms
ECS
Dev
N
To Devices (HW or SW)
Clara Gaspar, March 2003
9
Hierarchical control
Each node is able to:
Summarize information
(for the above levels)
“Expand” actions
(to the lower levels)
Implement specific behaviour
& Take local decisions
DCS
Tracke
r
H
V
Tem
p
Muon
H
V
GA
S
Sequence & Automate operations
Recover errors
User Interfacing
Present information and receive commands
Clara Gaspar, March 2003
10
Run Control
Each node (and its sub-tree)
Can run in stand-alone (i.e. can be partitioned)
Can be controlled independently (by an authorized User
Interface)
Run Control
Is the interface to the
Root node
 If the tree is partitioned
there can be several
Run Controls.

Clara Gaspar, March 2003
11
Generic SW Architecture
LHC
GAS
...
DetDcs1
SubSys
1
DAQ
DCS
DetDcs
N
SubSys
2
Dev
1
Dev
2
DetDaq
1
SubSysN
Dev
3
DSS
...
Commands
T.S
.
Status & Alarms
ECS
Dev
N
To Devices (HW or SW)
Clara Gaspar, March 2003
12
DCS Devices
High Voltages, Low Voltages, etc.
Try to use Commercial HW (CAEN,WIENER,ISEG,etc)
Interface: OPC Servers
(provided by manufacturer)
Analog & Digital IO
(temperatures, etc.)
ELMBs / OPC Servers
Commercial (e.g. WAGO)
PLC
FieldBus
Node
Others:
Node
FieldBus
Node
Experimental equipment
PLCs (Siemens, Schneider) / OPC Servers
Home made / DIM or OPC Servers
Clara Gaspar, March 2003
13
Front-End Electronics Devices
In Radiation Areas
Mainly needed I2C and JTAG
SPECS (evolved from ATLAS SPAC)
• Serial Protocol
• 10Mb/s
• Slave is radiation tolerant
S
I2C
JTAG
S
I2C
JTAG
S
I2C
JTAG
ELMB
• CAN protocol (0.5Mb/s)
SPECS
CAN
M
Clara Gaspar, March 2003
(~1000+)
14
Electronics Boards Devices
Electronics in barracks
Front-ends, Readout Units, Timing
and Fast Control components, etc.
Credit Card PC’s
66  85  12 mm3
Pentium Compatible CPU
Linux/DIM (no PVSS)
S
I2C
JTAG
Bus
S
I2C
JTAG
Bus
S
I2C
JTAG
Bus
Ethernet
(~500+)
Clara Gaspar, March 2003
15
Control of Electronics Boards
Read/write registers
Load FPGAs
Clara Gaspar, March 2003
16
Event Filter Farm Devices
CPU Infrastructure
Reboot/Download
Monitoring
SFC
• CPU usage, memory,
IO, etc.
Trigger Algorithms
Configure
Monitor
Start/Stop
Event Builder Switch
SFC
SFC
CPU
CPU
CPU
CPU
.
.
.
.
.
.
CPU
CPU
Clara Gaspar, March 2003
CPU
...
CPU
.
.
.
CPU
17
Control System Monitoring
The Monitoring of the Control System
itself is very similar to the Monitoring of
the CPUs in the Event Filter Farm:
Controls PCs, Credit Card PC’s, etc:
Remote Boot & SW download
CPU Monitoring
• CPU usage, memory, IO, etc.
Processes (control processes: PVSS, OPC, etc.)
• Configure, Monitor, Start/Stop
Clara Gaspar, March 2003
18
CPU Monitoring prototype
Try to use IT developments
Clara Gaspar, March 2003
19
EFF Algorithm Control
Functionality needed:
(Online environment: no disk/no files, low CPU usage)
Configure algorithm sequences and algorithm
parameters
Start/Stop processing
Monitor counters, errors, histograms, etc.
Add histograms from several nodes
 Needed to interface offline software
framework ( GAUDI ) to ECS
 Interfaced PVSS to ROOT
Clara Gaspar, March 2003
20
Online Gaudi
Clara Gaspar, March 2003
21
Proof of Concept
TestBeam Setup:
Run Control based on JCOP framework
Interfacing to (old) Cascade DAQ
Integrated PS/SPS information
Integrated CPU/process monitoring
Started integrating DCS devices:
Platform Motors
Environment monitoring (ELMB)
High Voltages
Clara Gaspar, March 2003
22
Test Beam Run Control
Clara Gaspar, March 2003
23
LHCb view on JCOP
The best way to achieve a homogeneous and
maintainable control system (and to save
manpower) is to do the maximum in common:
 Promote standardization (for ex. HW) so that
common components can be (re)used.
Devise a generic architecture which fits all four
experiments and “isolate” its building bocks.
Identify commonly needed building blocks and develop
them in common once.
Provide guidelines so that other components can be
developed in a re-usable way and distributed to the
community.
Clara Gaspar, March 2003
24
LHCb and JCOP
LHCb is heavily investing in JCOP
LHCb’s Control System relies on the success
of the following JCOP projects:
The Control Framework (in particular):
Hierarchical control tools addressing: partitioning, automation,
error recovery, etc.
Configuration tools using an external database, allowing for the
storage and retrieval of all HW and SW definitions and
configuration parameters for different running modes.
Interface to a conditions database for storing parameters
acquired by the control system
Modules for the exchange of information with other CERN
domains: LHC machine, Technical Services, Safety System, etc.
“Plug and play” modules for the monitoring and control of
commonly used equipment
Clara Gaspar, March 2003
25
LHCb and JCOP
Infrastructure Related Projects:
Power Distribution and Rack Control
Cooling and Ventilation Control
Gas Control
Detector Safety System
Magnet Control
Similar philosophy for All Services (hopefully):
Implementation, Support and Maintenance by the
provider (ST/EL & EP/ESS, ST/CV, Gas WG, JCOP, EP/TA3)
• Including closed loop control (PLC) programming, if any.
• Intervention picket
High level operation by Experiment (Sub Detectors)
• Start, Stop, Set parameters
Clara Gaspar, March 2003
26