October 2009
1
This pages are meant to initiate people who are not familiar with ECS (Experiment Control System), and Muon
ECS in particular, to perform shifts in global data taking run. In this sense, information given here are very basic and in general refer to a non-expert use of the system: setting-up of the system for DAQ runs, first check of proper running conditions, first simple interventions in case of simple frequently encountered problems.
This basic manual can also be a starting point for a richer documentation while the operator skill increases and the control system becomes more stable and complete.
Please note also that the ECS system is still under development. Changes on the panels and even on the structure is still possible. This pages should not be meant as a user manual of a delivered project/system, but just to gain a first idea on system operation.
The LHCb Experiment Control System (ECS) is based on the PVSS framework. You can find some documentation on PVSS and LHCb ECS at page http://lhcb-online.web.cern.ch/lhcb-online/ecs/default.htm
and in particular the tutorial http://lhcb-online.web.cern.ch/lhcb-online/ecs/powerpoint/LHCb_ECS_Tutorial.ppt
which we warmly suggest you to read.
The ECS is based on so-called PVSS projects running on dedicated online PCs. Each project runs on independent partitions. The MUON ECS has several project types, controlling:
• FEB (Front End Boards, i. e. CARDIAC via the Service Boards)
• ODE (Off Detector Electronics Boards)
• LV and HV
• Gas, Temperature, RAC power
• Tell1
• FSM (Finite State Machine & Hierarchy) projects
The last projects do not directly control any hardware but the hierarchical structure of the ECS itself. The FSM allows the ECS commands to circulate from top to bottom of the structure, thus making possible the communications among the complete system.
2
Front-end Boards (CARDIAC):
2 ASIC
-CARIOCA (ASD chip)
-DIALOG (logic, timing and monitoring)
DIALOG is configured and read back by SB (Service
Board).
Service Boards are placed in an off detector rack.
The same rack houses the
PDM board (Pulse
Distribution Module) : it generates pulses in a chosen phase with the LHC machine clock for the time alignment of the muon system.
The IB (Intermediate Board) make logical combinations of the front-end channels (logical channels) where the combination is not possible at DIALOG level
The ODE (Off Detector Electronic Board) receive signals from the FEBs (or IBs), synchronize and displace them to the L0 trigger. Upon reception of a trigger signal data are sent to the TELL1 board and from here to the DAQ (next slide for picture)
3
L0 Muon Trigger processors
Muon TELL1
DAQ
ODIN
The ODIN board distributes all orbit signals (clock, trigger, L0 reset,…) to the ODEs, Muon
TELL1s and Trigger processors (to all boards that are synchronuos and have the TTCrx chip)
4
C side A side
M1
M3
M4
M2
M5
Q4 Q1 z
Q3 Q2
Interaction point
5
The structure of ECS consists of 3 types of objects: Device Unit (DU),
Control Unit (CU) and Logical Unit (LU).
DU is an object controlling a hardware component (e.g. a CARDIAC board)
CU is just a hierarchical object, passing commands and states on a hierarchical node
LU is useful to group different DU together and is rarely used in our case
The hierarchical structure allows also including and excluding portions of the system.
Following the general LHCb structure, also the Muon hierarchy is organized in two independent and totally symmetrical parts, MUONA and MUONC, controlled by a higher node, MUON. The deeper the hierarchical level, the wider the understanding needed of system and hardware details. However, in “normal” conditions it should not be necessary to go further than the MUONA/C level.
EACH PVSS PROJECT RUNS ON AN ONLINE PC . On Control Room PCs you will find shortcuts to the online PCs where really the projects run
6
An example for the C side
MUON
MUONA MUONC
The Green bubbles are Control Unit (CU) of the:
•PVSS project MUSIDES
•PC: mudaqa01w
MUONC_DAQ MUONC_DCS
MUON_DCS_LV
MUONC_HV
Inside this CU there is the hierarchy of the
Muon detector High Voltage
•PVSS project MUHVC01 (PNPI - R3 R4)
•PC: muhvc01
•PVSS project MUHVC02 (CAEN – R1 R2)
•PC: muhvc02
Inside this CU there is the hierarchy of the Muon detector electronics:
Chambers/ODE/Tell1
•PVSS project MUDAQC
•PC: muecsc01w
Inside this CU there is the hierarchy of the Muon detector Low Voltage
•PVSS project MUDCSLVC
•PC: mulvc01w
7
An example for the C side
MUONC_DAQ
MUONC_DAQ_Q3 MUONC_DAQ_Q4 MUONC_DAQL1
MUONC_DAQ_Q3M45C MUONC_DAQ_Q3_M1C MUONC_DAQ_Q3_M23C
MUONC_DAQ_Q3_M23C_M2
MUONC_DAQ_Q3_M23C_M3
MUONC_DAQ_Q3_M23C_CRC
Inside this bubble there is the hierarchy of the DAQ tell1 boards witch take the data coming from the
ODEs:
•PVSS project MUDAQL1
•PC: mudaq01
Inside this bubble there is the
SB/PDM crate for station
M2&M3 Q3&Q4
•PVSS project MUDAQC23
•PC: mudaqc03w
8
An example for the C side
MUONC_DAQ_Q3_M23C
MUONC_DAQ_Q3_M23C_M2
Q3_M3_R1
Q3_M3_R1_HU1
MUONC_DAQ_Q3_M23C_M3
Q3_M3_R2
MUONC_DAQ_Q3_M23C_CRC
Q3_M3_R3 Q3_M3_R4
Hardware Unit
Q3_M3_R1_HU2
HU: Hardware Unit is made by One ODE and all the
Chambers connected to it
Chambers ----------------------
•PVSS project MUDAQC23
•PC: mudaqc03w
•ODE ---------------------------
•PVSS project MUODEC01
•PC: muodec01w
Chambers
9
Use
Feb 2&3/C
Feb 4&5/C
Feb 1/C
Feb 1/A
Feb 2&3/A
Feb 4&5/A
FSM/C
FSM/A
LV/C
LV/A
Comp name
MUDAQC03w
MUDAQC04w
MUDAQC01w
MUDAQA01w
MUDAQA03w
MUDAQA04w
MUECSC01W
MUECSA01W
MULVC01w
MULVA01w
PVSS Id
260
261
262
263
264
265
279
280
210
211
Project Name
MUDAQC23
MUDAQC45
MUDAQC1
MUDAQA1
MUDAQA23
MUDAQA45
MUDAQC
MUDAQA
MUDCSLVC
MUDCSLVA
HV/C
HV/C
HV/A
HV/A
MUHVC01
MUHVC02w
MUHVA01
MUHVA02w
212
213
214
215
MUHVC01
MUHVC02
MUHVA01
MUHVA02
ODEQ3/C
ODEQ4/C
ODEQ1/A
ODEQ2/A
T&P
MUODEC01w
MUODEC02w
MUODEA01w
MUODEA02w
MUDCS01w
266
267
268
269
270
MUODEC01
MUODEC02
MUODEA01
MUODEA02
MUDCSSEN
TELL1
M5 :
M4 :
M3 :
M2 :
M2:
M1:
M1: mudaq01 216 MUDAQL1 mutella01 (Q1/Q2) mutellc01 (Q4/Q3) mutella02 (Q1/Q2) mutellc02 (Q4/Q3) mutella03 (Q1/Q2) mutellc03 (Q4/Q3) mutella04 (Q1) mutellc04 (Q4) mutella05 (Q2) mutellc05 (Q3) mutella06 (Q1) mutellc06 (Q4) mutella07 (Q2) mutellc07 (Q4)
MUON TOP mudaqa01w MUSIDES
Here a complete list of the online PCs and the relative projects running on them is given
Comments:
1.
2.
3.
Access to this machines to start/stop/operate the PVSS projects should normally not be needed but for experts.
As a general rule, PC names ending with W are Windows-based PCS,
Linux otherwise.
Some of the reference experts are:
D. Pinci for FEBs,
S. Cadeddu for FSM,
M. Lenzi for LV and HV,
M. Carletti for ODE,
G. Graziani for Tell1 and Muon
Monitor,
C. Deplano.
10
HV
HV
A SIDE
HV
HV LV
LV
DCS
MUON
C SIDE
ODE
ODE
CMB
CMB
TELL1
CMB
CMB
ODE
ODE
11
5
ECS operation is based on control panels, giving information on system status and allowing intervention of the operator to trigger a status change. Although intuitive, it is important to give some explanations on the use of the panel buttons.
Sub-System buttons: clicking with the left mouse button opens the corresponding CU
{
Sub-System state, visualizing the current state. Clicking with the left mouse button gives the list of possible actions which can be done from it.
Lock button: the system can be accessed/controlled by different parts of
ECS. This button says who is presently
TAKING the system and its inclusion/exclusion in the hierarchy. This info can be gained by placing the mouse tip on the lock. Before operating on a given system/subsystem you need to TAKE it (if possible). During Global data taking “your” system must be controlled by the global manager and the muon operator must release it properly (see next pages).
Using the left mouse button it is possible to change the lock status.
Presently loaded recipe. A recipe is the collection of all the configuration parameters/variables needed for a specific data-taking run. Usually the shifter should not choose one recipe rather than another (it is already loaded).
If further information is needed on recipe handling, for the moment you have to refer to an expert.
MUONA CU Panel
NEVER use this button to dismiss the panel
ALWAYS USE the Close button
12
1. How to ...
In normal conditions, the PVSS projects (FEBs, ODE, FSM) are up and running. This condition should be reached automatically at each power-on of the machines where the projects are installed
(see list on next page). At the moment (June 08), this is implemented only for the LV control projects, but will be done soon also for the FEBs and the ODEs.
Being in this condition, the steps to set-up the system for DAQ are the following: i.
On any computer of the on-line cluster*, go to directory G:\online\ecs\Shortcuts38\MUON ii.
Launch (double click) the two shortcuts MUSIDES_UI_DEN and MUSIDES_UI_FSM . This visualizes the two control windows for DEN (Device Editor Navigator) and FSM (Finite State
Machine) Hierarchy (see next page).
Launching the shortcuts does not start the DEN and FSM processes (which are already running within the PVSS project), but just allows using the corresponding control windows from your screen.
(*you need an account within the on-line cluster: ask Niko Neufeld if you haven’t any.)
13
FSM panel
The FSM panel you should have opened allows visualizing the status of the full
MUON hierarchy and opening panels related to any hierarchical level.
Concerning the DU/CU status, the color code is as follows: grey for DEAD/NOT INCLUDED yellow for NOT READY orange for UNKNOWN/NOT ALLOCATED red for blue for green for
ERROR
READY or ACTIVE (TFC case)
RUNNING
Now, placing the cursor tip on the MUON box and clicking the RIGHT mouse button, opens the MUON CU, which represents the top level of our hierarchy, see next page(s).
This box accesses the TELL1 CU (refer to page 15)
14
In order to allow the global manager driving the MUON
System, the MUON must be
NOT_READY and its subsystems NOT_ALLOCATED.
All the locks should be grey and open, without any slash on them (which means Not
Included).
Although this condition appears to be verified in the shown panel, it is equally necessary to check the
MUONA and MUONC CU.
Checking deeper levels is not required to a shifter.
The first operation is TAKING the MUON System under your control. Click (LEFT mouse button) on the lock and select
TAKE. The lock goes green and closed.
A panel automatically pops up, giving the list of
LockedOut nodes and of CU which were not possible to take (more details on page
12).
Now OPEN (double click) the two sides (A & C)
15
(The MUONC panel has similar face)
If everything is correct, the panel appears in the configuration shown.
In this conditions, you can release the MUON system in order to make it possible the control by the global console (go to next page)
16
In order to release the MUON system, click on the green lock and choose ReleaseAll in the pop-up menu (the Release button releases this level of
CU but not those below it).
The MUON status goes to
NOT_READY and the system is free to be used by the global console.
Note:
Case of a subsystem taken by someone else.
Placing the mouse on, you obtain info on owner and status. In Global DAQ, the global manager decides what is to be included/excluded.
You have to take care only of
MUON, putting it in regular shape.
17
2. How to ...
Prepare the MUON system for global DAQ run when something is not the right way .
iv.
v.
vi.
i.
ii.
iii.
Suppose that opening the MUONA panel, a different status is shown
(see figure):
Some subsystems appear to be locked out and the TFC is
ACTIVE, while the right status is
NOT_ALLOCATED for the TFC and NOT_READY for the others
{
In such a case, these are the steps to follow:
Release MUON (ReleaseAll)
TAKE MUONA (where the problem was found)
ACTIVE Stop run
READY
READY RESET
NOT-READY
DEALLOCATE
ReleaseAll (in the lock)
This should take again the
MUONA in the right shape
18
When you
TAKE a
System, an Action
Report appears.
In some cases a subsystem cannot be taken because its state is DEAD
(i.e. The FSM cannot communicate with it, or the FSM is blocked there)
DEN panel
The DEN panel (you got from page 7) is to be used in case of DEAD states : click on the Start/Restart All button for the relevant stage of the hierarchy
(use the FSM tab). This activate the FSM only on the relevant stage and should cure the DEAD state.
19
When the the system is being reset or configured, it may happen that the system enter an ERROR state .
This state propagates from the DU/CU which originates it, up to the top level of the hierarchy (MUON in our case).
Frequently this is caused by the Tell1
CCard, needing to be restarted (see next page).
(see next slide also to know how to get to the
MUONA(C)_DAQL1 panel).
Here “ERROR” appears
20
Here is how to (possibly) cure a
TELL1 ERROR state
1.
Descend the hierachy starting from the MUON CU panel: MUON MUONA (e.g.) MUONA_DAQ
MUONA_DAQL1(see previous slide). Alternatively, you can access the Tell1 panel directly from the FSM
(see page 8).
2.
Spot the Tell1 which is in error and identify the corresponding CCPC (see subsystems in the panel and the given correspondance table).
3.
Log in on the CCPC using your user/pwd and start the ccserver (using putty, for example).
4.
Type the command “sudo service ccserv restart”. You should obtain a double ok. If not, try to repeat the command.
This restarts the ccserver and should solve the ERROR
Side A
PC TELL1 mutella01 MUTELLA_M5Q12 mutella02 MUTELLA_M4Q12 mutella03 MUTELLA_M3Q12 mutella04 MUTELLA_M2Q2 mutella05 MUTELLA_M2Q1 mutella06 MUTELLA_M1Q1
Mutella07 MUTELLA_M1Q2
Side C
PC TELL1 mutellc01 MUTELLC_M5Q34 mutellc02 MUTELLC_M4Q34 mutellc03 MUTELLC_M3Q34 mutellc04 MUTELLC_M2Q4 mutellc05 MUTELLC_M2Q3 mutellc06 MUTELLC_M1Q4 mutellc07 MUTELLC_M1Q3
21
Take before
And release all when finished
22
The LV domain is reachable from the MUON panel, from the
MUONA_DCS and MUONC_DCS hierarchy.
Click directly on the relevant CU (RIGHT mouse button), or start from the top level clicking on MUONA_DCS on the FSM panel. The corresponding panel appears.
From this panel you can descend the hierarchy across quadrants and stations reaching the LV controls (see next slides)
23
At this level, the different sub-systems of LV are visible and accessible
Q1M23A panel: DAQ tab
The DAQ tab refers to ODE, SB
& PDM, IB crates
Act only on the FSM buttons to switch on/off the sub-systems.
This part of the panel should be used only to check the effects of the action, not to cause them
24
The FEE tab refers to the
CARDIAC boards
Q1M23A panel: FEE tab
25
Starting the MuonMoni Presenter
The better way to check the global status of the detector is looking at the data quality by means of the online Presenter.
To start the Presenter:
1.
2.
3.
4.
5.
Launch Exceed on your PC. There is a small problem with the new Exceed: the default configuration settings are re-set by the installer.
Host access is now per default *DISABLED*. To change this open Exceed -> Tools ->
Configuration -> Security and remove the access control.
Log on mudaq01 (*) type: . /group/muon/scripts/lhcbsetup.sh
set local display: export DISPLAY=pcname:0.0 (**) run presenter:
/group/online/presenter/run_presenter.sh
(*) Log-in using putty:
- session: mudaq01
- connection
SSH
X11 (enable x11 forwarding) open and login
(**) PC name : see label glued on the monitor
See the twiki page https://twiki.cern.ch/twiki/bin/view/LHCb/MuonDaqMonitoring about the use of the Presenter
26
For example: tell1 PC
The PC is: mudaq01 (login with Putty)
Command line to open the “gedi” panel: PVSS00ui –proj MUDAQL1 –m gedi
On “gedi” go:
- Module
- Vision
- localdisk/PVSS/MUDAQL1/panel
- /grup/muon/fwComponent_MUDAQL1/panel
To open the “DEN” panel
- select the folder fwDeviceEditorNavigator
- open fwDeviceEditorNavigator.pnl
27
1.
Go into MUON_DAQ panel
2.
Create a new recipe
• Select the parameters needed
(Thresholds, Masks, …)
• Select the detector regions/side
• Push “set the configuration to region …”
• Check “re-load timing result” if the timing parameter have been changed
• write the recipe name
• Push “Create Recipe”
• Wait until the process-bar arrive at 100%
3.
Upload an existing recipe
• Select the detector region/side
• Push “set the configuration to region …”
• Select a recipe from the
“Recipe List”
• Push “upload Recipe”
• Wait until the process-bar arrive at 100%
28
1.
2.
3.
1.
2.
3.
4.
5.
6.
Configure the ODE starting from the timing file
1.
2.
3.
Go into the HU you need from the Musides shortcut and reset the ODE board
Push “go to expert” and “continue”
Into the tab “commands” push “Load HW Conf File”
1.
Select the file you want write into the hardware
Save a PULSING recipe
Go into the corresponding ODE PC
Open the ODE panel you want to change from the DEN
Push “recipes…”
Select the recipe you want create or overwrite (for example PULSING )
Create the recipe “Create from hardware”
Select “PULSING” recipe and push “edit”
1.
2.
3.
4.
Check that all “GOLCTRL” value are 0x00
1.
If not: push “edit” into “table Mode”
Write 0x00 for all the column “GOLCTRL”
Push “enter” into the keyboard: the changes you made became yellow
Push “save” and “close”
2.
3.
4.
5.
Save a PHYSICS recipe
1.
Go into the corresponding ODE PC
Open the ODE panel you want to change from the DEN
6.
7.
Push “recipes…”
Select “PHYSICS” recipe and push “edit”
Take note of the value of the TTCrx registers “FD1” and “CD” and “close”
Create the recipe “Create from hardware”
Select “PHYSICS” recipe and push “edit”
1.
2.
3.
4.
push “edit” into “table Mode”
Wrote the value you have toke note for TTCrx “FD1” and “CD” registers
Push “enter” into the keyboard: the changes you made became yellow
Push “save” and “close”
29
Chambers/CARDIAC
SB/PDM IB ODE
HV distributor
LV Marathon
LV distributor
(RCM/ACDC)
PCs with PVSS projects and SYSTEC witch connect ECS signals from SB/PDM/ODE to the PC)
Tell1 boards for
DAQ and L0 trigger
IB
ODE
SB/PDM
30