The CryoLinks and their Control System
A description of the mechanics, sensors and actuators and controls.
Fred Schimmel, Nikhef - Amsterdam
March 28, 2014
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
Table of Contents
1
Introduction..................................................................................................................................... 4
2
Some mechanical aspects. .............................................................................................................. 6
2.1
The CryoLink. ........................................................................................................................... 6
2.2
The Phase Separator. .............................................................................................................. 7
2.3
The Exhaust. ............................................................................................................................ 7
3
Control concept. .............................................................................................................................. 8
4
Sensors. ......................................................................................................................................... 10
5
6
4.1
Temperature-sensors. ........................................................................................................... 10
4.2
Level-sensors. ........................................................................................................................ 11
4.3
Pressure-sensors. .................................................................................................................. 12
4.4
Displacement -sensors. ......................................................................................................... 12
Control devices. ............................................................................................................................. 13
5.1
Control valves. ....................................................................................................................... 13
5.2
Solid-State Relay. ................................................................................................................... 13
Hardware Interlocks. ..................................................................................................................... 14
6.1
Incoming interlocks. .............................................................................................................. 14
6.2
Outgoing interlocks. .............................................................................................................. 14
7
The Control system Hardware....................................................................................................... 15
8
Overview of the CryoLink Control system Software. .................................................................... 19
9
8.1
FPGA. ..................................................................................................................................... 19
8.2
RealTime system.................................................................................................................... 19
8.3
Expert User Interface. ........................................................................................................... 20
8.4
Simulation program for testing. ............................................................................................ 20
Details of the CryoLink Control system Software. ......................................................................... 22
9.1
Details of the FPGA code. ...................................................................................................... 22
9.1.1
FPGA – Pt100. ................................................................................................................ 22
9.1.2
FPGA – Analogue Input.................................................................................................. 22
9.1.3
FPGA – Digital Input Interlocks. ..................................................................................... 23
9.1.4
FPGA – Digital Input Capacitive Level sensors............................................................... 23
9.1.5
FPGA – PID-loop. ........................................................................................................... 23
9.1.6
FPGA – Watchdog-loop. ................................................................................................ 24
A.Schimmel,
Nikhef, Amsterdam
-2-
-Document1-
Printed: 10-2-2014 20:31
9.2
The CryoLinks and their Control System.
Details of the RT-program code. ........................................................................................... 24
9.2.1
RT – UI Commands Loop. .............................................................................................. 25
9.2.2
RT – Message Handling Loop......................................................................................... 25
9.2.3
RT – Modbus Interface Loop. ........................................................................................ 25
9.2.4
RT – Watchdog Loop. .................................................................................................... 26
9.2.5
RT –System Health & FPGA Monitoring Loop. ............................................................. 26
9.2.6
RT – Logging Loop. ......................................................................................................... 26
10
Details of the Expert User Interface (EUI) program. ................................................................. 27
10.1
11
Details of the Expert User Interface (EUI) program code...................................................... 30
Appendices. ............................................................................................................................... 32
11.1
Appendix A. Level reference sensors .................................................................................... 32
11.2
Appendix B. Some LabVIEW terminology.............................................................................. 32
11.3
Appendix C. Network Streams .............................................................................................. 34
11.4
Appendix D. Shared Variables ............................................................................................... 34
11.5
Appendix E. Operational States............................................................................................. 34
11.6
Appendix F. Commands supported by the ‘RT Message Handling Loop’.............................. 35
11.7
Appendix G. Local Logfiles .................................................................................................... 36
11.8
Appendix H. Commands supported by the ‘UI Message Loop’ ............................................ 37
A.Schimmel,
Nikhef, Amsterdam
-3-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
1 Introduction.
The main parts of the Virgo experiment are two, three kilometer long, round vacuum tubes of about
1m diameter, forming the arms of a huge interferometer. They are positioned perpendicular to each
other and come almost together on one side in the central building. On that point the injection and
detection equipment is located (see the picture on the front page for an impression).
When the detector is in use, high power photon beam travels through each arm to and fro between
the highly stabilized mirrors at the ends. In order to minimize scattering of the photons, which
degrades the coherence of the beams of light, the quality of the vacuum has to be as good as
possible. The Advanced Virgo project will upgrade the existing detector in a number of ways; a
significant improvement of the vacuum is one of them. It is investigated that the main component in
the residual gas is water, originating from the vacuum enclosures of the suspension systems for the
end-mirrors. Trapping the water molecules will bring the end pressure to ~1E-9mBar, instead of
~1E-7mBar in the current situation.
Under high vacuum conditions the residual atoms and molecules are bouncing against the inner
surfaces of the vacuum vessel, without interaction with each other. To catch the water molecules
before they enter the interferometer arms, two meter long cryogenic sections (-196⁰C) will be
installed near the four end-mirrors. When water molecules hit the cold surface, they will stick to it
and remain trapped until the section is allowed to become warm (the regeneration process).
These four devices, (one on each end of the two arms) are called the CryoLinks. Nikhef has taken
responsibility to build the cryogenic systems and the controls to operate them in a stable and reliable
way.
Figure 1 - Artist impression CryoLink
A.Schimmel,
Nikhef, Amsterdam
-4-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
According to the naming convention of the European Gravitational Observatory (EGO), the four
CryoLink systems are named:
-
CryoLink-WE
CryoLink-WI
CryoLink-NI
CryoLink-NE
West arm – End (opposite to Interference point)
West arm – Injection (near Interference point)
North arm – Injection (near Interference point)
North arm – End (opposite to Interference point)
Figure 2 - Simplified layout Virgo with CryoLinks
A.Schimmel,
Nikhef, Amsterdam
-5-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
2 Some mechanical aspects.
2.1 The CryoLink.
The main part of a CryoLink is the cryogenic vessel, consisting of two aluminum tubes of different
diameter (1120 and 980mm) located around each other, but not circle symmetric: the inner tube’s
center is 32mm shifted downward relative to the center of the outer one (see Figure 3). The outer
ends are connected to each other by semi-circular plates, resulting in a closed vessel with a small
distance (~26mm) between the walls at the bottom and about 90mm at the top side.
Figure 3 - schematic view CryoLink (cut)
When this vessel is filled with liquid Nitrogen to a level above the top of the inner tube, the whole
inner tube is in contact with boiling Nitrogen, ensuring an equal temperature on the inner surface,
while there is a relative big surface for the vapor bubbles to escape from the liquid. This design is
meant to reduce the vibrations of the device due to the boiling of the nitrogen, to a minimum. The
top part of the outer cylinder maintains a low temperature by heat conduction through the
aluminum wall plus the contact with cold gaseous N2 at the inside.
The CryoLink construction is an integral part of the interferometer vacuum system, as each one is
mounted directly to the vacuum-vessel of an end-mirror suspension system. Therefor it is important
A.Schimmel,
Nikhef, Amsterdam
-6-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
that the CryoLinks hardly produce noise and/or vibrations, which may deteriorate the performance
of the interferometer. The design of the CryoLink is strongly dedicated to this constraint; vibrations
measurements on the tested prototype show only very low levels, while the main frequencies are
outside the region of interest of the Virgo experiment.
The cryogenic vessel itself is hanging in the bigger stainless steel vacuum vessel. It is suspended on
the outside of the vacuum vessel by strong rubber “balloons” filled with pressurized air (about 4.6
Bar). The feedthroughs are stainless steel bellows to minimize the transfer of vibrations from the
cold vessel to the vacuum vessel.
2.2 The Phase Separator.
The liquid N2 for the CryoLinks is supplied from big storage containers under several Bars pressure,
which causes the boiling temperature of the supplied liquid to be some degrees higher than under
atmospheric pressure. To obtain liquid which boils at the actual ambient pressure, Phase Separators
are installed.
A Phase Separator is an isolated cryogenic container in which a level of liquid N2 is maintained, that
boils at ambient pressure. The liquid nitrogen, supplied from the storage container is cooled to the
lower boiling temperature by “flash evaporation” of a (small) fraction of the liquid: the energy from
the lowering of temperature is consumed by the evaporation of some liquid. As the evaporation heat
is about 100 times higher than the specific heat of the same amount of liquid N2, the evaporation in
percentage is roughly equal to the number of degrees the temperature has to decrease.
The Phase Separator is mounted at such level that its outlet is positioned ~20cm above the inlet of
the CryoLink. This height difference plus the level of the liquid N2 in the Phase Separator delivers the
(hydrostatic) pressure over the control-valve for the CryoLink.
2.3 The Exhaust.
The evaporated Nitrogen leaves the CryoLink via a tube of 100mm, which finally ends outside the
building. Due to this big diameter the gas moves through the exhaust pipe with a laminar flow. The
absence of turbulences reduces the noise to a minimum. As the gas enters the exhaust at -196⁰C, the
exhaust will adopt a temperature much below the dew-point of the ambient air, which could cause
the forming of ice and water on the outside. As the ambient temperature varies, the thickness of the
ice-layer will vary as well, giving dripping water on places one does not want to have it.
From practical experience it is known that preventing the forming of ice/water by isolating such a
cold pipe is almost impossible. Over time moisture will inevitably enter the isolation layer, freeze and
deteriorate it.
Applying a vacuum isolation is fine, but expensive to build, certainly when a long exhaust is required.
With this in mind it is decided to apply a heater at the beginning of the exhaust, which elevates the
temperature of the Nitrogen above the dew-point of the ambient air, allowing a non-isolated pipe to
be used.
A.Schimmel,
Nikhef, Amsterdam
-7-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
3 Control concept.
To ensure a smooth, optimal functioning the level of liquid N2 in a CryoLink should be kept stable at a
height of several centimeters above the top of the inner wall of the cryogenic vessel. Despite two
radiation there is a heat-flow to the cold vessel, causing evaporation of liquid N2. This evaporation is
compensated by an adjustable flow of liquid N2 from the Phase Separator. An electro-pneumatic
control valve in the connection adjusts the flow, according to the output-signal of a PID-controller 1.
The controller has a wanted level (the Setpoint) and a measured level (sensor PT02) as inputs (see
Figure 4, middle part).
Figure 4 - overview PID-loops for controls
The flow through a control valve (such as CV02) is determined by several parameters, among which
the pressure drop over the valve. As described before, the hydrostatic pressure of the difference in
height between the liquid level in the Phase Separator and the inlet of the CryoLink determines the
pressure drop over CV02. To obtain an optimal control behavior, the pressure drop parameter should
remain constant, implying that the level in the Phase Separator must be kept constant. This is
achieved by another PID-controller, which adjusts the flow of liquid N2 from the Storage Tank to the
Phase Separator by control valve CV01 (see Figure 4, left part).
The temperature of the evaporated Nitrogen in the exhaust is controlled by the third PID-controller,
(see Figure 4, right part). The temperature of the gas is measured with temperature-sensor TT07, the
PID-controller adjusts the dissipated power in the heater to keep the measured value close to the
Setpoint.
For safety reasons the CryoLink Control system must autonomously switch to the ‘Safe State’ in case
the conditions for normal operation (e.g. malfunctioning HW, incoming interlocks, operator
command) are absent. The ‘Safe State’ means that the two control valves (CV01 and CV02) are closed
preventing supply of liquid N2. This ‘Safe State’ is implemented by stopping the PID-controllers for
the levels of the Phase Separator and the CryoLink and adjusting the valves to setting 0; the PIDcontroller for the heater in the exhaust should run always, as the evaporation continues as long as
1
Proportional-, Integral- and Differential- Controller, a widely applied control loop feedback mechanism to
keep a process-variable close to a wanted value. See < http://en.wikipedia.org/wiki/PID_controller> for details.
A.Schimmel,
Nikhef, Amsterdam
-8-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
liquid N2 is present. And as this controller keeps the temperature of the gas in the exhaust at the
wanted value (e.g. 15⁰C) there is no need to put it in a Safe State. In case there is no cold N2 gas
leaving the CryoLink the heater will not dissipate power, so the temperature will not exceed the
setpoint value.
A.Schimmel,
Nikhef, Amsterdam
-9-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
4 Sensors.
4.1 Temperature-sensors.
On the inner surface of the cryogenic vessel, six Pt100 temperature sensors are mounted, scattered
over the geometry of the construction. These sensors are connected in the four-wire configuration
to allow compensation for the wire- resistance and are monitored with dedicated Pt100 readout
devices. The heat dissipation in the sensors is dependent on the temperature: at 0⁰C it is 0.1mW, at
the liquid N2 boiling temperature about 0.02mW.
To control the dissipation in the 500W heater in the exhaust-pipe, a Pt100 sensor is mounted
downstream the heater section to measure the resulting temperature of the gas.
Figure 5 - Overview sensors and actuators (note the different layout)
A.Schimmel,
Nikhef, Amsterdam
-10-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
4.2 Level-sensors.
As mentioned before, for stable operation of a CryoLink and Phase Separator, a constant level of
liquid Nitrogen is required. To measure the level of liquid N2 two techniques are common:
 measure the hydrostatic pressure, resulting from the weight of the column of liquid N2
formed by the contents of the cryogenic vessel,
 measure the capacity of two conductors close to each other: the difference of the dielectric
constants of gaseous and liquid Nitrogen makes the capacity linear related to the height of
the liquid level.
Both techniques are applied. The commercially obtained Phase Separator has a capacitive level
sensor. The CryoLink has a differential pressure sensor as primary sensor for the level and two
capacitive sensors for redundancy reasons.
One side of a differential pressure sensor (PT02, 100mBar FS) is connected to the outlet for the
evaporated Nitrogen while the other side is connected to the bottom of the cryogenic vessel by a
thin tube. It is proven that the heat-leak through the wall of the thin tube will cause the tube to be
filled with gaseous Nitrogen. The pressure on this side relates linear to the height of the liquid
Nitrogen inside the vessel. The maximum pressure follows directly from the geometric data (max.
height at -196⁰C is 1078mm) and physical constants (density liquid N2 = 0.802kg/l and the gravity
constant 9.81m/s^2) resulting in a pressure of 84.36mbar when the cryogenic vessel is fully filled.
For the capacitive level measurement two sensors are installed: LT03, a long, bended set of
aluminum strips form the capacitor for a level indication during filling and emptying. A small set of
parallel strips, mounted under about 45⁰, form another capacitor that may indicate the level above
the top of the inner cylinder of the cryogenic vessel with much greater precision (LT02) (see
Figure 6).
Figure 6- Capacitive level sensors (schematic)
A.Schimmel,
Nikhef, Amsterdam
-11-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
To enable the calibration of the capacitive level-sensors, three Pt100 sensors are mounted on
well-known positions in the level range of LT02. These Pt100’s are connected to a 24V DC supply by a
1kΩ series resistor, creating a dissipation of ~0.02W. Due to this dissipation, the temperature of the
Pt100 in gaseous N2 is considerable higher than when it is immersed in liquid N2. This arrangement
results in different voltage-drop over the Pt100, depending on the condition around it (see below)
On ambient temperature
In cold gaseous N2
In liquid N2
V = 2.5V
V ~ 0.8V
V ~ 0.47V
When the voltage over a Pt100 sensor shifts from ~0.8V to below 0.5V, the level of the liquid N2 is at
the height of the sensor. By this provision a precise check on the readout of the pressure level-sensor
and the capacitive level-sensors is possible.
For the readout of each capacitive level-sensor an oscillator-unit is installed, which converts the
actual capacity of the sensor to a pulse-train with a frequency that is proportional to the actual
sensor-capacity. The pulses are counted in software every second, which delivers the actual
frequency with 1Hz precision. Based this frequency the level can be calculated; the required
coefficients for the linear equation are determined by an auto-calibration facility (see Appendix A for
details). The calibration data are stored in files to allow the control program to retrieve the most
recent data after (re-)booting of the control system.
4.3 Pressure-sensors.
A differential pressure-sensor (PT01, 1000mB FullScale) is on one side connected to the gaseous N2
outlet, the other side is open to the ambient air. As this sensor measures the over-pressure in the
exhaust, a possible obstruction (e.g. forming of ice in the exhaust that narrows the diameter) may be
observed.
The pressure of the supplied pressurized air is monitored with an absolute pressure sensor (PT03,
10 bar FullScale).
4.4 Displacement -sensors.
On the location of the two suspensions of the cryogenic vessel, LVDT’s2 (15mm range, 10m
resolution) are mounted to monitor the vertical movement of the cold vessel, relative to the vacuum
container. These displacement sensors will detect failure of the rubber suspension bellows and/or
the control of the air pressure in the bellows. As the bellows act like a spring, the LVDT readout is to
some extend a relative measure for the total weight of the cold vessel (Aluminum and liquid N2).
2
Linear Variable Differential Transformer, a device to convert position or displacement into an electrical signal,
see <http://en.wikipedia.org/wiki/LVDT> for more information.
A.Schimmel,
Nikhef, Amsterdam
-12-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
5 Control devices.
5.1 Control valves.
In the supply line of the liquid N2 to the Phase Separator and in the isolated tube from the Phase
Separator to the CryoLink, two cryogenic control valves are mounted to dose the amount of liquid
flowing into the vessels. The valves are vacuum isolated and have an dedicated layout to reduce the
heat-leak to a minimum. An electric-pneumatic controller determines the opening of the control
valves by adjusting the pressure under a spring loaded membrane proportional to the electrical
control signal.
5.2 Solid-State Relay.
The quantity of power, dissipatedby the heater of the exhaust pipe, is controlled by a ‘Solid-State
Relay’ (SSR) module. This electronic device can switch on-off a substantial electrical power with a
repetition rate up to 50Hz. By adjusting the ‘on’ period relative to the ‘cycle-time’ (= on+off period),
the effective power is can be varied from 0% to 100%.
A.Schimmel,
Nikhef, Amsterdam
-13-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
6 Hardware Interlocks.
6.1 Incoming interlocks.
It has been agreed that the CryoLink Controls will monitor a number of incoming interlocks and react
on them in a predictable way. To allow additional functionality in the future, seven incoming
interlocks are wired and monitored. In a later state an additional functionality is requested. This
addition takes one spare (GR04), meaning there are now three spare incoming interlocks left.
The functionality is:
- GR01 Vacuum OK?
Isolation vacuum below 1E-7?
- GR02 Vacuum Control PLC OK? PLC system vacuum functions OK?
- GR03 liqN2 supply OK?
The supply system for liq.N2 functions OK?
- GR04 Half CL-level OK?
Decrease the level- limit for ‘level not_too_low’ to 50%
- GR05 Spare #5
- GR06 Spare #6
- GR07 Spare #7
6.2 Outgoing interlocks.
Also a list of outgoing interlocks is defined. Five outgoing interlocks were specified, two spares are
added for future extensions.
The functionality is:
- CL01 Controls OK?
CryoLink Control system functions OK
- CL02 CL in Operation?
All conditions for operation are met
- CL03 CL-temp not_too_high?
The CryoLink temperature is below -193⁰C
- CL04 CL-level not_too_high?
The liquid N2 level in the CryoLink is below the upper limit
- CL05 CL-level not_too_low?
The liquid N2 level in the CryoLink above the lower limit *)
- CL06 Spare #6
- CL07 Spare #7
*) the value of the lower-limit is dependent on the incoming interlock ‘GR4 Half CL-level OK?’. When
not set (<5V on input) the lower level limit is 5mm above the top of the inner cylinder of the cold
vessel. When set (>10V on input) a level of more than 50% of the height of the cold vessel is enough
to signal ‘CL05 CL-level not_too_low?’ = OK.
A.Schimmel,
Nikhef, Amsterdam
-14-
-Document1-
The CryoLinks and their Control System.
Printed: 10-2-2014 20:31
7 The Control system Hardware.
To perform all control functions for the operation of a CryoLink (readout of all sensors, steering of
the control-valves, implementation of the PID-controllers and handling of the interlocks) a National
Instruments™ cRIO (=compact Reconfigurable Input and Output) system is selected.
The cRIO system consists of a cRIO-9112 8-slot chassis with a build-in Virtex-5 LX 30 FPGA 3and a
cRIO-9014 Real-Time PowerPC (RT-PC), running a real-time OS4 (VxWorks). The RT-PC has 128MB
volatile memory and a 2GB SSD (solid state disk), a 10/100MB Ethernet interface and an USB 2.0
connector for optional coupling of an external storage device.
The chassis slots can be populated with C-series I/O modules; nowadays many models are available
for all kind of interfacing. For the controls of the CryoLinks the following modules are selected:
 NI-9217 100Ω RTD5 module, delivers excitation and digitizes 4 resistances with a 24bit ADC6,
 NI-9205 16ch diff. Analogue Input module, digitizes analogue input signals with a 16bit ADC,
up to 250kS/s (kiloSamples per second),
 NI-9265 4ch Analogue Output 0…20mA current output module, applies a 16bit DAC7, up to
100kS/s,
 NI-9375 16ch Digital Input + 16ch Digital Output, sinks resp. sources 24V signals.
Figure 7 - cRIO-chassis with the selected C-modules (connectors not mounted)
3
Field Programmable Gate Array, see < http://en.wikipedia.org/wiki/Field-programmable_gate_array>.
4
Operating System, a collection of software modules required to run programs on a computer.
5
Resistance Thermometer Detector, a temperature sensor based on measuring resistance of a material with a
well-known R-T relation.
6
Analogue to Digital Converter, electronic device that converts an analogue electrical value to a digital value.
7
Digital to Analogue Converter, electronic device that converts a digital value to an analogue electrical value.
A.Schimmel,
Nikhef, Amsterdam
-15-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
The inserted modules are directly connected to the FPGA to allow high speed interfacing of the
readout and control functions, see Figure 8.
The programmer can define a number of variables which “cross” the border between the FPGA and
the RT-system by enabling writing and reading of variable-registers in the FPGA-code.
Figure 8 - Schematic view FPGA-IO
The control system is designed to provide a maximal reliability, i.e. the critical functionality
(digitalization of the signals, the PID-controllers, the steering the Control-valves and the heater and
handling of the interlocks) is running in the FPGA and will fail only when the power is cut or when a
hardware failure occurs. In fact, the essential part of the CryoLink Control system is functioning when
the FPGA runs and the C-series modules have no error, even if the RT-program crashes.
The less critical functions, like logging of the data, running a Modbus8 interface for the connection to
the Tango9 control system and the connection of a User-Interface (UI), are handled by the Real-Time
system. Both the FPGA-code and the RealTime-code start running automatically after power-up of
the cRIO system, independent of a User-Interface being connected or not.
8
Modbus is a serial communication protocol, introduced in 1979 for communication with Programmable Logic
Controllers. Nowadays a TCP/IP implementation is available, which is applied. For details, see:
<http://en.wikipedia.org/wiki/Modbus>
9
Tango is a free open source device-oriented controls toolkit for controlling any kind of hardware or software
and building SCADA systems. It is used for controlling synchrotrons, lasers, physics experiments in over 20 sites.
It is being actively developed by a consortium of research institutes. For details, see:
<http://en.wikipedia.org/wiki/Tango>
A.Schimmel,
Nikhef, Amsterdam
-16-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
To enable visualization of the control system and to allow user interaction, an Expert User-Interface
program (EUI) is created. This program may run on any computer that can make a TCP/IP connection
with the cRIO-system. The UI Front-Panel shows a schematic view of the CryoLink system with actual
values of sensors and settings. This interface is exclusively meant for diagnosis and adjusting and/or
tuning of parameters by an expert.
Figure 9 schematically shows the relations between FPGA, RT-program and EUI.
Figure 9 - Schematic overview Control Software
The Phase Separator (PS) and all other equipment of the CryoLink Control System is assembled in a
dedicated 19” cabinet that is positioned close to the CryoLink vacuum vessel, because the isolated
transfer-line for the transport of liquid N2 from the Phase Separator to the CryoLink has a limited
length (~1.8m).
The vessel of the Phase Separator is mounted in a dedicated support inside the upper part of the
cabinet; the top-cover of the cabinet is removed to accommodate the PS at the required height.
At the front side the cRIO chassis with the interface modules, a very simple control panel and the
24V= power supply are mounted (see Figure 10).
A.Schimmel,
Nikhef, Amsterdam
-17-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
On some installation locations it will be impossible to open the back-door of the cabinet. Therefor
the terminals for the connection of the 230V~ power, the exhaust-heater and the incoming and
outgoing interlocks are located on the front-side of the cabinet.
Figure 10 - 19"cabinet with Phase-Separator
A.Schimmel,
Nikhef, Amsterdam
-18-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
8 Overview of the CryoLink Control system Software.
All software (the FPGA-code, RealTime application, as well the UI-program) is developed in a
LabVIEW based program environment. LabVIEW© is a graphical programming tool, developed by
National Instruments, where the instructions are pictograms and the data going in and out is
represented as “wires”, similar to an electrical scheme. LabVIEW is introduced more than 20 years
ago, originally as tool for technicians to create their instruments in software, in contrary to HW
devices. This history explains the term VI (Virtual Instrument) which still is commonly used to indicate
a piece LabVIEW code. See Appendix B for more information.
8.1 FPGA.
The programming of the FPGA is done in the same way as an ordinary LabVIEW program, however
with some constraints which are directly related to the way a FPGA functions. A number of LabVIEW
primitives are not available (impossible to implement) or strongly discouraged because their
implementation will consume a considerable part of the (limited) FPGA resources. For instance:
when floating point numbers are required, it is advised to apply ‘FixedPoint’ variables instead of
‘Floats’. Implementation of arithmetic with ‘FixedPoint’ variables requires much less gates and
control-lines.
When the FPGA program is completed, the programmer can start a dedicated compilation tool that
transforms the LabVIEW code to a FPGA ‘bitfile’ that can be loaded to the FPGA and run.
The FPGA can have connections to the outside world in two ways:
 C-series IO-modules. A cRIO chassis has a number of slots where C-series modules of the
wanted functionality can be inserted in to connect sensors, switches, actuators, etc. to the
cRIO system.
 Software Controls (inputs for data to the FPGA) and Indicators (outputs for data from the
FPGA) together can be defined as an interface between the FPGA code and the application
running in the real-time system (see Figure 9).
The program running in the RT-system (from now indicated as ‘RT-program’) can send values to the
controls and read values from the controls and indicators defined in the FPGA-code. The update rate
is determined by the RT-program as the readout is done by polling (see Figure 9, lower part).
8.2 RealTime system.
In the design of the RT program special care is taken to prevent blocking of the program execution in
case an action takes some time. The functions to be performed are logically split and implemented in
separate while-loops which execute totally independent of each other, like threads in modern
computers.
As the RT system should be able to run “headless” (no display, no keyboard), its design is different
from an ordinary program. Errors should not stop the RT-program but should be logged and handled
in a smart way to ensure continuation of the controls. Popup panels have no use, there is no
operator to look at and act on it. A watchdog mechanism is implemented to monitor the “health” of
A.Schimmel,
Nikhef, Amsterdam
-19-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
the communication between the RT-program and the FPGA. In case of a timeout the RT-system will
be rebooted after which the RT-program will restart automatically.
Once installed on the EGO-site, the CryoLinks must be integrated in the Tango system for ‘Slow
Control’. The Tango communication is based on Modbus over TCP/IP and is capable to communicate
with ‘non-Tango’ Modbus Slaves. Each CryoLink Control system contains a Modbus Slave process
which holds the actual values of all relevant parameters. The Tango system can poll the Modbus
Slaves to retrieve the actual state of each of the CryoLinks . Beside this readout, three commands
from the Tango system are accepted and executed by the CryoLink Controls. These commands are:
 Reset Interlock errors?
 Go to Operation?
 Stop liqN2 Supply?
This mechanism is dedicated to remote monitoring and very simple remote control from a central
control room.
8.3 Expert User Interface.
For dedicated local access to a CryoLink Control system by an expert (for troubleshooting, updates,
tuning of the PID-controller parameters, etc.) the Expert User Interface (EUI) program can be used.
This expert program will be available on a dedicated laptop; when connected to the Virgo computer
network infrastructure, the expert can command the EUI program to setup some dedicated
communication channels with the RT-program of a CryoLink, enabling ‘live control’ of the system it is
connected to.
A special while-loop in the RT-program polls if a EUI-program tries to establish a connection. When
such an attempt is detected, two “Network Streams” (point-to-point SW-communication channels
using a proprietary protocol, see Appendix C for details) are configured, one for receiving
commands/data, the other for sending replies/data. In the “connected” state this loop functions as
communication channel with the UI-program. Received messages are checked on validity and sent to
the ‘action-queue’ of a simple finite-state-machine that handles the requests. When an UIapplication is connected, the RT-system will autonomously report important information and react
on requests and new settings sent by the UI via the Network-Stream mechanism.
Beside the Network Streams a number of ‘Shared-Variables’ are activated. The ‘Shared-Variable’
mechanism allows a programmer to exchange data between applications in an easy, transparent
way. See Appendix D for details.
The EUI-program monitors the Shared Variables (only read values in case of the CryoLinks) and
updates its display when new values are reported.
8.4 Simulation program for testing.
Testing of the FPGA-code, the RT-program and the EUI with the real Phase Separator and CryoLink is
quite hard to perform, because the processes are very slow (it takes days to cool the CryoLink to 196⁰C and >36H to lower the level from 95% to 50%). To allow testing of the software itself and the
implemented algorithms and procedures, a software simulation of the CryoLink and Phase Separator
is made.
To connect the simulation code to the cRIO system, the readout of the input C-modules in the FPGA
can be switched to reading a set of simulated values and the values, written to the output modules,
are made available for the simulation program. In this way the interaction with the real HW can be
switched to interaction with the simulation code that fakes the HW. See Figure 11 for a scheme of
A.Schimmel,
Nikhef, Amsterdam
-20-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
the routing of the data. The SW-switch of the inputs and outputs is depicted as a multi-pole electrical
switch.
Figure 11 - Schematic view implementation of Simulation code
WARNING: the remote Tango control system should check if the ‘Simulation-mode’ is <off> when the
detector is in operation to prevent illegal CryoLink status information.
A.Schimmel,
Nikhef, Amsterdam
-21-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
9 Details of the CryoLink Control system Software.
The control system for a CryoLink consists of three pieces of code that run on three different
hardware platforms:
- the FPGA code is translated to a ‘bitfile’ that automatically is loaded into the FPGA and
started,
- the Real-Time code, a ‘top-level VI’ that contains the additional functionality to run the
control system and interfaces. This VI is compiled and linked into an executable (from now
called the ‘RT-process’) which is stored on the Disk-drive of the Real-Time computer,
- the Expert User Interface (EUI), a ‘top-level VI’ that acts as graphical user interface which can
connect to each of the four CryoLink Control Systems, is a LabVIEW program that runs on a
laptop with a network connection.
The FPGA-code and RT-process run on the hardware of the cRIO-chassis with C-modules as hardware
interfaces; every CryoLink is equipped with such a chassis. The laptop is meant as a portable console
for the expert. It must be connected to the Virgo LAN (Local Area Network), most likely close to the
CryoLink of interest.
Although the four CryoLink Control Systems will be monitored and logged by the Tango control
system, every system logs its own data also locally. The ReadData and WriteData are to tabseparated ASCII logfiles. The status-information goes to an ordinary text-file. All records start with a
Timestamp on a new line. These logfiles contain data of maximal 24H, starting at 00:00 or the first
moment data is written; the filenames contain the date. For more details, see Appendix E.
9.1 Details of the FPGA code.
9.1.1 FPGA – Pt100.
For the readout of the seven Pt100 temperature sensors two NI 9217 modules, dedicated for precise
(24bits) digitalization of 4-wire 100Ω RTD’s, are applied. These modules convert the measured
resistances to a many digit decimal representations of the resistances. The non-linear mapping of
resistance to temperature is done in software. As the temperature of the N2 in the exhaust is
required as input for the Heater PID-controller, a simple linear conversion is used in the FPGA code to
obtain the gas temperature. The error is limited because the range for this conversion is small (80⁰C).
All channels of the RTD-modules are read in a 1 sec. infinite loop, which is more than enough for the
CryoLink Controls because the big mass of the system prohibits fast changes. The readings are quite
stable which makes additional improvement of data quality unnecessary.
9.1.2 FPGA – Analogue Input.
The signals of all other analogue sensors are digitized by the NI-9205 Analogue Input module. This
module can digitize 32 analogue 0…10Volt input channels in ‘Single Ended’ mode or 16 analogue
channels in ‘Differential’ mode. In the selected Differential mode the voltage difference between the
two signal inputs of one channel is measured, independent from the common voltage swing of the
signals wires. Without this precaution, the resulting digital values appeared to be rather noisy. To
improve the signal quality, all channels are sampled at a 1024 Hz repetition rate and averaged every
250ms. The averaging reduces the random noise level significantly. On top of this, the data of two
channels is fed to other moving average algorithms to be smoothed even more. Especially the
readout of the differential pressure sensor PT02 needs additional smoothing, as the process which
produces the pressure difference is somewhat unstable and this signal is used as process value for
the PID-controller for the CryoLink liquid-level. The readout of LT01, representing the level of liquid
A.Schimmel,
Nikhef, Amsterdam
-22-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
N2 in the Phase Separator, goes through an 8-value averaging, PT02 is fed to a 32-value averaging
algorithm.
The net result of the averaging is that the readout values are much more stable; however at the price
of a certain delay for the channels with additional averaging. A step in the LT01 signal will be totally
effectuated after 2 seconds, for PT02 this is 8 seconds.
9.1.3 FPGA – Digital Input Interlocks.
The state of the 7 incoming HW interlocks is polled by the readout of the lower 8 Digital inputs of the
NI 9375 module in an eternal loop with 1 second repetition period. In the same loop some control
parameters are logically combined to determine if the CryoLink Controls should switch to or leave
the ‘Safe State’. The control system will switch to ‘Safe State’ when:
- one/more of the incoming interlocks GR01, GR02 or GR03 switch <off>
- the watchdog mechanism that monitors the communication between the RT-program and the
FPGA produces a timeout
- the operator gives the software command ‘Switch PIDloops <off>’
When the conditions have become ok again, the system will not leave the ‘Safe State’ until the
operator explicitly has given the command ‘Reset Interlock?’ supported by the Tango Modbusinterface. This condition is defined by EGO as requirement for the control software.
9.1.4 FPGA – Digital Input Capacitive Level sensors.
The digital inputs 14 and 15 of the NI 9375 are read at the maximum repetition rate of the module,
100kHz. At every cycle is determined if the each of the inputs has changed from 1 to 0; the number
of these transitions is counted and the values of the two counters is exported every second. The
pulse-signals of the two oscillator units (convert the capacities of the two capacitive level sensors
LT02 and LT03 to pulse-trains, connected to these two digital inputs are counted here. The measured
frequencies will be converted to liquid level values by a calibration/conversion algorithm in the RTprogram.
9.1.5 FPGA – PID-loop.
The PID-loop is an eternal loop which cycles at a 1Hz repetition rate. Beside the execution of the PIDcontrol algorithms and the required conversions of the input- and output-data, some other functions
are implemented here for practical reasons. These are:
1. the determination of the operational state of the Phase Separator and the CryoLink
2. the writing of two channels of the NI 9265 0…20mA AO-module
3. the writing of a number of Digital Output channels of the NI 9375 DIO-module.
The operational state of the Phase Separator and the CryoLink is one of the defined states,
depending on the actual values of a number of parameters. The parameters and the applied
algorithm for the Phase Separator are different from that for the CryoLink. For details, see
Appendix F.
The outputs of the two PID-controllers for the liquid N2 levels are converted to 4…20mA values in
order to adjust the control valves CV01 and CV02; writing these values to channel 0 and 1 of the
NI 9265 effectuate the adjustments.
The output of the PID-controller for the Heater is converted to a Pulse Width Modulation (PWM)
<on>/<off> signal with a 1Hz cycletime; the length of the <on> period (the duty-factor) varies
between 0.0 and 1.0 sec (Ξ 0…100% power). This PWM signal is sent to Digital Output channel 8 of
the NI 9375, which is connected to the control input of a power Solid State Relay (SSR); the SSR
outputs a chopped 230V AC voltage and current to the heater elements on the exhaust.
A.Schimmel,
Nikhef, Amsterdam
-23-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
9.1.6 FPGA – Watchdog-loop.
The watchdog-loop is an eternal loop which cycles at maximum speed of the FPGA. Once enabled by
the RT-process it reads every iteration a boolean input that is cleared after reading. It calculates the
elapsed time between reading a ‘True’ and the previous reading of a ‘True’. If the elapsed time
exceeds the configured ‘Watchdog timeout period’ the loop is stopped, the event is reported to the
StatusLog and the RT-system is sent a signal to reboot itself. As the RT-system is configured to start
the RT-process automatically after startup, the Control system will continue to run after the short
(<30sec) interruption by the reboot.
9.2 Details of the RT-program code.
After startup a number of initialization actions are performed, like opening the interface to the FPGA
(which starts the FPGA-code if not running yet), initialization of the Logging tool, initialization of the
Modbus-Interface and the creation of the ‘Message Queue’. When finished, the RT-program starts
the six, independent eternal loops, described here below.
Although the loops are indicated as ‘eternal’, error-conditions may cause a loop to stop cycling.
Depending on the specific situation, a re-start is attempted or the loop is definitely abandoned,
which is followed by forcing all loops to stop, reporting the errors and finally a reboot of the RTsystem. The boot-sequence of the RT-system is configured to start the RT-program automatically,
which ensures the continuation of the controls. The RT-program has provisions to return to the
conditions before the reboot: the values of the write-parameters and a set of important read-values
(calibration- parameters for LT02 and LT03) are restored from dedicated files.
Figure 12- LabVIEW BlockDiagram of RT-program
A.Schimmel,
Nikhef, Amsterdam
-24-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
The Real-Time VI contains six parallel loops (see Figure 12):
1. UI Command Loop - Receive messages from UI Main VI,
2. Message Handling Loop - handle all messages coming from the UI Command Loop, itself, the
Watchdog Loop, and the Monitoring Loop,
3. Modbus Loop - handle all communication with the Tango control system by Modbus
messages,
4. Watchdog Loop - Send periodic 'pets' to FPGA Main VI to ensure communication is
maintained,
5. System Health and FPGA Monitoring Loop - Send RT diagnostic information and FPGA read
data to UI Main VI, handle setting changes from UI Main VI,
6. Handle Logging Loop - write new messages from LogBuffers to LogFiles.
9.2.1 RT – UI Commands Loop.
After initialization at startup this infinite loop checks every iteration if an Expert User Interface
applications attempts to make connection to the RT-program. If an attempt is detected, two Network
Streams are set up:
- the ‘UI Command Stream’ for messages from the EUI to the RT-program
- the ‘RT Reply Stream’ for message from the RT-program to the EUI)
The loop’s behavior changes to just forwarding received messages to the ‘RT – Message Handling
Loop’. To do so, the received messages are appended to the ‘Message Queue’ (which is accessible in
all parts of the RT-program). The ‘RT – Message Handling Loop’ monitors the queue for new
messages to be handled.
The message format is a LabVIEW ‘Cluster’ (which is equivalent to a ‘structure’ in C) of two elements:
first the ‘enumerated’ variable ‘Command’, followed by the ‘variant’ variable ‘Data’.
9.2.2 RT – Message Handling Loop.
This eternal loop polls every iteration if a message can be extracted from the ‘Message Queue’. Upon
success the retrieved message is split in the enumerated ‘Command’ and the variant ‘Data’. The
value of ‘Command’ is interpreted in a multi-value conditional (equivalent to the ‘switch()’ statement
in C), specifying the action(s) to be executed for the supported commands. See Appendix G for a list
of commands plus a short explanation.
The other RT-loops can also append command-messages to the ‘Message Queue’ to let the RTprogram execute the specified ‘Command’ (possibly with ‘Data’). By outsourcing the execution itself
to the ‘RT – Message Handling Loop’, the other RT-loops can quickly continue their iterations even
when the required action is time-consuming.
9.2.3 RT – Modbus Interface Loop.
The Modbus interface contains a list with actual values of all relevant variables of the CryoLink
Control System, formatted and accessible according to the Modbus specifications.
On request of the EGO system integrator, all variables in the Modbus interface are stored as U16. To
obtain the values in the correct format, the following conversions are applied:
bool
U16
Single precision float (SGL)
0 = False, 1 = True
none
join 2 subsequent U16 ‘s to a U32, typecast to SGL
The list of parameters and their datatypes exported by the Modbus Interface is described in a
separate document (‘CryoLink_ModbusInterface_ParameterSpecs.xlsx’).
A.Schimmel,
Nikhef, Amsterdam
-25-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
9.2.4 RT – Watchdog Loop.
At startup of the RT-Watchdog Loop the Watchdog function in the FPGA is enabled. After enabling
the endless RT- Watchdog Loop starts sending a ‘True’ to the ‘Pet Watchdog’ control of the FPGA
every 200ms.
9.2.5 RT –System Health & FPGA Monitoring Loop.
This important endless loop reads every second the values of all relevant read-variables from the
FPGA. The retrieved values are fed in groups of similar kind to the VI ‘RT-Handle_ReadDataBlock_Admin.vi’, which interprets/converts and stores the values internally. Every 5 sec. the values of
all read-variables are written to the ‘ReadData-Logfile’ on the local disk-drive of the RT-system.
Besides that, some other functionality is executed in this loop: the auto-calibration of the capacitive
level-sensors (if the conditions are present), the setting of parameters for an automated cooldown
procedure and the reporting of Alarm messages.
The FPGA-code checks if the various level values are in range. The failure of every check is bitwise
coded in the so-called ‘Level-Safety’ word which is exported to the RT-program. In the ‘RT – System
Health and FPGA Monitoring Loop’ another set of checks is executed on a number of FPGA-values.
The results of these checks are coded in the so-called ‘Non-Level Safety’ word. The two ‘safety
words’ are interpreted every second. Resulting Alarm messages (if any) are inserted to the Message
Queue for further processing.
9.2.6 RT – Logging Loop.
The RT-program applies three different logfiles: a Status-log, a ReadData-log and a WriteData-log.
The logging facility contains three circular string buffers (the Log-buffers) for temporary storage of
the three types of log-messages. The data to be logged is converted to strings before submitting to
the appropriate Log-buffer.
This eternal loop checks every second if new strings submitted to one or more Log-buffers. The
administration for each buffer contains a read- and a write-pointer, indicating the index of the most
recent string written to file and the index of the last submitted string. By comparing these pointers
the program concludes if there is data to be written to file; after writing to file the read-pointer is set
to the index of the last written string.
Writing to logfiles may be done in two ways:
- Fast but unsafe: the logfiles are opened at start and each write action adds its data to the
file. In case of a program crash it depends on the Operating System if the file will be closed
correctly.
- Save but slow: the logfiles are opened and closed for each write action to prevent data-loss
in case of a program crash.
This buffered handling of Log-messages prevents delays in the routines which generate the Logmessages because the message just has to be copied to the appropriate Log-buffer. The writing to
file is handled ‘Safe but slow’ in this independent loop where some delay of opening and closing of
files is no problem.
When the Logging Loop is slowed down too much (this should never happen) it may occur that the
write-pointer passes the read-pointer, meaning that strings are written to buffer locations that were
not written to file. The overwritten data is lost; the program detects such situation at the first
iteration of the Logging Loop after overwriting has happened and reports the event plus the number
of strings that were lost.
A.Schimmel,
Nikhef, Amsterdam
-26-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
10 Details of the Expert User Interface (EUI) program.
Each CryoLink is equipped with a cRIO-chassis with C-modules and a RT-controller on which the
FPGA-code and the RT-program runs. The four control systems are in principle identical; the only real
difference is their individual names and network addresses. In contrary to the cRIO-systems there is
only one laptop on which only one EUI application may be running. This application has a pushbutton ‘Connect’ on its panel; when not connected, clicking this button results in a popup-window
that shows a list of the four Virgo CryoLink systems and (for testing) a system ‘CryoLink-TEST’. By
clicking on one of the names the expert (for whom this EUI is meant) can select the system he or she
wants to connect to, see Figure 13.
Figure 13 - Select CryoLink system to connect
Clicking on the ‘Connect’ button when a CryoLink Controls system already is connected will end the
current connection: the Network Streams and Shared Variables are destroyed and the EUI
application is waiting for the next user input. In the “un-connected state” all controls and indicators
are disabled (greyed out), to indicate that there is no actual data to be displayed.
When the selected CryoLink is connected, the graphical interface is showing the actual values of the
CryoLink Controls system (see Figure 14 for an example).
In contrary to the functionality of the Modbus interface (all variables can be read, only three
variables can be modified) the EUI allows the expert to alter all writeable variables. Adjusting
parameters to un-appropriate values may result in serious malfunctioning of the controls.
The use of EUI should be restricted to people with sufficient knowledge of the CryoLink
Controls.
A.Schimmel,
Nikhef, Amsterdam
-27-
-Document1-
The CryoLinks and their Control System.
Printed: 10-2-2014 20:31
Figure 14 - Example of the EUI panel with some explanation
The RT-program keeps track of the values of a number of ReadData parameters plus timestamp
every 5 seconds for the last 24 hours in a set of ring-buffers, called: ‘Historical_Data_Cluster’. When
the EUI application makes a connection to a CryoLink Controls system, the whole
‘Historical_Data_Cluster’ of that system is copied to the EUI application; from that moment the EUI
takes over the updating of its historical data administration to keep it up-to-date. This mechanism
allows the EUI to present graphs of the stored parameters against time during the past 24H. These
graphs can be invoked by clicking on a select number of indicators on the main panel. To limit the
number of graphs to be supported, related parameters are shown in the same graph, see next table:
Read parameter
Graph
Indicator to click
TT01, TT02, TT03, TT04, TT05, TT06 (⁰C)
TTcal11, TTcal12, TTcal13,
PT01, PT02, PT03, LT01, LT02, LT03
DT01, DT02
LT01, LT02, LT03 PT02 (in %FS)
CL-temperatures
AI-values
TT01, TT02, TT03, TT04, TT05, TT06
DT01, DT02, PT01, PT02
Levels
PS PID-in, PS PID-out (0..100%)
CL PID-in, CL PID-out (0..100%)
Heater PID-in, PS PID-out (0..100%)
TT07
Level-PS PID-in/out
Level-CL PID-in/out
Heater PID-in/out
Heater Temperature
PS-level,
Level by LT02, Level by LT03, level by PT02
PS-PV, PS-PIDoutp
CL-PV, CL-PIDoutp
Heater-PV, Heater-PIDoutp
Temp after Heater
A.Schimmel,
Nikhef, Amsterdam
-28-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
Every invoked graph will appear in a separate window that may be repositioned on the screen. To
close a graph window one should click on the ‘Dismiss’ button in the window.
LabVIEW graphs are very flexible and user friendly: one can enable/disable traces (named ‘plots’ in
LabVIEW), enable/disable auto-scaling, define which of the predefined scales is used for each ‘plot’,
zoom in on a part of the graph, undo the zoom for the X- or Y-axis, define a cursor and move it
through the graph. Also the color and line-type of each ‘plot’, the representation of the data-points
and some more properties can be adjusted. And all these manipulations can be done ‘runtime’, in a
“live”, active graph.
The settings of the PID-parameters can be modified by typing a new value in a field of the PIDcontroller of choice: the new value is transferred to the RT-program, which sends it to the related
input of the FPGA where the PID-algorithm is executed.
The expert should realize that the PID-controllers for the levels in the Phase Separator and the
CryoLink can be switched <off> by the system, when one/more of the required conditions for
‘(go to) Operation’ is/are not present. When the PID-controllers are switched <off> the outputs are
set to the minimum value, which results in closure of the control-valves for the supply of liquid N2 to
the Phase Separator and the CryoLink. The setting of the valves are shown in the EUI-panel, the state
of the level PID-controllers (running or not running) is indicated by the color of the valves (green,
resp. red).
One can think of the logic for the CryoLink safety in the following way:
Figure 15 - Logic for running 'Level PID-control'
From the scheme above (Figure 15) one can see that the level PID-controllers can be ‘not running’
because of being disabled (one/more incoming interlock is/are False or the RT-program has detected
an error) or being instructed to stop (operator pressed the button ‘Stop liqN2 supply’ or has set the
boolean ‘go to Operation?’ to False (= ‘go to Bakeout-conditions’) ).
The “enabled/disabled” state is shown by the indicator ‘Level-Ctrl possible’ (light green) /
‘Level-Ctrl <disabled>’ (dark green). When the conditions are ok again, the system will not
automatically return to the ‘enabled’ state: the operator should explicitly click on the button
A.Schimmel,
Nikhef, Amsterdam
-29-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
‘Press to reset interlock error’ to return to the state ‘Level-Ctrl enabled’ (this will only succeed
when the conditions are ok).
The boolean switch ‘(going to) BAKEOUT, press to go to Operation’ / ‘(going to) OPERATION, press
to go to Bakeout conditions’ has an important function. The following table shows the resulting
actions for either of the two states (when the PID-controllers are running):
Actual state CryoLink
No liquid N2,
Temp-top > 0⁰C
No liquid N2,
Temp-top > -90⁰C
No liquid N2,
Temp-top < -90⁰C
Some liquid N2,
Temp-top > -180⁰C
Some liquid N2,
Temp-top < -180⁰C
Level liquid N2 >
1050mm
(going to) OPERATION,
press to go to Bakeout conditions
PID level-PS <auto>, setp = 50%
PID level-CL <manual>, outp= 40%
PID level-PS <auto>, setp = 50%
PID level-CL <manual>, outp= 30%
PID level-PS <auto>, setp = 50%
PID level-CL <manual>, outp= 20%
PID level-PS <auto>, setp = 50%
PID level-CL <manual>, outp= 40%
PID level-PS <auto>, setp = 50%
PID level-CL <manual>, outp= 80%
PID level-PS <auto>, setp = 50%
PID level-CL <auto>, setp = 62%
(going to) BAKEOUT,
press to go to Operation
CV01 <closed>
CV02 <closed>
CV01 <closed>
CV02 <closed>
CV01 <closed>
CV02 <closed>
CV01 <closed>
CV02 <closed>
CV01 <closed>
CV02 <closed>
CV01 <closed>
CV02 <closed>
In the state ‘(going to) BAKEOUT’, this switch sets the condition ‘run level PID’ to False, resulting in
stopped PID’s and closed valves. The liquid N2 in the Phase Separator and CryoLink will continue to
evaporate without supply and and the CryoLink finally ends in warmed up to ambient temperature.
The liquid N2 contents of the CryoLink will be evaporated in about 4 days.
With the boolean switch is TRUE = ‘(going to) OPERATION’, plus the condition ‘Level-Ctrl disabled’ =
False, the level PID-controllers will run. Depending on the actual control-state, the settings for the
two control-valves are adjusted to appropriate settings, to have an automated cooldown procedure
which finally ends in the control-state ‘Operation’. The ‘run-state’ of the level PID-controllers is
shown by the indicator ‘Level-Ctrl runs’ (light green) / ‘Level-Ctrl NOT active’ (dark green).
10.1 Details of the Expert User Interface (EUI) program code.
The EUI program consists of some code for initialization (runs once after startup) and four whileloops:
1. Event Handling Loop
2. UI Message Loop
3. Monitoring Loop
4. Simulation Loop
1. The ‘Event Handling Loop’ specifies a number of controls on the Front Panel to be monitored:
when the operator modifies a control (adjust another value), the related piece of code will be
executed. This mechanism allows “Event driven programming”: the program just waits for the
occurrence of one of the predefined events to execute the specified code.
2. The ‘UI Message Loop’ monitors a Message-queue: when a message is submitted to the queue,
this loop will extract the message, interpret the command and execute the related code. The
A.Schimmel,
Nikhef, Amsterdam
-30-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
message consists of a Cluster (equivalent to a structure in C) of two elements: ‘Command’ (type
‘enumerated’) and ‘Data’ (type ‘variant’). The content of ‘Data’ is specific for the value of
‘Command’. For details on the coded commands (see Appendix H).
3. The ‘Monitoring Loop’ monitors if the RT-program has sent a message to the’ RT Message
Stream’. If a message is received, a dedicated sub-VI interprets the message and performs the
specified action. The message structure and the mechanism to handle the command are similar
to that of the ‘UI Message Loop’. When no message is present in the ‘RT Message Stream’ the
program reads the specified ‘Shared Variables’ (the most recent values of a set parameters, set
by the RT-program) and updates the related indicators on the Front-Panel. After an update the
program waits 1 sec. and repeats the check on the ‘RT Message Stream’, etc.
4. The ‘Simulation Loop’ is implemented for testing purposes: a mechanism to “fake” the HW-IO in
software enables running the cRIO system without the sensors and actuators connected. Also
many scenarios of operation can be simulated without the need to wait for long time to get the
physical conditions on certain values (for instance, a simulation may start with a cold, half –filled
CryoLink to check if the last stage of filling is handled correctly).
A.Schimmel,
Nikhef, Amsterdam
-31-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
11 Appendices.
11.1 Appendix A. Level reference sensors
The special Pt100 sensors TTcal11, TTcal12 and TTcal1 are mounted on specific heights above the top
of the inner cylinder of the cryogenic vessel, being:
Calibr-Pt100
TTcal11
TTcal12
TTcal13
Height
5 mm
15 mm
70 mm
The RT-program monitors the ‘state’ of each sensor; in case of the state-change from ‘Cold-Dry’ to
‘Cold-Wet’, the liquid N2 level is at the height of the Pt100 that shows the change. The autocalibration procedure administrates the frequency of both oscillator units for the Pt100 that
changed. It keeps track of the last two measured frequencies at the state-change for each of the
three Pt100’s. The resulting tables (one for each sensor) look like:
Level [%]
5.56
16.7
72.2
Freq-1
9567
9478
9155
Freq-2
9570
9474
9149
After each update of the table, the procedure makes a new linear fit of ‘Height’ as function of the
frequency and the new coefficients are stored for future use.
The capacitive sensor LT03 is capable to measure the liquid N2 level over (almost) the full height of
the cold vessel. The TTcal-sensors are located at 92.2% of the full height and higher, making the
calibration for this sensor inaccurate for lower levels. To overcome this, a fourth level-value and its
frequencies are administrated. The level-value is derived for the readout of PT02, the “trigger” to
administrate a level and the related frequency is the value of PT02; when this sensor measures a
level between 10 and 15% of the full height, the calibration-data for LT03 at that level are updated.
This normally will happen during the filling of the cryogenic vessel and during the ‘boiling dry’ phase,
prior to the regeneration of a CryoLink.
11.2 Appendix B. Some LabVIEW terminology
A program in LabVIEW is often called the ‘top-level VI’, referencing a hierarchy of routines. The ‘toplevel VI’ calls other routines (called ‘sub-VI’s), which in turn call other sub-VI’s, etc.
A LabVIEW VI consists of two parts: the ‘Front Panel’, which always resembles a Graphical User
Interface (GUI) with inputs (called ‘Controls’) and outputs (called ‘Indicators’) and the ‘Block Diagram’
where the actions for execution of the VI are defined. A Front Panel may be simply a way to specify
parameters for the VI it belongs to (like arguments of a function or procedure in other programming
languages) or be more like a GUI: a graphical representation of information plus a way for the user to
interact with the program (change values and/or behavior).
A.Schimmel,
Nikhef, Amsterdam
-32-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
Figure 16 - Front Panel and Block Diagram of a simple LabVIEW VI
The controls and indicators on the Front Panel are represented as small rectangles on the Block
Diagram; a control “delivers” a value that may be handled in the code of the VI, an indicator can
”receive” a value from the code and represent it on the Front Panel. The black lines in Figure 16 mark
the relations. When executed, the value of a Control (or another “source” of data) is “flowing” along
the line of the diagram (called a ‘wire’) to the object on the other end; that object receives the value
(or values when more lines are connected) and execute, producing a new value that “flows” to the
next object, etc.
Every VI has an Icon; the programmer can edit the icon to make it representative for the VI. The icon
has a second representation, called the “Connector-pane”. This pane is divided in a number of small
rectangles, named “Terminals”, see in the right upper corner of Figure 16. To allow the use of the
code of a VI in another VI, the programmer can “link” Controls and Indicators to the Terminals on the
Connector pane, see the red lines in Figure 16, where the Controls ‘#iterations’, ‘Numeric’ and
‘Numeric 2’ and the Indicator ‘Array’ are linked. The icon of VI ‘A’ can be placed on the diagram of
another VI ‘B’, with wires connected to the Terminals of its Connector-Pane representation. When VI
‘B’ executes, the value on the wires are inputs for VI ‘A’ and the results on its output(s) flow further
on through the code of VI ‘B’.
In the more than 20 years of its existence, LabVIEW has evolved to a rich, versatile programming tool
with an enormous quantity of ready-to-use high level routines, instrument drivers (code to interface
A.Schimmel,
Nikhef, Amsterdam
-33-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
computer-controllable devices) and examples, which allows a programmer to concentrate on the
main tasks instead of many details.
11.3 Appendix C. Network Streams
National Instruments™ has developed a ‘Network Stream’ mechanism for the creation of
unidirectional point-to-point communication channels between distributed processes, based on
TCP/IP. This mechanism allows inter-process communication within the same computer as well
between separate systems with an Ethernet connection.
11.4 Appendix D. Shared Variables
National Instruments™ has developed a proprietary ‘Shared Variable’ concept as a kind of “super
global” variable. A Shared Variable can be considered as a global that is usable in separate programs,
possibly running on separate computers. The mechanism is called the”NI Publish-Subscribe Protocol”
(NI-PSP), which makes the most recent value of a “Shared Variable” available to all subscribers. One
of the computers should run the “Shared Variable Engine” (SVE) as administrative base station; every
update is sent to the SVE, which pushes this new value to every subscriber. Detailed information can
be found on < http://www.ni.com/white-paper/4679/en/>
11.5 Appendix E. Operational States
The operational state of the Phase Separator (PS) and the CryoLink (CL) is one of the following:
- @Ambient Temp
- Cooling down
- Filling liqN2
- Operation
- Boiling dry
- Warming up
- Error
- Unknown
The state ‘@Ambient Temp’ is only used for the CL (the PS has no temperature sensors).
The conditions that determine the operational state of the Phase Separator are:
1. User command ‘To Operation?’
2. Level-sensor LT01 ≥ 5%
3. Level-sensor LT01 ≥ 50%
4. Level-sensor LT01 < 80%
If condition 2 is met, there must be liquid N2 present in the PS; this distinguishes ‘Cooling down’ /
‘Warming up’ from ‘Filling liquid N2’ / ‘Boiling dry’. Condition 1 distinguishes ‘Cooling down’ / ‘Filling
liquid N2’ from ‘Boiling dry’ / ‘Warming up’. The state ‘Operation’ is selected when conditions 1, 3
and 4 are met.
The parameters that determine the operational state of the CryoLink are:
1. All temperature sensors CL > 0⁰C?
2. User command ‘To Operation?’
3. Level-sensor PT02 ≥ 5%
4. Level-sensor LT03 ≥ 5%
5. Level-sensor PT02 ≥ 92%
6. Level-sensor PT02 < 98%
7. Level-sensor PT02 ≥ 50%
A.Schimmel,
Nikhef, Amsterdam
-34-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
If condition 3 and 4 is met, there must be liquid N2 present in the CL; this distinguishes ‘Cooling
down’ / ‘Warming up’ from ‘Filling liquid N2’ / ‘Boiling dry’. Condition 1 distinguishes ‘Cooling down’
/ ‘Filling liquid N2’ from ‘Boiling dry’ / ‘Warming up’. Normally, the state ‘Operation’ is selected when
conditions 2, 5 and 6 are met, but when the incoming interlock ‘GR04 Half CL-level OK?’ is set the
logic is changed to conditions 2, 5 and 7.
11.6 Appendix F. Commands supported by the ‘RT Message Handling Loop’
The commands handled by the ‘RT Message Handling Loop’ of the RT-program are:
- Initialize
initialize the VI ‘RT_Handle_Write-DataBlock_Admin.vi’ (contains a list of actual values of all write
parameters) and let that VI retrieve the actual values from the FPGA.
- Disconnect UI
flush the ‘RT Writer Stream’, than delete it and delete the ‘UI Writer Stream’.
- Cleanup HD – remove old Logfiles
obtain a list of all logfiles on the Disk-drive of the RT-system, sorted by date; delete oldest Logfiles
until the free space on the drive exceeds the specified limit (500MB).
- Send Settings PIDloops to UI
if a EUI-application is connected: fill a structure with the actual settings of the Phase Separator
PID-controller, send it to the EUI. Repeat this for the settings of the PID-controller for the
CryoLink-level and the exhaust Heater.
Send lonely Booleans to UI
if an EUI-application is connected: send the actual values of a set of Boolean parameters (‘Reset
Interlock Error’, ‘Disable Level-Ctrl in PIDloop’, ‘Switch-to-Operation’, ‘CL-level_by_diffP?’,
‘Controls_Manual_Simulated’ and ‘Controls_SW_Simulated’) to the EUI.
- Write Error to LogFile, report to UI
report the error in ‘Data’ to the StatusLog; if a EUI is connected, send the same message to the
EUI for reporting in the StatusLog window.
- Write Error to LogFile, report to UI
report the status-message in ‘Data’ to the StatusLog; if a EUI is connected, send the same
message to the EUI for reporting in the StatusLog window.
- Write Error to LogFile, report to UI
report the alarm-message in ‘Data’ to the StatusLog; if a EUI is connected, send the same message
to the EUI for reporting in the Alarm window.
- Report Level-Safety Mess to UI
interpret the bits in Level-Safety value in ‘Data’; report the message(s) to the StatusLog; if a EUI is
connected, send the same message to the EUI for reporting in the Alarm window
- Report Non-Level-Safety Mess to UI
interpret the bits in Non-Level-Safety value in ‘Data’; report the message(s) to the StatusLog; if a
EUI is connected, send the same message to the EUI for reporting in the Alarm window.
- Wr ‘Reset Interlock Err’
write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA.
- Wr ‘Disable PID level Control’
write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA.
- Wr ‘Switch to Operation’
write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA.
- Wr ‘Switch to Operation’
A.Schimmel,
Nikhef, Amsterdam
-35-
-Document1-
Printed: 10-2-2014 20:31
-
-
-
-
-
-
-
-
-
The CryoLinks and their Control System.
write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the values to the related input of the FPGA.
Wr ‘CL-level by DiffP?’
write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the values to the related input of the FPGA.
Wr ‘PSlev PID-param’
write the cluster values specified in ‘Data’ to the appropriate entries in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related inputs of the FPGA.
Wr ‘CLlev PID-param’
write the cluster values specified in ‘Data’ to the appropriate entries in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related inputs of the FPGA.
Wr ‘Heater PID-param’
write the cluster values specified in ‘Data’ to the appropriate entries in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related inputs of the FPGA.
Wr ‘Simulated IO?’
write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA.
Wr ‘Simul_RTD[ ]’
write the array of values specified in ‘Data’ to the FPGA-control ‘Simul_RTD[ ]’, write ‘True’ to the
FPGA control ‘Simulated CL?’.
Wr ‘Simul_AIval[ ]’
write the array of values specified in ‘Data’ to the FPGA-control ‘Simul_AIval[ ]’.
Wr ‘Simul_DI7:0’
write the value specified in ‘Data’ to the FPGA-control ‘Simul_DI7:0’.
Update ReadData[ ]
sends values of all ReadData to the EUI (to update the display).
Update WriteData[ ]
sends values of all WriteData to the EUI (to update the display).
Update HistoricalData
sends values of HistoricalData (the values of a subset of the read-variables during the last 24H,
plus timestamps) to the EUI (to enable graphs over the last 24H, even when the EUI is connected
recently).
Exit
deletes the ‘Message Queue’, the ‘UI Write Stream’ and the ‘RT Writer Stream’, reports the
reception of the ‘Exit’ command to the StatusLog, writes ‘True’ to the global ‘ All RT Loop Stop’
and stops its own loop. When all other loops have ended the program reports any detected error
and finally reboots the RT-system (implying a restart of the RT-program as well).
default
reports the unsupported command as error.
11.7 Appendix G. Local Logfiles
The local logfiles of each CryoLink Control System are written on the Disk-drive of the RT-system. The
RT-system runs an ftp-server, by which the files may be copied to another computer. Because the
ftp-interface is not very user-friendly, a dedicated application is created to facilitate the copying in an
easy way.
To avoid directories containing lots of files, the storage on the RT-system is structured in the
following way:
A.Schimmel,
Nikhef, Amsterdam
-36-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
Logical structure
Logging BaseDirectory
ReadData
Year-dir
Month-dir
Files
Month-dir
Files
Example
c:\logs\CryoLink_Data
ReadData
Year_2014
Mon_01
Log2014Jan01.txt
Log2014Jan02.txt
Mon_02
Log2014Feb01.txt
Log2014Feb02.txt
WriteData
Year-dir
WriteData
Year_2014
Mon_01
Log2014Jan01.txt
Log2014Jan02.txt
Month-dir
Files
StatusLog
Year-dir
StatusLog
Year_2014
Mon_01
Log2014Jan01.txt
Log2014Jan02.txt
Month-dir
Files
11.8 Appendix H. Commands supported by the ‘UI Message Loop’
The commands handled by the ‘UI Message Loop’ of the EUI-program are:
Initialize
Clear the name and IP-address of the CryoLink on the Front-Panel, send the command ‘Disable FPcontrols, -indicators’ to the Message-Queue.
- Disable FP-controls, -indicators
Disable and grey-out all controls and indicators with a ‘reference’, administrated in the globals
‘UI-Controls-Ref[ ]’ and ‘UI-Indicator-Ref[ ]’
- Enable FP-controls, -indicators
Enable all controls and indicators with a ‘reference’, administrated in the globals ‘UI-ControlsRef[ ]’ and ‘UI-Indicator-Ref[ ]’
- Connect
Execute the VI ‘UI-Select_CryoLink_and_IPnumber.vi’ which is configured to show its Front-Panel
when running. The operator can select a CryoLink he/she want to become connected. With the
selected IP-number, the VI ‘UI-Initiate Connection.vi’ will realize the connection by creating the
required Network Streams and Shared Variables. Finally the command ‘Enable FP-controls, indicators’ is sent to the Message-Queue and the commands ‘Update ReadData[ ]’, ‘Update
WriteData[ ]’ and ‘Update Historical Data’ are written to the ‘UI Command Stream’ to become
sent to the RT-program.
- Close Connections
The two Network Streams and the Shared Variables are destroyed and the indicator ‘Connected?’
is set to False.
-
A.Schimmel,
Nikhef, Amsterdam
-37-
-Document1-
Printed: 10-2-2014 20:31
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
The CryoLinks and their Control System.
Disconnect UI
Clear the name and IP-address of the CryoLink on the Front-Panel, set the indicator ‘Connected?’
to False and write the commands ‘Disable FP-controls, -indicators’ and ‘Close Connections’ to the
Message Queue with priority.
Send settings PIDloops to UI
If the EUI is connected: write the ‘Send settings PIDloops to UI’ to the ‘UI Command Stream’ to
become sent to the RT-program.
Send lonely Booleans to UI
If the EUI is connected: write the ‘Send lonely Booleans to UI’ to the ‘UI Command Stream’ to
become sent to the RT-program.
Write Error to Logfile, report to UI
If the EUI is connected: write the ‘Write Error to Logfile, report to UI’ to the ‘UI Command Stream’
to become sent to the RT-program.
Write Status to Logfile, report to UI
If the EUI is connected: write the ‘Write Status to Logfile, report to UI’ to the ‘UI Command
Stream’ to become sent to the RT-program.
Write Alarm to Logfile, report to UI
If the EUI is connected: write the ‘Write Alarm to Logfile, report to UI’ to the ‘UI Command
Stream’ to become sent to the RT-program.
Wr ‘Reset Interlock Err’
If the EUI is connected: write the value True for the control ‘Reset Interlock Err’ to the ‘UI
Command Stream’ to become sent to the RT-program. After a 2 sec. wait, write False in the same
way (control is a push-button).
Wr ‘Disable PID level-Ctrl’
If the EUI is connected: write the value True for the control ‘Disable PID level-Ctrl’ to the ‘UI
Command Stream’ to become sent to the RT-program. After a 2 sec. wait, write False in the same
way (control is a push-button).
Wr ‘Switch to Operation’
If the EUI is connected: write the value of the control ‘Switch to Operation’ to the ‘UI Command
Stream’ to become sent to the RT-program.
Wr ‘CL-level by diff-P?’
If the EUI is connected: write the value of the control ‘CL-level by diff-P?’ to the ‘UI Command
Stream’ to become sent to the RT-program.
Wr ‘PSlev PID-param’
If the EUI is connected: write the values of the control-cluster ‘PSlev PID-param’ to the ‘UI
Command Stream’ to become sent to the RT-program.
Wr ‘CLlev PID-param’
If the EUI is connected: write the values of the control-cluster ‘CLlev PID-param’ to the ‘UI
Command Stream’ to become sent to the RT-program.
Wr ‘Heater PID-param’
If the EUI is connected: write the values of the control-cluster ‘Heater PID-param’ to the ‘UI
Command Stream’ to become sent to the RT-program.
Wr ‘Simulated IO?’
If the EUI is connected: write the value of the control ‘Simulated IO?’ to the ‘UI Command Stream’
to become sent to the RT-program.
Exit
Write True to the global ‘All UI Loop Stop’, write command ‘Disconnect UI’ to the ‘UI Command
Stream’, than destroy the ‘UI Command Stream’ and ‘RT Message Stream’, destroy all Shared
Variables, stop the ‘Event Handling Loop’, release the Message Queue and stop the ‘UI Message
A.Schimmel,
Nikhef, Amsterdam
-38-
-Document1-
Printed: 10-2-2014 20:31
The CryoLinks and their Control System.
Loop’. When all these actions have been executed the ‘UI Main.vi’ has no active code and stops
running.
A.Schimmel,
Nikhef, Amsterdam
-39-
-Document1-