The CryoLinks and their Control System A description of the mechanics, sensors and actuators and controls. Fred Schimmel, Nikhef - Amsterdam March 28, 2014 Printed: 10-2-2014 20:31 The CryoLinks and their Control System. Table of Contents 1 Introduction..................................................................................................................................... 4 2 Some mechanical aspects. .............................................................................................................. 6 2.1 The CryoLink. ........................................................................................................................... 6 2.2 The Phase Separator. .............................................................................................................. 7 2.3 The Exhaust. ............................................................................................................................ 7 3 Control concept. .............................................................................................................................. 8 4 Sensors. ......................................................................................................................................... 10 5 6 4.1 Temperature-sensors. ........................................................................................................... 10 4.2 Level-sensors. ........................................................................................................................ 11 4.3 Pressure-sensors. .................................................................................................................. 12 4.4 Displacement -sensors. ......................................................................................................... 12 Control devices. ............................................................................................................................. 13 5.1 Control valves. ....................................................................................................................... 13 5.2 Solid-State Relay. ................................................................................................................... 13 Hardware Interlocks. ..................................................................................................................... 14 6.1 Incoming interlocks. .............................................................................................................. 14 6.2 Outgoing interlocks. .............................................................................................................. 14 7 The Control system Hardware....................................................................................................... 15 8 Overview of the CryoLink Control system Software. .................................................................... 19 9 8.1 FPGA. ..................................................................................................................................... 19 8.2 RealTime system.................................................................................................................... 19 8.3 Expert User Interface. ........................................................................................................... 20 8.4 Simulation program for testing. ............................................................................................ 20 Details of the CryoLink Control system Software. ......................................................................... 22 9.1 Details of the FPGA code. ...................................................................................................... 22 9.1.1 FPGA – Pt100. ................................................................................................................ 22 9.1.2 FPGA – Analogue Input.................................................................................................. 22 9.1.3 FPGA – Digital Input Interlocks. ..................................................................................... 23 9.1.4 FPGA – Digital Input Capacitive Level sensors............................................................... 23 9.1.5 FPGA – PID-loop. ........................................................................................................... 23 9.1.6 FPGA – Watchdog-loop. ................................................................................................ 24 A.Schimmel, Nikhef, Amsterdam -2- -Document1- Printed: 10-2-2014 20:31 9.2 The CryoLinks and their Control System. Details of the RT-program code. ........................................................................................... 24 9.2.1 RT – UI Commands Loop. .............................................................................................. 25 9.2.2 RT – Message Handling Loop......................................................................................... 25 9.2.3 RT – Modbus Interface Loop. ........................................................................................ 25 9.2.4 RT – Watchdog Loop. .................................................................................................... 26 9.2.5 RT –System Health & FPGA Monitoring Loop. ............................................................. 26 9.2.6 RT – Logging Loop. ......................................................................................................... 26 10 Details of the Expert User Interface (EUI) program. ................................................................. 27 10.1 11 Details of the Expert User Interface (EUI) program code...................................................... 30 Appendices. ............................................................................................................................... 32 11.1 Appendix A. Level reference sensors .................................................................................... 32 11.2 Appendix B. Some LabVIEW terminology.............................................................................. 32 11.3 Appendix C. Network Streams .............................................................................................. 34 11.4 Appendix D. Shared Variables ............................................................................................... 34 11.5 Appendix E. Operational States............................................................................................. 34 11.6 Appendix F. Commands supported by the ‘RT Message Handling Loop’.............................. 35 11.7 Appendix G. Local Logfiles .................................................................................................... 36 11.8 Appendix H. Commands supported by the ‘UI Message Loop’ ............................................ 37 A.Schimmel, Nikhef, Amsterdam -3- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 1 Introduction. The main parts of the Virgo experiment are two, three kilometer long, round vacuum tubes of about 1m diameter, forming the arms of a huge interferometer. They are positioned perpendicular to each other and come almost together on one side in the central building. On that point the injection and detection equipment is located (see the picture on the front page for an impression). When the detector is in use, high power photon beam travels through each arm to and fro between the highly stabilized mirrors at the ends. In order to minimize scattering of the photons, which degrades the coherence of the beams of light, the quality of the vacuum has to be as good as possible. The Advanced Virgo project will upgrade the existing detector in a number of ways; a significant improvement of the vacuum is one of them. It is investigated that the main component in the residual gas is water, originating from the vacuum enclosures of the suspension systems for the end-mirrors. Trapping the water molecules will bring the end pressure to ~1E-9mBar, instead of ~1E-7mBar in the current situation. Under high vacuum conditions the residual atoms and molecules are bouncing against the inner surfaces of the vacuum vessel, without interaction with each other. To catch the water molecules before they enter the interferometer arms, two meter long cryogenic sections (-196⁰C) will be installed near the four end-mirrors. When water molecules hit the cold surface, they will stick to it and remain trapped until the section is allowed to become warm (the regeneration process). These four devices, (one on each end of the two arms) are called the CryoLinks. Nikhef has taken responsibility to build the cryogenic systems and the controls to operate them in a stable and reliable way. Figure 1 - Artist impression CryoLink A.Schimmel, Nikhef, Amsterdam -4- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. According to the naming convention of the European Gravitational Observatory (EGO), the four CryoLink systems are named: - CryoLink-WE CryoLink-WI CryoLink-NI CryoLink-NE West arm – End (opposite to Interference point) West arm – Injection (near Interference point) North arm – Injection (near Interference point) North arm – End (opposite to Interference point) Figure 2 - Simplified layout Virgo with CryoLinks A.Schimmel, Nikhef, Amsterdam -5- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 2 Some mechanical aspects. 2.1 The CryoLink. The main part of a CryoLink is the cryogenic vessel, consisting of two aluminum tubes of different diameter (1120 and 980mm) located around each other, but not circle symmetric: the inner tube’s center is 32mm shifted downward relative to the center of the outer one (see Figure 3). The outer ends are connected to each other by semi-circular plates, resulting in a closed vessel with a small distance (~26mm) between the walls at the bottom and about 90mm at the top side. Figure 3 - schematic view CryoLink (cut) When this vessel is filled with liquid Nitrogen to a level above the top of the inner tube, the whole inner tube is in contact with boiling Nitrogen, ensuring an equal temperature on the inner surface, while there is a relative big surface for the vapor bubbles to escape from the liquid. This design is meant to reduce the vibrations of the device due to the boiling of the nitrogen, to a minimum. The top part of the outer cylinder maintains a low temperature by heat conduction through the aluminum wall plus the contact with cold gaseous N2 at the inside. The CryoLink construction is an integral part of the interferometer vacuum system, as each one is mounted directly to the vacuum-vessel of an end-mirror suspension system. Therefor it is important A.Schimmel, Nikhef, Amsterdam -6- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. that the CryoLinks hardly produce noise and/or vibrations, which may deteriorate the performance of the interferometer. The design of the CryoLink is strongly dedicated to this constraint; vibrations measurements on the tested prototype show only very low levels, while the main frequencies are outside the region of interest of the Virgo experiment. The cryogenic vessel itself is hanging in the bigger stainless steel vacuum vessel. It is suspended on the outside of the vacuum vessel by strong rubber “balloons” filled with pressurized air (about 4.6 Bar). The feedthroughs are stainless steel bellows to minimize the transfer of vibrations from the cold vessel to the vacuum vessel. 2.2 The Phase Separator. The liquid N2 for the CryoLinks is supplied from big storage containers under several Bars pressure, which causes the boiling temperature of the supplied liquid to be some degrees higher than under atmospheric pressure. To obtain liquid which boils at the actual ambient pressure, Phase Separators are installed. A Phase Separator is an isolated cryogenic container in which a level of liquid N2 is maintained, that boils at ambient pressure. The liquid nitrogen, supplied from the storage container is cooled to the lower boiling temperature by “flash evaporation” of a (small) fraction of the liquid: the energy from the lowering of temperature is consumed by the evaporation of some liquid. As the evaporation heat is about 100 times higher than the specific heat of the same amount of liquid N2, the evaporation in percentage is roughly equal to the number of degrees the temperature has to decrease. The Phase Separator is mounted at such level that its outlet is positioned ~20cm above the inlet of the CryoLink. This height difference plus the level of the liquid N2 in the Phase Separator delivers the (hydrostatic) pressure over the control-valve for the CryoLink. 2.3 The Exhaust. The evaporated Nitrogen leaves the CryoLink via a tube of 100mm, which finally ends outside the building. Due to this big diameter the gas moves through the exhaust pipe with a laminar flow. The absence of turbulences reduces the noise to a minimum. As the gas enters the exhaust at -196⁰C, the exhaust will adopt a temperature much below the dew-point of the ambient air, which could cause the forming of ice and water on the outside. As the ambient temperature varies, the thickness of the ice-layer will vary as well, giving dripping water on places one does not want to have it. From practical experience it is known that preventing the forming of ice/water by isolating such a cold pipe is almost impossible. Over time moisture will inevitably enter the isolation layer, freeze and deteriorate it. Applying a vacuum isolation is fine, but expensive to build, certainly when a long exhaust is required. With this in mind it is decided to apply a heater at the beginning of the exhaust, which elevates the temperature of the Nitrogen above the dew-point of the ambient air, allowing a non-isolated pipe to be used. A.Schimmel, Nikhef, Amsterdam -7- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 3 Control concept. To ensure a smooth, optimal functioning the level of liquid N2 in a CryoLink should be kept stable at a height of several centimeters above the top of the inner wall of the cryogenic vessel. Despite two radiation there is a heat-flow to the cold vessel, causing evaporation of liquid N2. This evaporation is compensated by an adjustable flow of liquid N2 from the Phase Separator. An electro-pneumatic control valve in the connection adjusts the flow, according to the output-signal of a PID-controller 1. The controller has a wanted level (the Setpoint) and a measured level (sensor PT02) as inputs (see Figure 4, middle part). Figure 4 - overview PID-loops for controls The flow through a control valve (such as CV02) is determined by several parameters, among which the pressure drop over the valve. As described before, the hydrostatic pressure of the difference in height between the liquid level in the Phase Separator and the inlet of the CryoLink determines the pressure drop over CV02. To obtain an optimal control behavior, the pressure drop parameter should remain constant, implying that the level in the Phase Separator must be kept constant. This is achieved by another PID-controller, which adjusts the flow of liquid N2 from the Storage Tank to the Phase Separator by control valve CV01 (see Figure 4, left part). The temperature of the evaporated Nitrogen in the exhaust is controlled by the third PID-controller, (see Figure 4, right part). The temperature of the gas is measured with temperature-sensor TT07, the PID-controller adjusts the dissipated power in the heater to keep the measured value close to the Setpoint. For safety reasons the CryoLink Control system must autonomously switch to the ‘Safe State’ in case the conditions for normal operation (e.g. malfunctioning HW, incoming interlocks, operator command) are absent. The ‘Safe State’ means that the two control valves (CV01 and CV02) are closed preventing supply of liquid N2. This ‘Safe State’ is implemented by stopping the PID-controllers for the levels of the Phase Separator and the CryoLink and adjusting the valves to setting 0; the PIDcontroller for the heater in the exhaust should run always, as the evaporation continues as long as 1 Proportional-, Integral- and Differential- Controller, a widely applied control loop feedback mechanism to keep a process-variable close to a wanted value. See < http://en.wikipedia.org/wiki/PID_controller> for details. A.Schimmel, Nikhef, Amsterdam -8- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. liquid N2 is present. And as this controller keeps the temperature of the gas in the exhaust at the wanted value (e.g. 15⁰C) there is no need to put it in a Safe State. In case there is no cold N2 gas leaving the CryoLink the heater will not dissipate power, so the temperature will not exceed the setpoint value. A.Schimmel, Nikhef, Amsterdam -9- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 4 Sensors. 4.1 Temperature-sensors. On the inner surface of the cryogenic vessel, six Pt100 temperature sensors are mounted, scattered over the geometry of the construction. These sensors are connected in the four-wire configuration to allow compensation for the wire- resistance and are monitored with dedicated Pt100 readout devices. The heat dissipation in the sensors is dependent on the temperature: at 0⁰C it is 0.1mW, at the liquid N2 boiling temperature about 0.02mW. To control the dissipation in the 500W heater in the exhaust-pipe, a Pt100 sensor is mounted downstream the heater section to measure the resulting temperature of the gas. Figure 5 - Overview sensors and actuators (note the different layout) A.Schimmel, Nikhef, Amsterdam -10- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 4.2 Level-sensors. As mentioned before, for stable operation of a CryoLink and Phase Separator, a constant level of liquid Nitrogen is required. To measure the level of liquid N2 two techniques are common: measure the hydrostatic pressure, resulting from the weight of the column of liquid N2 formed by the contents of the cryogenic vessel, measure the capacity of two conductors close to each other: the difference of the dielectric constants of gaseous and liquid Nitrogen makes the capacity linear related to the height of the liquid level. Both techniques are applied. The commercially obtained Phase Separator has a capacitive level sensor. The CryoLink has a differential pressure sensor as primary sensor for the level and two capacitive sensors for redundancy reasons. One side of a differential pressure sensor (PT02, 100mBar FS) is connected to the outlet for the evaporated Nitrogen while the other side is connected to the bottom of the cryogenic vessel by a thin tube. It is proven that the heat-leak through the wall of the thin tube will cause the tube to be filled with gaseous Nitrogen. The pressure on this side relates linear to the height of the liquid Nitrogen inside the vessel. The maximum pressure follows directly from the geometric data (max. height at -196⁰C is 1078mm) and physical constants (density liquid N2 = 0.802kg/l and the gravity constant 9.81m/s^2) resulting in a pressure of 84.36mbar when the cryogenic vessel is fully filled. For the capacitive level measurement two sensors are installed: LT03, a long, bended set of aluminum strips form the capacitor for a level indication during filling and emptying. A small set of parallel strips, mounted under about 45⁰, form another capacitor that may indicate the level above the top of the inner cylinder of the cryogenic vessel with much greater precision (LT02) (see Figure 6). Figure 6- Capacitive level sensors (schematic) A.Schimmel, Nikhef, Amsterdam -11- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. To enable the calibration of the capacitive level-sensors, three Pt100 sensors are mounted on well-known positions in the level range of LT02. These Pt100’s are connected to a 24V DC supply by a 1kΩ series resistor, creating a dissipation of ~0.02W. Due to this dissipation, the temperature of the Pt100 in gaseous N2 is considerable higher than when it is immersed in liquid N2. This arrangement results in different voltage-drop over the Pt100, depending on the condition around it (see below) On ambient temperature In cold gaseous N2 In liquid N2 V = 2.5V V ~ 0.8V V ~ 0.47V When the voltage over a Pt100 sensor shifts from ~0.8V to below 0.5V, the level of the liquid N2 is at the height of the sensor. By this provision a precise check on the readout of the pressure level-sensor and the capacitive level-sensors is possible. For the readout of each capacitive level-sensor an oscillator-unit is installed, which converts the actual capacity of the sensor to a pulse-train with a frequency that is proportional to the actual sensor-capacity. The pulses are counted in software every second, which delivers the actual frequency with 1Hz precision. Based this frequency the level can be calculated; the required coefficients for the linear equation are determined by an auto-calibration facility (see Appendix A for details). The calibration data are stored in files to allow the control program to retrieve the most recent data after (re-)booting of the control system. 4.3 Pressure-sensors. A differential pressure-sensor (PT01, 1000mB FullScale) is on one side connected to the gaseous N2 outlet, the other side is open to the ambient air. As this sensor measures the over-pressure in the exhaust, a possible obstruction (e.g. forming of ice in the exhaust that narrows the diameter) may be observed. The pressure of the supplied pressurized air is monitored with an absolute pressure sensor (PT03, 10 bar FullScale). 4.4 Displacement -sensors. On the location of the two suspensions of the cryogenic vessel, LVDT’s2 (15mm range, 10m resolution) are mounted to monitor the vertical movement of the cold vessel, relative to the vacuum container. These displacement sensors will detect failure of the rubber suspension bellows and/or the control of the air pressure in the bellows. As the bellows act like a spring, the LVDT readout is to some extend a relative measure for the total weight of the cold vessel (Aluminum and liquid N2). 2 Linear Variable Differential Transformer, a device to convert position or displacement into an electrical signal, see <http://en.wikipedia.org/wiki/LVDT> for more information. A.Schimmel, Nikhef, Amsterdam -12- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 5 Control devices. 5.1 Control valves. In the supply line of the liquid N2 to the Phase Separator and in the isolated tube from the Phase Separator to the CryoLink, two cryogenic control valves are mounted to dose the amount of liquid flowing into the vessels. The valves are vacuum isolated and have an dedicated layout to reduce the heat-leak to a minimum. An electric-pneumatic controller determines the opening of the control valves by adjusting the pressure under a spring loaded membrane proportional to the electrical control signal. 5.2 Solid-State Relay. The quantity of power, dissipatedby the heater of the exhaust pipe, is controlled by a ‘Solid-State Relay’ (SSR) module. This electronic device can switch on-off a substantial electrical power with a repetition rate up to 50Hz. By adjusting the ‘on’ period relative to the ‘cycle-time’ (= on+off period), the effective power is can be varied from 0% to 100%. A.Schimmel, Nikhef, Amsterdam -13- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 6 Hardware Interlocks. 6.1 Incoming interlocks. It has been agreed that the CryoLink Controls will monitor a number of incoming interlocks and react on them in a predictable way. To allow additional functionality in the future, seven incoming interlocks are wired and monitored. In a later state an additional functionality is requested. This addition takes one spare (GR04), meaning there are now three spare incoming interlocks left. The functionality is: - GR01 Vacuum OK? Isolation vacuum below 1E-7? - GR02 Vacuum Control PLC OK? PLC system vacuum functions OK? - GR03 liqN2 supply OK? The supply system for liq.N2 functions OK? - GR04 Half CL-level OK? Decrease the level- limit for ‘level not_too_low’ to 50% - GR05 Spare #5 - GR06 Spare #6 - GR07 Spare #7 6.2 Outgoing interlocks. Also a list of outgoing interlocks is defined. Five outgoing interlocks were specified, two spares are added for future extensions. The functionality is: - CL01 Controls OK? CryoLink Control system functions OK - CL02 CL in Operation? All conditions for operation are met - CL03 CL-temp not_too_high? The CryoLink temperature is below -193⁰C - CL04 CL-level not_too_high? The liquid N2 level in the CryoLink is below the upper limit - CL05 CL-level not_too_low? The liquid N2 level in the CryoLink above the lower limit *) - CL06 Spare #6 - CL07 Spare #7 *) the value of the lower-limit is dependent on the incoming interlock ‘GR4 Half CL-level OK?’. When not set (<5V on input) the lower level limit is 5mm above the top of the inner cylinder of the cold vessel. When set (>10V on input) a level of more than 50% of the height of the cold vessel is enough to signal ‘CL05 CL-level not_too_low?’ = OK. A.Schimmel, Nikhef, Amsterdam -14- -Document1- The CryoLinks and their Control System. Printed: 10-2-2014 20:31 7 The Control system Hardware. To perform all control functions for the operation of a CryoLink (readout of all sensors, steering of the control-valves, implementation of the PID-controllers and handling of the interlocks) a National Instruments™ cRIO (=compact Reconfigurable Input and Output) system is selected. The cRIO system consists of a cRIO-9112 8-slot chassis with a build-in Virtex-5 LX 30 FPGA 3and a cRIO-9014 Real-Time PowerPC (RT-PC), running a real-time OS4 (VxWorks). The RT-PC has 128MB volatile memory and a 2GB SSD (solid state disk), a 10/100MB Ethernet interface and an USB 2.0 connector for optional coupling of an external storage device. The chassis slots can be populated with C-series I/O modules; nowadays many models are available for all kind of interfacing. For the controls of the CryoLinks the following modules are selected: NI-9217 100Ω RTD5 module, delivers excitation and digitizes 4 resistances with a 24bit ADC6, NI-9205 16ch diff. Analogue Input module, digitizes analogue input signals with a 16bit ADC, up to 250kS/s (kiloSamples per second), NI-9265 4ch Analogue Output 0…20mA current output module, applies a 16bit DAC7, up to 100kS/s, NI-9375 16ch Digital Input + 16ch Digital Output, sinks resp. sources 24V signals. Figure 7 - cRIO-chassis with the selected C-modules (connectors not mounted) 3 Field Programmable Gate Array, see < http://en.wikipedia.org/wiki/Field-programmable_gate_array>. 4 Operating System, a collection of software modules required to run programs on a computer. 5 Resistance Thermometer Detector, a temperature sensor based on measuring resistance of a material with a well-known R-T relation. 6 Analogue to Digital Converter, electronic device that converts an analogue electrical value to a digital value. 7 Digital to Analogue Converter, electronic device that converts a digital value to an analogue electrical value. A.Schimmel, Nikhef, Amsterdam -15- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. The inserted modules are directly connected to the FPGA to allow high speed interfacing of the readout and control functions, see Figure 8. The programmer can define a number of variables which “cross” the border between the FPGA and the RT-system by enabling writing and reading of variable-registers in the FPGA-code. Figure 8 - Schematic view FPGA-IO The control system is designed to provide a maximal reliability, i.e. the critical functionality (digitalization of the signals, the PID-controllers, the steering the Control-valves and the heater and handling of the interlocks) is running in the FPGA and will fail only when the power is cut or when a hardware failure occurs. In fact, the essential part of the CryoLink Control system is functioning when the FPGA runs and the C-series modules have no error, even if the RT-program crashes. The less critical functions, like logging of the data, running a Modbus8 interface for the connection to the Tango9 control system and the connection of a User-Interface (UI), are handled by the Real-Time system. Both the FPGA-code and the RealTime-code start running automatically after power-up of the cRIO system, independent of a User-Interface being connected or not. 8 Modbus is a serial communication protocol, introduced in 1979 for communication with Programmable Logic Controllers. Nowadays a TCP/IP implementation is available, which is applied. For details, see: <http://en.wikipedia.org/wiki/Modbus> 9 Tango is a free open source device-oriented controls toolkit for controlling any kind of hardware or software and building SCADA systems. It is used for controlling synchrotrons, lasers, physics experiments in over 20 sites. It is being actively developed by a consortium of research institutes. For details, see: <http://en.wikipedia.org/wiki/Tango> A.Schimmel, Nikhef, Amsterdam -16- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. To enable visualization of the control system and to allow user interaction, an Expert User-Interface program (EUI) is created. This program may run on any computer that can make a TCP/IP connection with the cRIO-system. The UI Front-Panel shows a schematic view of the CryoLink system with actual values of sensors and settings. This interface is exclusively meant for diagnosis and adjusting and/or tuning of parameters by an expert. Figure 9 schematically shows the relations between FPGA, RT-program and EUI. Figure 9 - Schematic overview Control Software The Phase Separator (PS) and all other equipment of the CryoLink Control System is assembled in a dedicated 19” cabinet that is positioned close to the CryoLink vacuum vessel, because the isolated transfer-line for the transport of liquid N2 from the Phase Separator to the CryoLink has a limited length (~1.8m). The vessel of the Phase Separator is mounted in a dedicated support inside the upper part of the cabinet; the top-cover of the cabinet is removed to accommodate the PS at the required height. At the front side the cRIO chassis with the interface modules, a very simple control panel and the 24V= power supply are mounted (see Figure 10). A.Schimmel, Nikhef, Amsterdam -17- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. On some installation locations it will be impossible to open the back-door of the cabinet. Therefor the terminals for the connection of the 230V~ power, the exhaust-heater and the incoming and outgoing interlocks are located on the front-side of the cabinet. Figure 10 - 19"cabinet with Phase-Separator A.Schimmel, Nikhef, Amsterdam -18- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 8 Overview of the CryoLink Control system Software. All software (the FPGA-code, RealTime application, as well the UI-program) is developed in a LabVIEW based program environment. LabVIEW© is a graphical programming tool, developed by National Instruments, where the instructions are pictograms and the data going in and out is represented as “wires”, similar to an electrical scheme. LabVIEW is introduced more than 20 years ago, originally as tool for technicians to create their instruments in software, in contrary to HW devices. This history explains the term VI (Virtual Instrument) which still is commonly used to indicate a piece LabVIEW code. See Appendix B for more information. 8.1 FPGA. The programming of the FPGA is done in the same way as an ordinary LabVIEW program, however with some constraints which are directly related to the way a FPGA functions. A number of LabVIEW primitives are not available (impossible to implement) or strongly discouraged because their implementation will consume a considerable part of the (limited) FPGA resources. For instance: when floating point numbers are required, it is advised to apply ‘FixedPoint’ variables instead of ‘Floats’. Implementation of arithmetic with ‘FixedPoint’ variables requires much less gates and control-lines. When the FPGA program is completed, the programmer can start a dedicated compilation tool that transforms the LabVIEW code to a FPGA ‘bitfile’ that can be loaded to the FPGA and run. The FPGA can have connections to the outside world in two ways: C-series IO-modules. A cRIO chassis has a number of slots where C-series modules of the wanted functionality can be inserted in to connect sensors, switches, actuators, etc. to the cRIO system. Software Controls (inputs for data to the FPGA) and Indicators (outputs for data from the FPGA) together can be defined as an interface between the FPGA code and the application running in the real-time system (see Figure 9). The program running in the RT-system (from now indicated as ‘RT-program’) can send values to the controls and read values from the controls and indicators defined in the FPGA-code. The update rate is determined by the RT-program as the readout is done by polling (see Figure 9, lower part). 8.2 RealTime system. In the design of the RT program special care is taken to prevent blocking of the program execution in case an action takes some time. The functions to be performed are logically split and implemented in separate while-loops which execute totally independent of each other, like threads in modern computers. As the RT system should be able to run “headless” (no display, no keyboard), its design is different from an ordinary program. Errors should not stop the RT-program but should be logged and handled in a smart way to ensure continuation of the controls. Popup panels have no use, there is no operator to look at and act on it. A watchdog mechanism is implemented to monitor the “health” of A.Schimmel, Nikhef, Amsterdam -19- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. the communication between the RT-program and the FPGA. In case of a timeout the RT-system will be rebooted after which the RT-program will restart automatically. Once installed on the EGO-site, the CryoLinks must be integrated in the Tango system for ‘Slow Control’. The Tango communication is based on Modbus over TCP/IP and is capable to communicate with ‘non-Tango’ Modbus Slaves. Each CryoLink Control system contains a Modbus Slave process which holds the actual values of all relevant parameters. The Tango system can poll the Modbus Slaves to retrieve the actual state of each of the CryoLinks . Beside this readout, three commands from the Tango system are accepted and executed by the CryoLink Controls. These commands are: Reset Interlock errors? Go to Operation? Stop liqN2 Supply? This mechanism is dedicated to remote monitoring and very simple remote control from a central control room. 8.3 Expert User Interface. For dedicated local access to a CryoLink Control system by an expert (for troubleshooting, updates, tuning of the PID-controller parameters, etc.) the Expert User Interface (EUI) program can be used. This expert program will be available on a dedicated laptop; when connected to the Virgo computer network infrastructure, the expert can command the EUI program to setup some dedicated communication channels with the RT-program of a CryoLink, enabling ‘live control’ of the system it is connected to. A special while-loop in the RT-program polls if a EUI-program tries to establish a connection. When such an attempt is detected, two “Network Streams” (point-to-point SW-communication channels using a proprietary protocol, see Appendix C for details) are configured, one for receiving commands/data, the other for sending replies/data. In the “connected” state this loop functions as communication channel with the UI-program. Received messages are checked on validity and sent to the ‘action-queue’ of a simple finite-state-machine that handles the requests. When an UIapplication is connected, the RT-system will autonomously report important information and react on requests and new settings sent by the UI via the Network-Stream mechanism. Beside the Network Streams a number of ‘Shared-Variables’ are activated. The ‘Shared-Variable’ mechanism allows a programmer to exchange data between applications in an easy, transparent way. See Appendix D for details. The EUI-program monitors the Shared Variables (only read values in case of the CryoLinks) and updates its display when new values are reported. 8.4 Simulation program for testing. Testing of the FPGA-code, the RT-program and the EUI with the real Phase Separator and CryoLink is quite hard to perform, because the processes are very slow (it takes days to cool the CryoLink to 196⁰C and >36H to lower the level from 95% to 50%). To allow testing of the software itself and the implemented algorithms and procedures, a software simulation of the CryoLink and Phase Separator is made. To connect the simulation code to the cRIO system, the readout of the input C-modules in the FPGA can be switched to reading a set of simulated values and the values, written to the output modules, are made available for the simulation program. In this way the interaction with the real HW can be switched to interaction with the simulation code that fakes the HW. See Figure 11 for a scheme of A.Schimmel, Nikhef, Amsterdam -20- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. the routing of the data. The SW-switch of the inputs and outputs is depicted as a multi-pole electrical switch. Figure 11 - Schematic view implementation of Simulation code WARNING: the remote Tango control system should check if the ‘Simulation-mode’ is <off> when the detector is in operation to prevent illegal CryoLink status information. A.Schimmel, Nikhef, Amsterdam -21- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 9 Details of the CryoLink Control system Software. The control system for a CryoLink consists of three pieces of code that run on three different hardware platforms: - the FPGA code is translated to a ‘bitfile’ that automatically is loaded into the FPGA and started, - the Real-Time code, a ‘top-level VI’ that contains the additional functionality to run the control system and interfaces. This VI is compiled and linked into an executable (from now called the ‘RT-process’) which is stored on the Disk-drive of the Real-Time computer, - the Expert User Interface (EUI), a ‘top-level VI’ that acts as graphical user interface which can connect to each of the four CryoLink Control Systems, is a LabVIEW program that runs on a laptop with a network connection. The FPGA-code and RT-process run on the hardware of the cRIO-chassis with C-modules as hardware interfaces; every CryoLink is equipped with such a chassis. The laptop is meant as a portable console for the expert. It must be connected to the Virgo LAN (Local Area Network), most likely close to the CryoLink of interest. Although the four CryoLink Control Systems will be monitored and logged by the Tango control system, every system logs its own data also locally. The ReadData and WriteData are to tabseparated ASCII logfiles. The status-information goes to an ordinary text-file. All records start with a Timestamp on a new line. These logfiles contain data of maximal 24H, starting at 00:00 or the first moment data is written; the filenames contain the date. For more details, see Appendix E. 9.1 Details of the FPGA code. 9.1.1 FPGA – Pt100. For the readout of the seven Pt100 temperature sensors two NI 9217 modules, dedicated for precise (24bits) digitalization of 4-wire 100Ω RTD’s, are applied. These modules convert the measured resistances to a many digit decimal representations of the resistances. The non-linear mapping of resistance to temperature is done in software. As the temperature of the N2 in the exhaust is required as input for the Heater PID-controller, a simple linear conversion is used in the FPGA code to obtain the gas temperature. The error is limited because the range for this conversion is small (80⁰C). All channels of the RTD-modules are read in a 1 sec. infinite loop, which is more than enough for the CryoLink Controls because the big mass of the system prohibits fast changes. The readings are quite stable which makes additional improvement of data quality unnecessary. 9.1.2 FPGA – Analogue Input. The signals of all other analogue sensors are digitized by the NI-9205 Analogue Input module. This module can digitize 32 analogue 0…10Volt input channels in ‘Single Ended’ mode or 16 analogue channels in ‘Differential’ mode. In the selected Differential mode the voltage difference between the two signal inputs of one channel is measured, independent from the common voltage swing of the signals wires. Without this precaution, the resulting digital values appeared to be rather noisy. To improve the signal quality, all channels are sampled at a 1024 Hz repetition rate and averaged every 250ms. The averaging reduces the random noise level significantly. On top of this, the data of two channels is fed to other moving average algorithms to be smoothed even more. Especially the readout of the differential pressure sensor PT02 needs additional smoothing, as the process which produces the pressure difference is somewhat unstable and this signal is used as process value for the PID-controller for the CryoLink liquid-level. The readout of LT01, representing the level of liquid A.Schimmel, Nikhef, Amsterdam -22- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. N2 in the Phase Separator, goes through an 8-value averaging, PT02 is fed to a 32-value averaging algorithm. The net result of the averaging is that the readout values are much more stable; however at the price of a certain delay for the channels with additional averaging. A step in the LT01 signal will be totally effectuated after 2 seconds, for PT02 this is 8 seconds. 9.1.3 FPGA – Digital Input Interlocks. The state of the 7 incoming HW interlocks is polled by the readout of the lower 8 Digital inputs of the NI 9375 module in an eternal loop with 1 second repetition period. In the same loop some control parameters are logically combined to determine if the CryoLink Controls should switch to or leave the ‘Safe State’. The control system will switch to ‘Safe State’ when: - one/more of the incoming interlocks GR01, GR02 or GR03 switch <off> - the watchdog mechanism that monitors the communication between the RT-program and the FPGA produces a timeout - the operator gives the software command ‘Switch PIDloops <off>’ When the conditions have become ok again, the system will not leave the ‘Safe State’ until the operator explicitly has given the command ‘Reset Interlock?’ supported by the Tango Modbusinterface. This condition is defined by EGO as requirement for the control software. 9.1.4 FPGA – Digital Input Capacitive Level sensors. The digital inputs 14 and 15 of the NI 9375 are read at the maximum repetition rate of the module, 100kHz. At every cycle is determined if the each of the inputs has changed from 1 to 0; the number of these transitions is counted and the values of the two counters is exported every second. The pulse-signals of the two oscillator units (convert the capacities of the two capacitive level sensors LT02 and LT03 to pulse-trains, connected to these two digital inputs are counted here. The measured frequencies will be converted to liquid level values by a calibration/conversion algorithm in the RTprogram. 9.1.5 FPGA – PID-loop. The PID-loop is an eternal loop which cycles at a 1Hz repetition rate. Beside the execution of the PIDcontrol algorithms and the required conversions of the input- and output-data, some other functions are implemented here for practical reasons. These are: 1. the determination of the operational state of the Phase Separator and the CryoLink 2. the writing of two channels of the NI 9265 0…20mA AO-module 3. the writing of a number of Digital Output channels of the NI 9375 DIO-module. The operational state of the Phase Separator and the CryoLink is one of the defined states, depending on the actual values of a number of parameters. The parameters and the applied algorithm for the Phase Separator are different from that for the CryoLink. For details, see Appendix F. The outputs of the two PID-controllers for the liquid N2 levels are converted to 4…20mA values in order to adjust the control valves CV01 and CV02; writing these values to channel 0 and 1 of the NI 9265 effectuate the adjustments. The output of the PID-controller for the Heater is converted to a Pulse Width Modulation (PWM) <on>/<off> signal with a 1Hz cycletime; the length of the <on> period (the duty-factor) varies between 0.0 and 1.0 sec (Ξ 0…100% power). This PWM signal is sent to Digital Output channel 8 of the NI 9375, which is connected to the control input of a power Solid State Relay (SSR); the SSR outputs a chopped 230V AC voltage and current to the heater elements on the exhaust. A.Schimmel, Nikhef, Amsterdam -23- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 9.1.6 FPGA – Watchdog-loop. The watchdog-loop is an eternal loop which cycles at maximum speed of the FPGA. Once enabled by the RT-process it reads every iteration a boolean input that is cleared after reading. It calculates the elapsed time between reading a ‘True’ and the previous reading of a ‘True’. If the elapsed time exceeds the configured ‘Watchdog timeout period’ the loop is stopped, the event is reported to the StatusLog and the RT-system is sent a signal to reboot itself. As the RT-system is configured to start the RT-process automatically after startup, the Control system will continue to run after the short (<30sec) interruption by the reboot. 9.2 Details of the RT-program code. After startup a number of initialization actions are performed, like opening the interface to the FPGA (which starts the FPGA-code if not running yet), initialization of the Logging tool, initialization of the Modbus-Interface and the creation of the ‘Message Queue’. When finished, the RT-program starts the six, independent eternal loops, described here below. Although the loops are indicated as ‘eternal’, error-conditions may cause a loop to stop cycling. Depending on the specific situation, a re-start is attempted or the loop is definitely abandoned, which is followed by forcing all loops to stop, reporting the errors and finally a reboot of the RTsystem. The boot-sequence of the RT-system is configured to start the RT-program automatically, which ensures the continuation of the controls. The RT-program has provisions to return to the conditions before the reboot: the values of the write-parameters and a set of important read-values (calibration- parameters for LT02 and LT03) are restored from dedicated files. Figure 12- LabVIEW BlockDiagram of RT-program A.Schimmel, Nikhef, Amsterdam -24- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. The Real-Time VI contains six parallel loops (see Figure 12): 1. UI Command Loop - Receive messages from UI Main VI, 2. Message Handling Loop - handle all messages coming from the UI Command Loop, itself, the Watchdog Loop, and the Monitoring Loop, 3. Modbus Loop - handle all communication with the Tango control system by Modbus messages, 4. Watchdog Loop - Send periodic 'pets' to FPGA Main VI to ensure communication is maintained, 5. System Health and FPGA Monitoring Loop - Send RT diagnostic information and FPGA read data to UI Main VI, handle setting changes from UI Main VI, 6. Handle Logging Loop - write new messages from LogBuffers to LogFiles. 9.2.1 RT – UI Commands Loop. After initialization at startup this infinite loop checks every iteration if an Expert User Interface applications attempts to make connection to the RT-program. If an attempt is detected, two Network Streams are set up: - the ‘UI Command Stream’ for messages from the EUI to the RT-program - the ‘RT Reply Stream’ for message from the RT-program to the EUI) The loop’s behavior changes to just forwarding received messages to the ‘RT – Message Handling Loop’. To do so, the received messages are appended to the ‘Message Queue’ (which is accessible in all parts of the RT-program). The ‘RT – Message Handling Loop’ monitors the queue for new messages to be handled. The message format is a LabVIEW ‘Cluster’ (which is equivalent to a ‘structure’ in C) of two elements: first the ‘enumerated’ variable ‘Command’, followed by the ‘variant’ variable ‘Data’. 9.2.2 RT – Message Handling Loop. This eternal loop polls every iteration if a message can be extracted from the ‘Message Queue’. Upon success the retrieved message is split in the enumerated ‘Command’ and the variant ‘Data’. The value of ‘Command’ is interpreted in a multi-value conditional (equivalent to the ‘switch()’ statement in C), specifying the action(s) to be executed for the supported commands. See Appendix G for a list of commands plus a short explanation. The other RT-loops can also append command-messages to the ‘Message Queue’ to let the RTprogram execute the specified ‘Command’ (possibly with ‘Data’). By outsourcing the execution itself to the ‘RT – Message Handling Loop’, the other RT-loops can quickly continue their iterations even when the required action is time-consuming. 9.2.3 RT – Modbus Interface Loop. The Modbus interface contains a list with actual values of all relevant variables of the CryoLink Control System, formatted and accessible according to the Modbus specifications. On request of the EGO system integrator, all variables in the Modbus interface are stored as U16. To obtain the values in the correct format, the following conversions are applied: bool U16 Single precision float (SGL) 0 = False, 1 = True none join 2 subsequent U16 ‘s to a U32, typecast to SGL The list of parameters and their datatypes exported by the Modbus Interface is described in a separate document (‘CryoLink_ModbusInterface_ParameterSpecs.xlsx’). A.Schimmel, Nikhef, Amsterdam -25- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 9.2.4 RT – Watchdog Loop. At startup of the RT-Watchdog Loop the Watchdog function in the FPGA is enabled. After enabling the endless RT- Watchdog Loop starts sending a ‘True’ to the ‘Pet Watchdog’ control of the FPGA every 200ms. 9.2.5 RT –System Health & FPGA Monitoring Loop. This important endless loop reads every second the values of all relevant read-variables from the FPGA. The retrieved values are fed in groups of similar kind to the VI ‘RT-Handle_ReadDataBlock_Admin.vi’, which interprets/converts and stores the values internally. Every 5 sec. the values of all read-variables are written to the ‘ReadData-Logfile’ on the local disk-drive of the RT-system. Besides that, some other functionality is executed in this loop: the auto-calibration of the capacitive level-sensors (if the conditions are present), the setting of parameters for an automated cooldown procedure and the reporting of Alarm messages. The FPGA-code checks if the various level values are in range. The failure of every check is bitwise coded in the so-called ‘Level-Safety’ word which is exported to the RT-program. In the ‘RT – System Health and FPGA Monitoring Loop’ another set of checks is executed on a number of FPGA-values. The results of these checks are coded in the so-called ‘Non-Level Safety’ word. The two ‘safety words’ are interpreted every second. Resulting Alarm messages (if any) are inserted to the Message Queue for further processing. 9.2.6 RT – Logging Loop. The RT-program applies three different logfiles: a Status-log, a ReadData-log and a WriteData-log. The logging facility contains three circular string buffers (the Log-buffers) for temporary storage of the three types of log-messages. The data to be logged is converted to strings before submitting to the appropriate Log-buffer. This eternal loop checks every second if new strings submitted to one or more Log-buffers. The administration for each buffer contains a read- and a write-pointer, indicating the index of the most recent string written to file and the index of the last submitted string. By comparing these pointers the program concludes if there is data to be written to file; after writing to file the read-pointer is set to the index of the last written string. Writing to logfiles may be done in two ways: - Fast but unsafe: the logfiles are opened at start and each write action adds its data to the file. In case of a program crash it depends on the Operating System if the file will be closed correctly. - Save but slow: the logfiles are opened and closed for each write action to prevent data-loss in case of a program crash. This buffered handling of Log-messages prevents delays in the routines which generate the Logmessages because the message just has to be copied to the appropriate Log-buffer. The writing to file is handled ‘Safe but slow’ in this independent loop where some delay of opening and closing of files is no problem. When the Logging Loop is slowed down too much (this should never happen) it may occur that the write-pointer passes the read-pointer, meaning that strings are written to buffer locations that were not written to file. The overwritten data is lost; the program detects such situation at the first iteration of the Logging Loop after overwriting has happened and reports the event plus the number of strings that were lost. A.Schimmel, Nikhef, Amsterdam -26- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 10 Details of the Expert User Interface (EUI) program. Each CryoLink is equipped with a cRIO-chassis with C-modules and a RT-controller on which the FPGA-code and the RT-program runs. The four control systems are in principle identical; the only real difference is their individual names and network addresses. In contrary to the cRIO-systems there is only one laptop on which only one EUI application may be running. This application has a pushbutton ‘Connect’ on its panel; when not connected, clicking this button results in a popup-window that shows a list of the four Virgo CryoLink systems and (for testing) a system ‘CryoLink-TEST’. By clicking on one of the names the expert (for whom this EUI is meant) can select the system he or she wants to connect to, see Figure 13. Figure 13 - Select CryoLink system to connect Clicking on the ‘Connect’ button when a CryoLink Controls system already is connected will end the current connection: the Network Streams and Shared Variables are destroyed and the EUI application is waiting for the next user input. In the “un-connected state” all controls and indicators are disabled (greyed out), to indicate that there is no actual data to be displayed. When the selected CryoLink is connected, the graphical interface is showing the actual values of the CryoLink Controls system (see Figure 14 for an example). In contrary to the functionality of the Modbus interface (all variables can be read, only three variables can be modified) the EUI allows the expert to alter all writeable variables. Adjusting parameters to un-appropriate values may result in serious malfunctioning of the controls. The use of EUI should be restricted to people with sufficient knowledge of the CryoLink Controls. A.Schimmel, Nikhef, Amsterdam -27- -Document1- The CryoLinks and their Control System. Printed: 10-2-2014 20:31 Figure 14 - Example of the EUI panel with some explanation The RT-program keeps track of the values of a number of ReadData parameters plus timestamp every 5 seconds for the last 24 hours in a set of ring-buffers, called: ‘Historical_Data_Cluster’. When the EUI application makes a connection to a CryoLink Controls system, the whole ‘Historical_Data_Cluster’ of that system is copied to the EUI application; from that moment the EUI takes over the updating of its historical data administration to keep it up-to-date. This mechanism allows the EUI to present graphs of the stored parameters against time during the past 24H. These graphs can be invoked by clicking on a select number of indicators on the main panel. To limit the number of graphs to be supported, related parameters are shown in the same graph, see next table: Read parameter Graph Indicator to click TT01, TT02, TT03, TT04, TT05, TT06 (⁰C) TTcal11, TTcal12, TTcal13, PT01, PT02, PT03, LT01, LT02, LT03 DT01, DT02 LT01, LT02, LT03 PT02 (in %FS) CL-temperatures AI-values TT01, TT02, TT03, TT04, TT05, TT06 DT01, DT02, PT01, PT02 Levels PS PID-in, PS PID-out (0..100%) CL PID-in, CL PID-out (0..100%) Heater PID-in, PS PID-out (0..100%) TT07 Level-PS PID-in/out Level-CL PID-in/out Heater PID-in/out Heater Temperature PS-level, Level by LT02, Level by LT03, level by PT02 PS-PV, PS-PIDoutp CL-PV, CL-PIDoutp Heater-PV, Heater-PIDoutp Temp after Heater A.Schimmel, Nikhef, Amsterdam -28- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. Every invoked graph will appear in a separate window that may be repositioned on the screen. To close a graph window one should click on the ‘Dismiss’ button in the window. LabVIEW graphs are very flexible and user friendly: one can enable/disable traces (named ‘plots’ in LabVIEW), enable/disable auto-scaling, define which of the predefined scales is used for each ‘plot’, zoom in on a part of the graph, undo the zoom for the X- or Y-axis, define a cursor and move it through the graph. Also the color and line-type of each ‘plot’, the representation of the data-points and some more properties can be adjusted. And all these manipulations can be done ‘runtime’, in a “live”, active graph. The settings of the PID-parameters can be modified by typing a new value in a field of the PIDcontroller of choice: the new value is transferred to the RT-program, which sends it to the related input of the FPGA where the PID-algorithm is executed. The expert should realize that the PID-controllers for the levels in the Phase Separator and the CryoLink can be switched <off> by the system, when one/more of the required conditions for ‘(go to) Operation’ is/are not present. When the PID-controllers are switched <off> the outputs are set to the minimum value, which results in closure of the control-valves for the supply of liquid N2 to the Phase Separator and the CryoLink. The setting of the valves are shown in the EUI-panel, the state of the level PID-controllers (running or not running) is indicated by the color of the valves (green, resp. red). One can think of the logic for the CryoLink safety in the following way: Figure 15 - Logic for running 'Level PID-control' From the scheme above (Figure 15) one can see that the level PID-controllers can be ‘not running’ because of being disabled (one/more incoming interlock is/are False or the RT-program has detected an error) or being instructed to stop (operator pressed the button ‘Stop liqN2 supply’ or has set the boolean ‘go to Operation?’ to False (= ‘go to Bakeout-conditions’) ). The “enabled/disabled” state is shown by the indicator ‘Level-Ctrl possible’ (light green) / ‘Level-Ctrl <disabled>’ (dark green). When the conditions are ok again, the system will not automatically return to the ‘enabled’ state: the operator should explicitly click on the button A.Schimmel, Nikhef, Amsterdam -29- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. ‘Press to reset interlock error’ to return to the state ‘Level-Ctrl enabled’ (this will only succeed when the conditions are ok). The boolean switch ‘(going to) BAKEOUT, press to go to Operation’ / ‘(going to) OPERATION, press to go to Bakeout conditions’ has an important function. The following table shows the resulting actions for either of the two states (when the PID-controllers are running): Actual state CryoLink No liquid N2, Temp-top > 0⁰C No liquid N2, Temp-top > -90⁰C No liquid N2, Temp-top < -90⁰C Some liquid N2, Temp-top > -180⁰C Some liquid N2, Temp-top < -180⁰C Level liquid N2 > 1050mm (going to) OPERATION, press to go to Bakeout conditions PID level-PS <auto>, setp = 50% PID level-CL <manual>, outp= 40% PID level-PS <auto>, setp = 50% PID level-CL <manual>, outp= 30% PID level-PS <auto>, setp = 50% PID level-CL <manual>, outp= 20% PID level-PS <auto>, setp = 50% PID level-CL <manual>, outp= 40% PID level-PS <auto>, setp = 50% PID level-CL <manual>, outp= 80% PID level-PS <auto>, setp = 50% PID level-CL <auto>, setp = 62% (going to) BAKEOUT, press to go to Operation CV01 <closed> CV02 <closed> CV01 <closed> CV02 <closed> CV01 <closed> CV02 <closed> CV01 <closed> CV02 <closed> CV01 <closed> CV02 <closed> CV01 <closed> CV02 <closed> In the state ‘(going to) BAKEOUT’, this switch sets the condition ‘run level PID’ to False, resulting in stopped PID’s and closed valves. The liquid N2 in the Phase Separator and CryoLink will continue to evaporate without supply and and the CryoLink finally ends in warmed up to ambient temperature. The liquid N2 contents of the CryoLink will be evaporated in about 4 days. With the boolean switch is TRUE = ‘(going to) OPERATION’, plus the condition ‘Level-Ctrl disabled’ = False, the level PID-controllers will run. Depending on the actual control-state, the settings for the two control-valves are adjusted to appropriate settings, to have an automated cooldown procedure which finally ends in the control-state ‘Operation’. The ‘run-state’ of the level PID-controllers is shown by the indicator ‘Level-Ctrl runs’ (light green) / ‘Level-Ctrl NOT active’ (dark green). 10.1 Details of the Expert User Interface (EUI) program code. The EUI program consists of some code for initialization (runs once after startup) and four whileloops: 1. Event Handling Loop 2. UI Message Loop 3. Monitoring Loop 4. Simulation Loop 1. The ‘Event Handling Loop’ specifies a number of controls on the Front Panel to be monitored: when the operator modifies a control (adjust another value), the related piece of code will be executed. This mechanism allows “Event driven programming”: the program just waits for the occurrence of one of the predefined events to execute the specified code. 2. The ‘UI Message Loop’ monitors a Message-queue: when a message is submitted to the queue, this loop will extract the message, interpret the command and execute the related code. The A.Schimmel, Nikhef, Amsterdam -30- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. message consists of a Cluster (equivalent to a structure in C) of two elements: ‘Command’ (type ‘enumerated’) and ‘Data’ (type ‘variant’). The content of ‘Data’ is specific for the value of ‘Command’. For details on the coded commands (see Appendix H). 3. The ‘Monitoring Loop’ monitors if the RT-program has sent a message to the’ RT Message Stream’. If a message is received, a dedicated sub-VI interprets the message and performs the specified action. The message structure and the mechanism to handle the command are similar to that of the ‘UI Message Loop’. When no message is present in the ‘RT Message Stream’ the program reads the specified ‘Shared Variables’ (the most recent values of a set parameters, set by the RT-program) and updates the related indicators on the Front-Panel. After an update the program waits 1 sec. and repeats the check on the ‘RT Message Stream’, etc. 4. The ‘Simulation Loop’ is implemented for testing purposes: a mechanism to “fake” the HW-IO in software enables running the cRIO system without the sensors and actuators connected. Also many scenarios of operation can be simulated without the need to wait for long time to get the physical conditions on certain values (for instance, a simulation may start with a cold, half –filled CryoLink to check if the last stage of filling is handled correctly). A.Schimmel, Nikhef, Amsterdam -31- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. 11 Appendices. 11.1 Appendix A. Level reference sensors The special Pt100 sensors TTcal11, TTcal12 and TTcal1 are mounted on specific heights above the top of the inner cylinder of the cryogenic vessel, being: Calibr-Pt100 TTcal11 TTcal12 TTcal13 Height 5 mm 15 mm 70 mm The RT-program monitors the ‘state’ of each sensor; in case of the state-change from ‘Cold-Dry’ to ‘Cold-Wet’, the liquid N2 level is at the height of the Pt100 that shows the change. The autocalibration procedure administrates the frequency of both oscillator units for the Pt100 that changed. It keeps track of the last two measured frequencies at the state-change for each of the three Pt100’s. The resulting tables (one for each sensor) look like: Level [%] 5.56 16.7 72.2 Freq-1 9567 9478 9155 Freq-2 9570 9474 9149 After each update of the table, the procedure makes a new linear fit of ‘Height’ as function of the frequency and the new coefficients are stored for future use. The capacitive sensor LT03 is capable to measure the liquid N2 level over (almost) the full height of the cold vessel. The TTcal-sensors are located at 92.2% of the full height and higher, making the calibration for this sensor inaccurate for lower levels. To overcome this, a fourth level-value and its frequencies are administrated. The level-value is derived for the readout of PT02, the “trigger” to administrate a level and the related frequency is the value of PT02; when this sensor measures a level between 10 and 15% of the full height, the calibration-data for LT03 at that level are updated. This normally will happen during the filling of the cryogenic vessel and during the ‘boiling dry’ phase, prior to the regeneration of a CryoLink. 11.2 Appendix B. Some LabVIEW terminology A program in LabVIEW is often called the ‘top-level VI’, referencing a hierarchy of routines. The ‘toplevel VI’ calls other routines (called ‘sub-VI’s), which in turn call other sub-VI’s, etc. A LabVIEW VI consists of two parts: the ‘Front Panel’, which always resembles a Graphical User Interface (GUI) with inputs (called ‘Controls’) and outputs (called ‘Indicators’) and the ‘Block Diagram’ where the actions for execution of the VI are defined. A Front Panel may be simply a way to specify parameters for the VI it belongs to (like arguments of a function or procedure in other programming languages) or be more like a GUI: a graphical representation of information plus a way for the user to interact with the program (change values and/or behavior). A.Schimmel, Nikhef, Amsterdam -32- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. Figure 16 - Front Panel and Block Diagram of a simple LabVIEW VI The controls and indicators on the Front Panel are represented as small rectangles on the Block Diagram; a control “delivers” a value that may be handled in the code of the VI, an indicator can ”receive” a value from the code and represent it on the Front Panel. The black lines in Figure 16 mark the relations. When executed, the value of a Control (or another “source” of data) is “flowing” along the line of the diagram (called a ‘wire’) to the object on the other end; that object receives the value (or values when more lines are connected) and execute, producing a new value that “flows” to the next object, etc. Every VI has an Icon; the programmer can edit the icon to make it representative for the VI. The icon has a second representation, called the “Connector-pane”. This pane is divided in a number of small rectangles, named “Terminals”, see in the right upper corner of Figure 16. To allow the use of the code of a VI in another VI, the programmer can “link” Controls and Indicators to the Terminals on the Connector pane, see the red lines in Figure 16, where the Controls ‘#iterations’, ‘Numeric’ and ‘Numeric 2’ and the Indicator ‘Array’ are linked. The icon of VI ‘A’ can be placed on the diagram of another VI ‘B’, with wires connected to the Terminals of its Connector-Pane representation. When VI ‘B’ executes, the value on the wires are inputs for VI ‘A’ and the results on its output(s) flow further on through the code of VI ‘B’. In the more than 20 years of its existence, LabVIEW has evolved to a rich, versatile programming tool with an enormous quantity of ready-to-use high level routines, instrument drivers (code to interface A.Schimmel, Nikhef, Amsterdam -33- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. computer-controllable devices) and examples, which allows a programmer to concentrate on the main tasks instead of many details. 11.3 Appendix C. Network Streams National Instruments™ has developed a ‘Network Stream’ mechanism for the creation of unidirectional point-to-point communication channels between distributed processes, based on TCP/IP. This mechanism allows inter-process communication within the same computer as well between separate systems with an Ethernet connection. 11.4 Appendix D. Shared Variables National Instruments™ has developed a proprietary ‘Shared Variable’ concept as a kind of “super global” variable. A Shared Variable can be considered as a global that is usable in separate programs, possibly running on separate computers. The mechanism is called the”NI Publish-Subscribe Protocol” (NI-PSP), which makes the most recent value of a “Shared Variable” available to all subscribers. One of the computers should run the “Shared Variable Engine” (SVE) as administrative base station; every update is sent to the SVE, which pushes this new value to every subscriber. Detailed information can be found on < http://www.ni.com/white-paper/4679/en/> 11.5 Appendix E. Operational States The operational state of the Phase Separator (PS) and the CryoLink (CL) is one of the following: - @Ambient Temp - Cooling down - Filling liqN2 - Operation - Boiling dry - Warming up - Error - Unknown The state ‘@Ambient Temp’ is only used for the CL (the PS has no temperature sensors). The conditions that determine the operational state of the Phase Separator are: 1. User command ‘To Operation?’ 2. Level-sensor LT01 ≥ 5% 3. Level-sensor LT01 ≥ 50% 4. Level-sensor LT01 < 80% If condition 2 is met, there must be liquid N2 present in the PS; this distinguishes ‘Cooling down’ / ‘Warming up’ from ‘Filling liquid N2’ / ‘Boiling dry’. Condition 1 distinguishes ‘Cooling down’ / ‘Filling liquid N2’ from ‘Boiling dry’ / ‘Warming up’. The state ‘Operation’ is selected when conditions 1, 3 and 4 are met. The parameters that determine the operational state of the CryoLink are: 1. All temperature sensors CL > 0⁰C? 2. User command ‘To Operation?’ 3. Level-sensor PT02 ≥ 5% 4. Level-sensor LT03 ≥ 5% 5. Level-sensor PT02 ≥ 92% 6. Level-sensor PT02 < 98% 7. Level-sensor PT02 ≥ 50% A.Schimmel, Nikhef, Amsterdam -34- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. If condition 3 and 4 is met, there must be liquid N2 present in the CL; this distinguishes ‘Cooling down’ / ‘Warming up’ from ‘Filling liquid N2’ / ‘Boiling dry’. Condition 1 distinguishes ‘Cooling down’ / ‘Filling liquid N2’ from ‘Boiling dry’ / ‘Warming up’. Normally, the state ‘Operation’ is selected when conditions 2, 5 and 6 are met, but when the incoming interlock ‘GR04 Half CL-level OK?’ is set the logic is changed to conditions 2, 5 and 7. 11.6 Appendix F. Commands supported by the ‘RT Message Handling Loop’ The commands handled by the ‘RT Message Handling Loop’ of the RT-program are: - Initialize initialize the VI ‘RT_Handle_Write-DataBlock_Admin.vi’ (contains a list of actual values of all write parameters) and let that VI retrieve the actual values from the FPGA. - Disconnect UI flush the ‘RT Writer Stream’, than delete it and delete the ‘UI Writer Stream’. - Cleanup HD – remove old Logfiles obtain a list of all logfiles on the Disk-drive of the RT-system, sorted by date; delete oldest Logfiles until the free space on the drive exceeds the specified limit (500MB). - Send Settings PIDloops to UI if a EUI-application is connected: fill a structure with the actual settings of the Phase Separator PID-controller, send it to the EUI. Repeat this for the settings of the PID-controller for the CryoLink-level and the exhaust Heater. Send lonely Booleans to UI if an EUI-application is connected: send the actual values of a set of Boolean parameters (‘Reset Interlock Error’, ‘Disable Level-Ctrl in PIDloop’, ‘Switch-to-Operation’, ‘CL-level_by_diffP?’, ‘Controls_Manual_Simulated’ and ‘Controls_SW_Simulated’) to the EUI. - Write Error to LogFile, report to UI report the error in ‘Data’ to the StatusLog; if a EUI is connected, send the same message to the EUI for reporting in the StatusLog window. - Write Error to LogFile, report to UI report the status-message in ‘Data’ to the StatusLog; if a EUI is connected, send the same message to the EUI for reporting in the StatusLog window. - Write Error to LogFile, report to UI report the alarm-message in ‘Data’ to the StatusLog; if a EUI is connected, send the same message to the EUI for reporting in the Alarm window. - Report Level-Safety Mess to UI interpret the bits in Level-Safety value in ‘Data’; report the message(s) to the StatusLog; if a EUI is connected, send the same message to the EUI for reporting in the Alarm window - Report Non-Level-Safety Mess to UI interpret the bits in Non-Level-Safety value in ‘Data’; report the message(s) to the StatusLog; if a EUI is connected, send the same message to the EUI for reporting in the Alarm window. - Wr ‘Reset Interlock Err’ write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA. - Wr ‘Disable PID level Control’ write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA. - Wr ‘Switch to Operation’ write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA. - Wr ‘Switch to Operation’ A.Schimmel, Nikhef, Amsterdam -35- -Document1- Printed: 10-2-2014 20:31 - - - - - - - - - The CryoLinks and their Control System. write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the values to the related input of the FPGA. Wr ‘CL-level by DiffP?’ write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the values to the related input of the FPGA. Wr ‘PSlev PID-param’ write the cluster values specified in ‘Data’ to the appropriate entries in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related inputs of the FPGA. Wr ‘CLlev PID-param’ write the cluster values specified in ‘Data’ to the appropriate entries in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related inputs of the FPGA. Wr ‘Heater PID-param’ write the cluster values specified in ‘Data’ to the appropriate entries in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related inputs of the FPGA. Wr ‘Simulated IO?’ write the boolean value specified in ‘Data’ to the appropriate entry in ‘RT_Handle_WriteDataBlock_Admin.vi’, which forwards the value to the related input of the FPGA. Wr ‘Simul_RTD[ ]’ write the array of values specified in ‘Data’ to the FPGA-control ‘Simul_RTD[ ]’, write ‘True’ to the FPGA control ‘Simulated CL?’. Wr ‘Simul_AIval[ ]’ write the array of values specified in ‘Data’ to the FPGA-control ‘Simul_AIval[ ]’. Wr ‘Simul_DI7:0’ write the value specified in ‘Data’ to the FPGA-control ‘Simul_DI7:0’. Update ReadData[ ] sends values of all ReadData to the EUI (to update the display). Update WriteData[ ] sends values of all WriteData to the EUI (to update the display). Update HistoricalData sends values of HistoricalData (the values of a subset of the read-variables during the last 24H, plus timestamps) to the EUI (to enable graphs over the last 24H, even when the EUI is connected recently). Exit deletes the ‘Message Queue’, the ‘UI Write Stream’ and the ‘RT Writer Stream’, reports the reception of the ‘Exit’ command to the StatusLog, writes ‘True’ to the global ‘ All RT Loop Stop’ and stops its own loop. When all other loops have ended the program reports any detected error and finally reboots the RT-system (implying a restart of the RT-program as well). default reports the unsupported command as error. 11.7 Appendix G. Local Logfiles The local logfiles of each CryoLink Control System are written on the Disk-drive of the RT-system. The RT-system runs an ftp-server, by which the files may be copied to another computer. Because the ftp-interface is not very user-friendly, a dedicated application is created to facilitate the copying in an easy way. To avoid directories containing lots of files, the storage on the RT-system is structured in the following way: A.Schimmel, Nikhef, Amsterdam -36- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. Logical structure Logging BaseDirectory ReadData Year-dir Month-dir Files Month-dir Files Example c:\logs\CryoLink_Data ReadData Year_2014 Mon_01 Log2014Jan01.txt Log2014Jan02.txt Mon_02 Log2014Feb01.txt Log2014Feb02.txt WriteData Year-dir WriteData Year_2014 Mon_01 Log2014Jan01.txt Log2014Jan02.txt Month-dir Files StatusLog Year-dir StatusLog Year_2014 Mon_01 Log2014Jan01.txt Log2014Jan02.txt Month-dir Files 11.8 Appendix H. Commands supported by the ‘UI Message Loop’ The commands handled by the ‘UI Message Loop’ of the EUI-program are: Initialize Clear the name and IP-address of the CryoLink on the Front-Panel, send the command ‘Disable FPcontrols, -indicators’ to the Message-Queue. - Disable FP-controls, -indicators Disable and grey-out all controls and indicators with a ‘reference’, administrated in the globals ‘UI-Controls-Ref[ ]’ and ‘UI-Indicator-Ref[ ]’ - Enable FP-controls, -indicators Enable all controls and indicators with a ‘reference’, administrated in the globals ‘UI-ControlsRef[ ]’ and ‘UI-Indicator-Ref[ ]’ - Connect Execute the VI ‘UI-Select_CryoLink_and_IPnumber.vi’ which is configured to show its Front-Panel when running. The operator can select a CryoLink he/she want to become connected. With the selected IP-number, the VI ‘UI-Initiate Connection.vi’ will realize the connection by creating the required Network Streams and Shared Variables. Finally the command ‘Enable FP-controls, indicators’ is sent to the Message-Queue and the commands ‘Update ReadData[ ]’, ‘Update WriteData[ ]’ and ‘Update Historical Data’ are written to the ‘UI Command Stream’ to become sent to the RT-program. - Close Connections The two Network Streams and the Shared Variables are destroyed and the indicator ‘Connected?’ is set to False. - A.Schimmel, Nikhef, Amsterdam -37- -Document1- Printed: 10-2-2014 20:31 - - - - - - - - - - - - - - - The CryoLinks and their Control System. Disconnect UI Clear the name and IP-address of the CryoLink on the Front-Panel, set the indicator ‘Connected?’ to False and write the commands ‘Disable FP-controls, -indicators’ and ‘Close Connections’ to the Message Queue with priority. Send settings PIDloops to UI If the EUI is connected: write the ‘Send settings PIDloops to UI’ to the ‘UI Command Stream’ to become sent to the RT-program. Send lonely Booleans to UI If the EUI is connected: write the ‘Send lonely Booleans to UI’ to the ‘UI Command Stream’ to become sent to the RT-program. Write Error to Logfile, report to UI If the EUI is connected: write the ‘Write Error to Logfile, report to UI’ to the ‘UI Command Stream’ to become sent to the RT-program. Write Status to Logfile, report to UI If the EUI is connected: write the ‘Write Status to Logfile, report to UI’ to the ‘UI Command Stream’ to become sent to the RT-program. Write Alarm to Logfile, report to UI If the EUI is connected: write the ‘Write Alarm to Logfile, report to UI’ to the ‘UI Command Stream’ to become sent to the RT-program. Wr ‘Reset Interlock Err’ If the EUI is connected: write the value True for the control ‘Reset Interlock Err’ to the ‘UI Command Stream’ to become sent to the RT-program. After a 2 sec. wait, write False in the same way (control is a push-button). Wr ‘Disable PID level-Ctrl’ If the EUI is connected: write the value True for the control ‘Disable PID level-Ctrl’ to the ‘UI Command Stream’ to become sent to the RT-program. After a 2 sec. wait, write False in the same way (control is a push-button). Wr ‘Switch to Operation’ If the EUI is connected: write the value of the control ‘Switch to Operation’ to the ‘UI Command Stream’ to become sent to the RT-program. Wr ‘CL-level by diff-P?’ If the EUI is connected: write the value of the control ‘CL-level by diff-P?’ to the ‘UI Command Stream’ to become sent to the RT-program. Wr ‘PSlev PID-param’ If the EUI is connected: write the values of the control-cluster ‘PSlev PID-param’ to the ‘UI Command Stream’ to become sent to the RT-program. Wr ‘CLlev PID-param’ If the EUI is connected: write the values of the control-cluster ‘CLlev PID-param’ to the ‘UI Command Stream’ to become sent to the RT-program. Wr ‘Heater PID-param’ If the EUI is connected: write the values of the control-cluster ‘Heater PID-param’ to the ‘UI Command Stream’ to become sent to the RT-program. Wr ‘Simulated IO?’ If the EUI is connected: write the value of the control ‘Simulated IO?’ to the ‘UI Command Stream’ to become sent to the RT-program. Exit Write True to the global ‘All UI Loop Stop’, write command ‘Disconnect UI’ to the ‘UI Command Stream’, than destroy the ‘UI Command Stream’ and ‘RT Message Stream’, destroy all Shared Variables, stop the ‘Event Handling Loop’, release the Message Queue and stop the ‘UI Message A.Schimmel, Nikhef, Amsterdam -38- -Document1- Printed: 10-2-2014 20:31 The CryoLinks and their Control System. Loop’. When all these actions have been executed the ‘UI Main.vi’ has no active code and stops running. A.Schimmel, Nikhef, Amsterdam -39- -Document1-