Latest Developments in Image Processing on JET by Andrea Murari1, J.Vega2, T.Craciunescu3, P.Arena4, D.Mazon5, L.Gabellieri6, M.Gelfusa7, D.Pacella6, S.Palazzo4, A.Romano6, J.F.Delmond8, A. De Maack9 , T.Lesage8 2 1 3 6 5 68 4 9 7 University of Rome “Tor Vergata” CODAS: Raw Data Total Raw data: a record of almost 35 Gbytes per shot has been reached which keeps JET increase in stored information in line with the Moore law. JET Database exceeds 100 Terabytes About 50% are images Cameras: Visualization In total more than 30 cameras operational (PIW protection). New visualization tools are indispensable for the analysis (PinUp) A new specialist is rostered in the control room: the VSO (Viewing Systems Officer) Goals of Imaging in JET Goals of imaging: o Imaging of the IR emission from the wall for portection and physics studies o Imaging of edge instabilities (ELMs, MARFEs etc) for phyics and to assess their effects on the wall. o Overview of the general discharge behaviour Issues of Imaging in JET Issues posed by the exploitation of images: o Information retrieval (discussed in detail last meeting) o Image registration (vibrations and interference) o Integration of models (see V.Martin Talk) o Real time identification of events o Extraction of quantitative information for physics studies (see T.Craciunescu Talk) Mathematical indicators • 8 different mathematical indicators for vibration detection have been investigated: • • • • • • • • Normalized cross-correlation Shannon entropy Tsallis entropy Renyi entropy Alpha entropy Shannon mutual information Tsallis mutual information Renyi mutual information Normalized crosscorrelation Entropy Mutual information Normalized cross-correlation Additive and Non additive entropy Shannon Entropy pi : probability of finding the system in each possible state i (or residual i) k : Total number of possible states (or number of possible residuals) Tsallis entropy / Sq entropy q : degree of non-additivity Equal when q 1 Applications of non additive entropy Shannon entropy is additive because it assumes that there are no correlations between the systems being added Tsallis entropy is not additive because it can take into account these correlations. Tsallis entropy is not additive. For a sum of two systems A1 and A2 Sq (A1 + A2 ) = Sq (A1) + Sq (A2) + (1-q) Sq (A1) Sq (A2) Tsallis entropy is finding many applications from statistical mechanics to signal processing, image processing etc Application of Tsallis entropy to image registration In the case of camera movements, the difference between two frames presents long range correlations These long range correlations, which are less pronounced, in case of objects moving in the still field of view of a camera, can be emphasised by the proper selection of q in the Tsallis entropy. Tsallis Entropy: higher sensitivity Background Matrix Object Matrix q=0.1 Red: Tsallis entropy versus row shift Blue: Shannon entropy vs row shift Shannon Shannon Entropy Entropy :: 0 Sq entropy : 0 0.61 0.81 Sq entropy : 3.16 3.99 Shannon Entropy : +0.23 Sq entropy : +0.83 Mutual information Renyi definition Image registration: diagnostic The Wide Angle Camera KL7 provides a view of the main vessel in the IR •The Camera seats at the end of and endoscope with many optical components whose position is not monitored • No reliable reference points in the field of view Statistics of frames observed in JET • A database of 69 videos and almost 40000 frames has been analysed manually to determine the cases with movements. Plasma current between 2 and 3.5 MA Toroidal field between 1.9 and 3.4T All the major typical events are included Comparison Entropies The vertical lines indicate the period with vibrations Comparison Mutual Informations Statistics: Threshold • Method: determination of a threshold discriminating between the frames with and without movements Mouvement No mouvement Succes Rate: Overview Conclusions Frame where no movement is wrongly detected 14.84 Frame where movement is wrongly detected 3.78 Threshold % of good results Normalized cross-correlation 0.94 71.66 Shannon entropy 1.6 84.17 15.35 0.48 Shannon mutual information 0.62 78.09 0.47 21.44 25 86.19 6.66 7.15 0.58 79.98 0.48 19.54 8 84.70 15.14 0.16 1.28 79.80 2.58 17.62 Tsallis entropy Tsallis mutual information Renyi entropy Renyi mutual information • The result is that entropy of Tsallis is the best among the other entropies. • The mutual information with Tsallis definition is the best definitions among from the definition of mutual information and NCC. Success Rate: missed and false alarms Tsallis entropy analysis False alarms 6,66% 7,15% Missed alarms Correct analysis Frame where no mouvement is wrongly detected Frame where mouvement is wrongly detected 86,19% Succesfull identifications Registration: Method Comparison • A synthetic videos has been shifted by 10 rows and then two of the best indicators have been tried to register it. Shift Application to video 73851, frame 786 • Frame 786 is chosen among frames with vibrations. The result of the Tsallis mutual information, which is shown below, is the matrix must be shift by two rows leftwards. Verification • Subtraction of the frame affected by the movement and the reference frame before and after the registration shows a clear improvement. More effective in the main chamber because the divertor is affected by ELMs Mean(value of pixel)=1.3864 Mean(value of pixel)=1.2788 Image Analysis: Hot spot detection The white areas represent the potential hot regions, parts of the wall which reach a to high temperature. 11,300 frames have been analysed manually • Infrared Wide Angle View: Size of IR images: 496x560 pixels A C++ algorithm to be run on a serial machine has been developed to automatically identify the hot spots (100% success rate in terms of image processing not physics) Assumption: the temperature map provided is correct Reference serial algorithm: computational time • For traditional serial algorithms, the computational time depends on the content of the image. A potential problem for real time applications Computational time versus number of white pixels Computational time evolution during a discharge Cellular Nonlinear Networks • CNNs are a new computational paradigm. If supported by an adequate memory they have the same computational power of Universal Turing machines but with the benefit of parallelism. • Array of cells – Information for each cell: • State (mapped to greyscale value) • Input • Output (dependent on state) – Each cell is connected to a set of neighbours (usually belonging to a 3x3 square) – A state equation defines the time evolution of the cell: xij xij Ai, j; k , l ykl Bi, j; k, l ukl zij C ( k ,l )Sr ( i , j ) C ( k ,l )Sr ( i , j ) where xij is the state of the cell, ykl the output and ukl the input. Cellular Nonlinear Networks xij xij Ai, j; k , l y C ( k ,l )Sr ( i , j ) kl Bi, j; k, l u C ( k ,l )Sr ( i , j ) kl zij • A, B: feedback and input synaptic operators – They define how the state evolves and how neighbour cells influence it. – For image processing, they define the kind of filter implemented by the CNN, and are usually 3x3 matrices a-1,-1 a-1,0 a-1,1 yi-1,j-1 yi-1,j yi-1,j+1 a0,-1 a0,0 a0, 1 yi,j-1 yi,j yi,j+1 a1,-1 a1,0 a1,1 yi+1,j-1 yi+1,j yi+1,j+1 • zij is a bias constant. Summation of dot products • The set (A, B, z) is called a template. Nonlinear (morphological) operators can be implemented 1. Directed Growing Shadow • This template create “shadows” from white pixels by increasing the objects. The template was customized so that the main direction of growth is horizontal. This template allows merging small close regions – this corresponds to the clustering operation of the serial algorithm. To be classified as hot spot To be eliminated 2. ConcaveFiller • The ConcaveFiller template is applied in order to avoid that the following shrinking phase might separate the regions unified by DirectedGrowingShadow. S.Palazzo, A.Murari et al REVIEW OF SCIENTIFIC INSTRUMENTS 81, 083505 2010 3. Object Decreasing Object Decreasing is applied in order to• Object Removal allows rescale the objects back to their original size, to remove “small objects” while keeping the merge regions united. How to implement different processing algorithms to different parts of the images? Space-varying CNNs • The implementation approach is based on the definitions of regions in the input image. • The image is divided into a grid of rectangular cells (regions), by specifying the coordinates of the grid’s rows and columns. • Each region is then assigned its own sequence of templates, which can differ from other regions in terms of number of templates to be applied, number of iterations or templates’ coefficients. Mathematics already developed. • The total computation time will depend on the longest template sequence among all regions. CNN implementation on FPGA Core array architecture A core takes as input a stripe of the image (or the output of the upper-row core) and computes the next iteration. All cores in a column process the same part of the image. All cores in a row execute the same iteration (on different input stripes). Parallelism is provided by adding columns to the array – that is, by dividing the image into more parts, to be independently processed. Hot spot detection • The new algorithm divides the image into different number of regions on which it is possible to: – Apply customized temperature thresholds, for example a higher one in the bottom-left divertor’s region. – Apply region-specific template sequences, in order to improve the global detection accuracy. Deterministic computational time Implementation with FPGA using cores Total computation time with a 100 MHz clock and 1 column of cores: 106 ∙10 ns = 10 ms → Maximum frame rate: 100 fps It is possible to increase the frame rate by adding parallelism, i.e. more columns in the core array architecture. With a 10-column core array, the computation time is reduced to 1 ms, and the maximum input frame rate becomes 1000 fps. Conclusions o Bidimensional measurements are the new frontier in plasma physics (they are a step forward comparable to profiles) oVideos contain a wealth of information which can give a very significant contribution to both the understanding of the physics and the real time control of fusion plasmas (including protection) o Image manipulation: many tools are on the market but they are not always exactly what is needed and therefore significant level of development is required Apha entropy Tsallis definition Results • The figure below shows the Iα entropy. This entropy does not provide coherent and understable results so it will not be used in the following.