Latest Developments in Image Processing on
JET
by Andrea Murari1, J.Vega2, T.Craciunescu3, P.Arena4,
D.Mazon5, L.Gabellieri6, M.Gelfusa7, D.Pacella6, S.Palazzo4,
A.Romano6, J.F.Delmond8, A. De Maack9 , T.Lesage8
2
1
3
6
5
68
4
9
7 University of Rome
“Tor Vergata”
CODAS: Raw Data
Total Raw data: a record
of almost 35 Gbytes per
shot has been reached
which keeps JET increase
in stored information in
line with the Moore law.
JET Database exceeds
100 Terabytes
About 50% are images
Cameras: Visualization
In total more than 30
cameras operational (PIW
protection).
New visualization tools
are indispensable for
the analysis (PinUp)
A new specialist is
rostered in the control
room: the VSO (Viewing
Systems Officer)
Goals of Imaging in JET
Goals of imaging:
o Imaging of the IR emission from the wall for
portection and physics studies
o Imaging of edge instabilities (ELMs, MARFEs
etc) for phyics and to assess their effects on
the wall.
o Overview of the general discharge behaviour
Issues of Imaging in JET
Issues posed by the exploitation of images:
o Information retrieval (discussed in detail last
meeting)
o Image registration (vibrations and interference)
o Integration of models (see V.Martin Talk)
o Real time identification of events
o Extraction of quantitative information for
physics studies (see T.Craciunescu Talk)
Mathematical indicators
• 8 different mathematical indicators for
vibration detection have been investigated:
•
•
•
•
•
•
•
•
Normalized cross-correlation
Shannon entropy
Tsallis entropy
Renyi entropy
Alpha entropy
Shannon mutual information
Tsallis mutual information
Renyi mutual information
Normalized crosscorrelation
Entropy
Mutual
information
Normalized cross-correlation
Additive and Non additive entropy
Shannon Entropy
pi : probability of finding the system in
each possible state i (or residual i)
k : Total number of possible states
(or number of possible residuals)
Tsallis entropy / Sq entropy
q : degree of non-additivity
Equal when q  1
Applications of non additive entropy
Shannon entropy is additive because it assumes that there are
no correlations between the systems being added
Tsallis entropy is not additive because it can take into account
these correlations.
Tsallis entropy is not additive. For a sum of two systems
A1 and A2
Sq (A1 + A2 ) = Sq (A1) + Sq (A2) + (1-q) Sq (A1) Sq (A2)
Tsallis entropy is finding many applications from
statistical mechanics to signal processing, image
processing etc
Application of Tsallis entropy
to image registration
In the case of camera movements, the difference between
two frames presents long range correlations
These long range correlations, which are less pronounced, in
case of objects moving in the still field of view of a camera,
can be emphasised by the proper selection of q in the Tsallis
entropy.
Tsallis Entropy: higher sensitivity
Background Matrix
Object Matrix q=0.1
Red: Tsallis entropy versus row shift
Blue: Shannon entropy vs row shift
Shannon
Shannon Entropy
Entropy :: 0
Sq entropy
: 0
0.61
0.81
Sq entropy
:
3.16
3.99
Shannon Entropy :
+0.23
Sq entropy
:
+0.83
Mutual information
Renyi definition
Image registration: diagnostic
The Wide Angle Camera KL7
provides a view of the main vessel
in the IR
•The Camera seats at the end of and
endoscope with many optical components
whose position is not monitored
• No reliable reference points in the field
of view
Statistics of frames observed in JET
• A database of 69 videos and almost 40000 frames has been
analysed manually to determine the cases with movements.
Plasma current between 2 and 3.5 MA
Toroidal field between 1.9 and 3.4T
All the major typical events are included
Comparison Entropies
The vertical lines indicate the period with vibrations
Comparison Mutual Informations
Statistics: Threshold
• Method: determination of a threshold discriminating
between the frames with and without movements
Mouvement
No
mouvement
Succes Rate: Overview
Conclusions
Frame where no
movement is
wrongly
detected
14.84
Frame where
movement is
wrongly
detected
3.78
Threshold
% of good
results
Normalized cross-correlation
0.94
71.66
Shannon entropy
1.6
84.17
15.35
0.48
Shannon mutual information
0.62
78.09
0.47
21.44
25
86.19
6.66
7.15
0.58
79.98
0.48
19.54
8
84.70
15.14
0.16
1.28
79.80
2.58
17.62
Tsallis entropy
Tsallis mutual information
Renyi entropy
Renyi mutual information
• The result is that entropy of Tsallis is the best among the other
entropies.
• The mutual information with Tsallis definition is the best definitions
among from the definition of mutual information and NCC.
Success Rate: missed and false alarms
Tsallis entropy analysis
False alarms
6,66%
7,15%
Missed alarms
Correct analysis
Frame where no mouvement is wrongly detected
Frame where mouvement is wrongly detected
86,19%
Succesfull
identifications
Registration: Method Comparison
• A synthetic videos has been shifted by 10 rows and then two
of the best indicators have been tried to register it.
Shift
Application to video 73851, frame 786
• Frame 786 is chosen among frames with vibrations.
The result of the Tsallis mutual information, which is
shown below, is the matrix must be shift by two rows
leftwards.
Verification
• Subtraction of the frame affected by the movement and
the reference frame before and after the registration
shows a clear improvement. More effective in the main
chamber because the divertor is affected by ELMs
Mean(value of
pixel)=1.3864
Mean(value of
pixel)=1.2788
Image Analysis: Hot spot detection
The white areas represent the
potential hot regions, parts of the
wall which reach a to high
temperature.
11,300 frames have been analysed
manually
• Infrared Wide Angle
View: Size of IR
images: 496x560
pixels
A C++ algorithm to be run on a
serial machine has been developed
to automatically identify the hot
spots (100% success rate in terms
of image processing not physics)
Assumption: the temperature
map provided is correct
Reference serial algorithm:
computational time
• For traditional serial algorithms, the computational time
depends on the content of the image. A potential problem for
real time applications
Computational
time versus
number of
white pixels
Computational
time
evolution
during a
discharge
Cellular Nonlinear Networks
• CNNs are a new computational paradigm. If supported by an
adequate memory they have the same computational power
of Universal Turing machines but with the benefit of
parallelism.
• Array of cells
– Information for each cell:
• State (mapped to greyscale value)
• Input
• Output (dependent on state)
– Each cell is connected to a set of
neighbours (usually belonging to a 3x3
square)
– A state equation defines the time
evolution of the cell:
xij   xij 
 Ai, j; k , l ykl   Bi, j; k, l ukl  zij
C ( k ,l )Sr ( i , j )
C ( k ,l )Sr ( i , j )
where xij is the state of the cell, ykl the output and ukl the input.
Cellular Nonlinear Networks
xij   xij 
 Ai, j; k , l y
C ( k ,l )Sr ( i , j )
kl

 Bi, j; k, l u
C ( k ,l )Sr ( i , j )
kl
 zij
• A, B: feedback and input synaptic operators
– They define how the state evolves and how neighbour
cells influence it.
– For image processing, they define the kind of filter
implemented by the CNN, and are usually 3x3 matrices
a-1,-1
a-1,0
a-1,1
yi-1,j-1
yi-1,j
yi-1,j+1
a0,-1
a0,0
a0, 1
yi,j-1
yi,j
yi,j+1
a1,-1
a1,0
a1,1
yi+1,j-1
yi+1,j
yi+1,j+1
• zij is a bias constant.
Summation of dot
products
• The set (A, B, z) is called a template. Nonlinear
(morphological) operators can be implemented
1. Directed Growing Shadow
• This template create “shadows” from white pixels by
increasing the objects. The template was customized so
that the main direction of growth is horizontal.
This template allows
merging small close
regions – this corresponds
to the clustering operation
of the serial algorithm.
To be classified
as hot spot
To be eliminated
2. ConcaveFiller
• The ConcaveFiller template is applied in order to avoid
that the following shrinking phase might separate the
regions unified by DirectedGrowingShadow.
S.Palazzo, A.Murari et al
REVIEW OF SCIENTIFIC
INSTRUMENTS 81,
083505 2010
3. Object Decreasing
Object Decreasing is applied in order to• Object Removal allows
rescale the objects back to their original size, to remove “small
objects”
while keeping the merge regions united.
How to implement different processing algorithms to
different parts of the images?
Space-varying CNNs
• The implementation approach is based
on the definitions of regions in the
input image.
• The image is divided into a grid of
rectangular cells (regions), by
specifying the coordinates of the
grid’s rows and columns.
• Each region is then assigned its own
sequence of templates, which can
differ from other regions in terms of
number of templates to be applied,
number of iterations or templates’
coefficients. Mathematics already
developed.
• The total computation time will depend
on the longest template sequence
among all regions.
CNN implementation on FPGA

Core array architecture




A core takes as input a stripe of the
image (or the output of the upper-row
core) and computes the next iteration.
All cores in a column process the
same part of the image.
All cores in a row execute the same
iteration (on different input stripes).
Parallelism is provided by adding
columns to the array – that is, by
dividing the image into more parts,
to be independently processed.
Hot spot detection
• The new algorithm divides the image
into different number of regions on
which it is possible to:
– Apply customized temperature thresholds,
for example a higher one in the bottom-left
divertor’s region.
– Apply region-specific template sequences,
in order to improve the global detection
accuracy.




Deterministic computational time
Implementation with FPGA using cores
Total computation time with a 100 MHz clock and 1 column of cores:
106 ∙10 ns = 10 ms → Maximum frame rate: 100 fps
It is possible to increase the frame rate by adding parallelism, i.e. more
columns in the core array architecture. With a 10-column core array,
the computation time is reduced to 1 ms, and the maximum input frame
rate becomes 1000 fps.
Conclusions
o Bidimensional measurements are the new frontier in
plasma physics (they are a step forward comparable
to profiles)
oVideos contain a wealth of information which can
give a very significant contribution to both the
understanding of the physics and the real time
control of fusion plasmas (including protection)
o Image manipulation: many tools are on the market
but they are not always exactly what is needed and
therefore significant level of development is required
Apha entropy
Tsallis definition
Results
• The figure below shows the Iα entropy. This
entropy does not provide coherent and
understable results so it will not be used in the
following.