Sequential Analysis of Medical Images
Using Neural Networks
Marek Czeleń, Grzegorz Wojtas, Zbigniew Mikrut1
Institute of Automatics, University of Mining and Metallurgy
Mickiewicza 30, 30-059 Kraków, Poland
e-mail: zibi@biocyb.ia.agh.edu.pl
Abstract. A problem frequently encountered in the analysis of medical images is reaching
a correct and reliable diagnosis of the examined organ, particularly by persons with little
experience, e.g. first year medical students. That fact has become an inspiration for construction of an assisting program, based on application of three neural networks: 1) moving
the attention point along the selected object, 2) elaborating the diagnosis for the area in the
current viewport, 3) reporting the end of the examination.
In the present work a summary is done for almost a hundred of experiments, testing the efficiency of all those neural networks in the analysis of raw medical images. The final version of the algorithm uses backpropagation type networks: it operates on square areas
which are 31x31 pixels in size for the first and third networks and 91x91 pixels in size for
the second (diagnosing) network. The achieved level of correct recognition (about 80%) is
a very good result, if one takes into account rather low quality (dynamic range) of the analysed images and the observed natural interferences.
1
Introduction
With the increasing presence of the equipment for digital recording and visualisation of medical images more and more needs are expressed, concerning not only
the quality increase of the registered images but also their intelligent analysis.
Such systems are narrowly specialised applications, the role of which is to suggest
the answer, leaving the final decision to the human staff. Most of such applications performs an overall analysis of the selected object, applying various preprocessing methods, segmentation, detection and classification of preselected features, both by neural [3, 8] and non-neural [5] methods.
An interesting alternative is provided by a sequential object analysis, for which
a distant and unequal led ideal is the human vision system. It seems the neural
networks are one of the few possibilities for realisation of that technique [6, 1].
This concerns both implementation of moving the so called "attention point" and
the analysis (diagnosing) of the currently "seen" image window.
In order to preserve a consistent approach to the problem for the present study
it was decided that raw data or slightly preprocessed medical image data will be
1
The work was partially supported by the Polish Committee for Scientific Research under
grant No. T11E 007 17.
used as input. It reflects the authors' belief that an adequately complicated neural
network will be able to extract the interesting image areas by itself. Other concepts
of recognition based on raw images can be found in papers [7, 2, 4].
The material selected for the study consisted of ERCP1 X-ray images, registered for several patients (see Fig. 3). The task for the created application was a
preliminary examination of the pancreatic duct area preselected by the human operator. During the pancreas area tracking the application should be able to provide
a current diagnosis, looking for the following pathological changes:
 branches characteristic for the acute pancreas inflammation
 swellings and constrictions characteristic for the pancreas cancer
 presence of cysts.
2
Methods and Experiments
In order to select an appropriate form of the input data and the neural network
structures, several experiments have been carried out, during which the networks
were learned and their structure was being adjusted. Therefore it is not possible to
strictly divide the description of the study into the classical stages, i.e. design of
the input data form, choice of the network structure and the experiments themselves. The concepts presented below are results from positively completed tests.
2.1
Task Distribution
The artificial neural networks operating on raw images are able to perform well in
only one learned task. That is a limitation for the sequential image analysis procedure, because for a complete analysis of the pancreatic duct image in each step it
is necessary to determine the motion direction, elaborate the local diagnosis and
possibly evaluate the condition for the algorithm termination. Therefore it has
been decided that three independent networks will be used:
1) the first network determined the motion direction for the "focus of attention"
point using a small (31x31 pixels) area of the image, what was equivalent to determination of a new actual position
2) the second network elaborated the local diagnosis at the actual position using a
bigger image area than the previous network (91x91 pixels), which was reduced to the area 31x31 pixels in size
3) the third network, operating on the same image area as the diagnosing network,
determined whether any part of the pancreatic duct can be seen. It was one of
the conditions required for program termination, in the case when all the objects has been analysed or in a situation when the first network lost the tracked
object.
1
Endoscopic Retrograde CholangioPancreatography
2.2
Construction of the Learning Sequences
During the construction of the learning sequences the main objective was to ensure that they are properly representative. For the network setting the motion direction the images and the respective motion directions have been selected in such
a way that the direction distribution was reasonably uniform or the forward motion
was slightly preferred. Obviously the images should be consistent (noncontradicting) what has been finally ensured by proposing a concept of "virtual
vehicle" (see sect. 2.4).
Another problem was the necessity of making a decision concerning the choice
of the algorithm used for determination of the motion direction for the "narrow"
and "wide" objects (see Fig. 3a and 3b respectively). When the pancreatic duct is
narrow, then bright areas can be seen on both sides. On the other hand when it is
wide, then the bright area can be detected on one side only. Frequently other objects are visible in the image window, which can interfere in the process of direction determination (e.g. in almost every image a duodenoscope tube - a device for
dosage the contrast medium - can be seen). Such situations have also been taken
into account during the construction of the learning sequence.
It was also ensured that the numbers of images representing particular diseases
(pathological conditions) were approximately the same. The images have been
registered at different angles, so that after the learning stage the network would be
able to carry out a proper evaluation. In the construction of the learning set for the
network responsible for analysis termination both the images containing parts of
the pancreatic duct and images without such parts have been taken into account.
The ratio of the respective images was kept close to the 1:1.
2.3
Preprocessing
The ERCP image express particularly low dynamic range (see Fig. 1 upper row).
The preliminary experiments have shown that the neural network was not able to
correct the dynamic range on its own, and - what is a consequence of the above that execution of a simple preprocessing essentially affects the obtained results.
From several examined methods of contrast enhancement based on the histogram
analysis one was selected, consisting of histogram stretching to the full brightness
range accompanied by aggregation of 10% brightest and 10% darkest pixels. Several examples of application of the above mentioned operation are shown in Fig.1.
Fig. 1. Image contrasting
using the histogram analysis
2.4
Moving the Focus of Attention
The best results have been obtained by implementing the "virtual vehicle" concept, in which the window to be analysed behaves like a small vehicle, moving
along the pancreatic duct as if it was a road. During consecutive steps the window
is turned in such a way that its centre is kept in the same location, while the lateral
edge (with respect to the motion direction) is kept parallel to the segment connecting the actual analysis window centre with its previous location (Fig.2)
previous locations
Previous location
Fig.2.The virtual vehicle
concept
For such a situation the network "sees" similarly oriented images and is able to
determine the direction of motion for the attention point with its proper orientation
("forward" direction). The rotation angle is determined using the actual and previous location of the window centre by the program. Therefore the neural network
should be able to recognise relative direction (with respect to the previous motion
direction) within the [-90,90] degrees.
The tests have not confirmed the utility of methods based on the log-polar
transform and similar concepts. It was a direct consequence of considerable information averaging near the edges of the input image area, what is an advantage
only when the tracked object is within the window. The attempts of application of
"receptor fields", detecting lines of various inclinations have lead to better results
than the transforms mentioned above, but still the results were worse then for the
pure contrast enhancement. An attempt of applying a Kohonen network, as a feature detecting network, did not work at all. Also the combining of contrasted image with the information taken from the receptor field or with the Kohonen network did not improve the correctness of the network operation (see Tab. 1).
3
Results and Discussion
The structures of all the networks tested are listed in Table 1, while several examples of analysis are presented in Fig.3.
In the cases when the pancreatic duct was located too close to other objects entering the "region of interest" of the direction determining network or when it
crossed other objects (Fig.3a) the network was not able to determine properly the
motion direction because the information about context (surrounding) was not sufficient. Possible extension of the window on one hand would increase the amount
of available information, but on the other hand it would complicate the analysis by
introducing more visible organs.
a)
b)
c)
Fig. 3. (a) Analysis of a regular pancreas; NN diagnosis: 84.6% of the pancreas length does
not exhibit any pathological changes. (b) Pancreatic duct with cysts; NN diagnosis: 38.5%
of the pancreas length exhibits cysts, 22.9% exhibits constrictions. (c) Regular pancreatic
duct; NN diagnosis: 78.1% of the pancreas length does not exhibit pathological changes.
Table 1. Summary of data representations and network architectures tested (X)
Data
representation
Log-Polar
Network structure
[type, inputs, structure]
Bp, (8x5), (100:100:2)
Bp, (12x6), (100:100:8)
Retina
Bp, (68), (15:15:5)
Bp, (130), (15:15:5)
Bp, (68), (100:100:16)
Square window Bp, (31x31), (15:15:5)
of analysis
Bp, (31x31), (100:100:2)
Bp, (31x31), (100:100:5)
Bp, (31x31), (25:25:5)
Bp, (31x31), (10:2)
Receptor fields Bp, (var), (15:15:5)
Hybrid netKh, (31x31), (5:5) + Bp, (25), (15:15:5)
work
Kh, (31x31), (5:5) + Bp, (25), (5:5:5)
Kh, (31x31), (5:5) + Bp, (31x31+5), (15:15:5)
Bp – backpropagation
Kh – Kohonen SOM
1
X
X
X
X
X
X
X
X
NN
2
3
X
X
X
X
X
X
The pancreases of regular shapes are better suited for the analysis than the
pathological ones (see Fig. 3). Particularly difficult cases are the ducts with cancer
symptoms, because the constrictions in their sections are seen as discontinuities,
what may lead to errors in the procedure of determination of motion direction.
For the diagnosing network the interference from another objects (e.g. the
spine) has lead to erroneous detection of branching. However the network used for
determination of the conditions of analysis termination correctly reported the fact
of absence of the pancreatic duct on its ends or in the case of deviation from the
proper route within the duct area. The errors were generated mainly on discontinuities, which the network was not able to distinguish from the situation when two
various objects were present.
4
Conclusions
The obtained results prove that the analysis of raw images using neural networks
is possible and effective. The diagnostic recognition level about 80% presents a
very good result, if the low quality of the ERCP images and the presence of natural interference are taken into account. The result is comparable to the 85% recognition level reported in the paper [5], whereas it should be mentioned that in the
cited work structural descriptions (grammars) have been applied to a preselected
(binary) object.
The presented method can be also applied to the analysis of medical images
containing objects of classes similar to the ERCP images, e.g. coronary vessels or
eyeball vessels. Having this in mind refinement of the whole algorithm is planned,
by introduction of feedback worked-out in the diagnosing network. The feedback's
task would include a temporary reduction of the step length between consecutive
attention shifts (at present the step length is constant) in order to provide a possibility of more detailed analysis by the recognising network in the cases which are
difficult to diagnose. That would be a next step towards the functional similarity
of the proposed algorithm to the movements of the human eyeball, executed during realisation of the diagnostic task, which have been registered in our Biocybernetic Laboratory.
References
1. van de Laar P., Heskes T., Gielen S. (1997) Task-Dependent Learning of Attention.
Neural Networks vol.10 no 6, Pergamon
2. Lindblad T., Lindsey C., Padgett M., Vanyachine A., Zanevsky I. (1998) Digital X-Ray
image processing using biologically inspired methods. Proc. IJCNN, Anchorage, Alaska.
3. Mikrut Z., Tadeusiewicz R. (2000) Neural Networks applied to digital image processing
and recognition, in: Biocybernetics and Biomedical Engineering, Vol. 6, Ed. Nałęcz M.
AOW EXIT, Warsaw (in Polish)
4. Neubauer C. (1998) Evaluation of Convolutional Neural Networks for Visual Recognition. IEEE Trans. Neural Networks, Vol. 9, No. 4
5. Ogiela M., Tadeusiewicz R. (2000) Syntactic Pattern Recognition for X-Ray Diagnosis
of Pancreatic Duct. Eng. in Medicine and Biology, IEEE Magazine, 19, 6.
6. Postma E., Herik H., Hudson P. (1997) SCAN: A Scalable Model of Attentional Selection. Neural Networks vol.10 No. 6, Pergamon
7. Sjostrom P. J., Wahlberd L.U. (1999) Automated cell recognition and counting based on
combination of artificial neural networks and standard image analysis methods. Proc. 3rd
Int. Conf. Cognitive and Neural Systems, Boston
8. Żurada J. M. (1992) Introduction to artificial neural networks. West Publ., St. Paul New
York.